Skip to content

br_ms_sim - #1475

Closed
laribritto wants to merge 28 commits into
mainfrom
sim_mortalidade
Closed

br_ms_sim#1475
laribritto wants to merge 28 commits into
mainfrom
sim_mortalidade

Conversation

@laribritto

@laribritto laribritto commented Mar 24, 2026

Copy link
Copy Markdown
Collaborator

Descrição do PR:

  • Dados de Mortalidade do DataSUS atualizados p/ os anos de 2023 e 2024. Os dados foram baixados pelo pySUS.

Summary by CodeRabbit

  • New Features

    • Added mortality data processing capabilities for Brazilian health statistics, including automated validation, standardized transformation, and regional data organization.
  • Updates

    • Extended data model year range from 2022 to 2024, providing access to more recent mortality records.

@laribritto laribritto self-assigned this Mar 24, 2026
@laribritto laribritto added test-dev-model [PR] Roda testes DBT nos models modificados em basedosdados-dev check-metadata [PR] Dispara validação de metadados entre BigQuery e API de produção labels Mar 24, 2026
@laribritto
laribritto marked this pull request as ready for review March 24, 2026 13:39
@coderabbitai

coderabbitai Bot commented Mar 26, 2026

Copy link
Copy Markdown
📝 Walkthrough

Walkthrough

The PR extends the year partition range of a SQL model from 2022 to 2024 and introduces a new Jupyter notebook that orchestrates SIM mortality data ingestion, transformation, and schema normalization, culminating in partitioned CSV outputs and dictionary coverage tracking.

Changes

Cohort / File(s) Summary
SQL Configuration Update
models/br_ms_sim/br_ms_sim__microdados.sql
Updated partition_by.range end year for ano field from 2022 to 2024; no logic or source changes.
Data Processing Pipeline
models/br_ms_sim/code/datasus_sim.ipynb
New notebook that loads SIM metadata via pysus, reads local Parquet files partitioned by UF and year 2023, applies column name mapping from raw to normalized schema, validates against expected columns, creates missing fields, writes partitioned CSV output by UF code, and updates dictionary coverage flags in an architecture CSV.

Sequence Diagram(s)

sequenceDiagram
    participant Notebook as Jupyter Notebook
    participant pysus as pysus Library
    participant LocalFS as Local Parquet Files
    participant Transform as Data Transform
    participant Output as Output CSV
    participant Dict as Dictionary CSV
    participant Arch as Architecture CSV

    Notebook->>pysus: Load SIM metadata & CID10 files
    pysus-->>Notebook: Metadata loaded
    Notebook->>LocalFS: Read Parquet files (ano=2023)
    LocalFS-->>Notebook: Parquet data returned
    Notebook->>Transform: Extract UF from filenames
    Notebook->>Transform: Apply column name mapping
    Notebook->>Transform: Validate against expected columns
    Notebook->>Transform: Create missing fields & reorder
    Transform-->>Notebook: Transformed dataframe
    Notebook->>Output: Partition by sigla_uf & write CSVs
    Output-->>Notebook: CSVs written
    Notebook->>Dict: Load dictionary CSV
    Dict-->>Notebook: Dictionary metadata
    Notebook->>Arch: Update covered_by_dictionary flags
    Arch-->>Notebook: Architecture CSV updated
Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~55 minutes

Poem

🐰 A notebook hops through SIM's domain,
Parquets loaded, columns tamed,
Partitions dance by state and year,
Dictionary covers all held dear! 📊✨

🚥 Pre-merge checks | ✅ 1 | ❌ 2

❌ Failed checks (2 warnings)

Check name Status Explanation Resolution
Title check ⚠️ Warning The title 'br_ms_sim' is only a model/dataset name and lacks a required keyword prefix (e.g., [Feature], [Data], [Bugfix]) and does not clearly describe what changed in the changeset. Update the title to follow the template format: include a keyword prefix in brackets (e.g., '[Data]') followed by a clear description of the main change, such as '[Data] br_ms_sim - Update mortality data to 2023 and 2024'.
Description check ⚠️ Warning The description is incomplete; it lacks required sections such as 'Detalhes Técnicos' (pipeline changes, schema changes, performance impact), 'Teste e Validações', 'Riscos e Mitigações', and 'Dependências' specified in the template. Expand the description to include all required sections from the template: technical details, testing evidence, risk assessment with mitigation plans, and dependency information.
✅ Passed checks (1 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch sim_mortalidade

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 Ruff (0.15.7)
models/br_ms_sim/code/datasus_sim.ipynb

Unexpected end of JSON input


Comment @coderabbitai help to get the list of available commands and usage tips.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 6

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@models/br_ms_sim/code/datasus_sim.ipynb`:
- Line 84: The code is hard-coded to only process 2023 where the literal
year=[2023] appears; update those occurrences (the year parameter passed into
the data-slicing/query functions) to include 2024 or make the argument dynamic
(e.g., year=[2023, 2024] or use a variable like years = [2023, 2024] passed into
the same calls) so the 2024 slice is generated; search for the exact token
year=[2023] in the notebook and replace each instance (including the calls that
build the workflow/slices) accordingly.
- Around line 99-135: The notebook contains executed outputs (HTML/JS blobs and
sample data rows) that should be removed before merging—clear all cell outputs
in models/br_ms_sim/code/datasus_sim.ipynb (notably the output block shown
between the diff and the other large output ranges at 276-577, 793-1336, and
1405-1412), remove any embedded dataset rows and HTML/JS artifacts, and save the
notebook with outputs cleared; you can do this manually in Jupyter/Colab or run
a CI-friendly tool (e.g., nbstripout or jupyter nbconvert --clear-output) to
ensure the file committed contains only the notebook source without execution
outputs.
- Around line 53-83: The UF list passed to sim.get_files when creating sp_cid10
contains a stray entry "dados_mortalidade" instead of the Distrito Federal code;
update the uf array in the sim.get_files call (the sp_cid10 assignment) by
replacing "dados_mortalidade" with "DF" so the Distrito Federal is included and
the fetch works correctly.
- Line 1432: The cell assigns arquitetura by reading a CSV from an invalid
placeholder path; update the read to point to a real, checked-in input and add a
clear fallback/error: replace the "/[arquitetura] microdados.csv" placeholder
with the actual repository CSV filename (or a relative path inside the project)
used for this dataset, ensure the variable arquitetura is loaded via pd.read_csv
with that real filename, and optionally add a simple existence check (or
try/except around the pd.read_csv) to raise a descriptive error if the file is
still missing so the notebook fails fast and clearly.
- Around line 750-768: You're overwriting potentially valid source values by
unconditionally setting columns like codigo_bairro_ocorrencia,
codigo_bairro_residencia, crm, etc. on dados_mortalidade; instead, use the
computed colunas_faltantes set to only create/backfill those missing columns
(e.g., check membership in colunas_faltantes before assigning defaults) so
existing renamed/mapped fields are preserved; update the block that assigns ano,
codigo_bairro_ocorrencia, codigo_bairro_residencia, crm,
data_recebimento_original, sequencial_obito to conditionally assign defaults
only when the column is in colunas_faltantes.
- Around line 763-779: The exported CSV contains raw date/time tokens (e.g.
"01012023", "0610") that BigQuery safe_cast cannot parse; before writing
dados_mortalidade.to_csv() convert those columns in the notebook (before the
line that selects colunas_esperadas) to ISO date/time strings (YYYY-MM-DD for
dates, HH:MM:SS for times) using pandas operations: parse with explicit formats
(e.g., dayfirst format for DDMMYYYY), zero-pad and insert separators for times,
and overwrite fields like data_obito, hora_obito, data_nascimento,
data_atestado, data_investigacao, data_cadastro, data_recebimento,
data_recebimento_original, etc.; alternatively update the SQL model
(br_ms_sim__microdados.sql) to use PARSE_DATE/PARSE_TIME with explicit format
strings if you prefer to keep raw tokens.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 21f92ae1-4777-4897-8741-47cac60cbf39

📥 Commits

Reviewing files that changed from the base of the PR and between 8e57cb9 and f779ec9.

📒 Files selected for processing (2)
  • models/br_ms_sim/br_ms_sim__microdados.sql
  • models/br_ms_sim/code/datasus_sim.ipynb

Comment on lines +53 to +83
"sp_cid10 = sim.get_files(\n",
" \"CID10\",\n",
" uf=[\n",
" \"AC\",\n",
" \"AL\",\n",
" \"AP\",\n",
" \"AM\",\n",
" \"BA\",\n",
" \"CE\",\n",
" \"dados_mortalidade\",\n",
" \"ES\",\n",
" \"GO\",\n",
" \"MA\",\n",
" \"MT\",\n",
" \"MS\",\n",
" \"MG\",\n",
" \"PA\",\n",
" \"PB\",\n",
" \"PR\",\n",
" \"PE\",\n",
" \"PI\",\n",
" \"RJ\",\n",
" \"RN\",\n",
" \"RS\",\n",
" \"RO\",\n",
" \"RR\",\n",
" \"SC\",\n",
" \"SP\",\n",
" \"SE\",\n",
" \"TO\",\n",
" ],\n",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Replace the stray UF entry with DF.

Line 62 is not a UF code, and DF is missing from the list. On rerun this either fails the fetch or ships an incomplete extract without Distrito Federal.

🐛 Minimal fix
-        "dados_mortalidade",
+        "DF",
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
"sp_cid10 = sim.get_files(\n",
" \"CID10\",\n",
" uf=[\n",
" \"AC\",\n",
" \"AL\",\n",
" \"AP\",\n",
" \"AM\",\n",
" \"BA\",\n",
" \"CE\",\n",
" \"dados_mortalidade\",\n",
" \"ES\",\n",
" \"GO\",\n",
" \"MA\",\n",
" \"MT\",\n",
" \"MS\",\n",
" \"MG\",\n",
" \"PA\",\n",
" \"PB\",\n",
" \"PR\",\n",
" \"PE\",\n",
" \"PI\",\n",
" \"RJ\",\n",
" \"RN\",\n",
" \"RS\",\n",
" \"RO\",\n",
" \"RR\",\n",
" \"SC\",\n",
" \"SP\",\n",
" \"SE\",\n",
" \"TO\",\n",
" ],\n",
"sp_cid10 = sim.get_files(\n",
" \"CID10\",\n",
" uf=[\n",
" \"AC\",\n",
" \"AL\",\n",
" \"AP\",\n",
" \"AM\",\n",
" \"BA\",\n",
" \"CE\",\n",
" \"DF\",\n",
" \"ES\",\n",
" \"GO\",\n",
" \"MA\",\n",
" \"MT\",\n",
" \"MS\",\n",
" \"MG\",\n",
" \"PA\",\n",
" \"PB\",\n",
" \"PR\",\n",
" \"PE\",\n",
" \"PI\",\n",
" \"RJ\",\n",
" \"RN\",\n",
" \"RS\",\n",
" \"RO\",\n",
" \"RR\",\n",
" \"SC\",\n",
" \"SP\",\n",
" \"SE\",\n",
" \"TO\",\n",
" ],\n",
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 53 - 83, The UF list
passed to sim.get_files when creating sp_cid10 contains a stray entry
"dados_mortalidade" instead of the Distrito Federal code; update the uf array in
the sim.get_files call (the sp_cid10 assignment) by replacing
"dados_mortalidade" with "DF" so the Distrito Federal is included and the fetch
works correctly.

" \"SE\",\n",
" \"TO\",\n",
" ],\n",
" year=[2023],\n",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

The notebook still only processes 2023.

Lines 84, 149, 582, 763, and 1352 all pin the workflow to 2023, so the 2024 slice from the PR objective is not generated here.

Also applies to: 149-149, 582-582, 763-763, 1352-1352

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` at line 84, The code is hard-coded
to only process 2023 where the literal year=[2023] appears; update those
occurrences (the year parameter passed into the data-slicing/query functions) to
include 2024 or make the argument dynamic (e.g., year=[2023, 2024] or use a
variable like years = [2023, 2024] passed into the same calls) so the 2024 slice
is generated; search for the exact token year=[2023] in the notebook and replace
each instance (including the calls that build the workflow/slices) accordingly.

Comment on lines +99 to +135
"outputs": [
{
"data": {
"text/plain": [
"[DOAC2023.dbc,\n",
" DOAL2023.dbc,\n",
" DOAM2023.dbc,\n",
" DOAP2023.dbc,\n",
" DOBA2023.dbc,\n",
" DOCE2023.dbc,\n",
" DODF2023.dbc,\n",
" DOES2023.dbc,\n",
" DOGO2023.dbc,\n",
" DOMA2023.dbc,\n",
" DOMG2023.dbc,\n",
" DOMS2023.dbc,\n",
" DOMT2023.dbc,\n",
" DOPA2023.dbc,\n",
" DOPB2023.dbc,\n",
" DOPE2023.dbc,\n",
" DOPI2023.dbc,\n",
" DOPR2023.dbc,\n",
" DORJ2023.dbc,\n",
" DORN2023.dbc,\n",
" DORO2023.dbc,\n",
" DORR2023.dbc,\n",
" DORS2023.dbc,\n",
" DOSC2023.dbc,\n",
" DOSE2023.dbc,\n",
" DOSP2023.dbc,\n",
" DOTO2023.dbc]"
]
},
"execution_count": 6,
"metadata": {},
"output_type": "execute_result"
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Clear the notebook outputs before merging.

This commit stores executed Colab HTML/JS blobs plus mortality sample rows in git. That makes the diff noisy and republishes dataset contents unnecessarily.

Also applies to: 276-577, 793-1336, 1405-1412

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 99 - 135, The notebook
contains executed outputs (HTML/JS blobs and sample data rows) that should be
removed before merging—clear all cell outputs in
models/br_ms_sim/code/datasus_sim.ipynb (notably the output block shown between
the diff and the other large output ranges at 276-577, 793-1336, and 1405-1412),
remove any embedded dataset rows and HTML/JS artifacts, and save the notebook
with outputs cleared; you can do this manually in Jupyter/Colab or run a
CI-friendly tool (e.g., nbstripout or jupyter nbconvert --clear-output) to
ensure the file committed contains only the notebook source without execution
outputs.

Comment on lines +750 to +768
"colunas_faltantes = set(colunas_esperadas) - set(dados_mortalidade.columns)\n",
"\n",
"print(sorted(colunas_faltantes))"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "LOwNbdq0c1M0"
},
"outputs": [],
"source": [
"dados_mortalidade[\"ano\"] = 2023\n",
"dados_mortalidade[\"codigo_bairro_ocorrencia\"] = None\n",
"dados_mortalidade[\"codigo_bairro_residencia\"] = None\n",
"dados_mortalidade[\"crm\"] = None\n",
"dados_mortalidade[\"data_recebimento_original\"] = None\n",
"dados_mortalidade[\"sequencial_obito\"] = None"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟠 Major

Backfill only the truly missing columns here.

You already compute colunas_faltantes, but Lines 764-768 then hard-reset a hand-picked subset. Because the rename map can already populate fields like codigo_bairro_ocorrencia, codigo_bairro_residencia, and crm, this can wipe valid source values.

🩹 Safer backfill
 dados_mortalidade["ano"] = 2023
-dados_mortalidade["codigo_bairro_ocorrencia"] = None
-dados_mortalidade["codigo_bairro_residencia"] = None
-dados_mortalidade["crm"] = None
-dados_mortalidade["data_recebimento_original"] = None
-dados_mortalidade["sequencial_obito"] = None
+for coluna in colunas_faltantes - {"ano"}:
+    if coluna not in dados_mortalidade.columns:
+        dados_mortalidade[coluna] = None
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 750 - 768, You're
overwriting potentially valid source values by unconditionally setting columns
like codigo_bairro_ocorrencia, codigo_bairro_residencia, crm, etc. on
dados_mortalidade; instead, use the computed colunas_faltantes set to only
create/backfill those missing columns (e.g., check membership in
colunas_faltantes before assigning defaults) so existing renamed/mapped fields
are preserved; update the block that assigns ano, codigo_bairro_ocorrencia,
codigo_bairro_residencia, crm, data_recebimento_original, sequencial_obito to
conditionally assign defaults only when the column is in colunas_faltantes.

Comment on lines +763 to +779
"dados_mortalidade[\"ano\"] = 2023\n",
"dados_mortalidade[\"codigo_bairro_ocorrencia\"] = None\n",
"dados_mortalidade[\"codigo_bairro_residencia\"] = None\n",
"dados_mortalidade[\"crm\"] = None\n",
"dados_mortalidade[\"data_recebimento_original\"] = None\n",
"dados_mortalidade[\"sequencial_obito\"] = None"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {
"id": "THGugqmEf38l"
},
"outputs": [],
"source": [
"dados_mortalidade = dados_mortalidade[colunas_esperadas]"

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

# First, locate and examine the notebook file
find . -type f -name "datasus_sim.ipynb" -o -name "*br_ms_sim*" | head -20

Repository: basedosdados/pipelines

Length of output: 216


🏁 Script executed:

# Search for the SQL model file mentioned
find . -type f -name "*br_ms_sim__microdados.sql"

Repository: basedosdados/pipelines

Length of output: 110


🏁 Script executed:

# Get an overview of the directory structure for br_ms_sim
find . -path "*br_ms_sim*" -type f | head -30

Repository: basedosdados/pipelines

Length of output: 227


🏁 Script executed:

# Get the size of the notebook to determine how to read it
wc -l ./models/br_ms_sim/code/datasus_sim.ipynb

Repository: basedosdados/pipelines

Length of output: 112


🏁 Script executed:

# Check the SQL model for safe_cast and PARSE_DATE usage
cat -n ./models/br_ms_sim/br_ms_sim__microdados.sql

Repository: basedosdados/pipelines

Length of output: 6487


🏁 Script executed:

# Extract the notebook content to understand the date handling
# Notebooks are JSON, so let's read it and search for date-related code
python3 << 'EOF'
import json
import re

with open('./models/br_ms_sim/code/datasus_sim.ipynb', 'r') as f:
    notebook = json.load(f)

# Look for cells containing "data_obito", "hora_obito", "data_nascimento", etc.
# and CSV export logic
for i, cell in enumerate(notebook['cells']):
    if cell['cell_type'] == 'code':
        source = ''.join(cell['source']) if isinstance(cell['source'], list) else cell['source']
        # Search for date-related operations and CSV export
        if any(keyword in source.lower() for keyword in ['data_obito', 'hora_obito', 'data_', 'to_csv', 'strftime', 'format', 'normalize']):
            print(f"=== CELL {i} (around line ~{i*10}) ===")
            print(source[:1000])
            print("\n")

EOF

Repository: basedosdados/pipelines

Length of output: 4459


🏁 Script executed:

# Let's also search for specific lines mentioned in the review (763-779 and 1376-1382)
# Convert notebook to readable text to find those line references
python3 << 'EOF'
import json

with open('./models/br_ms_sim/code/datasus_sim.ipynb', 'r') as f:
    notebook = json.load(f)

# Get all source code concatenated with line numbers
all_lines = []
for cell_idx, cell in enumerate(notebook['cells']):
    if cell['cell_type'] == 'code':
        source = ''.join(cell['source']) if isinstance(cell['source'], list) else cell['source']
        for line in source.split('\n'):
            all_lines.append(line)

# Print lines around 763-779 and 1376-1382
print("=== LINES 755-785 ===")
for i in range(755, min(785, len(all_lines))):
    print(f"{i}: {all_lines[i]}")

print("\n=== LINES 1368-1390 ===")
for i in range(1368, min(1390, len(all_lines))):
    print(f"{i}: {all_lines[i]}")

EOF

Repository: basedosdados/pipelines

Length of output: 112


🏁 Script executed:

# Also search for date formatting patterns and CSV export in the notebook
rg -A 5 -B 5 "(to_csv|strftime|datetime|date_obito|hora_obito)" ./models/br_ms_sim/code/datasus_sim.ipynb

Repository: basedosdados/pipelines

Length of output: 3838


Normalize date/time fields in the notebook before exporting to CSV.

The exported CSVs contain raw date tokens (01012023 as DDMMYYYY, 0610 as HHMM) that safe_cast in br_ms_sim__microdados.sql cannot parse. BigQuery's safe_cast(x as date) and safe_cast(x as time) expect standard formats (YYYY-MM-DD and HH:MM:SS); any mismatch silently produces NULL values. Convert these fields in the notebook to standard formats before calling .to_csv(), or switch the SQL model to use PARSE_DATE and PARSE_TIME with explicit format strings.

Affected fields: data_obito, hora_obito, data_nascimento, data_atestado, data_investigacao, data_cadastro, data_recebimento, data_recebimento_original, and others.

Also applies to: 1376-1382

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 763 - 779, The exported
CSV contains raw date/time tokens (e.g. "01012023", "0610") that BigQuery
safe_cast cannot parse; before writing dados_mortalidade.to_csv() convert those
columns in the notebook (before the line that selects colunas_esperadas) to ISO
date/time strings (YYYY-MM-DD for dates, HH:MM:SS for times) using pandas
operations: parse with explicit formats (e.g., dayfirst format for DDMMYYYY),
zero-pad and insert separators for times, and overwrite fields like data_obito,
hora_obito, data_nascimento, data_atestado, data_investigacao, data_cadastro,
data_recebimento, data_recebimento_original, etc.; alternatively update the SQL
model (br_ms_sim__microdados.sql) to use PARSE_DATE/PARSE_TIME with explicit
format strings if you prefer to keep raw tokens.

"\n",
"colunas_unicas = colunas_unicas.tolist()\n",
"\n",
"arquitetura = pd.read_csv(\"/[arquitetura] microdados.csv\")\n",

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🔴 Critical

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
# Locate likely architecture inputs and confirm whether the current literal path
# corresponds to anything checked into the repository.
fd -HI '.*arquitetura.*\.csv$' .
fd -HI '.*microdados.*\.csv$' .
rg -n 'covered_by_dictionary|\[arquitetura\] microdados\.csv|arquitetura\.csv'

Repository: basedosdados/pipelines

Length of output: 9373


Point arquitetura to a real checked-in input.

Line 1432 reads from /[arquitetura] microdados.csv, which is a placeholder path that does not exist in the repository. On a clean checkout, this code will fail with FileNotFoundError. The data file needs to be checked in or the path updated to reference an actual input file.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@models/br_ms_sim/code/datasus_sim.ipynb` at line 1432, The cell assigns
arquitetura by reading a CSV from an invalid placeholder path; update the read
to point to a real, checked-in input and add a clear fallback/error: replace the
"/[arquitetura] microdados.csv" placeholder with the actual repository CSV
filename (or a relative path inside the project) used for this dataset, ensure
the variable arquitetura is loaded via pd.read_csv with that real filename, and
optionally add a simple existence check (or try/except around the pd.read_csv)
to raise a descriptive error if the file is still missing so the notebook fails
fast and clearly.

@aspeddro aspeddro left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@laribritto olhei os dados de 2023 e 2024. Tem alguns problemas:

  • A coluna data_obito tá nula. Nos anos anteriores a 2023 têm valores.
  • As colunas raca_cor, estado_civil, escolaridade, ocupacao está com um espaço em branco (veja que eu selecionei com o mouse, destacado em azul). Se não tem valor deve ser null. Quando é nulo o BQ mosta null em cinza. Muitas colunas tem um espaço em branco.

Verifica as colunas novamente.

Image

@folhesgabriel

Copy link
Copy Markdown
Collaborator

Atualização feita em #1573

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

check-metadata [PR] Dispara validação de metadados entre BigQuery e API de produção test-dev-model [PR] Roda testes DBT nos models modificados em basedosdados-dev

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants