br_ms_sim - #1475
Conversation
📝 WalkthroughWalkthroughThe PR extends the year partition range of a SQL model from 2022 to 2024 and introduces a new Jupyter notebook that orchestrates SIM mortality data ingestion, transformation, and schema normalization, culminating in partitioned CSV outputs and dictionary coverage tracking. Changes
Sequence Diagram(s)sequenceDiagram
participant Notebook as Jupyter Notebook
participant pysus as pysus Library
participant LocalFS as Local Parquet Files
participant Transform as Data Transform
participant Output as Output CSV
participant Dict as Dictionary CSV
participant Arch as Architecture CSV
Notebook->>pysus: Load SIM metadata & CID10 files
pysus-->>Notebook: Metadata loaded
Notebook->>LocalFS: Read Parquet files (ano=2023)
LocalFS-->>Notebook: Parquet data returned
Notebook->>Transform: Extract UF from filenames
Notebook->>Transform: Apply column name mapping
Notebook->>Transform: Validate against expected columns
Notebook->>Transform: Create missing fields & reorder
Transform-->>Notebook: Transformed dataframe
Notebook->>Output: Partition by sigla_uf & write CSVs
Output-->>Notebook: CSVs written
Notebook->>Dict: Load dictionary CSV
Dict-->>Notebook: Dictionary metadata
Notebook->>Arch: Update covered_by_dictionary flags
Arch-->>Notebook: Architecture CSV updated
Estimated code review effort🎯 4 (Complex) | ⏱️ ~55 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Warning There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure. 🔧 Ruff (0.15.7)models/br_ms_sim/code/datasus_sim.ipynbUnexpected end of JSON input Comment |
There was a problem hiding this comment.
Actionable comments posted: 6
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@models/br_ms_sim/code/datasus_sim.ipynb`:
- Line 84: The code is hard-coded to only process 2023 where the literal
year=[2023] appears; update those occurrences (the year parameter passed into
the data-slicing/query functions) to include 2024 or make the argument dynamic
(e.g., year=[2023, 2024] or use a variable like years = [2023, 2024] passed into
the same calls) so the 2024 slice is generated; search for the exact token
year=[2023] in the notebook and replace each instance (including the calls that
build the workflow/slices) accordingly.
- Around line 99-135: The notebook contains executed outputs (HTML/JS blobs and
sample data rows) that should be removed before merging—clear all cell outputs
in models/br_ms_sim/code/datasus_sim.ipynb (notably the output block shown
between the diff and the other large output ranges at 276-577, 793-1336, and
1405-1412), remove any embedded dataset rows and HTML/JS artifacts, and save the
notebook with outputs cleared; you can do this manually in Jupyter/Colab or run
a CI-friendly tool (e.g., nbstripout or jupyter nbconvert --clear-output) to
ensure the file committed contains only the notebook source without execution
outputs.
- Around line 53-83: The UF list passed to sim.get_files when creating sp_cid10
contains a stray entry "dados_mortalidade" instead of the Distrito Federal code;
update the uf array in the sim.get_files call (the sp_cid10 assignment) by
replacing "dados_mortalidade" with "DF" so the Distrito Federal is included and
the fetch works correctly.
- Line 1432: The cell assigns arquitetura by reading a CSV from an invalid
placeholder path; update the read to point to a real, checked-in input and add a
clear fallback/error: replace the "/[arquitetura] microdados.csv" placeholder
with the actual repository CSV filename (or a relative path inside the project)
used for this dataset, ensure the variable arquitetura is loaded via pd.read_csv
with that real filename, and optionally add a simple existence check (or
try/except around the pd.read_csv) to raise a descriptive error if the file is
still missing so the notebook fails fast and clearly.
- Around line 750-768: You're overwriting potentially valid source values by
unconditionally setting columns like codigo_bairro_ocorrencia,
codigo_bairro_residencia, crm, etc. on dados_mortalidade; instead, use the
computed colunas_faltantes set to only create/backfill those missing columns
(e.g., check membership in colunas_faltantes before assigning defaults) so
existing renamed/mapped fields are preserved; update the block that assigns ano,
codigo_bairro_ocorrencia, codigo_bairro_residencia, crm,
data_recebimento_original, sequencial_obito to conditionally assign defaults
only when the column is in colunas_faltantes.
- Around line 763-779: The exported CSV contains raw date/time tokens (e.g.
"01012023", "0610") that BigQuery safe_cast cannot parse; before writing
dados_mortalidade.to_csv() convert those columns in the notebook (before the
line that selects colunas_esperadas) to ISO date/time strings (YYYY-MM-DD for
dates, HH:MM:SS for times) using pandas operations: parse with explicit formats
(e.g., dayfirst format for DDMMYYYY), zero-pad and insert separators for times,
and overwrite fields like data_obito, hora_obito, data_nascimento,
data_atestado, data_investigacao, data_cadastro, data_recebimento,
data_recebimento_original, etc.; alternatively update the SQL model
(br_ms_sim__microdados.sql) to use PARSE_DATE/PARSE_TIME with explicit format
strings if you prefer to keep raw tokens.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 21f92ae1-4777-4897-8741-47cac60cbf39
📒 Files selected for processing (2)
models/br_ms_sim/br_ms_sim__microdados.sqlmodels/br_ms_sim/code/datasus_sim.ipynb
| "sp_cid10 = sim.get_files(\n", | ||
| " \"CID10\",\n", | ||
| " uf=[\n", | ||
| " \"AC\",\n", | ||
| " \"AL\",\n", | ||
| " \"AP\",\n", | ||
| " \"AM\",\n", | ||
| " \"BA\",\n", | ||
| " \"CE\",\n", | ||
| " \"dados_mortalidade\",\n", | ||
| " \"ES\",\n", | ||
| " \"GO\",\n", | ||
| " \"MA\",\n", | ||
| " \"MT\",\n", | ||
| " \"MS\",\n", | ||
| " \"MG\",\n", | ||
| " \"PA\",\n", | ||
| " \"PB\",\n", | ||
| " \"PR\",\n", | ||
| " \"PE\",\n", | ||
| " \"PI\",\n", | ||
| " \"RJ\",\n", | ||
| " \"RN\",\n", | ||
| " \"RS\",\n", | ||
| " \"RO\",\n", | ||
| " \"RR\",\n", | ||
| " \"SC\",\n", | ||
| " \"SP\",\n", | ||
| " \"SE\",\n", | ||
| " \"TO\",\n", | ||
| " ],\n", |
There was a problem hiding this comment.
Replace the stray UF entry with DF.
Line 62 is not a UF code, and DF is missing from the list. On rerun this either fails the fetch or ships an incomplete extract without Distrito Federal.
🐛 Minimal fix
- "dados_mortalidade",
+ "DF",📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| "sp_cid10 = sim.get_files(\n", | |
| " \"CID10\",\n", | |
| " uf=[\n", | |
| " \"AC\",\n", | |
| " \"AL\",\n", | |
| " \"AP\",\n", | |
| " \"AM\",\n", | |
| " \"BA\",\n", | |
| " \"CE\",\n", | |
| " \"dados_mortalidade\",\n", | |
| " \"ES\",\n", | |
| " \"GO\",\n", | |
| " \"MA\",\n", | |
| " \"MT\",\n", | |
| " \"MS\",\n", | |
| " \"MG\",\n", | |
| " \"PA\",\n", | |
| " \"PB\",\n", | |
| " \"PR\",\n", | |
| " \"PE\",\n", | |
| " \"PI\",\n", | |
| " \"RJ\",\n", | |
| " \"RN\",\n", | |
| " \"RS\",\n", | |
| " \"RO\",\n", | |
| " \"RR\",\n", | |
| " \"SC\",\n", | |
| " \"SP\",\n", | |
| " \"SE\",\n", | |
| " \"TO\",\n", | |
| " ],\n", | |
| "sp_cid10 = sim.get_files(\n", | |
| " \"CID10\",\n", | |
| " uf=[\n", | |
| " \"AC\",\n", | |
| " \"AL\",\n", | |
| " \"AP\",\n", | |
| " \"AM\",\n", | |
| " \"BA\",\n", | |
| " \"CE\",\n", | |
| " \"DF\",\n", | |
| " \"ES\",\n", | |
| " \"GO\",\n", | |
| " \"MA\",\n", | |
| " \"MT\",\n", | |
| " \"MS\",\n", | |
| " \"MG\",\n", | |
| " \"PA\",\n", | |
| " \"PB\",\n", | |
| " \"PR\",\n", | |
| " \"PE\",\n", | |
| " \"PI\",\n", | |
| " \"RJ\",\n", | |
| " \"RN\",\n", | |
| " \"RS\",\n", | |
| " \"RO\",\n", | |
| " \"RR\",\n", | |
| " \"SC\",\n", | |
| " \"SP\",\n", | |
| " \"SE\",\n", | |
| " \"TO\",\n", | |
| " ],\n", |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 53 - 83, The UF list
passed to sim.get_files when creating sp_cid10 contains a stray entry
"dados_mortalidade" instead of the Distrito Federal code; update the uf array in
the sim.get_files call (the sp_cid10 assignment) by replacing
"dados_mortalidade" with "DF" so the Distrito Federal is included and the fetch
works correctly.
| " \"SE\",\n", | ||
| " \"TO\",\n", | ||
| " ],\n", | ||
| " year=[2023],\n", |
There was a problem hiding this comment.
The notebook still only processes 2023.
Lines 84, 149, 582, 763, and 1352 all pin the workflow to 2023, so the 2024 slice from the PR objective is not generated here.
Also applies to: 149-149, 582-582, 763-763, 1352-1352
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` at line 84, The code is hard-coded
to only process 2023 where the literal year=[2023] appears; update those
occurrences (the year parameter passed into the data-slicing/query functions) to
include 2024 or make the argument dynamic (e.g., year=[2023, 2024] or use a
variable like years = [2023, 2024] passed into the same calls) so the 2024 slice
is generated; search for the exact token year=[2023] in the notebook and replace
each instance (including the calls that build the workflow/slices) accordingly.
| "outputs": [ | ||
| { | ||
| "data": { | ||
| "text/plain": [ | ||
| "[DOAC2023.dbc,\n", | ||
| " DOAL2023.dbc,\n", | ||
| " DOAM2023.dbc,\n", | ||
| " DOAP2023.dbc,\n", | ||
| " DOBA2023.dbc,\n", | ||
| " DOCE2023.dbc,\n", | ||
| " DODF2023.dbc,\n", | ||
| " DOES2023.dbc,\n", | ||
| " DOGO2023.dbc,\n", | ||
| " DOMA2023.dbc,\n", | ||
| " DOMG2023.dbc,\n", | ||
| " DOMS2023.dbc,\n", | ||
| " DOMT2023.dbc,\n", | ||
| " DOPA2023.dbc,\n", | ||
| " DOPB2023.dbc,\n", | ||
| " DOPE2023.dbc,\n", | ||
| " DOPI2023.dbc,\n", | ||
| " DOPR2023.dbc,\n", | ||
| " DORJ2023.dbc,\n", | ||
| " DORN2023.dbc,\n", | ||
| " DORO2023.dbc,\n", | ||
| " DORR2023.dbc,\n", | ||
| " DORS2023.dbc,\n", | ||
| " DOSC2023.dbc,\n", | ||
| " DOSE2023.dbc,\n", | ||
| " DOSP2023.dbc,\n", | ||
| " DOTO2023.dbc]" | ||
| ] | ||
| }, | ||
| "execution_count": 6, | ||
| "metadata": {}, | ||
| "output_type": "execute_result" | ||
| } |
There was a problem hiding this comment.
Clear the notebook outputs before merging.
This commit stores executed Colab HTML/JS blobs plus mortality sample rows in git. That makes the diff noisy and republishes dataset contents unnecessarily.
Also applies to: 276-577, 793-1336, 1405-1412
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 99 - 135, The notebook
contains executed outputs (HTML/JS blobs and sample data rows) that should be
removed before merging—clear all cell outputs in
models/br_ms_sim/code/datasus_sim.ipynb (notably the output block shown between
the diff and the other large output ranges at 276-577, 793-1336, and 1405-1412),
remove any embedded dataset rows and HTML/JS artifacts, and save the notebook
with outputs cleared; you can do this manually in Jupyter/Colab or run a
CI-friendly tool (e.g., nbstripout or jupyter nbconvert --clear-output) to
ensure the file committed contains only the notebook source without execution
outputs.
| "colunas_faltantes = set(colunas_esperadas) - set(dados_mortalidade.columns)\n", | ||
| "\n", | ||
| "print(sorted(colunas_faltantes))" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "LOwNbdq0c1M0" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dados_mortalidade[\"ano\"] = 2023\n", | ||
| "dados_mortalidade[\"codigo_bairro_ocorrencia\"] = None\n", | ||
| "dados_mortalidade[\"codigo_bairro_residencia\"] = None\n", | ||
| "dados_mortalidade[\"crm\"] = None\n", | ||
| "dados_mortalidade[\"data_recebimento_original\"] = None\n", | ||
| "dados_mortalidade[\"sequencial_obito\"] = None" |
There was a problem hiding this comment.
Backfill only the truly missing columns here.
You already compute colunas_faltantes, but Lines 764-768 then hard-reset a hand-picked subset. Because the rename map can already populate fields like codigo_bairro_ocorrencia, codigo_bairro_residencia, and crm, this can wipe valid source values.
🩹 Safer backfill
dados_mortalidade["ano"] = 2023
-dados_mortalidade["codigo_bairro_ocorrencia"] = None
-dados_mortalidade["codigo_bairro_residencia"] = None
-dados_mortalidade["crm"] = None
-dados_mortalidade["data_recebimento_original"] = None
-dados_mortalidade["sequencial_obito"] = None
+for coluna in colunas_faltantes - {"ano"}:
+ if coluna not in dados_mortalidade.columns:
+ dados_mortalidade[coluna] = None🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 750 - 768, You're
overwriting potentially valid source values by unconditionally setting columns
like codigo_bairro_ocorrencia, codigo_bairro_residencia, crm, etc. on
dados_mortalidade; instead, use the computed colunas_faltantes set to only
create/backfill those missing columns (e.g., check membership in
colunas_faltantes before assigning defaults) so existing renamed/mapped fields
are preserved; update the block that assigns ano, codigo_bairro_ocorrencia,
codigo_bairro_residencia, crm, data_recebimento_original, sequencial_obito to
conditionally assign defaults only when the column is in colunas_faltantes.
| "dados_mortalidade[\"ano\"] = 2023\n", | ||
| "dados_mortalidade[\"codigo_bairro_ocorrencia\"] = None\n", | ||
| "dados_mortalidade[\"codigo_bairro_residencia\"] = None\n", | ||
| "dados_mortalidade[\"crm\"] = None\n", | ||
| "dados_mortalidade[\"data_recebimento_original\"] = None\n", | ||
| "dados_mortalidade[\"sequencial_obito\"] = None" | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": { | ||
| "id": "THGugqmEf38l" | ||
| }, | ||
| "outputs": [], | ||
| "source": [ | ||
| "dados_mortalidade = dados_mortalidade[colunas_esperadas]" |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, locate and examine the notebook file
find . -type f -name "datasus_sim.ipynb" -o -name "*br_ms_sim*" | head -20Repository: basedosdados/pipelines
Length of output: 216
🏁 Script executed:
# Search for the SQL model file mentioned
find . -type f -name "*br_ms_sim__microdados.sql"Repository: basedosdados/pipelines
Length of output: 110
🏁 Script executed:
# Get an overview of the directory structure for br_ms_sim
find . -path "*br_ms_sim*" -type f | head -30Repository: basedosdados/pipelines
Length of output: 227
🏁 Script executed:
# Get the size of the notebook to determine how to read it
wc -l ./models/br_ms_sim/code/datasus_sim.ipynbRepository: basedosdados/pipelines
Length of output: 112
🏁 Script executed:
# Check the SQL model for safe_cast and PARSE_DATE usage
cat -n ./models/br_ms_sim/br_ms_sim__microdados.sqlRepository: basedosdados/pipelines
Length of output: 6487
🏁 Script executed:
# Extract the notebook content to understand the date handling
# Notebooks are JSON, so let's read it and search for date-related code
python3 << 'EOF'
import json
import re
with open('./models/br_ms_sim/code/datasus_sim.ipynb', 'r') as f:
notebook = json.load(f)
# Look for cells containing "data_obito", "hora_obito", "data_nascimento", etc.
# and CSV export logic
for i, cell in enumerate(notebook['cells']):
if cell['cell_type'] == 'code':
source = ''.join(cell['source']) if isinstance(cell['source'], list) else cell['source']
# Search for date-related operations and CSV export
if any(keyword in source.lower() for keyword in ['data_obito', 'hora_obito', 'data_', 'to_csv', 'strftime', 'format', 'normalize']):
print(f"=== CELL {i} (around line ~{i*10}) ===")
print(source[:1000])
print("\n")
EOFRepository: basedosdados/pipelines
Length of output: 4459
🏁 Script executed:
# Let's also search for specific lines mentioned in the review (763-779 and 1376-1382)
# Convert notebook to readable text to find those line references
python3 << 'EOF'
import json
with open('./models/br_ms_sim/code/datasus_sim.ipynb', 'r') as f:
notebook = json.load(f)
# Get all source code concatenated with line numbers
all_lines = []
for cell_idx, cell in enumerate(notebook['cells']):
if cell['cell_type'] == 'code':
source = ''.join(cell['source']) if isinstance(cell['source'], list) else cell['source']
for line in source.split('\n'):
all_lines.append(line)
# Print lines around 763-779 and 1376-1382
print("=== LINES 755-785 ===")
for i in range(755, min(785, len(all_lines))):
print(f"{i}: {all_lines[i]}")
print("\n=== LINES 1368-1390 ===")
for i in range(1368, min(1390, len(all_lines))):
print(f"{i}: {all_lines[i]}")
EOFRepository: basedosdados/pipelines
Length of output: 112
🏁 Script executed:
# Also search for date formatting patterns and CSV export in the notebook
rg -A 5 -B 5 "(to_csv|strftime|datetime|date_obito|hora_obito)" ./models/br_ms_sim/code/datasus_sim.ipynbRepository: basedosdados/pipelines
Length of output: 3838
Normalize date/time fields in the notebook before exporting to CSV.
The exported CSVs contain raw date tokens (01012023 as DDMMYYYY, 0610 as HHMM) that safe_cast in br_ms_sim__microdados.sql cannot parse. BigQuery's safe_cast(x as date) and safe_cast(x as time) expect standard formats (YYYY-MM-DD and HH:MM:SS); any mismatch silently produces NULL values. Convert these fields in the notebook to standard formats before calling .to_csv(), or switch the SQL model to use PARSE_DATE and PARSE_TIME with explicit format strings.
Affected fields: data_obito, hora_obito, data_nascimento, data_atestado, data_investigacao, data_cadastro, data_recebimento, data_recebimento_original, and others.
Also applies to: 1376-1382
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` around lines 763 - 779, The exported
CSV contains raw date/time tokens (e.g. "01012023", "0610") that BigQuery
safe_cast cannot parse; before writing dados_mortalidade.to_csv() convert those
columns in the notebook (before the line that selects colunas_esperadas) to ISO
date/time strings (YYYY-MM-DD for dates, HH:MM:SS for times) using pandas
operations: parse with explicit formats (e.g., dayfirst format for DDMMYYYY),
zero-pad and insert separators for times, and overwrite fields like data_obito,
hora_obito, data_nascimento, data_atestado, data_investigacao, data_cadastro,
data_recebimento, data_recebimento_original, etc.; alternatively update the SQL
model (br_ms_sim__microdados.sql) to use PARSE_DATE/PARSE_TIME with explicit
format strings if you prefer to keep raw tokens.
| "\n", | ||
| "colunas_unicas = colunas_unicas.tolist()\n", | ||
| "\n", | ||
| "arquitetura = pd.read_csv(\"/[arquitetura] microdados.csv\")\n", |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
# Locate likely architecture inputs and confirm whether the current literal path
# corresponds to anything checked into the repository.
fd -HI '.*arquitetura.*\.csv$' .
fd -HI '.*microdados.*\.csv$' .
rg -n 'covered_by_dictionary|\[arquitetura\] microdados\.csv|arquitetura\.csv'Repository: basedosdados/pipelines
Length of output: 9373
Point arquitetura to a real checked-in input.
Line 1432 reads from /[arquitetura] microdados.csv, which is a placeholder path that does not exist in the repository. On a clean checkout, this code will fail with FileNotFoundError. The data file needs to be checked in or the path updated to reference an actual input file.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_ms_sim/code/datasus_sim.ipynb` at line 1432, The cell assigns
arquitetura by reading a CSV from an invalid placeholder path; update the read
to point to a real, checked-in input and add a clear fallback/error: replace the
"/[arquitetura] microdados.csv" placeholder with the actual repository CSV
filename (or a relative path inside the project) used for this dataset, ensure
the variable arquitetura is loaded via pd.read_csv with that real filename, and
optionally add a simple existence check (or try/except around the pd.read_csv)
to raise a descriptive error if the file is still missing so the notebook fails
fast and clearly.
aspeddro
left a comment
There was a problem hiding this comment.
@laribritto olhei os dados de 2023 e 2024. Tem alguns problemas:
- A coluna
data_obitotá nula. Nos anos anteriores a 2023 têm valores. - As colunas
raca_cor,estado_civil,escolaridade,ocupacaoestá com um espaço em branco (veja que eu selecionei com o mouse, destacado em azul). Se não tem valor deve sernull. Quando é nulo o BQ mosta null em cinza. Muitas colunas tem um espaço em branco.
Verifica as colunas novamente.
|
Atualização feita em #1573 |
Descrição do PR:
Summary by CodeRabbit
New Features
Updates