Sobe código de limpeza de br_tse_eleicoes 1990-2024 - #1467
Conversation
|
@rdahis vamos manter o código em stata no repositório SDK. Neste repo, somente em Pyhton/SQL; |
|
Usaremos o código em stata como base para a criação dos novos em python. Tenho algumas dúvidas:
|
Stata hoje em dia nem entra nos repositórios da BD 😸
Raramente. Eu já vi ser feito mas é raro.
Fora o código o e guia de uso, não. Tem muita coisa comentada no código em si. |
|
Iniciei o mapeamento do dataset
ufs = [
"AC","AL","AP","AM",
"BA","CE","DF","ES","GO",
"MA","MT","MS","MG",
"PA","PB",
"PR","PE","PI",
"RJ","RN","RS","RO","RR",
"SC","SP","SE","TO"
]
anos = [
2024, 2022,
2020,
2018, 2016, 2014, 2012, 2010,
2008, 2006, 2004, 2002, 2000,
1998, 1996, 1994
]Embora nem todas as combinações sejam esperadas, esse mapeamento permite identificar:
Achados atuais
Interpretação
Próximos passos
|
|
Pessoal, estou fazendo a conversão do código para Python com Claude Code aqui. Numa boa, é o mais eficiente. Eu já tenho todo o contexto dessa base. Subirei as atualizações no PR em breve. |
📝 WalkthroughWalkthroughA comprehensive Stata data pipeline is added for processing and normalizing Brazilian TSE (electoral commission) election datasets. The pipeline includes a main orchestration script, helper functions for data cleaning, and multiple subordinate processing scripts that handle candidates, parties, results, voter profiles, financial records, and vacancies across 1994–2024. Changes
Estimated code review effort🎯 5 (Critical) | ⏱️ ~120 minutes Poem
🚥 Pre-merge checks | ✅ 1 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (1 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 9
Note
Due to the large number of review comments, Critical severity comments were prioritized as inline comments.
🟠 Major comments (12)
models/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.do-46-46 (1)
46-46:⚠️ Potential issue | 🟠 MajorThe DF/2014 biometrics exception is currently a no-op.
This creates
situacao_biometrica, but the next block keepscd_mun_sit_biometricaand later renames it tosituacao_biometria. The special-case variable is dropped before it can affect the output.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.do` at line 46, The DF/2014 special-case creates situacao_biometrica but is a no-op because later code continues to use cd_mun_sit_biometrica and then renames it to situacao_biometria; update the DF/2014 branch so it sets the same upstream variable used later (cd_mun_sit_biometrica) or ensure the later renaming/logic uses situacao_biometrica instead; specifically modify the conditional that currently does `if "`uf'" == "DF" & `ano' == 2014 gen situacao_biometrica = ""` to assign or adjust cd_mun_sit_biometrica (or merge situacao_biometrica into cd_mun_sit_biometrica before the subsequent block) so the DF/2014 exception actually affects the eventual situacao_biometria output.models/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.do-105-107 (1)
105-107:⚠️ Potential issue | 🟠 MajorFilter to matched municipalities after the merge.
Dropping only
_merge == 2keeps every master row whoseid_municipio_tseis not in the directory, withid_municipiomissing. That lets placeholder or invalid municipality codes survive into the yearly output.🐛 Suggested fix
- merge m:1 id_municipio_tse using `diretorio' - drop if _merge == 2 - drop _merge + merge m:1 id_municipio_tse using `diretorio', keep(3) nogen🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.do` around lines 105 - 107, The current merge (merge m:1 id_municipio_tse using `diretorio') only drops _merge == 2, leaving master-only rows (with missing id_municipio) in output; change the filter to keep only matched rows by replacing "drop if _merge == 2" with "keep if _merge == 3" (or "drop if _merge != 3"), then remove the _merge variable as before (drop _merge) so only rows with matched id_municipio_tse → id_municipio survive.models/br_tse_eleicoes/code/sub/resultados_uf.do-63-75 (1)
63-75:⚠️ Potential issue | 🟠 MajorComplex import fallback logic may load unexpected data.
Multiple
cap importstatements execute sequentially (lines 63-74), each potentially overwriting the previous import. If an earlier import succeeds but the later one also succeeds for a different file, the wrong data may be processed. The subsequentifblocks (lines 76-156) then apply year-specific transformations that may not match the actual loaded data.Consider restructuring to ensure only one import succeeds per iteration, with explicit error handling.
💡 Suggested restructure
local imported = 0 cap import delimited "input/votacao_candidato_uf/votacao_candidato_uf_`ano'_br/VOTACAO_CANDIDATO_UF_`ano'_`estado'.txt", ... if _rc == 0 local imported = 1 if `imported' == 0 { cap import delimited "input/votacao_candidato_uf/VOTACAO_CANDIDATO_`ano'/VOTACAO_CANDIDATO_UF_`ano'_`estado'.txt", ... if _rc == 0 local imported = 1 } if `imported' == 0 { cap import delimited "input/votacao_candidato_uf/VOTACAO_CANDIDATO_UF_`ano'/VOTACAO_CANDIDATO_UF_`ano'_`estado'.txt", ... if _rc == 0 local imported = 1 } if `imported' == 0 { di as error "No file found for `ano' `estado'" continue }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/resultados_uf.do` around lines 63 - 75, The current sequence of cap import delimited calls (the three imports referencing votacao_candidato_uf_`ano'_br and two VOTACAO_CANDIDATO_UF_`ano' paths) can overwrite earlier successful imports; change the logic so only one import is accepted: attempt each cap import in order, check _rc after each import and set a local flag (e.g., local imported = 1) when one succeeds, and skip subsequent imports if imported==1; if none succeed, emit an explicit error/di and continue; retain the special-case drop in 1 for `ano' == 1933 | `ano' == 1934 but apply it only after confirming imported==1.models/br_tse_eleicoes/code/sub/candidatos.do-65-66 (1)
65-66:⚠️ Potential issue | 🟠 MajorPII fields (CPF, email) in output dataset.
The dataset includes
cpfandAlso applies to: 101-102, 137-138
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/candidatos.do` around lines 65 - 66, The output currently renames and exposes raw PII via the statements "ren v21 cpf" and "ren v22 email" (and similar renames at the other occurrences), so either remove these fields from the published dataset or pseudonymize them before output; update the renames to not propagate raw CPF/email (remove those ren v21/v22 lines or replace with masked/anonymized equivalents) and apply the same change to the other occurrences referenced (the ren statements around lines 101-102 and 137-138) so no raw CPF or email leaves the final output.models/br_tse_eleicoes/code/sub/detalhes_votacao_secao.do-123-125 (1)
123-125:⚠️ Potential issue | 🟠 MajorPotential unintended data loss when handling duplicates.
The current logic tags duplicates and then drops ALL rows where
dup > 0. This means if there are 2 identical records, both are dropped, not just the duplicate. If the intent is to keep one copy of each unique record, useduplicates dropinstead.💡 Suggested fix if intent is to keep one copy
-duplicates tag ano turno tipo_eleicao sigla_uf id_municipio_tse zona secao cargo, gen(dup) // idealmente usar o codigo_eleicao (v7) para identificar -drop if dup > 0 -drop dup +duplicates drop ano turno tipo_eleicao sigla_uf id_municipio_tse zona secao cargo, forceIf the intent is genuinely to remove all ambiguous records, add a comment explaining this decision.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_secao.do` around lines 123 - 125, The current sequence using "duplicates tag ano turno tipo_eleicao sigla_uf id_municipio_tse zona secao cargo, gen(dup)" followed by "drop if dup > 0" and "drop dup" will remove all copies of duplicated records; if you intend to keep one copy, replace that sequence with Stata's "duplicates drop ano turno tipo_eleicao sigla_uf id_municipio_tse zona secao cargo" to retain a single observation per group (or alternatively add a clear comment above the existing "drop if dup > 0" explaining that the deliberate intent is to remove all ambiguous records if that is desired).models/br_tse_eleicoes/code/build.do-11-19 (1)
11-19:⚠️ Potential issue | 🟠 MajorHardcoded absolute paths will break for other developers.
The script uses user-specific paths (
~/Downloads/dados_TSE,~/Dropbox/BD/sdk/...) that won't work on other machines. Consider using relative paths or environment variables for portability.Additionally, there's an inconsistency: function definitions (lines 13-19) use absolute paths while build scripts (lines 25-40) use relative paths.
💡 Suggested approach
-cd "~/Downloads/dados_TSE" +// Set root directory via environment variable or relative path +local root_dir = cond("$TSE_DATA_DIR" != "", "$TSE_DATA_DIR", ".") +cd "`root_dir'" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/clean_string.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_tipo_eleicao.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_instrucao.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_estado_civil.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_resultado.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_partido.do" -do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/limpa_candidato.do" +do "code/fnc/clean_string.do" +do "code/fnc/limpa_tipo_eleicao.do" +do "code/fnc/limpa_instrucao.do" +do "code/fnc/limpa_estado_civil.do" +do "code/fnc/limpa_resultado.do" +do "code/fnc/limpa_partido.do" +do "code/fnc/limpa_candidato.do"🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/build.do` around lines 11 - 19, The build script uses hardcoded absolute paths (e.g., cd "~/Downloads/dados_TSE" and do "~/Dropbox/BD/sdk/bases/br_tse_eleicoes/code/fnc/clean_string.do") which breaks portability; change the script to derive those locations from a configurable base variable (e.g., BASE_DATA_DIR or SDK_DIR) or use project-relative paths, then reference that variable when calling the functions (clean_string.do, limpa_tipo_eleicao.do, limpa_instrucao.do, limpa_estado_civil.do, limpa_resultado.do, limpa_partido.do, limpa_candidato.do) so both the build and function includes use the same indirection and no user-specific absolute paths remain.models/br_tse_eleicoes/code/sub/prestacao_contas.do-1107-1117 (1)
1107-1117:⚠️ Potential issue | 🟠 MajorPersist monetary values as numeric columns.
valor_receitaandvalor_despesaare normalized withsubinstr(..., ",", ".", .)throughout the file, but neverdestringed before these final saves.normalizacao_particao.doonly reorders/exports these tables, so the published finance outputs keep amount fields as strings.Suggested fix
+capture confirm variable valor_receita +if !_rc destring valor_receita, replace force + +capture confirm variable valor_despesa +if !_rc destring valor_despesa, replace force + compress save "output/receitas_candidato_`ano'.dta", replace @@ +capture confirm variable valor_receita +if !_rc destring valor_receita, replace force + +capture confirm variable valor_despesa +if !_rc destring valor_despesa, replace force + compress save "output/despesas_candidato_`ano'.dta", replaceAlso applies to: 2043-2053
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/prestacao_contas.do` around lines 1107 - 1117, The money fields valor_receita and valor_despesa are still strings (they were normalized with subinstr but not converted); before the final compress/save in prestacao_contas.do, use destring on these variables (e.g., destring valor_receita, replace force and destring valor_despesa, replace force) to convert them to numeric and handle non-numeric characters, then optionally format or label them; apply the same change in the other similar block that handles the other partitioned save (the block that mirrors this logic later in the file where valor_receita/valor_despesa are produced).models/br_tse_eleicoes/code/sub/normalizacao_particao.do-94-95 (1)
94-95:⚠️ Potential issue | 🟠 MajorPersist the cleaned candidate lookup before downstream sections reuse it.
Step 3 writes the post-cleanup candidate table only to a tempfile. Later sections in this same script reopen
output/norm_candidatos.dta, so results, bens, and finance merges use the pre-cleanup lookup whileoutput/candidatos/ano=*/candidatos.csvuses the cleaned one.Suggested fix
compress + save "output/norm_candidatos.dta", replace + export delimited "output/norm_candidatos.csv", replace + tempfile candidatos save `candidatos'Also applies to: 102-160
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/normalizacao_particao.do` around lines 94 - 95, The script currently writes the cleaned candidate lookup only to a tempfile so downstream merges reopen the old pre-cleanup file; change the save/export commands (the save "output/norm_candidatos.dta", replace and export delimited "output/norm_candidatos.csv", replace) to persist the cleaned lookup into output/norm_candidatos.dta and output/norm_candidatos.csv (remove tempfile usage) so subsequent sections that reopen output/norm_candidatos.dta will use the cleaned table; apply the same change to the duplicate blocks around lines referenced (the subsequent 102-160 region) so all downstream merges (results, bens, finance) consume the cleaned lookup.models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do-217-220 (1)
217-220:⚠️ Potential issue | 🟠 MajorGuard blank
data_eleicaovalues before reformatting.When
data_eleicaois empty, this generates--, and the secondreplacedoes not clear it. Invalid dates will then leak intooutput/detalhes_votacao_municipio_zona_*.dta.Suggested fix
foreach k in eleicao { - replace data_`k' = substr(data_`k', 7, 4) + "-" + substr(data_`k', 4, 2) + "-" + substr(data_`k', 1, 2) + replace data_`k' = substr(data_`k', 7, 4) + "-" + substr(data_`k', 4, 2) + "-" + substr(data_`k', 1, 2) if data_`k' != "" replace data_`k' = "" if real(substr(data_`k', 1, 4)) < 1900 }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do` around lines 217 - 220, The loop over "foreach k in eleicao" should guard empty or too-short data_eleicao values before using substr; change the transform on data_`k' so you first set it to missing/"" if it is empty or length < 10 (or if substr would produce non-numeric year), then only run the YYYY-MM-DD reconstruction and the year check; in practice wrap the current replace lines in a conditional like "if data_`k' != "" & strlen(data_`k')>=10" (or pre-emptively replace data_`k' = "" if strlen(data_`k')<10) so data_`k' never becomes "--" and the year check using real(substr(...)) only runs on valid strings.models/br_tse_eleicoes/code/sub/agregacao.do-195-216 (1)
195-216:⚠️ Potential issue | 🟠 MajorRestore the intended year range here.
This block currently exports
detalhes_votacao_municipioonly for 2024, so every 1994-2022 year built upstream disappears from the aggregated outputs. The inline comment suggests this was meant to be temporary.Suggested fix
-foreach ano of numlist 2024 { // 1994(2)2024 { +foreach ano of numlist 1994(2)2024 {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/agregacao.do` around lines 195 - 216, The year loop is hardcoded to only 2024 (foreach ano of numlist 2024) which drops earlier years; restore the intended range by changing the numlist to the original sequence (e.g., 1994(2)2024 or whatever full range the pipeline expects) so the foreach ano loop (and subsequent mkdir, local subdirs, import delimited, collapse, gen, export delimited steps) runs for all years rather than just 2024.models/br_tse_eleicoes/code/sub/resultados_secao.do-164-189 (1)
164-189:⚠️ Potential issue | 🟠 MajorMerge nominal and legend votes on the full election key.
Both temp tables are collapsed by
ano id_eleicao tipo_eleicao data_eleicao turno ..., but themerge 1:1dropstipo_eleicaoanddata_eleicaofrom the key. On legacy inputs whereid_eleicao == "", that can either mismatch rows from different elections or make the merge fail on non-unique keys.Suggested fix
- merge 1:1 ano id_eleicao turno sigla_uf id_municipio_tse zona secao cargo numero_partido using "tmp/votos_legenda.dta", nogenerate + merge 1:1 ano id_eleicao tipo_eleicao data_eleicao turno sigla_uf id_municipio_tse zona secao cargo numero_partido using "tmp/votos_legenda.dta", nogenerate🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/resultados_secao.do` around lines 164 - 189, The merge drops tipo_eleicao and data_eleicao from the merge key even though the datasets were collapsed by them; update the merge to include those fields so the 1:1 key matches the collapse (i.e. change the merge call in resultados_secao.do to merge 1:1 ano id_eleicao tipo_eleicao data_eleicao turno sigla_uf id_municipio_tse zona secao cargo numero_partido using "tmp/votos_legenda.dta", nogenerate) so legacy rows with id_eleicao == "" don’t collide or produce non-unique matches.models/br_tse_eleicoes/code/sub/prestacao_contas.do-1513-1521 (1)
1513-1521:⚠️ Potential issue | 🟠 MajorNormalize 2012 expense dates with the same path used elsewhere.
This branch prefixes
20to the whole raw string and never runs the usualdespesadate reformat loop, sodata_despesais saved in a one-off malformed format only for 2012.Suggested fix
- replace data_despesa = "20" + data_despesa if length(data_despesa) == 8 + replace data_despesa = substr(data_despesa, 1, 10) + replace data_despesa = "0" + data_despesa if length(data_despesa) == 9 + replace data_despesa = substr(data_despesa, 7, 4) + "-" + substr(data_despesa, 4, 2) + "-" + substr(data_despesa, 1, 2) if length(data_despesa) > 0🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/prestacao_contas.do` around lines 1513 - 1521, The branch that handles 2012 currently does only replace data_despesa = "20" + data_despesa (when length==8) which produces a malformed date because it skips the existing despesa date reformat loop; change this so after prefixing the year you run the same despesa reformat logic used elsewhere (i.e., the loop/code that normalizes data_despesa into day/month/year format) instead of leaving the one-off format—update the block containing data_despesa, valor_despesa and the subsequent gen id_eleicao lines so data_despesa is passed through the unified reformat routine rather than only prepending "20".
🟡 Minor comments (9)
models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do-71-77 (1)
71-77:⚠️ Potential issue | 🟡 MinorFragile assumption: script fails if
avanteparty file is missing.Line 71 unconditionally loads
filiacao_avanteas the base dataset. If thefiliados_avante.csvinput file doesn't exist, the script will fail here—even though other party files may have been processed successfully.Consider making the base dataset selection dynamic by using the first available party tempfile, or checking existence before loading.
Suggested approach for Python refactor
When refactoring to Python, use a pattern that collects all successfully processed DataFrames into a list and concatenates them at the end, avoiding dependence on any specific party being present.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do` around lines 71 - 77, The code unconditionally does use `filiacao_avante' which breaks if that tempfile/input is missing; change the logic to select a base dataset dynamically or guard the load with an existence check: iterate `partidos', check each `filiacao_`partido'' exists (e.g., capture confirm file or fileexists) and when you find the first existing tempfile set it as the base (replace the `use filiacao_avante' line), then append the rest; alternatively collect all existing `filiacao_...'' files into a list first and loop to append them so the script does not depend on `filiacao_avante' being present.models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do-233-278 (1)
233-278:⚠️ Potential issue | 🟡 MinorPotential error when dataset is empty after filtering.
The
keepcommand at line 276 is outside theif _N > 0block. Iftmp/para_2a_rodada.dtaloads with observations but they're all filtered out bykeep if N_nomes == 1at line 267, subsequentkeepandsavewould operate on an empty dataset, which is fine. However, if the dataset is empty from the start (_N == 0), the variables referenced inkeepmay not exist, causing a runtime error.Consider wrapping the final
keepandsavein the same conditional or usingcaptureto handle this edge case gracefully.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do` around lines 233 - 278, The final keep (keep id_candidato_bd_str titulo_eleitoral nome) and save ("tmp/2a_rodada.dta", replace) can error when the dataset is empty because they live outside the if _N > 0 block; wrap those two statements in the same if _N > 0 conditional (or alternatively guard with a check for variable existence or use capture) so they only run when observations/variables created by the id_candidato_bd_str / N_nomes logic exist (reference: the existing if _N > 0 block, the N_nomes filter, and the final keep/save calls).models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do-6-9 (1)
6-9:⚠️ Potential issue | 🟡 MinorYear range mismatch between v1 and v2 variants.
The v1 variant loads data through 2022 (line 7:
2000(2)2022), while the v2 variant at line 172 loads through 2024 (2000(2)2024). This inconsistency means v1 will be missing the 2024 election data.If both variants are intended for comparable outputs, update v1 to include 2024:
Proposed fix
use "output/candidatos_1998.dta", clear -foreach ano of numlist 2000(2)2022 { +foreach ano of numlist 2000(2)2024 { append using "output/candidatos_`ano'.dta" }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do` around lines 6 - 9, The foreach loop that appends year files uses the range "2000(2)2022" and therefore omits 2024; update the numlist in the foreach statement (the foreach ano of numlist ... loop) to match the v2 range "2000(2)2024" so the code appends the 2024 candidato file as well.models/br_tse_eleicoes/code/sub/perfil_eleitorado_municipio_zona.do-34-34 (1)
34-34:⚠️ Potential issue | 🟡 Minor
instrucaofield is not normalized.The
instrucao(education level) field is extracted butlimpa_instrucaois never called to normalize it, unlike incandidatos.do. The function is loaded bybuild.dobut not invoked here. Similarly,estado_civilandgenerocould benefit from normalization for consistency.💡 Suggested addition after line 76
// Normalize categorical fields limpa_instrucao // Add other normalizations as needed: // limpa_estado_civilAlso applies to: 52-52
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_municipio_zona.do` at line 34, The extracted categorical fields are not being normalized: invoke limpa_instrucao after the instrucao extraction in perfil_eleitorado_municipio_zona.do to normalize education levels, and likewise call limpa_estado_civil and limpa_genero (or their implemented equivalents) after extracting estado_civil and genero so values match the normalized forms used elsewhere (e.g., candidatos.do); update the code to call limpa_instrucao (and add limpa_estado_civil / limpa_genero calls) immediately after the relevant extraction blocks to ensure consistent categorical normalization.models/br_tse_eleicoes/code/sub/perfil_eleitorado_municipio_zona.do-17-18 (1)
17-18:⚠️ Potential issue | 🟡 MinorSilent failure if input file is missing.
Same issue as in
vagas.do- usingcapon both imports without checking_rcmeans no warning is raised if neither file exists.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_municipio_zona.do` around lines 17 - 18, The two cap import delimited commands silently fail if files are missing; change this by checking the result code after each import (inspect _rc) or use fileexists() before importing: try the first import ("perfil_eleitorado_`ano'.txt"), if _rc!=0 then try the CSV import ("perfil_eleitorado_`ano'.csv"), and if that also fails emit an error message and exit (e.g., using display and exit or error codes) so the script doesn't continue silently; update the import lines in perfil_eleitorado_municipio_zona.do to implement this check around the import delimited calls.models/br_tse_eleicoes/code/sub/vagas.do-2-4 (1)
2-4:⚠️ Potential issue | 🟡 MinorIncorrect comment - copy-paste error.
The header comment says "build: candidatos" but this file builds the "vagas" (vacancies) dataset.
📝 Suggested fix
//----------------------------------------------------------------------------// -// build: candidatos +// build: vagas //----------------------------------------------------------------------------//🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/vagas.do` around lines 2 - 4, Update the incorrect header comment at the top of the file (the comment that currently reads "build: candidatos") to accurately reflect the dataset this file builds by changing it to "build: vagas"; locate and edit the header comment block in models/br_tse_eleicoes/code/sub/vagas.do (the top comment lines that start with "// build:") so the wording matches the file purpose.models/br_tse_eleicoes/code/sub/resultados_uf.do-2-4 (1)
2-4:⚠️ Potential issue | 🟡 MinorIncorrect comment - file processes UF-level results.
The header comment says "resultados municipio-zona" but this script processes UF-level election results.
📝 Suggested fix
//----------------------------------------------------------------------------// -// build: resultados municipio-zona +// build: resultados uf //----------------------------------------------------------------------------//🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/resultados_uf.do` around lines 2 - 4, Update the top-of-file header comment that currently reads "resultados municipio-zona" to accurately describe the script as processing UF-level election results (for example, "resultados UF" or "resultados por UF"); search for the exact string "resultados municipio-zona" in the file and replace it with the corrected UF-level description and adjust surrounding comment lines for consistency.models/br_tse_eleicoes/code/sub/candidatos.do-170-172 (1)
170-172:⚠️ Potential issue | 🟡 MinorMerge result not validated.
The
merge 1:1withnogeneratesuppresses the_mergevariable, making it impossible to verify if all records matched. If records exist only inbasicoor only incomplementar, they may be silently mishandled.💡 Suggested validation
-merge 1:1 id_eleicao sequencial using `complementar', nogenerate +merge 1:1 id_eleicao sequencial using `complementar' +assert _merge == 3 | _merge == 1 // Allow unmatched from master +drop _mergeOr at minimum, log a warning for unmatched records.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/candidatos.do` around lines 170 - 172, The merge between `basico` and `complementar` uses "merge 1:1 id_eleicao sequencial using `complementar', nogenerate" which suppresses the _merge flag so you cannot validate matched/unmatched rows; remove the nogenerate option (or re-run the merge without it) so the _merge variable is created, then add post-merge checks on _merge in the same routine (e.g., count or loop over _merge==1, _merge==2, _merge==3) and log warnings for any unmatched records (or handle them explicitly) to ensure no rows were silently omitted.models/br_tse_eleicoes/code/sub/vagas.do-42-43 (1)
42-43:⚠️ Potential issue | 🟡 MinorSilent failure if input file is missing.
Using
capon both import attempts means the script continues silently if neither.txtnor.csvexists. This could lead to confusing errors downstream when variables are missing.💡 Suggested approach
cap import delimited using "input/consulta_vagas/consulta_vagas_`ano'/consulta_vagas_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear -cap import delimited using "input/consulta_vagas/consulta_vagas_`ano'/consulta_vagas_`ano'_`uf'.csv", delim(";") varn(nonames) stringcols(_all) clear +if _rc != 0 { + cap import delimited using "input/consulta_vagas/consulta_vagas_`ano'/consulta_vagas_`ano'_`uf'.csv", delim(";") varn(nonames) stringcols(_all) clear + if _rc != 0 { + di as error "Warning: No input file found for `ano'_`uf'" + continue + } +}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/vagas.do` around lines 42 - 43, The import block silently fails because both import delimited calls are prefixed with capture; replace the silent-capture pattern with an explicit existence check and a single import attempt: use fileexists("input/consulta_vagas/consulta_vagas_`ano'/consulta_vagas_`ano'_`uf'.txt") and fileexists(...".csv") to detect which file to load, call import delimited (without capture) on the existing file, and if neither exists call display as error (or error) and exit so the script fails fast; locate and update the two import delimited calls shown and add the fileexists checks and error/exit handling around them.
🧹 Nitpick comments (15)
models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do (3)
73-73: Remove commented debug code.The commented fragment
//"estado'" == "ac" &` appears to be leftover debug code and should be cleaned up.- if !("`partido'" == "avante") { //"`estado'" == "ac" & + if !("`partido'" == "avante") {🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do` at line 73, Remove the leftover debug comment from the conditional that checks the partido variable: in the if statement using "`partido'" (the line starting with if !("`partido'" == "avante") {), delete the trailing commented fragment //"`estado'" == "ac" & so the condition remains clean and syntactically correct; verify spacing and braces around the if ("`partido'" check) are preserved after removal.
51-55: Date conversion assumes DD/MM/YYYY format without validation.The substring-based date transformation assumes input is always in
DD/MM/YYYYformat. If the source data has inconsistent formats or invalid dates, this will produce malformedYYYY-MM-DDstrings silently.For the Python refactor, consider using proper date parsing with validation.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do` around lines 51 - 55, The current loop over varlist data_* uses brittle substring operations to build YYYY-MM-DD (replace `k' = substr...), which fails on non-DD/MM/YYYY inputs; update the refactor so each data_* field is parsed and validated (e.g., in Python use datetime.strptime inside a try/except or use dateutil.parser.parse with strict format checking) and only convert valid dates to ISO format (YYYY-MM-DD), otherwise set the field to a clear missing value (empty string or None); ensure this replacement logic is applied for each variable in the varlist data_* (the same loop that currently does the substr/replace) and include logging or a counter for malformed rows so invalid dates are detectable.
44-48: Silent data handling may mask issues.
- Line 44:
forceoption silently converts non-numeric values inid_municipio_tse,zona,secaoto missing (.)- Line 47:
drop if _merge == 2silently discards municipalities present in the directory but absent in the affiliation dataFor the Python refactor, consider adding logging/warnings when:
- Values fail numeric conversion
- Records are dropped due to unmatched municipality IDs
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/filiacao_partidaria.do` around lines 44 - 48, The destring/merge steps silently lose information; before converting id_municipio_tse, zona, secao to numeric, detect and log any non-numeric values (count and a small sample) and record their row IDs so they can be inspected (replace conversion should still proceed but with a warning), and when performing the merge with `diretorio` detect right-only keys (records present in `diretorio` but not in the affiliation table — the analog of _merge == 2), log the count and a small sample of those id_municipio_tse values and DO NOT silently drop them without a logged reason; implement these checks in the Python refactor around the functions that perform the numeric coercion (the destring equivalent for id_municipio_tse, zona, secao) and the merge/join step using `diretorio` so you can trace and review conversion failures and unmatched municipality IDs.models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do (2)
74-89: Duplicated hard-coded name corrections.These manual name corrections (lines 75-89) are duplicated verbatim in the v2 section (lines 242-256). Consider extracting these into a separate data file or lookup table to:
- Avoid duplication and potential sync issues
- Make corrections easier to maintain and audit
- Facilitate the planned Python conversion
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do` around lines 74 - 89, The duplicated hard-coded corrections for aux_primeira/aux_ultima (the multiple replace statements like replace aux_primeira = "elves" if aux_primeira == "elvis" & aux_ultima == "leite" and the identical block in the v2 section) should be moved into a single lookup source (e.g., a CSV/JSON or a shared dictionary) and both sections should load and apply that table instead of repeating literal replace lines; update the cria_id_candidato.do logic to read the shared corrections table and iterate applying corrections to aux_primeira and aux_ultima (use the existing symbols aux_primeira, aux_ultima and the replace operation) so future edits occur in one place and the same table can be re-used by the planned Python conversion.
1-162: Dead code: v1 and v2 outputs are not consumed by the active pipeline.Per the context from
normalizacao_particao.do(lines 50-60), the merge operations usingoutput/id_candidato_bd_v1.dtaandoutput/id_candidato_bd_v2.dtaare commented out. Onlytmp/cpf_titulo_eleitoral.dta(lines 336-353) is actively used.Given this file is in the
old/subdirectory and the PR objective mentions conversion to Python, consider:
- Adding a comment at the top of the file documenting that these variants are deprecated
- Removing the dead code if it won't be needed for the Python conversion, or
- Consolidating v1/v2 logic if both variants are needed for reference
Also applies to: 167-329
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/old/cria_id_candidato.do` around lines 1 - 162, This file produces id_candidato_bd_v1 (variable and file output "output/id_candidato_bd_v1.dta") but those v1/v2 artifacts are not consumed by the active pipeline; mark the script as deprecated and either remove or consolidate dead branches: add a top-of-file comment stating the file is deprecated/archival (mentioning id_candidato_bd_v1 and any id_candidato_bd_v2 references), then either delete the unused save/append blocks that write "output/id_candidato_bd_v1.dta" and related tmp/*.dta outputs (e.g., tmp/1a_rodada.dta, tmp/2a_rodada.dta) if they won't be needed for the Python conversion, or merge their logic into a single canonical path and update the final save to a single expected artifact name; ensure any renamed variable id_candidato_bd_v1 and the final save use the consolidated output name so downstream scripts reference one current file.models/br_tse_eleicoes/code/fnc/limpa_partido.do (1)
5-47: Normalize the party field inside this helper first.The current callers in
models/br_tse_eleicoes/code/sub/partidos.do:167-172,models/br_tse_eleicoes/code/sub/resultados_municipio_zona.do:201-220, andmodels/br_tse_eleicoes/code/sub/resultados_uf.do:168-180do not runclean_stringonsiglaorsigla_partido. Because every rule here is exact-match and case/whitespace sensitive, any new variant will bypass the mapping and fragment the party dimension.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/fnc/limpa_partido.do` around lines 5 - 47, The helper limpa_partido.do performs exact, case-sensitive replacements on `var' causing fragmented party values; normalize the input first by applying the project's clean_string (and/or trimming and uppercasing) to the party variable before any replace rules (i.e., call clean_string on `var' at the top of limpa_partido.do after local var `2' is set), then proceed with the existing conditional replace blocks (preserving `ano' checks and the same replacement targets) so all comparisons match normalized sigla/sigla_partido inputs.models/br_tse_eleicoes/code/fnc/limpa_candidato.do (1)
5-8: Make these key overrides conditional.These
replaces will overwrite future raw refreshes too. Sincecpf/titulo_eleitoralare padded inmodels/br_tse_eleicoes/code/sub/candidatos.do:220-252and then used as dedupe/merge keys inmodels/br_tse_eleicoes/code/sub/normalizacao_particao.do:23-38, a later source correction would be silently forced back to the hard-coded value. Prefer matching the known-bad source value or moving these exceptions into a small audited corrections table.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/fnc/limpa_candidato.do` around lines 5 - 8, The unconditional replace statements in limpa_candidato.do are forcing overrides that will persist across future raw refreshes; change each replace to be conditional on the field currently matching the known-bad source value (i.e., only replace cpf/titulo_eleitoral when the present value equals the incorrect original string) or move these exceptions into a small, audited corrections table applied during normalization (see candidatos.do padding logic and normalizacao_particao.do dedupe/merge usage) so fixes don’t silently reapply to corrected source data.models/br_tse_eleicoes/code/sub/resultados_uf.do (1)
53-53: Complex year iteration pattern is hard to follow.The numlist
1945 1947 1955(5)1965 1950(4)1990 1989creates a non-obvious sequence. Consider adding a comment listing the actual years processed, or using explicit year lists for clarity.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/resultados_uf.do` at line 53, The foreach loop using the numlist in the line "foreach ano of numlist 1945 1947 1955(5)1965 1950(4)1990 1989" is hard to read; update it to an explicit, easy-to-read year list or add an inline comment enumerating the exact years produced so future readers can immediately see the sequence. Locate the foreach that iterates the local "ano" and either expand the numlist into a plain space-separated list of years or append a clear comment (e.g., "// years: 1945 1947 1950 ...") describing the full set so the iteration is unambiguous.models/br_tse_eleicoes/code/sub/candidatos.do (1)
220-252: Complex identifier padding logic is difficult to maintain.The CPF and
titulo_eleitoralpadding logic handles many edge cases with repetitive conditional statements. Consider extracting this into a reusable helper program for clarity and testability.💡 Example helper approach
// In a separate helper file: cap program drop pad_identifier program pad_identifier args varname target_length forval i = 1/`=`target_length'-1' { local zeros = "0" * (`target_length' - `i') replace `varname' = "`zeros'" + `varname' if length(`varname') == `i' } end // Usage: pad_identifier cpf 11 pad_identifier titulo_eleitoral 12🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/candidatos.do` around lines 220 - 252, The current file contains repetitive, hard-to-maintain padding logic for cpf and titulo_eleitoral; extract this into a reusable helper program (e.g., pad_identifier) that accepts the variable name and target length, moves the repeated conditional/prefix logic into a loop, and then call pad_identifier for cpf (target 11) and titulo_eleitoral (target 12); update any special-case transformations for 12-length titulo_eleitoral (the substr-based space fixes) to run either inside pad_identifier when detecting internal spaces or in a small companion helper (e.g., normalize_titulo_with_internal_spaces) and replace the inline replace statements with calls to these helpers.models/br_tse_eleicoes/code/sub/perfil_eleitorado_local_votacao.do (2)
50-52: Inconsistent sentinel value handling.
telefone,latitude, andlongitudereplace-1with empty string"", while other scripts (e.g.,vagas.do,detalhes_votacao_secao.do) replace-1with missing.for numeric fields. Afterdestringon line 63-64,latitudeandlongitudebecome numeric, so empty strings become missing anyway, buttelefoneremains a string with empty values instead of missing.Consider using a consistent approach across all scripts.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_local_votacao.do` around lines 50 - 52, The sentinel "-1" is being replaced inconsistently: telefone is set to empty string "" while latitude and longitude are set to "" then later destringed (lines around the destring for latitude/longitude), causing telefone to remain a string rather than a numeric missing value as in other scripts; change the replacements to use the numeric missing marker for consistency — replace telefone, latitude, and longitude with . when the value equals "-1" so that after destring (see the destring block around latitude and longitude) all three fields are treated uniformly as missing numeric values.
54-62: Commented-out normalization code.Multiple
clean_stringcalls are commented out without explanation. If normalization is intentionally skipped (e.g., to preserve original formatting), add a comment explaining why. Otherwise, consider removing or enabling this code.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_local_votacao.do` around lines 54 - 62, Several normalization calls (clean_string) for fields tipo_secao_agregada, tipo, situacao, situacao_zona, situacao_secao, situacao_localidade, and situacao_secao_acessibilidade are commented out with no explanation; either re-enable them or remove them and document the decision. If normalization is required, uncomment the clean_string calls for those symbols (or call the appropriate normalizer function used elsewhere) so fields are normalized; if normalization is intentionally skipped to preserve original formatting, replace the commented block with a short explanatory comment stating why (e.g., "preserve original formatting for downstream matching") and remove the dead code to avoid confusion. Ensure the change references the same identifiers (clean_string and the listed field names) so reviewers can verify the intended behavior.models/br_tse_eleicoes/code/fnc/clean_string.do (2)
87-151: Consider removing commented-out code.The
clean_string_properprogram is entirely commented out. If it's not needed, consider removing it to reduce noise. If it may be useful later, document why it's disabled or move it to a separate file.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/fnc/clean_string.do` around lines 87 - 151, The file contains a fully commented-out program clean_string_proper (using proper(), usubinstr(), subinstr(), strtrim()) which should be removed or documented; either delete the entire commented block to reduce noise, or move it to a separate legacy/disabled file and add a short comment explaining why it’s retained (e.g., "legacy proper-case handling for non-ASCII chars") and when to re-enable, so maintainers can find clean_string_proper and related function calls (proper/usubinstr) easily.
64-64: Duplicate double-space replacement.The replacement of double spaces (
" "→" ") is performed twice (lines 64 and 76). A single pass after all other replacements would suffice.Also applies to: 76-76
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/fnc/clean_string.do` at line 64, The duplicate double-space replacement occurs twice for the same variable (`1') in clean_string.do; remove the earlier occurrence and keep a single qui replace `1' = subinstr(`1', " ", " ", .) after all other subinstr/replace operations so the double-space collapse is done once as a final cleanup (locate the two identical calls and delete the first, leaving the final pass).models/br_tse_eleicoes/code/sub/detalhes_votacao_secao.do (1)
106-119: Division by zero risk in proportion calculations.If
comparecimentois 0 or missing, the proportion calculations will produce missing or undefined values. This may be acceptable, but consider adding explicit handling or a note.🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_secao.do` around lines 106 - 119, The proportion calculations risk division by zero when comparecimento or aptos are zero/missing; update each generation to explicitly guard using conditional expressions: for proporcao_comparecimento use aptos>0 (e.g., gen proporcao_comparecimento = cond(aptos>0, 100 * comparecimento / aptos, .)), and for proporcao_votos_nominais/proporcao_votos_legenda/proporcao_votos_brancos/proporcao_votos_nulos use comparecimento>0 (e.g., gen proporcao_votos_nominais = cond(comparecimento>0, 100 * votos_nominais / comparecimento, .)); ensure you reference the existing variable names (proporcao_comparecimento, proporcao_votos_nominais, proporcao_votos_legenda, proporcao_votos_brancos, proporcao_votos_nulos, comparecimento, aptos, votos_nominais, votos_legenda, votos_brancos, votos_nulos) and preserve the label lines (la var ...) unchanged.models/br_tse_eleicoes/code/sub/partidos.do (1)
129-130: Fix the federation field typo before exporting it.
nome_federacacaolooks like a misspelling ofnome_federacao, and this file persists that name from 2022 onward. Fixing it later will be a breaking schema rename for downstream users.Suggested rename
- ren v20 nome_federacacao + ren v20 nome_federacao @@ - foreach k in nome_coligacao nome_federacacao sigla_federacao composicao_federacao nome_coligacao composicao_coligacao { + foreach k in nome_coligacao nome_federacao sigla_federacao composicao_federacao nome_coligacao composicao_coligacao {Also applies to: 162-163
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@models/br_tse_eleicoes/code/sub/partidos.do` around lines 129 - 130, The file renames contain a typo: change the field rename "ren v20 nome_federacacao" to "ren v20 nome_federacao" (and similarly correct the duplicated typo at the other occurrence around the block referenced as lines 162-163) so the exported schema uses nome_federacao consistently; update both rename statements in partidos.do and any nearby code that relies on v20 to ensure the correct field name is propagated without introducing a downstream-breaking rename.
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 3731dcd5-ce8c-4264-997e-75a959f5820f
📒 Files selected for processing (25)
models/br_tse_eleicoes/code/build.domodels/br_tse_eleicoes/code/fnc/clean_string.domodels/br_tse_eleicoes/code/fnc/limpa_candidato.domodels/br_tse_eleicoes/code/fnc/limpa_estado_civil.domodels/br_tse_eleicoes/code/fnc/limpa_instrucao.domodels/br_tse_eleicoes/code/fnc/limpa_partido.domodels/br_tse_eleicoes/code/fnc/limpa_resultado.domodels/br_tse_eleicoes/code/fnc/limpa_tipo_eleicao.domodels/br_tse_eleicoes/code/sub/agregacao.domodels/br_tse_eleicoes/code/sub/candidatos.domodels/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.domodels/br_tse_eleicoes/code/sub/detalhes_votacao_secao.domodels/br_tse_eleicoes/code/sub/detalhes_votacao_uf.domodels/br_tse_eleicoes/code/sub/normalizacao_particao.domodels/br_tse_eleicoes/code/sub/old/cria_id_candidato.domodels/br_tse_eleicoes/code/sub/old/filiacao_partidaria.domodels/br_tse_eleicoes/code/sub/partidos.domodels/br_tse_eleicoes/code/sub/perfil_eleitorado_local_votacao.domodels/br_tse_eleicoes/code/sub/perfil_eleitorado_municipio_zona.domodels/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.domodels/br_tse_eleicoes/code/sub/prestacao_contas.domodels/br_tse_eleicoes/code/sub/resultados_municipio_zona.domodels/br_tse_eleicoes/code/sub/resultados_secao.domodels/br_tse_eleicoes/code/sub/resultados_uf.domodels/br_tse_eleicoes/code/sub/vagas.do
| cap import delimited using "input/detalhe_votacao_munzona/detalhe_votacao_munzona_`ano'/detalhe_votacao_munzona_`ano'_`estado'.txt", delimiter(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited using "input/detalhe_votacao_munzona/detalhe_votacao_munzona_`ano'/detalhe_votacao_munzona_`ano'_`estado'.csv", delimiter(";") varn(nonames) stringcols(_all) clear |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's locate and read the file
find . -name "detalhes_votacao_municipio_zona.do" -type fRepository: basedosdados/pipelines
Length of output: 134
🏁 Script executed:
# Read the relevant section with context
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
cat -n "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" | sed -n '1,80p'
fiRepository: basedosdados/pipelines
Length of output: 4179
🏁 Script executed:
# Also check lines around 217-220 mentioned in the internal scratchpad
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
cat -n "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" | sed -n '210,230p'
fiRepository: basedosdados/pipelines
Length of output: 1064
🏁 Script executed:
# Check the full file length to understand scope
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
wc -l "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do"
fiRepository: basedosdados/pipelines
Length of output: 136
🏁 Script executed:
# Check the end of the file to see save/append logic
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
cat -n "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" | tail -70
fiRepository: basedosdados/pipelines
Length of output: 2773
🏁 Script executed:
# Also search for tempfile and save patterns
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
rg -n "tempfile|save|append" "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do"
fiRepository: basedosdados/pipelines
Length of output: 314
🏁 Script executed:
# Check specifically for any data validation after the imports
if [ -f "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" ]; then
cat -n "models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do" | sed -n '40,50p'
fiRepository: basedosdados/pipelines
Length of output: 706
Add check to ensure import succeeded before processing.
Both imports are wrapped in cap, silently suppressing errors if neither file exists for a given state/year. The code proceeds immediately to line 42 and beyond with no validation that data was actually loaded. If both imports fail, the previous iteration's data remains in memory, gets processed, and is saved to detalhes_ano'_estado'' at line 238–239 under the wrong state's name. This contaminated dataset then gets appended (lines 248–253) into the final output, silently corrupting data with no error indication.
Add a check after line 41 such as if _N == 0 error or similar to abort if neither file exists for the current state/year combination.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_municipio_zona.do` around
lines 40 - 41, After the two cap import delimited lines for
detalhe_votacao_munzona_`ano'_`estado' check whether data was actually loaded by
testing _N (e.g., if _N == 0 error "no input for `ano' `estado'"), so you abort
instead of processing stale data; reference the import commands and the output
target detalhes_`ano'_`estado' in the error message to make failures visible and
prevent contaminated datasets from being saved or appended.
| cap import delimited using "input/detalhe_votacao_uf/DETALHE_VOTACAO_UF_`ano'/DETALHE_VOTACAO_UF_`ano'_`uf'.txt", delimiter(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited using "input/detalhe_votacao_uf/DETALHE_VOTACAO_UF_`ano'/DETALHE_VOTACAO_UF_`ano'.txt", delimiter(";") varn(nonames) stringcols(_all) clear |
There was a problem hiding this comment.
Abort on missing UF inputs here as well.
These cap import delimited calls swallow file-not-found errors, but the loop immediately starts transforming whatever dataset is already in memory. If both paths fail for one ano/uf, the previous UF's data can be re-saved under the current partition.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_uf.do` around lines 38 - 39,
The two lines using "cap import delimited" for DETALHE_VOTACAO_UF_`ano'_`uf'.txt
and DETALHE_VOTACAO_UF_`ano'.txt must not silently swallow missing-file errors;
either remove the leading "cap" or add an explicit file existence check before
importing (use fileexists("input/detalhe_votacao_uf/...`ano'...`uf'.txt") /
fileexists("...`ano'.txt")) and if neither file exists call display/_error and
exit to abort processing for that ano/uf; update the import logic around the
import delimited calls in detalhes_votacao_uf.do so the code only proceeds when
a valid file was found and successfully imported.
| order ano turno id_eleicao tipo_eleicao data_eleicao sigla_uf cargo /// | ||
| secoes_anuladas secoes_sem_funcionamento zonas_eleitorais juntas_apuradoras votos_anulados_apurado_separado secoes_totalizadas /// | ||
| aptos comparecimento abstencoes votos_validos votos_brancos votos_nulos votos_legenda /// | ||
| proporcao_* |
There was a problem hiding this comment.
order references columns this script never creates.
The file only keeps/renames v3-v22, so id_eleicao and data_eleicao do not exist at this point. This order will stop the build before the final save. Either keep/rename those source columns earlier or remove them from this list.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/detalhes_votacao_uf.do` around lines 141 -
144, The ORDER clause references columns that aren't created (id_eleicao,
data_eleicao) after the keep/rename of v3-v22; either add those columns into the
earlier keep/rename step (so id_eleicao and data_eleicao are produced/renamed
from their source vX fields) or remove them from the order list; locate the
ORDER statement named "order" and either (A) update the keep/rename mapping that
converts v3-v22 to include the mappings that produce id_eleicao and
data_eleicao, or (B) delete id_eleicao and data_eleicao from the order list so
only existing columns (the renamed v3-v22 names and proporcao_*) remain.
| use "output/norm_candidatos.dta", clear | ||
| keep if mod(ano, 4) == 0 | ||
| keep ano tipo_eleicao sigla_uf id_municipio_tse cargo titulo_eleitoral numero | ||
| tempfile candidatos_mod0 | ||
| save `candidatos_mod0' | ||
|
|
||
| use "output/norm_candidatos.dta", clear | ||
| keep if mod(ano, 4) == 2 & cargo != "presidente" | ||
| keep ano tipo_eleicao sigla_uf cargo titulo_eleitoral numero | ||
| tempfile candidatos_mod2_estadual | ||
| save `candidatos_mod2_estadual' | ||
|
|
||
| use "output/norm_candidatos.dta", clear | ||
| keep if mod(ano, 4) == 2 & cargo == "presidente" | ||
| keep ano tipo_eleicao cargo titulo_eleitoral numero | ||
| tempfile candidatos_mod2_presid | ||
| save `candidatos_mod2_presid' |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, let's check the file structure and read the relevant lines
cd models/br_tse_eleicoes/code/sub/ 2>/dev/null && pwd || echo "Directory not found, searching for the file"
find . -name "normalizacao_particao.do" 2>/dev/null | head -5Repository: basedosdados/pipelines
Length of output: 143
🏁 Script executed:
# Read lines 763-779 to see the current code
sed -n '763,779p' models/br_tse_eleicoes/code/sub/normalizacao_particao.doRepository: basedosdados/pipelines
Length of output: 677
🏁 Script executed:
# Read lines 885-901 to see the second occurrence
sed -n '885,901p' models/br_tse_eleicoes/code/sub/normalizacao_particao.doRepository: basedosdados/pipelines
Length of output: 677
🏁 Script executed:
# Search for where these temporary files are used (candidatos_mod0, candidatos_mod2_estadual, candidatos_mod2_presid)
rg "candidatos_mod0|candidatos_mod2_estadual|candidatos_mod2_presid" models/br_tse_eleicoes/code/sub/normalizacao_particao.do -A 3 -B 1Repository: basedosdados/pipelines
Length of output: 4339
🏁 Script executed:
# Check prestacao_contas.do for how numero_candidato is imported
find . -name "prestacao_contas.do" -type fRepository: basedosdados/pipelines
Length of output: 119
🏁 Script executed:
# Check prestacao_contas.do for numero_candidato import with stringc
grep -n "numero_candidato\|stringc" models/br_tse_eleicoes/code/sub/prestacao_contas.do | head -20Repository: basedosdados/pipelines
Length of output: 1144
🏁 Script executed:
# Look for the complete import section in prestacao_contas.do
sed -n '1,100p' models/br_tse_eleicoes/code/sub/prestacao_contas.do | grep -A 5 -B 5 "stringc\|numero"Repository: basedosdados/pipelines
Length of output: 667
🏁 Script executed:
# Let's search for where the 763-779 temp files are used - look for the merge operations
sed -n '760,850p' models/br_tse_eleicoes/code/sub/normalizacao_particao.do | grep -A 10 "merge\|candidatos_mod"Repository: basedosdados/pipelines
Length of output: 1438
🏁 Script executed:
# Check what data is being merged in from the finance files around the merge operations
sed -n '750,1050p' models/br_tse_eleicoes/code/sub/normalizacao_particao.do | grep -B 5 "merge.*candidatos_mod0" | head -30Repository: basedosdados/pipelines
Length of output: 418
Add tostring numero, replace before saving candidate temporary files for finance merges.
The temporary files created at these locations are used in finance merge operations (receitas_candidato and despesas_candidato) where numero_candidato is imported as a string via stringc(_all) in prestacao_contas.do and renamed to numero. Without converting the candidate lookup tables to string type first, these merges will fail due to type mismatch.
Suggested fix
use "output/norm_candidatos.dta", clear
keep if mod(ano, 4) == 0
keep ano tipo_eleicao sigla_uf id_municipio_tse cargo titulo_eleitoral numero
+tostring numero, replace
tempfile candidatos_mod0
save `candidatos_mod0'
use "output/norm_candidatos.dta", clear
keep if mod(ano, 4) == 2 & cargo != "presidente"
keep ano tipo_eleicao sigla_uf cargo titulo_eleitoral numero
+tostring numero, replace
tempfile candidatos_mod2_estadual
save `candidatos_mod2_estadual'
use "output/norm_candidatos.dta", clear
keep if mod(ano, 4) == 2 & cargo == "presidente"
keep ano tipo_eleicao cargo titulo_eleitoral numero
+tostring numero, replace
tempfile candidatos_mod2_presid
save `candidatos_mod2_presid'Also applies to: 885-901
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/normalizacao_particao.do` around lines 763 -
779, Temporary candidate tempfiles are saving the variable numero as numeric
which breaks later finance merges; before each save of the tempfiles
candidatos_mod0, candidatos_mod2_estadual, and candidatos_mod2_presid (the
blocks that use "output/norm_candidatos.dta" and then save `candidatos_mod0' /
`candidatos_mod2_estadual' / `candidatos_mod2_presid') run tostring numero,
replace to convert numero to string; apply the same change in the analogous
block around the later section referenced (lines 885-901) so the temp lookup
tables match numero_candidato imported as string in prestacao_contas.do.
| cap import delimited "input/consulta_coligacao/CONSULTA_LEGENDA_`ano'/CONSULTA_LEGENDA_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/consulta_coligacao/consulta_legendas_`ano'/CONSULTA_LEGENDA_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/consulta_coligacao/consulta_legendas_`ano'/consulta_legendas_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/consulta_coligacao/consulta_legendas_`ano'/consulta_legendas_`ano'_`uf'.csv", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/consulta_coligacao/consulta_coligacao_`ano'/consulta_coligacao_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/consulta_coligacao/consulta_coligacao_`ano'/consulta_coligacao_`ano'_`uf'.csv", delim(";") varn(nonames) stringcols(_all) clear |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -name "partidos.do" -type fRepository: basedosdados/pipelines
Length of output: 111
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && wc -l partidos.doRepository: basedosdados/pipelines
Length of output: 81
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && sed -n '30,60p' partidos.doRepository: basedosdados/pipelines
Length of output: 1554
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && sed -n '1,80p' partidos.doRepository: basedosdados/pipelines
Length of output: 3912
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && sed -n '41,120p' partidos.doRepository: basedosdados/pipelines
Length of output: 2529
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && sed -n '120,180p' partidos.doRepository: basedosdados/pipelines
Length of output: 1633
🏁 Script executed:
cd models/br_tse_eleicoes/code/sub && sed -n '175,224p' partidos.doRepository: basedosdados/pipelines
Length of output: 1408
Add an error check after the fallback imports to prevent silent data corruption.
The six cap import delimited calls suppress all errors, allowing the code to proceed even if all imports fail. When none of the paths exist for the current (ano, uf) pair, Stata silently retains the previous UF's dataset in memory. The schema-mapping block then executes on this stale data and saves it to tempfile partidos_uf'_ano'', corrupting that year's append with wrong state data.
Verify that at least one import succeeded before proceeding to schema mapping (e.g., if _N > 0 or explicit nobs check), or fail fast if all paths are missing.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/partidos.do` around lines 41 - 46, After the
series of fallback cap import delimited attempts, add a check that the import
actually loaded data (e.g., verify _N>0 or use c(N) / count) before running the
schema-mapping block and saving to tempfile partidos_`uf'_`ano'; if no rows were
loaded, abort or skip this (ano, uf) with an explicit error/log message and
non-zero exit so you don't process stale memory from the previous UF. Place this
check immediately after the final cap import delimited command (the block of six
cap import delimited lines) and ensure the code branches to skip mapping/saving
when the dataset is empty.
| cap import delimited "input/perfil_eleitorado_secao/perfil_eleitor_secao_`ano'_`uf'.txt", delim(";") stringcols(_all) clear | ||
| cap import delimited "input/perfil_eleitorado_secao/perfil_eleitor_secao_`ano'_`uf'.csv", delim(";") stringcols(_all) clear | ||
|
|
||
| } | ||
| else { | ||
|
|
||
| cap import delimited "input/perfil_eleitorado_secao/perfil_eleitor_secao_`ano'_`uf'/perfil_eleitor_secao_`ano'_`uf'.txt", delim(";") stringcols(_all) clear | ||
| cap import delimited "input/perfil_eleitorado_secao/perfil_eleitor_secao_`ano'_`uf'/perfil_eleitor_secao_`ano'_`uf'.csv", delim(";") stringcols(_all) clear |
There was a problem hiding this comment.
Abort the iteration when neither raw file exists.
Both imports are under cap, but the loop never checks whether one succeeded before it starts keep/rename. If both paths miss, the dataset already in memory from the previous UF is transformed and saved again under the current ano/uf.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/perfil_eleitorado_secao.do` around lines 35 -
42, The loop uses suppressed imports (cap import delimited) but never checks if
either import actually succeeded before running the subsequent keep/rename that
assume a dataset is loaded; add explicit checks after the two cap import
delimited attempts: capture the return codes (e.g., store _rc into locals like
rc1 and rc2 after each cap), and if both indicate failure (both nonzero), abort
this iteration (use continue or skip the downstream keep/rename/save logic) and
log an informative message; otherwise proceed only when at least one import
succeeded. Ensure these checks surround the block that performs keep/rename so
you never transform the previous UF’s in-memory data when both files are
missing.
| cap import delimited "input/bem_candidato/bem_candidato_`ano'/bem_candidato_`ano'_`uf'.txt", clear delim(";") varnames(nonames) stringc(_all) | ||
| cap import delimited "input/bem_candidato/bem_candidato_`ano'/bem_candidato_`ano'_`uf'.csv", clear delim(";") varnames(nonames) stringc(_all) |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
find . -type f -name "prestacao_contas.do" | head -20Repository: basedosdados/pipelines
Length of output: 119
🏁 Script executed:
cat -n ./models/br_tse_eleicoes/code/sub/prestacao_contas.do | head -100Repository: basedosdados/pipelines
Length of output: 3472
🏁 Script executed:
cat -n ./models/br_tse_eleicoes/code/sub/prestacao_contas.do | sed -n '11,80p'Repository: basedosdados/pipelines
Length of output: 2149
Add data validation after fallback imports to prevent stale state from previous UF iteration.
Both cap import delimited commands on lines 13–14 suppress errors, but if both fail, the previous UF's data remains in memory and gets processed and saved as the current UF's dataset at line 69. This causes data corruption when combined at lines 74–79.
The script should either explicitly clear data before imports, validate that at least one import succeeded, or fail fast on missing files.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/prestacao_contas.do` around lines 13 - 14,
Before attempting the fallback imports, explicitly clear the in-memory dataset
and validate that an import succeeded: run a plain clear (to remove previous UF
data) before the two cap import delimited attempts; after each cap import check
_rc and set a flag (e.g., local imported = 1) if successful; if neither import
succeeded (imported missing or _rc != 0 for both) abort or skip processing for
that UF (display an error and continue) so you never reach the subsequent
save/processing steps that write the stale data used later in the save/combine
logic (the save at the end of the UF loop and the combine block that follows).
| if `ano' == 1998 & "`uf'" != "BRASIL" { | ||
| keep if cargo == "presidente" | ||
| } |
There was a problem hiding this comment.
The 1998 party filter is inverted.
This is the opposite of the candidate handling at Lines 223-230. As written, non-BRASIL files keep only presidente while the BRASIL file stays unfiltered, which will either duplicate presidential rows or discard non-presidential state results.
Suggested fix
- if `ano' == 1998 & "`uf'" != "BRASIL" {
- keep if cargo == "presidente"
- }
+ if `ano' == 1998 {
+ if "`uf'" == "BRASIL" {
+ keep if cargo == "presidente"
+ }
+ else {
+ drop if cargo == "presidente"
+ }
+ }🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/resultados_municipio_zona.do` around lines
440 - 442, The 1998 party filter is inverted: change the condition so only the
BRASIL (national) file is filtered to keep presidente; specifically, in the if
that currently reads checking `ano' == 1998 & "`uf'" != "BRASIL", flip the uf
comparison to == ("`uf'" == "BRASIL") so the block that runs keep if cargo ==
"presidente" applies only to the BRASIL case (variables: ano, uf, cargo).
| cap import delimited "input/votacao_secao/votacao_secao_`ano'_`uf'/votacao_secao_`ano'_`uf'.txt", delim(";") varn(nonames) stringcols(_all) clear | ||
| cap import delimited "input/votacao_secao/votacao_secao_`ano'_`uf'/votacao_secao_`ano'_`uf'.csv", delim(";") varn(nonames) stringcols(_all) clear |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
# First, check file existence and size
wc -l models/br_tse_eleicoes/code/sub/resultados_secao.doRepository: basedosdados/pipelines
Length of output: 121
🏁 Script executed:
# Read the file with context around lines 47-48
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | head -70Repository: basedosdados/pipelines
Length of output: 3648
🏁 Script executed:
# Look for the broader loop structure and any validation after these imports
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | sed -n '40,100p'Repository: basedosdados/pipelines
Length of output: 2002
🏁 Script executed:
# Check for any validation or error checking after the imports
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | sed -n '45,120p'Repository: basedosdados/pipelines
Length of output: 2538
🏁 Script executed:
# Search for any `scalar` or `if` statements that might check dataset state
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | grep -E "(nobs|_N|if \(c\(|error|confirm)"Repository: basedosdados/pipelines
Length of output: 48
🏁 Script executed:
# Check the end of the loop and how data is handled
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | sed -n '115,160p'Repository: basedosdados/pipelines
Length of output: 1643
🏁 Script executed:
# Look for where data is saved/appended in the loop
cat -n models/br_tse_eleicoes/code/sub/resultados_secao.do | tail -100Repository: basedosdados/pipelines
Length of output: 3497
Validate that at least one section-result file was successfully imported before running transformations.
Both imports use cap to suppress errors. If neither the .txt nor .csv file exists, both import commands fail silently and the previous iteration's dataset persists in memory. The subsequent transforms (lines 50–90+) then run against this stale data, relabeling it with the current loop's ano and uf values before saving to tmp/resultados_secao_uf'.dta`. This mislabeled data is then split and appended to the final output, corrupting the results.
Add a validation check (e.g., if _N == 0) after both import attempts to halt execution when neither file exists.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@models/br_tse_eleicoes/code/sub/resultados_secao.do` around lines 47 - 48,
After the two cap import delimited attempts for votacao_secao_`ano'_`uf' (.txt
and .csv), add a validation that the dataset in memory is non-empty (e.g., check
_N) and abort the loop/run if no rows were imported; specifically, after the
import commands referencing votacao_secao_`ano'_`uf' check if _N == 0 and call
an appropriate stop/exit/continue to avoid running the subsequent transforms
that relabel and save stale data for the current ano and uf.
|
Fechando pois decidimos não subir código de Stata nesse repositório. O PR #1476 já subindo o código refatorado em Python. |
3 participants
Subindo para o repositório
pipelineso que antes estava só emsdk(em https://github.com/basedosdados/sdk/tree/master/bases/br_tse_eleicoes/code).Esse código é rodado na minha máquina a partir de todos os dados baixados do repositório de dados abertos do TSE. Eu cheguei a subir todos os dados raw no nosso storage. Se quisermos manter esses dados guardados como arquivo (acho que deveríamos), eu precisaria atualizá-los até 2024.
Se formos refatorar para Python, será a partir desse código.
Summary by CodeRabbit