Feat/prefect3 flows migration#1553
Draft
folhesgabriel wants to merge 57 commits into
Draft
Conversation
- ibge_inflacao/_run_ibge_inflacao: gate is_outdated check behind force_run - br_ibge_ipca, br_ibge_inpc factories: propagate force_run - guia de migração: documenta o padrão e a ressalva de verbosidade - flows migrados: marca os 8 flows IBGE como pendentes de re-deploy
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
📝 WalkthroughWalkthroughThis PR executes a comprehensive migration of the Prefect orchestration framework from 0.x/2.x to Prefect 3. Changes span dependency updates, utilities refactoring into focused modules, deployment infrastructure (Docker image, CI/CD workflows, deployment scripts), shared task updates, and individual flow migrations with renamed objects and inline scheduling. ChangesPrefect 3 Migration
Sequence DiagramssequenceDiagram
participant dev as Developer
participant gh as GitHub
participant deploy as deploy_flows.py
participant prefect as Prefect API
participant gcs as GCS/Storage
gh->>gh: Push to main or feat/prefect3
gh->>deploy: Trigger cd-prefect3 workflow
deploy->>deploy: load_flows_from_file(path)
deploy->>prefect: flow.from_source(repo, branch).deploy(pool_name, schedule)
prefect->>gcs: Store flow definition
prefect-->>deploy: deployment_id
deploy-->>gh: Success/fail exit status
sequenceDiagram
participant Flow as Prefect Flow
participant Task as run_dbt task
participant dbtRunner as dbtRunner
participant GCS as GCS Artifacts
Flow->>Task: run_dbt(dataset_id, model)
Task->>Task: Validate dbt_command
Task->>Task: Resolve model path
Task->>dbtRunner: invoke(run_results_path, ...)
dbtRunner-->>Task: InvocationResult
Task->>GCS: Upload dbt artifacts (finally block)
Task-->>Flow: Success or raise exception
Estimated code review effort🎯 4 (Complex) | ⏱️ ~75 minutes Possibly related issues
Suggested labels
🚥 Pre-merge checks | ✅ 3 | ❌ 2❌ Failed checks (2 warnings)
✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Contributor
|
@folhesgabriel esse pull request tem conflitos 😩 |
for more information, see https://pre-commit.ci
- Refactor br_inmet_bdmep__microdados flow with canonical Prefect 3 template (force_run guard, on_failure Discord hook, deploy_schedules cron) - Move tasks/utils/constants from pipelines/datasets/br_inmet_bdmep/ to pipelines/crawler/inmet_bdmep/ to avoid pipelines.datasets.__init__ poisoning - Drop legacy schedules.py (replaced by flow.deploy_schedules) - Add README to pipelines/crawler/ibge_inflacao explaining the IBGE SIDRA API request cap (100k values) and why force_run=false is required for inflation flows - Update prefect3-flows-migrados.md: mark IPCA/INPC flows as Dev OK (validated via 4a only); add br_inmet_bdmep__microdados as Pronto p/ deploy
- Refactor to factory pattern over 4 tables (microdados, densidade_municipio, densidade_brasil, densidade_uf). - Shared _run_anatel_banda_larga_fixa in pipelines/crawler/anatel/banda_larga_fixa/flows.py. - Update tracking and runbook (dbt failures after staging upload are non-blocking for migration validation).
…atasets/__init__ trigger
…ublica_distribuicao as dev ok
…br_ms_sinan as dev ok
…_rais/tse to prefect 3
for more information, see https://pre-commit.ci
…age to avoid 403 on requester-pays user_project
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Centralizar a migração de flows do prefect 0.15 p/ 3.0
Summary by CodeRabbit
New Features
Documentation
Chores