Add BAM-MP-core Preview model#371
Open
Godorina wants to merge 6 commits into
Open
Conversation
# Conflicts: # site/src/figs/cumulative-precision-recall.jsonl
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Run ID: 📒 Files selected for processing (1)
🚧 Files skipped from review as they are similar to previous changes (1)
📝 WalkthroughWalkthroughAdds BAM-MP-core to the benchmark through model registration, a YAML model card, discovery and thermal-conductivity evaluation scripts, and BAM-MP-core records in site metric datasets. ChangesBAM-MP-core model addition
Estimated code review effort: 4 (Complex) | ~60 minutes Possibly related PRs
Suggested labels: Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 2
🧹 Nitpick comments (2)
models/BAM/test_bam_mp_core_kappa.py (1)
175-197: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winRecord the model package version in
run_paramsfor reproducibility.
versionscaptures onlynumpyandtorch. For a benchmark submission, the model package (bam-torch) and ideallyase/phono3pyversions materially affect reproducibility of the kappa numbers.♻️ Suggested change
- "versions": {dep: version(dep) for dep in ("numpy", "torch")}, + "versions": { + dep: version(dep) + for dep in ("numpy", "torch", "ase", "phono3py", "bam-torch") + },🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@models/BAM/test_bam_mp_core_kappa.py` around lines 175 - 197, The run_params metadata in test_bam_mp_core_kappa.py is missing the model package version, so update the versions field in the run_params dictionary to also record bam-torch and, if available in this benchmark environment, ase and phono3py alongside numpy and torch. Use the existing version(...) pattern in the run_params construction so the resulting run_params.json captures all key package versions needed for reproducibility.models/BAM/test_bam_mp_core_discovery.py (1)
93-124: 🗄️ Data Integrity & Integration | 🔵 Trivial | ⚡ Quick winCheckpoint download has no checksum verification.
ensure_checkpointonly checksContent-Lengthmatches the downloaded size, unlikeDataFiles.path(enums.pymaybe_auto_download_file(..., md5=expected_md5)) which verifies an md5 checksum. A truncated-but-matching-length or corrupted download of the model checkpoint would silently proceed to load a bad checkpoint. Consider adding an md5/sha256 check against a known value, mirroring the existingDataFilespattern in the codebase.🛡️ Sketch of adding checksum verification
-def ensure_checkpoint(path: Path) -> Path: +def ensure_checkpoint(path: Path, expected_md5: str | None = None) -> Path: if path.is_file(): print(f"Using BAM-MP-core checkpoint: {path}") return path ... tmp_path.replace(path) finally: tmp_path.unlink(missing_ok=True) + if expected_md5 is not None: + import hashlib + actual = hashlib.md5(path.read_bytes()).hexdigest() # noqa: S324 + if actual != expected_md5: + path.unlink(missing_ok=True) + raise OSError(f"Checkpoint md5 mismatch: expected {expected_md5}, got {actual}") print("Download complete.") return path🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@models/BAM/test_bam_mp_core_discovery.py` around lines 93 - 124, The ensure_checkpoint helper currently validates only file size, so a corrupted BAM checkpoint can still be accepted. Update ensure_checkpoint to verify the downloaded file against a known checksum (md5 or sha256) before replacing the final path, ideally by mirroring the existing DataFiles path / maybe_auto_download_file pattern in enums.py and using the same checksum value source for the BAM checkpoint. Keep the current Content-Length and empty-file checks, but add a checksum validation step in ensure_checkpoint so bad downloads are rejected before the checkpoint is loaded.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@models/BAM/test_bam_mp_core_discovery.py`:
- Around line 358-406: The per-structure blanket exception handling in the
relaxation loop can hide a systemic failure and let the job finish with almost
no successful structures. Update the try/except around the
FIRE/FrechetCellFilter relaxation block to fail fast when failures become
repetitive or exceed an early threshold, using the existing n_failed/n_success
counters in the main loop to detect a run-wide problem. Keep the first few
traceback prints for debugging, but re-raise or abort after too many consecutive
or total failures so the workflow cannot continue silently into downstream
artifact writing with an empty result set.
- Around line 371-374: The relaxation test currently relies on
dyn.optimizable.gradient_norm(...), which depends on ASE’s internal Optimizable
API and can fail on older ase releases. Update the test to avoid that
dyn.optimizable dependency by using only stable Optimizer/Atoms methods, or
explicitly raise the minimum ASE version so the Optimizable interface is
guaranteed. Keep the existing dyn.get_number_of_steps() usage and adjust the
final_fmax calculation in the BAM test helper accordingly.
---
Nitpick comments:
In `@models/BAM/test_bam_mp_core_discovery.py`:
- Around line 93-124: The ensure_checkpoint helper currently validates only file
size, so a corrupted BAM checkpoint can still be accepted. Update
ensure_checkpoint to verify the downloaded file against a known checksum (md5 or
sha256) before replacing the final path, ideally by mirroring the existing
DataFiles path / maybe_auto_download_file pattern in enums.py and using the same
checksum value source for the BAM checkpoint. Keep the current Content-Length
and empty-file checks, but add a checksum validation step in ensure_checkpoint
so bad downloads are rejected before the checkpoint is loaded.
In `@models/BAM/test_bam_mp_core_kappa.py`:
- Around line 175-197: The run_params metadata in test_bam_mp_core_kappa.py is
missing the model package version, so update the versions field in the
run_params dictionary to also record bam-torch and, if available in this
benchmark environment, ase and phono3py alongside numpy and torch. Use the
existing version(...) pattern in the run_params construction so the resulting
run_params.json captures all key package versions needed for reproducibility.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: bd5bfe0f-70c8-4cd1-93d3-70e6baa29a64
📒 Files selected for processing (15)
matbench_discovery/enums.pymodels/BAM/bam-mp-core.ymlmodels/BAM/test_bam_mp_core_discovery.pymodels/BAM/test_bam_mp_core_kappa.pysite/src/figs/box-hull-dist-errors.jsonlsite/src/figs/cumulative-precision-recall.jsonlsite/src/figs/element-prevalence-vs-error.jsonlsite/src/figs/hist-clf-pred-hull-dist.jsonlsite/src/figs/hist-largest-each-errors-fp-diff.jsonlsite/src/figs/kappa-103-analysis.jsonlsite/src/figs/roc-models.jsonlsite/src/figs/rolling-mae-vs-hull-dist.jsonlsite/src/figs/scatter-largest-each-errors-fp-diff.jsonlsite/src/figs/scatter-largest-fp-diff-each-error.jsonlsite/src/routes/models/per-element-each-errors.jsonl
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Could a maintainer please apply the
ingest-modellabel to run the full upstream ingest forbam-mp-core?This PR replaces/continues #355, and the branch has been updated with latest
main.Checklist
Please open PRs as
draftand only mark asready to reviewafter checking off the following items:models/BAM/bam-mp-core.ymlfor my submission.BAMis the name of the architecture andbam-mp-core.ymlincludes things like author details, training set names and important hyperparameters.Model.bam_mp_coreenum on theModelenum inenums.py.2026-05-09-wbm-IS2RE.csv.gz).2026-05-09-wbm-geo-opt-FIRE.jsonl.gz).2026-05-09-kappa-103-FIRE-dist=0.03-fmax=1e-4-symprec=1e-5.json.gz).<yyyy-mm-dd>-diatomics.json.gz). Not submitted for BAM-MP-core.models/BAM/bam-mp-core.yml).test_bam_mp_core_discovery.pyandtest_bam_mp_core_kappa.py) that generated the prediction files.uv run python scripts/ingest_model.py bam-mp-coreto check metadata and generate the plots needed for submission.uvx prek) and ensured all checks are passing.Marking your PR as
ready to reviewwill trigger an automated code review, please address any issues raised by the automated review. For the maintainers minimizing the final human review process enables us to merge your submissions much faster!Additional Information (Optional)
train_<arch_name>.py) if I trained a model specifically for this benchmark.readme.mdwith additional details about my model.Summary by CodeRabbit