Conversation
…cessing utilities - Implemented GK2A data model for converting GK2A data to UWISC format. - Added NASA data model for converting NASA data to UWISC format. - Introduced shared utilities in base_data_model.py for preprocessing tasks. - Created pytest fixtures for testing in conftest.py. - Added tests for base data model and GK2A data model functionalities.
… and timestamp improvements refactor: update method names for clarity and consistency test: add tests for GK2A output filename and improve existing test structure
Contributor
There was a problem hiding this comment.
Pull request overview
Adds a shared BaseUwiscDataModel preprocessing pipeline plus a new GK2A converter, and refactors the existing NasaDataModel to inherit from it. Also adds a multi-pattern CLI helper, pytest fixtures, and unit tests for the new helpers.
Changes:
- New
nsrdb/preprocessing/base_data_model.pywith the shared conversion pipeline,expand_input_patterns, andrun_data_model_jobs(supports multiple recursive glob patterns and grouped inputs). - New
nsrdb/preprocessing/gk2a_data_model.pyhandling per-channel files, raw count → radiance/temperature/reflectance conversion, 4× coarsening, GK2A timestamp parsing, and cloud-phase remapping. NasaDataModelrefactored onto the shared base; NASA CLI now acceptsnargs='+'patterns; minor docstring/TODO touch-ups inspa.pyandvariable_factory.py.
Reviewed changes
Copilot reviewed 8 out of 8 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| nsrdb/preprocessing/base_data_model.py | Adds shared UWISC conversion pipeline, timestamp parsing, dim/coord normalization, batch runner. |
| nsrdb/preprocessing/gk2a_data_model.py | New GK2A-specific subclass, multi-file combining/coarsening, timestamp grouping. |
| nsrdb/preprocessing/nasa_data_model.py | Refactored to subclass the shared base; CLI accepts multiple input patterns. |
| nsrdb/solar_position/spa.py | Docstring fixes for azimuth() / zenith(). |
| nsrdb/data_model/variable_factory.py | Adds a TODO comment on the variable mapping. |
| tests/preproc/conftest.py | New shared pytest fixtures. |
| tests/preproc/test_base_data_model.py | Tests for dim remapping and multi-pattern job dispatch. |
| tests/preproc/test_gk2a_data_model.py | Tests for grouping, naming, file combining/coarsening, cloud-phase remap. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+181
to
+195
| def get_solar_zenith(self, ds): | ||
| """Derive the solar zenith angle for the dataset.""" | ||
| lats = ds['latitude'].values | ||
| lons = ds['longitude'].values | ||
| return SolarPosition._zenith( | ||
| self.time_index, lats, lons | ||
| ) | ||
|
|
||
| def get_solar_azimuth(self, ds): | ||
| """Derive the solar azimuth angle for the dataset.""" | ||
| lats = ds['latitude'].values | ||
| lons = ds['longitude'].values | ||
| return SolarPosition._azimuth( | ||
| self.time_index, lats, lons | ||
| ) |
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
…tion from hpc node failures
…3325684192 Use public `SolarPosition` API for UWISC solar-angle derivation
github-actions Bot
pushed a commit
that referenced
this pull request
Jun 8, 2026
Bespoke data model for GK2A data. NASA and GK2A data models now subclass base model. Eventually this should be refactored to rely more on nsrdb_vars.csv for variable definitions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
…cessing utilities