Merge Feature/v0.7.5 into main#46
Merged
Merged
Conversation
A malformed data entry whose RVA falls outside any section currently propagates pefile.PEFormatError out of build_resource_structure. Catch it narrowly and emit -1 as the raw_offset sentinel; the validator's existing data_raw < 0 arm already maps this to RESOURCE_DATA_OUT_OF_BOUNDS. No schema change; no new reason codes.
Add depth parameter to validate_directory, flag depth-2 entries that
use names instead of LCIDs, and flag data entries that appear outside
the Language layer.
New reason codes:
RESOURCE_DIRECTORY_LANGUAGE_NOT_ID
RESOURCE_DATA_AT_INVALID_DEPTH
parser(version_info): add deterministic VS_VERSIONINFO extractor
Independent module; locates the first RT_VERSION leaf under a stable
(name_id, language_id) ordering and decodes VS_VERSIONINFO, VS_FIXEDFILEINFO,
StringFileInfo and VarFileInfo purely from bytes. Never raises; emits
tombstone tags for sub-structure failures.
validator(version_info): structural validation against decoded blob
Emits RESOURCE_VERSIONINFO_INVALID_{HEADER,FIXEDINFO,STRINGFILEINFO,VARFILEINFO}
when the decoded blob is malformed. Absence of RT_VERSION is not treated
as a defect.
…or_version_info. - 100% line coverage on the resource validator additions. - Defensive-path coverage for every except clause via monkeypatched struct.error injection and narrow-except negative tests.
New module independent of pefile's DIRECTORY_ENTRY_EXPORT interpretation. Uses pefile only to locate the export directory and read raw bytes via pe.get_data; all structural fields are decoded directly via struct.unpack_from. Deterministic extraction of: - 40-byte IMAGE_EXPORT_DIRECTORY header - Export Address Table (EAT) - Export Name Pointer Table (ENPT) - Export Ordinal Table (EOT) - Per-function name resolution and forwarder detection by RVA range - Per-entry ASCII name string reading with bounded scan length - Forwarder string structural validation against DllName.SymbolName and DllName.#Ordinal grammar via a single conservative regex Determinism guarantees: - Bounded array reads with per-position fallback to None on truncation - All loops bounded by declared counts; no input can produce unbounded iteration - Sub-structure failures emit deterministic tombstone tags in truncations[] and per-entry errors[] rather than raising - Stable output regardless of platform or pefile version Never raises; returns None for absent export directory, or a dict shaped per the documented output contract. Refs: requirement 2 (export table refinement)
Maps parser_exports output to ten new reason codes: Directory anomalies: - EXPORT_DIRECTORY_INVALID_HEADER: malformed header or inconsistent declared counts vs RVAs - EXPORT_DIRECTORY_OUT_OF_BOUNDS: directory extends past SizeOfImage - EXPORT_TABLE_TRUNCATED: declared sub-table size exceeded available bytes Name pointer anomalies: - EXPORT_NAME_RVA_INVALID: name RVA missing, zero, or string unreadable - EXPORT_NAME_NOT_ASCII: name decoded but contains non-printable bytes - EXPORT_NAME_POINTER_TABLE_UNSORTED: ENPT violates PE-spec sort order - EXPORT_NAME_ORDINAL_INDEX_INVALID: EOT entry missing or >= NumberOfFunctions Function entry anomalies: - EXPORT_ORDINAL_OUT_OF_RANGE: Base + NumberOfFunctions - 1 exceeds u16 - EXPORT_FUNCTION_RVA_INVALID: address RVA exceeds SizeOfImage - EXPORT_FORWARDER_MALFORMED: forwarder unreadable or grammar violation Implementation choices: - Priority-resolved sub-reasons in details[reason] for name RVA and name encoding error classes, eliminating double-emission on entries with multiple matching parser tags - Per-entry single-issue emission discipline matching validator_version_info - Top-level decode failure short-circuits all sub-validation - Absence of export directory not treated as a defect (most EXEs have no exports) - Forwarder entries skip the function-RVA-in-image check (forwarders legitimately point inside the export directory) Registered in the dispatcher between validate_version_info and validate_entropy. Refs: requirement 2 (export table refinement)
New TypedDicts in internal_schema: - ExportDirectoryHeader: the 40-byte IMAGE_EXPORT_DIRECTORY fields - ExportFunctionEntry: per-function decoded view with name and forwarder resolution - ExportNamePointerEntry: per-name-pointer view with per-entry errors list - ExportStruct: top-level export struct shape produced by parser_exports InternalMetadata gains export_struct: Optional[ExportStruct]. Mirrors the typing discipline established for VersionInfoStruct and ResourcesStruct in earlier requirement 3 work. No runtime behaviour change.
The export struct is built alongside resources_struct and version_info_struct in the metadata builder. The new validate_exports validator is registered in STRUCTURAL_VALIDATORS between version_info and entropy. No public IOC contract change; export_struct remains in internal metadata pending the scheduled public-contract refresh.
Parser tests (~50 cases): - Locator paths: all four exception classes, zero RVA/size, valid extraction - Header decoder: valid, too short, empty, all-zero - Array readers: dword and word, zero count, zero RVA, read failure, short read with partial data, full read - ASCII string reader: zero RVA, read failure, empty, unterminated, valid, empty terminated, non-ASCII fallback, oversized buffer - Name validation: 10 parametrised cases including control chars and Unicode - Full roundtrips: minimal valid, forwarder detection by RVA range, invalid forwarder format, name pointer error classes, truncation propagation through to per-entry None handling, unused EAT slots - Output contract: required keys, list type guarantees - Determinism: identical output across 20 runs on valid and malformed input Validator tests (~40 cases): - Absence and top-level decode short-circuit - Placement: in-bounds, exceeds image, silent fallbacks - Truncations: single, multiple, per-table emission - Header consistency: each sub-reason independently, combined failures, header-none skip - Name pointer validation: clean, every error class, priority resolution for name_rva and name encoding errors, no double-emission - Name pointer ordering: sorted, unsorted, unreadable-entry skip, empty - Function entries: clean, ordinal range, RVA bounds, zero/None RVA, missing size_of_image fallback - Forwarders: valid, unreadable, malformed format, RVA check bypass - Combined anomalies and output contract - Determinism across 20 runs All defensive code paths exercised; one # pragma: no cover applied to _first_unsorted_index's defensive return (unreachable from caller).
- Extend the reason-codes reference with a new top-level section EXPORT ANOMALIES, structured into three subsections: Directory, Name Pointer, and Function Entry. - Add a dedicated EXPORT SUB-REASONS section documenting the details[reason] taxonomy for each code that carries one. The sub-reason list is treated as part of the public contract. - Add validator documentation section 2.6 for the exports validator, including an explicit determinism rationale paragraph matching the section 2.11 pattern. No code changes.
…w triggering on 2 contract fixtures.
Extends _parse_resources to expose the full ResourceEntry shape declared
in the public schema, with structured error reporting for entries that
cannot be fully decoded.
Output additions per entry:
- name: the resource's named identifier (e.g. MAINICON), previously
declared in the schema but not populated.
- codepage: from data_entry.struct.CodePage, typed as Optional[int]
(null when zero or absent).
- rva: data RVA, previously declared but not populated.
- raw_offset: file offset via get_offset_from_rva, with guarded exception
handling matching the pattern established in build_resource_structure.
- errors: tombstone list of per-entry computation failures:
* size_invalid (declared size <= 0)
* rva_invalid (negative RVA)
* data_out_of_bounds (RVA + size exceeds memory-mapped image)
* raw_offset_invalid (RVA-to-offset resolution failed)
Behavioural changes:
- Resources with invalid data are no longer silently dropped; they are
emitted with errors populated and the affected fields set to None.
This makes the resource count visible to consumers regardless of
per-entry health.
- Output list is sorted by (type, language, rva) for snapshot stability.
- Entropy is rounded to 4 decimal places for cross-platform float
determinism.
Bug fix included:
- Entropy is now computed by slicing get_memory_mapped_image() with the
RVA (correct) rather than the raw file offset (incorrect). The
memory-mapped image is indexed by RVA per pefile's contract; using
the raw offset sliced the wrong byte range and produced incorrect
entropy values for every resource. Caught before snapshot stamping.
No schema changes in this commit; ResourceEntry schema update lands
separately.
Refs: requirement 4 (resource metadata enrichment)
Schema changes: - ResourceEntry is now total=False with all fields Optional, accommodating per-entry computation failures. - New fields: codepage (Optional[int]), errors (Optional[List[str]]). - Existing fields retain their meanings; the schema now matches what the parser produces. LANGID decoder corrections in _decode_langid: - Removed the early-return guard, which was rejecting valid neutral-sublang LANGIDs (e.g., LANGID 0x0001 now correctly decodes as 'ar' for Arabic with sublang neutral). - All 'cannot decode' paths now return None instead of the magic string 'unknown'. The previous behaviour conflated several distinct states (not provided, structurally invalid, primary language unmapped) into one ambiguous string sentinel that was fragile for consumers. Compatibility: - Consumers checking need to update to . This is a deliberate semantic correction rather than a passive breaking change. - Output shape gains keys ( Usage: cpi code_page_file [-c] [-L] [-l] [-a|nnn] -c: input file is a single codepage -L: print header info (you don't want to see this) -l or no option: list all codepages contained in the file -a: extract all codepages from the file nnn (3 digits): extract codepage nnn from the file Example: cpi ega.cpi 850 will create a file 850.cp containing the requested codepage., ) and may include entries that previously didn't appear (those with errors). Existing per-entry field meanings are unchanged. Deferred follow-up (not in this commit): - SUBLANG table refinement. The current implementation models sublang values as language-independent, which is incorrect for multilingual edge cases (sublang 0x02 means UK English with primary English, but Swiss German with primary German). A flat LCID -> BCP-47 mapping is the structural fix; tracked as a separate ticket since current behaviour is correct for the common case. Snapshot refresh required for fixtures with resources. Refs: requirement 4 (resource metadata enrichment)schema(public): expand ResourceEntry and correct LANGID decoding Schema changes: - ResourceEntry is now total=False with all fields Optional, accommodating per-entry computation failures. - New fields: codepage (Optional[int]), errors (Optional[List[str]]). - Existing fields retain their meanings; the schema now matches what the parser produces. LANGID decoder corrections in _decode_langid: - Removed the early-return guard, which was rejecting valid neutral-sublang LANGIDs (e.g., LANGID 0x0001 now correctly decodes as 'ar' for Arabic with sublang neutral). - All 'cannot decode' paths now return None instead of the magic string 'unknown'. The previous behaviour conflated several distinct states (not provided, structurally invalid, primary language unmapped) into one ambiguous string sentinel that was fragile for consumers. Compatibility: - Consumers checking need to update to . This is a deliberate semantic correction rather than a passive breaking change. - Output shape gains keys ( Usage: cpi code_page_file [-c] [-L] [-l] [-a|nnn] -c: input file is a single codepage -L: print header info (you don't want to see this) -l or no option: list all codepages contained in the file -a: extract all codepages from the file nnn (3 digits): extract codepage nnn from the file Example: cpi ega.cpi 850 will create a file 850.cp containing the requested codepage., ) and may include entries that previously didn't appear (those with errors). Existing per-entry field meanings are unchanged. Deferred follow-up (not in this commit): - SUBLANG table refinement. The current implementation models sublang values as language-independent, which is incorrect for multilingual edge cases (sublang 0x02 means UK English with primary English, but Swiss German with primary German). A flat LCID -> BCP-47 mapping is the structural fix; tracked as a separate ticket since current behaviour is correct for the common case. Snapshot refresh required for fixtures with resources. Refs: requirement 4 (resource metadata enrichment)
…LANGID was incorrectly mapped as unknown, 2> regression guard for the common case to prove the broader changes didn't break anything obvious
Extends _parse_optional_header and _parse_header to expose security-relevant fields previously not captured in IOCX's public metadata, completing requirement 1 (Optional Header Metadata Enrichment). Optional Header additions: - dll_characteristics (raw int) - dll_characteristics_flags (decoded flag names, sorted by bit position for snapshot stability) - dll_characteristics_unknown_bits (hex string for any bits outside the known mask, or null) - win32_version_value (raw int, deprecated DWORD per PE spec, with Reserved1 field guard) - loader_flags (raw int, reserved DWORD per PE spec) - stack_reserve_size, stack_commit_size - heap_reserve_size, heap_commit_size Header additions: - subsystem_name (decoded string from IMAGE_SUBSYSTEM_* table, or null for unknown subsystem values) New constants module iocx.parsers.pe_constants: - SUBSYSTEM_NAMES: IMAGE_SUBSYSTEM_* lookup table (15 entries covering the well-known subsystems) - DLL_CHARACTERISTICS_FLAGS: IMAGE_DLLCHARACTERISTICS_* bit→name table (11 entries covering all defined bits) - DLL_CHARACTERISTICS_KNOWN_MASK: derived OR of all known bits, used to detect unknown bits per binary Implementation choices: - New fields use None as the missing-field default, distinct from 0 which indicates the binary really has that value. This is a deliberate divergence from the existing fields' default-to-zero convention; the semantic split between 'missing' and 'zero' is meaningful for security fields (DLL characteristics 0 means 'no security features enabled,' which is itself a signal). Existing fields retain default-to-zero for backward compatibility. - DLL characteristics decoding emits flag names in bit-position order, with any unknown bits exposed separately as a hex string. Consumers get a stable list plus complete information about non-decoded bits. - Subsystem name resolution follows the same pattern as the language / language_name pair added in requirement 4: raw value plus optional decoded string. Conservative invalid-field handling: each field is extracted independently via getattr with an appropriate default. Failure to extract one field does not affect others, satisfying the requirement's 'conservative handling of invalid fields' clause. No validator changes; no new reason codes. The validate_optional_header validator handles structural sanity checks (e.g., SizeOfImage consistency) and is unaffected by this enrichment. Refs: requirement 1 (Optional Header Metadata Enrichment)
…achine name decoding moved to the parser layer. Resource entropy statistics tolerate per-entry errors.
…ended new behaviour. Coverage at 100% over 1275 tests
…rser New module independent of pefile's DIRECTORY_ENTRY_DELAY_IMPORT interpretation. Uses pefile only to locate the delay-load directory, determine PE32 vs PE32+ via OPTIONAL_HEADER.Magic, and resolve RVAs to file offsets via pe.get_data. Deterministic extraction of: - 32-byte IMAGE_DELAY_IMPORT_DESCRIPTOR structures via struct.unpack_from - Descriptor array walk with zero-terminator detection and hard count limit (4096), with distinct truncation tags for each termination cause - v1 (modern RVA) vs v0 (legacy VA) attribute mode captured from raw Attributes field rather than silently coerced - Import Name Table (INT) and Import Address Table (IAT) thunk arrays with DWORD/QWORD sizing determined by Magic - Per-thunk decoding with high-bit ordinal detection - IMAGE_IMPORT_BY_NAME structures with bounded ASCII scan (1024 bytes) - Bound state detection by bound_iat_rva != 0 - INT/IAT length parity check at descriptor level - DLL name string reading with bounded scan (512 bytes) and structural ASCII validation Determinism guarantees: - All array walks bounded by zero terminator AND hard count limit - Sub-structure failures emit deterministic tombstone tags in truncations[] and per-entry errors[] - Never raises; pefile.PEFormatError and other exceptions are narrowly caught and converted to tombstone entries - Output dict shape stable regardless of malformation pattern Returns None for absent delay-load directory (most binaries don't use delay-loading); returns a structurally-rich dict otherwise per the documented output contract. Verified end-to-end against mspaint.exe with dumpbin /imports cross-check: 107 imports decoded from gdiplus.dll's delay-load directory with byte-exact agreement on DLL name, hint values, IAT addresses (5369363602 = 0x14009FC92), ordering, and bound state. Refs: delay-load import parsing (originally deferred; bundled into this release for cross-tool divergence methodology value)
Maps pe_delay_imports output to eight new reason codes: Directory anomalies: - DELAY_IMPORT_DIRECTORY_INVALID_HEADER: top-level decode failed - DELAY_IMPORT_DIRECTORY_OUT_OF_BOUNDS: directory extends past SizeOfImage - DELAY_IMPORT_TABLE_TRUNCATED: sub-table truncation or unterminated descriptor array (per-table emission via details[table]) Descriptor anomalies: - DELAY_IMPORT_DESCRIPTOR_INVALID: per-descriptor INT or IAT structural error (priority-resolved sub-reasons via details[reason], scoped by details[table]) - DELAY_IMPORT_DLL_NAME_INVALID: DLL name RVA unusable (priority-resolved sub-reasons: dll_name_rva_zero, read_failed, unterminated, dll_name_not_printable, non_ascii) - DELAY_IMPORT_INT_IAT_MISMATCH: INT and IAT have different lengths, violating the parallel-array invariant - DELAY_IMPORT_ATTRIBUTES_LEGACY_VA_MODE: descriptor uses v0 (pre-Windows 2000) mode where table fields are raw virtual addresses rather than RVAs Entry anomalies: - DELAY_IMPORT_ENTRY_INVALID: per-import entry malformed (priority-resolved sub-reasons: int_entry_missing, int_entry_zero, ordinal_zero, name_read_failed, name_too_short, hint_unpack_failed, name_unterminated, name_non_ascii, name_not_printable) Implementation choices: - Priority-resolved sub-reasons via module-level _PRIORITY constants, matching the discipline established in validator_exports - Single-issue-per-pathology emission for per-descriptor and per-entry checks; cross-table consistency check (INT/IAT mismatch) emitted as its own reason code rather than per-entry cascade - Top-level decode failure short-circuits all sub-validation - Absence of delay-load directory not treated as a defect (most binaries don't use delay-loading) - Bound state is normal behaviour; bound_iat_rva != 0 is not flagged - v0 attribute mode emitted as a dedicated reason code rather than silently coerced, supporting the cross-tool divergence demonstration this validator was scoped to enable Registered in the dispatcher between validate_exports and validate_entropy, completing the import/export structural validator cluster ahead of the entropy/derived layer. Refs: delay-load import parsing
New TypedDicts in internal_schema: - DelayImportEntry: per-import decoded view (ordinal vs name, hint, IAT value, per-entry errors) - DelayImportDescriptor: per-descriptor view including Attributes, v1/v0 mode flag, all five sub-table RVAs, bound state, and the imports list - DelayImportStruct: top-level delay-load struct shape produced by parser_delay_imports InternalMetadata gains delay_import_struct: Optional[DelayImportStruct]. Mirrors the typing discipline established for ResourcesStruct, VersionInfoStruct, and ExportStruct in earlier requirements. No runtime behaviour change.
The delay-load struct is built alongside resources_struct, version_info_struct, and export_struct in the metadata builder. The new validate_delay_imports validator is registered in STRUCTURAL_VALIDATORS between validate_exports and validate_entropy. No public IOC contract change; delay_import_struct remains in internal metadata pending the scheduled coordinated public-contract refresh that will also promote version_info_struct, export_struct, and load_config metadata.
The delay-load struct is built alongside resources_struct, version_info_struct, and export_struct in the metadata builder. The new validate_delay_imports validator is registered in STRUCTURAL_VALIDATORS between validate_exports and validate_entropy. All reason codes surfaced by the validator are added in this commit
Parser tests (~70 cases): - Locator paths: missing data directory, IndexError, AttributeError, zero RVA, zero size, valid placement - PE32 vs PE32+ Magic detection - 32-byte IMAGE_DELAY_IMPORT_DESCRIPTOR decoding: valid, all zero (terminator), unpack failure - Descriptor array walk: single descriptor, multiple descriptors, zero terminator, declared size exhaustion, hard count limit - INT/IAT thunk arrays: empty, terminator, truncation, max imports, unpack failure - IMAGE_IMPORT_BY_NAME structure: valid, too short, unterminated, non-ASCII, hint unpack failure - DLL name string reading: valid, zero RVA, read failed, unterminated, non-printable, non-ASCII - Ordinal entry decoding: high bit set, ordinal zero - Name entry decoding: high bit clear, valid name resolution - v1 vs v0 attribute mode capture - Bound state detection - Full roundtrip on simulated mspaint-like single-descriptor binary - Output contract: required keys, list type guarantees - Determinism: identical output across 20 runs on valid and malformed input Validator tests (~50 cases): - Absence and top-level decode short-circuit - Placement: in-bounds, exceeds image, silent fallbacks - Truncations: single, multiple, per-table emission - Per-descriptor: every error class, priority resolution, no double-emission - DLL name validation: all sub-reasons with priority order pinned - INT/IAT mismatch: pure mismatch and cascade with entry errors - v0 attribute mode: single emission and cascade with downstream errors - Per-entry: every error class with priority order pinned - Combined anomalies and output contract - Determinism across 20 runs All defensive code paths exercised via monkeypatched struct.error injection. No # pragma: no cover added; all defensive branches reachable through targeted input.
- Extend the reason-codes reference with a new top-level Delay-Load
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.