sdk-api + parse
Turn MSDN's Windows SDK API and Windows Driver DDI docs into a .json file, fast.
Now with --mode driver for windows-driver-docs-ddi: IRQL constraints, KMDF/UMDF version tags, target-type, tech.root, etc.
# install uv once (Windows)
winget install astral-sh.uv
# clone with submodules
git clone --recurse-submodules https://github.com/cristeigabriela/sparse.git
cd sparse
# parse sdk-api (the submodule path is the default for --mode sdk)
uv run python sparse.py -o sdk-api.json
# parse windows-driver-docs-ddi
uv run python sparse.py --mode driver -o driver-docs.jsonuv handles the venv and dependencies automatically (see pyproject.toml). No pip install needed.
The latest builds of both files are attached to the rolling latest release. They get rebuilt on every push to main (skip the release by including norelease in the commit message, e.g. docs(norelease): tweak README).
usage: sparse.py [-h] [--mode {sdk,driver}] [-o OUTPUT] [--chunk-size CHUNK_SIZE] [--workers WORKERS] [--silent]
input_dir
Parse MSDN Native Function .md files to JSON (sdk-api or windows-driver-docs-ddi)
positional arguments:
input_dir Root content directory to search for nf-*.md files
options:
-h, --help show this help message and exit
--mode {sdk,driver} Frontmatter dialect: 'sdk' for sdk-api (default), 'driver' for windows-driver-docs-ddi
-o, --output OUTPUT Output JSON file path
--chunk-size CHUNK_SIZE
Files per worker chunk
--workers WORKERS Number of parallel workers, or "max" to use _MAX_WINDOWS_WORKERS
--silent Suppress all output
Each --mode defaults input_dir to the matching submodule, so the usual invocations are short:
# sdk-api (default mode)
uv run python sparse.py -o sdk-api.json
# windows-driver-docs-ddi
uv run python sparse.py --mode driver -o driver-docs.jsonPass an explicit path to override the default — useful for pointing at a checkout outside the repo:
uv run python sparse.py --mode driver -o driver-docs.json /path/to/windows-driver-docs-ddi/wdk-ddi-src/content/
~8 seconds to finish 43,337 entries on
Intel(R) Core(TM) i9-14900HX, 16c/32t, default settings.
Real output from the parser, side-by-side. --mode driver adds the extra fields shown on the right (include_header, target_type, kmdf_ver, umdf_ver, tech_root, irql, irql_raw); the base shape is shared.
Long string values are trimmed inline (
…marker) so the table fits without a horizontal scrollbar. Seeexample/output.jsonandexample/output-driver.jsonfor the full verbatim output.
--mode sdk + CreateFileW |
--mode driver + WdfCollectionAdd |
|---|---|
{
"CreateFileW": {
"header": "fileapi.h",
"lib": "Kernel32.lib",
"dll": "Kernel32.dll",
"min_client_version": "Windows XP [desktop apps only]",
"min_server_version": "Windows Server 2003 [desktop apps only]",
"metadata": {
"UID": "NF:fileapi.CreateFileW",
"title": "CreateFileW function (fileapi.h)",
"description": "Creates or opens a file or I/O device. … (Unicode)",
"api_type": [
"DllExport"
],
"api_location": [
"api-ms-win-core-file-l1-2-5.dll",
"api-ms-win-core-file-l1-2-4.dll",
"api-ms-win-core-file-l1-2-3.dll",
"Kernel32.dll",
"API-MS-Win-Core-File-l1-1-0.dll",
"KernelBase.dll",
"API-MS-Win-Core-File-l1-2-0.dll",
"API-MS-Win-Core-File-l1-2-1.dll",
"API-MS-Win-Core-File-l1-2-2.dll",
"API-MS-Win-DownLevel-Kernel32-l1-1-0.dll",
"MinKernelBase.dll"
],
"api_name": [
"CreateFile",
"CreateFileA",
"CreateFileW"
]
},
"params": {
"lpFileName": {
"directions": [
"in"
],
"values": {}
},
"dwDesiredAccess": {
"directions": [
"in"
],
"values": {}
},
"dwShareMode": {
"directions": [
"in"
],
"values": {
"FILE_SHARE_DELETE": 4,
"FILE_SHARE_READ": 1,
"FILE_SHARE_WRITE": 2
}
},
"lpSecurityAttributes": {
"directions": [
"in",
"optional"
],
"values": {}
},
"dwCreationDisposition": {
"directions": [
"in"
],
"values": {
"CREATE_ALWAYS": 2,
"CREATE_NEW": 1,
"OPEN_ALWAYS": 4,
"OPEN_EXISTING": 3,
"TRUNCATE_EXISTING": 5
}
},
"dwFlagsAndAttributes": {
"directions": [
"in"
],
"values": {
"FILE_ATTRIBUTE_ARCHIVE": 32,
"FILE_ATTRIBUTE_ENCRYPTED": 16384,
"FILE_ATTRIBUTE_HIDDEN": 2,
"FILE_ATTRIBUTE_NORMAL": 128,
"FILE_ATTRIBUTE_OFFLINE": 4096,
"FILE_ATTRIBUTE_READONLY": 1,
"FILE_ATTRIBUTE_SYSTEM": 4,
"FILE_ATTRIBUTE_TEMPORARY": 256,
"FILE_FLAG_BACKUP_SEMANTICS": 33554432,
"FILE_FLAG_DELETE_ON_CLOSE": 67108864,
"FILE_FLAG_NO_BUFFERING": 536870912,
"FILE_FLAG_OPEN_NO_RECALL": 1048576,
"FILE_FLAG_OPEN_REPARSE_POINT": 2097152,
"FILE_FLAG_OVERLAPPED": 1073741824,
"FILE_FLAG_POSIX_SEMANTICS": 16777216,
"FILE_FLAG_RANDOM_ACCESS": 268435456,
"FILE_FLAG_SESSION_AWARE": 8388608,
"FILE_FLAG_SEQUENTIAL_SCAN": 134217728,
"FILE_FLAG_WRITE_THROUGH": 2147483648,
"SECURITY_ANONYMOUS": null,
"SECURITY_CONTEXT_TRACKING": null,
"SECURITY_DELEGATION": null,
"SECURITY_EFFECTIVE_ONLY": null,
"SECURITY_IDENTIFICATION": null,
"SECURITY_IMPERSONATION": null
}
},
"hTemplateFile": {
"directions": [
"in",
"optional"
],
"values": {}
}
}
}
} |
{
"WdfCollectionAdd": {
"header": "wdfcollection.h",
"lib": "Wdf01000.sys (KMDF); WUDFx02000.dll (UMDF)",
"dll": null,
"min_client_version": null,
"min_server_version": null,
"metadata": {
"UID": "NF:wdfcollection.WdfCollectionAdd",
"title": "WdfCollectionAdd function (wdfcollection.h)",
"description": "The WdfCollectionAdd method adds a specified framework object to an object collection.",
"api_type": [
"LibDef"
],
"api_location": [
"Wdf01000.sys",
"Wdf01000.sys.dll",
"WUDFx02000.dll",
"WUDFx02000.dll.dll"
],
"api_name": [
"WdfCollectionAdd"
]
},
"params": {
"Collection": {
"directions": [
"in"
],
"values": {}
},
"Object": {
"directions": [
"in"
],
"values": {}
}
},
"include_header": "Wdf.h",
"target_type": "Universal",
"construct_type": null,
"kmdf_ver": "1.0",
"umdf_ver": "2.0",
"tech_root": "wdf",
"irql": {
"level": "DISPATCH_LEVEL",
"op": "<="
},
"irql_raw": "<= DISPATCH_LEVEL"
}
} |
The irql object normalizes ~95% of req.irql strings into {level, op} where level is one of PASSIVE_LEVEL, APC_LEVEL, DISPATCH_LEVEL, DPC_LEVEL, DEVICE_LEVEL, DIRQL, HIGH_LEVEL, IPI_LEVEL, or ANY, and op is one of <=, <, =, ==, >=, >, or null. The original string is always preserved in irql_raw.
Two pinned upstreams, both managed as git submodules:
| path | upstream | branch |
|---|---|---|
sdk-api/ |
MicrosoftDocs/sdk-api | default (docs) |
windows-driver-docs-ddi/ |
cristeigabriela/windows-driver-docs-ddi (fork) | integration/all-open-prs |
The driver-docs submodule points at a fork, not Microsoft's repo directly. It tracks my integration branch that bundles all my open patch PRs against MicrosoftDocs/windows-driver-docs-ddi so we parse the cleanest available data. We'll switch back to upstream if/once the PRs land.
To bump both submodules to the tip of their tracked branches:
./scripts/update-submodules.ps1
git diff --submodule=log # review what moved
git commit -am "chore: bump submodules"
Both sdk-api and windows-driver-docs-ddi keep their markdown under *-src/content/, split into per-header (or per-domain) folders.
We glob every file that starts with nf- (Native Function) and ends in .md, then run them through the parser Python module.
Each file has the same general shape:
- a YAML frontmatter block (where most of the metadata lives —
req.header,req.lib,req.irql, …) - a function/method/macro header (
# Foo function,# Foo macro,# IBar::Baz) — used to derive the top-level JSON key - zero or more
## -param Name [annotations]subheaders, sometimes containing an HTML table of constants
Driver mode is purely additive: the same parser, plus a handful of extra frontmatter keys and an IRQL normalizer (see parser/irql.py).
A long, long time ago, the person behind Hexacorn published "Hunting for Windows API prototypes and descriptions…", discussing existing projects that parse WinAPI metadata.
The following projects discussed are:
- msdocsviewer by Alexander Hanel
- Mandiant's IDA MSDN annotations
- etc.
They also discuss the history of how Microsoft distributed the documentation that is today available on MSDN in the past (HLP files, CHM files most recently, etc.)
Those are particularly helpful to avoid web requests to Microsoft's MSDN, and the (even more) unreliable process of parsing the HTML generated from the Markdown they are based on.
Today the golden standard is sdk-api (and windows-driver-docs-ddi / kernel / virt / etc. friends), as was first most aptly demonstrated by msdocsviewer in our community — and by Microsoft themselves in win32metadata.
Unfortunately, none of those generate lengthy, parsable metadata for every method.
This is why, I believe, the person behind Hexacorn went on to write their own parser (which, in its structure, is very similar to sparse).
Matter of fact, I'm actually fairly sure that they use the exact same expectations to extract data that we do, as we seem to have the same parsing limitations for common constant values.
So, in some ways, this is just an open-source way to reproduce the same outputs without needing to write your own parser for their data (like I have in the past!)