The one -Niuandmorluto
0baad58e60
* feat(ghidra): admit RISC-V ELF analysis through a pinned generic lane Refs #2063. Recognize `EM_RISCV` (243) as a provider-neutral family that carries its own ELF facts, and bound the first Ghidra lane to little-endian `ET_EXEC` images that declare the double-float ABI, pinned to `RISCV:LE:32:default` or `RISCV:LE:64:default` with the `gcc` compiler specification. Ghidra 12.1.4 is not reliable without the pin: its `riscv.opinion` pins `variant="default"` for the 64-bit constraint and leaves the 32-bit constraint with the comment that the loader "must choose between 'default' and 'AndeStar' variants". Measured on the installed release tree, a source-owned generic RV32 ELF with no Andes declaration imports as `RISCV:LE:32:AndeStar_v5:gcc`, while RV64 resolves to `RISCV:LE:64:default:gcc`. The lane therefore states the language per width and refuses any caller-selected processor outside it instead of inheriting the header guess. Refusals name the declaration rather than falling back to a guessed profile: big-endian images (Ghidra 12.1.4 ships no big-endian RISC-V language), non-`ET_EXEC` images, `EF_RISCV_RVE`, `EF_RISCV_TSO`, and every float ABI other than double, which the pinned generic compiler specifications model. The raw e_flags bits stay in Evidence as declarations, not as instruction semantics. Family recognition now has one owner for the ELF-metadata families: `ELF_METADATA_ARCHITECTURES`, `isElfMetadataArchitecture`, and `HeaderArchitecture` replace ten `Exclude<BinaryArchitecture, "mips">` sites that would each have needed a second hand-edited member. `ghidraPinnedLanguage` is that owner for the import commitment: the session readback previously re-derived only the MIPS pin, so a profile committing any second pinned language failed with a language-mismatch error before the bridge started. Hopper and IDA refuse RISC-V by name, matching the MIPS wording. Caller-supplied raw-binary entry keeps its existing four-family list; this lane is ELF-header admission only. Verification on macOS 15.7.7 arm64, Node 24.18.0, Ghidra 12.1.4 PUBLIC and JDK 21.0.12.1, with the host decompiler built from the release sources for `os/mac_arm_64`: `npm run check` passes; `npm run test:fast` passes 396 test files; `npm run verify:test-discovery` discovers all 917 test files exactly once. Source-owned RV32 and RV64 double-float `ET_EXEC` fixtures complete `rea analyze --provider ghidra` and `rea decompile rea_riscv_entry`, committing `RISCV:LE:32:default` and `RISCV:LE:64:default` with `language_selection: provider-pinned-v1`; the soft-float RV64 shape is refused with the named float-ABI reason, and the full refusal matrix including the RV32 and width-mismatch shapes is covered by unit tests. No `verify:ghidra:riscv` lane script is included in this change; the runs above were performed manually against the installed engine. Fork pull requests receive no CI check runs, so the real-provider scope above is the evidence available for review. * fix(ghidra): bound the RISC-V lane by declared flags and document it Address review feedback on #2151. `e_flags` bits outside `EF_RISCV_RVC | EF_RISCV_FLOAT_ABI` are now refused with the offending value named, so a reserved or newer producer declaration cannot enter the verified generic lane by falling through the RVE, TSO and float-ABI checks. Admission reads `e_type` only, so the refusal text no longer calls `ET_EXEC` static: a fixed-address declaration does not establish static linkage, and `PT_INTERP` and dynamic dependencies are not inspected. The session records that boundary as a limitation rather than inferring a load model. Add `docs/ghidra-riscv.md` in the shape of the MIPS lane guide (admission boundary, refusal table, why the language is pinned per class, Evidence fields, and what the fixtures do not establish), and correct `docs/firmware-analysis.md`, which still told users that MIPS and RISC-V decompilation had not been added. Verification after the change: `npm run check` and `npm run docs:check` pass; the RISC-V module tests pass (17 lane cases plus 9 parser cases); on Ghidra 12.1.4 with JDK 21.0.12.1 the RV32 and RV64 double-float fixtures still analyze and decompile with `RISCV:LE:32:default` / `RISCV:LE:64:default` and `language_selection: provider-pinned-v1`, the soft-float fixture is still refused by name, and the new linkage limitation appears in the emitted Evidence. * fix(ghidra): align RISC-V pinning with admission Reuse the complete lane admission boundary when selecting a language, retain unsigned unsupported flag bits in diagnostics, and derive float ABI metadata from the source declaration. * test(ghidra): verify RISC-V CLI and MCP workflows Build source-owned RV32/RV64 fixtures with Clang and LLD, establish independent ELF and symbol facts with LLVM readelf, and verify real provider evidence, instruction bytes, calls, decompilation, schema projections, snapshot replay, refusal diagnostics, source immutability and cleanup. Verified on macOS ARM64 with Ghidra 12.1.4, JDK 21 and LLVM/LLD 23.1.2. Dynamic linkage, big-endian, Andes and RVE/TSO workflows remain unverified; ISA attributes are not inspected. --------- Co-authored-by: morluto <76467478+morluto@users.noreply.github.com>
8.5 KiB
Firmware analysis
REA accepts Binwalk 3.1.x for region inspection and Unblob 26.6.x for explicit extraction. The report parsers are verified with Binwalk 3.1.0 and Unblob 26.6.4. Another build on those lines is accepted, the observed version is reported, and a limitation records that the parser was not verified against that build. This first firmware milestone is verified on Linux x64. No device, root privilege, mount, emulation or execution of extracted programs is needed. Arbitrary firmware bytes do not pass native executable admission; select a returned ELF separately for supported Ghidra analysis.
Caller-supplied tools
Provide existing tools using absolute executable paths, scoped to your shell:
export REA_BINWALK_COMMAND=/absolute/path/to/binwalk
export REA_UNBLOB_COMMAND=/absolute/path/to/unblob
REA does not install these tools, their dependencies or system packages. It
checks the tool's --version banner when executing a request. Binwalk must
print binwalk <version> and Unblob must print the version alone. Inspection and
extraction have independent prerequisites; one does not require the other.
Both require util-linux prlimit, normally /usr/bin/prlimit; override its
absolute path with REA_FIRMWARE_PRLIMIT_COMMAND if needed.
REA retains at most 8 MiB of combined stdout and stderr per command. Exceeding that diagnostic budget stops analysis with an explicit truncated-output error; child process limits do not bound parent-side pipe retention. Completed commands drain their output streams before recording diagnostics. Failed worker cleanup retains the worker and workspace for retry before another request or at shutdown.
Unblob needs the external extractor for the selected format on its inherited
PATH: for example, debugfs for ext2/ext3/ext4. Its default Landlock sandbox
and skip rules stay enabled. Sandbox startup failures remain actionable errors;
REA does not retry with --no-sandbox. Missing extractors reported in a valid
analysis report retain partial results and dependency names inline.
Upstream sources are preserved unmodified as optional pinned submodules:
git submodule update --init third_party/binwalk third_party/unblob
Normal REA builds and npm packages do not include or build these sources or engine binaries. See upstream provenance.
CLI and MCP
rea inspect-firmware-regions /path/to/firmware.bin
rea extract-firmware /path/to/firmware.bin /path/to/new-output
rea extract-firmware /path/to/firmware.bin /path/to/selected-output \
--offset 64 --length 4096 --max-depth 3
The corresponding MCP tools accept the same intent:
{
"name": "inspect_firmware_regions",
"arguments": { "path": "/path/to/firmware.bin" }
}
{
"name": "extract_firmware",
"arguments": {
"path": "/path/to/firmware.bin",
"output_directory": "/path/to/new-output",
"range": { "offset": 64, "length": 4096 },
"max_depth": 3
}
}
Both return Evidence with the original path and SHA-256, raw producer reports, actual executable path/digest, version and limitations. Binwalk signature UUIDs and Unblob blob IDs are producer-local identities; compare content hashes and source ranges across runs. A matching version does not attest an installed build's source revision, so that field remains unknown.
Interpret the results
- Region offsets refer to bytes in the original firmware.
reported_sizepreserves Binwalk's output;size_basis=provider_reported_validation_unknownreflects that Binwalk can replace unknown lengths with the next signature or EOF before serialization. Neither a signature nor its confidence byte proves a valid partition, encryption or runtime behavior. - Extraction returns regular file paths, independently verified SHA-256 values,
producer hashes when available, chunks, diagnostics and task derivations.
Unblob's task report is unordered; REA resolves relationships by paths and
blob IDs. Logical references use
$input,$output, or$output/<relative-path>so a real file named$inputstays distinct from the original input. Chunk offsets belong to the task's input file. Only chunks in the selected input have a knownroot_file_range, adjusted by the selected offset. - Decompressed children have unknown original byte ranges and runtime addresses. Task lineage does not establish a byte-for-byte mapping through decompression.
partialmarks unknown/encrypted chunks, reported extraction problems, unpublished links/special files or reached depth limits.completemeans the engine finished its configured extraction without those reported conditions; default skip rules and format coverage still apply. It does not establish complete firmware reconstruction.- The destination must be absent, absolute, with an existing parent. Publication uses exclusive, symlink-resistant writes and durable digest readback. Existing destinations are never overwritten. This is not an atomic directory rename: a new tree can be visible while files are written. Failed uncommitted output is rolled back. Symlinks and special files are recorded without publication; empty directories, ownership and executable permission bits are not reproduced.
Select a verified native child for the existing tools:
{
"name": "open_binary",
"arguments": {
"path": "/path/from/extraction/usr/bin/program",
"provider_id": "ghidra"
}
}
Then use analyze_function, inspect_native_api or the other native primitives.
Native architecture and executable validation still apply. Generic
MIPS ELF and RISC-V ELF lanes are separate
documents with their own admission boundaries. This milestone does
not add PPC decompilation, raw flash load profiles, bootloader/device
tree semantics, vulnerability scanning, emulation or hardware runtime capture.
Resource and lifecycle policy
- Nonempty regular firmware inputs: at most 128 MiB, copied and hashed into a private workspace before analysis. No in-place source modification.
- One queued heavy operation per provider instance; Binwalk one thread and Unblob one worker. No automatic parallel extraction engines.
- 120-second deadline; inherited 1 GiB per-process address-space limit,
120 CPU seconds and zero core-dump allowance through
prlimit. - JSON reports: at most 8 MiB and valid UTF-8; malformed encoding and oversized reports fail without repairing or truncating producer data. Diagnostic output is bounded with exact captured byte counts.
- Default staging budget: 256 MiB and 10,000 entries, configurable using
max_output_bytes/max_output_files(CLI--max-output-bytes/--max-output-files). Depth defaults to 3, allowed range 1–10.
Staging budgets are polled and rechecked after exit; short-lived overshoot is possible. The inherited file-size limit bounds each file, not total storage; address-space limits are per process, not an aggregate memory quota. Worker count is not CPU affinity for every external extractor. These limits do not claim cgroup/container containment. Cancellation cleans the owned process group before removing its workspace; uncertain cleanup retains resources and blocks further operations in that provider instance.
Verification
Generate small public, source-derived fixtures into ignored _reference/:
npm run fixtures:firmware
npm run verify:firmware
Fixture generation requires existing Python 3 and a host C compiler (cc);
override with REA_FIRMWARE_PYTHON / REA_FIRMWARE_CC. The independent oracle
records gzip offset/length and extracted file hashes. Real verification checks
CLI/MCP, full-file and selected-range extraction, unknown chunks, depth limits,
content parity and unchanged source bytes. No binary fixture is committed.
Additional lanes are explicit, so the base lane stays small:
# Existing mke2fs and debugfs required; put debugfs on the scoped PATH.
REA_FIRMWARE_VERIFY_EXT4=1 npm run fixtures:firmware
REA_FIRMWARE_VERIFY_EXT4=1 npm run verify:firmware
# Existing Ghidra 12.1.x and its declared JDK required.
REA_FIRMWARE_VERIFY_GHIDRA=1 npm run verify:firmware
REA_FIRMWARE_MKE2FS_COMMAND selects the optional ext4 fixture producer.
REA_FIRMWARE_FIXTURE_ROOT selects generated data storage.
REA_FIRMWARE_TEST_ENTRYPOINT can select an isolated installed npm package's
scripts/rea.mjs to verify the packaged CLI/MCP workflow. Real coverage here is
generated gzip/USTAR/ext4 with a Linux x64 ELF child; SquashFS, UBIFS, proprietary
firmware, other hosts and architectures remain unverified.