Nikon bodies from before ~2016 record no CropArea, so the decoder kept
their whole sensor data, less trailing dark columns. Where that data
holds masked lines above black or optical black, they were developed as
a coloured or dark bar on that edge: the D7000 and D5100's 36 columns at
~600 (#976), the D800's 14 (#914, the top in portrait), the D600 / D610's
8 (#1236), the D3200's 46, and others.
MEASURED_BODIES (crates/raw/src/vendor/nef.rs) now gives 44 more model /
data size / depth groups the frame of their camera JPEG inside the data,
next to the D4, D7100 and D90 entries already there: D7000, D5100, D800,
D800E, Df, the D4's 12-bit mode, D3, D3S, D700, D300, D300S, D600, D610,
D810, D4S, D5600, Coolpix A, D3000, D40X, D60, D80, D2Xs, D200, D3200
and Nikon 1 V1; and, on best-guess frames short of the evidence rule,
each with a comment naming its evidence, the D3100 (3 of 4 clean files
agree), D5000 (4 of 5, and the D90's sensor), D40 (2 of 2), D50 and D70
(one file each, agreeing) and D70s (next to them, between pixels).
Every best-guess frame keeps the masked lines out by several pixels.
An entry matches model, data size and depth exactly. Bodies whose data
holds only picture beyond the camera frame get no entry and render as
before.
`cargo xtask raw-frame FILE...` (crates/engine/examples/raw_frame.rs)
measures a body: it aligns each raw's embedded camera JPEG with the
sensor data (half-size luminance, gradient correlation, sub-pixel peak;
weak, ambiguous or between-pixel peaks are flagged), counts the edge
lines that hold no picture and reports what the decoder's crop keeps
beyond the camera frame. A masked line is flat and its two CFA phases
agree within the noise or within 0.4% of white, so filler with a small
fixed offset between them (D90, D5000) counts. It prints entries only
with evidence: at least 3 clean files agreeing exactly, of the model
(12- and 14-bit pooled) or, with at least one of its own, with sibling
models of the same readout (data and JPEG size) that agree on the same
frame. On the D7500, which records its CropArea, it finds that area
exactly; it declines the Z f, whose JPEG is distortion-corrected. It
works for any make whose raw embeds a full-size camera JPEG: on Sony,
Canon CR2 / CR3, Pentax and Samsung files it finds the decoder's own
crop (relative to the active area).
Every Nikon raw on raw.pixls.us without CropArea was surveyed this way
(its 138 CC0 samples, plus 124 other publicly posted sample files; only
the measurements are committed). Bodies whose data holds only picture
beyond the frame need no entry (D750, D5300, D5500, D3300, D2X, D2Hs,
D3X, D5200, Coolpix P6000, Nikon 1 AW1 / J3 / V2 / V3, the D800's crop
modes); raw-parity.md lists them, and no longer claims every NEF
coding but HE decodes. measured_bodies_are_consistent checks every entry lies
inside its data and is listed once; measured_older_dx_bodies_image_area
covers the D200, D3200, Nikon 1 V1 and best-guess entries.
Fixes#976Fixes#914Fixes#1236
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scan asked the disk about every library photo in the folder: is its
file there, and has it a sidecar under any of four names. On a network
share each question is a round trip, and the scan had just listed the
folder. Measured on a hard disk over SMB, for unchanged folders of 1,600
to 2,400 photos: 4.6 to 7.1 s of questions after a listing of 4 to 5 s.
- The walk hands each folder's listing over (`walk::files_in_noting`:
every entry's name and kind, hidden ones too, and whether that is all of
them). The scan walks the folder once, then checks the photos beside
the probing of new files.
- `sync::Listing` answers "is it there" / "is it a file" where that holds
on any file system: a name the folder lists is there; a plain ASCII name
is not there when the folder was listed to its end, all its names are
ASCII and none is that name in another case. Everything else is asked
of the disk as before (a link, a name Windows reads another way, a
folder that was not walked, and every name not listed in a folder that
holds a name outside ASCII).
Now under 0.05 s for those folders. Two things differ on purpose: a file
the folder lists is there though it can't be opened by its path (a folder
that may be listed but not searched, an over-long path), where such
photos were listed as missing; and the answers are the folder as the
walk saw it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Setting up Synchronize Folder, an import review or an import made a table
of every photo in the library on the session's thread before the first
file was looked at, and naming one photo's sidecar worked out who owns
every shared sidecar name in the library. In a library of 3.5 million
photos (`scan_setup_bench`, release, a folder of 500): 5.8 s before a
Synchronize scan, 2.6 s before an import review, 1.7 s before an import,
2.4 s for each sidecar written.
- A scan keeps the library as it is (photos are shared, not copied) and
looks up the files it finds (`library_files`) and the contents it probes
(`library_hashes`) on its own thread.
- An import looks the library up with its first files
(`ImportJob::look_up_library`), from the library and the options as they
were when it was set up.
- The owners of shared sidecar names are worked out for the photos asked
about (`StemOwners::among`): all five callers name only those photos.
- Synchronize Folder names its photos' sidecars in the scan, after it has
seen that the folder is there.
Now 0.9 s, 0.05 s, 0.05 s and under 1 s. What is found, and how sidecars
are named, is unchanged: the tests hold the owners against the table of
the whole library as it was made before, and the key made without
allocating against the key, for every string of up to six of the
characters that decide it.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- drag_latency (ui-egui example): the whole app driven headless at 60 frames a second; per develop control,
the latency of each drag tick until its draft is the loupe (or, with ZOOM, the zoom window) texture, the
ticks never shown, the UI thread's frame time and the time to the full-quality render after release.
- drag_tick_bench (engine example): the UI-thread work of one drag tick outside the render.
- render_bench: ONLY=window (a 1:1 zoom window cut from the original: first window, pan, slider changes;
GPU_ONLY=1, DUMP=file for pixel A/B), ONLY=load / load1 / jpeg (+ LOADONLY=1): the steps of opening a photo.
The window rows carry no colon in their names, so `cargo xtask bench` records each of them (parser test).
- open_timeline (ui-egui example): opening raws in Detail through the real app logic, headless with the GPU.
Co-Authored-By: Claude <noreply@anthropic.com>
Nikon bodies from before ~2016 record no CropArea, so the decoder kept
their whole sensor data, less trailing dark columns. Where that data
holds masked lines above black or optical black, they were developed as
a coloured or dark bar on that edge: the D7000 and D5100's 36 columns at
~600 (#976), the D800's 14 (#914, the top in portrait), and others.
MEASURED_BODIES (crates/raw/src/vendor/nef.rs) now gives 36 model /
data size / depth groups the frame of their camera JPEG inside the data:
D7000, D5100, D800, D800E, Df, the D4's 12-bit mode, D3, D3S, D700,
D300, D300S, D600, D610, D810, D4S, D5600, Coolpix A, D3000, D40X, D60,
D80 and D2Xs. An entry matches model, data size and depth exactly; the
measured white level becomes optional (only the D4's 14-bit mode has
one). Bodies whose data holds only picture beyond the camera frame get
no entry and render as before.
`cargo xtask raw-frame FILE...` (crates/engine/examples/raw_frame.rs)
measures a body: it aligns each raw's embedded camera JPEG with the
sensor data (half-size luminance, gradient correlation, sub-pixel peak;
weak, ambiguous or between-pixel peaks are flagged), counts the edge
lines that hold no picture and reports what the decoder's crop keeps
beyond the camera frame. It prints entries only with evidence: at least
3 clean files agreeing exactly, of the model (12- and 14-bit pooled) or,
with at least one of its own, with sibling models of the same readout
(data and JPEG size) that agree on the same frame. On the D7500, which
records its CropArea, it finds that area exactly; it declines the Z f,
whose JPEG is distortion-corrected.
It works for any make whose raw embeds a full-size camera JPEG: on
Sony, Canon CR2 / CR3, Pentax and Samsung files it finds the decoder's
own crop (relative to the active area).
docs/nikon-bodies.md surveys every Nikon raw on raw.pixls.us (98
models, 291 files; 138 CC0 samples measured): 22 models record
CropArea; of the 88 groups without it, 36 have entries, 13 are fine, 25
could not be aligned and 7 do not decode. The D3200, D3100, D200, D50,
D750 and Nikon 1 V1 / V2 are pending, each listed with the files it
needs; the doc also describes how to add a body. raw-parity.md no
longer claims every NEF coding but HE decodes.
measured_bodies_are_consistent checks every entry lies inside its data
and is listed once.
The evidence is committed in docs/nikon-bodies/: headers.json (every
file's raw size, depth and maker-note crop tags), samples.json (the 138
measured CC0 files with URL and sha256) and frames.jsonl (the tool's
per-file results). `cargo xtask corpus --download nikon` fetches the
samples (sha256 checked) so `cargo xtask raw-frame` reproduces every
count.
Fixes#976Fixes#914
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A batch ran its photos strictly one after another: each render is
row-parallel inside, but a photo's decode, its PNG/TIFF/WebP encode and
the wait on a GPU render are mostly serial, so a 32-core machine averaged
~9.5 busy cores on the CPU path and 1-2 with the GPU.
run_batch now keeps export_parallelism() photos in flight (cores / 4,
1..=4; one on wasm32) on scoped lane threads plus the calling thread.
Each holds an estimate of its working memory from a new export_gate
(twice the memory budget, separate from the decode gate it nests over),
so full-size 24 MP exports (~1.2 GB each) run two at a time by default.
Lanes run at most `lanes` photos past the last one placed, which also
bounds the finished files waiting for their turn. Paths, conflict
handling, original protection, writes, results and progress stay in
batch order (Placer), so the files are the same as one at a time; a
panic in a lane becomes that photo's error.
render_bench's batch scenario now goes through export_batch. 8 x 24 MP
JPEGs, 32 cores + RTX 4080 Laptop: CPU 3.87 s -> 2.35 s, GPU 4.21 s ->
1.82 s.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The RW2 reader reads PanasonicRaw tag 0x0119 (16 LE words) and expresses the
camera's own correction as the equivalent DNG WarpRectilinear in OpcodeList3,
so embedded_lens() turns it into the existing EmbeddedLens/EmbeddedWarp and
"Enable Profile Corrections" applies it like a DNG's warp (no new warp path;
Convert to DNG keeps it). Rule, from our own measurements against the
embedded camera JPEGs (clean-room; ExifTool tag names only):
corrected = rho*(1 + (w8 rho^2 + w4 rho^4 + w11 rho^6)/32768), rho = r/w12
(w12 = active-area half diagonal), output = corrected / (1 + w5/32768),
on when (w7 & 0xf) == 1. Chosen on 38 GX85 photos, held out on 35 photos
from 12 other bodies / 9 other lenses; corrected renders line up with the
camera JPEG within 0.75 px at 1440 px at the corners (up to 13 % of the
half diagonal before). Implausible values leave the photo uncorrected.
The loader now removes lens opcodes before the RW2 camera-look fit (its
binned proxy needs an empty OpcodeList3); Opcode::is_lens_correction()
replaces the duplicated matches. RENDER_CACHE_VERSION 11 -> 12.
Fix-ups: diagnostic overlays never populate the view-preview cache; Photo Merge's RenderRequest gets
the new overlay field.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Up to 8 OkLCh samples (pointColor.pick samples the colour after the mixer on a proxy render);
per sample hue/saturation/luminance shift, variance (compress/expand the spread around the
sample) and an overall + per-axis range with soft edges, applied after the colour mixer in
ColorOps and its WGSL twin (equivalence case). Indexed control specs
(pointColor.<i>.hueShift …) so develop.set/controls/MCP address each sample. Visualize range
is a render overlay (RenderRequest::overlay) drawn after either renderer: the sample's own
edit is left out and colours outside the range turn grey. Point Color flyout in the Color
section with swatches, eyedropper, sliders, visualize and delete.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- lightcraft-raw: RawImage::develop_binned(k) — k×k superpixel binning of the CFA
(Bayer k even, X-Trans k multiple of 3), black/white normalised row by row, default
crop only, clipping preserved for highlight reconstruction. 2560 px previews of a
24 MP raw bin 2×, thumbnails 4–8×; exports / 1:1 still demosaic (AHD) at full size.
- highlight::reconstruct and its pyramid fill are rayon-parallel (bit-exact).
- X-Trans directional demosaic: precomputed per-phase neighbour offsets and no
reflection in the interior, no per-pixel allocation (bit-exact, tested against the
reference formulation): 2.9 s → 0.75 s CPU at 24 MP.
- files::load_bytes runs on a rayon worker so its short parallel loops don't each
wait for the pool to wake (dominant cost on a loaded machine); LIGHTCRAFT_PROFILE
prints per-stage source timings; files::bin_factor picks the block size.
- render_bench ONLY=source: per-file decode / normalise / demosaic / binning / load.
- Fix: u64 overflow on hostile IFD sizes in DNG/ARW/NEF raw-IFD selection (proptest).
load_bytes(2560) CPU time, ARW a7 III 24 MP: 2347 → 240 ms; X-T20: 3120 → 1220 ms.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
`media::develop` renders on the GPU when available (views, render_now, exports;
thumbnails stay on the CPU) and falls back to the CPU otherwise. `app.gpu
{enabled?}` toggles it at runtime, `LIGHTCRAFT_GPU=0` disables it for the process,
`ui.inspect` reports the adapter under perf.gpu.
Performance: dropped device buffers are recycled once their thread has submitted
(fresh 100-300 MB allocations per pass cost as much as the passes), Gaussian
passes ping-pong between two buffers and read all channels of a pixel at once,
and long box radii slide over longer chunks. LIGHTCRAFT_PROFILE prints real GPU
stage timings. render_bench prints CPU and GPU columns and the CPU/GPU difference.
24 MP ARW, loaded machine, min of 7: cold loupe 415 -> 32 ms, exposure drag
22-42 -> 3.9 ms, NR drag (1920) 301+ -> 18 ms, export render 1037+ -> 291 ms;
max 1 LSB difference. docs/gpu-pipeline.md describes the implementation.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The guided base, clarity band, texture band and dark channel (and the paired blurs inside the
guided filter) are independent; each alone scales poorly (subsampled grids, short rows), so they now
run under raster::par_join. Same arithmetic, identical pixels. render_bench gains a per-feature
breakdown of the per-pixel stage (ONLY=finish).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Exports above 2560 px were rendered from the 2560 px preview proxy and upsampled. SourceLevel::Full
(native size; procedural demo scenes at their nominal size) is now selected for outputs larger than
a preview; the media cache keeps only the last full-size original. render_bench gains a batch-export
scenario (decode + render + JPEG per file).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
- pipeline: StageCache + render_cached: resampled source, white-balanced/retouched/denoised image
and each spatial plane (log L, guided base, clarity, texture, dark channel + airlight) are keyed
by exactly their inputs; keeps a draft-size and a full-size entry warm.
- Exposure moved after the spatial filters (applied per pixel; planes shifted by EV): identical up
to rounding (test: <= 2 LSB vs. exposing the source), so exposure drags hit the cache too.
- engine: RenderJob::draft / with_stages; ui: loupe and before views own a StageCache, loupe
renders at Draft quality while interacting.
- engine example render_bench (min-of-N wall + process CPU time).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
