mirror of
https://github.com/storytold/effectcraft.git
synced 2026-10-22 11:42:57 -05:00
bflatasticandClaude Opus 5.5
e43bf66775
- effectcraft-segment: a FaceModel interface (find a face in a region, follow it; a Topology maps the model's points to the tracker's landmarks and outline), a TensorFlow Lite reader and interpreter (float ops, float16 weights folded; matches ai-edge-litert to ~1e-6), and MediaPipe Face Landmarker (Google, Apache-2.0: BlazeFace + Face Mesh V2, 3.8 MB, pinned SHA-256). The registry gains a task and authors per model; installed weights get a NOTICE. - effectcraft-track: FaceTracker::new_with uses the model, falls back to the classical fitter when it finds no face; chin and jaw come from the outline either way so measurements agree. - Engine: roto_models becomes models (both tasks); face.models / face.model.* commands, Settings ▸ Face Tracking, `wait` on select/install/download for scripts; track.mask reports faceModel. - Credits: ATTRIBUTION.md lists the optional models and the Apache-2.0 specs the code follows. - Docs: README, ROADMAP (G7), gaps, parity (TRK-3 score unchanged pending G1), agents, preferences, architecture, segment README. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
65 lines
4.6 KiB
Markdown
65 lines
4.6 KiB
Markdown
# effectcraft-segment
|
|
|
|
Trained models for Roto Brush and face tracking, each kind behind one swappable interface.
|
|
|
|
- **`MaskModel`**: what Roto Brush asks of a model: given a frame and prompts (foreground /
|
|
background points, a box, a prior mask), the foreground probability of every pixel. Roto Brush
|
|
(`effectcraft-track`) only talks to this trait, so models can be added or swapped without
|
|
touching it. The classical graph-cut segmenter stays built in as the fallback.
|
|
- **`face::FaceModel`**: what face tracking asks of a model: find a face in a region of a frame,
|
|
then follow it from frame to frame. A model reports its own points plus a `Topology` naming
|
|
which are the tracker's landmarks and which trace the face outline, so the tracker never
|
|
depends on one model's layout. The classical face tracker stays built in as the fallback.
|
|
- **`MODELS`**: the registry. Every entry must be open source under a licence compatible with
|
|
EffectCraft's MIT OR Apache-2.0 (`ALLOWED_LICENCES`), with its task, authors, homepage, licence
|
|
URL, the official weights URL, size and SHA-256. `load` refuses anything else.
|
|
- **Weights are never bundled.** They are downloaded on demand (or installed from a file) by the
|
|
engine (`roto.model.*` / `face.model.*`, Settings ▸ Roto Brush / Face Tracking), verified
|
|
against the registry's SHA-256 (`sha256`, dependency-free) and kept with a notice naming their
|
|
authors, licence and source (`notice`).
|
|
- **`pt`**: reads PyTorch checkpoints (`torch.save` zip archives and the pickle subset state dicts
|
|
use), so official weights load exactly as published.
|
|
- **`tflite`**: reads TensorFlow Lite models (the FlatBuffer laid out by TensorFlow Lite's
|
|
`schema.fbs`) and runs the float operators small vision models use (convolutions, depthwise
|
|
convolutions, pooling, PReLU, padding, concatenation; float16 weights dequantised at load).
|
|
- **`mobilesam`**: MobileSAM in plain Rust: the TinyViT-5M image encoder and Segment Anything's
|
|
prompt encoder and two-way-transformer mask decoder.
|
|
- **`mediapipe`**: MediaPipe Face Landmarker: the BlazeFace short-range detector (SSD anchors,
|
|
weighted non-maximum suppression) and Face Mesh V2 (478 points), with the rotated crops
|
|
MediaPipe's face landmarker graph uses.
|
|
|
|
All of it runs on `nn` kernels (rayon, and ndarray's safe `general_mat_mul`, which picks AVX/FMA
|
|
kernels at run time). No ML framework, no unsafe code.
|
|
|
|
## Models
|
|
|
|
| id | Task | Model | Authors | Licence | File | Size |
|
|
|---|---|---|---|---|---|---|
|
|
| `mobilesam` | Roto Brush | [MobileSAM](https://github.com/ChaoningZhang/MobileSAM): Segment Anything with a 5M-parameter TinyViT encoder | Chaoning Zhang et al., building on Meta AI's Segment Anything and Microsoft's TinyViT | Apache-2.0 | `mobile_sam.pt` from the official repository | 40.7 MB |
|
|
| `mediapipe-face` | Face tracking | [MediaPipe Face Landmarker](https://developers.google.com/edge/mediapipe/solutions/vision/face_landmarker): BlazeFace detector and Face Mesh V2 | Google LLC | Apache-2.0 ([model card](https://storage.googleapis.com/mediapipe-assets/Model%20Card%20MediaPipe%20Face%20Mesh%20V2.pdf)) | `face_landmarker.task` (float16, version 1) from Google's model storage | 3.8 MB |
|
|
|
|
How they were checked:
|
|
|
|
- **MobileSAM**: the implementation follows the published architectures (MobileSAM, TinyViT and
|
|
Segment Anything papers and their Apache-2.0 reference code). Its output was checked against an
|
|
independent implementation (Hugging Face candle's, MIT/Apache-2.0) on the same weights:
|
|
embeddings within 0.3% RMS (exact vs. tanh GELU) and 99.9% of mask pixels equal.
|
|
- **MediaPipe Face Landmarker**: the interpreter's outputs match TensorFlow Lite's own
|
|
(`ai-edge-litert` 2.2.0) on both models to about one part in a million (a test pins them). The
|
|
whole pipeline's 478 points on a test portrait are 0.85 px from MediaPipe's own Python pipeline
|
|
on average (2.2 px at worst). Finding a face takes about 45 ms; following it, about 20 ms a
|
|
frame (32-thread CPU).
|
|
|
|
Tests that need the weights run when `EFFECTCRAFT_MOBILESAM` / `EFFECTCRAFT_FACE_LANDMARKER`
|
|
point at the files, and are skipped otherwise.
|
|
|
|
## Adding a model
|
|
|
|
1. Pick a model whose code **and** weights are open source under an allowed licence. Check the
|
|
weights' own terms: many models with MIT code ship weights trained on research-only data.
|
|
2. Add a `ModelInfo` to `MODELS` (its task and authors; pin the official file's SHA-256 and size),
|
|
and credit it in the root `ATTRIBUTION.md`.
|
|
3. Implement `MaskModel` or `FaceModel` for it and add its loader to `load`.
|
|
4. Test it against a reference implementation; Settings, the `roto.model.*` / `face.model.*`
|
|
commands and the tools pick it up from the registry.
|