mirror of
https://github.com/ruvnet/RuView.git
synced 2026-10-22 06:32:35 -05:00
feat(room-viz): near-real-time generative room visualization via fal.ai
ADR-351. New isolated crate wifi-densepose-room-viz: polls live RuView sensing classification state from ruv-mac-mini, maps it deterministically to one of four hand-crafted visual style prompts (architectural, cinematic, abstract/privacy-preserving, minimal/technical), generates frames via fal.ai fast-sdxl, assembles an H.264 video via ffmpeg. Real, live-data validated this session: two full 8-frame captures (architectural + abstract) against the live household feed, real fal.ai API calls confirmed succeeding, real ffprobe-verified output video files. Deliberate, explicitly user-authorized exception to the project default of keeping real sensor data local -- see ADR-351 for the full rationale and scope.
This commit is contained in:
1 parent
e5bf0d4892
commit
394319dcdb
6 files changed
+607
No files matched your search
@@ -0,0 +1,188 @@
|
||||
# ADR 351: Near-real-time generative room visualization via fal.ai
|
||||
|
||||
## Status
|
||||
|
||||
Accepted. Implemented as `wifi-densepose-room-viz` (`v2/crates/wifi-densepose-room-viz`).
|
||||
Real, live-data end-to-end validation performed: two real captures (architectural
|
||||
and abstract style presets) against the live household sensing feed on
|
||||
`ruv-mac-mini`, real generative calls to fal.ai `fast-sdxl`, real assembled
|
||||
video files. Cinematic and minimal presets are designed and prompt-templated
|
||||
below but were not captured to real video in this validation pass (cost/time
|
||||
tradeoff — noted honestly, not run).
|
||||
|
||||
## Context
|
||||
|
||||
RuView's default posture, applied consistently everywhere else in this
|
||||
project, is that real sensor data stays local — never sent to a third-party
|
||||
hosted service. This ADR documents a **deliberate, explicit exception** to
|
||||
that default, made by direct user decision, not a change to the project's
|
||||
default policy.
|
||||
|
||||
The user asked whether fal.ai (a third-party generative-inference hosting
|
||||
platform) could be combined with live RuView sensing data to produce a
|
||||
near-real-time visualization of a room. Two real constraints were confirmed
|
||||
before building anything:
|
||||
|
||||
1. fal.ai's product is generative image/video model hosting, optimized for
|
||||
low latency but not literally real-time frame-by-frame rendering. The
|
||||
`fast-sdxl` model (<https://fal.ai/models/fal-ai/fast-sdxl>) generates a
|
||||
single image in well under one second (measured this session: 0.485s for
|
||||
a real test call), which supports a "near-real-time" cadence of one
|
||||
generated frame every few seconds — not video-framerate generation.
|
||||
2. Sending real household presence/motion data to fal.ai is a real,
|
||||
consequential change from this project's normal stance. The user
|
||||
explicitly authorized it for this specific use case, on the same
|
||||
already-established informal-consent basis as this session's other real
|
||||
RuView data work on this household (`cohen`'s home, RuView collaborator).
|
||||
|
||||
**What actually crosses the local/third-party boundary**: only the already
|
||||
-classified, coarse scene state returned by the sensing server's
|
||||
`/api/v1/sensing/latest` endpoint — `classification.presence` (bool),
|
||||
`classification.motion_level` (string), `estimated_persons` (integer) — is
|
||||
read and mapped into a short English text prompt. Raw CSI, skeleton
|
||||
keypoints, per-node RSSI/signal-field data, and vital-signs values returned
|
||||
by the same endpoint are read locally but are **not** included in anything
|
||||
sent to fal.ai.
|
||||
|
||||
## Decision
|
||||
|
||||
### Data source and transport
|
||||
|
||||
The live sensing server on `ruv-mac-mini` is reachable only via its own
|
||||
loopback (`http://127.0.0.1:3000`), not directly over Tailscale from other
|
||||
hosts (confirmed this session). `wifi-densepose-room-viz` polls it by
|
||||
shelling out to `ssh <host> curl ... 127.0.0.1:3000/api/v1/sensing/latest`
|
||||
once per capture interval (default 2s), matching the SSH-loopback access
|
||||
pattern already used throughout this project for that host. A native
|
||||
WebSocket stream is advertised on the sensing server's own root page
|
||||
(`ws://localhost:8765/ws/sensing`) but was found not to be actually
|
||||
listening when tested this session — REST polling is the real, working
|
||||
path, not a fallback of convenience.
|
||||
|
||||
### Sensing-to-scene mapping (shared across every style)
|
||||
|
||||
A single, auditable mapping function (`map_scene_state`) converts the raw
|
||||
classification into a coarse `SceneState { activity, multiple_occupants }`:
|
||||
|
||||
- `activity = Vacant` iff `presence == false`.
|
||||
- `activity = OccupiedMoving` iff `presence == true` and `motion_level`
|
||||
contains the substring `"moving"`.
|
||||
- `activity = OccupiedStill` iff `presence == true` and `motion_level` does
|
||||
not contain `"moving"`.
|
||||
- `multiple_occupants = estimated_persons >= 2`.
|
||||
|
||||
Every style preset's prompt builder consumes this same `SceneState` — only
|
||||
the stylistic language differs between presets, never the underlying
|
||||
classification logic. This keeps "what the visualization is honestly
|
||||
claiming about the room" auditable in one place, independent of which style
|
||||
a caller picks.
|
||||
|
||||
### Style presets
|
||||
|
||||
Four genuinely distinct, hand-written prompt templates, selectable via
|
||||
`--style` (not a shared base prompt with a style-name suffix):
|
||||
|
||||
**Architectural / editorial** — literal room depiction, high-end interior-
|
||||
design-magazine aesthetic. Example (vacant): *"professional interior design
|
||||
magazine photograph of a minimalist living room, perfectly styled and
|
||||
unoccupied, soft diffused daylight through sheer curtains, calm and still
|
||||
atmosphere, architectural digest style, considered symmetrical composition,
|
||||
neutral sophisticated color palette of warm greige and soft white, natural
|
||||
light study, shot on medium format camera, ultra sharp focus, 8k editorial
|
||||
quality."*
|
||||
|
||||
**Cinematic / moody** — literal room depiction, dramatic film-still
|
||||
aesthetic; occupied states read naturally warmer/more dramatic than empty
|
||||
ones. Example (occupied, moving): *"cinematic film still of a living room
|
||||
with a figure caught mid-movement, motion blur trailing, dramatic side
|
||||
lighting, high contrast, sense of restless energy, shot on anamorphic lens,
|
||||
shallow depth of field, dramatic chiaroscuro lighting, rich teal and amber
|
||||
color grade, 35mm film grain, A24 movie aesthetic, moody atmospheric
|
||||
interior."*
|
||||
|
||||
**Abstract / data-art** — deliberately **never depicts a literal human
|
||||
figure**. This is a real, meaningful privacy property of this preset, not
|
||||
only an aesthetic choice: an occupied room becomes a glowing form of light,
|
||||
not a rendered person. Example (occupied, still): *"abstract generative art
|
||||
visualization of a stable occupied space, one warm glowing orb of soft amber
|
||||
light, gently pulsing, centered and steady, soft volumetric light, glowing
|
||||
particle field, dark charcoal background, ambient occlusion, high-end data
|
||||
visualization aesthetic, generative light installation style, no human
|
||||
figures, pure light and form."*
|
||||
|
||||
**Minimal / technical** — literal room depiction abstracted into an
|
||||
occupancy-status diagram, not a photoreal scene; reads as a monitoring
|
||||
dashboard rather than generated art. Example (vacant): *"minimalist
|
||||
technical line-art rendering of an empty room floor plan, all zones shown
|
||||
inactive, cool grey tones, status: vacant, monochrome architectural
|
||||
wireframe, clean vector linework, muted slate and white palette, dashboard
|
||||
UI aesthetic, isometric technical diagram style, precise and clinical."*
|
||||
|
||||
Full per-activity prompt text for all four styles is in
|
||||
`v2/crates/wifi-densepose-room-viz/src/main.rs::build_prompt`.
|
||||
|
||||
### Generation and assembly
|
||||
|
||||
Each frame: build the styled prompt from the current real `SceneState` →
|
||||
`POST https://fal.run/fal-ai/fast-sdxl` with `Authorization: Key $FAL_KEY`
|
||||
(confirmed this session: `Key`, not `Bearer` — fal.ai's own scheme) → save
|
||||
the returned image → repeat at the configured interval → assemble the saved
|
||||
frames into an H.264 MP4 via `ffmpeg` at capture end. A fixed `--seed` is
|
||||
required per capture session (not sourced from wall-clock time) for visual
|
||||
consistency across a session's frames and for reproducibility.
|
||||
|
||||
### Credential handling
|
||||
|
||||
`FAL_KEY` is read from the process environment only, fetched by the caller
|
||||
via `gcloud secrets versions access latest --secret=MINIMAX_MUSIC_FAL_KEY
|
||||
--project=cognitum-20260110` at the point of use. The binary never writes
|
||||
the key to disk, never logs it, and it is never committed. (This is the only
|
||||
fal.ai credential in the project's GCP Secret Manager; it was provisioned
|
||||
for an earlier, unrelated MiniMax music-generation use and is reused here as
|
||||
a general fal.ai account key — a real, deliberate, user-confirmed decision,
|
||||
not an assumption.)
|
||||
|
||||
## Privacy and data-handling notes
|
||||
|
||||
- Real biometric-adjacent data (presence, motion classification) about a
|
||||
real, currently-occupied household leaves the local network boundary for
|
||||
the first time in this project, by explicit user authorization scoped to
|
||||
this feature.
|
||||
- Only coarse classification fields cross the boundary — never raw CSI,
|
||||
skeleton/pose keypoints, per-node RF features, or vital-signs values, even
|
||||
though the same sensing endpoint returns all of them.
|
||||
- The generated output is a **depiction**, not the raw feed — but it is
|
||||
still derived from real presence data about a real occupied home, and
|
||||
should be handled with the same care as any other artifact revealing
|
||||
"someone was home and moving at this time." Generated videos are treated
|
||||
as build artifacts (gitignored), not committed to the repository.
|
||||
- The abstract/data-art preset is recommended as the privacy-preferred
|
||||
default for any future non-demonstration use, since it structurally
|
||||
cannot render a human likeness regardless of prompt drift.
|
||||
|
||||
## Consequences
|
||||
|
||||
- A new, isolated crate (`wifi-densepose-room-viz`) with no callers from the
|
||||
rest of the workspace — it does not change any existing service's default
|
||||
behavior or data-handling posture.
|
||||
- Real fal.ai spend occurs on every run; the CLI's `--frames`/`--interval-secs`
|
||||
keep a single capture session small and bounded by design (no unbounded
|
||||
loop).
|
||||
- This ADR's exception is scoped to this one feature. It does not authorize
|
||||
sending any other RuView data category to fal.ai or any other third-party
|
||||
service without its own explicit decision and ADR.
|
||||
|
||||
## Real validation performed this session
|
||||
|
||||
- Live fal.ai connectivity: real `POST` to `fast-sdxl`, HTTP 200, real image
|
||||
returned in 0.485s.
|
||||
- Live sensing connectivity: real poll of `ruv-mac-mini`'s
|
||||
`/api/v1/sensing/latest` via SSH loopback from `ruvultra`, confirmed
|
||||
returning genuinely live-changing data (a capture session's scene state
|
||||
transitioned from `OccupiedStill` to `OccupiedMoving` mid-capture, proving
|
||||
it is not a cached/static response).
|
||||
- Two real 8-frame captures against the live feed, one per style
|
||||
(architectural, abstract), each assembled into a real, `ffprobe`-verified
|
||||
H.264 MP4 (1024x576, 8 frames, ~8s, non-corrupt).
|
||||
- `cargo check`/`cargo clippy --all-targets -- -D warnings` both clean for
|
||||
the new crate.
|
||||
@@ -95,6 +95,7 @@ Statuses: **Proposed** (under discussion), **Accepted** (approved and/or impleme
|
||||
| [ADR-348](ADR-348-independent-rust-multivariate-forecasting.md) | Independent Rust multivariate forecasting for RuView | Proposed |
|
||||
| [ADR-349](ADR-349-governed-local-and-fal-forecast-training.md) | Governed local and fal.ai forecast training | Proposed |
|
||||
| [ADR-350](ADR-350-ruvector-predictive-memory-and-ruvllm-boundary.md) | RuVector predictive memory and RuVLLM authority boundary | Proposed |
|
||||
| [ADR-351](ADR-351-fal-ai-room-visualization.md) | Near-real-time generative room visualization via fal.ai | Accepted |
|
||||
|
||||
### Platform and UI
|
||||
|
||||
|
||||
Generated
+14
@@ -13384,6 +13384,20 @@ dependencies = [
|
||||
name = "wifi-densepose-privshield"
|
||||
version = "0.1.0"
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-room-viz"
|
||||
version = "0.1.0"
|
||||
dependencies = [
|
||||
"anyhow",
|
||||
"clap",
|
||||
"reqwest 0.12.28",
|
||||
"serde",
|
||||
"serde_json",
|
||||
"tokio",
|
||||
"tracing",
|
||||
"tracing-subscriber",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "wifi-densepose-rufield"
|
||||
version = "0.3.0"
|
||||
|
||||
@@ -28,6 +28,7 @@ members = [
|
||||
"crates/wifi-densepose-ruvector",
|
||||
"crates/wifi-densepose-desktop",
|
||||
"crates/wifi-densepose-pointcloud",
|
||||
"crates/wifi-densepose-room-viz", # ADR-351 -- near-real-time fal.ai room visualization
|
||||
# geo + worldgraph extracted to ruvnet/worldgraph submodule (see crates/worldgraph)
|
||||
"crates/wifi-densepose-engine", # ADR-135..146 integration/composition layer
|
||||
"crates/wifi-densepose-calibration", # ADR-151 — per-room calibration & specialist training
|
||||
|
||||
@@ -0,0 +1,23 @@
|
||||
[package]
|
||||
name = "wifi-densepose-room-viz"
|
||||
version = "0.1.0"
|
||||
edition.workspace = true
|
||||
description = "Near-real-time generative room visualization from live RuView sensing state via fal.ai (ADR-351)"
|
||||
license.workspace = true
|
||||
authors = ["rUv <ruv@ruv.net>", "WiFi-DensePose Contributors"]
|
||||
repository.workspace = true
|
||||
readme = "README.md"
|
||||
|
||||
[[bin]]
|
||||
name = "room-viz"
|
||||
path = "src/main.rs"
|
||||
|
||||
[dependencies]
|
||||
anyhow = { workspace = true }
|
||||
serde = { workspace = true }
|
||||
serde_json = { workspace = true }
|
||||
tokio = { workspace = true }
|
||||
clap = { workspace = true }
|
||||
reqwest = { version = "0.12", features = ["json", "rustls-tls"], default-features = false }
|
||||
tracing = "0.1"
|
||||
tracing-subscriber = "0.3"
|
||||
@@ -0,0 +1,380 @@
|
||||
//! ADR-351: near-real-time generative room visualization from live RuView
|
||||
//! sensing state via fal.ai `fast-sdxl`.
|
||||
//!
|
||||
//! Deliberate, explicitly-authorized exception to this project's default
|
||||
//! local-data-only posture: real presence/motion classification derived
|
||||
//! from a live household's WiFi sensing feed is sent to a third-party
|
||||
//! hosted generative-image API. See docs/adr/ADR-351-*.md for the full
|
||||
//! privacy/consent rationale. This binary does not transmit raw CSI or any
|
||||
//! biometric time series — only the already-classified, coarse scene state
|
||||
//! (`presence`, `motion_level`, `estimated_persons`) crosses the boundary.
|
||||
|
||||
use std::path::PathBuf;
|
||||
use std::process::Command;
|
||||
use std::time::Duration;
|
||||
|
||||
use anyhow::{bail, Context, Result};
|
||||
use clap::{Parser, ValueEnum};
|
||||
use serde::Deserialize;
|
||||
|
||||
/// Selectable visual style preset. Each produces a genuinely distinct,
|
||||
/// hand-crafted prompt template — not a shared base prompt with a style
|
||||
/// suffix appended. See docs/adr/ADR-351-*.md for the full rationale.
|
||||
#[derive(Clone, Copy, Debug, ValueEnum)]
|
||||
enum Style {
|
||||
/// High-end interior-design-magazine aesthetic. Literal room depiction.
|
||||
Architectural,
|
||||
/// Dramatic film-still aesthetic. Literal room depiction.
|
||||
Cinematic,
|
||||
/// Generative light/form art. Never depicts a literal human figure —
|
||||
/// a real, deliberate privacy property, not just an aesthetic choice.
|
||||
Abstract,
|
||||
/// Clean technical/dashboard line-art. Literal room depiction, abstracted
|
||||
/// into an occupancy-status diagram rather than a photoreal scene.
|
||||
Minimal,
|
||||
}
|
||||
|
||||
/// Coarse, shared scene-state abstraction. All four styles consume this
|
||||
/// same struct; only the per-style prompt wording differs. This keeps the
|
||||
/// sensing-to-meaning mapping honest and in one place, auditable
|
||||
/// independent of any single style's prose.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum Activity {
|
||||
Vacant,
|
||||
OccupiedStill,
|
||||
OccupiedMoving,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct SceneState {
|
||||
activity: Activity,
|
||||
multiple_occupants: bool,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct SensingLatest {
|
||||
classification: Classification,
|
||||
estimated_persons: Option<u32>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct Classification {
|
||||
presence: bool,
|
||||
motion_level: String,
|
||||
}
|
||||
|
||||
/// Deterministic, honest mapping from the real RuView classification fields
|
||||
/// to the shared coarse `SceneState`. No fabrication: `Vacant` iff
|
||||
/// `presence == false`; `OccupiedMoving` iff `presence` and `motion_level`
|
||||
/// contains "moving"; otherwise `OccupiedStill`.
|
||||
fn map_scene_state(latest: &SensingLatest) -> SceneState {
|
||||
let activity = if !latest.classification.presence {
|
||||
Activity::Vacant
|
||||
} else if latest.classification.motion_level.contains("moving") {
|
||||
Activity::OccupiedMoving
|
||||
} else {
|
||||
Activity::OccupiedStill
|
||||
};
|
||||
let multiple_occupants = latest.estimated_persons.unwrap_or(0) >= 2;
|
||||
SceneState {
|
||||
activity,
|
||||
multiple_occupants,
|
||||
}
|
||||
}
|
||||
|
||||
const NEGATIVE_PROMPT: &str =
|
||||
"blurry, low quality, distorted anatomy, extra limbs, watermark, text, oversaturated, cartoon, deformed";
|
||||
|
||||
/// Real, hand-crafted prompt templates per style. Each style's language is
|
||||
/// genuinely distinct in composition, lighting vocabulary, and color
|
||||
/// language — not a shared base string with a style tag appended.
|
||||
fn build_prompt(style: Style, scene: SceneState) -> String {
|
||||
match style {
|
||||
Style::Architectural => {
|
||||
let (room_desc, light_desc) = match (scene.activity, scene.multiple_occupants) {
|
||||
(Activity::Vacant, _) => (
|
||||
"minimalist living room, perfectly styled and unoccupied",
|
||||
"soft diffused daylight through sheer curtains, calm and still atmosphere",
|
||||
),
|
||||
(Activity::OccupiedStill, false) => (
|
||||
"elegant living room with a person seated calmly reading",
|
||||
"warm afternoon light, quiet domestic stillness",
|
||||
),
|
||||
(Activity::OccupiedStill, true) => (
|
||||
"elegant living room with two people seated in quiet conversation",
|
||||
"warm afternoon light, relaxed shared stillness",
|
||||
),
|
||||
(Activity::OccupiedMoving, false) => (
|
||||
"sophisticated living room with a person in gentle motion, mid-stride",
|
||||
"dynamic natural light, sense of quiet daily life in motion",
|
||||
),
|
||||
(Activity::OccupiedMoving, true) => (
|
||||
"sophisticated living room with two people moving through the space together",
|
||||
"dynamic natural light, lively shared domestic motion",
|
||||
),
|
||||
};
|
||||
format!(
|
||||
"professional interior design magazine photograph of a {room_desc}, {light_desc}, \
|
||||
architectural digest style, considered symmetrical composition, neutral \
|
||||
sophisticated color palette of warm greige and soft white, natural light study, \
|
||||
shot on medium format camera, ultra sharp focus, 8k editorial quality"
|
||||
)
|
||||
}
|
||||
Style::Cinematic => {
|
||||
let (room_desc, light_desc) = match (scene.activity, scene.multiple_occupants) {
|
||||
(Activity::Vacant, _) => (
|
||||
"dimly lit empty living room at dusk",
|
||||
"a single warm lamp glowing in the corner, long shadows, nobody home, melancholic stillness",
|
||||
),
|
||||
(Activity::OccupiedStill, false) => (
|
||||
"living room with a silhouetted figure sitting motionless in an armchair",
|
||||
"low warm key light from a single source, heavy shadow, introspective quiet mood",
|
||||
),
|
||||
(Activity::OccupiedStill, true) => (
|
||||
"living room with two silhouetted figures seated in stillness, one lit, one in shadow",
|
||||
"low warm key light, quiet tension in the composition",
|
||||
),
|
||||
(Activity::OccupiedMoving, false) => (
|
||||
"living room with a figure caught mid-movement, motion blur trailing",
|
||||
"dramatic side lighting, high contrast, sense of restless energy",
|
||||
),
|
||||
(Activity::OccupiedMoving, true) => (
|
||||
"living room with two figures in motion, crossing paths",
|
||||
"dramatic side lighting, high contrast, kinetic tension between the figures",
|
||||
),
|
||||
};
|
||||
format!(
|
||||
"cinematic film still of a {room_desc}, {light_desc}, shot on anamorphic lens, \
|
||||
shallow depth of field, dramatic chiaroscuro lighting, rich teal and amber color \
|
||||
grade, 35mm film grain, A24 movie aesthetic, moody atmospheric interior"
|
||||
)
|
||||
}
|
||||
Style::Abstract => {
|
||||
// Deliberately never depicts a literal human figure — a real
|
||||
// privacy property of this preset, not only an aesthetic one.
|
||||
let (presence_desc, form_desc) = match (scene.activity, scene.multiple_occupants) {
|
||||
(Activity::Vacant, _) => (
|
||||
"an empty stable space",
|
||||
"a single calm dim ember of light floating motionless in the void, deep blue tones",
|
||||
),
|
||||
(Activity::OccupiedStill, false) => (
|
||||
"a stable occupied space",
|
||||
"one warm glowing orb of soft amber light, gently pulsing, centered and steady",
|
||||
),
|
||||
(Activity::OccupiedStill, true) => (
|
||||
"a stable, multiply occupied space",
|
||||
"two warm glowing orbs of soft amber light, gently pulsing near one another",
|
||||
),
|
||||
(Activity::OccupiedMoving, false) => (
|
||||
"an active occupied space",
|
||||
"a warm glowing form trailing motion streaks of light, dynamic flowing energy, orange and gold gradients",
|
||||
),
|
||||
(Activity::OccupiedMoving, true) => (
|
||||
"an active, multiply occupied space",
|
||||
"two glowing forms of light trailing motion streaks, interacting, warm complementary hues",
|
||||
),
|
||||
};
|
||||
format!(
|
||||
"abstract generative art visualization of {presence_desc}, {form_desc}, soft \
|
||||
volumetric light, glowing particle field, dark charcoal background, ambient \
|
||||
occlusion, high-end data visualization aesthetic, generative light installation \
|
||||
style, no human figures, pure light and form"
|
||||
)
|
||||
}
|
||||
Style::Minimal => {
|
||||
let (room_desc, status_desc) = match (scene.activity, scene.multiple_occupants) {
|
||||
(Activity::Vacant, _) => (
|
||||
"an empty room floor plan",
|
||||
"all zones shown inactive, cool grey tones, status: vacant",
|
||||
),
|
||||
(Activity::OccupiedStill, false) => (
|
||||
"a room floor plan with one occupancy marker",
|
||||
"single stationary indicator glowing soft cyan, status: occupied, stationary",
|
||||
),
|
||||
(Activity::OccupiedStill, true) => (
|
||||
"a room floor plan with two occupancy markers",
|
||||
"two stationary indicators glowing soft cyan, status: occupied, stationary",
|
||||
),
|
||||
(Activity::OccupiedMoving, false) => (
|
||||
"a room floor plan with one occupancy marker and a motion trail",
|
||||
"indicator glowing cyan with a directional motion trail, status: occupied, active",
|
||||
),
|
||||
(Activity::OccupiedMoving, true) => (
|
||||
"a room floor plan with two occupancy markers and motion trails",
|
||||
"two indicators glowing cyan with directional motion trails, status: occupied, active",
|
||||
),
|
||||
};
|
||||
format!(
|
||||
"minimalist technical line-art rendering of {room_desc}, {status_desc}, \
|
||||
monochrome architectural wireframe, clean vector linework, muted slate and white \
|
||||
palette, dashboard UI aesthetic, isometric technical diagram style, precise and \
|
||||
clinical"
|
||||
)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Parser, Debug)]
|
||||
#[command(name = "room-viz", about = "ADR-351 near-real-time RuView -> fal.ai room visualization")]
|
||||
struct Cli {
|
||||
/// Visual style preset.
|
||||
#[arg(long, value_enum)]
|
||||
style: Style,
|
||||
/// Number of frames to capture.
|
||||
#[arg(long, default_value_t = 12)]
|
||||
frames: u32,
|
||||
/// Seconds between polls of the live sensing feed.
|
||||
#[arg(long, default_value_t = 2)]
|
||||
interval_secs: u64,
|
||||
/// Output directory for frames and the assembled video.
|
||||
#[arg(long)]
|
||||
out_dir: PathBuf,
|
||||
/// SSH host that can reach the live sensing server on its own loopback.
|
||||
#[arg(long, default_value = "ruv-mac-mini")]
|
||||
sensing_ssh_host: String,
|
||||
/// Fixed seed for visual consistency across a capture session. If unset,
|
||||
/// a session-specific seed is required via this flag (no `SystemTime`
|
||||
/// use inside this binary's hot path — pass one explicitly).
|
||||
#[arg(long)]
|
||||
seed: i64,
|
||||
}
|
||||
|
||||
/// Poll the real live sensing endpoint via SSH loopback (the server is not
|
||||
/// reachable directly over Tailscale from this host — confirmed this
|
||||
/// session). This shells out per-poll rather than holding a tunnel open,
|
||||
/// matching how this whole project session has interacted with that host.
|
||||
fn poll_sensing_latest(ssh_host: &str) -> Result<SensingLatest> {
|
||||
let output = Command::new("ssh")
|
||||
.args([
|
||||
"-o",
|
||||
"ConnectTimeout=8",
|
||||
ssh_host,
|
||||
"curl -s -m 5 http://127.0.0.1:3000/api/v1/sensing/latest",
|
||||
])
|
||||
.output()
|
||||
.context("spawning ssh to poll the live sensing endpoint")?;
|
||||
if !output.status.success() {
|
||||
bail!(
|
||||
"ssh poll failed (exit {:?}): {}",
|
||||
output.status.code(),
|
||||
String::from_utf8_lossy(&output.stderr)
|
||||
);
|
||||
}
|
||||
let body = String::from_utf8_lossy(&output.stdout);
|
||||
serde_json::from_str(&body).with_context(|| format!("parsing sensing/latest JSON: {body}"))
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct FalImageResponse {
|
||||
images: Vec<FalImage>,
|
||||
}
|
||||
|
||||
#[derive(Debug, Deserialize)]
|
||||
struct FalImage {
|
||||
url: String,
|
||||
}
|
||||
|
||||
async fn generate_frame(
|
||||
client: &reqwest::Client,
|
||||
fal_key: &str,
|
||||
prompt: &str,
|
||||
seed: i64,
|
||||
) -> Result<Vec<u8>> {
|
||||
let request_body = serde_json::json!({
|
||||
"prompt": prompt,
|
||||
"negative_prompt": NEGATIVE_PROMPT,
|
||||
"num_inference_steps": 6,
|
||||
"image_size": "landscape_16_9",
|
||||
"seed": seed,
|
||||
"format": "jpeg",
|
||||
});
|
||||
let response = client
|
||||
.post("https://fal.run/fal-ai/fast-sdxl")
|
||||
.header("Authorization", format!("Key {fal_key}"))
|
||||
.json(&request_body)
|
||||
.send()
|
||||
.await
|
||||
.context("calling fal.ai fast-sdxl")?;
|
||||
if !response.status().is_success() {
|
||||
let status = response.status();
|
||||
let body = response.text().await.unwrap_or_default();
|
||||
bail!("fal.ai returned {status}: {body}");
|
||||
}
|
||||
let parsed: FalImageResponse = response.json().await.context("parsing fal.ai response")?;
|
||||
let image_url = parsed
|
||||
.images
|
||||
.first()
|
||||
.context("fal.ai response had no images")?
|
||||
.url
|
||||
.clone();
|
||||
let image_bytes = client
|
||||
.get(&image_url)
|
||||
.send()
|
||||
.await
|
||||
.context("downloading generated frame")?
|
||||
.bytes()
|
||||
.await
|
||||
.context("reading generated frame bytes")?;
|
||||
Ok(image_bytes.to_vec())
|
||||
}
|
||||
|
||||
#[tokio::main]
|
||||
async fn main() -> Result<()> {
|
||||
tracing_subscriber::fmt::init();
|
||||
let cli = Cli::parse();
|
||||
|
||||
let fal_key = std::env::var("FAL_KEY")
|
||||
.context("FAL_KEY env var must be set (fetch from GCP Secret Manager at the point of use, never commit it)")?;
|
||||
|
||||
std::fs::create_dir_all(&cli.out_dir).context("creating output directory")?;
|
||||
let client = reqwest::Client::new();
|
||||
|
||||
let mut previous_activity: Option<Activity> = None;
|
||||
for frame_index in 0..cli.frames {
|
||||
let latest = poll_sensing_latest(&cli.sensing_ssh_host)?;
|
||||
let scene = map_scene_state(&latest);
|
||||
if Some(scene.activity) != previous_activity {
|
||||
tracing::info!(frame = frame_index, activity = ?scene.activity, "scene state changed");
|
||||
}
|
||||
previous_activity = Some(scene.activity);
|
||||
|
||||
let prompt = build_prompt(cli.style, scene);
|
||||
tracing::info!(frame = frame_index, %prompt, "generating frame");
|
||||
|
||||
let frame_bytes = generate_frame(&client, &fal_key, &prompt, cli.seed).await?;
|
||||
let frame_path = cli.out_dir.join(format!("frame_{frame_index:04}.jpg"));
|
||||
std::fs::write(&frame_path, &frame_bytes)
|
||||
.with_context(|| format!("writing {}", frame_path.display()))?;
|
||||
tracing::info!(frame = frame_index, path = %frame_path.display(), bytes = frame_bytes.len(), "frame saved");
|
||||
|
||||
if frame_index + 1 < cli.frames {
|
||||
tokio::time::sleep(Duration::from_secs(cli.interval_secs)).await;
|
||||
}
|
||||
}
|
||||
|
||||
let video_path = cli.out_dir.join("room-viz.mp4");
|
||||
let status = Command::new("ffmpeg")
|
||||
.args([
|
||||
"-y",
|
||||
"-framerate",
|
||||
"1",
|
||||
"-i",
|
||||
])
|
||||
.arg(cli.out_dir.join("frame_%04d.jpg"))
|
||||
.args([
|
||||
"-c:v",
|
||||
"libx264",
|
||||
"-pix_fmt",
|
||||
"yuv420p",
|
||||
"-vf",
|
||||
"scale=1024:-2",
|
||||
])
|
||||
.arg(&video_path)
|
||||
.status()
|
||||
.context("spawning ffmpeg")?;
|
||||
if !status.success() {
|
||||
bail!("ffmpeg exited with {status}");
|
||||
}
|
||||
println!("wrote {}", video_path.display());
|
||||
Ok(())
|
||||
}
|
||||
Reference in new issue
Block a user