Fixes#1019. A dashboard card grouped by a custom spool field counted more
spools than it listed: the group aggregate summed them, the card's own query
returned a subset.
Grouping and filtering disagreed about what "no value" means. group_by keys on
the decoded value, so a spool holding the JSON empty string a cleared text box
writes became its own group, separate from the null one; the client renders
both as "unassigned" and adds the counts. The card then fetches its spools with
`extra.<key>=`, whose empty branch matched only NULL and JSON null - never the
empty string. Built-in string columns have matched both NULL and '' since
forever (add_where_clause_str_opt), which is why grouping by location was fine.
- The empty extra-field filter matches a stored empty string too.
- Value-keyed group axes (extra fields, location, material) fold a blank into
NULL, so "unassigned" is one group rather than two the client can only draw
as one.
- Emptying a text extra field clears it instead of storing "", so no new spool
lands in the state above.
- The dashboard clears an extra field with a real JSON null. It sent the string
'null', which the API rejects as not-a-string, so dragging a spool onto the
unassigned card 400'd on every extra-field board.
A spool can hold its own full weight, empty-spool (tare) weight and price,
falling back to its filament's where it has none. The API has had all three
since #339, and the v1 client exposed them, but client_v2 never mapped
spool_weight at all and had no field for it or for the full weight.
The result was silent wrong arithmetic (#1013): the API gives every spool it
creates a copy of the filament's tare weight, so correcting the *filament*
later looked like it had worked — the spool panel showed the new figure — while
"adjust by measured weight" went on subtracting the old one. A reporter's empty
261 g spool came back with 129 g still on it, exactly the difference.
Both ends of a disagreement are now marked. The spool's field says what it
shadows, the filament's row says what applies instead and is struck through,
and the same treatment covers the manufacturer's empty-spool weight, which
seeds a filament and is then left behind. Only where the values actually
differ: merely having one of its own is already visible in the field, and
marking every spool would teach everyone to stop reading the marks.
Also fixes a 500 from PUT /spool/{id}/measure when nothing in the chain has a
tare weight — reachable in one click now that the field can be cleared.
The validator fix only guards the write path. Instances that already hit #780
keep 500ing on GET /filament because the stored value exceeds the response
model's max_length, and the UI can't repair what it can't list.
Filament.from_db now strips the # and returns None when the stored value still
isn't a valid 6/8 character code, so a single bad row no longer takes down the
whole list. A migration chained from 415a8f855e14 rewrites existing rows one by
one through op.get_bind() rather than with SQL string functions, so it behaves
the same on sqlite, postgres, mysql and cockroachdb. downgrade() is a no-op.
Searching a filament name only ever returned the filament: a spool matches
on its own text (comment, location, lot number, extra fields) or its exact
id, so a name that lives on the filament could not reach any of its spools.
The hit named the filament you own and then dead-ended.
Each filament result now carries its first few spools as pills, with an
overflow link to the filament for the rest. GET /search gains an opt-in
spools_per_filament parameter; at 0, the default, the response is byte for
byte what it was, so nothing else that reads the endpoint changes.
The spools are fetched in one query over the matched filaments and grouped
in Python rather than with a per-group window function, which keeps the
behaviour identical on all four supported databases, and yields the exact
total count for free. They obey allow_archived like the rest of the
response, and are ordered by id so "first" means the same thing everywhere.
Fixes#993
Sorting the library by "last used" led with everything that had never been
used, on PostgreSQL and CockroachDB. Those two default to NULLS FIRST on
DESC, while SQLite and MySQL treat NULL as the lowest value and put it last;
`last_used` is NULL until a spool is first used, so the default library
ordering came out inverted for half the supported databases (#984, #985).
Grouped views showed it worst: a group's date is the max over its spools, so
a filament holding nothing but unused spools aggregates to NULL and floated
to the top, above filaments used minutes ago.
order_by_clauses() now orders on an explicit "is it null" flag ahead of the
field itself, so an absent value sorts last in both directions everywhere.
It is written as `expr IS NULL` rather than SQLAlchemy's nullslast(), which
renders a literal NULLS LAST that MySQL and MariaDB reject. Applied to every
sort site -- spools, groups, filaments, vendors, extra fields -- so ordering
no longer depends on which database is underneath.
Verified against all four: 438 integration tests pass on SQLite, PostgreSQL,
MariaDB and CockroachDB, and both new regression tests fail on PostgreSQL
without the fix.
Claude-Session: https://claude.ai/code/session_01EaLK21WpQYDE1DkKwZHK3P
The color_hex and multi_color_hexes validators normalized a local copy of
the value (uppercasing it and removing a leading "#") in order to validate
it, but then returned the caller's original string. A "#"-prefixed 8-digit
color such as "#FF000000" was therefore stored with 9 characters, which no
longer fits the String(8) color_hex column, so the write failed and every
subsequent read of that filament returned a 500.
Both validators now return the value with the "#" removed, matching the
un-prefixed form the rest of the codebase and the frontend already assume.
Adds tests/test_color_hex_validation.py covering both validators for the
prefixed and un-prefixed, 6- and 8-digit forms.
The unit tests are supposed to run with no database, no server and no
built client, but tests/test_configjs.py imported spoolman.main, which
builds the SPA handler at module level and so needs client_v2/build to
exist. That passed locally, where the bundle is lying around, and failed
CI's style job, which never builds the client.
Moves the rendering into client.render_config_js, which the endpoint now
calls, so the test exercises the real code path without the app.
Task 10: config.js JSON-encodes the base path, so a backslash cannot
escape the string literal the way the old " check allowed; search.py
escapes LIKE wildcards at both its call sites, using a helper moved into
database/utils.py so it and extra_field_query share one implementation.
Task 9 was a pair of decisions rather than fixes. /info keeps its paths:
both clients render them and removing response fields breaks API v1, while
the disclosure buys nothing against an API that is unauthenticated anyway.
/metrics keeps its shape but now says what it publishes when switched on.
Also pins the unit tests to a throwaway data directory -- importing
spoolman.main creates one as a side effect, and it was the real one.
Defaulting it on was the wrong trade. The only hostname it can refuse is a
registrable public domain, so it broke exactly the reverse-proxy-on-a-real-
domain deployments that configure nothing today -- a 400 on every request
until they found a new variable -- while the deployments rebinding can
actually reach are addressed by IPs, single-label names and .local, which
it never refuses anyway. The group that breaks and the group that gains
barely overlap.
SPOOLMAN_ALLOWED_HOSTS is now the on switch as well as the allowlist, so
there is no state where an operator has to think about this without having
asked for it. SPOOLMAN_CORS_ORIGIN deliberately does not enable it, or
setting CORS for a Fluidd instance would start refusing your own hostname.
Closes the DNS rebinding hole, which the origin guard structurally cannot:
under rebinding the attacker's page keeps its own name, so Origin and Host
agree with each other and the origin guard trusts them both, while the
address behind that name is the victim's Spoolman instance.
The default allows everything a stranger cannot point at someone else's
machine -- any IP literal, any single-label name, and the non-registrable
suffixes -- because what the attack needs is a registrable public domain.
That leaves a reverse proxy on a real domain, which cannot be told apart
from the attack and so must declare itself via SPOOLMAN_ALLOWED_HOSTS.
Brings the branch up to 0.25.0 and picks up cosign image signing, the
armv7/arm64 smoke tests, the armv7 greenlet/libstdc++ fix, the
PUID/PGID and Kubernetes SPOOLMAN_PORT entrypoint guards, and the
externaldb unwritable-path fix.
The only conflict was publish-images' needs list, where both sides had
added a job: test-frontend-v2 here, test-multiarch-smoke on master.
Resolved as the union, so nothing publishes until every test job has
passed. The rest of the graph is unchanged, and the new smoke tests
inherit client_v2 coverage for free -- build-arm64 and build-armv7
already download the client_v2 artifact, so booting those images to
/health now also proves the default client is present in them.
Claude-Session: https://claude.ai/code/session_01UB5KocsTBm9GL5x3Uuup5F
The bump script has only ever touched client/package.json, so
client_v2 has sat at 0.0.1 since it was created. Nothing reads it --
the footer takes the version from GET /info -- but both clients ship in
every release, and a package that claims 0.0.1 while the release says
0.25.0 is confusing to anyone who looks.
The indent has to be per-client: each tree is formatted by its own
prettier config and client_v2 uses tabs, so writing 2 spaces back would
make `npm run lint` fail on the release commit itself.
Also brings client_v2 up to the current version so the first bump after
this is a normal increment rather than a jump from 0.0.1.
Claude-Session: https://claude.ai/code/session_01UB5KocsTBm9GL5x3Uuup5F
The clients are build artifacts and are not committed, so a source
checkout has none until `npm run build` has been run in the client
directory -- and a `git pull` that brings new client sources does not
bring a bundle. Spoolman then refuses to start with StaticFiles' own
message, "Directory 'client_v2/build' does not exist", which says
nothing about what is wrong or how to fix it.
This has always been true of client/dist, but it bites harder now: an
existing source install that upgrades into client_v2 has never built
this directory even once, so it hits the wall on the upgrade rather than
on a fresh install. Docker images and spoolman.zip both ship the bundles
pre-built, so neither is affected.
Check before handing the directory to StaticFiles and name the command
to run, derived from the client directory so it is right for whichever
client is mounted.
Claude-Session: https://claude.ai/code/session_01UB5KocsTBm9GL5x3Uuup5F
The legacy client got its manifest and icons from vite-plugin-pwa.
client_v2 has no such plugin, and nothing replaced them: static/ held
only favicon.svg, spoolman.svg and the Ko-fi logo. So the upgrade
silently un-installs the app for everyone who had added Spoolman to a
home screen -- no manifest means no standalone display mode, no
home-screen icon, and no install prompt ever offered again.
The manifest and icons are plain files in static/ rather than plugin
output. The icons are the v1 set, unchanged, so an already-installed
app keeps the icon it had; name, colors and display mode are v1's too,
and start_url resolves to the same URL at a root deploy, which keeps the
app identity stable across the upgrade instead of orphaning the old
install.
Every URL inside the manifest is relative to the manifest itself, so the
one file is correct under any SPOOLMAN_BASE_PATH with no rebuild. Only
the document's link to it needs help: the prerendered pages emit
"./manifest.webmanifest" and resolve fine, but the SPA fallback (200.html,
which serves deep routes like the /spool/show/<id> a printed QR points at)
carries absolute paths, so the existing base-path fixup gains two
patterns for the manifest and the touch icon.
Also register the .webmanifest media type. StaticFiles takes Content-Type
from the stdlib mimetypes registry, whose built-in table only learned
this extension in recent Pythons and otherwise falls back to the host's
/etc/mime.types -- absent in slim images. We support 3.10+, so leaving it
to the environment means some installs serve the manifest as text/plain.
Claude-Session: https://claude.ai/code/session_01UB5KocsTBm9GL5x3Uuup5F
Picking the empty option of a single-choice extra field did nothing --
in either client, on spools, filaments and vendors alike. Emptying an
integer, range, datetime or multi-choice field was just as stuck. There
was no way to get a value back to unset once it had one.
Nothing in the API could express it. A key left out of the map means
"leave it alone", and every value that could stand in for empty is
rejected by the field's type check: "" is not one of a choice field's
choices. So `extra` values now accept null, meaning the entity has no
value for that field -- no row is stored, which clears whatever was
there. Null was a 422 before it reached any logic, so nothing that used
to work behaves differently.
A filament's and a vendor's extra were also REPLACED wholesale by a
patch rather than merged per key, unlike a spool's since 28f6f8e
(v0.22.0). Anyone patching one field of a filament -- as client_v2 did
-- silently destroyed the rest of its custom fields; anyone relying on
the replacement to delete a key can now say so with null. All three
entities merge per key now, and the field descriptions say so instead
of saying nothing.
client_v2's saver was where the frontend half died: an emptied input
produced no patch at all. It sends the cleared key as null now, and
keeps sending only the keys that changed, which is what stops two people
editing different fields of one entity from overwriting each other. It
also captures the entity id when the edit is made -- the inspectors are
reused across selections rather than remounted, so a debounced patch
could otherwise land on whatever you clicked next.
The legacy client cleared a filament's or vendor's field by omitting it,
which the merge would now ignore, so its form sends null for a field
left empty. That also buys it clearing on spools, which merged (and so
ignored the omission) all along.
Claude-Session: https://claude.ai/code/session_01UB5KocsTBm9GL5x3Uuup5F
Renaming a dashboard card only worked for Location, which has a
server-side rename that moves every spool in one statement. A custom
field had no equivalent, so the card name was read-only unless the group
was empty -- renaming it from the client would have meant patching each
spool one at a time, and only the ones the board had actually paged in.
Generalise the location rename to the spool's other string fields:
PATCH /spool/field/{field} {"value": "Shelf A", "new_value": "Shelf B"}
-> {"spools_updated": 6}
where field is `location` or `extra.<key>`. Archived spools are included
-- the value is the value, and skipping them would leave half the spools
behind. Renaming onto a value already in use merges the two, though the
dashboard still refuses that so it can't happen by accident.
The old value travels in the body rather than the path, unlike
/location/{name}. Field keys are ^[a-z0-9_]+$ and so are path-safe, but
values are not: a location like "Rack A/B" needs an encoded slash, which
proxies routinely mangle. No reason to carry that flaw into the general
form.
Only fields the spool owns can be renamed. material/vendor/filament are
refused because they belong to another entity, and rewriting them "for
these spools" would change filaments that other spools share -- the same
rule that decides what the dashboard can group by, so grouping and
renaming now share one validator.
Like rename_location, no spool event is broadcast per row: the change is
one statement over what may be hundreds of spools, and that fan-out is
what the websocket layer avoids elsewhere.
Verified on all four databases, and in the browser on a 40-spool group
of which the board had loaded 30 -- all 40 moved, and the card renamed in
place without losing its position.
The locations board could only ever answer one question: where is this
spool. The same card-and-drag interaction works for any field a spool
carries, so make the field a view mode and keep the rest.
Location is still the default view and behaves exactly as before. The
other views come from the user's own spool fields: any single-value text
or choice field now appears in a Group-by menu. A choice field renders a
card per choice, including the empty ones, which turns a "Status" field
into a kanban board.
Which field a card belongs to is decided by one small interface
(src/lib/dashboard/fields.ts), so the page never special-cases a field.
A field only qualifies if the SPOOL OWNS IT, because a drag moves one
spool and must mean exactly one change. That rules out material and
vendor, where writing one spool's card would silently rewrite every other
spool of the same filament, and lot number, which is stamped on by the
manufacturer and would be recording something untrue. Those stay library
filters. It also rules out numbers, dates, booleans and multi-choice,
whose values don't name a card you can drag into.
Backing changes:
- /spool/group grows group_by=extra.<key> for a spool's custom fields.
The field table is aliased so it stays a distinct FROM element -- the
extra-field filters reference the un-aliased table inside correlated
subqueries, and without the alias those would correlate to this join
instead of standing alone. Grouping is rejected for field types whose
value isn't a single repeatable string. group_by is no longer a
Literal, so an unsupported value is a 400 rather than a 422.
- The saved layout is per field: dashboard_groups and
dashboard_spoolorders replace the location-only pair. The old
`locations` / `locations_spoolorders` are read once as a seed for the
location view and never written back, so an upgrade keeps the shelves
it had and the old client keeps working.
- The view mode lives in the URL (?by=) so a board can be bookmarked and
shared, and is mirrored to localStorage so a bare /dashboard resumes
the last one looked at. The URL wins when it says something; a stale
?by= naming a deleted field falls back to location.
Verified against SQLite, PostgreSQL, MariaDB and CockroachDB -- Postgres
caught a real difference here, it sorts NULLs last where SQLite sorts
them first, so the new group assertions are keyed rather than ordered.
_load_filaments memoizes the parsed catalog on the cache file's st_mtime alone.
Two writes landing in the same mtime tick therefore look identical to the cache and
the second one is never picked up -- the process keeps serving the stale parse until
some later sync happens to change the stamp.
The window is small on Linux, where st_mtime is nanosecond-resolution, but it is not
theoretical: Spoolman's data directory is routinely a bind mount, and filesystems
with 1-second mtime granularity are common there. A catalog refresh triggered close
after another write would be silently dropped.
Including st_size makes the key sensitive to any rewrite that changes length, which
is the realistic shape of a catalog update.
load_and_rewrite_fallback_base_path does blind string replacement on 200.html to
insert the operator's base path into SvelteKit's absolute asset references and its
hardcoded `base: ""`. str.replace on a pattern that isn't there is a silent no-op,
so a SvelteKit upgrade that changes any of those three emission shapes would ship a
fallback document whose assets 404 and whose client-side router builds URLs without
the base path.
That failure is invisible in development and in CI, both of which run at the root
path where this code doesn't execute at all. It would surface only on an operator's
reverse-proxied instance, as a blank page on exactly the routes the rewrite exists
to serve -- direct loads of /spool/show/<id>, the target of printed QR labels.
Check each pattern is present before rewriting and raise naming the ones that are
not. A build/server mismatch is a packaging error, so refusing to start with an
actionable message beats serving a broken page.
Comments only. 2924850 removed the filament->spool fan-out, but neither comment
named remaining_length, which is the field 6ade405 added the fan-out for. That
omission is how someone re-adds it.
Spell out that remaining_length is derived from density and diameter and is not
cached client-side -- SpoolInspector computes it from the $derived filament, so
the filament event alone refreshes it -- while the stored initial/remaining
weights are the part that genuinely needs reweighSpoolsOfFilament.
Two gaps in the new /search endpoint, neither yet released:
Archived spools were always included, unlike GET /spool. /search now takes
allow_archived (default false) and applies the same false-or-null clause
spool.find uses, so the two endpoints agree on what "archived" hides. The
exact-id shortcut honours it too -- otherwise searching an archived spool's id
surfaced it regardless.
Candidate selection ordered by id ASC before LIMIT 200, so on a database with
more than 200 matches for a term only the 200 lowest ids were ever ranked and a
newly added spool could not be found at all. Ordering descending makes the newest
rows the ones that get ranked, which is the useful bias for an inventory that
only grows. Ranking still re-sorts the survivors by match quality, so this
changes which rows are considered, not how results are ordered.
Integration tests cover both defaults and both opt-ins.
filament.update re-broadcast every spool of the filament (archived included)
after commit, so a filament shared by hundreds of spools turned a single PATCH
into hundreds of ORM loads and websocket frames inside one request. This is the
unbounded fan-out flagged in review; the same reasoning is why no equivalent was
ever added for vendor.update.
Subscribers now treat a filament event as invalidating the spools that reference
it. In client_v2 the cache is already normalized -- spools hold a filamentId and
resolve their filament through filamentById -- so nearly everything a spool
displays was live regardless. The exception is a spool with no initial_weight of
its own, whose initial/remaining weights the server derived from filament.weight.
The cache now recomputes exactly those, mirroring Spool.from_db's formula
including its clamp at 0, which needs used_weight and the initial_weight override
carried on the cached spool.
Reverts the behavior added in 6ade405 (unreleased). Its integration test is
repurposed to pin the decision from both sides: no spool event reaches a /spool
subscriber, and the filament event subscribers rely on is still emitted.
Spool.from_db reports remaining_weight as max(initial - used, 0), but
find_groups' remaining_expr omitted the clamp. A spool used past empty
therefore contributed a negative number to total_remaining_weight, so a
group's total no longer matched the sum of the spools it contains — two
500 g spools, one with 200 g used and one with 800 g, reported 0 g instead
of 300 g.
Mirror the clamp with CASE rather than func.greatest, which is not portable
across all four supported databases. Covered by a new integration test.
Each list endpoint parsed `sort` inline with a bare split(":"), so `?sort=name`
raised "not enough values to unpack" and `?sort=name:sideways` a KeyError from
SortOrder[...]. Both surfaced as a 500 (todo.md task 8).
parse_nested_field also gated on hasattr, so any attribute that merely existed
passed through to order_by() -- `?sort=metadata:asc` got as far as calling
.asc() on the MetaData object. It now checks the SQLAlchemy mapper's columns,
so only real columns are accepted; relationship traversal through filament. and
vendor. still works.
Sort parsing moves into a shared parse_sort() used by all four call sites, which
tolerates surrounding whitespace and rejects a missing colon, an empty field
name or an unknown direction with a message saying what was expected.
database/vendor.py turned out not to use parse_nested_field at all -- it did a
bare getattr(models.Vendor, fieldstr) -- so /vendor kept returning 500 even
after the helper was fixed. It now uses the shared helper like filament.py.
All three endpoints verified end-to-end: 400 for each malformed form, 200 for
valid single and multi-field sorts, and no unhandled exceptions in the log.
Two lower-severity findings from the audit (todo.md tasks 6 and 7).
A spreadsheet executes a cell that begins with =, +, -, @, tab or CR, so a
vendor named `=cmd|' /C calc'!A0` runs when someone opens the CSV export
(CWE-1236). Such cells are now prefixed with a single quote, which spreadsheets
read as "this is literal text" and do not display. Only strings are escaped --
numbers reach the writer as numeric types, so a negative number cannot be
mistaken for a formula and must not grow a stray quote. Header names need no
escaping: they are fixed attribute names or extra-field keys, which are already
constrained to ^[a-z0-9_]+$.
Exports also gained Content-Disposition, so they download under a sensible name
rather than rendering in the browser.
Extra-field values had no size limit and the columns are Text(), so a single
request could store megabytes: unbounded database growth, and the resulting
websocket event overflowed the 1 MiB default frame limit, which killed live
updates for every connected client. Values are now capped at the same 65535
characters as settings. The number of extra fields per entity is capped too --
the per-entity key count was already bounded, since unknown keys are rejected,
but the registry itself could be grown without limit, and it is cached in memory
and embedded in every entity response.
Finally, a setting larger than the column limit returned 500; the ValueError is
now caught and returned as a 400.
Verified end-to-end: a 2 MB extra value is refused on both create and update, an
oversized setting returns 400 and leaves the setting unset, normal values still
work, and a vendor named `=cmd|' /C calc'!A0` exports as `'=cmd|' /C calc'!A0`
while the JSON export keeps the raw name.
allow_origins=["*"] with allow_credentials=True makes Starlette echo the
caller's origin back and set Access-Control-Allow-Credentials, which tells the
browser it may send cookies to whichever site asked. Spoolman has no cookies of
its own, so this looked harmless, but plenty of instances sit behind a reverse
proxy that does the authentication with one -- and there it let any website the
user visited make authenticated requests as them. Now credentials are refused
whenever the resolved origin list contains "*" (todo.md task 5).
Adds a README Security section. The README has no configuration reference at
all -- that lives in the Wiki -- so rather than starting one, this documents the
things a user has to know before deciding where to run Spoolman: that there is
no authentication by design, that it belongs behind an authenticating reverse
proxy if it is reachable beyond the LAN, what SPOOLMAN_CORS_ORIGIN is for, why
"*" is a bad idea on a shared network, and that a reverse proxy needs to forward
the original Host.
Verified with SPOOLMAN_CORS_ORIGIN=* against a real server: the response now
carries `access-control-allow-origin: *` and no credentials header, where the
audit saw the caller's origin echoed with credentials allowed.
Rotating deletes the oldest restore point and shifts the rest down, so enough
calls in quick succession leave nothing but snapshots of whatever state the
database is in now. POST /backup takes no parameters, so a bodyless cross-origin
form post reached it; the audit destroyed a full six-deep history that way
(todo.md task 2).
The origin guard from the previous commit already refuses that request, but the
rotation itself should not be this easy to abuse, so two further changes:
Rotation is skipped when the database has not actually changed. Not by comparing
the live database file, which the audit suggested: under WAL the live file does
not yet contain recent commits, so comparing it would skip backups that were
genuinely needed. Instead the snapshot is written to a pending file first and
compared against the newest backup -- sqlite's backup API reads through the
connection, so it sees WAL content. Identical means the pending file is
discarded and the history is left alone. This is fail-safe: if the comparison
ever came out unequal for an unchanged database, the result is the old behaviour
of rotating, never a missed backup.
Rotation is also rate-limited to one per five minutes, returning the newest
existing backup instead. Never applied when there is no backup to fall back on.
BackupResponse gained a `created` field so a client can tell the difference.
Additive, so v1-compatible, but load-bearing: telling a user they have a fresh
restore point when they do not is exactly the wrong failure mode here.
Verified against a real server -- six rapid calls now leave exactly one backup
and never rotate.
Closes the two confirmed CSRF findings and the cross-site websocket hijack
from the backend security audit (todo.md tasks 1 and 3).
POST /setting/{key} took a bare `str` body, and FastAPI only JSON-parses
application/*json -- anything else arrived as raw bytes that lax-mode Pydantic
coerced. A text/plain body was therefore accepted, which is exactly what
<form enctype="text/plain"> sends, so any website the user visited could
rewrite base_url, label_designs, locations or extra_fields_* through their
browser. Now requires a JSON content type.
The audit suggested Body(media_type="application/json") for this, but that only
annotates the OpenAPI schema and is not enforced at runtime -- a text/plain post
to a pinned endpoint still returns 200. Hence an explicit dependency. Both
clients already send application/json, and a charset parameter is accepted, so
this is wire-compatible.
TrustedOriginMiddleware refuses state-changing HTTP methods and every websocket
handshake from an untrusted origin, using the helper added in the previous
commit. A middleware rather than per-endpoint dependencies so that none of the
nine websocket endpoints can be missed, and one added later cannot forget to
opt in. Reads are left alone -- the same-origin policy already covers them --
but every websocket is guarded, since websockets are exempt from CORS and
otherwise leak the whole inventory to any origin that opens one.
It is ordered inside the CORS middleware so a legitimate cross-origin client
can read the 403 body instead of an opaque CORS error.
Writing a malformed extra_fields_* array through the settings endpoint bypassed
validate_extra_field and left GET /field/{entity} permanently returning 500;
that array is now validated, and the registry cache is invalidated on write so
the /setting and /field paths no longer disagree until restart.
Verified against a real uvicorn server by replaying the audit's exploit: the
text/plain write is refused, a bodyless form post that would have reset a
setting is refused, a websocket from a foreign origin gets HTTP 403 on the
handshake, and clients that send no Origin at all -- Moonraker, OctoPrint,
Home Assistant, curl -- are unaffected on both HTTP and websockets.
Tasks 1-3 of the backend security audit all need to answer "is this browser
origin someone we trust". Build it once, driven entirely by the existing
SPOOLMAN_CORS_ORIGIN rather than a new env var.
- Unset means same-origin only, which is right for nearly every deployment.
Non-browser clients (Moonraker, OctoPrint) send no Origin and are unaffected.
- A list means those origins plus our own.
- "*" and debug mode mean "I do not want origin checks" and are honoured as
such, with a loud startup warning. An operator who writes "*" is opting out;
reinterpreting it as something narrower would leave them no way to say so.
Also accept X-Forwarded-Host, so a reverse proxy that rewrites Host (nginx
without proxy_set_header Host $host, Apache without ProxyPreserveHost On) does
not see the guard reject its own web UI. A malicious page cannot set that
header on any of the vectors the guard exists to stop: forms cannot set request
headers, nor can the browser WebSocket API, and a fetch that adds one needs a
CORS preflight it will not get.
get_cors_origin() now normalizes its entries -- it was a bare split(","), so
"https://a, https://b" yielded " https://b", which no Origin header can match.
CORS itself benefits from the same fix. The raw value is kept in the startup
log so a typo stays visible.
Adds tests/ for backend unit tests, runnable with `poe test` and wired into the
lefthook ci group. Nothing enforces this helper yet; that is tasks 1-3.
SvelteKit's adapter-static emits the SPA fallback (200.html) with absolute
asset paths (/_app/...) and a hardcoded base: "", since it can't know the
deploy base path at build time. Prerendered per-route documents use relative
paths and a runtime-computed base, so they work under any base path, but the
fallback is used for non-prerendered dynamic routes (/spool/show/<id>,
/filament/show/<id>) -- exactly the URLs printed into QR labels. Under a base
path those direct loads 404'd every asset and rendered a blank page.
Rewrite the fallback server-side using the base path the backend already
knows, mirroring the legacy-client asset rewrite: prefix /_app/ and /favicon
references and set the runtime base. Prerendered docs are still served
verbatim, and the legacy client is unaffected (its tweaked_html is always
produced when rewrite_asset_paths is on, so the file_response guard change is
equivalent there).
The fallback to an in-memory cache never worked when the data directory
itself is not writable. Evaluating cache_path calls get_cache_dir(), which
creates the data directory, so a PermissionError raised there propagates out
before cache_path is ever bound. The handler then dereferences it and dies
with "NameError: name 'cache_path' is not defined", which both hides the real
cause and takes the application down at import time, before the startup code
that would have reported the permission problem in plain language runs.
Take the path from the exception instead. It is set for both cases: the data
directory that could not be created, and the cache directory underneath it.
Verified against the published image by running it as a UID that owns neither
directory, which is what a restricted Kubernetes SecurityContext does:
docker run --rm --user 1000670000:0 ghcr.io/donkie/spoolman:0.24.0
Before, that failed with the NameError above. Now it logs "Ensure the path
/.local is writable" and carries on to the regular startup check.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
find_by_color loaded every filament as a full ORM object (with its vendor
join) into memory and computed delta-E in Python, then the endpoint re-queried
the matched ids anyway. The full-catalog ORM load dominated the cost: a color
search ran at a flat ~100ms regardless of how few results it returned, ~5-20x
slower than an equivalent regular filtered search, and each concurrent request
cloned the whole catalog into memory.
Scan only id/color_hex/multi_color_hexes and return the matched ids; callers
load full rows for just the matches. The matching algorithm is unchanged.
Color search now scales with the match count like any other filter: ~5-16x
faster here and the concurrency memory spike (+21MB at 8 concurrent) is gone.
Drops the "Slow operation!" note from the color_hex query param since it no
longer applies.
Verified across SQLite/Postgres/MariaDB/CockroachDB (372 integration tests
pass on each).
Claude-Session: https://claude.ai/code/session_014Cwk5Yyze9yPvkYkUtMAP4
Spool responses carry fields derived from the filament (remaining_length,
plus the initial_weight/price fall-backs). Editing a filament emitted only a
filament WS event, so subscribers to that filament's spools kept stale derived
values until each spool was next touched — visible in client_v2 as a length
inconsistent with the displayed density, but affecting every consumer.
filament.update now re-broadcasts the filament's spools (archived included)
as spool "updated" events after commit. Adds a WS integration test asserting
a density patch reaches /api/v1/spool with the new density and changed length.
Claude-Session: https://claude.ai/code/session_014Cwk5Yyze9yPvkYkUtMAP4
The spool listing already joins spool -> filament -> vendor, so joined
eager loading of all three `extra` collections multiplied their rows
together. With ~12 spool, ~9 filament and ~9 vendor extra fields, a
single GET /spool?limit=30 made the database materialize ~35k wide rows
(one per field combination) and then spent most of its time in
SQLAlchemy de-duplicating them back into 30 objects.
Switch the three `extra` relationships to lazy="selectin" so each
collection is fetched in its own WHERE id IN (...) query and nothing
multiplies. Measured on a 4911-spool SQLite database, GET /spool?limit=30
drops from 431 ms to 12 ms and limit=100 from 1358 ms to 23 ms, with
byte-identical responses across the spool/filament/vendor/group/search
endpoints and all 371 integration tests green on SQLite, PostgreSQL,
MariaDB and CockroachDB.
Claude-Session: https://claude.ai/code/session_014Cwk5Yyze9yPvkYkUtMAP4
The search treated the whole query as one substring match, so "bambu
petg-cf" had to appear verbatim in a single field and returned nothing,
even though "bambu" and "petg-cf" each matched on their own. Vendor-name
matching also ran as a separate query, so a term could never be satisfied
by the vendor while another was satisfied by the filament.
Split the query on whitespace and require every term to match some
searchable field, letting different terms match different fields:
- Filaments search vendor.name in the same pass as their own fields via an
outerjoin, replacing the separate vendor query. This is what makes
"bambu petg-cf" work.
- Extra-field matching moves from a follow-up query to a correlated EXISTS
inside each term's OR-clause, so extras participate in the AND too.
- Ranking generalizes to a per-field-kind base rank (native 0, vendor 2,
extra 4) with prefix hits beating substrings. A row ranks by its worst
term, so rows matching everything strongly sort first; the reported
match_field is the single best hit.
Single-term queries behave as before. Standard SQL only (ilike, correlated
EXISTS), so this holds on all four supported databases.
Claude-Session: https://claude.ai/code/session_014Cwk5Yyze9yPvkYkUtMAP4
The info endpoint called externaldb.get_external_db_name(), but the
api-layer externaldb module never re-exported that name from
spoolman.externaldb, so every /info request raised AttributeError ->
HTTP 500. Import the function directly from the core module and call it.
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
Add a shared ColorEditor component (single/multi toggle, per-color
pickers + hex fields, add/remove colors, illustrated coaxial vs
longitudinal cards that preview the actual colors) and wire it into
both the add-spool create form and the library filament inspector,
which also gains its first color picker.
Colors and multi_color_direction round-trip through the API via a new
colorFieldsToApi helper that keys single- vs multi-color by count and
nulls out the other fields, so switching between them stays consistent.
Make color type filterable in the library toolbar (Single / Coextruded
/ Longitudinal) backed by a new additive filament.multi_color_direction
query param on the spool and spool/group endpoints; the Single option
sends an empty value to match single-color (NULL) filaments.
Also tidy a few inspector field labels to reuse the canonical
filament.fields.* / vendor.fields.* message keys.
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
Build the new Svelte client (client_v2) in CI, bake it into the Docker
image alongside the legacy React client, and serve it by default. Set
SPOOLMAN_LEGACY_CLIENT=TRUE to fall back to the old client.
SinglePageApplication is generalized to serve either client: the React
client keeps its index.html + base-path rewriting, while the Svelte
client uses a 200.html SPA fallback, serves prerendered per-route pages,
and serves relative-path assets verbatim. The tweaked-HTML store is moved
off self.html, which StaticFiles uses as its HTML-mode flag.
The Playwright frontend suite targets the React DOM, so it's pinned to
the legacy client via SPOOLMAN_LEGACY_CLIENT for now.
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
The external filament catalog defaults to SpoolmanDB but operators can
point EXTERNAL_DB_URL at their own catalog. The UI hardcoded "SpoolmanDB"
in a few places, which is misleading in that case.
Add an EXTERNAL_DB_NAME env var (default "SpoolmanDB"), expose it via
GET /info, and have client_v2 read it into a serverInfo store. Badges and
the results header now show the configured name, and the localized
strings that reference the library take it as a {name} parameter.
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
Extend the cross-entity /search filament results to also include filaments
whose vendor name matches the query, so searching a manufacturer surfaces its
filaments (not just the vendor itself). Vendor-name matches rank below any
native-field match and report match_field "vendor.name". The extra-field
annotation block is factored into a helper to keep _search_filaments within the
complexity budget. The client search bar labels the new match_field as
"Manufacturer".
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
The external SpoolmanDB catalog (filaments.json, thousands of entries) was
served whole via GET /external/filament and filtered in the browser, so every
client downloaded the entire catalog just to search it.
Add a backend GET /external/filament/search?query=&limit= endpoint that filters
the catalog server-side (same word-AND, case-insensitive semantics as the old
client filter), memoizing the parsed catalog by file mtime. client_v2 now calls
it instead of downloading the whole file. The existing /external/filament
endpoint is unchanged, so external API consumers are unaffected. The small
materials.json is still fetched whole for new-filament presets.
Claude-Session: https://claude.ai/code/session_01VQQtwPBPiMCoZCpBbbqS64
Implements the search feature for client_v2: a new backend search endpoint
plus a results-panel UI, replacing the old query box that only filtered the
library list by filament name.
Backend (additive, API v1 backward compatibility preserved):
- New GET /api/v1/search returning categorized spool/filament/vendor results,
each annotated with the field that matched, plus an is_color_query flag.
- Case-insensitive text search over native text fields and text/choice extra
fields; exact spool-id match for numeric queries; color-similarity search
(CSS color name or hex) via find_by_color, ordered closest first.
- spoolman/colors.py resolves the CSS4 color names / hex codes to hex.
- Integration tests (run across SQLite/PostgreSQL/MySQL/CockroachDB in CI).
Frontend (client_v2):
- SearchBox overlay with Spools/Filaments/Manufacturers sections, per-result
match badges, keyboard navigation, a color-similarity threshold slider
(persisted to localStorage), and an English-only color-name hint.
- Clicking a result opens the matching inspector on the Library page.
- Removes the old q -> filament.name inline list filtering.
Claude-Session: https://claude.ai/code/session_01Ne7RsfHejKZX97uAq3aVMn
Adds a label design/print subsystem (canvas editor, QR/logo/paper
helpers, print layout panel) backed by a new label_designs setting,
plus a footer, datetime field component, and assorted fixes across
the library and settings views.
Introduces client_v2, a new SvelteKit-based client, alongside the
backend it depends on: a /spool/group endpoint that returns
per-group aggregates (count, in-use count, remaining weight,
last-used) grouped by filament, vendor, material or location, so the
client can paginate whole groups instead of fetching every spool.
Not yet complete/wired up for release.
* Add filtering and sorting for custom fields
* Fix tests and make code postgres compatible
* Fix bugs and expand test coverage for custom field filter/sort
* trigger CI
* fix: address extra field filter review feedback
* fix: clean up custom field table state handling
* docs: update extra field table view description
* fix: narrow boolean custom field filter syntax
* Expand test coverage for all custom field types/entities and fix range filter/sort
- Add comprehensive tests covering all 9 field types (text, integer, float,
boolean, single-choice, multi-choice, datetime, integer_range, float_range)
for filter and sort on spool, filament, and vendor entities
- Add invalid-filter 400 tests for float, integer_range, and float_range
- Fix integer_range/float_range filter to use LIKE pattern matching against
Python's deterministic json.dumps output instead of fragile .contains()
- Add integer_range/float_range sort support via a @compiles helper
(_JsonArrayFirstElement) that emits CAST(col AS JSON)->>0 on PostgreSQL
and JSON_EXTRACT(col, '$[0]') on SQLite/MariaDB
- Add logger and __all__ to extra_fields.py (aligns with PR #893)
- All tests verified passing on postgres, sqlite, and mariadb
* Add filter UI for all custom field types and fix range filters
- Add text input filter dropdown for text fields (substring search)
- Add range input filter dropdowns (min/max) for integer, float, integer_range, and float_range fields
- Add datetime picker filter dropdown for datetime fields
- Fix boolean "No" filter to use empty value so it correctly matches unset/false fields
- Fix integer_range/float_range filter to use numeric comparisons instead of LIKE-based exact matching (stored_min >= filter_min, stored_max <= filter_max)
- Add range filter support for integer/float fields (min:max format)
- Add _JsonArraySecondElement cross-database helper for extracting second JSON array element
* Range filter for integer/float fields and fix range_field filter semantics
Backend:
- Add _JsonArraySecondElement cross-database helper (mirrors _JsonArrayFirstElement)
- integer_range/float_range filter now uses numeric comparisons:
stored_min >= filter_min and stored_max <= filter_max (was LIKE exact match)
- integer/float filter now supports min:max range format in addition to exact match:
stored_value >= min and/or stored_value <= max
Frontend:
- integer and float fields now use a range filter dropdown (min/max inputs)
instead of a single exact-value input
Tests:
- Fix integer_range/float_range spool and vendor tests broken by new >= semantics
(filter values updated to discriminate between test entries)
- Add range filter tests (min only, max only, both) for integer and float fields
* Replace 12 copy-pasted numeric field tests with 4 parametrized tests
Introduce a _create_entity helper and @pytest.mark.parametrize("entity_type",
["spool", "filament", "vendor"]) to run each numeric field type test against
all three entity types from a single test function.
- Removes 12 individual tests (integer/float/integer_range/float_range × 3 entities)
- Adds 4 parametrized tests covering the same ground plus the previously missing
min-only / max-only range filter cases for filament and vendor
- Uses try/finally for cleanup so entities are always deleted even on assertion failure
* Add datetime range filter and restore integer/float range filter support
- Backend: add datetime range filter using '|' separator (ISO dates contain ':')
- Backend: restore integer/float range filter support (lost during attribution rewrite)
- Frontend: replace single DateTimePicker with From/To range pickers for datetime fields
* Use local timezone in filter picker; add range filter tests
The DateTimeRangeFilterDropdown was initializing pickers with dayjs.utc(),
causing them to display and accept UTC times directly. The entry form shows
local time and converts to UTC, so the filter was inconsistent: entering
the same displayed time would produce a filter value 2h offset from what
was stored (in UTC+2), making exact boundaries fail unexpectedly.
Fix: initialize picker values with dayjs() (local mode) so the filter picker
behaves the same as the entry form — shows local times and converts to UTC.
Tests: replace 3 separate entity-specific datetime tests with one parametrized
test covering spool/filament/vendor, and add range filter cases (start|, |end,
start|end).
* Replace all entity-specific tests with parametrized tests
Replace 16 separate spool/filament/vendor tests for text, boolean,
single-choice, multi-choice, and empty-filter with 5 parametrized tests
that run against all three entity types. Each test is now defined once
and executed 3 times, eliminating copy-paste and ensuring consistent
coverage across all entity types.
Parametrize invalid-filter-value 400 tests across all entity types
The non-numeric-value error uses "Invalid integer/float range filter value"
while the missing-colon error uses "Invalid range filter value". Both contain
"range filter value", so use that as the assertion substring.
* Align custom filter dropdowns with Ant Design choice filter style
All three custom filter dropdowns (text, number range, datetime range) now
match the built-in choice filter footer: a border-top separator, Reset as a
link button on the left (disabled when no filter is active), and OK as a
primary button on the right.
* fix(fields): correct custom-field filter edge cases across databases
- Float exact-match filters now compare numerically (cast to Float) instead of
by JSON string, so float fields stored as integer JSON (e.g. "2") or
non-canonical decimals (e.g. "2.50") match an equivalent filter.
- integer_range/float_range filter and sort no longer return HTTP 500 on
CockroachDB: the PostgreSQL '->>' JSON extraction is now also registered for
the cockroachdb dialect, which previously fell back to the unsupported
JSON_EXTRACT function.
- Text exact-match and single-choice equality use ensure_ascii=False so
non-ASCII values match how the frontend's JSON.stringify stores them.
Co-authored-by: Dieter Blomme <dieterblomme@gmail.com>
* test(fields): expand custom-field filter/sort coverage
Add tests covering filter composition (multiple extra fields, extra + built-in,
filter one field / sort another), pagination total-count, unknown-field
handling, text case-insensitivity/substring/non-ASCII, numeric multi-value OR,
boolean empty semantics and token rejection, multi-choice substring collisions,
empty filters on numeric fields, null-bounded ranges, and additional
invalid-value 400 paths.
Verified against sqlite, postgres, mariadb and cockroachdb.
Co-authored-by: Dieter Blomme <dieterblomme@gmail.com>
* style(client): format column.tsx with prettier
The custom-field filter dropdowns' Reset buttons were written as one-line
onClick handlers that prettier wraps across multiple lines. Apply prettier
so the CI style check passes.
Co-authored-by: Dieter Blomme <dieterblomme@gmail.com>
* fix(fields): decode JSON scalars in-DB for text/choice/datetime custom-field filters
Match text, single-choice and datetime custom-field filters against the
database-decoded JSON scalar (new _JsonScalarText cross-dialect helper:
#>> '{}' on postgres/cockroachdb, JSON_UNQUOTE(JSON_EXTRACT) on mysql,
json_extract on sqlite) instead of reconstructing the exact JSON
serialization the client wrote. This decouples matching from
json.dumps/JSON.stringify encoding quirks (non-ASCII escaping, surrounding
quotes). Also escape LIKE wildcards so '%' and '_' in text and multi-choice
queries match literally rather than acting as wildcards.
Verified against sqlite, postgres, mariadb and cockroachdb.
Co-authored-by: Dieter Blomme <dieterblomme@gmail.com>
* test(fields): cover LIKE-wildcard escaping in custom-field filters
Add parametrized tests (spool/filament/vendor) asserting that '%' and '_'
are matched literally in text and multi-choice filters, that a literal '/'
(the internal LIKE escape char) matches literally, and that single-choice
equality handles a '%' value exactly.
Co-authored-by: Dieter Blomme <dieterblomme@gmail.com>
---------
Co-authored-by: Donkie <daniel.cf.hultgren@gmail.com>
Co-authored-by: akira69 <akira69@gmail.com>
Donkie