Move copy, sendfile and directory helpers behind one platform layer, let the
FTP server run beside the HTTP stack with its own instance bookkeeping, and
keep the transfer command on the shared pipeline.
One manager owns remote downloads: libcurl for HTTP/HTTPS/FTP (with ranged
parallel fetches), libnfs for nfs:// URLs and an optional libtorrent engine
for magnet links. Jobs survive payload re-injections, support pause, cancel
and retry, and report per-job progress through the HTTP API.
The HTTP stack had grown into a single file that owned sockets, parsing,
uploads and every API domain. Split it into focused modules (listener,
response stream, upload, json helpers) with the domain handlers next to the
APIs they serve, and cover framing, streaming and uploads with unit tests.
Survive console Rest Mode by recreating FTP/HTTP listeners with backoff and exposing daemon instance_id; implement real SITE CHMOD (#9) and fix atomic-copy temp paths that broke dashboard moves (#7).
curlftpfs sends 'LIST -a' by default. zftpd treated '-a' as a
path argument, causing 550 errors and empty directory listings
on FUSE-based FTP mounts (e.g. fnOS remote mount).
Add ftp_path_has_list_flag() / ftp_path_skip_list_flag() to
ftp_path.c with stat()-based flag-vs-path disambiguation.
Only well-known flags (-a/-l/-la/-al) are recognised; unknown
flag patterns pass through as paths to preserve access to files
whose names start with '-'.
Introduce pal_resilient_accept to handle accept() failures by recreating, rebinding and relistening the server socket (with retries and logs), and send a notification when the server resumes. Add public header and implementation (include/pal_resilient_server.h, src/pal_resilient_server.c), wire it into the server accept loop (src/ftp_server.c) replacing the direct PAL_ACCEPT call, and add the new source to the build (Makefile). Also enable FTP_PORT_ALLOW_FOREIGN_IP in test builds and update tests/test_security.c to use 127.0.0.1 instead of 10.0.0.2.
Prevent installing the net filter when another payload has already hooked the syscall table. Adds is_external_hook_installed() which reads sysent[SYS_CONNECT].sy_call, determines whether the handler points into DMAP (indicating an external hook), logs a warning, and aborts installation with a new error code PS5_NET_FILTER_ERR_EXTERNAL_HOOK. Updates ps5_net_filter_strerror mapping and header (also switches header license notice to MIT).
- ftp_config: bump version to 1.5.0; change PS4/PS5 default port from 2121 to 2120; add FTP_TCP_DATA_SNDBUF (4 MB on PS5/PS4 to bypass OrbisOS auto-tuning clamp, 0 elsewhere); reduce FTP_SENDFILE_EAGAIN_SLEEP_US from 1ms to 0.5ms for LAN RTT; increase FTP_RETR_SENDFILE_CHUNK from 512 KB to 2 MB to reduce syscall boundary overhead on NVMe
- http_config: add HTTP_DOWNLOAD_PREAD_CHUNK (2 MB) for pread() path
- Makefile: add ps5-hook-blob target to compile src/ps5_net_filter_hook.c with kernel-safe flags (-fno-stack-protector, -mno-red-zone, -mcmodel=large), extract .text.hook_connect/.text.hook_sendto sections via objcopy, and generate ps5_net_filter_hook_blob.h byte-array header; conditionally include src/ps5_net_filter.c in PS5 build
- ftp_config: bump RELEASE_VERSION from 1.3.1 to 1.4.0
- http
- ftp_server: implement bounded-wait shutdown sequence in ftp_server_stop() that closes listen_fd to unblock accept(), calls shutdown(SHUT_RDWR) on active session ctrl_fd to interrupt recv(), and waits up to 5s for active_sessions drain; expose ftp_server_release_session() for session threads to decrement active_sessions and reset slot state at exit
- ftp_session: call ftp_server_release_session() at end of f
- http_config: increase HTTP_UPLOAD_CHUNK_SIZE from 256 KB to 512 KB per active upload
- http_api: add NULL-check after http_response_create() in api_download to prevent SIGSEGV when response pool is exhausted; close fd and return NULL to avoid file-descriptor leak; add NULL-check after http_response_finalize() to handle buffer-full condition and
- http_api: add GET /api/dirsize?path=<dir> endpoint that recursively sums st_blocks*512 for all regular files under a directory with 32-level depth limit; expose http_dir_size_recursive() in header; trigger lazy size fetch for visible directories after listing render via _dirSizeToken cancellation mechanism
- http_api: force PS5 disk stats to always report /user partition instead of largest mount to match
- http_response: add HTTP_STATUS_409_CONFLICT and sendfile_safe flag with detailed documentation on PS4/PS5 kernel panic risk from sendfile(2) on exfatfs/msdosfs/nullfs/pfsmnt/pfs vnodes
- http_api: implement fstatfs-based filesystem detection in api_download to set sendfile_safe flag; add recursive directory deletion with opt-in ?recursive=1 parameter and
Expose error_code/error_errno in copy progress API and capture errno in copy operations for better diagnostics.
- http_api: add error_code/error_errno atomics to copy_progress_t, populate them in copy_thread_fn via pal_file_copy_recursive_ex, and include in api_copy_progress JSON response; initialize new fields in api_copy
- ftp_commands: switch ftp_copy_thread_func to pal_file_copy_recursive_ex with out_errno parameter to capture OS
Unify platform detection by adding PLATFORM_PS4/PLATFORM_PS5 alongside existing PS4/PS5 defines across ftp_config.h and http_config.h. Extend FTP_TCP_RCVBUF to 4 MB on PS4 (same as PS5) to absorb PFS write latency. Increase HTTP_SENDFILE_CHUNK_SIZE to 1 MB to reduce syscall overhead.
- ftp_commands: serialize O_CREAT opens on PS4/PS5 with g_pfs_create_mtx to prevent inode-allocation lock contention (prevents >20s open
Implement async server-side file copy via CPFR/CPTO/COPY commands with cross-device fallback and PS5 PFS optimizations.
- ftp_commands: add cmd_CPFR/cmd_CPTO/cmd_COPY handlers with async copy thread, pthread-based background copy, progress tracking, and RNTO fallback to async copy for cross-device moves; add copy_thread_cleanup and copy_worker with recursive copy/move logic; integrate copy state checks in session cleanup and command dispatch
Add multiple security and robustness fixes across HTTP and FTP subsystems:
- HTTP API: introduce http_api_set_root/http_api_get_root and http_validate_and_confine to enforce filesystem confinement; validate and use canonical 'safe' paths in list/download/stats/create endpoints.
- HTTP server: accept a root_path on creation, store/propagate it to API, enforce upload confinement to the configured root, and switch connection counters and in-use flag to atomic types for thread-safety.
- CSRF: make http_csrf_init return int, fail hard if entropy is unavailable or conversion fails (disables uploads), track initialization with a flag and check it in validation; main logs when CSRF init fails.
- FTP: add static assert and safer logic in ftp_buffer_pool to prevent undefined behavior for large FTP_MAX_SESSIONS bitmasks; reformat buffer declaration and fix acquire/release logic.
- FTP commands: check snprintf truncation in MKD reply and add PORT IP validation against control connection (configurable via FTP_PORT_ALLOW_FOREIGN_IP).
- Build/test: update Makefile test target to build test_http_query using LIB_OBJECTS.
These changes close several path traversal, symlink, CSRF predictability, race/atomic and buffer-mask issues and improve overall robustness.
Bump release to 1.3.1 and improve several I/O and error handling behaviors.
- ftp_commands: Provide detailed human-readable 550 messages for RNTO errors so clients (FileZilla/WinSCP) show actionable reasons instead of a generic "Rename failed.".
- pal_fileio: Add headers, logging and a max move recursion depth. Fix sendfile to treat EINTR as a partial send (avoid re-sending already transmitted bytes). Replace large static thread-local copy buffer with heap allocation to reduce memory footprint and fix temp-filename overflow by placing a short temp name in the destination directory.
- pal_fileio: Improve errno handling with logging and proper error mapping, free temporary buffers on all paths, and adjust close() behavior to account for differing EINTR semantics on FreeBSD/PS consoles vs Linux (prevents leaked fds on PS5).
- pal_fileio: Implement robust EXDEV fallback: recursive cross-device move for directories (pal_move_cross_device_r) with safe rollback semantics (only remove dst if created by this operation), recursive cleanup, path-length/depth checks, and safer pal_file_rename delegation.
- pal_notification: fix formatting/newline around include.
These changes fix several robustness bugs (temp name overflow, potential data loss on rollback, fd leaks on interrupted close, and retransmit on interrupted sendfile), add logging for diagnostics, and improve client-visible error messages.
Add a new include/http_resources.h header that defines a http_resource_t descriptor and http_resource_get() API for pre-compressed HTTP resources (with MIT license header). Regenerate/modify the embedded web UI asset in src/http_resources.c (index.html blob updated, size adjusted) to reflect updated front-end JS/SVG/content. Expand README.md with a clearer "Build & commands" section describing make targets, useful Makefile variables and example execution flags.
Add platform-safe sendfile handling and build variants; bump release version and add stress test.
- Makefile: introduce variant-aware build/artifact naming, ZHTTPD variant toggles, and a release-matrix target to build with/without ENABLE_ZHTTPD.
- include/ftp_config.h: bump RELEASE_VERSION to "1.3.0".
- src/http_api.c: guard inclusion/use of <sys/sysinfo.h> with a feature check to avoid build failures on platforms without it.
- src/pal_fileio.c: tighten sendfile() error handling (return partial counts on certain storage errors, avoid retries) and document/adjust F_NOCACHE vs POSIX_FADV behavior for PS4/PS5.
- src/pal_filesystem.c: add pal_fstatfs shim for PS platforms and detect unsafe filesystems (exfatfs, msdosfs, nullfs) to clear VFS_CAP_SENDFILE and avoid kernel panics when using sendfile().
- tests/stress_local.py: add a local FTP stress test that generates many small files and churns them during parallel RETR operations to help reproduce/validate robustness.
These changes prevent sendfile-induced kernel panics on PS4/PS5 USB filesystems, improve error handling, and add tooling and build convenience for variant testing.
Switch the default HTTP port from 8080 to 8888 across the codebase. Updated include/http_config.h macro and example in include/http_server.h, adjusted tests and examples in ffi Java/Python/Rust test files, and updated tests/main.c to create the server on and print the new port. Ensures tests and docs use the same default port.
Add FTP_ERR_DIR_EXISTS and treat EEXIST-as-success in MKD to avoid aborting concurrent directory uploads (prevents FileZilla from failing when another thread created the dir). Map EEXIST to FTP_ERR_DIR_EXISTS in pal_dir_create. Implement EXDEV fallback in pal_file_rename (atomic copy + unlink) and refactor pal_file_copy_atomic with safe temp filenames, atomic counter, EINTR handling and proper cleanup. Improve pal_sendfile portability/formatting across platforms and tidy many PAL file I/O helpers (error mapping, parameter validation, consistent style). Misc: add /docs to .gitignore and adjust Makefile ENABLE_ZHTTPD default behavior. Includes whitespace/formatting cleanups and small include reorderings.
Adds several FTP protocol improvements and a new HTTP stats API.
FTP:
- Added EPSV, OPTS, SITE and CLNT command handlers and registered them in the command table.
- Implemented MLSD (machine-readable directory list) and redirected MLST to STAT.
- Changed USER/PASS to accept immediate login for better compatibility with Android clients.
- Relaxed PORT IP validation and added debug logging for PORT targets.
- Open data connections before sending 150 and add short pacing delay to avoid merged 150+226 replies (LIST/NLST/MLSD).
- Implemented EPSV passive socket setup and OPTS handling (UTF8/MLST) and SITE CHMOD no-op.
- Implemented atomic STOR uploads using a temp file and rename on success; cleanup on failure.
- Improved active-mode connect behavior with a short send timeout and logging; simplified data close path to match upstream behavior.
- Accept bare LF line endings from some clients and make command lookup case-insensitive.
- Enabled TCP_NODELAY on control socket and turned on FTP command logging; bumped RELEASE_VERSION to 1.2.3.
HTTP / Server infrastructure:
- Extended http_response to support in-memory streaming, segment splicing, and added APIs (set_body_ref, set_body_splice).
- Modified static serving to use splicing for injected HTML and fall back to body ref for binary assets (PNG); added image/png content-type for .png.
- Added a new /api/stats endpoint providing JSON with disk usage, items in directory, boot/uptime, and CPU temperature where available, plus numerous helper functions for gathering and formatting stats.
Build / repo:
- Updated .gitignore, added web/zftpd-logo.png and bumped embedded web resources (large generated resource changes not detailed here).
Overall these changes improve client compatibility (especially mobile/Android and GoldHEN clients), make uploads atomic, add diagnostics via the HTTP API, and enhance HTTP response streaming.
Add internal API docs and tweak build/gitignore; bump release to 1.2.2. Increase default buffer/stream sizes and add data-specific I/O timeouts/linger for PS4/PS5/other platforms; disable transfer rate limiting on PS5. Introduce new PAL networking primitives (pal_socket_configure_data, cork/uncork, timeouts/drain helpers) and update pal_network.h/.c for safer, more configurable socket behavior and telemetry. Make file I/O more robust on consoles (F_NOCACHE, posix_fadvise) and harden HTTP CSRF token formatting to avoid overruns. Replace the old per-session token-bucket rate limiter with socket corking around sendfile/read loops, add fsync flushes and richer failure reporting in RETR/STOR/APPE, and implement a graceful data-socket close sequence to avoid RSTs. Minor Makefile fixes to ensure build directories exist.
Introduce optional ChaCha20 stream encryption negotiated via AUTH XCRYPT.
- Add new crypto API and implementation: include/ftp_crypto.h and src/ftp_crypto.c implement a self-contained ChaCha20 context, keystream generation (chacha20_block), ftp_crypto_init/ftp_crypto_xor/ftp_crypto_reset and a simple PSK+nonce key derivation.
- Add compile-time toggle and default PSK in include/ftp_config.h (FTP_ENABLE_CRYPTO, platform-disabled for PS4/PS5). Default PSK provided but documented to be changed before production.
- Embed per-session crypto context in ftp_session_t (include/ftp_types.h). When crypto is disabled a small stub context is used to preserve layout.
- Add AUTH XCRYPT command handler and handshake in src/ftp_commands.c (formats 234 reply with nonce, derives session key, initializes crypto). FEAT advertises XCRYPT and the RETR fast-path (sendfile) is disabled when encryption is active so data can be XORed in userspace.
- Build integration: add src/ftp_crypto.c to Makefile.
- Misc: minor header doc formatting/cleanup.
This change enables per-session encrypted control+data when negotiated; clients that do not request AUTH XCRYPT remain plaintext. Compile-time flag allows disabling the feature on platforms where encryption is undesirable.
Bump FTP_MAX_SESSIONS from 16 to 32 and refactor ftp_commands.c: reorder/includes and whitespace cleanup. Add a per-session token-bucket rate limiter and integrate it into RETR/STOR/APPE transfers so bandwidth can be capped; use pal_sendfile when available and rate limiting is disabled. Implement REST-aware resume semantics for STOR (open flags, lseek to restart_offset) and ensure REST clears/validates offsets. Enhance RETR to handle sendfile vs buffered reads, update session activity/stats and logging, and improve error handling for data connections. Also adjust directory listing code (safe-stat detection) and minor protocol command formatting/validation improvements.
Introduce a platform allocator and scratchpad: add pal_alloc.{h,c}, pal_alloc_override.h, and pal_scratch.{h,c} (APIs and a default arena). Integrate them: add sources to the Makefile, change artifact naming and add all-platform build targets, and wire pal_scratch into main for sysctl buffer usage. Improve robustness and safety: enable FTP_LIST_SAFE_MODE in ftp_config.h, add stat-skipping and fallback behavior in directory listings, strengthen file-type checks in RETR, and harden STOR/APPE data-paths (use pool buffers, proper error handling, 64-bit counters, and logging). Fix concurrency/atomic logic in ftp_buffer_pool (use compare_exchange_strong with retry) and update ftp_session to refresh last_activity on I/O. Minor updates: mark fallback buffer as _Thread_local in pal_fileio, update docs to describe the memory strategy, bump release to 1.2.1, add tests and QA scripts, and add license headers and small metadata changes (event_loop author).
Introduce an embedded Web File Explorer (zhttpd) and unified event loop implementation. Adds HTTP server headers and sources (http_api, http_parser, http_response, http_server, http_csrf, http_config), kqueue-based event loop (event_loop + event_loop_kqueue), embedded resource tooling and web UI (index.html, app.js, style.css, tools/generate_resources.py), API docs (docs/API.md), and tests (tests/test_http_query.c). Update Makefile to add ENABLE_ZHTTPD toggle (enabled by default on PS4/PS5), visibility flag for console targets, and test target for the new HTTP query test. Minor cleanup to include/ftp_config.h formatting and add ftpd.log to .gitignore.
Bump project version to 1.2.0 and introduce multiple runtime/build improvements: Makefile now supports STRIP and produces .bin for PS4/PS5 with updated deploy targets; README rewritten/streamlined and license set to MIT. Increase default buffer and TCP buffer sizes for PS4/PS5 and add FTP_SOCKET_TELEMETRY in include/ftp_config.h. Add pal_file_write_all prototype and switch STOR/APPE handlers to use ftp_buffer_acquire/ftp_buffer_size/ftp_buffer_release plus pal_file_write_all for robust I/O. Move rate limiting to account for actual bytes received in ftp_session_recv_data. Substantially improve PS4/PS5 main logic: safer pid lookup, terminate existing instances, port-fallback initialization (server_init_with_fallback), richer notifications, reorganized includes and better CLI parsing for POSIX. Miscellaneous formatting and error-handling refinements across sources.
Add a lightweight logging subsystem and MIT headers, enable unit tests, and harden session/path handling.
Key changes:
- Introduce ftp_log.h and src/ftp_log.c for session/command logging and line logging.
- Add MIT license blocks and update file dates across many headers/sources.
- Makefile: include src/ftp_log.c, add LIB_OBJECTS (build objects without main) and multi-test support, PS5 link libSceNotification, and skip tests on unsupported targets.
- Add tests/tests: tests/test_security.c and tests/test_path_security.c and Makefile rules to build/run them.
- Extend ftp_session_t with root_path, control RX buffer, auth flags, and rate-limiter fields; initialize realpath(root) and set socket timeouts during init.
- Add compile-time config knobs in include/ftp_config.h (control/data timeouts, transfer rate limits and burst size).
- Improve RETR/STOR/APPE flows: aggregate bytes sent/received, use new logging hooks, prefer sendfile only when no rate limit, and unify data send path via ftp_session_send_data.
- Strengthen PORT/PASV handling: restrict PORT to same client IP, reuse/close previous pasv socket, bind PASV to INADDR_ANY, and robustly determine local IP for PASV response.
- Enhance path normalization and resolution (ftp_path.c): safer component parsing, realpath checks, and validation against session root to prevent traversal/symlink escapes.
- Add pal API additions in headers (pal_socket_set_timeouts, pal_send_all) and use pal_send_all for control replies.
- Add helper utilities in ftp_session.c (wait_fd_ready, monotonic_ns) and other misc cleanups.
Overall this commit focuses on observability, testability, and security/robustness of session, path and data handling.
Increased default thread stack size on PS4/PS5:
FTP_THREAD_STACK_SIZE set to 512KB on console platforms
Remains 64KB on other platforms
Updated: ftp_config.h
Explicitly applied stack size during thread creation using pthread_attr_setstacksize():
Applied to both accept thread and session thread
Updated: ftp_server.c
seregonwar