Merge feat/playback: session playback, synced audio and thumbnails
# Conflicts: # README.md # crates/clipbin-desktop/Cargo.toml # crates/clipbin-desktop/src/main.rs # crates/clipbin-media/src/lib.rs # docs/validation.md
This commit is contained in:
@@ -370,3 +370,99 @@ AV1 export is unvalidated because this GPU has no usable AV1 encoder. B-frame
|
||||
export and ranges containing an internal timestamp gap need additional media
|
||||
fixtures; dropped-frame ranges are explicitly rejected. Linux default-player
|
||||
and file-manager associations still need testing on a real desktop.
|
||||
|
||||
## Playback and thumbnails
|
||||
|
||||
Playback uses the shared `clipbin-media` crate (`ffmpeg-next` 9) without GPUI.
|
||||
One background decoder opens only completed, available segments. Session time
|
||||
comes from `Metadata::locate`; each file's timestamps are normalised independently.
|
||||
Seeking goes backwards to a keyframe, then decodes forward to the selected frame.
|
||||
A one-frame mailbox and coalesced seeks bound work; overdue frames are decoded
|
||||
but discarded before swscale. Playback threads have lower scheduling priority
|
||||
than recording. Leaving the editor closes its decoder and audio stream.
|
||||
|
||||
The editor uploads packed BGRA at its actual pixel size, maintains a 16:9
|
||||
preview, and releases old GPUI images. Space/K toggle playback; arrows/J/L step
|
||||
one frame. The timeline seeks and drags across files and gaps; completed footage
|
||||
from active sessions becomes browsable on the next 250 ms metadata refresh
|
||||
within the recorder's segment completion delay. Missing/removed footage advances
|
||||
on the wall clock and shows “No footage (not captured / cleaned up)”. There is
|
||||
no live recording preview.
|
||||
|
||||
Audio uses cpal and swresample at the output device's rate/channel count, with
|
||||
sample conversion in the device callback. OBS's combined third audio track is
|
||||
selected when three tracks exist; otherwise the first track is used. The queue
|
||||
holds at most 250 ms plus one decoded block. Callback timestamps, including DAC
|
||||
latency, drive the master clock; gaps and mute use the wall clock. Callbacks do
|
||||
no decoding and never wait for the queue lock. Mute and volume are in the editor;
|
||||
an unavailable device is shown explicitly while video remains usable. The Linux
|
||||
rig uses its existing null sink and ALSA's PipeWire plugin as a fallback if cpal's
|
||||
native PulseAudio backend refuses the device. Install/configure `pipewire-alsa`
|
||||
or an ALSA default routed to PulseAudio on systems without a physical ALSA card.
|
||||
|
||||
One low-priority worker produces 320×180 PNG thumbnails beside the first playable
|
||||
session segment and exported MP4s. Its request queue holds 16 paths and duplicate
|
||||
requests coalesce; card discovery and all filesystem work run off the UI thread.
|
||||
`clipbin_media::thumbnail::{generate,cache_path,Thumbnails}` accept any MP4/MKV.
|
||||
The gradient remains while a thumbnail is pending or footage is unavailable.
|
||||
|
||||
`scripts/validate/playback-fixture.sh` generates independent 1080p60 H.264/AAC
|
||||
segments with one-second keyframes and a three-tone audio fixture. It passed on
|
||||
2026-10-08: exact frame pixels at frame indices 0/1/59/60/61/119 in every file,
|
||||
real-time boundary crossing, gaps, superseded seeks, pause/seek clock maths,
|
||||
bounded frame placement, resampling 48 kHz to 44.1 kHz and combined-track selection.
|
||||
The ordinary workspace tests skip the generated-media assertions unless
|
||||
`CLIPBIN_TEST_MEDIA_DIR` points to the fixture; use the script to exercise them.
|
||||
|
||||
Windows: the cpal WASAPI dependency cross-check passed, as did the dedicated
|
||||
Windows thread-priority bridge. Core/OBS/probe strict MSVC cross-lint passed.
|
||||
The full media cross-lint is excluded on this Linux machine because the FFmpeg
|
||||
Windows development import libraries are unavailable. On Windows set `FFMPEG_DIR`
|
||||
to an FFmpeg **7.1 shared development** package with MSVC import libraries matching
|
||||
the OBS-bundled avcodec-61/avformat-61/avutil-59/swscale-8/swresample-5 DLLs. Linux
|
||||
links system FFmpeg 8. Actual Windows decoding, WASAPI device latency/seek behaviour,
|
||||
GPUI image presentation and thumbnail caching need a Windows machine. Hardware
|
||||
video decode is not used; HEVC/AV1 software performance and real output devices
|
||||
other than the rig null sink still need measurement.
|
||||
|
||||
Final playback validation passed on 2026-10-08, serially with other rigs stopped:
|
||||
|
||||
- `cargo fmt --all -- --check`, strict locked workspace clippy, and locked
|
||||
workspace tests (with `CLIPBIN_TEST_MEDIA_DIR=target/playback-fixture`).
|
||||
- `env PATH=/usr/bin:/bin:$HOME/.cargo/bin scripts/validate/linux-rig.sh`:
|
||||
all nine scenarios. An earlier concurrent run lost its H.264 probe to SIGKILL;
|
||||
the serial rerun passed without timeout or sandbox changes.
|
||||
- The same PATH with `scripts/validate/linux-session.sh`: automatic recording,
|
||||
exact segment placement, close-window survival, game exit, library reload and
|
||||
clean Quit. Repeated after the final keyboard integration, also passed.
|
||||
- `scripts/validate/linux-playback.sh target/playback-e2e-final-6`: moving preview
|
||||
(11.0% changed pixels inside the preview), active recording with zero reported
|
||||
recording drops, completed-segment availability, real session thumbnail,
|
||||
cross-file drag, exact post-boundary frame/time, cleaned-up gap, zoom, arrow
|
||||
stepping, device audio, half-volume and mute. The audio monitor RMS was 0.08044;
|
||||
the player asserts video time stays within 20 ms of the audio master clock.
|
||||
Screenshots were inspected: `sessions-thumbnails.png`, `editor-playing-1.png`,
|
||||
`editor-playing-2.png`, `editor-seek.png`, and `editor-gap.png` in that directory.
|
||||
|
||||
The rig runs on an Intel i5-12500 (12 logical CPUs). These are development builds
|
||||
with dependencies at opt-level 2, software video decoding, 1080p60 H.264 source
|
||||
and an 800×450 target (the editor used 740×416). Five serial repetitions of
|
||||
`target/debug/examples/playback-check <sessions-root> <id>` with
|
||||
`scripts/validate/measure-cpu.py`, inside the null-sink rig, measured process CPU
|
||||
at **16.7% median / 17.7% p95 of one core** (12.2–17.7% range). The filesystem cache
|
||||
was warm; each repetition reopened its decoders. Across presented frames,
|
||||
decode/scale time was **2.39 ms median / 8.80 ms p95**; maximum **101.5 ms** included
|
||||
opening files and decoding forward on seeks. This is a test pattern with gaps,
|
||||
not a worst-case game or a release-build benchmark. Per-run logs and CPU reports
|
||||
are in `target/playback-timing-final`; `report.json` stores the distributions.
|
||||
|
||||
During the active-session e2e, the player presented **59.2 frames/s** while
|
||||
recording remained at zero reported drops. GPUI upload/construction time was
|
||||
**0.22 ms median / 0.48 ms p95**. UI frame intervals (including vsync, not CPU work)
|
||||
were 16.2 ms median / 32.5 ms p95; the one-frame mailbox drops presentation work
|
||||
rather than allowing delay to accumulate. `CLIPBIN_TRACE_FRAMES=1` logs decoded
|
||||
video time, audio time, decode/scale work, uploads and actual timeline bounds.
|
||||
The headless e2e restores keyboard focus after virtual-pointer commands and
|
||||
holds virtual keys briefly: instantaneous virtual input can disappear before
|
||||
GPUI/Wayland dispatch. Keyboard actions update the editor entity directly,
|
||||
avoiding a nested window borrow during event dispatch.
|
||||
|
||||
Reference in New Issue
Block a user