videocube
videocube · sourcesVIDEOCUBE — the VIDEO version of the audio CUBE oscillator. It ingests THREE 60-frame video rings — A (FLOOR), B (WALL/connector), C (CEILING) — each generated LIVE from a connected video input or loaded from a .frametable.png atlas, and combines them exactly the way audio CUBE combines its three wavetables: an OCCUPANCY-WEIGHTED trilinear morph. For every output pixel it reads a colour from each ring, derives an occupancy position from the WALL luma, and blends the three by the SAME cube-dsp occ weights — MORPH cross-fades A↔C through B, CONNECT (+ CONNECT STRENGTH) reshapes/swells the interior binding, CRUSH posterizes colour depth, SPACE CRUSH mosaics the sampling grid, SPACE DIFFUSE smears the coords, MATERIAL switches soft-blend↔hard mosaic and WRAP mirror-folds — so video_out is a recognizable MORPHED IMAGE of the sources. It ALSO emits AUDIO: the same three rings are reduced to luma heightfields and scanned by the IDENTICAL cube-dsp surface-height slice the audio CUBE plays, so audio_out is a mono wavetable-oscillator drone whose timbre is driven by the SAME knobs (MORPH/CONNECT/CRUSH/SPACE/slice Y·ROT/WRAP/MATERIAL) — the picture and the sound move through one shared field structure. TUNE/FINE set pitch, LEVEL the gain, FOLD a west-coast wavefolder on the audio (audio-only). A global READER (SMOOTH/MORPH/CHAOS) reads each ring's 60-frame history, FREEZE holds the live rings and LIVE forces the real-time read. Beyond video_out + audio_out it exposes SIX dedicated slice-visualizer video jacks — SCOPE (the 256-sample surface-height wave the audio plays), SLICE (the 2-D cutting-plane cross-section), DEPTH (the intersection-depth heightmap, bright=deep/loud), and a SMOOTH/MORPH/CHAOS triptych (the cross-section forced through each reader treatment) — each a different readout of the SAME field/slice, each PER-PORT gated so an unpatched jack costs ZERO GPU, and the SLICE VIEW toggle (TEXTURED/XRAY/WEIGHTS) recolours slice+triptych. Rendered at half engine resolution with a renderer-gated tap count (the FrameTable budget ×3); the audio slice is GPU-reduced + recomputed off the audio thread on a throttle. v1 ships MONO-DRONE-first (poly/ADSR = follow-up). The occupancy combine + colour blend + luma reduction are a 1:1 CPU mirror unit-tested in $lib/video/videocube-core, reusing the audio-CUBE field math (cube-dsp) wholesale. All ports live on the yellow drill-down PATCH PANEL. Look-affecting new WebGL shader — held for owner visual preview.
the faceplate
inputs
| id | cable | what it does |
|---|---|---|
video_a | video | The LIVE source recorded, frame by frame, into ring A (FLOOR — one of the two morph ends / the bottom surface of the 3D solid). Ignored when slot A is loaded from a file. Unpatched (and no file) leaves ring A black. RGB video stream |
video_b | video | The LIVE source recorded into ring B (WALL / connector — the surface that binds A and C through the middle of the solid). Ignored when slot B is loaded from a file. RGB video stream |
video_c | video | The LIVE source recorded into ring C (CEILING — the other morph end / the top surface of the solid). Ignored when slot C is loaded from a file. RGB video stream |
morph_cv | cv | CV that modulates MORPH (the FLOOR↔CEILING cross-fade of the 3D fill, A ↔ C through B), swept linearly over 0..1. Drives both the volume and the audio field. control voltage (CV); modulates morph_fc (additive offset — ±1 CV sweeps the full range, centered on the knob) |
connect_cv | cv | CV that modulates CONNECT (how the WALL binds A and C — rounded soft connector ↔ hard linear ramp), swept linearly over 0..1. control voltage (CV); modulates connect (additive offset — ±1 CV sweeps the full range, centered on the knob) |
connect_strength_cv | cv | CV that modulates CONNECT STRENGTH (over-emphasises B in the mid-band, swelling the connector through the solid / bulging the audio field), swept linearly over 0..1. control voltage (CV); modulates connect_strength (additive offset — ±1 CV sweeps the full range, centered on the knob) |
crush_cv | cv | CV that modulates CRUSH (posterize the colour + amplitude-crush the field density AND the derived audio — the same crush levels), swept linearly over 0..1. control voltage (CV); modulates crush (additive offset — ±1 CV sweeps the full range, centered on the knob) |
space_crush_cv | cv | CV that modulates SPACE CRUSH (voxelize the field lookup — a chunky 3D mosaic in the picture and the identical voxelization of the audio field lookup), swept linearly over 0..1. control voltage (CV); modulates space_crush (additive offset — ±1 CV sweeps the full range, centered on the knob) |
space_diffuse_cv | cv | CV that modulates SPACE DIFFUSE (pull the sampling toward the field’s lowest-information face — a smear in the volume and the same coord warp in the audio scan), swept linearly over 0..1. control voltage (CV); modulates space_diffuse (additive offset — ±1 CV sweeps the full range, centered on the knob) |
slice_y_cv | cv | CV that modulates Y (the cutting slice plane height through the solid — the same plane the audio surface-height scan reads), swept linearly over 0..1. control voltage (CV); modulates slice_y (additive offset — ±1 CV sweeps the full range, centered on the knob) |
slice_rx_cv | cv | CV that modulates ROT X (Euler tilt of the cutting plane about X — the same plane the audio reads), swept linearly over -pi..pi. control voltage (CV); modulates slice_rx (additive offset — ±1 CV sweeps the full range, centered on the knob) |
slice_ry_cv | cv | CV that modulates ROT Y (Euler tilt of the cutting plane about Y), swept linearly over -pi..pi. control voltage (CV); modulates slice_ry (additive offset — ±1 CV sweeps the full range, centered on the knob) |
slice_rz_cv | cv | CV that modulates ROT Z (Euler tilt of the cutting plane about Z), swept linearly over -pi..pi. control voltage (CV); modulates slice_rz (additive offset — ±1 CV sweeps the full range, centered on the knob) |
view_x_cv | cv | CV that modulates VIEW X (orbit-camera ELEVATION — the HORIZONTAL axis of the VIEW joystick pad), swept linearly over -pi..pi. Picture only — reprojects the volumetric view without touching the sound or the cutting plane. control voltage (CV); modulates view_rot_x (additive offset — ±1 CV sweeps the full range, centered on the knob) |
view_y_cv | cv | CV that modulates VIEW Y (orbit-camera AZIMUTH — the VERTICAL axis of the VIEW joystick pad), swept linearly over -pi..pi. Picture only — orbits the volumetric view without touching the sound or the cutting plane. control voltage (CV); modulates view_rot_y (additive offset — ±1 CV sweeps the full range, centered on the knob) |
fold_cv | cv | CV that modulates FOLD (the west-coast wavefolder on the derived audio — audio-only, no image effect), swept linearly over 0..1. control voltage (CV); modulates fold (additive offset — ±1 CV sweeps the full range, centered on the knob) |
spread_cv | cv | CV that modulates SPREAD (the temporal reader window width — how far a frozen table oozes through time), swept linearly over 0..1. control voltage (CV); modulates spread (additive offset — ±1 CV sweeps the full range, centered on the knob) |
scan_cv | cv | CV that modulates SCAN (the reader-centre POSITION — moves the reading centre through the 60-frame ring / scrubs a frozen table through its ~2 seconds), swept linearly over 0..1 (wraps at the ring seam). Drives both the picture surfaces and the audio reduce. control voltage (CV); modulates scan (additive offset — ±1 CV sweeps the full range, centered on the knob) |
tune_cv | cv | CV that modulates TUNE (the derived oscillator pitch in semitones), swept linearly over -36..36. Affects the audio only, not the picture. control voltage (CV); modulates tune (additive offset — ±1 CV sweeps the full range, centered on the knob) |
chroma_depth_cv | cv | CV that modulates CHROMA DEPTH (the overall colour→sound intensity of the CHROMASTACK hue morph), swept linearly over 0..1. Audio-only — the picture already shows full colour; this drives how much the colour reshapes the DRONE. 0 = the pure luma carrier (today’s sound). control voltage (CV); modulates chroma_depth (additive offset — ±1 CV sweeps the full range, centered on the knob) |
motion_cv | cv | CV that modulates MOTION (how much frame-to-frame picture change adds "alive" energy/loudness to the derived audio), swept linearly over 0..1. Audio-only. 0 = pure content (a static colourful frame still sounds rich). control voltage (CV); modulates motion (additive offset — ±1 CV sweeps the full range, centered on the knob) |
hue_mode_cv | cv | CV/GATE that toggles the HUE MODE bank (the colour→timbre character): low = MUSICAL (tonal), high = INSTRUMENT (analog↔digital). Swept linearly and rounded, so it acts as a gate. Audio-only. control voltage (CV); modulates hue_mode (additive offset — ±1 CV sweeps the full range, centered on the knob) |
outputs
| id | cable | what it does |
|---|---|---|
video_out | video | The PRIMARY output: a VOLUMETRIC RAY-MARCH of the 3D occupancy solid, textured by the three source videos joined across a real depth axis, under an orbit camera — you look THROUGH the three videos. Includes the cutting slice plane (the plane the audio reads) and a cube wireframe. The card preview shows this output; rendered at quarter engine resolution and LINEAR-upscaled to half res. RGB video stream |
audio_out | audio | The derived audio: the reader-selected frame of each ring reduced to a luma heightfield and scanned by the audio-CUBE surface-height slice (the SAME field, the SAME plane as the picture), played as a mono wavetable oscillator at TUNE/FINE pitch and LEVEL gain. A continuous drone whose timbre is shaped by every field knob; silent until the audio worklet stands up (a moment after spawn). audio signal |
scope_out | video | A SCOPE TRACE of the exact 256-sample surface-height wave audio_out plays (cube-dsp.sampleSlice of the cutting plane, after FOLD) — the literal picture of the sound. A dedicated small video readout; PER-PORT gated (renders nothing until patched). Because it visualises the audio-derived wave, it is flat until the audio worklet stands up and the rings fill (the same warm-up as audio_out). Route it into a video mixer/monitor to SEE the drone move as you fly the slice. RGB video stream |
slice_out | video | The 2-D CROSS-SECTION where the cutting plane cuts the 3-D solid: for each pixel a point (a,b) on the tilted plane (positioned by Y / ROT exactly as the audio's slice ray) reads the three ring surfaces + the cube-dsp occupancy, coloured by the SLICE VIEW flavour — TEXTURED (the occupancy-weighted source videos on the plane), XRAY (grayscale occupancy density) or WEIGHTS (false-colour R=floor / G=ceiling / B=wall occupancy shares). Y and the rotations slide/tilt the cut through the solid; it honours the card's READER mode. PER-PORT gated. RGB video stream |
depth_out | video | The intersection-depth HEIGHTMAP: for every plane point (a,b) a ray marches the solid along the plane normal and accumulates how much material it crosses (the 2-D extension of the audio's 1-D surface-height scan), colour-mapped so BRIGHT = DEEP = LOUD. The picture of the audio's loudness field across the whole plane. The z-march step count is renderer-gated (SwiftShader-safe). PER-PORT gated. RGB video stream |
smooth_out | video | The slice cross-section (as slice_out, honouring SLICE VIEW) forced through the SMOOTH reader treatment — a trailing frame of each ring — regardless of the card's READER selector. Routed alongside morph_out + chaos_out it lets all three temporal treatments be compared/mixed at once. PER-PORT gated. RGB video stream |
morph_out | video | The slice cross-section forced through the MORPH reader treatment — the newest (crisp) frame of each ring — regardless of the card's READER selector. Differs from smooth_out (which trails) under motion. Honours SLICE VIEW. PER-PORT gated. RGB video stream |
chaos_out | video | The slice cross-section forced through the CHAOS reader treatment — a per-pixel dithered frame across each ring's 60-frame window — regardless of the card's READER selector. The noisiest of the triptych; differs from smooth_out/morph_out under motion. Honours SLICE VIEW. PER-PORT gated. RGB video stream |
params
| id | label | range | default | curve |
|---|---|---|---|---|
tune | tune | -36..36st | — | linear |
fine | fine | -100..100¢ | — | linear |
morph_fc | morph | 0..1 | — | linear |
connect | connect | 0..1 | — | linear |
connect_strength | cnct str | 0..1 | — | linear |
crush | crush | 0..1 | — | linear |
space_crush | space crush | 0..1 | — | linear |
space_diffuse | space diffuse | 0..1 | — | linear |
fold | fold | 0..1 | — | linear |
spread | spread | 0..1 | — | linear |
scan | scan | 0..1 | — | linear |
chroma_depth | chroma | 0..1 | — | linear |
motion | motion | 0..1 | — | linear |
slice_y | y | 0..1 | — | linear |
slice_rx | rot x | -3.1416..3.1416 | — | linear |
slice_ry | rot y | -3.1416..3.1416 | — | linear |
slice_rz | rot z | -3.1416..3.1416 | — | linear |
level | level | 0..2 | — | linear |
view_zoom | view zoom | 0.3..3 | — | linear |
view_rot_x | view x | -3.1416..3.1416 | — | linear |
view_rot_y | view y | -3.1416..3.1416 | — | linear |
view_rot_z | view z | -3.1416..3.1416 | — | linear |
wrap | wrap | 0..1 | — | discrete |
material | material | 0..1 | — | discrete |
screen_on | ray-march | 0..1 | — | discrete |
reader_mode | reader | 0..2 | — | discrete |
freeze | freeze | 0..1 | — | discrete |
live | live | 0..1 | — | discrete |
slice_view | slice view | 0..2 | — | discrete |
hue_mode | hue mode | 0..1 | — | discrete |
controls
| control | what it does |
|---|---|
| chroma | CHROMA DEPTH (0..1, default 0.6): the CHROMASTACK colour→sound intensity — how much the picture’s COLOUR reshapes the derived drone on top of the luma cube carrier. The reduced RGB is read as a hue histogram that blends 8 band-limited harmonic archetypes (red = dark/hollow → violet = bright/rich) with a hue-centroid spectral tilt, so a colour change AUDIBLY sweeps the timbre. 0 = the pure luma carrier (byte-identical to the old, static sound); a GRAYSCALE picture always collapses to that carrier regardless of this knob (the clean fallback). Audio-only. CV via the chroma depth input. |
| connect | CONNECT (0..1, default 0): reshapes how the WALL (B) binds A and C through the solid interior — 0 = a rounded/soft connector (circle occupancy), 1 = a hard linear ramp toward B (sawtooth-V). Same occ profile drives the picture and the audio. CV via the connect input. |
| cnct str | CONNECT STRENGTH (0..1, default 0): over-emphasises B's connector in the mid-band of the solid (B swells through the volume / bulges the audio field). 0 = the exact CONNECT shape. CV via the connect strength input. |
| crush | CRUSH (0..1, default 0): posterize the ray-march colour (RGB quantized) AND amplitude-crush the field density (opacity) and the derived audio (the same crush levels). 0 = clean. CV via the crush input. |
| fine | FINE (-100..100 cents, default 0): fine pitch trim of the derived audio between the semitone steps of TUNE. Audio-only. |
| fold | FOLD (0..1, default 0): a west-coast wavefolder applied to the derived audio waveform (adds harmonics). Audio-only — it has no image analog. CV via the fold input. |
| freeze | FREEZE (0/1, default 0): stops all LIVE rings from advancing so the held surfaces can be scrubbed with the slice/view controls. File-loaded rings are always frozen. Picture + (via the ring content) audio. |
| hue mode | HUE MODE (0/1, default 0 = MUSICAL): which colour→timbre CHARACTER bank drives the chroma morph. MUSICAL = tonal harmonic archetypes (red = root/dark/hollow → violet = bright/rich); INSTRUMENT = warm colours sound analog-ish, cool colours digital-ish. A front-panel toggle; CV-GATED (a gate high selects INSTRUMENT). Audio-only. |
| level | LEVEL (0..2, default 1): output gain on the derived audio (applied after FOLD). 1 = unity. Audio-only. |
| live | LIVE (0/1, default 0): forces the real-time / no-lag ring read (the newest frame) for the SMOOTH reader (MORPH is already newest; CHAOS keeps its per-pixel dither). Applies to BOTH the picture and the audio surface selection. |
| material | MATERIAL (0/1, default 0 = SMOOTH): SMOOTH renders the solid translucent, blending the three videos by their occupancy weights; HARD renders a binary solid where one surface wins per voxel (a hard-cut mosaic), and the same in the audio field. |
| morph | MORPH (0..1, default 0): cross-fades the FLOOR fill (surface A) toward the CEILING fill (surface C) of the 3D solid through the WALL (surface B) — 0 biases toward the floor, 1 toward the ceiling. Drives both the volume and the audio field. CV via the morph input. |
| motion | MOTION (0..1, default 0): how much frame-to-frame picture CHANGE adds "alive" energy/loudness to the derived audio — still = quiet, moving = alive. BLENDED with content, not a replacement: a static but colourful frame still sounds rich, and MOTION 0 is pure content. Audio-only. CV via the motion input. |
| reader | READER (0..2, default 0 = SMOOTH): which frame each ring contributes as its surface (global — all three rings), in BOTH the picture AND the derived audio — they read the SAME temporal frame (the unified-field promise). 0 = SMOOTH (a trailing frame, sub-frame smoothed), 1 = MORPH (the newest frame, crisp), 2 = CHAOS (a per-pixel dithered frame across the ring window). Because CHAOS is per-pixel in the picture, the 1-D audio scan reads its window-MEAN representative frame. |
| scan | SCAN (0..1, default 0): the reader-centre POSITION — the FrameTable MORPH partner to SPREAD. It moves the reading CENTRE through the whole 60-frame (~2-second) ring: 0 = today’s per-mode centre (byte-identical to the pre-scan read), and turning it up shifts the centre scan·(N−1) frames back, wrapping at the ring seam. Where SPREAD widens the blend at a fixed centre, SCAN walks that centre (crisp or widened) THROUGH the ring — so a FROZEN table can be SCRUBBED through its captured frames and oozes through the ~2 seconds exactly like FrameTable’s MORPH. Drives BOTH the picture surfaces and the audio reduce (image + drone in lockstep). For CHAOS it shifts the per-pixel dither base. CV via the scan input. |
| ray-march | SCREEN (0/1, default 1 = on): perf gate for the ray-march. When off AND video_out is unpatched the render is skipped (the rings keep capturing and the audio drone keeps running). |
| rot x | ROT X (-pi..pi, default 0): Euler tilt of the cutting plane about X — the same plane the audio reads and the render draws. CV via the rot x input. |
| rot y | ROT Y (-pi..pi, default 0): Euler tilt of the cutting plane about Y. CV via the rot y input. |
| rot z | ROT Z (-pi..pi, default 0): Euler tilt of the cutting plane about Z. CV via the rot z input. |
| slice view | SLICE VIEW (0..2, default 0 = TEXTURED): the colorize flavour for slice_out AND the smooth/morph/chaos triptych (it does NOT change video_out, the audio, or the field). 0 = TEXTURED (the occupancy-weighted source videos where the plane cuts solid), 1 = XRAY (grayscale occupancy density), 2 = WEIGHTS (false-colour occupancy shares — R = floor-fill, G = ceiling-fill, B = wall/connector). Picture-only; no CV. |
| y | Y (0..1, default 0.5): the height of the CUTTING SLICE PLANE through the solid. It is the exact plane the audio surface-height scan reads AND the plane drawn (tinted) in the volumetric render. CV via the y input. |
| space crush | SPACE CRUSH (0..1, default 0): voxelize the field lookup — snaps the (x,y,z) sampling coordinates to a chunky 3D voxel grid (a blocky solid) and the identical voxelization of the audio field lookup. 0 = transparent. CV via the space crush input. |
| space diffuse | SPACE DIFFUSE (0..1, default 0): pulls the field sampling toward the cube’s lowest-information face (computed from the field, latched on the field, matching the audio scan) — a smear/gravity in the volume and the same coord warp in the audio. 0 = off. CV via the space diffuse input. |
| spread | SPREAD (0..1, default 0): the TEMPORAL sampling window WIDTH of the reader (FrameTable-style). At 0 each ring surfaces one crisp frame (byte-identical to the pre-window read); opening it Hann-averages a ±window of ring frames into every surface — a wider window blends more of the ring, so a FROZEN table oozes through time. The reader mode (SMOOTH lag / MORPH newest) + SCAN pick the window CENTRE; SPREAD sets its WIDTH. Feeds BOTH the picture surfaces and the audio reduce, so image and drone flow together. CHAOS keeps its per-pixel single frame. CV via the spread input. |
| tune | TUNE (-36..36 st, default 0): coarse pitch of the derived audio oscillator in semitones. CV via the tune input. Audio-only. |
| Videocube a file input {n} | FLOOR table - loads a .frametable.png atlas (a lossless 10x6 = 60-tile contact sheet, the format FRAMETABLE saves) into ring A, replacing live capture and holding the ring frozen so the loaded content can be scanned with SCAN and SPREAD. Session-only in v1: nothing is written to disk or to the patch, so a reload returns the slot to LIVE. |
| Videocube a live {n} | FLOOR live - points ring A (the FLOOR surface, one of the two morph ends) back at whatever is patched into video_a, resuming live frame-by-frame capture. This is the way back from a loaded table: a slot stays on its file until you press this. Unpatched, the ring fills with black. |
| Videocube b file input {n} | WALL table - loads a .frametable.png atlas into ring B, the connector surface. Session-only in v1. |
| Videocube b live {n} | WALL live - points ring B (the WALL / connector surface that binds FLOOR and CEILING through the middle of the solid) back at video_b for live capture. The same reset as FLOOR live, on the ring that decides how the other two join through space. |
| Videocube c file input {n} | CEIL table - loads a .frametable.png atlas into ring C, the ceiling surface. Session-only in v1. |
| Videocube c live {n} | CEIL live - points ring C (the CEILING surface, the far morph end) back at video_c for live capture. The same reset as the other two. |
| view x | VIEW X (-pi..pi, default 0.6): orbit camera ELEVATION. Picture only — rotates the volumetric view up/down without touching the sound or the cutting plane. |
| view y | VIEW Y (-pi..pi, default 0.7): orbit camera AZIMUTH. Picture only — rotates the volumetric view left/right. |
| view z | VIEW Z (-pi..pi, default 0): orbit camera ROLL (a genuine roll of the view). Picture only. |
| view zoom | VIEW ZOOM (0.3..3, default 1): the orbit camera distance (camera dist = 2.6 / zoom). Picture only — a deliberate divergence from audio CUBE’s viz-only view: here it shapes the OUTPUT frame. |
| wrap | WRAP (0/1, default 0): OFF reads each source video with a single clamped sample (the plain look). ON extends the sampling domain and mirror-folds it, so the videos MIRROR-TILE across the cube — a kaleidoscopic fold at the faces/mid-planes in the volume, and the matching mirror-fold of the audio slice samples that reach beyond the cube. Governs BOTH the picture and the derived audio. |
source
videocube.ts on GitHub.