ashui-media

Reactive audio and video playback, codecs, streams and equalization for ashui.
https://github.com/rayzor-blade/ashui

To install, run:

haxelib install ashui-media 0.2.4 

See using Haxelib in Haxelib documentation for more information.

README.md

ashui-media

Audio/video playback components, equalization, codecs and bounded streams backed by hlavi. Install ashui-media in applications that need media support. Native playback owns audio output, decoding and the audio/video clock. Video frames upload to reusable GPU textures through CanvasKit, with ashui's transforms, clipping, rounded corners and opacity.

Setup

Install with haxelib:

haxelib install ashui-media

Add -lib ashui-media to your app's build.hxml. Haxelib installs ashui, components, canvaskit and hlavi as dependencies. The packages enable HXX and stage the native libraries beside the bytecode; run it with Ash.

Playback example

The existing MediaPlayback demo uses the default video controls over a blurred overlay, alongside a custom HXX audio player.

Video playback and a custom audio player

HXX tags

import ashui.media.Video;
import ashui.media.Audio;
import ashui.media.VideoFit;

return <div class="flex flex-col gap-4">
  <video src="/api/public/p/aHR0cHM6Ly9saWIuaGF4ZS5vcmcvcC8lN0J2aWRlb1BhdGglN0Q" autoplay={true} muted={true} loop={true} fit={VideoFit.Contain} />
  <audio src="/api/public/p/aHR0cHM6Ly9saWIuaGF4ZS5vcmcvcC8lN0JhdWRpb1BhdGglN0Q" />
</div>;

Both tags accept reactive src, autoplay, loop, volume (0–1), muted, equalizer, and controls, plus onReady, onEnded, onError, and id. Default controls are optional: controls={false} renders the surface and supplied children. Video.fit accepts Contain, Cover or Fill and can change reactively. Standard layout attributes and CSS classes apply to the component's root.

A component creates and closes its own player. Supplying player={controller} shares a caller-owned controller; unmounting the component leaves it alive. Use ref={videoRef} or ref={audioRef} to access an internally created controller through .player.

Custom playback UI

import ashui.components.Button;
import ashui.media.Player;
import ashui.media.Video;
import ashui.reactive.Computed;
import ashui.reactive.Owner;

var player = new Player(path, {muted: true});
var playing = Computed.make(() -> player.isPlaying());
Owner.onCleanup(player.close);
return <div class="flex flex-col gap-3">
  <video player={player} controls={false} />
  <button type="button" onClick={_ -> player.toggle()}>
    ${playing.get() ? "Pause" : "Play"}
  </button>
  <text>${player.position.get() + " / " + player.duration.get()}</text>
</div>;

Use <playback-controls player={player} /> to place the default controls separately. They reuse framework Button, Slider and Popover behavior, including keyboard navigation, focus and dismissal. HXX children can provide additional controls.

Video controls use a translucent, blurred overlay. The reactive controls prop shows or hides them without resizing the video. The toolbar includes 10-second backward/forward seeks, an icon Play/Pause button, timeline, elapsed/total time, volume and a repeat/restart menu. Timeline clicks, drags and key presses seek; playback position updates only move the thumb.

<volume-control player={player} /> opens a compact horizontal slider with a percentage and mute icon button.

Video controls fade out after two seconds without pointer movement during playback and return when the pointer moves over or re-enters the video. Paused controls stay visible, as do controls with keyboard focus or an open volume/menu popup. autoHideControls={false} keeps them visible; controlsHideDelay={3} changes the delay in seconds. Both the default controls and custom HXX children remain optional. video.controlsVisible reports the auto-hide state.

Controller API

new Player(?path, ?options) accepts autoplay, loop, volume, muted, equalizer and an optional animation scheduler. It works independently of UI owners.

MethodBehavior
load(path, ?autoplay)Replace the file; an empty path unloads it
play(), pause(), toggle()Control native playback; play restarts EOF
seek(seconds)Bound the seek to the file; update a paused video frame
setVolume(value), setMuted(value)Control volume while preserving the unmuted level
setLoop(value)Restart playback at EOF
setEqualizer(equalizer), clearEqualizer()Attach shared settings or restore flat playback
currentFrame()Clone the latest video frame; caller must close it
update()Poll on the opening thread; normally the UI scheduler does this
close() / dispose()Terminal, idempotent release of player and frames

Signals expose source, state, error, position, duration, volume, muted, looping, equalizer, videoWidth, videoHeight and frameVersion. Read them for UI state and use methods to control playback. Times are seconds.

The FSM manages Opening, Seeking, Buffering and stable playback states. Opening and seek timers are cancelled on transitions and disposal. stateName() returns the lowercase CSS state. isPlaying() reflects playback intent through buffering and seeks, including seeks started while paused.

Drive playback on the main event-loop thread. Native playback uses real time, independently of animation steps. A standalone host must pump its platform event loop as the framework's window loop does. Paused playback stops polling after its preview/seek settles; audio labels update at 10 Hz.

Equalizer

ashui.media.Equalizer wraps hlavi's native peaking-band DSP for both playback and decoded PCM. Developers can build their own controls using its reactive settings and methods. The default five bands are 60, 250, 1000, 4000 and 12000 Hz, enabled at zero gain and Q=1. Supply 1–16 frequencies for another arrangement.

import ashui.media.Equalizer;
import ashui.media.Audio;

var eq = new Equalizer();
eq.setGain(0, 4);
eq.setGain(4, 2);
eq.setPreamp(-6);
return <audio src="/api/public/p/aHR0cHM6Ly9saWIuaGF4ZS5vcmcvcC8lN0JhdWRpb1BhdGglN0Q" equalizer={eq} />;

<video equalizer={eq} /> accepts the same controller. equalizer can be a signal or expression; replacing it detaches the previous controller, and null restores flat playback. A controller can be shared by several players. Settings edits update all attached players immediately, including paused players, and persist across source reloads. Each player owns its independent filter history. Passing an equalizer does not change volume, mute or playback state.

MethodBehavior
new Equalizer(?frequencies)Create flat, enabled bands with the given center frequencies
setBand(index, frequency, gainDb, q)Configure and enable a band; Hz 1–96000, gain −24…+24 dB, Q 0.1–20
setGain(index, gainDb)Change gain and enable the band, preserving its frequency and Q
disableBand(index)Bypass one band while retaining its settings
setPreamp(gainDb)Set input headroom, −60…0 dB
setBypass(value)Bypass both bands and preamp, preserving the settings
flatten()Restore zero gains and preamp, retaining frequencies, Q and bypass
process(audioData)Borrow PCM input and return owned interleaved F32 output
reset()Clear the controller's PCM processing history, preserving settings
close() / dispose()Release DSP and detach every attached player; terminal and idempotent

Read bands[index].get() for {frequency, gainDb, q, enabled}, preamp.get() and bypassed.get() to observe settings. Change settings through the methods. revision changes on settings edits or close; disposed is also reactive. Invalid settings throw and preserve the current configuration. Boosted bands may need negative preamp to avoid clipping at native playback's output. The native DSP smooths settings changes over 10 ms; frequencies at or above half the input sample rate are inactive.

An equalizer created under an Owner closes during cleanup. Otherwise, call close explicitly. Players leave caller-owned equalizers open. Closing an equalizer detaches it and restores flat playback on attached players.

For PCM processing, use one controller per stream. process retains filter history across contiguous blocks and resets on timing or format changes. Close each returned block; the input remains caller-owned. Output preserves timestamps, sample rate and channel count. Use media.AudioEqualizer for raw native handles without reactive integration.

Native data API

Use hlavi's existing typed media package directly alongside ashui.media:

  • media.AudioData: PCM creation, format conversion, copying and cloning.
  • media.VideoFrame: pixels, timestamps, visible/display dimensions, color metadata, plane layouts, copying and cloning.
  • media.EncodedAudioChunk, media.EncodedVideoChunk: encoded bytes, timing, kind and copying.

Timestamps and durations on buffers/chunks are Int64 microseconds. Close each owned data handle and each PlaneLayouts returned by copyTo. Garbage collection does not release native media handles. See hlavi's MediaData example.

Encoding, decoding and streams

ashui.media.Codec uses hlavi's native codec workers. It exposes typed input and output rather than converting media into UI objects:

FactoryInput → output
Codec.audioEncoder({sampleRate, channels, ?codec, ?bitrate, ?limits})AudioData → EncodedAudioChunk
Codec.videoEncoder({width, height, ?codec, ?bitrate, ?framerate, ?limits})VideoFrame → EncodedVideoChunk
Codec.audioDecoder(configuration, ?limits)EncodedAudioChunk → AudioData
Codec.videoDecoder(configuration, ?limits)EncodedVideoChunk → VideoFrame

Default profiles are AAC-LC (Codec.AAC, mp4a.40.2) and H.264 Baseline level 3 (Codec.H264, avc1.42001E), matching hlavi's platform profiles. Audio defaults to 128 kbps; video to 2 Mbps and 30 fps. Supply the encoder's owned getConfiguration() result to a decoder or MP4 writer after the first output is ready, then close that configuration when it is no longer needed.

ashui.media.Stream creates typed native queues with audio(), video(), audioChunks(), videoChunks() or bytes(). Byte queues copy writes and preserve block boundaries. Frames/chunks are retained by native reference.

Both codecs and streams share MediaChannel:

MethodBehavior
tryWrite(input)Nonblocking; false means retain the input and retry after draining output
poll()media.StreamReadStatus.Pending, Ready or Ended
read()Transfer one ready output to its caller
finish()Stop accepting input, flush, and drain before EOF
close() / dispose()Cancel and release native resources; idempotent

An accepted write retains its input, so the caller can close its own frame or chunk handle immediately. A rejected write leaves ownership with the caller. Every native frame/chunk read must be closed by its caller. A channel created inside an ashui Owner closes on cleanup; standalone callers close it. The shared FSM exposes reactive state (Open, Backpressured, Draining, Ended, Failed, Closed) and error. Backpressure never drops frames.

import ashui.media.Codec;
import haxe.io.Bytes;
import media.StreamReadStatus;

var encoder = Codec.videoEncoder({width: 640, height: 360, framerate: 30,
  limits: {maxItems: 2, maxBytes: 4 * 1024 * 1024}});

// On a producer tick: keep frame if the write is rejected.
if (encoder.tryWrite(frame)) frame.close();

// On later host/UI ticks: consume output without waiting for it.
while (encoder.poll() == StreamReadStatus.Ready) {
  var packet = encoder.read();
  var payload = Bytes.alloc(haxe.Int64.toInt(packet.byteLength()));
  packet.copyTo(payload);
  packet.close();
  // payload now holds the encoded bytes; native packet ownership is released.
}

encoder.finish(); // Continue consuming until poll() reports Ended.
// Close after EOF, or close earlier to cancel.

MediaLimits bounds item count and bytes on each native queue. Defaults are 8 items and 16 MiB; valid ranges are 1–1024 items and 1 byte–256 MiB. Each frame/packet must fit its queue's byte budget. Keep a rejected item for retry and drain output while producing input to avoid blocking a full pipeline. PCM/AAC timestamps must be contiguous; video timestamps strictly increasing. All frame/chunk timestamps and durations are Int64 microseconds.

For containers, use hlavi's media.MediaMuxer (incremental MP4 writing) and media.MediaDemuxer (incremental local MP4 reading) directly. These expose the same nonblocking readiness and write-backpressure contracts. Finish the tracks and writer before opening an exported file with <video> or <audio>. The reader supports ordinary, unfragmented MP4 files; a byte queue alone does not add a network transport or streaming container parser.

Styling and rendering

The package's media stylesheet loads below application CSS. Parts are .ui-media-video, .ui-media-surface, .ui-media-audio, .ui-media-controls, .ui-media-timeline, .ui-media-toolbar, .ui-media-time, .ui-media-volume-control and .ui-media-error. Roots expose [data-state]; video also exposes [data-controls="visible"|"hidden"]. Customize --ui-media-height, --ui-media-bg, --ui-media-radius, --ui-media-controls-bg, --ui-media-fg and --ui-media-padding, --ui-media-overlay-bg, --ui-media-overlay-blur and --ui-media-controls-radius, or replace controls completely.

Playback currently supplies packed BGRA frames. The renderer also accepts RGBA, RGBX and BGRX, handles visible crops and display aspect ratios, and reuses its buffer/textures until dimensions or format change. It draws SDR color through the framework's normal color target; HDR, rotation metadata, planar YUV rendering and browser media need additional backend support.

Native file/codec/platform support follows hlavi (Apple, Windows, Android and Linux). Verification here is on macOS; other platform backends are provided by hlavi and are not exercised by this repository's macOS run.

Repository demos and verification

The source repository contains these development runners and scenes. Run the following from a configured framework checkout; these scripts are not included in the haxelib package.

tools/demo/run.sh tools/demo/MediaPlayback.hx
tools/snapshot/run.sh tools/demo/MediaPlayback.hx
tools/snapshot/run.sh tools/snapshot/scenes/MediaPlaybackTest.hx
tools/snapshot/run.sh tools/snapshot/scenes/MediaEqualizerTest.hx
tools/demo/run.sh tools/demo/MediaEncoding.hx
tools/snapshot/run.sh tools/demo/MediaEncoding.hx
tools/snapshot/run.sh tools/snapshot/scenes/MediaCodecTest.hx

Set ASHUI_VIDEO / ASHUI_AUDIO to absolute file paths; the demo defaults to tools/demo/assets/MediaPlayback.mp4, copied from hlavi's handpan example. HLAVI_HDLL=/path/to/xavi.hdll uses an existing native binary in the runners. ASHUI_MEDIA=1 opts in a scene which imports media indirectly. Scenes that import ashui.media opt in automatically.

The test generates a silent WAV, checks real audio playback, pause, seeking, looping, EOF/restart, failures, source replacement and native ownership. It also checks timeline clicks/drags/keys against decoded frame timestamps, auto-hide/hover/focus behavior, volume changes and control alignment, then captures contain/cover/fill video and controls offscreen. Offscreen verification pumps a hidden native window so asynchronous platform opening/seeking complete. It never advances playback with a simulated clock.

MediaEncoding generates frames, passes them through a bounded video stream and native H.264 encoder, writes an MP4, then uses the existing HXX video component to play it. MediaCodecTest verifies native AAC/H.264 encode/decode, MP4 export/demux, backpressure, EOF, ownership and reactive channel failures.

MediaEqualizerTest checks native gain/preamp/bypass against actual PCM, live settings changes, shared controllers, reactive HXX bindings, source reloads and native cleanup.

Licensed under Apache 2.0.

Contributors
darmie
rayzor
Version
0.2.4
Published
1 week ago
Dependencies
License
Apache

All libraries are free

Every month, more than a thousand developers use Haxelib to find, share, and reuse code — and assemble it in powerful new ways. Enjoy Haxe; It is great!

Explore Haxe

Haxe Manual

Haxe Code Cookbook

Haxe API documentation

You can try Haxe in the browser! try.haxe.org

Join us on GitHub!

Haxe is being developed on GitHub. Feel free to contribute or report issues to our projects.

Haxe on GitHub

⌂