Any venue.
One schema.
FerroFeed is a normalization engine, not a connector bundle. Implement one connector trait and any venue inherits a canonical schema with dual nanosecond timestamps and lossless decimals, every output sink, live subscription control, and a full Prometheus surface — soak-tested, in pure Rust.
Connectors
One connector trait. Any venue.
FerroFeed is a normalization engine, not a fixed set of integrations. A connector implements one trait — connect, subscribe, yield native messages — and inherits the entire pipeline: canonicalization, reconnect and backpressure isolation, every output sink, and the full metrics surface. Nothing in the core assumes a particular market.
The connector trait
Implement one async trait — open the socket, subscribe, and return a stream of native messages. Reconnection, canonicalization, sinks, and metrics come for free.
- One trait; the engine owns the rest of the pipeline
- Exponential reconnect backoff (capped at 30s)
- Per-connector bounded ingress queue — isolated backpressure
- Emits Connected / Disconnected / Reconnected status
Canonical events
Native protocols decode into one MarketEvent envelope — order books, trades, quotes, funding, bars — the same shape regardless of where the data came from.
- L2 price-level and L3 per-order book lifecycle
- Trades, top-of-book quotes, funding rates, OHLCV bars
- Lossless
rust_decimal— never floats - Canonical
BASE-QUOTEsymbols, dual ns timestamps
Asset-class agnostic
The same pipeline carries crypto order books, equity quotes, and options chains. Provenance survives where it matters — per-side market centers on quotes, no NBBO collapse — and venue quirks normalize at the boundary.
- Crypto, equities, and options on one pipeline
- Per-side market centers preserved on quotes
- Condition codes normalize to canonical
TradeFlags - Funding normalized to a per-8h fraction — consumers don't re-scale
Dynamic control
Change the live subscription set without a restart. A fail-open watcher reconciles the desired set from the ctrl:v1control plane and diffs it against what's currently subscribed.
- Add or drop subscriptions live — no process restart
- Fail-open reconcile with desired-set diffing
- Explicit absent-vs-empty semantics on the desired set
- ~1s to reconcile a 300-item diff (soak-measured)
Architecture
Per-venue isolation, one pipeline
Each venue runs in its own connector with its own bounded ingress queue. A round-robin fair merger forwards into one global mpsc channel. One overloaded venue cannot stall another.
Connect
Per-venue WebSocket lifecycle — connect, subscribe, reconnect with exponential backoff (capped at 30s), heartbeat. Per-venue bounded ingress queue isolates backpressure.
Normalize
Native exchange protocols decoded into the canonical MarketEvent model. Symbols mapped to BASE-QUOTE. All prices and quantities rust_decimal::Decimal — never floats.
Merge
Round-robin fair merger across per-venue queues forwards into one bounded mpsc channel (default capacity 4096). Drop accounting is per-venue and global.
Sink
Pluggable OutputSink trait. ndjson over stdout, file, or UDS broadcast; a lock-free SPSC ring buffer via mmap for shm consumers; or the versioned md:v1Redis wire — ET-dated daily streams plus TTL'd hot-state keys.
Use Cases
Six recipes from venue to consumer
FerroFeed is a binary + TOML config; the surface area you work with is a config file, a sink choice, and a wire contract. Each card below is a runnable recipe — venue subscription, multi-sink fan-out, co-located shared-memory consumer, the versioned Redis md:v1 wire, the canonical event envelope, and Prometheus observability.
Compose a consolidated tape from any venues
FerroFeed is config-driven — a single TOML lists venues, symbols, and channels, and the same binary runs them all concurrently with per-venue readiness gating and independent reconnect / backpressure isolation. required_for_ready = truecomposes a deterministic startup gate before the consolidated tape emits. Mixing asset classes is the same shape: add a venue block, pick its channels — the engine doesn't care whether it's a crypto book or an equity quote.
# ferro-feed.toml — one consolidated tape, any venues [runtime] timeout_s = 30 channel_cap = 4096 metrics_addr = "127.0.0.1:9898" [sink] type = "stdout" [[venues]] # a crypto book id = "venue_a" exchange = "<connector>" # any implemented connector symbols = ["BTC-USD"] channels = ["book_l2", "trades", "funding_rate"] required_for_ready = true [[venues]] # an equity quote id = "venue_b" exchange = "<connector>" symbols = ["AAPL"] channels = ["quotes", "trades"] required_for_ready = true # … add more [[venues]] blocks as needed
One process → many sinks, simultaneously
Four sink types ship in-tree: stdout (ndjson + jq), file (append-only ndjson for replay), uds (Unix domain socket with broadcast to N simultaneous listeners), and shm (lock-free SPSC ring with JSON or FlatBuffers payloads). Backpressure is per-sink with explicit drop accounting — a slow listener never starves the others.
# ferro-feed.toml — broadcast UDS to N listeners [sink] type = "uds" uds_path = "/tmp/ferro-feed.sock" # Connect any number of consumers: # $ socat -u UNIX-CONNECT:/tmp/ferro-feed.sock STDOUT | jq . # $ python record.py /tmp/ferro-feed.sock > 2026-05-13.ndjson # $ ./live-dashboard /tmp/ferro-feed.sock # Or fan to a file for replay later: [sink] type = "file" log_path = "/var/log/ferro-feed/2026-05-13.ndjson" # Drops are per-sink and counted in metrics: # ferrofeed_drops_total{sink="uds",reason="slow_listener"}
Read the shared-memory ring without a copy
The shm sink writes lock-free SPSC ring-buffer frames into an mmap region. Co-located consumers — a risk engine on the same NUMA node, a quote dashboard, a strategy process — read frames with atomic head/tail under Acquire/Release semantics. Zero syscalls on the steady state, sub-microsecond producer→consumer hand-off.
// Rust consumer co-located with FerroFeed (same host). // Open the mmap ring read-only and tail it. use memmap2::Mmap; use std::{fs::File, sync::atomic::{AtomicU64, Ordering}}; let file = File::open("/dev/shm/ferro-feed")?; let map = unsafe {{ Mmap::map(&file)? }}; let head: &AtomicU64 = bytemuck::from_bytes(&map[..8]); let mut local_seq = 0u64; loop { let published = head.load(Ordering::Acquire); while local_seq < published { let frame = read_frame(&map, local_seq); on_event(frame); // risk · routing · UI local_seq += 1; }} }}
One canonical envelope across every venue
Every emitted event — book delta, trade, funding rate, OHLCV bar — wears the same envelope: a typed venue_id, canonical symbol, dual nanosecond timestamps (exchange_ns and local_ns), and an EventKind discriminator. Prices and quantities are rust_decimal strings — no float math, no precision loss. Funding rates are normalized to a per-period rate with a typed period_hours so cross-venue comparisons work without bespoke conversion.
// Funding-rate event — native period normalized to 8h { "venue_id": "venue-a", "symbol": "BTC-PERP", "exchange_ns": 1715616000000000000, "local_ns": 1715616000042118400, "kind": "funding_rate", "funding_rate": { "rate": "0.00007248", "period_hours": 8, "predicted_rate": null, "next_funding_ns": 1715644800000000000 }} }} // Same shape from every venue. Period is on each // event — no implicit "venue X is 1h, divide by 8" // conversion code living on the consumer.
Prometheus metrics + per-venue drop accounting
Set runtime.metrics_addr and FerroFeed exposes a /metrics endpoint with per-venue throughput and drop counters, per-channel parse-error counters, reconnect counts, and pipeline depth. Drops are accounted per source — backpressure from a slow consumer never shows up as a parsing problem on the venue connector. Pair with the built-in ratatui dashboard for live ops.
# ferro-feed.toml [runtime] metrics_addr = "127.0.0.1:9898" stats_interval_s = 60 # Then scrape /metrics: # curl -s 127.0.0.1:9898/metrics # Counters you get for free: # ferrofeed_events_total{venue,kind} # ferrofeed_drops_total{venue,reason} # ferrofeed_reconnects_total{venue} # ferrofeed_parse_errors_total{venue,channel} # ferrofeed_channel_depth{venue} # ferrofeed_uptime_seconds{venue} # Suggested SLO: drops_total[5m] / events_total[5m] < 1e-5
Publish the versioned md:v1 Redis wire
The redis sink writes the md:v1 wire: ET-dated daily streams per asset class plus TTL'd hot-state keys (quote / trade / bar) with an embedded stale_after_ms and a server-side monotonicity guard. Consumers integrate against the documented wire, not this codebase — and reseed from hot-state keys after any gap. Options subscriptions are driven live through the ctrl:v1control plane (~1 s to reconcile a 300-contract diff), and ferro-feed archive tails the streams to durable .ndjson.zst segments with at-least-once, ack-after-durable semantics.
# ferro-feed.toml — publish the md:v1 wire to Redis [sink] type = "redis" redis_url = "redis://127.0.0.1:6379" redis_maxlen = 100_000_000 # runaway-producer breaker # Consume the daily stream (ET-dated) + hot-state keys: # XRANGE md:v1:options:2026-06-15 - + # GET md:v1:quote:SPY # TTL’d, stale_after_ms # Drive subscriptions live (no restart): # SET ctrl:v1:<venue>:desired ‘["<contract>", ...]’ # Archive streams to durable .ndjson.zst: ferro-feed archive --config feed.toml
Sinks
Where the data goes
Five sink types covering four integration shapes — line-oriented streaming, broadcast fan-out, zero-copy co-located consumers, and a versioned Redis wire — plus first-class observability for each.
jq, append-only file for recording and replay, and Unix domain socket with broadcast fan-out to multiple clients.mmap and poll the head atomic with Acquire ordering — zero copies, no locks, no syscalls. JSON or FlatBuffers payload.stale_after_ms and a server-side monotonicity guard. Paired with the ctrl:v1 control plane and an archive subcommand for durable .ndjson.zst capture./health and /ready endpoints. /ready returns 200 only when all required venues are connected.Code Quality
Schema as a contract
FerroFeed is consumed by downstream systems that depend on schema stability, lossless precision, observable backpressure, and a feed that stays up. Each is a release-gate concern, not a tolerated drift.
Schema Stability
The canonical MarketEvent with EventKind enum is the consumer contract. Schema changes are breaking and called out in release notes. Downstream routes on venue_id, never on symbol-pattern inference.
No Float Math
All prices and quantities use rust_decimal::Decimal. Loss of precision is a release-gate failure, not a tolerated drift. Tests cover round-trip equality across every connector.
Drop Accounting
Per-venue and global drop counters. Backpressure drops are observable and labeled, never silent. The bounded channel default (4096) is documented and the metric surface tells you when you should raise it.
Soak-Tested & Supervised
A full-session soak run cleared 41.4M events at a 5,577 msg/s peak with zero drops. Fatal conditions exit by design; systemd / launchd / compose units restart on any exit and a kill -9 mid-session resumes streaming well inside 30s.
Conventions
Schema contracts at the boundary
Cross-venue normalization hides where data comes from. These conventions make it explicit so downstream consumers can reproduce semantics without reading every connector.
Dual nanosecond timestamps
Every event carries exchange_ns (when the exchange recorded it) and local_ns (when FerroFeed received it). Both in nanoseconds since epoch. No millisecond rounding.
Canonical symbols
Symbols rendered as BASE-QUOTE regardless of exchange (e.g. BTC-USD). Each connector maps native venue formatting to the canonical form at the boundary.
Funding rate normalization
Every connector normalizes its native funding rate to a decimal fraction per 8-hour interval, whatever the venue's native period (1h, 4h, 8h). Downstream consumers compare like-for-like and never re-scale.
Venue routing
Every event carries venue_id set from the configured [[venues]].id. Downstream consumers route on venue_id, never on symbol patterns.
Schema
Canonical event envelope
One JSON envelope across every venue, every channel, every sink that emits text
stdout, file, uds, and shm with shm_encoding = "json" all emit events with this envelope. The shape is the same regardless of which venue produced it — and whether the channel is books, trades, quotes, or funding.
The kind field discriminates the event type. Nine kinds cover the full surface: trade, quote, book_update, book_snapshot, book_l3_update, book_l3_snapshot, funding_rate, status, ohlcv_bar.
// Canonical event envelope (ndjson) { "venue_id": "venue-a-futures", "exchange": "venue-a", "symbol": "BTC-USD", "timestamp": { "exchange_ns": 1705312200000000000, "local_ns": 1705312200100000000 }, "kind": { "type": "trade", ... } }
Talk to us
FerroFeed is a normalization engine — an integration layer over venues' own feeds, not a market-data product you subscribe to. Reach out for design-partner support, integration guidance, or additional venue coverage.
hello@morphiqlabs.comTell us about your use case
- Venues, symbols, and channels you need
- Consumer pattern — co-located shm, broadcast UDS, ndjson recording
- Throughput target and acceptable drop budget
- Integration timeline and existing infrastructure