Building an open source data streaming runtime with Rust and DataFusion
The columnar streaming engine behind Goldsky Turbo Pipelines: Apache DataFusion, plus a source that never ends.
Instant data for onchain finance
The borderless data platform for fintech and crypto businesses.
Proof of reserves, treasury, AML monitoring
Wallet balances and transfers
Deposit detection, cross-chain settlement
Tokenized RWA, prediction markets
From Flink to Streamling
We had a streaming platform. We learned from it. Then we built a lighter one.
Flink served us well, with a mature ecosystem and a big community.
It also cost us operational complexity and a real $/month floor.
Most of our workloads are CDC fan-out, enrichment, dedup and simple transforms.
None of them need distributed shuffle. We paid for distribution we could not use.
DataFusion and Arrow matured into a usable base for streaming (2022–2024).
5s from paused to running across our fleet. Customers feel it as the whole platform being faster.
Postgres builds every database layer in one binary
gram.y (Bison)views, rulescost-based, pg_statisticVolcano, tuple at a timeheap files, 8 KB pagesWAL, commit logMVCC snapshotsid, amountamount > 1000transactions
| id | from | to | amount | block | ts |
|---|---|---|---|---|---|
| 1 | 500 | ||||
| 2 | 1500 | ||||
| 3 | 2500 | ||||
| 4 | 750 | ||||
| 5 | 1200 | ||||
| 6 | 90 | ||||
| 7 | 3000 | ||||
| 8 | 640 |
The scan reads every row and column. The Filter above it drops 4 of 8 rows after they were already read.
A query engine you embed, then extend
Apache DataFusion is an extensible SQL query engine in Rust. It uses Apache Arrow as its in-memory format. Streamling runs on DataFusion 54.
Parser, planner, optimizer, UDFs and a vectorized executor are ready to use.
TableProviderSources and sinks become tablesExecutionPlanCustom physical operatorsExtensionPlannerLowers our custom nodesPhysicalOptimizerRuleRewrites the plan (we have 3)CDC fan-out, enrichment, dedup: no distributed shuffle needed. Pipelines go from paused to running in about 5s.
A streaming source is a TableProvider that never ends
ParquetSinkSELECT a, bWHERE …parquet · BoundedKafkaSinkSELECT a, bWHERE …kafka · UnboundedOnly the bottom node differs. Filter, Projection and UDFs above the scan are stock DataFusion.
Marker(N)Coordinator → sources, flows through the plan in batch metadataAck(N)Sinks → coordinator, after flushFinalizer(N)Sources commit offsetsOne YAML file: sources, transforms, sinks
_gs_opBuild from source and run your own pipeline
Needs Rust via rustup. The repo pins its toolchain.
One release binary: target/release/streamling.
A CSV in ./data/ acts as a bounded source.
pipeline.yamlSource → SQL filter → print sink.
Rows 2 and 3 print, then the job ends. Set type: continuous to keep polling.
# 1. clone git clone https://github.com/goldsky-io/streamling && cd streamling # 2. build cargo build --release --bin streamling # 3. data mkdir -p data && printf 'id,amount\n1,500\n2,1500\n3,2500\n4,750\n' > data/tx.csv # 4. pipeline.yaml sources: raw_transactions: type: file path: ./data/ format: csv primary_key: id mode: { type: bounded } transforms: large_transactions: type: sql primary_key: id sql: SELECT * FROM raw_transactions WHERE amount > 1000 sinks: print_large: { type: print, from: large_transactions } # 5. run ./target/release/streamling pipeline.yaml
What moving off Flink changed for Goldsky
Average CPU on a benchmarked production workload: read from Kafka, enrich, write into ClickHouse.
Moving our pipelines to Streamling cut Goldsky's cloud costs.
From a topology change to data written in the destination, including image download, pod creation and execution.
Streamling is open source today. Star it, clone it, raise an issue.
github.com/goldsky-io/streamling
streamling.dev
goldsky.com