A work-in-progress reference implementation for production-grade data platforms with open source, zero lock-in, and complete sovereignty — and how you can help
Every organization should own its data destiny. Yet most data platforms are built on closed ecosystems, vendor lock-in, and proprietary black boxes. You’re renting infrastructure, not building it. Your roadmap depends on a vendor’s priorities, not yours.
There’s a better way: build a sovereign data stack using open source tools. Full control. No vendor lock-in. Complete independence.
I’ve started building a reference implementation showing exactly how — and I need your help to make it real:
github.com/RonaldHensbergen/composable-data-stack/
Why Sovereignty Matters
Open source data infrastructure means:
- You own your architecture — no surprise pricing, license changes, or feature removals
- You control your data — it lives where you want, moves how you want, stays as long as you want
- You’re not locked in — switching tools doesn’t require rewriting your entire pipeline
- You can audit everything — security and compliance are transparent, not hidden behind NDAs
- You can contribute — fix bugs, add features, shape the tools your business depends on
Closed platforms promise simplicity. Open source promises freedom and long-term sustainability.
The Composable Data Stack Reference (In Development)
This project is a work in progress. The vision is clear; the implementation is just beginning.
We’re building a production-ready, open source data architecture composed of independent, interchangeable components:
Core Components (from the repo):
- Ingestion: Apache Airflow orchestrates data pipelines with full visibility and control
- Storage: PostgreSQL and DuckDB for flexible, embeddable data warehousing
- Transformation: dbt for SQL-first analytics engineering
- Monitoring: Prometheus and Grafana for observability across your entire stack
Extensions & Alternatives:
The beauty of open source is flexibility. Want real-time streaming instead of batch jobs? Swap in Apache Kafka. Need columnar analytics? DuckDB plugs in seamlessly. Want distributed compute? Add Apache Spark. Prefer different storage? Try MinIO (S3-compatible), Clickhouse, or Apache Iceberg for data lakes.
Each tool is independent. No vendor telling you they’re “optimized together.” You choose what works for your use case.
What Makes This Stack Sovereign
Open Source Foundation
Every piece is open source. No proprietary licenses. No surprise restrictions. You can run this on your own servers, on Kubernetes, on the cloud — wherever you want.
True Composability
- Airflow doesn’t care if you use PostgreSQL or DuckDB
- dbt works with any warehouse via adapters
- Kafka integrates with any downstream system
- Grafana monitors anything with an API
Replace any component without touching the rest. That’s real independence.
No Vendor Dependencies
- Apache Airflow: maintained by Apache Foundation, not a company’s product team
- PostgreSQL: 30+ year open source standard, community-driven
- dbt: core engine is open, thriving community ecosystem
- DuckDB: fast-moving open source, embeddable anywhere
- Kafka: battle-tested infrastructure, truly distributed
- Grafana: open source visualization with no phone-home requirements
You’re not betting your data infrastructure on a startup’s Series C funding round.
Current Status & How You Can Help
Where we are:
- Architecture and vision are defined
- Docker containers are being tested
- Core integration patterns are being documented
Where we need help:
- Testing Docker setups across different systems (Mac, Linux, Windows)
- Integration testing between components (Airflow → PostgreSQL → dbt, Kafka → DuckDB, etc.)
- Documentation and troubleshooting guides
- Example pipelines showing real data workflows
- Security hardening for production deployments
- Performance tuning recommendations
- Feedback on the architecture and design decisions
If you’ve built data infrastructure before, we need you. If you’re learning, this is a great way to understand how production systems fit together.
Building Your Own
The repository provides:
- Architecture diagrams showing how components connect
- Docker Compose examples to spin up the full stack locally (currently being tested)
- Integration patterns for Kafka, DuckDB, Grafana, and more
- Configuration templates for PostgreSQL, Airflow, dbt
- Documentation on swapping components and extending the stack
- Best practices for operating open source infrastructure at scale
Real-World Flexibility
Once fully tested, here’s what becomes possible:
- Real-time analytics? Add Kafka and a DuckDB sink. Nothing else changes.
- Petabyte-scale storage? Migrate from PostgreSQL to an object store with Iceberg or Delta. Your dbt jobs barely notice.
- Distributed compute? Plug in Spark for transformations. dbt adapters exist.
- Different warehouse? Switch to Clickhouse, Snowflake (yes, you can mix), or Redshift — dbt handles it.
With proprietary platforms, each upgrade is a risky migration. With this stack, you evolve gradually and safely.
The Sovereignty Dividend
Open source infrastructure costs more upfront — you operate it yourself. But you gain:
- Long-term cost predictability (no surprise price hikes)
- Portability (move between cloud providers, on-prem, hybrid without vendor lock-in)
- Security transparency (audit the code, find vulnerabilities faster)
- Community power (thousands of engineers improving the tools you depend on)
- Negotiating leverage (you can always self-host if vendors get too expensive)
Join Us
This project succeeds through community contribution. Whether you’re an experienced data engineer or someone building your first pipeline, there’s a way to help.
Check out the repository. Spin up the stack locally. Test the Docker containers. Report issues. Suggest improvements. Share your own integration patterns.
The goal: prove that you don’t need vendor lock-in to build world-class data infrastructure.
Your data is too important to rent. Let’s build a sovereign stack together.
Eerder gepubliceerd op Medium (engels).