data-lakehouse

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

A curated list of open source tools used in analytics platforms and data engineering ecosystem

End-to-end Data Lakehouse project built on Databricks, following the Medallion Architecture (Bronze, Silver, Gold). Covers real-world data engineering and analytics workflows using Spark, PySpark, SQL, Delta Lake, and Unity Catalog. Designed for learning, portfolio building, and job interviews.

Open-source data management for multimodal AI. Query, trace, and govern with a lineage-native, format-agnostic lakehouse for agents and teams. Supports biological formats and registries - by the creators of Scanpy. 🍊YC S22

Qbeast-spark: DataSource enabling multi-dimensional indexing and efficient data sampling. Big Data, free from the unnecessary!

Sample Data Lakehouse deployed in Docker containers using Apache Iceberg, Minio, Trino and a Hive Metastore. Can be used for local testing.

ETL / ELT / Reverse ETL Framework powered by DuckDB, designed to seamlessly integrate and process data from diverse sources. It leverages Markdown as a configuration medium, where YAML blocks define metadata for each data source, and embedded SQL blocks specify the extraction, transformation, and loading logic.

Hybrid recommender system + RAG chatbot for IT courses (FastAPI + Express/TS + PostgreSQL), with an independent Data Lakehouse module (Spark/Kafka/dbt/Trino) and full CI/CD

Floe: Policy-based table maintenance for Apache Iceberg

Repositório de desenvolvimento (MVP) para estruturação analítica do CadÚnico municipal. Sem dados pessoais — apenas código, dicionários, esquemas e amostras sintéticas. Alinhado à LGPD e boas práticas de governança.

Data Engine for AI/Algo Trading: Download/Stream -> Clean -> Store. Supports Data Lakehouse Architecture. Clean Once and Forget.

DatAasee - A Metadata-Lake for Libraries

SwiftLake: Java SQL engine built on Apache Iceberg and DuckDB for efficient lakehouse reads and writes

End-to-end retail lakehouse on Databricks: dual-path ingestion from PostgreSQL and S3, dbt-modeled galaxy schema with SCD2 history, Airflow orchestration with parallel DAG execution, CI-gated dbt tests, and a dual consumption layer via Power BI and a Genie Agent.

This repo provides a step-by-step approach to building a modern data warehouse using PostgreSQL. It covers the ETL (Extract, Transform, Load) process, data modeling, exploratory data analysis (EDA), and advanced data analysis techniques.

This repository is a place for the Data Warehousing course at the Information Systems & Analytics department, Santa Clara University.

Production-ready Apache Superset with DuckLake integration. Stateless analytics architecture using DuckDB for compute, PostgreSQL for metadata, and S3/GCS/MinIO for data lake storage. Includes Docker Compose, Kubernetes Helm charts, BigQuery Integration, and CI/CD workflows. Supports MotherDuck cloud integration.

Former fork combining dbt-fabric and dbt-fabricspark

End-to-end E-Commerce Lakehouse built on Databricks using Medallion Architecture (Bronze → Silver → Gold). Ingests raw CRM and ERP data, applies PySpark transformations, and delivers a star schema via Spark SQL stored as Delta Tables in Unity Catalog.

A project of creating a local data lakehouse using open-source tools and using Apache Iceberg as the open table format