Posts about Data Engineering
Practical data engineering: ingestion pipelines, repartitioning, open table formats and lightweight data platforms built with DuckDB and cloud services.
4 posts
Filter by tag
Filter by tag:
Introducing polyglot - A Rust/Wasm SQL transpilation library
polyglot transpiles SQL between 33 dialects like PostgreSQL, BigQuery and Snowflake. A Rust port of sqlglot, compiled to Wasm to run in the browser and Node.js.
Handling GTFS data with DuckDB
Load GTFS public transit schedule feeds into a DuckDB database, query stops, routes and trips with SQL, and export the data to Parquet.
Using DuckDB-WASM for in-browser Data Engineering
Analyze, transform and visualize local and remote data in the browser with DuckDB-WASM and SQL Workbench, including a full example data engineering pipeline.
Gathering and analyzing public cloud provider IP address data with DuckDB & Observable
Unify, clean and analyze the published IP address ranges of AWS, Azure, Google Cloud, Cloudflare, Oracle, DigitalOcean and Fastly with DuckDB and Observable.