tobilg.com

Introducing polyglot - A Rust/Wasm SQL transpilation library

Tobias Müller — Sat, 14 Feb 2026 16:56:46 GMT

What if you could take a SQL query written for PostgreSQL and transpile it to BigQuery, Snowflake, or any of 33 database dialects — entirely in the browser, with no server round-trip? That's what polyglot does.

SQL dialect fragmentation is a real problem. Every database has its own flavor: LIMIT vs TOP, ILIKE vs LOWER() LIKE, STRING vs VARCHAR vs TEXT. If you're building tools that work across databases, you end up writing dialect-specific code paths or maintaining parallel query sets. It's tedious and error-prone.

polyglot tackles this head-on. It's a SQL transpiler inspired by Python's sqlglot, but built from scratch in Rust, and can be compiled to WebAssembly. The result is a fast, portable library that works in the browser, in Node.js, or natively in any Rust project. It has 100% test fixture compliance to sqlglot, with more than 9k tests.

What is polyglot?

The architecture is straightforward: a core Rust library handles all the parsing, AST manipulation, and SQL generation. That core compiles to Wasm, which powers a TypeScript SDK (@polyglot-sql/sdk) for use in web and Node.js environments.

It supports 33 SQL dialects, including PostgreSQL, MySQL, BigQuery, Snowflake, DuckDB, SQLite, ClickHouse, Redshift, Spark, Trino, TSQL (SQL Server), Oracle, Databricks, Hive, Athena, Teradata, and more.

The key capabilities are:

Transpilation — convert SQL between any pair of supported dialects
Parsing — turn SQL strings into a fully-typed AST
Generation — produce SQL from AST nodes
Formatting — pretty-print SQL with proper indentation
Validation — check SQL for syntax and semantic errors
Builder API — construct queries programmatically with a fluent interface

The Rust Crate

The polyglot-sql crate exposes a clean, high-level API. The core functions cover the most common operations:

use polyglot_sql::{transpile, parse, generate, validate, DialectType};

// Transpile from PostgreSQL to BigQuery
let result = transpile(
    "SELECT id, name FROM users WHERE created_at > NOW() - INTERVAL '7 days'",
    DialectType::Postgres,
    DialectType::BigQuery,
)?;

println!("{}", result[0]);
// SELECT id, name FROM users WHERE created_at > CURRENT_TIMESTAMP() - INTERVAL 7 DAY

We can also parse SQL into an AST, manipulate it, and generate SQL back:

let ast = parse("SELECT 1 + 2", DialectType::Generic)?;
let sql = generate(&ast[0], DialectType::Postgres)?;

Builder API

For constructing queries programmatically, the fluent builder API avoids string concatenation entirely:

use polyglot_sql::builder::*;

let query = select(["id", "name", "email"])
    .from("users")
    .where_(col("age").gte(lit(18)).and(col("active").eq(lit(true))))
    .order_by(col("name").asc())
    .limit(100)
    .build()?;

This produces a proper AST that can be generated into any dialect. The builder supports joins, subqueries, aggregations, window functions, CTEs, and set operations (UNION, INTERSECT, EXCEPT).

Beyond the core API, the crate also includes scope analysis, column lineage tracking, AST traversal utilities, and optimizer passes — useful for building more sophisticated SQL tooling.

The TypeScript SDK

The TypeScript SDK (@polyglot-sql/sdk on npm)wraps the Wasm module and provides the same capabilities in JavaScript environments. Install it from npm:

npm install @polyglot-sql/sdk

Transpilation

import { transpile, Dialect } from "@polyglot-sql/sdk";

const result = transpile(
  "SELECT TOP 10 id, name FROM users WHERE name LIKE '%test%'",
  Dialect.TSQL,
  Dialect.PostgreSQL
);

console.log(result.sql);
// SELECT id, name FROM users WHERE name LIKE '%test%' LIMIT 10

Builder API

The TypeScript builder mirrors the Rust API:

import { select, col, lit, Dialect } from "@polyglot-sql/sdk";

const query = select(col("id"), col("name"), col("email"))
  .from("users")
  .where(col("active").eq(lit(true)))
  .orderBy(col("name").asc())
  .limit(25)
  .toSql(Dialect.PostgreSQL);

console.log(query);
// SELECT id, name, email FROM users WHERE active = TRUE ORDER BY name ASC LIMIT 25

Formatting

Pretty-printing SQL is a single function call:

import { format, Dialect } from "@polyglot-sql/sdk";

const result = format(
  "SELECT a.id, b.name, COUNT(*) FROM orders a JOIN users b ON a.user_id = b.id WHERE a.status = 'active' GROUP BY a.id, b.name HAVING COUNT(*) > 5",
  Dialect.PostgreSQL
);

console.log(result.sql);

SELECT
  a.id,
  b.name,
  COUNT(*)
FROM orders AS a
JOIN users AS b
  ON a.user_id = b.id
WHERE
  a.status = 'active'
GROUP BY
  a.id,
  b.name
HAVING
  COUNT(*) > 5

For quick browser integration, the SDK can also be loaded via CDN without a build step.

Integration Paths

Browser-based SQL Editors

Since @polyglot-sql/sdk can run as Wasm, it works directly in the browser with no backend. This makes it a natural fit for web-based SQL editors, data tools, and notebook interfaces that need dialect-aware transpilation or formatting.

CI/CD & Migration Pipelines

We can integrate the Rust crate or TypeScript SDK into build pipelines to validate SQL syntax, enforce dialect compatibility, or automatically convert queries during database migrations. The validate() function catches errors early, before they hit production.

Multi-database ORMs & Query Builders

The builder API can serve as a dialect-aware SQL backend for ORMs or custom query builders. Construct queries once using the fluent interface, then generate dialect-specific SQL at runtime.

Data Catalogs & Documentation Tools

Parsing SQL into a full AST enables extraction of table references, column lineage, and query structure. This is useful for data catalogs, documentation generators, and governance tools that need to understand what a query touches.

CLI Tools & Developer Utilities

The Rust crate can power command-line tools for SQL formatting, linting, or batch transpilation. It's fast enough for interactive use and handles large query files without issues.

Use Cases

Database migration: Converting thousands of queries from Oracle to PostgreSQL (or any other dialect pair) without manual rewrites
Multi-cloud analytics: Writing SQL once and transpiling to BigQuery, Snowflake, or Redshift depending on the target warehouse
SQL formatting & linting: Enforcing consistent SQL style across a codebase with pretty-printing
Query analysis & lineage: Parsing SQL to track which columns flow through transformations and which tables are referenced
Educational tools: Showing how the same query looks across different SQL dialects, side by side

Try It Out

The fastest way to see polyglot in action is the playground, where you can transpile and format SQL directly in the browser.

To get started in a JavaScript or TypeScript project:

npm install @polyglot-sql/sdk

import { transpile, Dialect } from "@polyglot-sql/sdk";

const result = transpile("SELECT * FROM t LIMIT 10", Dialect.PostgreSQL, Dialect.TSQL);
console.log(result.sql); // SELECT TOP 10 * FROM t

For Rust projects, add the crate to your Cargo.toml:

[dependencies]
polyglot-sql = "0.1"

You can find the source code on GitHub, the Rust docs on docs.rs, the TypeScript docs at polyglot.gh.tobilg.com, and the npm package at @polyglot-sql/sdk.

Summary

polyglot brings SQL transpilation to any environment — browser, server, or CLI — without depending on a Python runtime or external service. With 33 supported dialects, a fluent builder API, and full parsing/generation capabilities, it covers a wide range of SQL tooling needs.

The project is open source and actively developed. If you're working on SQL tooling, multi-database support, or migration pipelines, give it a try and let us know how it goes. Contributions, bug reports, and feature requests are all welcome on GitHub.

Custom DuckDB Wasm builds for Cloudflare Workers

Tobias Müller — Tue, 27 Jan 2026 19:47:43 GMT

What if you could run full SQL queries, including JOINs, aggregations, and even remote Parquet file reads, directly inside a Cloudflare Worker? No database server, no connection pool, no cold-start latency from external services. Just DuckDB, compiled to WebAssembly, running at the edge.

That's now possible with Ducklings, a minimal DuckDB WASM build I created specifically for browsers and serverless environments. In this post, I'll focus on the @ducklings/workers package, which makes it possible to deploy DuckDB to Cloudflare Workers for the first time (to my knowledge).

The challenge: DuckDB in resource-limited serverless environments like Workers

DuckDB has had a WASM build for a while through duckdb-wasm, but it was designed for browsers. Cloudflare Workers are a different beast: they don't have access to XMLHttpRequest, SharedArrayBuffer, or threads. Most critically, the Workers runtime doesn't support synchronous I/O, everything must go through async fetch().

This matters because DuckDB's httpfs extension, the one that lets you query remote Parquet, CSV, and JSON files over HTTP, relies on synchronous HTTP calls internally. In a browser, that works via XMLHttpRequest. In a Worker, it simply doesn't.

The solution is Emscripten's Asyncify, a compile-time transformation that lets synchronous C/C++ code call asynchronous JavaScript functions. Ducklings uses Asyncify to bridge DuckDB's synchronous HTTP calls to the Workers fetch() API, making httpfs work transparently.

What Ducklings provides

Ducklings is a from-scratch WASM build of DuckDB, optimized for minimal size:

Package	Size (gzipped)	API	HTTP transport
`@ducklings/workers`	~9.6 MB	Async	fetch() via Asyncify

The @ducklings/workers package includes Parquet, JSON, and httpfs extensions compiled in statically. No runtime extension loading, just what you need, baked into the binary.

The Workers build is larger because Asyncify adds instrumentation to every function in the call chain between DuckDB's HTTP layer and the JavaScript fetch() call. That's the price for making synchronous C++ code work in an async-only environment.

Getting started

Prerequisites

You'll need a Cloudflare Workers paid plan ($5/month). The DuckDB WASM binary is ~9.6 MB gzipped, which exceeds the free tier's 3 MB code size limit. The paid plan allows up to 10 MB.

Project setup

Create a new project and install dependencies:

mkdir duckdb-worker && cd duckdb-worker
npm init -y
npm install @ducklings/workers
npm install -D @cloudflare/vite-plugin vite wrangler typescript @cloudflare/workers-types

Configuration files

wrangler.jsonc - Cloudflare Workers configuration:

{
  "$schema": "node_modules/wrangler/config-schema.json",
  "name": "duckdb-worker",
  "main": "src/index.ts",
  "compatibility_date": "2026-01-11",
  "compatibility_flags": ["nodejs_compat"]
}

The nodejs_compat flag is required for some Node.js APIs that Emscripten's glue code uses.

vite.config.ts - Build configuration with the Ducklings Vite plugin:

import { defineConfig } from 'vite';
import { cloudflare } from '@cloudflare/vite-plugin';
import { ducklingsWorkerPlugin } from '@ducklings/workers/vite-plugin';

export default defineConfig({
  plugins: [
    ducklingsWorkerPlugin(),
    cloudflare(),
  ],
});

The ducklingsWorkerPlugin() handles resolving the WASM module import so that Vite and Wrangler can bundle it correctly.

tsconfig.json:

{
  "compilerOptions": {
    "target": "ES2022",
    "module": "ESNext",
    "moduleResolution": "bundler",
    "strict": true,
    "skipLibCheck": true,
    "types": ["@cloudflare/workers-types/2026-01-11"],
    "jsx": "react-jsx",
    "lib": ["ES2022"]
  },
  "include": ["src"]
}

package.json scripts:

{
  "type": "module",
  "scripts": {
    "dev": "vite dev",
    "build": "vite build",
    "deploy": "vite build && wrangler deploy"
  }
}

Writing the Worker

Here's a complete Worker that exposes a SQL query API:

src/index.ts:

import {
  init,
  DuckDB,
  version,
  tableToIPC,
  sanitizeSql,
  DuckDBError,
  type Connection,
} from '@ducklings/workers';
import wasmModule from '@ducklings/workers/wasm';

let db: DuckDB | null = null;
let conn: Connection | null = null;
let initialized = false;

async function ensureInitialized(): Promise<void> {
  if (initialized && db && conn) return;

  // Initialize the WASM module
  await init({ wasmModule });

  // Create database and connection
  db = new DuckDB();
  conn = db.connect();

  // Create sample data
  await conn.execute(`
    CREATE TABLE readings (
      sensor_id INTEGER,
      temperature DOUBLE,
      humidity DOUBLE,
      recorded_at TIMESTAMP
    )
  `);

  await conn.execute(`
    INSERT INTO readings VALUES
      (1, 22.5, 45.0, '2026-01-27 10:00:00'),
      (1, 23.1, 44.2, '2026-01-27 11:00:00'),
      (2, 19.8, 62.1, '2026-01-27 10:00:00'),
      (2, 20.3, 60.5, '2026-01-27 11:00:00')
  `);

  initialized = true;
}

export default {
  async fetch(request: Request): Promise<Response> {
    await ensureInitialized();

    const url = new URL(request.url);

    if (url.pathname === '/query' && request.method === 'POST') {
      const { sql } = await request.json() as { sql: string };

      // Block dangerous queries
      try {
        sanitizeSql(sql);
      } catch (e) {
        if (e instanceof DuckDBError) {
          return Response.json({ error: e.message }, { status: 400 });
        }
        throw e;
      }

      const rows = await conn!.query(sql);
      return Response.json({ data: rows, rowCount: rows.length });
    }

    if (url.pathname === '/stats') {
      const stats = await conn!.query(`
        SELECT
          sensor_id,
          AVG(temperature) AS avg_temp,
          AVG(humidity) AS avg_humidity,
          COUNT(*) AS reading_count
        FROM readings
        GROUP BY sensor_id
      `);
      return Response.json({ data: stats });
    }

    return Response.json({
      name: 'DuckDB Worker',
      version: version(),
      endpoints: {
        '/query': 'POST SQL queries',
        '/stats': 'GET sensor statistics',
      },
    });
  },
};

A few things to note about the Workers API:

init() takes a pre-compiled WASM module: In Workers, you import the WASM binary directly and pass it to init(). There's no fetch step, the module is bundled into the Worker.
All query methods are async: query(), execute(), queryArrow() all return Promises. This is because of Asyncify, under the hood, DuckDB's C++ code may need to pause and wait for an async fetch() call.
sanitizeSql() is built in: It blocks duckdb_secrets(), PRAGMA, COPY ... TO, and EXPORT DATABASE patterns. Use it when accepting SQL from untrusted sources.

Querying remote files

This is where it gets interesting. Because httpfs works in Workers, you can query remote Parquet, CSV, and JSON files directly:

// Remote Parquet file
const data = await conn.query(`
  SELECT *
  FROM 'https://example.com/data.parquet'
  LIMIT 100
`);

// Remote CSV
const csv = await conn.query(`
  SELECT *
  FROM read_csv('https://example.com/data.csv')
`);

// Remote JSON
const json = await conn.query(`
  SELECT *
  FROM read_json('https://example.com/data.json')
`);

No additional configuration needed as httpfs is compiled in and initialized automatically.

Accessing S3 and R2 storage

For private data, you can configure secrets to access S3-compatible storage. Here's how to set up Cloudflare R2:

interface Env {
  R2_ACCESS_KEY_ID: string;
  R2_SECRET_ACCESS_KEY: string;
  R2_ACCOUNT_ID: string;
}

async function ensureInitialized(env: Env): Promise<void> {
  // ... init code ...

  // Create R2 secret for accessing private buckets
  await conn.execute(`
    CREATE SECRET r2 (
      TYPE R2,
      KEY_ID '${env.R2_ACCESS_KEY_ID}',
      SECRET '${env.R2_SECRET_ACCESS_KEY}',
      ACCOUNT_ID '${env.R2_ACCOUNT_ID}'
    )
  `);
}

// Then query R2 files directly
const data = await conn.query(`
  SELECT * FROM 'r2://my-bucket/data/events.parquet'
`);

Set the secrets via Wrangler:

wrangler secret put R2_ACCESS_KEY_ID
wrangler secret put R2_SECRET_ACCESS_KEY
wrangler secret put R2_ACCOUNT_ID

The same approach works for AWS S3 (TYPE S3) and Google Cloud Storage (TYPE GCS).

Returning Arrow IPC

If your client understands Apache Arrow, you can return results in Arrow IPC format for efficient data transfer:

import { tableToIPC } from '@ducklings/workers';

// In your request handler:
const table = await conn.queryArrow('SELECT * FROM readings');
const ipcBytes = tableToIPC(table, { format: 'stream' });

return new Response(ipcBytes, {
  headers: {
    'Content-Type': 'application/vnd.apache.arrow.stream',
  },
});

Consumers can read the result with any Arrow-compatible library:

# Python
import pyarrow as pa
import requests

response = requests.get('https://your-worker.dev/arrow')
reader = pa.ipc.open_stream(response.content)
table = reader.read_all()

// JavaScript (Flechette)
import { tableFromIPC } from '@ducklings/workers';

const response = await fetch('https://your-worker.dev/arrow');
const bytes = new Uint8Array(await response.arrayBuffer());
const table = tableFromIPC(bytes);

Deploying

With everything in place, deploy to Cloudflare:

# Local development
npm run dev

# Deploy to production
npm run deploy

That's it. Wrangler handles bundling the WASM binary into the Worker.

Test it:

# Check the API
curl https://duckdb-worker.<your-subdomain>.workers.dev/

# Run a query
curl -X POST https://duckdb-worker.<your-subdomain>.workers.dev/query \
  -H 'Content-Type: application/json' \
  -d '{"sql": "SELECT 42 AS answer"}'

Prepared statements

For parameterized queries, use prepared statements to avoid SQL injection:

const stmt = await conn.prepare(
  'SELECT * FROM readings WHERE sensor_id = ? AND temperature > ?'
);
stmt.bindInt32(1, 1);
stmt.bindDouble(2, 22.0);

const results = await stmt.run();
await stmt.close();

Available binding methods: bindBoolean, bindInt32, bindInt64, bindFloat, bindDouble, bindString, bindBlob, bindNull, bindDate, bindTimestamp.

Streaming large results

For large result sets, avoid loading everything into memory:

const stream = await conn.queryStreaming('SELECT * FROM large_table');

for await (const chunk of stream) {
  // Process each chunk individually
  for (let row = 0; row < chunk.rowCount; row++) {
    const id = chunk.getInt32(row, 0);
    const name = chunk.getString(row, 1);
    // ...
  }
}

Limitations

A few things to be aware of:

Memory: Workers have a 128 MB memory limit. DuckDB itself uses a portion of that, so you'll want to keep datasets and query results reasonably sized.
WASM size: The ~9.6 MB gzipped binary requires a paid Workers plan ($5/month). The free tier caps at 3 MB.
No dynamic extension loading: Only Parquet, JSON, and httpfs are available. You can't INSTALL or LOAD other extensions at runtime.
Single-threaded: No parallel query execution. Workers are single-threaded by nature, and the WASM build has threading disabled.
In-memory only: No persistent storage. Each new Worker instance starts fresh. Use httpfs to read external data, or populate tables on init.

Wrapping up

Ducklings makes it possible to run DuckDB at the edge on Cloudflare Workers, something that wasn't feasible before due to the async I/O constraint. The @ducklings/workers package handles the Asyncify plumbing, provides a TypeScript API, and includes httpfs, Parquet, and JSON extensions out of the box.

This opens up some interesting use cases: SQL APIs without a database server, edge data transformations, Parquet-to-JSON converters, analytics endpoints that query directly from object storage, or lightweight ETL pipelines running globally.

The project is open source at github.com/tobilg/ducklings.

References

Using Iceberg Catalogs in the Browser with DuckDB-Wasm

Tobias Müller — Tue, 16 Dec 2025 15:42:18 GMT

With recent updates of DuckDB itself, DuckDB-Wasm and the Iceberg extension it is now possible to query Iceberg catalog directly from the browser, with no backends. involved.

Example clients that work:

💡

Be aware the credentials you’re using should not be shared. The below examples are for demo purposes, and not for public or production usage. It’s possible to set this up in a safe manner though, for example by granting a fine-grained AWS IAM role to users of a frontend application when they log in via Cognito. This will grant the users temporary permissions to use the respective services.

Disable the built-in http client in DuckDB-Wasm

To remotely querying Iceberg data, we first need to disable the internal http client of DuckDB-Wasm, and then use the official httpfs extension build:

-- Load the httpfs extension, this disables the built-in http client
LOAD http;

Load the Iceberg extension

You can then load the Iceberg extension:

-- Load Iceberg extension
LOAD iceberg;

Query Iceberg data in S3 Tables

I published another blog post “Query S3 Tables with DuckDB” in Q1/2025 which outlines the basic setup of a S3 Tables buckets, as well as a Namespace and an actual Table, please refer to it when trying to set this up in your environment.

So, if you’d like to attach an S3 Tables bucket as a database to DuckDB-Wasm, you can do this like the following:

ATTACH 'arn:aws:s3tables:us-east-1:12345678912:bucket/duckdb-test'
  AS test_db (
    TYPE iceberg,
    ENDPOINT_TYPE s3_tables
);

The attach target is the ARN of the S3 Table bucket.

You then can check which tables are available:

SHOW ALL TABLES;

┌──────────────┬─────────┬─────────────┬─────────────────────────────────────────────┬─────────────────────────┬───────────┐
│   database   │ schema  │    name     │                column_names                 │      column_types       │ temporary │
│   varchar    │ varchar │   varchar   │                  varchar[]                  │        varchar[]        │  boolean  │
├──────────────┼─────────┼─────────────┼─────────────────────────────────────────────┼─────────────────────────┼───────────┤
│ test_db      │ test    │ daily_sales │ [sale_date, product_category, sales_amount] │ [DATE, VARCHAR, DOUBLE] │ false     │
└──────────────┴─────────┴─────────────┴─────────────────────────────────────────────┴─────────────────────────┴───────────┘

To query the table, just use normal SQL statements like you would with any other local table:

SELECT * FROM test_db.test.daily_sales;
┌────────────┬──────────────────┬──────────────┐
│ sale_date  │ product_category │ sales_amount │
│    date    │     varchar      │    double    │
├────────────┼──────────────────┼──────────────┤
│ 2024-01-15 │ Laptop           │        900.0 │
│ 2024-01-15 │ Monitor          │        250.0 │
│ 2024-01-16 │ Laptop           │       1350.0 │
│ 2024-02-01 │ Monitor          │        300.0 │
│ 2024-02-01 │ Keyboard         │         60.0 │
│ 2024-02-02 │ Mouse            │         25.0 │
│ 2024-02-02 │ Laptop           │       1050.0 │
│ 2024-02-03 │ Laptop           │       1200.0 │
│ 2024-02-03 │ Monitor          │        375.0 │
└────────────┴──────────────────┴──────────────┘

Query Iceberg data in R2 Data Catalog / R2

There are a few steps to set up an R2 bucket for the usage as an Iceberg catalog.

Creating a R2 bucket

To create a R2 bucket, go to you account’s Cloudflare Dashboard, and in the left sidebar click on “Storage & databases” → “R2 Object Storage” and then on “Overview”:

After that, click on “Create bucket” in the top right corner. You’ll see the next screen below:

Setting a CORS policy

To set a CORS policy which is necessary to allow specific origin requests from the browser, you need to go to the “CORS” Policy” section and click on “Add”. Then, you can copy & paste the CORS policy below and click on “Save”.

💡

Make sure to add your frontend domain to the AllowedOrigins, otherwise this will NOT work.

[
  {
    "AllowedOrigins": [
      "https://sql-workbench.com",
      "https://embedded.sql-workbench.com",
      "https://terminal.sql-workbench.com",
      "https://shell.duckdb.org"
    ],
    "AllowedMethods": [
      "GET",
      "HEAD",
      "PUT",
      "POST"
    ],
    "AllowedHeaders": [
      "Authorization",
      "Cache-Control",
      "Content-Range",
      "Content-Type",
      "Content-Length",
      "Range",
      "X-Amz-Acl",
      "X-Amz-Content-Sha256",
      "X-Amz-Date",
      "X-Amz-Security-Token",
      "X-Iceberg-Access-Delegation",
      "X-Host-Override"
    ],
    "ExposeHeaders": [
      "Content-Type",
      "Access-Control-Allow-Origin",
      "ETag"
    ]
  }
]

Enabling R2 Data Catalog

Now, enable the R2 Data Catalog in your R2 bucket’s settings:

After clicking on “Enable”, it takes a few seconds, and you should see your Catalog URI and your Warehouse Name. Those are both needed when querying the data later.

Creating a Account Token for R2 / R2 Data Catalog access

To access the data via queries, DuckDB needs an Account API Token. In the left sidebar menu, click on “Manage account” → “Account API Tokens”

Then, on the next screen, click on “Create Token” and on the following screen on “Get started” in the Custom Token section.

Next, enter a name for your token, and create the two Permission entries as outlined below:

Click on “Continue to summary” and then create your access token

💡

Save the newly created Account API Token somewhere safe, you will not be able to retrieve it again. Only a rotation or new creation is possible in case you loose the token afterwards.

Querying data

You now have all the pieces you need for querying the data from your R2 Data Catalog.

As the R2 Data Catalog doesn’t send CORS headers, you need to use our cors.sqlqry.run CORS proxy, which provides a /iceberg route that needs to be postfixed with the R2 Data Catalog base URL concatenated with your Cloudflare Account ID and the name of the R2 bucket your created.

Create a DuckDB secret for the R2 Data Catalog as shown belog. All requests will subsequently use this secret to authorize themselves:

CREATE OR REPLACE SECRET r2secret (
    TYPE iceberg, 
    ENDPOINT 'https://cors.sqlqry.run/iceberg/catalog.cloudflarestorage.com/YOUR_CLOUDFLARE_ACCOUNT_ID/YOUR_BUCKET_NAME', 
    TOKEN 'YOUR_ACCOUNT_API_TOKEN'
);

Then, attach the remote Iceberg catalog as database by supplying the Warehouse Name from the R2 Data Catalog settings:

ATTACH IF NOT EXISTS 'YOUR_WAREHOUSE_NAME' AS r2lake (TYPE iceberg);

If you haven’t created a schema (namespace) yet, you can do this like this with plain SQL:

CREATE SCHEMA IF NOT EXISTS. r2lake.test;

Create an example table:

CREATE TABLE IF NOT EXISTS r2lake.test.sales (
    sale_id INT,
    product_id INT,
    quantity INT,
    price DECIMAL(10, 2),
    sale_date DATE
);

Insert data:

INSERT INTO r2lake.test.sales 
    (sale_id, product_id, quantity, price, sale_date) 
VALUES
    (1, 1, 10, 100.00, '2021-01-01'),
    (2, 2, 20, 234.45, '2021-01-02'),
    (3, 3, 30, 30.99, '2021-01-03');

Query the data:

SELECT * FROM r2lake.test.sales;

Summary

Thanks to the innovations of the DuckDB team, it’s now possible to directly query Iceberg catalogs from the browser. The use cases for this may be in-browser SQL clients for example, or demo applications that demonstrate how Iceberg itself works.

TypeScript scripts as DuckDB Table Functions

Tobias Müller — Wed, 10 Dec 2025 17:07:28 GMT

What if you could query any REST API, GraphQL endpoint, or web page directly from DuckDB using SQL? No ETL pipelines, no intermediate files, no complex setup - just write a TypeScript script and use it as a table function.

In this post, I'll show you how to combine DuckDB's shellfs and arrow extensions with Bun's zero-dependency scripts to create a powerful, flexible data pipeline that lets you query the world with SQL.

This post was inspired by Python Scripts as DuckDB Table Functions and Self-Contained TypeScript Programs Using Bun.

The “Magic”

The setup is surprisingly simple. DuckDB's shellfs extension can read data from shell command output, and the arrow extension can parse Apache Arrow IPC format. Combine this with Bun's ability to run TypeScript files directly (with automatic dependency installation), and you get a powerful pattern:

-- One-time setup in DuckDB
INSTALL shellfs FROM community;
INSTALL arrow FROM community;
LOAD shellfs;
LOAD arrow;

-- Create a macro to run Bun scripts as table functions
CREATE OR REPLACE MACRO bun(script, args := '') AS TABLE
SELECT * FROM read_arrow('bun ' || script || ' ' || args || ' |');

Now any TypeScript script that outputs Arrow IPC data becomes a queryable table:

SELECT * FROM bun('my-script.ts');
SELECT * FROM bun('fetch-data.ts', '--limit=100');

The Secret Sauce: json-to-arrow-ipc

The key ingredient is the json-to-arrow-ipc npm package. This package converts JSON to Arrow IPC format without dictionary encoding, making it fully compatible with DuckDB.

A minimal script looks like this:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const data = [{ name: 'Alice', age: 30 }, { name: 'Bob', age: 25 }];
process.stdout.write(jsonToArrowIPC(data));

Real-World Examples

Let's look at some practical examples that demonstrate the power of this approach.

Example 1: Query GitHub Repositories

Want to analyze the most popular TypeScript repositories on GitHub? Create github-repos.ts:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const language = args.find(a => a.startsWith('--language='))?.split('=')[1] ?? 'typescript';
const perPage = parseInt(args.find(a => a.startsWith('--per-page='))?.split('=')[1] ?? '30');

const response = await fetch(
  `https://api.github.com/search/repositories?q=language:${language}&sort=stars&order=desc&per_page=${perPage}`,
  {
    headers: {
      'Accept': 'application/vnd.github.v3+json',
      'User-Agent': 'duckdb-bun-script'
    }
  }
);

const { items } = await response.json();

const repos = items.map((repo: any) => ({
  name: repo.name,
  full_name: repo.full_name,
  description: repo.description,
  stars: repo.stargazers_count,
  forks: repo.forks_count,
  open_issues: repo.open_issues_count,
  language: repo.language,
  created_at: repo.created_at,
  updated_at: repo.updated_at,
  homepage: repo.homepage,
  topics: JSON.stringify(repo.topics),
}));

process.stdout.write(jsonToArrowIPC(repos));

Now query it with SQL:

-- Top 10 TypeScript repos by stars
SELECT name, stars, forks, description
FROM bun('github-repos.ts', '--language=typescript --per-page=100')
ORDER BY stars DESC
LIMIT 10;

-- Compare languages
SELECT
  'TypeScript' as language, stars, name
FROM bun('github-repos.ts', '--language=typescript --per-page=10')
UNION ALL
SELECT
  'Rust' as language, stars, name
FROM bun('github-repos.ts', '--language=rust --per-page=10')
ORDER BY stars DESC;

Example 2: Fetch Blog Posts from Hashnode GraphQL API

Hashnode provides a GraphQL API that's perfect for this pattern. Create hashnode-posts.ts:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const host = args.find(a => a.startsWith('--host='))?.split('=')[1] ?? 'tobilg.com';

const query = `
  query GetPublicationPosts($host: String!, $first: Int!) {
    publication(host: $host) {
      posts(first: $first) {
        edges {
          node {
            title
            slug
            brief
            publishedAt
            views
            reactionCount
            responseCount
            readTimeInMinutes
            tags {
              name
            }
            url
            author {
              username
            }
          }
        }
      }
    }
  }
`;

const response = await fetch('https://gql.hashnode.com/', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({
    query,
    variables: { host, first: 20 }
  }),
});

const { data } = await response.json();

const posts = data?.publication?.posts?.edges?.map((edge: any) => ({
  title: edge.node.title,
  slug: edge.node.slug,
  brief: edge.node.brief,
  published_at: edge.node.publishedAt,
  views: edge.node.views ?? 0,
  reactions: edge.node.reactionCount ?? 0,
  responses: edge.node.responseCount ?? 0,
  read_time_minutes: edge.node.readTimeInMinutes,
  tags: JSON.stringify(edge.node.tags?.map((t: any) => t.name) ?? []),
  url: edge.node.url,
  author: edge.node.author?.username ?? null,
})) ?? [];

process.stdout.write(jsonToArrowIPC(posts));

Query blog analytics with SQL:

-- All posts sorted by views
SELECT title, views, reactions, read_time_minutes, published_at
FROM bun('hashnode-posts.ts', '--host=tobilg.com')
ORDER BY views DESC;

-- Posts with most engagement
SELECT
  title,
  views,
  reactions,
  responses,
  (reactions + responses) as total_engagement
FROM bun('hashnode-posts.ts', '--host=tobilg.com')
ORDER BY total_engagement DESC
LIMIT 5;

-- Average read time
SELECT
  AVG(read_time_minutes) as avg_read_time,
  SUM(views) as total_views,
  SUM(reactions) as total_reactions
FROM bun('hashnode-posts.ts', '--host=tobilg.com');

Example 3: GitHub User Activity

Create github-user-events.ts to analyze recent GitHub activity:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const username = args.find(a => a.startsWith('--user='))?.split('=')[1] ?? 'tobilg';

const response = await fetch(
  `https://api.github.com/users/${username}/events/public?per_page=100`,
  {
    headers: {
      'Accept': 'application/vnd.github.v3+json',
      'User-Agent': 'duckdb-bun-script'
    }
  }
);

const events = await response.json();

const processed = events.map((event: any) => ({
  id: event.id,
  type: event.type,
  repo: event.repo?.name,
  created_at: event.created_at,
  action: event.payload?.action ?? null,
  ref: event.payload?.ref ?? null,
  ref_type: event.payload?.ref_type ?? null,
}));

process.stdout.write(jsonToArrowIPC(processed));

Analyze activity patterns:

-- Event type breakdown
SELECT type, COUNT(*) as count
FROM bun('github-user-events.ts', '--user=tobilg')
GROUP BY type
ORDER BY count DESC;

-- Most active repositories
SELECT repo, COUNT(*) as events
FROM bun('github-user-events.ts', '--user=tobilg')
GROUP BY repo
ORDER BY events DESC
LIMIT 10;

-- Activity by day of week
SELECT
  strftime(created_at::timestamp, '%A') as day_of_week,
  COUNT(*) as events
FROM bun('github-user-events.ts', '--user=tobilg')
GROUP BY day_of_week
ORDER BY events DESC;

Example 4: NPM Package Statistics

Create npm-package.ts to fetch package metadata:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const packageName = args.find(a => a.startsWith('--package='))?.split('=')[1] ?? 'duckdb';

const [registryResponse] = await Promise.all([
  fetch(`https://registry.npmjs.org/${packageName}`)
]);

const registry = await registryResponse.json();

const versions = Object.entries(registry.time || {})
  .filter(([key]) => key !== 'created' && key !== 'modified')
  .map(([version, time]) => ({
    package: packageName,
    version,
    published_at: time as string,
    description: registry.description,
    license: registry.license,
    homepage: registry.homepage,
  }));

process.stdout.write(jsonToArrowIPC(versions));

Create npm-downloads.ts for download stats:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const packageName = args.find(a => a.startsWith('--package='))?.split('=')[1] ?? 'duckdb';

const response = await fetch(
  `https://api.npmjs.org/downloads/range/last-month/${packageName}`
);

const data = await response.json();

const downloads = data.downloads?.map((d: any) => ({
  package: packageName,
  date: d.day,
  downloads: d.downloads,
})) ?? [];

process.stdout.write(jsonToArrowIPC(downloads));

Query package analytics:

-- Version history
SELECT version, published_at
FROM bun('npm-package.ts', '--package=apache-arrow')
ORDER BY published_at DESC
LIMIT 10;

-- Download trends
SELECT
  date,
  downloads,
  AVG(downloads) OVER (ORDER BY date ROWS BETWEEN 6 PRECEDING AND CURRENT ROW) as rolling_avg
FROM bun('npm-downloads.ts', '--package=duckdb')
ORDER BY date;

-- Compare package downloads
SELECT 'duckdb' as package, SUM(downloads) as total
FROM bun('npm-downloads.ts', '--package=duckdb')
UNION ALL
SELECT 'better-sqlite3' as package, SUM(downloads) as total
FROM bun('npm-downloads.ts', '--package=better-sqlite3');

Example 5: Hacker News Front Page

Create hn-frontpage.ts using the official Hacker News API:

#!/usr/bin/env bun

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const args = process.argv.slice(2);
const limit = parseInt(args.find(a => a.startsWith('--limit='))?.split('=')[1] ?? '30');

// Fetch top story IDs
const topStoriesResponse = await fetch('https://hacker-news.firebaseio.com/v0/topstories.json');
const topStoryIds: number[] = await topStoriesResponse.json();

// Fetch story details in parallel (limited to avoid rate limiting)
const storyIds = topStoryIds.slice(0, limit);
const stories = await Promise.all(
  storyIds.map(async (id) => {
    const response = await fetch(`https://hacker-news.firebaseio.com/v0/item/${id}.json`);
    return response.json();
  })
);

const processed = stories.map((story: any, index: number) => ({
  rank: index + 1,
  id: story.id,
  title: story.title,
  url: story.url ?? null,
  score: story.score,
  author: story.by,
  comments: story.descendants ?? 0,
  created_at: new Date(story.time * 1000).toISOString(),
  type: story.type,
}));

process.stdout.write(jsonToArrowIPC(processed));

Analyze the front page:

-- Current top stories
SELECT rank, title, score, comments, author
FROM bun('hn-frontpage.ts', '--limit=30')
ORDER BY rank;

-- Stories with high engagement ratio
SELECT
  title,
  score,
  comments,
  ROUND(comments::float / NULLIF(score, 0), 2) as comment_per_point
FROM bun('hn-frontpage.ts', '--limit=50')
WHERE score > 10
ORDER BY comment_per_point DESC
LIMIT 10;

-- Top domains
SELECT
  regexp_extract(url, 'https?://([^/]+)', 1) as domain,
  COUNT(*) as stories,
  AVG(score) as avg_score
FROM bun('hn-frontpage.ts', '--limit=100')
WHERE url IS NOT NULL
GROUP BY domain
ORDER BY stories DESC
LIMIT 10;

Configuration Options

The json-to-arrow-ipc package handles various data scenarios:

import { jsonToArrowIPC } from 'json-to-arrow-ipc';

const ipc = jsonToArrowIPC(data, {
  // Sample rows for schema inference (default: 100)
  schemaSampleSize: 50,

  // Handle schema mismatches: 'error' | 'skip' | 'coerce' (default: 'coerce')
  onSchemaMismatch: 'coerce',

  // Flatten nested objects with dot notation (default: false)
  flattenNestedObjects: true,

  // Serialize arrays as JSON strings (default: true)
  serializeArrays: true,
});

Tips and Best Practices

Handle pagination: For APIs with large datasets, implement pagination in your script and use command-line arguments to control limits.
Error handling: Add try-catch blocks and output empty arrays on error to prevent DuckDB query failures.
Rate limiting: Be mindful of API rate limits. Add delays between requests when fetching many resources.

Conclusion

Any data source with an API becomes a SQL table. Combined with DuckDB's analytical capabilities - window functions, CTEs, JSON extraction, and more - you have a lightweight but powerful tool for ad-hoc data analysis.

The combination of Bun's TypeScript execution, automatic dependency management, and DuckDB's extensibility creates a developer experience that's both powerful and simple.

The json-to-arrow-ipc package and example scripts are available at github.com/tobilg/duckdb-bun-scripts.

Use your favorite AI tool to read the lastest AWS News

Tobias Müller — Tue, 14 Oct 2025 17:16:50 GMT

You can use the unofficial AWS News MCP Server, which combines the news articles, blog posts and updates of more than 40 different AWS Newsfeeds to a single source.

Connecting to the AWS News MCP Server

You can use different protocols to access it:

https://awsnews.remotemcp.directory/mcp (Streaming HTTP, recommended)
https://awsnews.remotemcp.directory/sse (SSE, deprecated)

The MCP Server is available without authentication, free to use.

Available tools

Tool	Description	Parameters
countNews	Get the number of AWS news articles	-
getNewsStats	Get the AWS news statistics by date	Number of Days (optional, default: 90)
getNewsByDate	Get the AWS news articles by date	Date (in format YYYY-MM-DD)
getLatestNews	Get the latest AWS news articles	Limit (optional, default: 10)
searchNews	Search the AWS news articles	Query (required), Limit (optional, default: 10), SortBy (optional, allowed values: relevance, date, default: relevance)

Using Claude

To add the AWS News MCP Server to claude.ai, you can follow the connector documentation. Navigate to Settings → Connectors while logged-in, and enter the details as shown below:

Once you did this, you can click on the “Configure” button to allow unsupervised access if you like to, but this isn’t necessary to use the MCP Server itself. Otherwise, you’ll be asked if Claude is allowed to access it each time you use it.

You can then just create a new chat, and ask a question like

Show the latest 5 AWS news

and the answer from Claude will look like this (after you eventually had to confirm the MCP tool usage:

You can also ask about specific AWS services:

What are the latest 3 AWS news articles about EKS?

Or, ask about AWS News statistics:

Give me the AWS News stats for October 2025

You can even use Claude to analyze the data:

What were the top 5 services of the AWS news of October 13th 2025?

Using Claude Code

To add the AWS News MCP Server to Claude Code, you can check the documentation on how to add remote MCP Servers. It supports different installation scopes:

# Local scope (default)
claude mcp add --transport http aws-news https://awsnews.remotemcp.directory/mcp

# Project scope
claude mcp add --transport http aws-news --scope project https://awsnews.remotemcp.directory/mcp

# User scope
claude mcp add --transport http aws-news --scope user https://awsnews.remotemcp.directory/mcp

Using Cursor

To add the AWS News MCP Server to Cursor, you can just click on this link that will add it automatically, or you can install it manually via a mcp.json file, which you can place either in a project or or a user’s home directory:

{
  "mcpServers": {
    "aws-news": {
      "url": "https://awsnews.remotemcp.directory/mcp"
    }
  }
}

Using VS Code & GitHub Copilot

To add the AWS News MCP Server to VS Code for the usage with GitHub Copilot, you can install it manually via a mcp.json file, which supports different locations and scopes (e.g. workspace or user):

{
  "servers": {
    "aws-news": {
      "url": "https://awsnews.remotemcp.directory/mcp",
      "type": "http"
    }
  }
}

You can than use the MCP Server in the Agent Mode as described in the docs.

Using Amazon Q Developer

To add the AWS News MCP Server to Amazon Q Developer, you can follow the documentation and add a mcp.json file like this:

{
  "mcpServers": {
    "aws-news": {
      "url": "https://awsnews.remotemcp.directory/mcp",
      "type": "http"
    }
  }
}

After that, use the /mcp command to activate the AWS News MCP server.

Using Cloudflare AI Playground

The AI Playground is a free UI (e.g. if you don’t have a AI subscription) that allows users to explore different models and integrations like MCP Servers. Just head to https://playground.ai.cloudflare.com/ and configure the MCP server:

Then, click on the “Connect” button and you should see the following output with the available tools, and some debug output:

You then can ask questions in the main chat window like

Show the latest 3 AWS News articles

Summary

With the freely available remote AWS News MCP Server, it’s possible to access and analyze the latest AWS News articles by your favorite AI tools, such as Claude, Cursor, GitHub Copilot and others.

Using Amazon SageMaker Lakehouse with DuckDB

Tobias Müller — Sun, 08 Jun 2025 17:48:00 GMT

Preconditions

To use the Amazon SageMaker Lakehouse with DuckDB, you first have to create a S3 Table bucket, a namespace and an actual S3 Table. All those steps are described in my other blog post “Query S3 Tables with DuckDB”, so please make sure you followed the outlined (manual) steps of the Setting up a S3 Table section before continuing with this blog post.

The setup of the pretty complicated permissions on the AWS side is implemented according to their blog post “Access data in Amazon S3 Tables using PyIceberg through the AWS Glue Iceberg REST endpoint”

Create IAM role and policy

💡

As outlined in the previous blog post, you already created a S3 Table bucket called “duckdb-test” and a namespace “test“. In the following examples, I will not use my real AWS Account ID, but the artificial “123456789012” instead. Please keep in mind to use your actual AWS Account ID if you follow to implement this in your account.

Check your ARNs

Before creating your IAM policy, you need to check the ARNs of your S3 Table bucket, and also note the namespace.

So, if your S3 Table bucket lives in us-east-1, go to the AWS Console and check the ARN for it:

In our case it’s arn:aws:glue:us-east-1:123456789012:catalog/s3tablescatalog/duckdb-test and the namespace we created as of the referenced blog post is test.

Create the policy

Now, go the the IAM Policies screen of the AWS Console, and click on the Create policy button. Click on the JSON button, and paste the following JSON policy into the editor.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SagemakerLakehouseTest",
            "Effect": "Allow",
            "Action": [
                "glue:GetCatalog",
                "glue:GetDatabase",
                "glue:GetDatabases",
                "glue:GetTable",
                "glue:GetTables",
                "glue:CreateTable",
                "glue:UpdateTable"
            ],
            "Resource": [
                "arn:aws:glue:us-east-1:123456789012:catalog",
                "arn:aws:glue:us-east-1:123456789012:catalog/s3tablescatalog",
                "arn:aws:glue:us-east-1:123456789012:catalog/s3tablescatalog/duckdb-test",
                "arn:aws:glue:us-east-1:123456789012:table/s3tablescatalog/duckdb-test/test/*",
                "arn:aws:glue:us-east-1:123456789012:database/s3tablescatalog/duckdb-test/test"
            ]
        },
        {
            "Effect": "Allow",
            "Action": [
                "lakeformation:GetDataAccess"
            ],
            "Resource": "*"
        }
    ]
}

Then update your AWS Account ID, the S3 Table bucket name and the namespace accordingly if you chose other values for them.

Then press Next, and use s3tables-duckdb-test as policy name and save the policy with Save policy.

Create the role

Go to the IAM Roles screen of the AWS Console, and click on the Create role button.

💡

This will require an existing IAM role or IAM user. Normally, this will be a role you’re assuming via STS, or a user whose credentials you stored on the machine you’re using. The setup of this is not part of this blog post.

Now click on Next. Select your previously created policy s3tables-duckdb-test as trust policy by clicking the checkbox in front of the policy and clicking Next:

Enter the role name s3tables-duckdb-test-role and click on Create role.

Define access control using Lake Formation

Application integration setup

In Lake Formation, enable full table access for external engines to access data. This allows third-party applications to get the Lake Formation temporary credential using an IAM role(s) that has full permissions (ALL) on the requested table.

Go to the AWS Lake Formation Dashboard, and then on the Left Pane, expand the Administration section, then Application integration settings and choose Allow external engines to access data in Amazon S3 locations with full table access and click on Save:

Set up Lake Formation grants

For the newly created role s3tables-duckdb-test-role to be able to actually access the database and the tables, you need to provide database and table level permissions.

Database-level permissions

In the Lake Formation Data permissions, click on Grant. Then, in the Principals section, choose the radio button IAM users and roles, and from the drop-down choose s3tables-duckdb-test-role. In the LF-Tags or catalog resources section, choose Named Data Catalog resources:

Choose 123456789012:s3tablescatalog/duckdb-test for Catalog
Choose test for Databases
Choose CREATE TABLE, DESCRIBE for database permissions
Click on Grant

Table-level permissions

Choose 123456789012:s3tablescatalog/duckdb-test for Catalog
Choose test for Databases
Choose All Tables for tables
Choose SUPER for table permissions.
Click on Grant.

Quick check in Athena

We verify in Athena that the table daily_sales exists, and that we have data, before going to the DuckDB CLI to test the catalog integration:

Validate IAM role

You can validate the created IAM role on the target machine by issuing the following command (this assumes you have installed the AWS CLI):

aws sts assume-role --role-arn "arn:aws:iam::123456789012:role/s3tables-duckdb-test-role" --role-session-name s3tables-duckdb-test-role

Usage with DuckDB

Create DuckDB secret for catalog access

CREATE SECRET (
    TYPE s3,
    PROVIDER credential_chain,
    CHAIN sts,
    ASSUME_ROLE_ARN 'arn:aws:iam::123456789012:role/s3tables-duckdb-test-role',
    REGION 'us-east-1'
);

Attach the Lake Formation (Glue) catalog

ATTACH '123456789012:s3tablescatalog/duckdb-test' AS s3tables_datalake (
    TYPE ICEBERG,
    ENDPOINT_TYPE 'GLUE'
);

List tables

SHOW ALL TABLES;

Query table

 SELECT * FROM s3tables_datalake.test.daily_sales LIMIT 5;

CLI screenshot

Summary

We were able to show that the integration with the AWS Sagemaker Catalog (a.k.a. AWS Glue Iceberg REST endpoint) works with DuckDB. Once the quite tedious IAM role/policy, as well as the Lake Formation permission setup are done, the usage is pretty straight forward with simple CREATE SECRET and ATTACH statements.

References

DuckDB docs: Amazon Sagemaker Lakehouse
AWS Blog Post: Access data in Amazon S3 Tables using PyIceberg through the AWS Glue Iceberg REST endpoint
Blog post: Query S3 Tables with DuckDB

Welcome to the age of $10/month Lakehouses

Tobias Müller — Fri, 30 May 2025 18:22:31 GMT

Recap: Data Warehouses, Data Lakes, Lakehouses?

As a short recap, what do these mean, and how are they differentiated?

Modern Data Warehouses, like Amazon Redshift, Google BigQuery, and Snowflake, offer fast, SQL-optimized performance for structured data and BI workloads. Their columnar storage, advanced indexing, and automatic optimization make them ideal for analytics. However, they come at a premium: storage and compute are tightly coupled, and costs can rise sharply with large-scale, always-on workloads or frequent queries over massive datasets.

Data Lakes, typically built on Amazon S3, Azure Data Lake Storage (ADLS), Google Cloud Storage (GCS), or Cloudflare R2, provide a low-cost solution for storing raw structured and unstructured data. They scale effortlessly and are the foundation of many batch and streaming pipelines. While storage is cheap, performance suffers without significant investment in tooling, such as Apache Spark, Presto, or Hive, to support transformation and query execution. Data governance and consistency also become challenges as systems scale.

The Lakehouse architecture, popularized by platforms like Databricks or Snowflake with table formats like Delta Lake and Apache Iceberg, offers a compelling middle ground. Lakehouses combine the low-cost, schema-flexible storage of data lakes with the performance and ACID transaction support of warehouses. Technologies like Databricks Lakehouse Platform, Snowflake’s Unistore, or Dremio provide unified data management, allowing teams to run SQL analytics, data science, and streaming workloads directly on cloud object storage, without needing separate ETL into a warehouse.

From a cost-performance perspective, Lakehouses reduce redundancy and simplify infrastructure by eliminating the need to maintain both a lake and a warehouse. They support open formats (e.g., Parquet, ORC) and bring advanced features like time travel (see Apache Iceberg), schema evolution, and fine-grained access control which previously was reserved for traditional Data Warehouses into the Data Lake domain.

Ultimately, the choice depends on your workload characteristics, query latency requirements, data volume, and ecosystem alignment. But as the major cloud providers and vendors converge around the Lakehouse paradigm, it's becoming an increasingly attractive option for teams seeking agility, scalability, and cost control in modern data platforms.

Existing open table formats

There are different “table formats” for the Lakehouse architecture, such as Apache Iceberg and Delta Lake.

Apache Iceberg is an open table format designed for huge analytic datasets in distributed data processing systems like Apache Spark, Trino, Presto, Flink, and Hive. It was developed at Netflix and later donated to the Apache Software Foundation.

Delta Lake is an open-source storage layer that brings ACID transactions, scalable metadata handling, and unified streaming and batch processing to Apache Spark and other big data engines. Originally developed by Databricks, it is now part of the Linux Foundation.

💡

Because I’m a bit lazy sometimes, I asked ChatGPT the following: “You are an expert data engineer. Can you write a comparison summary of open table formats of both Apache Iceberg and Delta Lake? Use the same criteria while doing so”. The shortened output is found below. I tried to verify the claims as far as I could. If you find an error, please leave a comment.

Comparison

ACID Transactions

Iceberg:
Fully supports ACID transactions using snapshot isolation. Transactions are handled via metadata manifests and atomic commits to the metadata file. Suitable for both streaming and batch operations.
Delta Lake:
Implements ACID transactions via a write-ahead log (_delta_log). Also supports snapshot isolation, enabling reliable concurrent read/write operations in batch and streaming.

Schema Evolution

Iceberg:
Offers flexible schema evolution and supports adding, renaming, reordering, and deleting columns. Field IDs are used to track schema changes safely.
Delta Lake:
Supports additive schema evolution (e.g., adding columns). Reordering and renaming are partially supported and can be more restrictive than Iceberg. Dropping columns my require rewriting of the underlying Parquet files.

Partitioning

Iceberg:
Uses hidden partitioning, abstracting physical layout from logical queries. Automatically rewrites queries for efficient partition pruning.
Delta Lake:
Relies on explicit partition columns, which are visible to users. Partition pruning depends on query awareness of partition structure.

Time Travel

Iceberg:
Native time travel support via snapshots. You can query historical versions based on snapshot ID or timestamp.
Delta Lake:
Also supports time travel using version numbers or timestamps. Retains transaction logs that allow rollback and historical queries.

Performance and Metadata Scalability

Iceberg:
Uses a metadata tree structure (manifests and manifest lists) to avoid file listing and support petabyte-scale datasets. More scalable with large numbers of files and partitions.
Delta Lake:
Stores metadata in a log of JSON files, which can become large over time and may require compaction (OPTIMIZE and VACUUM) for performance.

Streaming Support

Iceberg:
Has native support for streaming writes and reads, especially with Apache Flink and Spark Structured Streaming. Streaming write support is still evolving in some engines.
Delta Lake:
Designed with stream-batch unification in mind. Seamless support for Structured Streaming in Spark.

File Format Support

Iceberg:
Supports Parquet, Avro, and ORC. File format agnostic, with clean decoupling of metadata and data.
Delta Lake:
Primarily based on Parquet. No support for other formats as of now.

Ecosystem and Engine Support

Iceberg:
Integrates with Apache Spark, Flink, Trino, Presto, Hive, Dremio, Snowflake, and more. Supported by AWS Glue, Snowflake, and others.
Delta Lake:
Strong integration with Databricks, Apache Spark, and supported in Presto, Trino, Flink (via Delta-RS). Emerging ecosystem via Delta Kernel and Delta-RS.

Operations and Tooling

Iceberg:
Growing support for table maintenance (compaction, expiration, snapshot management). Still evolving CLI tools.
Delta Lake:
Mature operational tooling via Databricks (e.g., OPTIMIZE, VACUUM, Delta Live Tables). Better UX for managed environments.

Summary

Criteria	Apache Iceberg	Delta Lake
ACID Transactions	✅ Yes (Snapshot-based)	✅ Yes (Log-based)
Schema Evolution	✅ Flexible (Field IDs, reorder, rename)	⚠️ Limited (mostly additive)
Partitioning	✅ Hidden, automatic pruning	⚠️ Explicit, user-managed
Time Travel	✅ Snapshots, timestamps	✅ Versioned log, timestamps
Metadata Performance	✅ Scalable (manifest-based)	⚠️ Requires periodic compaction
Streaming Support	⚠️ Evolving (good Flink/Spark support)	✅ Mature (best with Spark)
File Format Support	✅ Parquet, Avro, ORC	⚠️ Parquet only
Engine/Ecosystem Support	✅ Broad, vendor-neutral	⚠️ Strong Spark/Databricks focus
Operational Tooling	⚠️ Growing	✅ Advanced in Databricks

The new kid on the block: DuckLake

DuckLake was introduced on 2025-05-27 by the by the founders of DuckDB. There’s an interesting podcast where Mark & Hannes explain more about the motivation and goals behind it, viewing it is strongly recommended! Also, they wrote a manifesto: SQL as a Lakehouse format.

Its key features are:

Supports snapshots, time travel queries, schema evolution and partitioning
Can have as many lightweight snapshots as you want without frequent compacting steps
Allows concurrent access with ACID transactional guarantees over multi-table operations
Uses statistics for filter pushdown, enabling fast queries even on large datasets

There’s also a nice introduction and overview in Jordan Tigani’s blog post in the MotherDuck blog: “A Duck walks into a lake”

The big difference compared to Apache Iceberg and Delta Lake is that DuckLake uses a database as a backing store for the metadata, instead of keeping it in complex files as JSON and/or Avro on object storage, like both others do. Only to eventually have a data catalog put on top of that.

DuckLake stores data in Parquet, and metadata in relational tables, plain and simple. This has many benefits, e.g. faster lookups (SQL query vs. an eventual cascade of S3 HEAD/GET requests), and the catalog is already built-in.

The catalog database should be chosen upon the following criteria (according to the DuckLake website):

If you would like to perform local data warehousing with a single client, use DuckDB as the catalog database
If you would like to perform local data warehousing using multiple local clients, use SQLite as the catalog database
If you would like to operate a multi-user Lakehouse with potentially remote clients, choose a transactional client-server database system as the catalog database: MySQL or PostgreSQL

Even though it’s a very early version, which still misses some functionality, it’s already looking very useful from a (metadata) complexity and performance perspective.

The necessary relational database for multi-user scenarios adds some complexity, but there are some SaaS solutions on the market that reduce or actually get rid of the operative burden of managing relational databases. The leading principle still is the separation of storage and compute. More on that in the next chapter.

Is Big Data dead?

Coming back to the article’s title, where’s the promised $10/month Lakehouse at? And why $10/month?

In his article “Big Data is dead”, Jordan Tigani shared some interesting insights from his days at Google’s BigQuery. One was that the 90th percentile of all queries only queried 100MB data, and the 99th no more than 10GB.

Another one was that most people don’t actually have that much data, but some have real Big Data. “Among customers who were using the service heavily, the median data storage size was much less than 100 GB”:

Amazon released a dataset about the Redshift usage and query sizes, which Jordan analyzed in another blog post that supports his original article.

So, what’ the relation to this blog post?

Basically, it validates that most of the people/projects/companies do neither have “real” Big Data, nor run queries that require actual “real” Big Data tools that are distributed, horizontally scaling etc. etc.

Many of them though have established systems that are much too complex and/or much too expensive for their actual need.

The $10/month Lakehouse

Requirements

If we apply the “separation of storage and compute” paradigm, we need to find a supplier for the (object) storage part, and one for the compute. For the storage, we want to use a S3-compatible API for flexibility reasons.

As we want to leverage the new DuckLake format, and we want to eventually have multiple reader and writer processes. This means we do not “only” have to search for a compute provider, but also find a provider for a relational database for the DuckLake metadata, which enables the desired multi-user capabilities.

Furthermore, we’d like to spend as little time on managing actual infrastructure as possible. This leads us to using serverless services for both compute/querying, and metadata storage.

Going forward, we assume the following:

We have 250GB of data we want to analyze (as monthly average)
We would like to query the remote data via HTTPs endpoints, as well as from out local machines
We want the whole Lakehouse to run in “auto-pilot mode”, meaning we don’t want to manage the underlying infrastructure
We want SQL to be our primary/sole interface to the data, so we want to use the new DuckLake, based on DuckDB

A suggested solution

Storage

For storage, there are many providers on the market, be it the big public cloud providers like AWS, Azure, Cloudflare or Google Cloud, or others, like Hetzner, or, more specialized, Backblaze.

If we want to optimize on costs, Cloudflare R2 comes into focus, mainly because the free egress fees, a generous free tier, and generally cheap standard storage costs of $0.015 / GB-month.

R2 Free Tier

Class A operations are basically write or list operations, Class B operations are mainly read operations, that will occur much more often as write operations in a Lakehouse setting.

R2 Pricing after Free Tier

The calculated cost per month, based on our requirements for the storage part would be the following:

250 GB-month - 10 GB-month (free tier) * $0.015 = $3.60
Write requests: We assume that the 1 million requests / month are sufficient, $0
Read requests: We assume that the 10 million requests / month are sufficient, $0
Egress is free, $0

Overall costs for storage for a month: $3.60

Compute

When wanting to run DuckDB in a serverless fashion, some options come into one’s mind. E.g. we could use AWS services such as Fargate, running DuckDB in a container, or AWS Lambda, running it in a serverless function. Google Cloud has CloudRun, Azure has Container Instances.

Cloudflare will launch it’s Containers service in June 2025, which has a pricing model of $0.072 per vCPU-hour, and $0.009 per GB-hour. This is not necessarily cheaper than the other providers, but it comes with 1TB / month free egress from the containers. Because we’ll already be using R2, we’ll use it anyways, despite of being able to safe maybe $1 with other providers, based on our assumed usage (see below).

The monthly costs, based on some assumptions (containers use 4GB and 2 vCPUs when run, meaning $0.18 / hour / container), would be:

25.000 queries, with an average of 3s runtime (=): $0.18 (25.000 × 3 / 3600) = $3.75
Container will spin down after 1 minute of inactivity, so we add another $2 to accommodate this behavior

Overall costs for compute for a month: $5.75

Metadata storage

Because we’d like to use DuckLake in multi-user/writer mode, we’ll need either a MySQL or Postgres compatible database to store the metadata. There are many options on the market, like Supabase or Neon, which both provide serverless Postgres services.

We’ll choose Neon, because it has a more generous free tier, that offers up to 0.5 GB of database storage, and auto-scaling of up to 2 vCPUs and 8 GB of memory, having 190 compute-hours included. This should also fit to our assumed compute usage as outlined above:

Overall costs for metadata storage for a month: $0

Cumulative costs

From the costs we derived for each part of the stack, we can calculate the cumulative costs of the stack per month: storage $3.60, compute $5.75, and metadata storage $0. This means overall monthly costs of $9.35!

Deploy a Serverless DuckLake

💡

The Cloudflare Containers service is currently in closed beta, but I got access a few weeks ago. It will GA in June 2025, as stated in the introduction article.

GitHub repo

The repository with the code can be found at tobilg/cloudflare-ducklake:

Once you cloned the code to your local machine, you can start with the deployment:

git clone git@github.com:tobilg/cloudflare-ducklake.git && cd cloudflare-ducklake

Preconditions

To deploy the project, you need to have the following preconditions to be present on the machine you want to use:

Node 20 or higher
Docker Desktop
wrangler (newest version)
An IDE like VS Code with TypeScript support

Installing dependencies

To install the dependencies, please run

npm i

Build image locally

Before being able to build the Docker image locally, you have to download the DuckDB extensions we'd like to package into the image, so that they don't need to be downloaded on each container start:

scripts/download_extensions.sh

Once this is done, you can run the following to build the image locally:

npm run build:docker

Run the image locally

To run the newly built image locally, run

npm run dev:docker

To query the DuckDB API within the locally running container, use

curl --location 'http://localhost:8080/query' \
--header 'Content-Type: application/json' \
--data '{
  "query": "SELECT * FROM '\''https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet'\'' LIMIT 1000"
}'

💡

Currently it's not possible to use wrangler dev during local development. I guess this will eventually change once Containers become GA.

DuckDB API

DuckDB is exposed as a Hono.js-based API, that offers a few endpoints:

GET /: Will show a JSON welcome message
GET /_health: Enables potential container health checking (currently not used)
POST /query: Takes a application/json object body with a query property that contains the (encoded) SQL query. Returns the query result in application/json as well (see example above)

Securing the API

You can generate a unique API Token, e.g. with a tool like Strong Password Generator, and deploy a Workers secret named API_TOKEN that will automatically be used to secure the /query endpoint once it's present with the below script:

scripts/setup_api_secrets.sh

If you want to test this locally, too, you need to create a .dev.vars file in the root directory of this project. This will be appended for R2 Data Catalog & DuckLake usage later if you want to try these before deployment as well.

Creating a R2 bucket

If you want to either deploy the DuckLake or the Apache Iceberg integration via R2 Data Catalog, you have to set up a R2 bucket first. To create a new R2 bucket, you can run the following:

scripts/setup_r2.sh YOUR-BUCKET-NAME

where YOUR-BUCKET-NAME is your desired name for the bucket. It will automatically add the respective environment variables to .dev.vars. You'll also need the bucket name later for settng up the DuckLake secrets.

If you'd want to set a location hint or a jurisdiction, please edit the script accordingly before running it.

Setting up a catalog database

As described above, we chose Neon‘s free tier to act as the metadata storage provider. Once you initially signed-up, you can choose the project name, cloud provider and the region:

After you clicked on "Create project", you get directly taken to your Dashboard. The next step is to get the connection details. Therefore, click on the "Connect" button in the upper-right corner:

Note the following connection parameters, and their equivalent environment variable/secret names (and add them to the .dev.vars file):

The user (POSTGRES_USER)
The password (POSTGRES_PASSWORD)
The hostname (POSTGRES_HOST)
The database name (POSTGRES_DB)

You can also create another user, and use this if you don't want to use the main instance user.

Getting a R2 Access Token

Please follow the instructions in the R2 docs on how to create an API to ken.

You'll need to store this token in a secure location (and eventually in .dev.vars if you want to use the R2 Data Catalog when running the Docker image locally), as you'll need later for the R2 Data Catalog deployment.

Also, please note the the S3 APIs Access Key (R2_ACCESS_KEY_ID) and Secret Key (R2_SECRET_ACCESS_KEY), as well as your Cloudflare account ID (R2_ACCOUNT_ID), you'll need them in the next step. Put them in .dev.vars as well if you want to run it locally.

Create secrets for DuckLake

Running the following script will create eight new Workers secrets needed for deployment:

scripts/setup_ducklake_secrets.sh

Deployment

If you followed the above steps, you can now run the actual deployment. This will create a Worker, a DurableObject, build the Docker image locally and the upload it to the Cloudflare container registry.

💡

You need access to Cloudflare containers beta to be able to deploy it to Cloudflare!

npm run deploy

Running queries

Replace the WORKERS_URL with the real URL, and the API_TOKEN with your real API token:

curl --location 'https://WORKERS_URL/query' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer API_TOKEN' \
--data '{
  "query": "CREATE TABLE ducklake.orders AS SELECT * FROM '\''https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet'\''"
}'

The response should look like this:

[
  {"Count":"15000"}
]

Get the first row of the new table:

curl --location 'https://WORKERS_URL/query' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer API_TOKEN' \
--data '{
  "query": "SELECT * FROM ducklake.orders LIMIT 1"
}'

The response should look like this:

[
  {
    "o_orderkey": 1,
    "o_custkey": 370,
    "o_orderstatus": "O",
    "o_totalprice": 172799.49,
    "o_orderdate": {
      "days": 9497
    },
    "o_orderpriority": "5-LOW",
    "o_clerk": "Clerk#000000951",
    "o_shippriority": 0,
    "o_comment": "nstructions sleep furiously among "
  }
]

Deployment with the additional R2 Data Catalog / Iceberg integration

With the v1.3.0 release of DuckDB, it became possible to connect to the R2 Data Catalog. This means that you can now also read Iceberg data from R2's Object Storage directly from a SQL statement issued by DuckDB.

Enable R2 Data Catalog for existing R2 bucket

You can enable the Apache Iceberg support via the R2 Data Catalogby running the following command:

scripts/setup_r2_data_catalog.sh YOUR-BUCKET-NAME

where YOUR-BUCKET-NAME is the name for the bucket you used in the R2 bucket creation step before.

Getting the R2 Data Catalog information

The information about the R2 Data Catalog URI (R2_ENDPOINT secret) and warehouse name (R2_CATALOG secret) can be gathered by running:

scripts/get_catalog_info.sh YOUR-BUCKET-NAME

where YOUR-BUCKET-NAME is the name for the bucket you used in the R2 bucket creation step before.

Please also store this information, because you'll need in in the next step. Also, please add the three variables to .dev.vars if you want to use the R2 Data Catalog when running the Docker image locally.

Creating secrets for R2 Data Catalog

To create the necessary Workers secrets, run:

scripts/create_r2_data_catalog_secrets.sh

and copy & paste the respective values you noted in the last two steps.

Writing Iceberg example data

As we need some example data if we want to test the new Iceberg capabilities, we need to create this data manually with a Python script.

For convenience, we'll use Marimo for this. It requires a working Python installation on your machine.

Setup Marimo

To setup Marimo, run the following npm task:

npm run iceberg:setup

This will create a new directory and install Marimo and some dependencies in a virtual environment.

Create the Iceberg example data

To create the Iceberg example data, run the following npm task:

npm run iceberg:create

This will start Marimo, and load the respective Python script. You'll need to edit the variables for WAREHOUSE, CATALOG_URI and TOKEN with the values gathered in the last steps. This is also described in the R2 Data Catalog docs.

After you did that, you can run the Python cells in the Marimo notebook, and should end up with some data created in the R2 Bucket. You can check in the Cloudflare Dashboard, or via wrangler.

Running DuckDB with Iceberg support

Once you created the secrets as outlined above (all of them!), the application will automatically create a DuckDB secret for accessing the R2 Data Catalog, and attach the catalog as well under the name of r2lake when it starts. The example table created in the last step is called people and was created in the default schema.

Deployment

If you followed the above steps, you can now run the deployment. This will update the existing deployment when you already deployed the DuckLake configuration.

Hint: You need access to Cloudflare containers beta to be able to deploy it to Cloudflare!

npm run deploy

The wrangler deployment output shows the workers URL where the service can be reached. Note it down if you want to run test queries (WORKERS_URL in the step below).

Running queries

Replace the WORKERS_URL with the real URL, and the API_TOKEN with your real API token:

curl --location 'https://WORKERS_URL/query' \
--header 'Content-Type: application/json' \
--header 'Authorization: Bearer API_TOKEN' \
--data '{
  "query": "SELECT * FROM r2lake.default.people"
}'

This should return the response below:

[
  {
    "id": "1",
    "name": "Alice",
    "score": 80
  },
  {
    "id": "2",
    "name": "Bob",
    "score": 92.5
  },
  {
    "id": "3",
    "name": "Charlie",
    "score": 88
  }
]

Summary

We’ve demonstrated that it’s possible to build (and deploy) a Lakehouse infrastructure based on DuckLake and DuckDB, that can stay below $10 / month, while running completely serverless. This can be a suitable way to run a flexible Lakehouse for many smaller project or teams.

There’s no need anymore to build up complex and costly infrastructure projects on big public cloud providers.

Let me know if you have any feedback in the comments!

Using DuckDB databases as lightweight Data Lake access layer

Tobias Müller — Sat, 17 May 2025 22:00:00 GMT

Data Lakes come in a broad variety and lots of different flavors. AWS, Azure, Google Cloud, Snowflake, DataBricks, etc. they all have their specialties, strong and weak sides. Common among them is that the most, if not all, of them use Object Storage services such as Amazon S3 or Cloudflare R2 for storing their vast amount of data. Accessing those Data Lakes can involve pretty complicated IAM setups, OAuth, OIDC and other technologies in this space.

But if we step a few steps back from all the complexity: What’s the most lightweight and flexible way to provide access to data in Data Lakes?

For example, if we’d like to share open public transportation data, and we’d like to focus on costs and ease of use? With an interface that exists since more than 50 years?

Introducing: DuckDB databases as Data Lake access layer!

Jokes aside, I’m not the first one to ever suggest or use it, but I’ve hardly seen any written explanation of this yet, and also got some questions from the community about it (looking at you, David!), so I though it’d make sense to write a short blog post about this technique.

What are our goals?

We want to query the data with SQL
We want the data to be accessible via HTTP(S)
We want to download as little data as possible when querying it
We want (or already) have our data stored in Object Storage services like S3 or R2
We don’t need authentication and authorization to access the data (yet)

How does this work?

The basis of the approach is that we store our datasets on Object Storage. For example, you can store a table’s data as a single Parquet file, multiple Parquet files in a prefix with a naming pattern, or even as Hive-partitioned data. DuckDB can read (and write) them all. The latter two only via the S3-compatible API though, which needs some form of credentials, so for keeping our example simple, we focus on single Parquet files per table.

The other puzzle pieces are that DuckDB database files can contain view definitions that can reference remote Parquet files, and can be shared and attached from remote clients either via HTTP(S) or S3.

From a high-level perspective, our solution looks like this:

How can this be implemented?

The process to create such a "view database" consists of four main steps:

Exporting the data from a pre-made DuckDB database as Parquet files
Uploading the exported Parquet files to an Object Storage service
Creating the database file containing the views to the Object Storage files
Uploading the new database file to Object Storage as well, to be able to use the ATTACH statement of DuckDB

Example Use Case

I recently published another blog post about Handling GTFS data with DuckDB. The accompanying GitHub repository can be found at tobilg/duckdb-gtfs.

Preparation

As preparation, you’ll need to perform the steps outlined in the Usage chapter of the README. If you follow these instructions, it will generate a 1.4GB uncompressed DuckDB database file on your local machine, after it downloaded the example dataset’ raw data and loaded it into the default GTFS table schemas.

Export single Parquet files

The database can be exported as one Parquet file per table with a simple command (also available as bash script):

duckdb exported-data/providers/gtfs-de/full/data.duckdb -c "EXPORT DATABASE 'exported-data/providers/gtfs-de/full' (FORMAT parquet, COMPRESSION zstd);"

The result looks like this:

$ tree exported-data 
exported-data
└── providers
    └── gtfs-de
        └── full
            ├── agency.parquet
            ├── areas.parquet
            ├── attributions.parquet
            ├── booking_rules.parquet
            ├── calendar.parquet
            ├── calendar_dates.parquet
            ├── fare_attributes.parquet
            ├── fare_leg_join_rules.parquet
            ├── fare_leg_rules.parquet
            ├── fare_media.parquet
            ├── fare_products.parquet
            ├── fare_rules.parquet
            ├── fare_transfer_rules.parquet
            ├── feed_info.parquet
            ├── frequencies.parquet
            ├── levels.parquet
            ├── load.sql
            ├── location_group_stops.parquet
            ├── location_groups.parquet
            ├── networks.parquet
            ├── pathways.parquet
            ├── rider_categories.parquet
            ├── route_networks.parquet
            ├── routes.parquet
            ├── schema.sql
            ├── shapes.parquet
            ├── stop_areas.parquet
            ├── stop_times.parquet
            ├── stops.parquet
            ├── timeframes.parquet
            ├── transfers.parquet
            ├── translations.parquet
            └── trips.parquet

Not only have the Parquet files been produced, but also the load.sql and schema.sql scripts. We don’t need those for our use case though, so we can either ignore or delete them.

Upload Parquet files to Object Storage

The next step is to upload the newly created Parquet files to your Object Storage provider. In this example, I already have created a new R2 Bucket, and also assigned a custom domain for it (data.openrailway.dev), which is possible with a few clicks because the domain itself is hosted on Cloudflare as well.

💡

R2 offers free egress, compared to S3 which for example charges $0.09 per GB in us-east-1. For storage, S3 charges $0.023 / GB / month, whereas R2 charges $0.015 / GB / month.This can incur large costs if your data is used a lot (reminder: Our use case is providing public transportation datasets for analysis, so we don’t really now how many people will use them, and how). I’d recommend to make an explicit decision here based on your use case.

If you want to enable browser-based clients as well, make sure that you also setup a CORS policy. With R2, this can be configured via API or via the Dashboard and looks like this:

[
  {
    "AllowedOrigins": [
      "http://localhost:3000",
      "http://localhost:5174",
      "https://sql-workbench.com"
    ],
    "AllowedMethods": [
      "GET",
      "HEAD"
    ],
    "AllowedHeaders": [
      "*"
    ],
    "ExposeHeaders": [
      "ETag"
    ]
  }
]

This enables local development, and also SQL Workbench as browser-based query interface (more of this later). Add your own domains appropriately.

Once you prepared this, you can go on and upload the data to your Object Storage bucket, either with tools like rclone or directly via the Dashboard of R2 or the S3 Console.

Creating the “view database”

As the data is now available via HTTP(S), we can now create a new DuckDB database that contains the VIEWs that reference the data, namely the Parquet files, on Object Storage. This leverages the read_parquet() function, with which you can load remote data via HTTP(S).

An example for our use case (try it with SQL Workbench):

SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/agency.parquet');

The output will look like this:

We now prepare a SQL script that will create a view for each remote Parquet file:

CREATE VIEW agency AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/agency.parquet');
CREATE VIEW areas AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/areas.parquet');
CREATE VIEW attributions AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/attributions.parquet');
CREATE VIEW booking_rules AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/booking_rules.parquet');
CREATE VIEW calendar AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/calendar.parquet');
CREATE VIEW calendar_dates AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/calendar_dates.parquet');
CREATE VIEW fare_leg_join_rules AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_leg_join_rules.parquet');
CREATE VIEW fare_leg_rules AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_leg_rules.parquet');
CREATE VIEW fare_media AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_media.parquet');
CREATE VIEW fare_products AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_products.parquet');
CREATE VIEW fare_transfer_rules AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_transfer_rules.parquet');
CREATE VIEW feed_info AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/feed_info.parquet');
CREATE VIEW frequencies AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/frequencies.parquet');
CREATE VIEW levels AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/levels.parquet');
CREATE VIEW location_groups AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/location_groups.parquet');
CREATE VIEW networks AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/networks.parquet');
CREATE VIEW rider_categories AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/rider_categories.parquet');
CREATE VIEW shapes AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/shapes.parquet');
CREATE VIEW timeframes AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/timeframes.parquet');
CREATE VIEW translations AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/translations.parquet');
CREATE VIEW fare_attributes AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_attributes.parquet');
CREATE VIEW routes AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/routes.parquet');
CREATE VIEW route_networks AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/route_networks.parquet');
CREATE VIEW stops AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/stops.parquet');
CREATE VIEW stop_areas AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/stop_areas.parquet');
CREATE VIEW trips AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/trips.parquet');
CREATE VIEW fare_rules AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/fare_rules.parquet');
CREATE VIEW location_group_stops AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/location_group_stops.parquet');
CREATE VIEW pathways AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/pathways.parquet');
CREATE VIEW stop_times AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/stop_times.parquet');
CREATE VIEW transfers AS SELECT * FROM read_parquet('https://data.openrailway.dev/providers/gtfs-de/full/transfers.parquet');

To create the actual “view database”, we can use the following command:

duckdb exported-data/providers/gtfs-de/full/database.duckdb < queries/providers/gtfs-de/full/create_view_database.sql

This will create a new database in the exported-data/providers/gtfs-de/full directory named database.duckdb.

We can now inspect its size:

$ ls -la exported-data/providers/gtfs-de/full/*.duckdb
-rw-r--r--@ 1 tmueller  staff  274432 18 Mai 14:32 exported-data/providers/gtfs-de/full/database.duckdb

This means the “view database” has ONLY 268kb in size!

Testing the “view database”

We can now test if it works locally by doing a SHOW TABLES:

$ duckdb exported-data/providers/gtfs-de/full/database.duckdb
v1.2.2 7c039464e4
Enter ".help" for usage hints.
D show tables;
┌──────────────────────┐
│         name         │
│       varchar        │
├──────────────────────┤
│ agency               │
│ areas                │
│ attributions         │
│ booking_rules        │
│ calendar             │
│ calendar_dates       │
│ fare_attributes      │
│ fare_leg_join_rules  │
│ fare_leg_rules       │
│ fare_media           │
│ fare_products        │
│ fare_rules           │
│ fare_transfer_rules  │
│ feed_info            │
│ frequencies          │
│ levels               │
│ location_group_stops │
│ location_groups      │
│ networks             │
│ pathways             │
│ rider_categories     │
│ route_networks       │
│ routes               │
│ shapes               │
│ stop_areas           │
│ stop_times           │
│ stops                │
│ timeframes           │
│ transfers            │
│ translations         │
│ trips                │
├──────────────────────┤
│       31 rows        │
└──────────────────────┘

Also, we can run some test queries on the remote data, and show the time they take:

$ duckdb exported-data/providers/gtfs-de/full/database.duckdb
v1.2.2 7c039464e4
Enter ".help" for usage hints.
D .timer on
D SELECT count(*) FROM trips;
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    1630671     │
│ (1.63 million) │
└────────────────┘
Run Time (s): real 0.554 user 0.175864 sys 0.089233
D SELECT count(*) FROM stops;
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│    678388    │
└──────────────┘
Run Time (s): real 0.336 user 0.006173 sys 0.002544
D SELECT count(*) FROM stop_times;
┌─────────────────┐
│  count_star()   │
│      int64      │
├─────────────────┤
│    32228711     │
│ (32.23 million) │
└─────────────────┘
Run Time (s): real 0.367 user 0.051620 sys 0.009298
D SELECT * FROM stops LIMIT 10;
┌─────────┬───────────┬──────────────────────┬───────────────┬───────────┬───────────┬───┬──────────┬───────────────┬────────────────┬───────────────┬─────────────────────┬──────────┬───────────────┐
│ stop_id │ stop_code │      stop_name       │ tts_stop_name │ stop_desc │ stop_lat  │ … │ stop_url │ location_type │ parent_station │ stop_timezone │ wheelchair_boarding │ level_id │ platform_code │
│ varchar │  varchar  │       varchar        │    varchar    │  varchar  │  double   │   │ varchar  │     int32     │    varchar     │    varchar    │        int32        │ varchar  │    varchar    │
├─────────┼───────────┼──────────────────────┼───────────────┼───────────┼───────────┼───┼──────────┼───────────────┼────────────────┼───────────────┼─────────────────────┼──────────┼───────────────┤
│ 199269  │ NULL      │ 's-Heerenberg Goud…  │ NULL          │ NULL      │  51.87225 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 358723  │ NULL      │ 's-Heerenberg Goud…  │ NULL          │ NULL      │  51.87228 │ … │ NULL     │          NULL │ 199269         │ NULL          │                NULL │ NULL     │ NULL          │
│ 536901  │ NULL      │ 's-Heerenberg Mole…  │ NULL          │ NULL      │  51.87632 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 532065  │ NULL      │ 's-Heerenberg Mole…  │ NULL          │ NULL      │  51.87632 │ … │ NULL     │          NULL │ 536901         │ NULL          │                NULL │ NULL     │ NULL          │
│ 666544  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │  51.87479 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 269106  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │ 51.874844 │ … │ NULL     │          NULL │ 666544         │ NULL          │                NULL │ NULL     │ NULL          │
│ 540312  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │ 51.874737 │ … │ NULL     │          NULL │ 666544         │ NULL          │                NULL │ NULL     │ NULL          │
│ 615572  │ NULL      │ 's-Hertogenbosch     │ NULL          │ NULL      │  51.69054 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 525421  │ NULL      │ 's-Hertogenbosch     │ NULL          │ NULL      │  51.69054 │ … │ NULL     │          NULL │ 615572         │ NULL          │                NULL │ NULL     │ NULL          │
│ 198290  │ NULL      │ 12 Apostel           │ NULL          │ NULL      │ 52.052963 │ … │ NULL     │          NULL │ 655876         │ NULL          │                NULL │ NULL     │ NULL          │
├─────────┴───────────┴──────────────────────┴───────────────┴───────────┴───────────┴───┴──────────┴───────────────┴────────────────┴───────────────┴─────────────────────┴──────────┴───────────────┤
│ 10 rows                                                                                                                                                                       15 columns (13 shown) │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Run Time (s): real 0.909 user 0.214754 sys 0.023443

As you might have noticed, we counted the records of a remote >32 million records file in less than 0.4 seconds!

We can also switch on the enable_http_logging flag to see the HTTP(S) requests performed by DuckDB when querying the remote files:

D SET enable_http_logging = true;
Run Time (s): real 0.001 user 0.000300 sys 0.000365
D SELECT * FROM stops LIMIT 10;
HTTP Request:
  HEAD /providers/gtfs-de/full/stops.parquet
  Accept: */*
  Host: data.openrailway.dev
  User-Agent: cpp-httplib/0.14.3

HTTP Response:
  200 OK 
  Accept-Ranges: bytes
  alt-svc: h3=":443"; ma=86400
  cf-cache-status: DYNAMIC
  CF-RAY: 9423a96b0dd4d38c-FRA
  Connection: keep-alive
  Content-Length: 14580912
  Date: Sun, 18 May 2025 12:40:15 GMT
  ETag: "5d654fe45888665fcaff7f2831180088"
  Last-Modified: Sun, 18 May 2025 10:42:40 GMT
  NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
  Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=8J9y/6mVjzeKKD/RHRLvPc53+xxuxj+n8m6FugMHTbB0N8cD0l18HfCieD6GeWrGlFB1vCMf7PnzSxj8oCYJYgsQHvcJCxnw5OEIkgv9C4FWDrkv1jgTKj3N4IfoIwJGLMqAMAQ+LaEssRo5FN9DPSmWsg=="}],"group":"cf-nel","max_age":604800}
  Server: cloudflare
  server-timing: cfL4;desc="?proto=TCP&rtt=9270&min_rtt=8571&rtt_var=3023&sent=6&recv=7&lost=0&retrans=0&sent_bytes=2869&recv_bytes=556&delivery_rate=469577&cwnd=33&unsent_bytes=0&cid=6800b4b191d1dbe2&ts=165&x=0"
  Vary: Accept-Encoding

HTTP Request:
  GET /providers/gtfs-de/full/stops.parquet
  Accept: */*
  Host: data.openrailway.dev
  Range: bytes=14580904-14580911
  User-Agent: cpp-httplib/0.14.3

HTTP Response:
  206 Partial Content 
  Accept-Ranges: bytes
  alt-svc: h3=":443"; ma=86400
  cf-cache-status: DYNAMIC
  CF-RAY: 9423a96bbf92d38c-FRA
  Connection: keep-alive
  Content-Length: 8
  Content-Range: bytes 14580904-14580911/14580912
  Date: Sun, 18 May 2025 12:40:16 GMT
  ETag: "5d654fe45888665fcaff7f2831180088"
  Last-Modified: Sun, 18 May 2025 10:42:40 GMT
  NEL: {"success_fraction":0,"report_to":"cf-nel","max_age":604800}
  Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v4?s=095gOTFBeOfulP4iVVnDyvqk/FLW3exKsuA0e2t3/kNqb2QNnBtswCgTkKoGlL3F8qnNYZjdlljl2wTDb1+KWcrhAYrVAC5sUOHEnjkVzYixd1OQq/WFlHh3/u9HRTqo+UbBxBJKXIS+yxJHwlIHZDzFXQ=="}],"group":"cf-nel","max_age":604800}
  Server: cloudflare
  server-timing: cfL4;desc="?proto=TCP&rtt=9152&min_rtt=8571&rtt_var=1901&sent=9&recv=10&lost=0&retrans=0&sent_bytes=4253&recv_bytes=737&delivery_rate=469577&cwnd=35&unsent_bytes=0&cid=6800b4b191d1dbe2&ts=245&x=0"
  Vary: Accept-Encoding

// SNIP //

┌─────────┬───────────┬──────────────────────┬───────────────┬───────────┬───────────┬───┬──────────┬───────────────┬────────────────┬───────────────┬─────────────────────┬──────────┬───────────────┐
│ stop_id │ stop_code │      stop_name       │ tts_stop_name │ stop_desc │ stop_lat  │ … │ stop_url │ location_type │ parent_station │ stop_timezone │ wheelchair_boarding │ level_id │ platform_code │
│ varchar │  varchar  │       varchar        │    varchar    │  varchar  │  double   │   │ varchar  │     int32     │    varchar     │    varchar    │        int32        │ varchar  │    varchar    │
├─────────┼───────────┼──────────────────────┼───────────────┼───────────┼───────────┼───┼──────────┼───────────────┼────────────────┼───────────────┼─────────────────────┼──────────┼───────────────┤
│ 199269  │ NULL      │ 's-Heerenberg Goud…  │ NULL          │ NULL      │  51.87225 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 358723  │ NULL      │ 's-Heerenberg Goud…  │ NULL          │ NULL      │  51.87228 │ … │ NULL     │          NULL │ 199269         │ NULL          │                NULL │ NULL     │ NULL          │
│ 536901  │ NULL      │ 's-Heerenberg Mole…  │ NULL          │ NULL      │  51.87632 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 532065  │ NULL      │ 's-Heerenberg Mole…  │ NULL          │ NULL      │  51.87632 │ … │ NULL     │          NULL │ 536901         │ NULL          │                NULL │ NULL     │ NULL          │
│ 666544  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │  51.87479 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 269106  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │ 51.874844 │ … │ NULL     │          NULL │ 666544         │ NULL          │                NULL │ NULL     │ NULL          │
│ 540312  │ NULL      │ 's-Heerenberg Muzi…  │ NULL          │ NULL      │ 51.874737 │ … │ NULL     │          NULL │ 666544         │ NULL          │                NULL │ NULL     │ NULL          │
│ 615572  │ NULL      │ 's-Hertogenbosch     │ NULL          │ NULL      │  51.69054 │ … │ NULL     │             1 │ NULL           │ NULL          │                NULL │ NULL     │ NULL          │
│ 525421  │ NULL      │ 's-Hertogenbosch     │ NULL          │ NULL      │  51.69054 │ … │ NULL     │          NULL │ 615572         │ NULL          │                NULL │ NULL     │ NULL          │
│ 198290  │ NULL      │ 12 Apostel           │ NULL          │ NULL      │ 52.052963 │ … │ NULL     │          NULL │ 655876         │ NULL          │                NULL │ NULL     │ NULL          │
├─────────┴───────────┴──────────────────────┴───────────────┴───────────┴───────────┴───┴──────────┴───────────────┴────────────────┴───────────────┴─────────────────────┴──────────┴───────────────┤
│ 10 rows                                                                                                                                                                       15 columns (13 shown) │
└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
Run Time (s): real 0.578 user 0.026387 sys 0.018349

We can see the range requests in action, first sending a HEAD request, and then followed by multiple (logs are truncated for brevity) GET range requests.

To even get some more insights, we can prefix the query with EXPLAIN ANALYZE and have a look how the query is executed, along with some stats:

D EXPLAIN ANALYZE SELECT * FROM stops LIMIT 10;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││    Query Profiling Information    ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
EXPLAIN ANALYZE SELECT * FROM stops LIMIT 10;
┌─────────────────────────────────────┐
│┌───────────────────────────────────┐│
││         HTTPFS HTTP Stats         ││
││                                   ││
││            in: 2.5 MiB            ││
││            out: 0 bytes           ││
││              #HEAD: 1             ││
││              #GET: 3              ││
││              #PUT: 0              ││
││              #POST: 0             ││
││             #DELETE: 0            ││
│└───────────────────────────────────┘│
└─────────────────────────────────────┘
┌────────────────────────────────────────────────┐
│┌──────────────────────────────────────────────┐│
││              Total Time: 0.490s              ││
│└──────────────────────────────────────────────┘│
└────────────────────────────────────────────────┘
┌───────────────────────────┐
│           QUERY           │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      EXPLAIN_ANALYZE      │
│    ────────────────────   │
│           0 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│      STREAMING_LIMIT      │
│    ────────────────────   │
│          10 Rows          │
│          (0.00s)          │
└─────────────┬─────────────┘
┌─────────────┴─────────────┐
│         TABLE_SCAN        │
│    ────────────────────   │
│         Function:         │
│        READ_PARQUET       │
│                           │
│        Projections:       │
│          stop_id          │
│         stop_code         │
│         stop_name         │
│       tts_stop_name       │
│         stop_desc         │
│          stop_lat         │
│          stop_lon         │
│          zone_id          │
│          stop_url         │
│       location_type       │
│       parent_station      │
│       stop_timezone       │
│    wheelchair_boarding    │
│          level_id         │
│       platform_code       │
│                           │
│         4096 Rows         │
│          (0.22s)          │
└───────────────────────────┘
Run Time (s): real 0.491 user 0.194019 sys 0.022184

Uploading the “view database” to Object Storage

For the final step to use the DuckDB database file containing the views to the remote Parquet files, we need to upload the database file itself to Object Storage as well.

For our example, we’ll upload it to the same directory as the Parquet files, meaning that it’s available at:

https://data.openrailway.dev/providers/gtfs-de/full/database.duckdb

Using the DuckDB database

DuckDB is able to ATTACH remote databases via HTTP(S) or the S3 API, meaning that you can both use it in the local DuckDB CLI, or with another, for example in-browser, tool like SQL Workbench.

$ duckdb                                                     
v1.2.2 7c039464e4
Enter ".help" for usage hints.
Connected to a transient in-memory database.
Use ".open FILENAME" to reopen on a persistent database.
D ATTACH 'https://data.openrailway.dev/providers/gtfs-de/full/database.duckdb' as gtfs_de_full (READ_ONLY);
D USE gtfs_de_full;
D SHOW TABLES;
┌──────────────────────┐
│         name         │
│       varchar        │
├──────────────────────┤
│ agency               │
│ areas                │
│ attributions         │
│ booking_rules        │
│ calendar             │
│ calendar_dates       │
│ fare_attributes      │
│ fare_leg_join_rules  │
│ fare_leg_rules       │
│ fare_media           │
│ fare_products        │
│ fare_rules           │
│ fare_transfer_rules  │
│ feed_info            │
│ frequencies          │
│ levels               │
│ location_group_stops │
│ location_groups      │
│ networks             │
│ pathways             │
│ rider_categories     │
│ route_networks       │
│ routes               │
│ shapes               │
│ stop_areas           │
│ stop_times           │
│ stops                │
│ timeframes           │
│ transfers            │
│ translations         │
│ trips                │
├──────────────────────┤
│       31 rows        │
└──────────────────────┘
D SELECT count(*)::INT AS cnt FROM gtfs_de_full.stop_times;
┌─────────────────┐
│       cnt       │
│      int32      │
├─────────────────┤
│    32228711     │
│ (32.23 million) │
└─────────────────┘

To query the data in your browser, just click on the this link to SQL Workbench. The result will look like this:

Summary

In this blog post, we were able to show that DuckDB database files containing views to static Parquet files are a very lightweight solution to provide a single, highly available entry point to datasets that need to be queryable with SQL.

It intentionally doesn’t talk about authentication and authorization, which will be covered in another subsequent blog post soon.

Handling GTFS data with DuckDB

Tobias Müller — Fri, 16 May 2025 22:00:00 GMT

The General Transit Feed Specification (GTFS) is a standardized, open data format for public transportation schedules and geographic information. In practice, a GTFS feed is simply a ZIP archive of text (CSV) tables - such as stops.txt, routes.txt, and trips.txt - that together describe an agency’s routes, stops, schedules, and fare rules.

By following this common schema, transit agencies can publish their data so that any GTFS-compatible application (trip planners, mapping tools, or analytics pipelines) can consume it easily. Because GTFS is an open standard, it has become the go-to format for thousands of transit agencies worldwide.

This broad adoption means developers can integrate multiple feeds into a single pipeline without writing custom parsers. For example, many journey-planning apps and mapping services rely on GTFS feeds to deliver route and schedule information to users.

The specification even separates static and dynamic data: the Schedule portion provides planned timetables (in CSV), while the Realtime extension (using Protocol Buffers) adds live trip updates, vehicle positions, and service alerts.

The main benefits of GTFS are:

It defines a consistent schema (stops, routes, trips, etc), so users / developers only need to learn one format for all agencies
Interoperability: Any GTFS-compliant tool can read feeds from any transit provider, enabling seamless integration across different platforms and systems
Plain-text and open: The format is based on simple CSV files, which are easy to generate, ingest (egvia SQL or pandas), and maintain without proprietary software
Static + real-time: GTFS distinguishes fixed schedules from live updates. The static GTFS-Schedule files cover routes and timetables, while GTFS-Realtime (Protocol Buffers) provides real-time trip updates, vehicle positions, and alerts.
Ecosystem and tooling: A large community supports GTFS with open documentation and many libraries/validators (e.g. GTFS-kit in Python), so developer can leverage existing tools rather than build custom parsers

Creating a DuckDB database for GTFS Schedule data

As the GTFS Schedule standard uses ZIP-compressed CSV files, it’s not possible out of the box to directly analyze datasets with the most tools.

Also, a lot of GTFS datasets don’t publish all dataset files, as some of the data files are optional, additionally not all fields in each those files are required as well:

File Name	Presence	Description
agency.txt	Required	Transit agencies with service represented in this dataset.
stops.txt	Conditionally Required	Stops where vehicles pick up or drop off riders. Also defines stations and station entrances. Optional if demand-responsive zones are defined in locations.geojson, required otherwise.
routes.txt	Required	Transit routes. A route is a group of trips that are displayed to riders as a single service.
trips.txt	Required	Trips for each route. A trip is a sequence of two or more stops that occur during a specific time period.
stop_times.txt	Required	Times that a vehicle arrives at and departs from stops for each trip.
calendar.txt	Conditionally Required	Service dates specified using a weekly schedule with start and end dates. Required unless all dates of service are defined in calendar_dates.txt, optional otherwise.
calendar_dates.txt	Conditionally Required	Exceptions for the services defined in calendar.txt. Required if calendar.txt is omitted (must then contain all dates of service), optional otherwise.
fare_attributes.txt	Optional	Fare information for a transit agency’s routes.
fare_rules.txt	Optional	Rules to apply fares for itineraries.
timeframes.txt	Optional	Date and time periods to use in fare rules for fares that depend on date/time factors. Forbidden if `network_id` exists in routes.txt, optional otherwise.
shapes.txt	Optional	Rules for mapping vehicle travel paths (route alignments).
frequencies.txt	Optional	Headway (time between trips) for headway-based service or compressed fixed-schedule service.
transfers.txt	Optional	Rules for making connections at transfer points between routes.
pathways.txt	Optional	Pathways linking together locations within stations.
levels.txt	Conditionally Required	Levels within stations. Required when describing pathways with elevators (`pathway_mode=5`), optional otherwise.
location_groups.txt	Optional	A group of stops that together indicate locations where a rider may request pickup or drop off.
location_group_stops.txt	Optional	Rules to assign stops to location groups.
locations.geojson	Optional	Zones for rider pickup/drop-off requests by on-demand services, represented as GeoJSON polygons.
booking_rules.txt	Optional	Booking information for rider-requested services.
translations.txt	Optional	Translations of customer-facing dataset values.
feed_info.txt	Conditionally Required	Dataset metadata, including publisher, version, and expiration information. Required if translations.txt is provided, recommended otherwise.
attributions.txt	Optional	Dataset attributions.

Taken that into account, the plan for being able to use GTFS Schedule datasets with DuckDB is: Create a DuckDB database that contains all tables with all columns. The DDL for the tables will be derived from the official standard’s website. We want to use the foreign key relationships outlined in the standard, and thus need to make sure that the tables are created in the right order.

The database creation script can be found at queries/create_gtfs_database.sql.

This results in a database that has the following Entity Relationship Diagram (ERD):

Creating a DuckDB GTFS database from an example dataset

To load some sample dataset, we need to choose one of the many available data sources. For German data, the gtfs.de website provides some interesting datasets.

There’s one dataset that contains the full train and local traffic for Germany: de_full, and that’s the one we’ll use. It contains 1.6 million trips, 663 thousand stops, and more than 32 million stop times.

The accompanying GitHub repository tobilg/duckdb-gtfs has the necessary scripts available. You’ll need a locally installed DuckDB CLI to be able to follow the next steps on your machine.

Preparations

You need to clone the repository from GitHub first:

git clone git@github.com:tobilg/duckdb-gtfs.git && cd duckdb-gtfs

Downloading the dataset

To download the example dataset (218MB zipped, around 1.4GB unzipped), you can run the download script:

scripts/providers/gtfs-de/full/download_data.sh

Loading the dataset in a new GTFS database

As outlined above, not all dataset providers use the full set of tables in their datasets, so we’ll have to write a custom database creation script for each provider. In our case, the SQL for the example dataset looks like this (see load_data.sql):

INSERT INTO agency (agency_id,agency_name,agency_url,agency_timezone,agency_lang) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/agency.txt', delim = ',', header = true);

INSERT INTO attributions (attribution_id,organization_name,is_producer,attribution_url,attribution_email) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/attributions.txt', delim = ',', header = true);

INSERT INTO calendar (monday,tuesday,wednesday,thursday,friday,saturday,sunday,start_date,end_date,service_id) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/calendar.txt', delim = ',', header = true, dateformat = '%Y%m%d');

INSERT INTO calendar_dates (service_id,exception_type,"date") 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/calendar_dates.txt', delim = ',', header = true, columns = {
  'service_id': 'VARCHAR',
  'exception_type': 'INTEGER',
  'date': 'DATE'
}, dateformat = '%Y%m%d', ignore_errors = true);

INSERT INTO feed_info (feed_publisher_name,feed_publisher_url,feed_lang,feed_version,feed_contact_email,feed_contact_url) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/feed_info.txt', delim = ',', header = true, dateformat = '%Y%m%d');

INSERT INTO routes (route_long_name,route_short_name,agency_id,route_type,route_id) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/routes.txt', delim = ',', header = true);

INSERT INTO trips (route_id,service_id,trip_id) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/trips.txt', delim = ',', header = true, ignore_errors = true);

INSERT INTO stops (stop_name,parent_station,stop_id,stop_lat,stop_lon,location_type) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/stops.txt', delim = ',', header = true);

INSERT INTO stop_times (trip_id,arrival_time,departure_time,stop_id,stop_sequence,pickup_type,drop_off_type) 
SELECT * FROM read_csv('source-data/providers/gtfs-de/full/stop_times.txt', delim = ',', header = true, ignore_errors = true);

Now, run the database creation script (this will take some time depending on your machine):

scripts/providers/gtfs-de/full/create_database.sh

The resulting DuckDB database file can be found at exported-data/providers/gtfs-de/full/data.duckdb.

Querying the new GTFS database

To query the new GTFS database, you can just use the newly created database file with your local DuckDB CLI:

duckdb exported-data/providers/gtfs-de/full/data.duckdb

v1.2.2 7c039464e4
Enter ".help" for usage hints.
D show tables;
┌──────────────────────┐
│         name         │
│       varchar        │
├──────────────────────┤
│ agency               │
│ areas                │
│ attributions         │
│ booking_rules        │
│ calendar             │
│ calendar_dates       │
│ fare_attributes      │
│ fare_leg_join_rules  │
│ fare_leg_rules       │
│ fare_media           │
│ fare_products        │
│ fare_rules           │
│ fare_transfer_rules  │
│ feed_info            │
│ frequencies          │
│ levels               │
│ location_group_stops │
│ location_groups      │
│ networks             │
│ pathways             │
│ rider_categories     │
│ route_networks       │
│ routes               │
│ shapes               │
│ stop_areas           │
│ stop_times           │
│ stops                │
│ timeframes           │
│ transfers            │
│ translations         │
│ trips                │
├──────────────────────┤
│       31 rows        │
└──────────────────────┘
D SELECT count(*) FROM stop_times;
┌─────────────────┐
│  count_star()   │
│      int64      │
├─────────────────┤
│    32228711     │
│ (32.23 million) │
└─────────────────┘
D SELECT count(*) FROM stops;
┌──────────────┐
│ count_star() │
│    int64     │
├──────────────┤
│    678388    │
└──────────────┘
D SELECT count(*) FROM trips;
┌────────────────┐
│  count_star()  │
│     int64      │
├────────────────┤
│    1630671     │
│ (1.63 million) │
└────────────────┘

Export the data in Parquet format

If you want to export the data in an optimized format like Parquet, this is very straight forward with DuckDB (as demonstrated with the export_data.sh script:

duckdb exported-data/providers/gtfs-de/full/data.duckdb -c "EXPORT DATABASE 'exported-data/providers/gtfs-de/full' (FORMAT parquet);"

This will export each table as a Parquet file in the exported-data/providers/gtfs-de/full/ directory.

Summary

Using DuckDB for storing and analyzing GTFS Schedule data seems like a great choice. For example, the CSV reader of DuckDB can handle a lot of different error scenarios, which are quite common with data providers.

We successfully created a default GTFS Schedule database, that can be loaded with data from different providers with custom loading scripts. Those require specific effort to create, but as the result is a standardized database schema, it can create value for data-driven organizations to do so.

Being able to export the data to storage-optimized formats out-of-the-box is also a great benefit of DuckDB. We were able to shrink the 1.4GB unzipped data to 118MB in compressed Parquet files.

If you want to query the data live and in your browser, you can use SQL Workbench.

Cost-efficient event ingestion into Iceberg S3 Tables on AWS

Tobias Müller — Thu, 10 Apr 2025 19:22:36 GMT

Amazon S3 Tables was launched on December 3rd 2024, and provides you “storage that is optimized for tabular data such as daily purchase transactions, streaming sensor data, and ad impressions in Apache Iceberg format”.

While S3 Tables can be queried with a variety of AWS services, such as Amazon Athena, Amazon EMR, Apache Spark or, even better, DuckDB, the ingestion of data is currently pretty limited. You can insert data via SQL queries in Athena, EMR Spark clusters, and as a pretty new feature, Amazon Data Firehose.

For use cases that involve (mass) event ingestion, which also want to be cost-efficient, using Athena would be impractical (and slow), whereas doing so via EMR Spark clusters is very costly. Contrary to that, Amazon Data Firehose’s pricing is strictly pay-per-use for Apache Iceberg tables (not sure why the use a different name for S3 Tables though):

For Firehose streams that is configured with Apache Iceberg Tables as a destination, you will be billed for the amount of data processed to the destination. Pricing is billed per GB ingested with no 5KB increments. If data processed bytes before delivery is more than the ingested bytes due to custom Lambda processing, then the additional bytes are also billed. Additional bytes are billed at the same rate as shown in Kinesis Data Streams as a source to Apache Iceberg tables as a destination for all sources of ingestion including Direct PUT.

So that’s a pretty sweet deal at $0.075/GB (for the first 250TB/month) one might say!

Example Use Case

How would a event ingestion pipeline that lands the data in S3 Tables look like? A typical example use case would be web analytics, where client browsers send requests containing information about the browser, the used devices, user location and potentially many more details to a backend. There can be different event types, such as normal pageviews, or tracking events of varying structure.

The backend then ingests the events, validates, cleans, transform and stores them, so that different metrics can be calculated in later steps. For example visitor count, pageview counts, session length etc.

For sending the events, we will use the OwnStats Client, which is a plugin for getanalytics.io. Alternatively, you can also sent sample events via HTTP clients as well.

Architectural Implementation Options

But how should the actual events be “landed” on AWS, if we consider them as typical JSON payloads? Below you’ll find some architectural variants that achieve the same or similar goals, but with different services involved. As we’d like to optimize for costs, we want to focus on services that are pay-per-use, but also discuss break-even points compared to fixed-priced services as well.

Variant 1: API Gateway, Lambda and Kinesis Data Firehose to S3

The usual suspect for event ingestion would be an API created with API Gateway fronting Lambda functions, then publishing the events to a Kinesis Data Firehose stream that forwards the data to S3 Tables, or more “classical” to S3 as Hive-partitioned Parquet files:

Variant 2: CloudFront, Realtime Logs, Kinesis Data Streams & Firehose to S3 via Lambda

CloudFront basically replaces API Gateway here, by hosting a 1×1 pixel GIF file that can be used to pass event payloads to via GET request query strings, which is the same method a lot of web analytics providers are using (such as Google Analytics).

CloudFront Realtime Logs forward the 100% “sample” of the logs containing the actual event payloads to a Kinesis Data Stream, which itself feeds the events to a Kinesis Data Firehose stream that uses a processing Lambda function that decodes the event payloads to a tabular form, and also adds Hive partitioning info back to the stream. Firehose then writes the data in batches to S3.

Unfortunately, it’s currently not possible for the Realtime Logs to directly write to Kinesis Data Firehose, instead of Kinesis Data Streams first, which imposes a fixed cost “tax” on this architecture of $0.015/hour, which means at least $10.80/month.

This architecture is implemented at OwnStats, which is also further documented and available on GitHub as well.

Variant 3: CloudFront, CloudWatch Logs, Kinesis Data Firehose to S3 via Lambda

In this architecture variant, the CloudFront Realtime Logs and the Kinesis Data Stream are replaced by a CloudWatch Logs Delivery, which forward the standard logs (v2!) from CloudFront to the Kinesis Data Firehose.

This gets rid of the fixed costs for the Kinesis Data Stream (as outlined above), and replaces this with a pay-per-use pricing of $0.25/GB for the Logs Delivery. This means if you ingest less than 40GB/month of events, you’ll definitely spent less with this variant.

The data doesn’t land in S3 as Hive-partitioned Parquet files in this case, but in the self-optimizing Apache Iceberg format in a S3 Table. The processing Lambda function is updated to support the routing of the events to the correct S3 Table, as well as parsing the raw logs from CloudFront, and transform them to the right tabular format.

Caveats

There are currently multiple downsides of this approach:

There is no complete CloudFormation support for S3 Tables, meaning that you can use it to manage S3 Table Buckets, but not Namespaces or actual S3 Tables. This works only with the CLI, SDKs or the API. This means that we can’t deploy this as one single stack, but have to take a multi-step/stack approach with manual interactions.
The forwarding of the data from the Kinesis Data Firehose to the S3 Table only works if a LakeFormation Data Lake Administrator has been set up. Furthermore, the IAM role you deploy the CloudFormation stacks with needs to have the correct permissions / managed policy as well.
When creating the LakeFormation Permission via CloudFormation, there seems to be a strange bug that even when you define a DependsOn dependency for the permission that needs to be there before creating the Kinesis Data Firehose resource which has a reference to the actual Namespace and S3 Table, you’ll be seeing the following error:

Role arn:aws:iam::112233445566:role/rolename is not authorized to perform: glue:GetTable for the given table or the table does not exist

If the Kinesis Data Firehose resource is created in another subsequent CloudFormation stack, it works. This seems to be some kind of race condition in the CloudFormation service.
S3 Tables currently only support the Apache Iceberg primitive types as per the docs. As we’d like to store and query JSON payloads, it’d be much nicer if it would also support semi-structured types like variant.

S3 Tables supports all Apache Iceberg primitive types. For more information, see the Apache Iceberg documentation.

Nevertheless, we want to move forward to the implementation, while taking the caveats into account.

Implementation of the Event Ingestion Pipeline

As Variant 1 has been implemented by many other people already, and Variant 2 is implemented for example by OwnStats as well (while producing fixed costs), Variant 3 shall be be the focus because it offers an overall pay-per-use approach, is technically novel and thus potentially more interesting for the community, even though it has the outlined caveats.

The focus lies on implementing the Example Use Case of web analytics events, as outlined above, and builds upon the insights gained while creating OwnStats, a completely serverless web analytics

Prerequisites

The following prerequisites need to be available on the machine you want to deploy from:

AWS IAM credentials that have the respective permissions to deploy the different stack’s resources
A Node.js installation (> v18, including npm)
A global installation of the Serverless Framework (v3, not v4!) via npm i -g serverless@3

Structure of the project

The Iceberg Event Pipeline project consists of three sequentially deployable CloudFormation stacks, as well as a set of scripts that create the Namespace and the actual S3 Table resources, as they currently have no CloudFormation support themselves.

Base Stack

The Base Stack contains the CloudFront Distribution, the S3 Bucket for the Backup of non-forwardable data from the Kinesis Data Firehose to the S3 Table, and the S3 Table Bucket.

Processing Stack

The Processing Stack contains the Kinesis Data Firehose Policy, the LakeFormation permissions, the Glue Database Resource Link, and the Lambda function for the event processing.

Storage Stack

The Storage Stack contains the actual Kinesis Data Firehose, the CloudWatch Logs Delivery as well as it’s Source and Destination.

Configuration

In the root folder of the project, there’s a file called config.json, which contains the project’s configuration that’s used during the stack deployments. It looks like this:

{
  "profile": "default",
  "stage": "prd",
  "region": "us-east-1",
  "s3": {
    "tableBucket": "iceberg-event-analytics-data",
    "namespace": "iceberg_event_analytics",
    "tableName": "event_data"
  }
}

You can configure the AWS credentials profile name, the stage, and the AWS region you want to deploy the stacks to. In the s3 object, the S3 Tables-related resource names can be configured. You can also just use the defaults.

S3 Table Structure

The structure of the target table is defined in a JSON file as well, which is used during the manual creation of the S3 Table resources (see below). The table structure is reused from the OwnStats incoming_events table, though slightly adapted/simplified.

Column name	Data type	Is partition key?	Description
event_id	uuid	yes	The domain name
event_date	string	no	The date of the event (YYYY-MM-DD), as string
event_timestamp	timestamp	no	The exact event timestamp
arrival_timestamp	timestamp	no	The exact timestamp when the event arrived in the Kinesis Data Stream
edge_city	string	no	The name of the edge city (all edge location info is derived from the `x-edge-location` field in the logs)
edge_state	string	no	The state of the edge location
edge_country	string	no	The country of the edge location
edge_country_code	string	no	The country code of the edge location
edge_latitude	float	no	The latitude of the edge location
edge_longitude	float	no	The longitude of the edge location
edge_id	string	no	The original id of the edge location
domain_name	string	no	The domain name where the event was recorded
referrer	string	no	The referrer
referrer_domain_name	string	no	The domain name of the referrer
browser_name	string	no	The name of the browser
browser_version	string	no	The version of the browser
browser_os_name	string	no	The OS name of the browser
browser_os_version	string	no	The OS version of the browser
browser_timezone	string	no	The timezone of the browser
browser_language	string	no	The language of the browser
device_type	string	no	The device type
device_vendor	string	no	The device vendor
device_outer_resolution	string	no	The outer resolution of the device
device_inner_resolution	string	no	The inner resolution of the device
device_color_depth	int	no	The color depth of the device
device_platform	string	no	The platform of the device
device_memory	float	no	The memory of the device (in MB)
device_cores	int	no	The number of cores of the device
utm_source	string	no	Identifies which site sent the traffic
utm_campaign	string	no	Identifies a specific product promotion or strategic campaign
utm_medium	string	no	Identifies what type of link was used, such as cost per click or email
utm_content	string	no	Identifies what specifically was clicked to bring the user to the site
utm_term	string	no	Identifies search terms
request_url	string	no	The full requested URL
request_path	string	no	The path of the requested URL
request_query_string	string	no	The query string of the requested URL
request_bytes	int	no	The size of the request in bytes
request_status_code	int	no	The HTTP status code of the request
request_cache_status	string	no	The CloudFront cache status
request_delivery_time_ms	int	no	The time in ms it took for CloudFront to complete the request
request_asn	int	no	The ASN of the requestor
event_type	string	yes	The type of the event (`pageview` or `track`)
event_name	string	no	The name of the event for tracking events
event_data	string	no	The stringified event payload for tracking events

Deployment

To be able to deploy the project, you have to have set up all the above mentioned requirements on the machine you want to deploy from.

💡

You need to deploy the stacks / do the manual steps as outlined below, otherwise it won’t work as they depend on each other.

Deployment of the Base Stack

cd base-stack

# Deploy Base Stack
sls deploy

# Return to project base directory
cd ..

Manual creation of S3 Table resources & Asset sync for CloudFront

cd manual-setup

# Install dependencies
npm i 

# Create S3 Table resources
npm run create-resources

# Sync assets to S3 Bucket that backs the Cloudfront Distribution
npm run sync-s3

# Return to project base directory
cd ..

Deployment of the Processing Stack

cd processing-stack

# Install dependencies
npm i

# Deploy Processing Stack
sls deploy

# Return to project base directory
cd ..

Deployment of the Storage Stack

cd storage-stack

# Deploy Storage Stack
sls deploy

# Return to project base directory
cd ..

Sending test events

If everything went smoothly, you can now send send events to your pipeline, which will land in the S3 Table. Therefore, you can either use the OwnStats client as stated in the Use Case description, or use provided script that produces the same request signature:

cd manual-setup

# Send event
npm run send-event -- "My Event" '{"testevent":true,"message":"foobar"}'

Querying the data in the S3 Table

Using Athena

You can query the data with Athena in the AWS Console. Please make sure that you select the correct catalog and database before you run a sample query on the event_data table:

SELECT * FROM event_data ORDER BY arrival_timestamp DESC

If everything worked, you should see your events in the result browser:

Using DuckDB

I recently wrote a blog post about querying S3 Tables with DuckDB, please have a look to learn how you can do it.

Verdict

Creating low-cost, pay-per-use event data pipelines which store their data in the Apache Iceberg format on AWS is possible. Currently, it still requires a lot of configuration efforts, and doesn’t support single-stack IaC, neither via CloudFormation nor the AWS CDK.

This is a mayor hurdle in the DX from my current point of view, especially in the light of the very recent product releases from Cloudflare on 2025-04-10:

While those are still beta releases, it’s visible that the innovation in the event processing / efficient Data Lake storage space is accelerating. Once Cloudflare enables that Pipelines can directly write Iceberg tables on R2, together with the R2 Data Catalog and the upcoming containers product (e.g. running DuckDB as an API), you can host your complete Lakehouse data stack on Cloudflare. With potentially much less complexity compared to AWS at the moment.

Also from a cost perspective, this is bound to gain some share of the market, especially for smaller, non-enterprise companies

Workers Paid ($5 / month)

Ingestion First 50 GB per month included

Delivery to R2 First 50 GB per month included

	Workers Paid ($5 / month)
Ingestion	First 50 GB per month included
Delivery to R2	First 50 GB per month included

In this scenario, you’d pay $5 for ingesting 50GB of data to R2, while on AWS you’d already have to pay $12.5 for the CloudWatch Logs Delivery to Kinesis Data Firehose only.

Also, there’s much less complexity from a IaC perspective compared to the pretty huge required infrastructure on AWS. I’m curious how / if AWS will react.

Query S3 Tables with DuckDB

Tobias Müller — Fri, 14 Mar 2025 18:35:31 GMT

DuckDB has gained a new feature in preview, that allows querying of Iceberg data in AWS S3 Tables.

Setting up a S3 Table

There are multiple steps which need to be performed to set up a S3 Table that can be then queried with tools like DuckDB. As the Infrastructure as Code (IaC) support from is not yet complete, a manual setup in the AWS Console will be used.

Creating a S3 Table Bucket

Open the AWS Console, and go to “Amazon S3” → “Table Buckets”, and then click on the “Create table bucket” button. Enter a name for the table bucket.

If you do this the first time, also activate the “Integration with AWS analytics services” for the specific region.

Press the “Create table bucket” button again, and the Table Bucket will be created.

Creating a S3 Table via Athena

As a next step, you can create a S3 Table with Athena. Click on the “Create table with Athena” button:

If you haven’t setup a Namespace yet, you need to do this in the next step, before you can actually create the S3 Table:

If you created a namespace before, you can select it and finally continue to the S3 Table creation, by clicking on “Create table with Athena”:

This will open a new tab with the Athena console:

💡

You need to create a Athena query result location before you can actually run queries, but this is not part of this blog post

For our testing purposes, we can just use the example table DDL that Athena provides. Select the CREATE TABLE statement, and click on the “Run” button:

Yay, we successfully created a new S3 Table with Iceberg support!

Inserting some test data via Athena

As we want to query some data from DuckDB later, we need to insert some sample data in our newly created table as well. Therefore, just uncomment the INSERT statement below the CREATE TABLE statement, and click on the “Run” button:

This takes surprisingly long for those few records, but if successful looks like this:

Now, we have established the infrastructure and data we need for actually start to query the S3 Table with DuckDB!

IAM permissions

But before, you need to make sure the the IAM role you’re planning to use for the test has the appropriate permissions. Otherwise you’ll not be able to perform the queries.

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3tables:*",
                "YOUR OTHER IAM permissions",
            ],
            "Resource": "*"
        }
    ]
}

The `s3tables:*` is the relevant part, as it will enable you to use all IAM actions respective to the s3tables service.

Going forward, it’s assumed that you have setup your AWS credentials on the machine you want to use DuckDB from to query the S3 Table, the details are not covered in this article, but a starting point in the docs is here.

Using the latest DuckDB version

Before continuing, please make sure that you have installed the version 1.2.1 of DuckDB. Otherwise this will not work!

Installing extensions

Open DuckDB from your terminal:

$ duckdb --version
v1.2.1 8e52ec4395

$ duckdb

The first step is to install the relevant extensions (aws, httpsfs and iceberg) in their nightly versions (latest):

FORCE INSTALL aws FROM core_nightly;
FORCE INSTALL httpfs FROM core_nightly;
FORCE INSTALL iceberg FROM core_nightly;

Creating a S3 secret

As the next step, you need to create a secret, so that DuckDB can automatically authenticate you via the credential chain provider (you need to have set this up beforehand, see above):

CREATE SECRET (
  TYPE s3,
  PROVIDER credential_chain
);

Attach the database

Next, we need to attach the actual remote database to our local DuckDB instance. Please make sure that the ARN is correct, you can copy the ARN from the Table Buckets details:

Replace your ARN in the statement below, and run it:

ATTACH 'arn:aws:s3tables:us-east-1:12345678912:bucket/duckdb-test'
  AS test_db (
    TYPE iceberg,
    ENDPOINT_TYPE s3_tables
);

Show all tables in the Table Bucket’s Namespace

Run the below statement, which will show you a list of your S3 Tables:

SHOW ALL TABLES;

┌──────────────┬─────────┬─────────────┬─────────────────────────────────────────────┬─────────────────────────┬───────────┐
│   database   │ schema  │    name     │                column_names                 │      column_types       │ temporary │
│   varchar    │ varchar │   varchar   │                  varchar[]                  │        varchar[]        │  boolean  │
├──────────────┼─────────┼─────────────┼─────────────────────────────────────────────┼─────────────────────────┼───────────┤
│ test_db      │ test    │ daily_sales │ [sale_date, product_category, sales_amount] │ [DATE, VARCHAR, DOUBLE] │ false     │
└──────────────┴─────────┴─────────────┴─────────────────────────────────────────────┴─────────────────────────┴───────────┘

Querying the S3 Table

Finally, you can now query your S3 Table:

select * from test_db.test.daily_sales;
┌────────────┬──────────────────┬──────────────┐
│ sale_date  │ product_category │ sales_amount │
│    date    │     varchar      │    double    │
├────────────┼──────────────────┼──────────────┤
│ 2024-01-15 │ Laptop           │        900.0 │
│ 2024-01-15 │ Monitor          │        250.0 │
│ 2024-01-16 │ Laptop           │       1350.0 │
│ 2024-02-01 │ Monitor          │        300.0 │
│ 2024-02-01 │ Keyboard         │         60.0 │
│ 2024-02-02 │ Mouse            │         25.0 │
│ 2024-02-02 │ Laptop           │       1050.0 │
│ 2024-02-03 │ Laptop           │       1200.0 │
│ 2024-02-03 │ Monitor          │        375.0 │
└────────────┴──────────────────┴──────────────┘

References

The following article is a great start:

DuckDB Blog: Amazon S3 Tables in DuckDB

There are some videos from the Amazon Pi Day where this was introduced:

The following PRs are relevant for this new feature

An test with some example code can be found under

test_glue.test

Querying IP addresses and CIDR ranges with DuckDB

Tobias Müller — Fri, 20 Sep 2024 22:00:00 GMT

I had a use case that eventually required performing IP address lookups in a given list of CIDR ranges, as I maintain an open source project that gathers IP address range data from public cloud providers, and also wrote an article in my blog about analyzing this data as well.

I had a look at the DuckDB docs, and found the inet extension that provides some functionality around IP addresses. Unfortunately, it offers no direct way to determine whether a given IP address is part of a CIDR range.

After some research, I found a StackOverflow answer that showed how this could be done with basic functions in Postgres. I decided to create three DuckDB macros to implement the needed IP address lookup functionality.

The plan

Basically, both the starting (network) and ending (broadcast) IP addresses of a CIDR range need to be cast to integers, to be able to determine if a given IP address (also cast to an integer) lies within the derived integer value boundaries.

Deriving the network address value

CREATE OR REPLACE MACRO network_from_cidr(cidr_range) AS (
    cast(string_split(string_split(cidr_range, '/')[1], '.')[1] as bigint) * (256 * 256 * 256) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[2] as bigint) * (256 * 256      ) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[3] as bigint) * (256            ) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[4] as bigint)
);

Deriving the broadcast address value

CREATE OR REPLACE MACRO broadcast_from_cidr(cidr_range) AS (
    cast(string_split(string_split(cidr_range, '/')[1], '.')[1] as bigint) * (256 * 256 * 256) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[2] as bigint) * (256 * 256      ) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[3] as bigint) * (256            ) +
    cast(string_split(string_split(cidr_range, '/')[1], '.')[4] as bigint)) + 
    cast(pow(256, (32 - cast(string_split(cidr_range, '/')[2] as bigint)) / 8) - 1 as bigint
);

Final macro for the IP address lookup

CREATE OR REPLACE MACRO ip_within_cidr(ip, cidr_range) AS (
    network_from_cidr(ip || '/32') >= network_from_cidr(cidr_range) AND network_from_cidr(ip || '/32') <= broadcast_from_cidr(cidr_range)
);

Sample SQLs

-- Transform to network start address
select network_from_cidr('4.0.0.0/8')
-- Transform to broadcast address
select broadcast_from_cidr('4.0.0.0/8')
-- Checks
select ip_within_cidr('4.0.0.0', '4.0.0.0/8') -- true
select ip_within_cidr('4.255.255.255', '4.0.0.0/8') -- true
select ip_within_cidr('3.255.255.255', '4.0.0.0/8') -- false
select ip_within_cidr('5.0.0.0', '4.0.0.0/8') -- false

Try it out yourself

You can try the described functionalities in my free online SQL Workbench by clicking on this link. It will open a new browser window, and create all three macros and run a test query accordingly.

Chat with a Duck

Tobias Müller — Tue, 16 Apr 2024 23:24:58 GMT

A while ago I published sql-workbench.com and the accompanying blog post called "Using DuckDB-WASM for in-browser Data Engineering". The SQL Workbench enables its users to analyze local or remote data directly in the browser.

This lowers the bar regarding the infrastructure needed to get started with Data Analysis and Data Engineering. No databases must be installed on servers or developer machines, no data is sent to the internet and SaaS or Cloud providers. The sole interface to the data you want to analyze is SQL.

But what if a user is new in this space, and has no SQL skills yet? Or, if he/she has SQL skills, but wants to query the data without having to understand the data model first?

Enter DuckDB-NSQL

LLMs (Large Language Models) can be used to generate code from natural language questions. Popular examples include GitHub Copilot, OpenAI's GPT-4 or Meta's Code Llama.

DuckDB-NSQL is a Text-to-SQL model created by NumberStation for MotherDuck. It's hosted on HuggingFace and on Ollama, and can be use with different LLM runtimes. There are also some nice blog posts that are worth a read:

The model was specifically trained for the DuckDB SQL syntax with 200k DuckDB Text-to-SQL pairs, and is based on the Llama-2 7B model.

Bring your own AI

If you want to enable SQL Workbench's privacy first AI integration for SQL generation, you first have to install Ollama on your local machine.

Once you installed Ollama, you can either download the relevant DuckDB-NSQL model beforehand, or have it automatically downloaded on the first usage.

If you want to pull the model yourself, you can do this in your terminal (after you installed Ollama) by issuing the following command:

ollama pull duckdb-nsql:7b

Please be aware that the default model has a size of 3.8GB, which can take a while to download, depending on your internet connection speed. There are smaller quantized models as well, but be aware that the answer quality might be lower with them.

Once the model is downloaded, Ollama can be started from your terminal:

OLLAMA_ORIGINS="https://sql-workbench.com" ollama serve

Setting the OLLAMA_ORIGINS environment variable to https://sql-workbench.com is necessary to enable CORS from the SQL Workbench running in your browser for your locally running Ollama server.

You can enable the AI feature in SQL Workbench:

Then, you can ask your questions after a specific comment string like below:

--ai your natural language question

To execute the prompt, you have two options:

Press ALT + g to generate the SQL
Press ALT + r to run the generated SQL directly

So if you created a table named "locations" beforehand, that has a column named "country", the following would generate an appropriate SQL:

--ai distinct country from locations
SELECT distinct country FROM locations

The generated SQL is automatically inserted below the closest prompt comment string. In case you have multiple comment strings in the current SQL Workbench tab, the one closest to the actual cursor position is used.

You can also ask questions regarding about remote files (Parquet, CSV), but don't expect the answer quality to be very good, because the schema will be unknown to the model.

Explore AWS IAM data with the help of AI

AWS publishes its Service Authorization Reference documentation, and there's a Github repository that transforms the published data automatically to Parquet, CSV and JSON data formats every night at 4AM UTC.

Each Parquet file represents a table in a relational data model that looks like this:

With the help of SQL Workbench and its underlying DuckDB-WASM instance, it's possible to load the remote data from GitHub into our browser's memory.

To do this, the following SQL statements need to be executed, either by copy & pasting the code, or automatically via the SQL Workbench's shareable query feature by clicking on the duck below (this will open in a new browser window/tab, redirected via dub.co):

The create schema will then be used as input for the prompt which is sent to Ollama in the background by the browser.

SQL script


CREATE TABLE services (
  service_id INTEGER PRIMARY KEY, 
  "name" VARCHAR, 
  prefix VARCHAR, 
  reference_url VARCHAR 
);

CREATE TABLE actions (
  action_id INTEGER PRIMARY KEY, 
  service_id INTEGER, 
  "name" VARCHAR, 
  reference_url VARCHAR, 
  permission_only_flag BOOLEAN, 
  access_level VARCHAR,
  FOREIGN KEY (service_id) REFERENCES services (service_id)
);

CREATE TABLE condition_keys (
  condition_key_id INTEGER PRIMARY KEY, 
  "name" VARCHAR, 
  reference_url VARCHAR, 
  description VARCHAR, 
  "type" VARCHAR
);

CREATE TABLE resource_types (
  resource_type_id INTEGER PRIMARY KEY, 
  service_id INTEGER, 
  "name" VARCHAR, 
  reference_url VARCHAR, 
  arn_pattern VARCHAR,
  FOREIGN KEY (service_id) REFERENCES services (service_id)
);

CREATE TABLE resource_types_condition_keys (
  resource_type_condition_key_id INTEGER PRIMARY KEY, 
  resource_type_id INTEGER, 
  condition_key_id INTEGER,
  FOREIGN KEY (resource_type_id) REFERENCES resource_types (resource_type_id),
  FOREIGN KEY (condition_key_id) REFERENCES condition_keys (condition_key_id)
);

CREATE TABLE actions_resource_types (
  action_resource_type_id BIGINT PRIMARY KEY, 
  action_id INTEGER, 
  resource_type_id INTEGER, 
  required_flag BOOLEAN,
  FOREIGN KEY (action_id) REFERENCES actions (action_id)
);

CREATE TABLE actions_condition_keys (
  action_condition_key_id BIGINT PRIMARY KEY, 
  action_resource_type_id BIGINT, 
  action_id INTEGER, 
  condition_key_id INTEGER,
  FOREIGN KEY (action_id) REFERENCES actions (action_id),
  FOREIGN KEY (condition_key_id) REFERENCES condition_keys (condition_key_id)
);

CREATE TABLE actions_dependant_actions (
  action_dependent_action_id INTEGER PRIMARY KEY, 
  action_resource_type_id BIGINT, 
  action_id INTEGER, 
  dependent_action_id INTEGER,
  FOREIGN KEY (action_id) REFERENCES actions (action_id),
  FOREIGN KEY (action_resource_type_id) REFERENCES actions_resource_types (action_resource_type_id)
);

INSERT INTO services SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_services.parquet';
INSERT INTO resource_types SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_resource_types.parquet';
INSERT INTO condition_keys SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_condition_keys.parquet';
INSERT INTO actions SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_actions.parquet';
INSERT INTO resource_types_condition_keys SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_resource_types_condition_keys.parquet';
INSERT INTO actions_resource_types SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_actions_resource_types.parquet';
INSERT INTO actions_condition_keys SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_actions_condition_keys.parquet';
INSERT INTO actions_dependant_actions SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/parquet/aws_actions_dependant_actions.parquet';

Example prompts & results

Prompt

--ai distinct service names that contain 'S3'

Result

Prompt

--ai count distinct action names for service name contains S3

Result

Prompt

--ai show all 'Write' access level action for service 'Amazon S3'

Result

Prompt

--ai first 10 actions for service 'Amazon CloudFront'

Result

Prompt

--ai service name with least actions

Result

Prompt

--ai service name with most resource types

Result

Prompt

--ai count actions names of service 'Amazon S3'

Result

Prompt

--ai top 10 resource types by services

Result

Demo video

https://youtu.be/rTuCec_fhlk

Summary

Using a locally hosted LLM together with an in-browser SQL Workspace enables a cost-effective and privacy-friendly way to use state of the art tools without needing to rely on third party services.

Using DuckDB-WASM for in-browser Data Engineering

Tobias Müller — Sat, 27 Jan 2024 23:00:00 GMT

Introduction

DuckDB, the in-process DBMS specialized in OLAP workloads, had a very rapid growth during the last year, both in functionality, but also popularity amongst its users, but also with developers that contribute many projects to the Open Source DuckDB ecosystem.

DuckDB cannot "only" be run on a variety of Operating Systems and Architectures, there's also a DuckDB-WASM version, that allows running DuckDB in a browser. This opens up some very interesting use cases, and is also gaining a lot of traction in the last 12 months.

Use Case: Building a SQL Workbench with DuckDB-WASM

One of the first things that came to my mind once I learned about the existence of DuckDB-WASM was that it could be used to create an online SQL Workbench, where people could interactively run queries, show their results, but also visualize them. DuckDB-WASM sits at its core, providing the storage layer, query engine and many things more...

You can find the project at

It's built with the following core technologies / frameworks:

It's hosted as a static website export / single page application on AWS using

CloudFront as CDN
S3 as file hosting service
ACM for managing certificates
Route53 for DNS

If you're interested in the hosting setup, you can have a look at https://github.com/tobilg/serverless-aws-static-websites which can deploy such static websites on AWS via IaC with minimum effort.

Using the SQL Workbench

There are many possibilities how you can use the SQL Workbench, some are described below

Overview

When you open sql-workbench.com for the first time, you can see that the workbench is divided in three different areas:

On the left, there's the "Local Tables" area, that will display the created tables of you ran queries such as CREATE TABLE names (name VARCHAR), or used the drag-and-drop area on the lower left corner to drop any CSV, Parquet or Arrow file on it (details see below).
The upper main editor area is the SQL editor, where you can type your SQL queries. You're already presented with some example queries for different types of data once the page is loaded.
The lower main result area where the results of the ran queries will be shown, or alternatively, the visualizations of these results.

💡

You can adjust the respective heights of the main areas by dragging the lever in the middle.

Running SQL queries

To run your first query, select the first line of SQL, either with your keyboard or with your mouse, and press the key combination CMD + Enter of you're on a Mac, or Ctrl + Enter if you're on a Windows or Linux machine.

The result of the query that was executed can then be found in the lower main area as a table:

💡

Queries will only be executed if one or more queries are selected. If multiple queries shall be executed, make sure you use a semicolon at the end of each query. Otherwise an error will be displayed.

Running multiple queries

You can also run multiple queries sequentially, e.g. to create a table, insert some records, and display the results:

CREATE TABLE first_names (name VARCHAR, birth_cnt integer);

INSERT INTO first_names (name, birth_cnt) VALUES ('Liam', 20456);
INSERT INTO first_names (name, birth_cnt) VALUES ('Noah', 18621);
INSERT INTO first_names (name, birth_cnt) VALUES ('Oliver', 15076);
INSERT INTO first_names (name, birth_cnt) VALUES ('James', 12028);
INSERT INTO first_names (name, birth_cnt) VALUES ('Elijah', 11979);
INSERT INTO first_names (name, birth_cnt) VALUES ('William', 11282);
INSERT INTO first_names (name, birth_cnt) VALUES ('Henry', 11221);
INSERT INTO first_names (name, birth_cnt) VALUES ('Lucas', 10909);
INSERT INTO first_names (name, birth_cnt) VALUES ('Benjamin', 10842);
INSERT INTO first_names (name, birth_cnt) VALUES ('Theodore', 10754);

SELECT * FROM first_names;

When you copy & paste the above SQLs, select them and run them, the result looks like this:

You can see on the left-hand side the newly created table first_names, that can be reused for other queries without having to reload the data again.

💡

Only the result of the last-run query will be displayed in the lower main result area!

If you want to open a new SQL Workbench and directly run the above query, please click on the image below:

💡

The data is persisted until you reload the overall SQL Workbench page.

Querying data you have on your machine

To try this, you can for example download a list of AWS Services as a CSV from

https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/csv/aws_services.csv

This file has four columns, service_id, name, prefix and reference_url. Once you downloaded the file, you can simply drag-and-drop from the folder you downloaded it to to the area in the lower left corner of the SQL Workbench:

A table called aws_services.csv has now been automatically created, which you can query via SQLs, for example:

SELECT name, prefix from 'aws_services.csv';

If you want to open a new SQL Workbench and directly run the above query, please click on the image below:

Querying and visualizing remote data

DuckDB-WASM supports the loading of compatible data in different formats (e.g. CSV, Parquet or Arrow) from remote http(s) sources. Other data formats that can be used include JSON, but this requires the loading of so-called DuckDB extensions.

💡

It's necessary that the websites hosting the data add the relevant CORS headers, otherwise the browser (not DuckDB-WASM or the SQL Workbench) will forbid the loading of the files and show an error message instead

In this example, we will use data about AWS CloudFront Edge Locations, that is available at tobilg/aws-edge-locations with this query:

SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-edge-locations/main/data/aws-edge-locations.parquet';

The result will look like this:

We now want to create a bar chart of the data, showing the number of Edge Locations by country and city. This can be done by hovering over the result table, and clicking on the small "configure" button that looks like a wrench which subsequently appears on the upper right corner of the table:

You then see the overview of the available columns, and the current visualization type (Datagrid)

To get an overview of the possible visualization types click on the Datagrid icon:

Then select "Y Bar". This will give you an initial bar char:

But as we want to display the count of Edge Locations by country and city, we need to drag-and-drop the columns country and city to the "Group By" area:

We can now close the configuration menu to see the chart in it's full size:

There are many other visualization types from which you can choose from, such as Treemaps and Sunbursts, as well as Map Scatters:

Exporting visualizations and data

You can also export the visualizations, as well as the data. Just click on "Export" and type in a "Save as" name, and select the output format you want to download:

The data can be downloaded as CSV, JSON or Arrow file. Here's the CSV example:

"country (Group by 1)","count"
,517
"Argentina",3
"Australia",10
"Austria",3
"Bahrain",2
"Belgium",1
"Brazil",21
"Bulgaria",3
"Canada",8
"Chile",6
"China",8
"Colombia",3
"Croatia",1
"Czech Republic",1
"Denmark",3
"Finland",4
"France",17
"Germany",37
"Greece",1
"Hungary",1
"India",48
"Indonesia",5
"Ireland",2
"Israel",2
"Italy",16
"Japan",27
"Kenya",1
"Korea",8
"Malaysia",2
"Mexico",4
"Netherlands",5
"New Zealand",2
"Nigeria",1
"Norway",2
"Oman",1
"Peru",2
"Philippines",2
"Poland",5
"Portugal",1
"Romania",1
"Singapore",7
"South Africa",2
"Spain",12
"Sweden",4
"Switzerland",2
"Taiwan",3
"Thailand",2
"UAE",4
"UK",30
"United States",179
"Vietnam",2

And here's the exported PNG:

Exporting the HTML version will give you an interactive graph with hovering etc. Furthermore, you can also change the theme for the different visualizations:

This is also reflected in the exported graphs:

Using the schema browser

The schema browser can be found on the left-hand side. It's automatically updated after each executed query, so that all schema operations can be captured. On table-level, the columns, constraints and indexes are shown:

If you right-click on a table name, a context menu is shown that has different options:

Generating scripts based on the table definition
Truncating, deleting or summarizing the table
Viewing table data (all records, first 10 and first 100)

Once clicked, those menu items will create a new tab (see below), and generate and execute the appropriate SQL statements:

Using query tabs

Another new feature is the possibility to have multiple query tabs. Those are either automatically created by context menu actions, or the user that clicks on the plus icon:

Each tab can be closed by clicking on the "x" icon next to the tab name.

Generating data models

If users have created some tables, it's then possible to create a data model from the schema metadata. If the tables also have foreign key relationships, those are also shown in the diagram. Just click on the "Data Model" menu entry on the lower left corner.

Under the hood, this feature generates Mermaid Entity Relationship Diagram code, that is dynamically rendered as a graph.

Using the query history

Each query that is issued in the current version of the SQL Workbench is recorded for the so-called query history. It can be accessed by clicking on the "Query History" menu entry in the lower left corner. Once clicked, there's an overlay on the right-hand side with the list of the issued queries.

The newest queries can be found on top of the list, and with each query listed, there's also an indication when the query was run, and how long it took to execute.

With the trash icon in the top-right corner, the complete query history can be truncated. Also, single query history entries can be deleted, as well as specific queries can be re-run in a new tab by clicking "Replay query" in the menu that's present for each query history entry.

Example Data Engineering pipeline

Dataset & Goals

A well-known dataset is the NYC TLC Trip Record dataset. It can be found freely available on the website of NYC Taxi and Limousine Commission website. It also comes with some explanations and additional lookup data. In this example, we focus on the yellow taxi data.

The goal of this example pipeline is to create a clean Data Mart from the given trip records and location data, being able to support some basic analysis of the data via OLAP patterns.

Source Data analysis

On the NYC TLC website, there's a PDF file explaining the structure and contents of the data. The table structure can be found below, the highlighted columns indicate dimensional values, for which we'll build dimension tables for in the later steps.

Column name	Description
VendorID	A code indicating the TPEP provider that provided the record.
tpep_pickup_datetime	The date and time when the meter was engaged.
tpep_dropoff_datetime	The date and time when the meter was disengaged.
Passenger_count	The number of passengers in the vehicle. This is a driver-entered value.
Trip_distance	The elapsed trip distance in miles reported by the taximeter.
PULocationID	TLC Taxi Zone in which the taximeter was engaged
DOLocationID	TLC Taxi Zone in which the taximeter was disengaged
RateCodeID	The final rate code in effect at the end of the trip.
Store_and_fwd_flag	This flag indicates whether the trip record was held in vehicle memory before sending to the vendor, aka “store and forward,” because the vehicle did not have a connection to the server.
Payment_type	A numeric code signifying how the passenger paid for the trip.
Fare_amount	The time-and-distance fare calculated by the meter.
Extra	Miscellaneous extras and surcharges. Currently, this only includes the $0.50 and $1 rush hour and overnight charges.
MTA_tax	$0.50 MTA tax that is automatically triggered based on the metered rate in use.
Improvement_surcharge	$0.30 improvement surcharge assessed trips at the flag drop. The improvement surcharge began being levied in 2015.
Tip_amount	Tip amount – This field is automatically populated for credit card tips. Cash tips are not included.
Tolls_amount	Total amount of all tolls paid in trip.
Total_amount	The total amount charged to passengers. Does not include cash tips.
Congestion_Surcharge	Total amount collected in trip for NYS congestion surcharge.
Airport_fee	$1.25 for pick up only at LaGuardia and John F. Kennedy Airports

There's an additional CSV file for the so-called Taxi Zones, as well as a SHX shapefile containing the same info, but with an additional geo information. The structure is the following:

Column name	Description
LocationID	TLC Taxi Zone, corresponding to the PULocationID and DOLocationID columns in the trip dataset
Borough	The name of the NYC borough this Taxi Zone is in
Zone	The name of the Taxi Zone
service_zone	Can either be "Yellow Zone" or "Boro Zone"

Target Data Model

The target data model is derived from the original trip record data, with extracted dimension tables plus a new date hierarchy dimension. Also, the naming schema gets unified and cleaned up.

It is modeled as a so-called Snowflake Schema (check the Mermaid source):

Loading & Transforming the data

The loading & transforming of the data is divided in multiple steps:

Generating the dimensional tables and values from the given dataset information
Generating a date hierarchy dimension
Loading and transforming the trip data
- Replacing values with dimension references
- Clean the column naming
- Unify values

Generating the dimensional tables

We use the given dataset information from the PDF file to manually create dimension tables and their values:

-- Install and load the spatial extension
INSTALL spatial;
LOAD spatial;

-- Create temporary table
CREATE TABLE tmp_service_zones AS (
    SELECT 
        DISTINCT service_zone 
    FROM 
        'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv' 
    WHERE 
        service_zone != 'N/A' 
    ORDER BY 
        service_zone
); 

-- Create dim_zone_type table
CREATE TABLE dim_zone_type (
    zone_type_id INTEGER PRIMARY KEY,
    name VARCHAR
);

-- Insert dim_zone_type table
INSERT INTO dim_zone_type
SELECT 
    -1 as zone_type_id, 
    'N/A' as name 
UNION ALL
SELECT 
    (rowid + 1)::INTEGER as zone_type_id, 
    service_zone as name 
FROM 
    tmp_service_zones 
; 

-- Drop table tmp_service_zones
DROP TABLE tmp_service_zones; 

-- Create temporary table
CREATE TABLE tmp_borough AS (
    SELECT 
        DISTINCT borough 
    FROM 
        'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv' 
    WHERE
        borough != 'Unknown'
    ORDER BY 
        borough
); 

-- Create dim_borough table
CREATE TABLE dim_borough (
    borough_id INTEGER PRIMARY KEY,
    name VARCHAR
);

-- Insert dim_borough table
INSERT INTO dim_borough
SELECT 
    -1 as borough_id, 
    'N/A' as name 
UNION ALL
SELECT 
    (rowid + 1)::INTEGER as borough_id, 
    borough as name 
FROM 
    tmp_borough 
; 

-- Drop temporary table
DROP TABLE tmp_borough;

-- Create dim_zone table
CREATE TABLE dim_zone (
    zone_id INTEGER PRIMARY KEY,
    zone_type_id INTEGER,
    borough_id INTEGER,
    name VARCHAR,
    geojson VARCHAR,
    FOREIGN KEY (zone_type_id) REFERENCES dim_zone_type (zone_type_id),
    FOREIGN KEY (borough_id) REFERENCES dim_borough (borough_id)
);

-- Insert dim_zone table
INSERT INTO dim_zone 
SELECT DISTINCT
    CASE
        WHEN csv.LocationID IS NOT NULL THEN csv.LocationID::INT
        ELSE raw.LocationID
    END AS zone_id, 
    zt.zone_type_id, 
    CASE
        WHEN b.borough_id IS NOT NULL THEN b.borough_id
        ELSE -1
    END AS borough_id,
    CASE
        WHEN csv.Zone IS NOT NULL THEN csv.Zone
        ELSE raw.zone
    END AS name, 
    raw.geojson 
FROM 
    (
        SELECT 
            LocationID, 
            borough,
            zone,  
            geojson 
        FROM 
            (
                SELECT 
                    LocationID, 
                    borough, 
                    zone, 
                    rank() OVER (PARTITION BY LocationID ORDER BY Shape_Leng) AS ranked, 
                    ST_AsGeoJSON(ST_Transform(geom, 'ESRI:102718', 'EPSG:4326')) AS geojson 
                FROM ST_Read('https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.shx') 
            ) sub
        WHERE 
            sub.ranked = 1 
    ) raw 
FULL OUTER JOIN 
    (
        SELECT DISTINCT 
            LocationID,
            Zone, 
            service_zone 
        FROM 
            'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv' 
    ) csv 
ON 
    csv.LocationId = raw.LocationId 
FULL OUTER JOIN 
    dim_zone_type zt 
ON 
    csv.service_zone = zt.name 
FULL OUTER JOIN 
    dim_borough b 
ON 
    b.name = raw.borough
WHERE
    zone_id IS NOT NULL
ORDER BY
    zone_id
;

-- Create dim_rate_code table
CREATE TABLE dim_rate_code (
    rate_code_id INTEGER PRIMARY KEY,
    name VARCHAR
);

INSERT INTO dim_rate_code (rate_code_id, name) VALUES (1, 'Standard rate');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (2, 'JFK');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (3, 'Newark');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (4, 'Nassau or Westchester');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (5, 'Negotiated fare');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (6, 'Group ride');
INSERT INTO dim_rate_code (rate_code_id, name) VALUES (99, 'N/A');

-- Create dim_payment_type table
CREATE TABLE dim_payment_type (
    payment_type_id INTEGER PRIMARY KEY,
    name VARCHAR
);

INSERT INTO dim_payment_type (payment_type_id, name) VALUES (1, 'Credit card');
INSERT INTO dim_payment_type (payment_type_id, name) VALUES (2, 'Cash');
INSERT INTO dim_payment_type (payment_type_id, name) VALUES (3, 'No charge');
INSERT INTO dim_payment_type (payment_type_id, name) VALUES (4, 'Dispute');
INSERT INTO dim_payment_type (payment_type_id, name) VALUES (5, 'Unknown');
INSERT INTO dim_payment_type (payment_type_id, name) VALUES (6, 'Voided trip');

-- Create dim_vendor table
CREATE TABLE dim_vendor (
    vendor_id INTEGER PRIMARY KEY,
    name VARCHAR
);

INSERT INTO dim_vendor (vendor_id, name) VALUES (1, 'Creative Mobile Technologies');
INSERT INTO dim_vendor (vendor_id, name) VALUES (2, 'VeriFone Inc.');

-- Create dim_stored_type table
CREATE TABLE dim_stored_type (
    stored_type_id INTEGER PRIMARY KEY,
    name VARCHAR
);

INSERT INTO dim_stored_type (stored_type_id, name) VALUES (1, 'Store and forward trip');
INSERT INTO dim_stored_type (stored_type_id, name) VALUES (2, 'Not a store and forward trip');

Generating a date hierarchy dimension

-- Create dim_date table
CREATE TABLE dim_date (
    day_dt DATE PRIMARY KEY,
    day_name VARCHAR,
    day_of_week INT,
    day_of_month INT,
    day_of_year INT,
    week_of_year INT,
    month_of_year INT,
    month_name VARCHAR,
    year INT
);

INSERT INTO dim_date
SELECT 
    date_key AS day_dt,
    DAYNAME(date_key)::VARCHAR AS day_name,
    ISODOW(date_key)::INT AS day_of_week,
    DAYOFMONTH(date_key)::INT AS day_of_month,
    DAYOFYEAR(date_key)::INT AS day_of_year, 
    WEEKOFYEAR(date_key)::INT AS week_of_year,
    MONTH(date_key)::INT AS month_of_year,
    MONTHNAME(date_key)::VARCHAR AS month_name,
    YEAR(date_key)::INT AS year
FROM 
    (
        SELECT 
            CAST(RANGE AS DATE) AS date_key 
        FROM 
            RANGE(DATE '2005-01-01', DATE '2030-12-31', INTERVAL 1 DAY)
    ) generate_date
;

Loading and transforming the trip data

-- Create sequence for generating trip_ids
CREATE SEQUENCE trip_id_sequence START 1;

-- Create fact_trip table
CREATE TABLE fact_trip (
    trip_id INTEGER  DEFAULT nextval('trip_id_sequence') PRIMARY KEY,
    pickup_zone_id INTEGER,
    pickup_dt DATE,
    pickup_ts TIMESTAMP,
    dropoff_zone_id INTEGER,
    dropoff_dt DATE,
    dropoff_ts TIMESTAMP,
    rate_code_id INTEGER, 
    stored_type_id INTEGER, 
    payment_type_id INTEGER, 
    vendor_id INTEGER, 
    passenger_count DOUBLE,
    trip_distance_miles DOUBLE,
    fare_amount DOUBLE,
    extra_amount DOUBLE,
    mta_tax_amount DOUBLE,
    improvement_surcharge_amount DOUBLE,
    tip_amount DOUBLE,
    tolls_amount DOUBLE,
    congestion_surcharge_amount DOUBLE,
    airport_fee_amount DOUBLE,
    total_amount DOUBLE
);

-- Deactivating FK relationships for now, due to performance issues when inserting 3 million records
-- FOREIGN KEY (pickup_zone_id) REFERENCES dim_zone (zone_id),
-- FOREIGN KEY (dropoff_zone_id) REFERENCES dim_zone (zone_id),
-- FOREIGN KEY (pickup_dt) REFERENCES dim_date (day_dt),
-- FOREIGN KEY (dropoff_dt) REFERENCES dim_date (day_dt),
-- FOREIGN KEY (rate_code_id) REFERENCES dim_rate_code (rate_code_id),
-- FOREIGN KEY (stored_type_id) REFERENCES dim_stored_type (stored_type_id),
-- FOREIGN KEY (payment_type_id) REFERENCES dim_payment_type (payment_type_id),
-- FOREIGN KEY (vendor_id) REFERENCES dim_vendor (vendor_id)

-- Insert transformed fact data
INSERT INTO fact_trip
SELECT 
    nextval('trip_id_sequence') AS trip_id,
    PULocationID::INT as pickup_zone_id,
    tpep_pickup_datetime::DATE as pickup_dt,
    tpep_pickup_datetime AS pickup_ts,
    DOLocationID::INT as dropoff_zone_id,
    tpep_dropoff_datetime::DATE as dropoff_dt,
    tpep_dropoff_datetime AS dropoff_ts,
    RatecodeID::INT AS rate_code_id,
    CASE
        WHEN store_and_fwd_flag = 'Y' THEN 1
        WHEN store_and_fwd_flag = 'N' THEN 2
    END AS stored_type_id,
    payment_type::INT AS payment_type_id,
    VendorID::INT AS vendor_id,
    passenger_count,
    trip_distance AS trip_distance_miles,
    fare_amount,
    extra AS extra_amount,
    mta_tax AS mta_tax_amount,
    improvement_surcharge AS improvement_surcharge_amount,
    tip_amount,
    tolls_amount,
    congestion_surcharge AS congestion_surcharge_amount,
    airport_fee AS airport_fee_amount,
    total_amount
FROM 
    'https://data.quacking.cloud/nyc-taxi-data/yellow_tripdata_2023-01.parquet'
;

Data Analysis

The following analyses are just examples on how you could analyze the data set. Feel free to think about your own questions for the dataset, and try to build queries yourselves!

Preparation

DuckDB supports the SUMMARIZE command can help you understand the final data in the fact table before querying it. It launches a query that computes a number of aggregates over all columns, including min, max, avg, std and approx_unique:

The output already shows some "interesting" things, such as pickup_dt being far in the past, e.g. 2008-12-31, and the dropoff_dt with similar values (2009-01-01).

Most utilized trip locations

With this analysis, we want to have a look at the 20 most frequented trips from pickup zone to dropoff zone:

SELECT
    pz.name || ' -> ' || dz.name AS trip_description,
    count(DISTINCT ft.trip_id)::INT AS trip_count,
    sum(ft.passenger_count)::INT AS passenger_count,
    sum(ft.total_amount)::INT AS total_amount,
    sum(ft.trip_distance_miles)::INT AS trip_distance_miles,
    (sum(trip_distance_miles)/count(DISTINCT ft.trip_id))::DOUBLE AS trip_distance_miles_avg,
    (sum(ft.total_amount)/count(DISTINCT ft.trip_id))::DOUBLE AS total_amount_avg,
    (sum(ft.passenger_count)/count(DISTINCT ft.trip_id))::DOUBLE AS passenger_count_avg
FROM
    fact_trip ft
INNER JOIN
    dim_zone pz
ON
    pz.zone_id = ft.pickup_zone_id
INNER JOIN
    dim_zone dz
ON
    dz.zone_id = ft.dropoff_zone_id
GROUP BY
    trip_description
ORDER BY
    trip_count DESC
LIMIT 20;

On a M2 Mac Mini with 16GB RAM, aggregating the 3 million trips takes around 900ms. The result looks like the following:

We can now also create a Y Bar chart showing the (total) trip count, passenger count and trip distance by top 20 most frequented trips:

End result:

Trip frequency by weekday and time of day

To inspect the traffic patterns, we want to analyze the trip frequency by weekday and time of day (aggregated on hourly level). Therefore, we make use of DuckDB's advanced timestamp/time handling functions:

SELECT
    dd.day_name,
    dd.day_of_week,
    datepart('hour', time_bucket(INTERVAL '1 HOUR', ft.pickup_ts)) day_hour,
    count(DISTINCT ft.trip_id)::INT AS trip_count,
FROM
    fact_trip ft
INNER JOIN
    dim_date dd
ON
    dd.day_dt = ft.pickup_dt
GROUP BY
    dd.day_name,
    dd.day_of_week,
    day_hour
ORDER BY
    dd.day_of_week,
    day_hour;

Then, we can configure a Y Bar chart that can show us the number of trips by weekday and hour:

End result:

With the latest version of sql-workbench.com it's possible to share both queries and the (customized) visualization of the last executed query. Therefore, you write your queries, run them to check whether they work, and then update the visualization configuration.

When you did that, you can then click on "Share queries" in the lower left corner of the SQL Workbench. The toggle will let you choose whether you want to copy the visualization configuration as well, or not.

If you want to run the complete pipeline to build our dimensional data model, you can click on the link below (this can take from 10 to 120 seconds depending on your machine and internet connection speed, as approximately 50MB of data will be downloaded):

💡

The above link unfortunately has to be routed through https://dub.co as URL shortener, because HashNode doesn't support very long URLs as links!

Attaching remote databases

Since DuckDB v1.0.0, it is possible to attach a remote database via HTTPS or S3. As an example, you could use the below statements to load and query a remote dataset of AWS IAM data from GitHub:

ATTACH 'https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/db/iam.duckdb' AS iam (READ_ONLY);

SELECT 
  s.name as service_name, 
  count(distinct a.name)::int action_cnt 
FROM 
  iam.services s 
INNER JOIN 
  iam.actions a ON s.service_id = a.service_id 
GROUP BY ALL 
ORDER BY action_cnt desc 
LIMIT 25;

Click on the image below to execute the queries:

Conclusion

With DuckDB-WASM and some common web frameworks, it's pretty easy and fast to create custom data applications that can handle datasets in the size of millions of records.

Those applications are able to provide a very lightweight approach to working with different types of data (such as Parquet, Iceberg, Arrow, CSV, JSON or spatial data formats), whether locally or remote (via HTTP(S) of S3-compatible storage services), thanks to the versatile DuckDB engine.

Users can interactively work with the data, create data pipelines by using raw SQL, and iterate until the final desired state has been achieved. The generated data pipeline queries can easily be shared with simple links to sql-workbench.com, so that other collaborators can continue to iterate on the existing work, or even create new solutions with it.

Once a data pipeline has been finalized, it could for example be deployed to DuckDB instances running in own cloud accounts of the users. A great example would be running DuckDB in AWS Lambda, e.g. for repartitioning Parquet data in S3 nightly, or automatically running reports based on aggregation pipelines etc.

The possibilities are nearly endless, so I'm very curious what you all build with this great technology! Thanks for reading this length article, I'm happy to answer any questions in the comments.

Retrieving Lambda@Edge CloudWatch Logs

Tobias Müller — Fri, 26 Jan 2024 20:11:22 GMT

What is Lambda@Edge

AWS Lambda@Edge is an extension of the traditional AWS Lambda service, but with a crucial twist – it brings serverless computing capabilities closer to the end-users.

In essence, Lambda@Edge empowers developers to run custom code in response to specific CloudFront events. Whether it's tailoring content based on user location, device type, or handling real-time data processing at the edge, Lambda@Edge functions open up a world of possibilities for optimizing content delivery.

Lambda@Edge functions have the following features:

They can access public internet
They can be run before or after your cache
They allow developers to view and modify not only the client request/response but also the origin request/response
They can read the body of the request
They can use a much bigger package size for code compared to CloudFront Functions (1MB for a client request/response trigger and 50MB for an origin request/response trigger)
They allow up to 5 seconds for a client request/response trigger and up to 30 seconds for an origin request/response trigger.

How can you retrieve the Lambda@Edge logs?

Due to the fact that Lambda@Edge functions run in the CloudFront Regional Edge Caches, the CloudWatch logs cannot only be found in us-east-1, but potentially in all other AWS regions that support Regional Edge Caches.

The Lambda@Edge testing and debugging guide states

When you review CloudWatch log files or metrics when you're troubleshooting errors, be aware that they are displayed or stored in the Region closest to the location where the function executed. So, if you have a website or web application with users in the United Kingdom, and you have a Lambda function associated with your distribution, for example, you must change the Region to view the CloudWatch metrics or log files for the London AWS Region.

So, one simple solution to automatically get the logs from all regions would be to write a script that searches for relevant CloudWatch Logs groups in each region, and displays their respective LogStream entries.

This script could look like this

#!/bin/bash

FUNCTION_NAME=$1
for region in $(aws --output text ec2 describe-regions | cut -f 4) 
do
  echo "Checking $region"
  for loggroup in $(aws --output text logs describe-log-groups --log-group-prefix "/aws/lambda/us-east-1.$FUNCTION_NAME" --region $region --query 'logGroups[].logGroupName')
  do
    echo "Found '$loggroup' in region $region"
    for logstream in $(aws --output text logs describe-log-streams --log-group-name $loggroup --region $region --query 'logStreams[].logStreamName')
    do
      aws --output text logs get-log-events --log-group-name $loggroup --region $region --log-stream-name $logstream | cat
    done
  done
done

If you save this script as cf-logs.sh, after giving it execution rights with chmod +x cf-logs.sh, can then be started with ./cf-logs.sh YOUR_FUNCTION_NAME, where YOUR_FUNCTION_NAME is the real Lambda@Edge function name.

This solution is great for quick log viewing while you're still developing your edge-enabled application, but surely is not a sustainable one when running in production.

Aggregating Lambda@Edge logs with Kinesis

For a production setup, you'd probably want to be able to aggregate and store the individual logs coming from the different regions in one place. A possible solution is to stream them to a Kinesis Firehose Delivery Stream, which then stores the logs in S3.

An example implementation can be found at https://gist.github.com/heitorlessa/5d2295655f9d76483969d215986e53b0

Please be aware that this will incur additional fixed and variable costs, so please review AWS' pricing of the used services.

List of free AWS Knowledge Badges

Tobias Müller — Fri, 01 Sep 2023 10:32:10 GMT

As the Skillbuilder website is sometimes a bit hard to navigate, here's the full list of free badges you can do on AWS Skillbuilder:

Serverless Maps for fun and profit

Tobias Müller — Mon, 07 Aug 2023 13:10:07 GMT

Introduction

In today's data-driven world, interactive and visually appealing web-based maps have become an integral part of countless applications and services. Whether it's for navigation, location-based services, or data visualization, delivering seamless and lightning-fast user experiences is paramount. In this article, we explore how to host web-based maps on Amazon Web Services (AWS) in a serverless manner, exploring how this approach not only boosts speed but also brings significant cost advantages.

Web-Based Maps: A Key to Engaging User Experiences

The ubiquity of smartphones and the proliferation of internet-connected devices have transformed how we interact with the digital world. Whether we're exploring a new city, tracking our fitness activities, or monitoring real-time data, web-based maps provide a dynamic and immersive way to engage users. However, delivering smooth and responsive map experiences can be challenging, especially as data sizes and user demands continue to grow.

Traditionally, hosting web-based maps required dedicated servers or complex infrastructure setups, which often resulted in high operational costs and scalability limitations. Fortunately, the advent of serverless computing and the capabilities offered by AWS have strongly improved how we can deploy and manage web applications, including mapping services.

Serverless map hosting: A Game-Changer

AWS' serverless services allow developers to build and run applications without worrying about provisioning and managing servers. This approach not only simplifies development but also offers numerous advantages, particularly when it comes to hosting web-based maps.

Speed and Responsiveness: With serverless hosting via CloudFront and S3, the infrastructure caches data on the edge. This means that users will experience near-instantaneous map loading and seamless interactions, translating to heightened satisfaction and engagement.
Cost Efficiency: One of the most compelling aspects of the serverless approach is its cost-effectiveness. Traditional hosting methods often require paying for idle server resources, which can quickly add up and strain budgets. In contrast, serverless computing on AWS charges you only for the actual compute resources consumed during map requests. As a result, you can significantly reduce operational costs and allocate resources more efficiently.

In this comprehensive guide, we'll walk you through the steps of setting up a serverless web-based map hosting solution on AWS. From leveraging AWS Lambda for dynamic map tile querying to utilizing S3 for scalable and cost-effective data storage and using CloudFront for edge caching, you'll learn how to harness the full potential of AWS services for a top-notch mapping experience.

Choosing a map tile format

In the realm of digital maps, efficiency and performance are paramount. Whether you're navigating through city streets or exploring remote terrains, quick loading times and seamless zooming can make all the difference in delivering a superior user experience. Enter PMTiles, a cutting-edge map tile format that is redefining how we interact with digital maps.

What are PMTiles?

Traditional map tile formats, such as the popular PNG or JPEG images, can be bulky and slow to load, especially when dealing with intricate cartography or high-resolution imagery. PMTiles was specifically designed to address these limitations, offering a lightweight and performant solution for delivering map tiles.

According to the protomaps.com website

PMTiles is a single-file archive format for pyramids of tiled data. A PMTiles archive can be hosted on a storage platform like S3, and enables low-cost, zero-maintenance map applications.

Concepts include

A general format for tiled data addressable by Z/X/Y coordinates, which can be cartographic basemap vector tiles, remote sensing observations, JPEG images, or more.
Readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.
The arrangement of tiles and directories is designed to minimize the amount of overhead requests when panning and zooming.

The current specification of PMTiles v3 can be found on GitHub.

Why PMTiles matter?

Faster Loading Times: PMTiles leverages the power of vector data to store map tiles, which means that the tiles are smaller in size compared to raster image formats like PNG or JPEG. This smaller size translates to quicker loading times, enabling a seamless map browsing experience even in areas with limited network connectivity.
Efficient Storage: Due to their compact size, PMTiles require less storage space than traditional map tile formats. This advantage is particularly significant for applications with large map datasets, as it reduces the infrastructure costs associated with storing and serving the maps.
Dynamic Styling: PMTiles offer the flexibility of dynamic styling, allowing developers to modify the appearance of map tiles on the fly. By adjusting colors, labels, and map features in real-time, developers can create customized maps that cater to the specific needs of their users.
Offline Accessibility: One of the most impressive features of PMTiles is its ability to support offline access. By preloading PMTiles on a user's device, applications can ensure uninterrupted map access, even when an internet connection is not available. This functionality is invaluable for users in remote locations or areas with unstable network coverage. Also, usage of cache headers like Cache-Control, when used in a browser, can save bandwidth and rendering time.

How can PMTiles be used?

PMTiles can be used with common maps rendering engines like Leaflet, MapLibre GL or OpenLayers by using the protomaps.js library. Examples of how to integrate it can be found on the website, or the index.html and basemap.html examples in our repo.

Generating PMTiles from OpenStreetMap data

The first step for hosting your self-hosted maps can be downloading publically available map data from OpenStreetMap. This data is then to be transformed into a compatible basemap layer for use with PMTiles.

See the documentation about vector basemap layers to get an idea of how this works in general. The following is necessary to mention though (from protomaps.com):

The organization of features with layers and tags is specific to Protomaps services; this means that map styles are not directly portable with other systems such as OpenMapTiles or Mapzen tiles

This step can be automated to a high extent.

Solution

Enter ServerlessMaps! The project is outlined in the following paragraphs.

Architecture

In our example implementation, we will only rely on a very few services:

AWS Lambda (proxying the map tile requests to the S3 origin)
Amazon CloudFront (globally distributed CDN)
Amazon S3 (storing the PMTiles files, and the map example website)
Amazon CloudWatch (keeping the logs of the Lambda function)

The overall architecture looks like this:

Deployment

The deployment of ServerlessMaps must be done in multiple steps. From a high-level perspective, they are

Setting up the project from GitHub locally
Setting up the local environment to prepare the basemaps
Build the desired basemaps as PMTiles file
Deploy the serverless infrastructure on AWS
Upload the basemaps and example websites to S3 (done automatically)

Setting up the project locally

To clone the project to your local machine, please use

git clone https://github.com/serverlessmaps/serverlessmaps.git

in your desired directory. Then, do a cd serverlessmaps in the directory you ran the above command.

Setting up the local environment

This step assumes that you're using MacOS as the operating system. If so, you can run

scripts/install_macos.sh

to install the dependencies needed to build the basemaps as PMTiles file. This will install the latest OpenJDK and Maven, which are both necessary for the basemaps build.

If your system is not using MacOS, you can also install those two dependencies manually.

The next step is to compile the Planetiler build profile, that later can generate the PMTiles file:

scripts/compile_basemaps_builder.sh

This will create a new directory builder that will contain the JAR with the runnable builder.

Build the desired basemaps

To build a basemap with the before compiled builder, run

scripts/build_pmtiles.sh

This will build a map with a default area of Hamburg / Germany. If you want to generate maps for other areas, have a look at the OSM (sub-)region names, e.g. on the https://download.geofabrik.de/ server.

For example, if you'd like to generate a map for the whole of Europe, you could run

scripts/build_pmtiles.sh europe

Please be aware that this will run for several hours depending on your machine, and will generate a PMTiles file of around 45GB. This file will take some time to upload to S3 in the next step as well. The recommendation is if you just want to try out this project, use the default hamburg sub-region, which is around 35MB.

Deploy the serverless infrastructure

This project assumes that you already have set up your AWS credentials locally so that the Serverless framework can use it accordingly.

To deploy the serverless AWS infrastructure, you can do a cd iac from the project's root directory, and then use

sls deploy

to deploy the necessary stack.

You can customize some parameters for the deployment:

region: The AWS region you want to deploy your stack to (default: us-east-1)
stage: The stage name (default: prd)
cors: The allowed hostname for the CORS header (default: *)

The following will deploy the stack to the eu-central-1 region with the stage name dev and the allowed CORS hostname mymapservice.xyz:

sls deploy --region eu-central-1 --stage dev --cors mymapservice.xyz

Stack output

The deployment of the stack will generate an output like this on the console:

---------------------------------------------------------------------------------

-> The map can be viewed at https://d1b056iuztreqte.cloudfront.net/#13.54/53.54958/9.99286

-> The basemap themes can be viewed at https://d1b056iuztreqte.cloudfront.net/basemap.html#13.54/53.54958/9.99286

-> Please manually do a 'npm run website:sync' on your console to sync the static website assets if you changed them after the last deployment

-> Afterwards, run 'npm run website:invalidate' to invalidate the website's CloudFront distribution cache

---------------------------------------------------------------------------------

Automatically created files

There will be two automatically created files based on the setting you chose before:

website/urlConfig.js: This contains the CloudFront Distribution hostname (variable tilesDistributionHostname) for the caching of the PMTiles. This is assigned by CloudFront during deployment.
website/tilePath.js: This contains the needed tilePath variable, which depends on the area you chose for the basemap. This is generated by the scripts/build_pmtiles.sh script automatically.

Upload the basemaps and example websites

The basemap that was generated in the step before the deployment, and the two example websites are synched automatically to the website S3 bucket.

If you want to deploy your web application/website, you need to run the sync manually via npm run website:sync. After that, the CloudFront cache needs to be invalidated as well to show the new content. This can be done via npm run website:invalidate, both from the iac directory.

Result

If everything went well, you can access the URL (https://d1b056iuztreqte.cloudfront.net/#13.54/53.54958/9.99286 in the above example output) to view your basic map:

Conclusion

Hosting web-based maps on AWS in a serverless manner opens up a world of opportunities for delivering blazing-fast, interactive, and cost-efficient map services. By taking advantage of AWS's scalable infrastructure and pay-as-you-go pricing model, you can create an outstanding user experience without breaking the bank.

Gathering and analyzing public cloud provider IP address data with DuckDB & Observerable

Tobias Müller — Wed, 26 Apr 2023 16:06:39 GMT

As organizations increasingly adopt the public cloud, managing the networking and security aspects of cloud computing becomes more complex. One of the challenges that cloud administrators face is, especially in a hybrid cloud environment, keeping track of the IP address ranges of the public cloud providers, which all use different file formats to publish their IP address range data. The formats include deeply nested JSONs, CSVs as well as plain text.

The goal of this article is to outline how this data can be unified, cleaned and made available on a platform that makes it easy for users to consume. Furthermore, some interesting statistics can be derived from those public datasets.

The data and the source code can be found at:

https://github.com/tobilg/public-cloud-provider-ip-ranges

Data sources

The (incomplete) list of public cloud providers that are publishing their IPv4 CIDR blocks are:

You can click on the individual links to view or download the data manually.

HINT: The published lists of IP address ranges don't represent the overall IP address space that each of these providers are possessing.

To get the complete IP range, data from organizations like ARIN would need to be added as well. For simplicity and brevity, only the publically downloadable info was used.

Retrieving, cleaning, storing and exporting data

The overall data engineering process is divided into multiple steps:

Identify data sources (see above) and define the common schema
Retrieving the raw data from public data sources (via HTTP)
Cleaning the retrieved data (e.g. remove duplicates)
Storing the data in a common schema, so that it can be aggregated and analyzed for different public cloud providers at once
Exporting the stored data into different file formats, so that as many different types of clients can make use of it

Additionally, we'd like to keep the costs low, and the infrastructure as simple as possible. That's why DuckDB is chosen as the database layer, which offers a rich and advanced set of features to handle (read and write) different file formats, as well as it can read directly from remote data sources via HTTP, only by using SQL. That saves additional effort for out-of-band ETL.

Furthermore, to share the data, we chose GitHub, which is free to use for the scope of our use case. Most importantly, it allows us to store the exported data files in our repository. To run the overall process, GitHub Actions are used as they also offer a free usage tier, and have everything we need to create the described data pipeline.

Common data schema

After inspecting the data source files, the derived unified schema for all loaded data sources will look like this:

Column name	Data type	Description
cloud_provider	VARCHAR	The public cloud provider's name
cidr_block	VARCHAR	The CIDR block, e.g. `10.0.0.0/32`
ip_address	VARCHAR	The IP address, e.g. `10.0.0.0`
ip_address_mask	INTEGER	The IP address mask, e.g. `32`
ip_address_cnt	INTEGER	The number of IP addresses in this CIDR block
region	VARCHAR	The public cloud provider region information (if given)

Creating the cloud provider tables

At first, we'll create a table in DuckDB for each of the public cloud providers. If DuckDB is installed (see docs) and in the PATH, we can execute SQL scripts like this:

# $DATA_PATH is the location of the DuckDB database file (this is important, because otherwise an in-memory table will be automatically created that will not be able to persist the data
# $SCRIPT is the path to the SQL script that shall be executed
duckdb $DATA_PATH < $SCRIPT.sql

Each table has different SQLs, as the data sources (contents and formats) of each provider are different.

Before starting, we need to make sure that the httpfs extension is installed and loaded (as we use remote datasets):

INSTALL httpfs;
LOAD httpfs;

AWS table

CREATE TABLE aws_ip_data AS (
  SELECT DISTINCT
    prefixes.cidr_block,
    prefixes.ip_address,
    prefixes.ip_address_mask,
    CAST(POW(2, 32-prefixes.ip_address_mask) AS INTEGER) AS ip_address_cnt,
    prefixes.region
  FROM (
    SELECT
      prefix_object.ip_prefix AS cidr_block,
      STR_SPLIT(prefix_object.ip_prefix, '/')[1] AS ip_address,
      CAST(STR_SPLIT(prefix_object.ip_prefix, '/')[2] AS INTEGER) AS ip_address_mask,
      prefix_object.region
    FROM (
      SELECT UNNEST(prefixes) AS prefix_object FROM 'https://ip-ranges.amazonaws.com/ip-ranges.json'
    )
  ) prefixes
);

Azure table

CREATE TABLE azure_ip_data AS (
  SELECT DISTINCT
    prefixes AS cidr_block,
    STR_SPLIT(prefixes, '/')[1] AS ip_address,
    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,
    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,
    CASE
      WHEN region = '' THEN 'No region'
      ELSE region
    END AS region
  FROM (
    SELECT DISTINCT
      prop.region AS region,
      UNNEST(prop.addressPrefixes) AS prefixes
    FROM (
      SELECT 
        values.properties AS prop
      FROM (
        SELECT 
          UNNEST(values) AS values
        FROM
          read_json_auto('https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20230417.json', maximum_object_size=10000000)
      )
    )
  )
  WHERE
    prefixes NOT LIKE '%::%'
);

CloudFlare table

CREATE TABLE cloudflare_ip_data AS (
  SELECT DISTINCT
    prefixes AS cidr_block,
    STR_SPLIT(prefixes, '/')[1] AS ip_address,
    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,
    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,
    'No region' AS region
  FROM (
    SELECT
      column0 AS prefixes
    FROM
      read_csv_auto('https://www.cloudflare.com/ips-v4')
  )
);

DigitalOcean table

CREATE TABLE digitalocean_ip_data AS (
  SELECT DISTINCT
    prefixes AS cidr_block,
    STR_SPLIT(prefixes, '/')[1] AS ip_address,
    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,
    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,
    'No region' AS region
  FROM (
    SELECT
      column0 AS prefixes
    FROM
      read_csv_auto('https://digitalocean.com/geo/google.csv')
    WHERE
      column0 NOT LIKE '%::%'
  )
);

Fastly table

CREATE TABLE fastly_ip_data AS (
  SELECT DISTINCT
    prefixes AS cidr_block,
    STR_SPLIT(prefixes, '/')[1] AS ip_address,
    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,
    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,
    'No region' AS region
  FROM (
    SELECT
      UNNEST(addresses) AS prefixes
    FROM
      read_json_auto('https://api.fastly.com/public-ip-list')
  )
);

Google Cloud table

CREATE TABLE google_ip_data AS (
  SELECT DISTINCT
    prefixes.cidr_block,
    prefixes.ip_address,
    prefixes.ip_address_mask,
    CAST(pow(2, 32-prefixes.ip_address_mask) AS INTEGER) AS ip_address_cnt,
    prefixes.region
  FROM (
    SELECT
      prefix_object.ipv4Prefix AS cidr_block,
      str_split(prefix_object.ipv4Prefix, '/')[1] AS ip_address,
      CAST(str_split(prefix_object.ipv4Prefix, '/')[2] AS INTEGER) AS ip_address_mask,
      prefix_object.scope as region
    FROM (
      SELECT unnest(prefixes) AS prefix_object FROM 'https://www.gstatic.com/ipranges/cloud.json'
    )
    WHERE
      prefix_object.ipv4Prefix IS NOT NULL
  ) prefixes
);

Oracle Cloud table

CREATE TABLE oracle_ip_data AS (
  SELECT DISTINCT
    prefixes.cidr AS cidr_block,
    STR_SPLIT(prefixes.cidr, '/')[1] AS ip_address,
    CAST(STR_SPLIT(prefixes.cidr, '/')[2] AS INTEGER) AS ip_address_mask,
    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes.cidr, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,
    CASE
      WHEN region = '' THEN 'No region'
      ELSE region
    END AS region
  FROM (
    SELECT DISTINCT
      region,
      UNNEST(cidrs) AS prefixes
    FROM (
      SELECT 
        regions.region AS region,
        regions.cidrs AS cidrs
      FROM (
        SELECT 
          UNNEST(regions) AS regions
        FROM
          read_json_auto('https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json', maximum_object_size=10000000)
      )
    )
  )
);

Create a combined view

The next step is to create a new view (ip_data) that combines our tables for the individual cloud providers. We can then use this view later to compare the different cloud providers.

The view definition looks like this:

CREATE VIEW ip_data AS (
  SELECT 'AWS' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM aws_ip_data
  UNION ALL
  SELECT 'Azure' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM azure_ip_data
  UNION ALL
  SELECT 'CloudFlare' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM cloudflare_ip_data
  UNION ALL
  SELECT 'DigitalOcean' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM digitalocean_ip_data
  UNION ALL
  SELECT 'Fastly' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM fastly_ip_data
  UNION ALL
  SELECT 'Google Cloud' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM google_ip_data
  UNION ALL
  SELECT 'Oracle' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM oracle_ip_data
);

Export the data

To be able to use the data with other tools, we need to export the data to different formats, in our case CSV and Parquet. You can review the executed queries in the repository.

-- Only an example, this needs to be done for all providers as well!

-- Export complete data as CSV
COPY (SELECT * FROM ip_data ORDER BY cloud_provider, cidr_block) TO 'data/providers/all.csv' WITH (HEADER 1, DELIMITER ',');

-- Export complete data as Parquet
COPY (SELECT * FROM ip_data ORDER BY cloud_provider, cidr_block) TO 'data/providers/all.parquet' (FORMAT 'parquet', COMPRESSION 'SNAPPY');

Analyze the data

As we now prepared our data, we can start analyzing it. Therefore, we'll use an ObservableHQ notebook, where we'll upload the all.csv file to.

Overall IP address counts

Total number and value of IP addresses

An astonishing insight is that both AWS and Azure have more than six times as many IP addresses available as their next competitor.

Also, the market values of their IP addresses are nearly four billion Dollars according to a market analysis.

CIDR masks distribution by public cloud provider

It's remarkable that, although AWS and Azure have similar absolute numbers of IP addresses, the type of CIDR blocks / IP ranges strongly differ: AWS owns very few very large IP ranges, whereas Azure owns very many rather small IP ranges, and just a few very large ones:

Another view is the filterable table for this data:

AWS CIDR masks

Azure CIDR masks

CloudFlare CIDR masks

DigitalOcean CIDR masks

Fastly CIDR masks

Google Cloud CIDR masks

Oracle Cloud CIDR masks

Conclusion

In this article, we described a simple and straightforward way to gather and transform data in different formats with DuckDB, as well as export it to a common schema as CSV and Parquet files.

Furthermore, we leveraged DuckDB on Observable to analyze and display the data in beautiful and interactive graphs.

By using GitHub Actions free tier as our "runtime" for the data processing via Bash and SQL scripts, and hosting our data in a GitHub repo (also covered by the free tier), we were able to show that data pipelines like covered use case can be built without accruing infrastructure costs. Also, the Observable pricing model supports our analyses for free.

Casual data engineering, or: A poor man's Data Lake in the cloud - Part I

Tobias Müller — Mon, 24 Apr 2023 06:00:40 GMT

In the age of big data, organizations of all sizes are collecting vast amounts of information about their operations, customers, and markets. To make sense of this data, many are turning to data lakes - centralized repositories that store and manage data of all types and sizes, from structured to unstructured. However, building a data lake can be a daunting task, requiring significant resources and expertise.

For enterprises, this often means using SaaS solutions like Snowflake, Dremio, DataBricks or the like. Or, go all-in on the public cloud provider offerings from AWS, Azure and Google Cloud. But what if, as recent studies show, the data sizes aren't as big as commonly thought? Is it really necessary to spend so much money on usage and infrastructure?

In this blog post, we'll walk you through the steps to create a scalable, cost-effective data lake on AWS. Whether you're a startup, a small business, or a large enterprise, this guide will help you unlock the power of big data without breaking the bank (also see the excellent "Big data is dead" blog post by Jordan Tigani).

Modern Data Lake basics

The definition of what a Data Lake is, is probably slightly different depending on whom you're asking (see AWS, Google Cloud, Azure, DataBricks, IBM or Wikipedia). What is common to all these definitions and explanations is that it consists of different layers, such as ingestion, storage, processing and consumption. There can be several other layers as well, like cataloging and search, as well as a security and governance layer.

This is outlined in the excellent AWS article "AWS serverless data analytics pipeline reference architecture", which shall be the basis for this blog post:

Separation of storage & compute

Modern data lakes have revolutionized the way organizations handle big data. A data lake is a central repository that allows organizations to store all types of data, both structured and unstructured, at any scale. The flexibility and scalability of data lakes enable organizations to perform advanced analytics and gain insights that can drive business decisions. One of the key architectural patterns that modern data lakes follow is the separation of storage and compute.

Traditionally, data storage and processing were tightly coupled in data warehouses. However, in modern data lakes, data is stored in a separate layer from the computational layer that processes it. Data storage is handled by a data storage layer, while data processing is done by a compute layer. This approach allows organizations to scale storage and compute independently, enabling them to process vast amounts of data without incurring significant costs.

This has several advantages, which include:

Scalability: It allows organizations to scale each layer independently. The storage layer can be scaled up or down depending on the amount of data being stored, while the compute layer can be scaled up or down depending on the processing requirements.
Cost Savings: Decoupling storage and compute can significantly reduce costs. In traditional data warehouses, organizations must provision sufficient storage and processing power to handle peak loads. This results in underutilized resources during periods of low demand, leading to the wastage of resources and increased costs. In modern data lakes, organizations can store data cheaply and only provision the necessary compute resources when required, leading to significant cost savings.
Flexibility: Organizations can use a range of storage options, including object storage, file storage, and block storage, to store their data. This flexibility allows organizations to choose the most appropriate storage option for their data, depending on factors such as cost, performance, and durability.
Performance: In traditional data warehouses, data is moved from storage to processing, which can be slow and time-consuming, leading to performance issues. In modern data lakes, data is stored in a central repository, and processing is done where the data resides. This approach eliminates the need for data movement, leading to faster processing and improved performance.

Optimized file formats

As an example, Parquet is an open-source columnar storage format for data lakes that is widely used in modern data lakes. Parquet stores data in columns rather than rows, which enables it to perform selective queries faster and more efficiently than traditional row-based storage formats.

Additionally, Parquet supports compression, which reduces storage requirements and improves data processing performance. It's supported by many big data processing engines, including Apache Hadoop, Apache Spark, Apache Drill and many services of public cloud providers, such as Amazon Athena and AWS Glue.

Hive partitioning & query filter pushdown

The so-called "Hive partitioning" is a technique used in data lakes that involves dividing data into smaller, more manageable parts, called partitions, based on specific criteria such as date, time, or location.

Partitioning can help improve query performance and reduce data processing time by allowing users to select only the relevant partitions, rather than scanning the entire dataset.

Query filter pushdown is another optimization technique used in Apache Hive and other services that involves pushing down query filters into the storage layer, allowing it to eliminate irrelevant data before processing the query.

Combining Hive partitioning and query filter pushdown can result in significant performance gains in data processing, as the query filters can eliminate large amounts of irrelevant data at the partition level, reducing the amount of data that needs to be processed. Therefore, Hive partitioning and query filter pushdown are essential techniques for optimizing data processing performance in data lakes.

Repartitioning of data

Repartitioning Parquet data in data lakes is a useful technique that involves redistributing data across partitions based on specific criteria. This technique can help optimize query performance and reduce data shuffling during big data processing.

For instance, if a large amount of data is stored in a single partition, querying that data may take longer than if the data were spread across several partitions. Or, you could write aggregation queries whose output contains much less data, which could improve query speeds significantly.

The use case

Data privacy and GDPR are pretty talked-about topics in recent years. A lot of existing web tracking solutions were deemed as non-compliant, especially in the EU. Thus, individuals and companies had to eventually change their Web Analytics providers, which lead to a rise of new, data privacy-focussing companies in this space (e.g. Fathom Analytics, SimpleAnalytics, and Plausible just to name a few).

The pricing of those providers can get relatively steep quite fast if you have a higher amount of pageviews ($74/mo for 2m at Fathom, €99/mo for 1m at SimpleAnalytics, €89/mo for 2m at Plausible). Also, if you're using a provider, you're normally not owning your data.

So, let's try to build a web tracking and analytics service on AWS for the cheap, while owning our data, adhering to data privacy laws and using scalable serverless cloud services to avoid having to manage infrastructure by ourselves. And have some fun and learn a bit while doing it :-)

High-level architecture

The overall architecture for the outlined use case looks like this:

The details will be described further for each layer in the coming paragraphs. For brevity, the focus lies on the main data processing layers. Other layers, such as cataloging, consumption and security & governance, are eventually handled in other upcoming blog posts.

Serving layer

The serving layer is not a part of the data lake. Its main goal is to serve static assets, such as the tracking JavaScript libraries (those will be covered in more detail in another part of this series), and the 1x1 pixel GIF files that are used as endpoints that the tracking library can push its gathered data to. This is done by sending the JSON payload as URL-encoded query strings.

In our use case, we want to leverage existing AWS services and optimize our costs, while providing great response times. From an architectural perspective, there are many ways we could set up this data-gathering endpoint. Amazon CloudFront is a CDN that has currently over 90 edge locations worldwide, thus providing great latencies compared to classical webservers or APIs that are deployed in one or more regions.

It also has a very generous free tier (1TB outgoing traffic, and 10M requests), and with its real-time logs feature a great and very cost-effective way ($0.01 for every 1M log lines) to set up such an endpoint by just storing a 1x1px GIF with appropriate caching headers, to which the JavaScript tracking library will send its payload to as an encoded query string.

CloudFront can use S3 as a so-called origin (where the assets will be loaded from if they aren't yet in the edge caches), and that's where the static asset data will be located. Between the CloudFront distribution and the S3 bucket, an Origin Access Identity will be created, which enables secure communication between both services and avoids that the S3 bucket needs to be publicly accessible.

To configure CloudFront real-time logs that contain the necessary information, a RealtimeLogConfig needs to be created. This acts as "glue" between the CloudFront distribution and the Kinesis Data Stream that consumes the logs:

CFRealtimeLogsConfig:
  Type: AWS::CloudFront::RealtimeLogConfig
  Properties: 
    EndPoints: 
      - StreamType: Kinesis
        KinesisStreamConfig:
          RoleArn: !GetAtt 'AnalyticsKinesisDataRole.Arn'
          StreamArn: !GetAtt 'AnalyticsKinesisStream.Arn'
    Fields: 
      - timestamp
      - c-ip
      - sc-status
      - cs-uri-stem
      - cs-bytes
      - x-edge-location
      - time-taken
      - cs-user-agent
      - cs-referer
      - cs-uri-query
      - x-edge-result-type
      - asn
    Name: '${self:service}-cdn-realtime-log-config'
    # IMPORTANT: This setting make sure we receive all the log lines, otherwise it's just sampled!
    SamplingRate: 100

Ingestion layer

The ingestion layer mainly consists of two services: A Kinesis Data Stream, which is the consumer of the real-time logs feature of CloudFront, and a Kinesis Data Firehose Delivery Stream, which will back up the raw data in S3, and also store the data as partitioned parquet files in another S3 bucket. Both S3 buckets are part of the storage layer.

The Kinesis Data Stream (one shard in provisioned mode) provides an ingest capacity of 1 MB/second or 1,000 records/second, for a price of $0.015/hour in us-east-1, and $0.014 per 1M PUT payload units. It forwards the incoming data to the Kinesis Data Firehose Delivery Stream, whose pricing is more complex. The ingestion costs $0.029/GB, the format conversion $0.018/GB, and the dynamic partitioning $0.02/GB. That sums up to $0.067/GB ingested and written to S3, plus the S3 costs of $0.005/1k PUT object calls.

The Kinesis Data Firehose Delivery Stream uses data transformation and dynamic partitioning with a Lambda function, which cleans, transforms and enriches the data so that it can be stored in S3 as parquet files with appropriate Hive partitions.

The Delivery Stream has so-called BufferingHints, which either define from which size (from 1 to 128MB) or in which interval (between 60 to 900 seconds) the data is flushed to S3. The interval defines the minimum latency at which the data gets persisted in the data lake. The Lambda function is part of the processing layer and is discussed below.

The CloudFormation resource definition for the Kinesis Data Firehose Delivery Stream can be found below. It sources its variables from the serverless.yml:

AnalyticsKinesisFirehose:
  Type: 'AWS::KinesisFirehose::DeliveryStream'
  Properties:
    DeliveryStreamName: ${self:custom.kinesis.delivery.name}
    DeliveryStreamType: KinesisStreamAsSource
    # Source configuration
    KinesisStreamSourceConfiguration:
      KinesisStreamARN: !GetAtt 'AnalyticsKinesisStream.Arn'
      RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'
    # Necessary configuration to transfrom and write data to S3 as parquet files
    ExtendedS3DestinationConfiguration:
      BucketARN: !GetAtt 'CleanedBucket.Arn'
      BufferingHints:
        IntervalInSeconds: ${self:custom.kinesis.delivery.limits.intervalInSeconds}
        SizeInMBs: ${self:custom.kinesis.delivery.limits.sizeInMB}
      # This enables logging to CloudWatch for better debugging possibilities
      CloudWatchLoggingOptions:
        Enabled: True
        LogGroupName: ${self:custom.logs.groupName}
        LogStreamName: ${self:custom.logs.streamName}
      DataFormatConversionConfiguration:
        Enabled: True
        # Define the input format
        InputFormatConfiguration: 
          Deserializer: 
            OpenXJsonSerDe: 
              CaseInsensitive: True
        # Define the output format
        OutputFormatConfiguration: 
          Serializer: 
            ParquetSerDe: 
              Compression: SNAPPY
              WriterVersion: V1
        # The schema configuration based on Glue tables
        SchemaConfiguration: 
          RoleArn: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'
          DatabaseName: '${self:custom.glue.database}'
          TableName: 'incoming_events'
      # Enable dynamic partitioning
      DynamicPartitioningConfiguration:
          Enabled: True
      # Enable Lambda function for pre-processing the Kinesis records
      ProcessingConfiguration:
        Enabled: True
        Processors: 
          - Type: Lambda
            Parameters: 
              - ParameterName: NumberOfRetries
                ParameterValue: 3
              - ParameterName: BufferIntervalInSeconds
                ParameterValue: 60
              - ParameterName: BufferSizeInMBs
                ParameterValue: 3
              - ParameterName: LambdaArn
                ParameterValue: !GetAtt 'ProcessKinesisRecordsLambdaFunction.Arn'
      # Enable backups for the raw incoming data
      S3BackupMode: Enabled
      S3BackupConfiguration:
        BucketARN: !GetAtt 'RawBucket.Arn'
        BufferingHints:
          IntervalInSeconds: ${self:custom.kinesis.delivery.limits.intervalInSeconds}
          SizeInMBs: ${self:custom.kinesis.delivery.limits.sizeInMB}
        # Disable logging to CloudWatch for raw data
        CloudWatchLoggingOptions:
          Enabled: false
        CompressionFormat: GZIP
        Prefix: '${self:custom.prefixes.raw}'
        ErrorOutputPrefix: '${self:custom.prefixes.error}'
        RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'
      RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'
      # Define output S3 prefixes
      Prefix: '${self:custom.prefixes.incoming}/domain_name=!{partitionKeyFromLambda:domain_name}/event_type=!{partitionKeyFromLambda:event_type}/event_date=!{partitionKeyFromLambda:event_date}/'
      ErrorOutputPrefix: '${self:custom.prefixes.error}'

Processing layer

The processing layer consists of two parts, the Lambda function that is used for the dynamic partitioning of the incoming data, and a Lambda function that uses the COPY TO PARTITION BY feature of DuckDB to aggregate and repartition the ingested, enriched and stored page views data.

Data transformation & Dynamic partitioning Lambda

Data transformation is a Kinesis Data Firehose Delivery Stream feature that enables the cleaning, transformation and enrichment of incoming records in a batched manner. In combination with the dynamic partitioning feature, this provides powerful data handling capabilities with the data still being "on stream". When writing data to S3 as parquet files, a schema configuration in the form of a Glue Table needs to be defined as well to make it work (see "Cataloging & search layer" below).

It's necessary to define some buffer configuration for the Lambda function, meaning that you need to specify the time interval of 60 seconds (this will add a max delay of one minute to the stream data), the size in MB (between 0.2 and 3), and the number of retries (3 is the only usable default).

The input coming from the Kinesis Data Firehose Delivery Stream are a base64 encoded strings that contain the loglines coming from the CloudFront distribution:

MTY4MjA4NDI0MS40NjlcdDIwMDM6ZTE6YmYxZjo3YzAwOjhlYjoxOGY4OmExZmI6OWRhZFx0MzA0XHQvaGVsbG8uZ2lmP3Q9cHYmdHM9MTY4MjA4MzgwNDc2OCZ1PWh0dHBzJTI1M0ElMjUyRiUyNTJGbXlkb21haW4udGxkJTI1MkYmaG49bXlkb21haW4udGxkJnBhPSUyNTJGJnVhPU1vemlsbGElMjUyRjUuMCUyNTIwKE1hY2ludG9zaCUyNTNCJTI1MjBJbnRlbCUyNTIwTWFjJTI1MjBPUyUyNTIwWCUyNTIwMTBfMTVfNyklMjUyMEFwcGxlV2ViS2l0JTI1MkY1MzcuMzYlMjUyMChLSFRNTCUyNTJDJTI1MjBsaWtlJTI1MjBHZWNrbyklMjUyMENocm9tZSUyNTJGMTEyLjAuMC4wJTI1MjBTYWZhcmklMjUyRjUzNy4zNiZpdz0xMjkyJmloPTkyNiZ0aT1NeSUyNTIwRG9tYWluJnc9MzQ0MCZoPTE0NDAmZD0yNCZsPWRlLURFJnA9TWFjSW50ZWwmbT04JmM9OCZ0ej1FdXJvcGUlMjUyRkJlcmxpblx0Nzg5XHRIQU01MC1QMlx0MC4wMDFcdE1vemlsbGEvNS4wJTIwKE1hY2ludG9zaDslMjBJbnRlbCUyME1hYyUyME9TJTIwWCUyMDEwXzE1XzcpJTIwQXBwbGVXZWJLaXQvNTM3LjM2JTIwKEtIVE1MLCUyMGxpa2UlMjBHZWNrbyklMjBDaHJvbWUvMTEyLjAuMC4wJTIwU2FmYXJpLzUzNy4zNlx0LVx0dD1wdiZ0cz0xNjgyMDgzODA0NzY4JnU9aHR0cHMlMjUzQSUyNTJGJTI1MkZteWRvbWFpbi50bGQlMjUyRiZobj1teWRvbWFpbi50bGQmcGE9JTI1MkYmdWE9TW96aWxsYSUyNTJGNS4wJTI1MjAoTWFjaW50b3NoJTI1M0IlMjUyMEludGVsJTI1MjBNYWMlMjUyME9TJTI1MjBYJTI1MjAxMF8xNV83KSUyNTIwQXBwbGVXZWJLaXQlMjUyRjUzNy4zNiUyNTIwKEtIVE1MJTI1MkMlMjUyMGxpa2UlMjUyMEdlY2tvKSUyNTIwQ2hyb21lJTI1MkYxMTIuMC4wLjAlMjUyMFNhZmFyaSUyNTJGNTM3LjM2Jml3PTEyOTImaWg9OTI2JnRpPU15JTI1MjBEb21haW4mdz0zNDQwJmg9MTQ0MCZkPTI0Jmw9ZGUtREUmcD1NYWNJbnRlbCZtPTgmYz04JnR6PUV1cm9wZSUyNTJGQmVybGluXHRIaXRcdDMzMjBcbg==

After decoding, the logline is visible and contains the info from the real-time log fields, which are tab-separated and contain newlines:

1682084241.469\t2003:e1:bf1f:7c00:8eb:18f8:a1fb:9dad\t304\t/hello.gif?t=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\t789\tHAM50-P2\t0.001\tMozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/112.0.0.0%20Safari/537.36\t-\tt=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\tHit\t3320\n

During transformation and enrichment, the following steps are followed:

Validating the source record
Enriching the browser and device data from the user agent string
Determine whether the record was generated by a bot (by user agent string)
Add nearest geographical information based on edge locations
Compute referer
Derive requested URI
Compute UTM data
Get the event type (either a page view or a tracking event)
Build the time hierarchy (year, month, day, event timestamp)
Compute data arrival delays (data/process metrics)
Generate hashes for page view, daily page view and daily visitor ids (later used to calculate page views and visits)
Add metadata with the partition key values (in our case, the partition keys are domain_name, event_date, and event_type), to be able to use the dynamic partitioning feature

The generated JSON looks like this:

{
  "result": "Ok",
  "error": null,
  "data": {
    "event_year": 2023,
    "event_month": 4,
    "event_day": 21,
    "event_timestamp": "2023-04-21T13:30:04.768Z",
    "arrival_timestamp": "2023-04-21T13:37:21.000Z",
    "arrival_delay_ms": -436232,
    "edge_city": "Hamburg",
    "edge_state": null,
    "edge_country": "Germany",
    "edge_country_code": "DE",
    "edge_latitude": 53.630401611328,
    "edge_longitude": 9.9882297515869,
    "edge_id": "HAM",
    "referer": null,
    "referer_domain_name": "Direct / None",
    "browser_name": "Chrome",
    "browser_version": "112.0.0.0",
    "browser_os_name": "Mac OS",
    "browser_os_version": "10.15.7",
    "browser_timezone": "Europe/Berlin",
    "browser_language": "de-DE",
    "device_type": "Desktop",
    "device_vendor": "Apple",
    "device_outer_resolution": "3440x1440",
    "device_inner_resolution": "1292x926",
    "device_color_depth": 24,
    "device_platform": "MacIntel",
    "device_memory": 8,
    "device_cores": 8,
    "utm_source": null,
    "utm_campaign": null,
    "utm_medium": null,
    "utm_content": null,
    "utm_term": null,
    "request_url": "https://mydomain.tld/",
    "request_path": "/",
    "request_query_string": "t=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\t789\tHAM50-P2\t0.001\tMozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/112.0.0.0%20Safari/537.36\t-\tt=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin",
    "request_bytes": 789,
    "request_status_code": 304,
    "request_cache_status": "Hit",
    "request_delivery_time_ms": 1,
    "request_asn": 3320,
    "request_is_bot": 0,
    "event_name": null,
    "event_data": null,
    "page_view_id": "f4e1939bc259131659b00cd5f73e55a5bed04fbfa63f095b561fd87009d0a228",
    "daily_page_view_id": "7c82d13036aa2cfe04720e0388bb8645eb90de084bd50cf69356fa8ec9d8b407",
    "daily_visitor_id": "9f0ac3a2560cfa6d5c3494e1891d284225e15f088414390a40fece320021a658",
    "domain_name": "mydomain.tld",
    "event_date": "2023-04-21",
    "event_type": "pageview"
  },
  "metadata": {
    "partitionKeys": {
      "domain_name": "mydomain.tld",
      "event_date": "2023-04-21",
      "event_type": "pageview"
    }
  }
}

Then, the following steps are done by the Lambda function:

Encode the JSON stringified records in base64 again
Return them to the Kinesis Data Firehose Delivery Stream, which will then persist the data based on the defined prefix in the S3 bucket for incoming data.

Aggregation Lambda

As the ingested data contains information on a single request level, it makes sense to aggregate the data so that queries can be run optimally, and query response times are reduced.

The aggregation Lambda function is based on tobilg/serverless-parquet-repartitioner, which also has an accompanying blog post that explains in more detail how the DuckDB Lambda Layer can be used to repartition or aggregate existing data in S3.

The Lambda function is scheduled to run each night at 00:30AM, which makes sure that all the Kinesis Firehose Delivery Stream output files of the last day have been written to S3 (this is because the maximum buffer time is 15 minutes).

When it runs, it does three things:

Create a session aggregation, that derives the session information and whether single requests were bounces or not
Calculate the pageviews and visitor numbers, broken down by several dimensions which are later needed for querying (see stats table below)
Store the extraction of the event data separately, newly partitioned by event_name to speed up queries

The queries can be inspected in the accompanying repository to get an idea about the sophisticated query patterns DuckDB supports.

Storage layer

The storage layer consists of three S3 buckets, where each conforms to a zone outlined in the reference architecture diagram (see above):

A raw bucket, where the raw incoming data to the Kinesis Firehose Delivery Stream is backed up to (partitioned by event_date)
A cleaned bucket, where the data is stored by the Kinesis Firehose Delivery Stream (partitioned by domain_name, event_date and event_type)
A curated bucket, where the aggregated pageviews and visitors data are stored (partitioned by domain_name and event_date), as well as the aggregated and filtered events (partitioned by domain_name, event_date and event_name)

Cataloging & search layer

The Kinesis Data Firehose Delivery Stream needs a Glue table that holds the schema of the parquet files to be able to produce them (incoming_events table). The stats and the events tables are aggregated daily from the base incoming_events table via cron jobs scheduled by Amazon EventBridge Rules at 00:30 AM.

incoming_events table

This table stores the events that are the result of the data transformation and dynamic partitioning Lambda function. The schema for the table incoming_events looks like this:

Column name	Data type	Is partition key?	Description
domain_name	string	yes	The domain name
event_date	string	yes	The date of the event (YYYY-MM-DD), as string
event_type	string	yes	The type of the event (`pageview` or `track`)
event_year	int	no	The year of the event_date (YYYY)
event_month	int	no	The month of the event (MM)
event_day	int	no	The day of the event (DD)
event_timestamp	timestamp	no	The exact event timestamp
arrival_timestamp	timestamp	no	The exact timestamp when the event arrived in the Kinesis Data Stream
arrival_delay_ms	int	no	The difference between event_timestamp and arrival_timestamp in milliseconds
edge_city	string	no	The name of the edge city (all edge location info is derived from the `x-edge-location` field in the logs)
edge_state	string	no	The state of the edge location
edge_country	string	no	The country of the edge location
edge_country_code	string	no	The country code of the edge location
edge_latitude	float	no	The latitude of the edge location
edge_longitude	float	no	The longitude of the edge location
edge_id	string	no	The original id of the edge location
referrer	string	no	The referrer
referrer_domain_name	string	no	The domain name of the referrer
browser_name	string	no	The name of the browser
browser_version	string	no	The version of the browser
browser_os_name	string	no	The OS name of the browser
browser_os_version	string	no	The OS version of the browser
browser_timezone	string	no	The timezone of the browser
browser_language	string	no	The language of the browser
device_type	string	no	The device type
device_vendor	string	no	The device vendor
device_outer_resolution	string	no	The outer resolution of the device
device_inner_resolution	string	no	The inner resolution of the device
device_color_depth	int	no	The color depth of the device
device_platform	string	no	The platform of the device
device_memory	float	no	The memory of the device (in MB)
device_cores	int	no	The number of cores of the device
utm_source	string	no	Identifies which site sent the traffic
utm_campaign	string	no	Identifies a specific product promotion or strategic campaign
utm_medium	string	no	Identifies what type of link was used, such as cost per click or email
utm_content	string	no	Identifies what specifically was clicked to bring the user to the site
utm_term	string	no	Identifies search terms
request_url	string	no	The full requested URL
request_path	string	no	The path of the requested URL
request_query_string	string	no	The query string of the requested URL
request_bytes	int	no	The size of the request in bytes
request_status_code	int	no	The HTTP status code of the request
request_cache_status	string	no	The CloudFront cache status
request_delivery_time_ms	int	no	The time in ms it took for CloudFront to complete the request
request_asn	int	no	The ASN of the requestor
request_is_bot	int	no	If the request is categorized as a bot, the value will be `1`, if not `0`
event_name	string	no	The name of the event for tracking events
event_data	string	no	The stringified event payload for tracking events
page_view_id	string	no	The unique pageview id
daily_page_view_id	string	no	The unique daily pageview id
daily_visitor_id	string	no	The unique daily visitor id

stats table

The pageviews and visitor aggregation table. Its schema looks like this:

Column name	Data type	Is partition key?	Description
domain_name	string	yes	The domain name
event_date	string	yes	The date of the event (YYYY-MM-DD), as string
event_hour	int	no	The hour part of the event timestamp
edge_city	string	no	The name of the edge city (all edge location info is derived from the `x-edge-location` field in the logs)
edge_country	string	no	The country of the edge location
edge_latitude	float	no	The latitude of the edge location
edge_longitude	float	no	The longitude of the edge location
referrer_domain_name	string	no	The domain name of the referrer
browser_name	string	no	The name of the browser
browser_os_name	string	no	The OS name of the browser
device_type	string	no	The device type
device_vendor	string	no	The device vendor
utm_source	string	no	Identifies which site sent the traffic
utm_campaign	string	no	Identifies a specific product promotion or strategic campaign
utm_medium	string	no	Identifies what type of link was used, such as cost per click or email
utm_content	string	no	Identifies what type of link was used, such as cost per click or email
utm_term	string	no	Identifies search terms
request_path	string	no	The path of the requested URL
page_view_cnt	int	no	The number of page views
visitor_cnt	int	no	The number of daily visitors
bounces_cnt	int	no	The number of bounces (visited only one page)
visit_duration_sec_avg	int	no	The average duration of a visit (in seconds)

events table

The schema for the table events looks like this:

Column name	Data type	Is partition key	Description
domain_name	string	yes	The domain name
event_date	string	yes	The date of the event (YYYY-MM-DD), as string
event_name	string	yes	The name of the event for tracking events
event_timestamp	timestamp	no	The exact event timestamp
edge_city	string	no	The name of the edge city (all edge location info is derived from the `x-edge-location` field in the logs)
edge_country	string	no	The country of the edge location
edge_latitude	float	no	The latitude of the edge location
edge_longitude	float	no	The longitude of the edge location
request_path	string	no	The path of the requested URL
page_view_id	string	no	The unique pageview id
daily_visitor_id	string	no	The unique daily visitor id
event_data	string	no	The stringified event payload for tracking events

Consumption layer

The consumption layer will be part of another blog post in this series. Stay tuned! Until it's released, you can have a look at tobilg/serverless-duckdb to get an idea of how the data could potentially be queried in a serverless manner.

Wrapping up

In this article, you learned some basic principles of modern data lakes in the introduction. After that, it described how to build a serverless, near-realtime data pipeline leveraging AWS services and DuckDB on these principles, by the example of a web analytics application.

The example implementation of this article can be found on GitHub at ownstats/ownstats. Feel free to open an issue in case something doesn't work as expected, or if you'd like to add a feature request.

The next posts in this series will be

Part II: Building a lightweight JavaScript library for the gathering of web analytics data
Part III: Consuming the gathered web analytics data by building a serverless query layer
Part IV: Building a frontend for web analytics data

Using DuckDB to repartition parquet data in S3

Tobias Müller — Sun, 26 Feb 2023 17:45:04 GMT

Since release v0.7.1, DuckDB has the ability to repartition data stored in S3 as parquet files by a simple SQL query, which enables some interesting use cases.

Why not use existing AWS services?

If your data lake lives in AWS, a natural choice for ETL pipelines would be existing AWS services such as Amazon Athena. Unfortunately, Athena has pretty tight limits on the number of partitions that can be written by a single query (only up to 100). You, therefore, would need to create your workaround logic to be able to adhere to the limits, while still being able to do the operations you want. Additionally, Athena in some cases takes several 100ms to even start a query.

This is where DuckDB comes into play because it (theoretically) supports an unlimited amount of Hive partitions, and offers very fast queries on partitioned parquet files.

Use case

A common pattern to ingest streaming data and store it in S3 is to use Kinesis Data Firehose Delivery Streams, which can write the incoming stream data as batched parquet files to S3. You can use custom S3 prefixes with it when using Lambda processing functions, but by default, you can only partition the data by the timestamp (the timestamp the event reached the Kinesis Data Stream, not the event timestamp!).

So, a few common use cases for data repartitioning could include:

Repartitioning the written data for the real event timestamp if it's included in the incoming data
Repartitioning the data for other query patterns, e.g. to support query filter pushdown and optimize query speeds and costs
Aggregation of raw or preprocessed data, and storing them in an optimized manner to support analytical queries

We want to be able to achieve this without having to manage our own infrastructure and built-upon services, which is the reason we want to be as serverless as possible.

Solution

As described before, Amazon Athena only has a partition limit of 100 partitions when writing data. DuckDB doesn't have this limitation, that's why we want to be able to use it for repartitioning.

This requires that we can deploy DuckDB in a serverless manner. The choice is to run it in Lambda functions, which can be provisioned with up to 10GB of memory (meaning 6 vCPUs), and a maximum runtime of 900 seconds / 15 minutes. This should be enough for most repartitioning needs, because the throughput from/to S3 is pretty fast. Also, we want to be able to run our repartition queries on flexible schedules, that's why we'll use EventBridge Rules with a schedule.

The project can be found at https://github.com/tobilg/serverless-parquet-repartitioner, and just needs to be configured and deployed.

Architecture overview

Configuration

Mandatory configuration settings

S3 bucket name: You need to specify the S3 bucket where the data that you want to repartition resides (e.g. my-source-bucket)
Custom repartitioning query: You can write flexible repartitioning queries in the DuckDB syntax. Have a look at the examples in the httpfs extension docs. You need to update this, as the template uses only example values!

Optional configuration settings

S3 region: The AWS region your S3 bucket is deployed to (if different from the region the Lambda function is deployed to)
The schedule: The actual schedule on why the Lambda function is run. Have a look at the Serverless Framework docs to find out what the potential settings are.
DuckDB memory limit: The memory limit is influenced by the function memory setting (automatically)
DuckDB threads count: Optionally set the max thread limit (on Lambda, this is set automatically by the amount of memory the functions has assigned), but with this setting, you can influence how many files are written per partition. If you set a lower thread count than available, this means that the computation will not use all available resources for the sake of being able to set the number of generated files! Ideally, rather align the amount of memory you assign to the Lambda function.
Lambda timeout: The maximum time a Lambda function can run is currently 15min / 900sec. This means that if your query takes longer than that, it will be terminated by the underlying Firecracker engine.

Using different source/target S3 buckets

If you're planning to use different S3 buckets as sources and targets for the data repartitioning, you need to adapt the iamRoleStatements settings of the function.

Here's an example with minimal privileges:

iamRoleStatements:
  # Source S3 bucket permissions
  - Effect: Allow
    Action:
      - s3:ListBucket
    Resource: 'arn:aws:s3:::my-source-bucket'
  - Effect: Allow
    Action:
      - s3:GetObject
    Resource: 'arn:aws:s3:::my-source-bucket/*'
  # Target S3 bucket permissions
  - Effect: Allow
    Action:
      - s3:ListBucket
      - s3:AbortMultipartUpload
      - s3:ListMultipartUploadParts
      - s3:ListBucketMultipartUploads
    Resource: 'arn:aws:s3:::my-target-bucket'
  - Effect: Allow
    Action:
      - s3:GetObject
      - s3:PutObject
    Resource: 'arn:aws:s3:::my-target-bucket/*'

A query for this use case would look like this:

COPY (SELECT * FROM parquet_scan('s3://my-source-bucket/input/*.parquet', HIVE_PARTITIONING = 1)) TO 's3://my-starget-bucket/output' (FORMAT PARQUET, PARTITION_BY (column1, column2, column3), ALLOW_OVERWRITE TRUE);

Deployment

After you cloned this repository to your local machine and cd'ed in its directory, the application can be deployed like this (don't forget a npm i to install the dependencies!):

$ sls deploy

This will deploy the stack to the default AWS region us-east-1. In case you want to deploy the stack to a different region, you can specify a --region argument:

$ sls deploy --region eu-central-1

The deployment should take 2-3 minutes.

Checks and manual triggering

You can manually invoke the deployed Lambda function by running

$ sls invoke -f repartitionData

After that, you can check the generated CloudWatch logs by issuing

$ sls logs -f repartitionData

If you don't see any DUCKDB_NODEJS_ERROR in the logs, everything ran successfully, and you can have a look at your S3 bucket for the newly generated parquet files.

Costs

Using this repository will generate costs in your AWS account. Please refer to the AWS pricing docs for the respective services before deploying and running it.

Summary

We were able to show a possible serverless solution to repartition data that is stored in S3 as parquet files, without limitations imposed by certain AWS services. With the solution shown, we can use plain and simple SQL queries, instead of having to eventually use external libraries etc.

References

Serverless parquet repartitioner repo: https://github.com/tobilg/serverless-parquet-repartitioner

Using DuckDB in AWS Lambda

Tobias Müller — Sun, 12 Feb 2023 16:41:41 GMT

Prelude

DuckDB is an open-source in-process SQL OLAP database management system that has recently gained significant public interest due to its unique architecture and impressive performance benchmarks.

Unlike traditional databases that are designed to handle a wide variety of use cases, DuckDB is built specifically for analytical queries and is optimized to perform extremely well in these scenarios. This focus on analytics has allowed DuckDB to outperform traditional databases by several orders of magnitude, making it a popular choice for data scientists and analysts who need to process large datasets quickly and efficiently.

As DuckDB is designed to be a highly efficient and scalable database system, which makes it a perfect fit for serverless architectures that allow developers to build and run applications and services without having to manage infrastructure.

DuckDB's ability to handle large datasets in a memory-efficient manner makes it an ideal choice for serverless environments. Being able to read columnar storage formats like Parquet or Apache Arrow tables from local, S3 or HTTP sources, DuckDB can quickly scan and aggregate large amounts of data without having to load it all into memory, reducing the amount of memory required to perform complex analytical queries. This allows for cost savings, as serverless environments typically charge for both compute and memory resources.

Existing AWS services, such as Athena or RDS, don't provide the same functionalities, and also have different scaling and pricing models. That's why it makes sense to explore ways to run DuckDB as an analytical service on AWS.

How to run DuckDB in AWS Lambda?

The goal of this article is to use DuckDB on Node.js runtimes (12, 14, 16 and 18), thus it is necessary to find a way to make DuckDB usable with Lambda. The first idea was to simply use the existing DuckDB npm package and use the default packaging mechanisms when deploying Lambda functions. Unfortunately, this idea proved impossible due to downstream package problems and different build Operating Systems (Lambda needs statically linked binaries built on Amazon Linux).

Generally, there are several ways to package dependencies in AWS Lambda functions with Node.js runtimes. Using bundlers like WebPack or ESbuild is probably the most used option at the moment.

Another one is using AWS Lambda layers for distributing dependencies to Lambda functions, allowing developers to manage common components and libraries across multiple functions in a centralized manner. By creating a layer for the dependencies, developers can avoid having to include them in each function’s deployment package. This helps reduce the size of the deployment package and makes it easier to manage updates to the dependencies. Moreover, using a Lambda layer can also improve the performance of Lambda functions.

Building DuckDB for AWS Lambda

So, how can we achieve to build a DuckDB version that can be used with Node.js runtimes on AWS Lambda?

We have to use a compatible environment when compiling the DuckDB binary, to avoid GLIBC incompatibilities etc. This means that we have to use an Amazon Linux distribution to build DuckDB, and enable static linking.
We have to solve the downstream package problems stated above, which make it impossible to use the default package.
On a side note, we want to enable the current features like COPY TO PARTITIONED BY and improved Parquet file reading, thus requiring a build from the master branch of DuckDB.

I created https://github.com/tobilg/duckdb-nodejs-layer to achieve this. It uses GitHub Actions to automatically trigger a build of DuckDB from the current master, package it as AWS Lambda layer and automatically upload it to all AWS regions. Feel free to have a look at the source code, and open a GitHub issue in case you find some errors or ideas for improvement.

Using the DuckDB Lambda layer

Depending on your preferred framework, the methods to use a Lambda layer are different. You can find the respective docs of the most common frameworks below:

The ARNs follow the following logic:

arn:aws:lambda:$REGION:041475135427:layer:duckdb-nodejs-layer:$VERSION

You can find the list of ARNs for all regions at https://github.com/tobilg/duckdb-nodejs-layer#usage.

I created an example repository with the Serverless Framework at https://github.com/tobilg/serverless-duckdb that uses API Gateway and Lambda to provide an endpoint to which SQL queries can be issued, which the built-in DuckDB then executes. Let's walk through it!

Requirements

You'll need a current v3 version installation of the Serverless Framework on the machine you're planning to deploy the application from.

Also, you'll have to set up your AWS credentials according to the Serverless docs.

Configuration

DuckDB is automatically configured to use the HTTPFS extension and uses the AWS credentials that are given to your Lambda function by its execution role. This means you can potentially query data that is available via HTTP(S) or in AWS S3 buckets.

If you want to also query data (e.g. Parquet files) that resides in one or more S3 buckets, you'll have to adjust the iamRoleStatements part of the function configuration in the serverless.yml file. Just replace the YOUR-S3-BUCKET-NAME with your actual S3 bucket name.

  query:
    handler: src/functions/query.handler
    memorySize: 10240
    timeout: 30
    iamRoleStatements:
      - Effect: Allow
        Action:
          - s3:GetObject
        Resource: 'arn:aws:s3:::YOUR-S3-BUCKET-NAME/*'
      - Effect: Allow
        Action:
          - s3:ListBucket
        Resource:
          - 'arn:aws:s3:::YOUR-S3-BUCKET-NAME'
    layers:
      - 'arn:aws:lambda:${self:provider.region}:041475135427:layer:duckdb-nodejs-layer:3'
    events:
      - http:
          path: ${self:custom.api.version}/query
          method: post
          cors: true
          private: true

Deployment

After you cloned this repository to your local machine and cd'ed in its directory, the application can be deployed like this (don't forget a npm i to install the dependencies):

$ sls deploy

This will deploy the stack to the default AWS region us-east-1. In case you want to deploy the stack to a different region, you can specify a --region argument:

$ sls deploy --region eu-central-1

The deployment should take 2-3 minutes. Once the deployment is finished, you should find some output in your console that indicates the API Gateway endpoint URL and the API Key:

api keys:
  DuckDBKey: REDACTED
endpoint: POST - https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query

Usage

You can now query your DuckDB endpoint via HTTP requests (don't forget to exchange REDACTED with your real URL and API Key), e.g.

curl --location --request POST 'https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query' \
--header 'x-api-key: REDACTED' \
--header 'Content-Type: application/json' \
--data-raw '{
    "query": "SELECT avg(c_acctbal) FROM '\''https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet'\'';"
}'

The query results will look to this:

[
    {
        "avg(c_acctbal)": 4454.577060000001
    }
]

Example queries

Remote Parquet Scans:
  SELECT count(*) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/lineitem.parquet';
  SELECT count(*) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet';
  SELECT avg(c_acctbal) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet';
  SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 10;

Remote Parquet/Parquet Join:
  SELECT n_name, count(*)
  FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet',
       'https://shell.duckdb.org/data/tpch/0_01/parquet/nation.parquet'
  WHERE c_nationkey = n_nationkey GROUP BY n_name;

Conclusion

We were able to show that it's possible to package and use DuckDB in a Lambda function, as well as to run performant queries on remote data with this setup.

But we need to keep in mind that this is just a showcase. The example application doesn't solve a lot of issues we'd have to solve if we'd want to run this in a distributed manner:

A query planner and router, that scales DuckDB instances and redistributes the queries to the "query backend" functions, as well as unites the query results before passing them back to the query issuer
"Query stickiness": The example is stateless, meaning that even if you'd load data into a memory table, you'd not be able to be sure that you'd reach the same function instance with a subsequent query due to Lambda's scaling/execution model
Running DuckDB "only" in Lambda functions may not be the most performant way when AWS Fargate and very large EC2 instances exist
The example application uses API Gateway as the event source for the Lambda function, which means the maximum runtime of the queries can be 30 seconds, which is unrealistic for large datasets or complicated queries. In real-world scenarios, the Lambda function would need to be triggered asynchronously, e.g. via SNS or SQS. This also means that the queries probably can't follow a strict request/response model.

References

Building a global reverse proxy with on-demand SSL support

Tobias Müller — Tue, 10 Jan 2023 23:25:26 GMT

Motivation

Who needs a reverse proxy with on-demand SSL support? Well, think about services as Hashnode, which also runs this blog, or Fathom and SimpleAnalytics. A feature that all those services have in common? They all are enabling their customers to bring their own, custom domain names. The latter two services use them to bypass adblockers, with the intention that customers can track all their pageviews and events, which potentially wouldn't be possible otherwise because the service's own domain are prone to DNS block lists. Hashnode is using them to enable their customers to host their blogs under their own domain names.

Requirements

What are the functional & non-functional requirements to build such a system? Let's try to recap:

We want to be able to use a custom ("external") domain, e.g. subdomain.customdomain.tld to redirect to another domain, such as targetdomain.tld via a CNAME DNS record
The custom domains need to have SSL/TLS support
The custom domains need to be configurable, without changing the underlying infrastructure
We want to make sure that the service will only create and provide the certificates for whitelisted custom domains
We want to optimize the latency of individual requests, thus need to support a scalable and distributed infrastructure
We want to be as Serverless as possible
We want to optimize for infrastructure costs (variable and fixed)
We want to build this on AWS
The service needs to be deployable/updateable/removable via Infrastructure as Code (IaC) in a repeatable manner

Architectural considerations

Looking at the above requirements, what are the implications from an architectural point of view? Which tools and services are already on the market? What does AWS as Public Cloud Provider offer for our use case?

For the main requirements of a reverse proxy server with automatic SSL/TLS support, Caddy seems to be an optimal candidate. As it is written in Go, it can run in Docker containers very well and can be used on numerous operating systems. This means we have the options to either run it on EC2 instances or ECS/Fargate if we decide to run it in containers. The latter would cater to the requirement to run as Serverless as possible. It has modules to store the generated SSL/TLS on-demand certificates in DynamoDB or S3.

Also, the whitelisting of custom domains is possible for those certificates, by providing an additional backend service, which Caddy can ask whether a requested custom domain is allowed to use or not.

A challenge is that none of those modules are contained in the official Caddy builds, meaning that we'd have to build a custom version of Caddy to be able to use those storage backends.

Regarding the requirement of global availability and short response times, AWS Global Accelerator is a viable option, as it can provide a global single static IP address endpoint for multiple, regionally distributed services. In our use case, those services would be our Caddy installations.

Running Caddy itself, as said before, is possible via Containers or EC2 instances / VMs. As the services will run the whole time and assumingly don't need a lot of resources if not under heavy load, we assume that 1 vCPU and 1 GB of memory should be enough.

When projecting this on the necessary infrastructure, the cost comparison between Containers and VMs looks like the following (for simplification, we just compare the fixed costs, ignore variable costs such as egress traffic, and assume the us-east-1 region is used):

Containers

Fargate task with 1 vCPU and 1 GB of memory for each Caddy regional instance
- Price: ($0.04048/vCPU hour + $0.004445/GB hour) * 720 hours (30 days) = $32.35 / 30 days
ALB to make the Fargate tasks available to the outside world
- Price: ($0.0225 per ALB-hour + $0.008 per LCU-hour) * 720 hours (30 days) = $21.96 / 30 days

In combination, it would cost $54.31 to run this setup for 30 days.

EC2 instances / VMs

A t2.micro instance with 1 vCPU and 1 GB f memory for each Caddy regional instance
- Price: $0.0116/hour on-demand * 720 hours (30 days) = $8.35
There's no need for a Load Balancer in front of the EC2 instances, as Global Accelerator can directly use them
- Price: $0

In combination, it would cost $8.35 to run this setup for 30 days.

Additional costs are:

AWS Global Accelerator
- Price: $0.025 / hour * 720 hours (30 days) = $18
DynamoDB table
- Price: (5000 reads/day * $0.25/million + 50 writes/day * $1.25/million) * 30 days = $0.0375 reads + 0.001875 writes = $0.04
Lambda function (128 MB / 0.25 sec avg. duration / 5000 req./day)
- Price: ($0.0000166667 / GB-second + $0.20 per 1M req.) = basically $0

Resulting architecture

Based on the calculated fixed costs, we decided to use EC2 instances instead of Fargate tasks, which will save us a decent amount of money even for one Caddy instance. The estimated costs for running this architecture for 30 days are $26.39.

As one of our requirements was that we can roll out this infrastructure potentially on a global scale, we need to be able to deploy the EC2 instances with the Caddy servers in different AWS regions, as well as having multiple instances in the same region.

Furthermore, we could use DynamoDB Global Tables to achieve a global distribution of the certificates to get faster response times, but deem it as out of scope for this article.

The final architecture:

Implementation

To implement the described architecture, we must take several steps. First of all, we must build a custom version of Caddy that includes the DynamoDB module, which then enables us to use DynamoDB as certificate store.

Custom Caddy build

This can be achieved via a custom build process leveraging Docker images of AmazonLinux 2, as found at tobilg/aws-caddy-build.

build.sh (parametrized custom build of Caddy via Docker)

#!/usr/bin/env bash

set -e

# Set OS (first script argument)
OS=${1:-linux}

# Set Caddy version (second script argument)
CADDY_VERSION=${2:-v2.6.2}

# Create release folders
mkdir -p $PWD/releases $PWD/temp_release

# Run build
docker build --build-arg OS=$OS --build-arg CADDY_VERSION=$CADDY_VERSION -t custom-caddy-build .

# Copy release from image to temporary folder
docker run -v $PWD/temp_release:/opt/mount --rm -ti custom-caddy-build bash -c "cp /tmp/caddy-build/* /opt/mount/"

# Copy release to releases
cp $PWD/temp_release/* $PWD/releases/

# Cleanup
rm -rf $PWD/temp_release

Dockerfile (will build Caddy with the DynamoDb and S3 modules)

FROM amazonlinux:2

ARG CADDY_VERSION=v2.6.2
ARG OS=linux

# Install dependencies
RUN yum update -y && \
  yum install golang -y

RUN GOBIN=/usr/local/bin/ go install github.com/caddyserver/xcaddy/cmd/xcaddy@latest

RUN mkdir -p /tmp/caddy-build && \
  GOOS=${OS} xcaddy build ${CADDY_VERSION} --with github.com/ss098/certmagic-s3 --with github.com/silinternational/certmagic-storage-dynamodb/v3 --output /tmp/caddy-build/aws_caddy_${CADDY_VERSION}_${OS}

That's it for the custom Caddy build. You don't need to build this yourself, as the further steps use the release I built and uploaded to GitHub.

Reverse Proxy Service

The implementation of the reverse proxy service can be found at tobilg/global-reverse-proxy.

Just clone it via git clone https://github.com/tobilg/global-reverse-proxy.git to your local machine, and configure it as described below.

Prerequisites

Serverless Framework

You need to have a recent (>=3.1.2) version of the Serverless Framework installed globally on your machine. If you haven't, you can run npm i -g serverless to install it.

Valid AWS credentials

The Serverless Framework relies on already configured AWS credentials. Please refer to the docs to learn how to set them up on your local machine.

EC2 key already configured

If you want to interact with the deployed EC2 instance(s), you need to add your existing public SSH key or create a new one. Please have a look at the AWS docs to learn how you can do that.

Please also note the name you have given to the newly created key, as you will have to update the configuration of the proxy server(s) stack.

Infrastructure as Code overview

The infrastructure consists of three different stacks:

A stack for the domain whitelisting service, and the certificate table in DynamoDB
A stack for the proxy server(s) itself, which can be deployed multiple times if you want high (global) availability and fast latencies
A stack for the Global Accelerator, and the according DNS records

Most important parts

The main functionality, the reverse proxy based on Caddy, is deployed via an EC2 instance. Its configuration, the so-called Caddyfile, is, together with the CloudFormation resource for the EC2 instance, the most important part.

Caddyfile

This configuration enables the reverse proxy, the on-demand TLS feature and DynamoDB storage for certificates. It's automatically parametrized via the generated /etc/caddy/environment file (see ec2.yml below). There's a systemctl service for Caddy generated, based on our configuration derived from the serverless.yml, as well.

{
        admin off
        on_demand_tls {
                ask {$DOMAIN_SERVICE_ENDPOINT}
        }

        storage dynamodb {$TABLE_NAME} {
                aws_region {$TABLE_REGION}
        }
}

:80 {
       respond /health "Im healthy" 200
}

:443 {
        tls {$LETSENCRYPT_EMAIL_ADDRESS} {
                on_demand
        }

        reverse_proxy https://{$TARGET_DOMAIN} {
                header_up Host {$TARGET_DOMAIN}
                header_up User-Custom-Domain {host}
                header_up X-Forwarded-Port {server_port}

                health_timeout 5s
        }
}

ec2.yml (extract)

The interesting part is the UserData script, which is run automatically when the EC2 instance starts. It does the following:

Download the custom Caddy build with DynamoDB support
Prepare a group and a user for Caddy
Create the caddy.service file for systemctl
Create the Caddyfile (as outlined above)
Create the environment file (/etc/caddy/environment)
Enable & reload the systemctl service

Resources:
  EC2Instance:
    Type: AWS::EC2::Instance
    Properties:
      InstanceType: '${self:custom.ec2.instanceType}'
      KeyName: '${self:custom.ec2.keyName}'
      SecurityGroups: 
        - !Ref 'InstanceSecurityGroup'
      ImageId: 'ami-0b5eea76982371e91' # Amazon Linux 2 AMI
      IamInstanceProfile: !Ref 'InstanceProfile'
      UserData: !Base64 
        'Fn::Join':
          - ''
          - - |
              #!/bin/bash -xe
            - |
              sudo wget -O /usr/bin/caddy "https://github.com/tobilg/aws-caddy-build/raw/main/releases/aws_caddy_v2.6.2_linux"
            - |
              sudo chmod +x /usr/bin/caddy
            - |
              sudo groupadd --system caddy
            - |
              sudo useradd --system --gid caddy --create-home --home-dir /var/lib/caddy --shell /usr/sbin/nologin --comment "Caddy web server" caddy
            - |
              sudo mkdir -p /etc/caddy
            - |
              sudo echo -e '${file(./configs.js):caddyService}' | sudo tee /etc/systemd/system/caddy.service
            - |
              sudo printf '${file(./configs.js):caddyFile}' | sudo tee /etc/caddy/Caddyfile
            - |
              sudo echo -e "TABLE_REGION=${self:custom.caddy.dynamoDBTableRegion}\nTABLE_NAME=${self:custom.caddy.dynamoDBTableName}\nDOMAIN_SERVICE_ENDPOINT=${self:custom.caddy.domainServiceEndpoint}\nLETSENCRYPT_EMAIL_ADDRESS=${self:custom.caddy.letsEncryptEmailAddress}\nTARGET_DOMAIN=${self:custom.caddy.targetDomainName}" | sudo tee /etc/caddy/environment
            - |
              sudo systemctl daemon-reload
            - |
              sudo systemctl enable caddy
            - |
              sudo systemctl start --now caddy

global-accelerator.yml

The Global Accelerator CloudFormation resource wires the EC2 instance(s) to its kind-of global load balancer. This is then referenced by the dns-record.yml, which assigns the configured domain name to the Global Accelerator.

Resources:
  Accelerator:
    Type: AWS::GlobalAccelerator::Accelerator
    Properties:
      Name: 'External-Accelerator'
      Enabled: true

  Listener:
    Type: AWS::GlobalAccelerator::Listener
    Properties:
      AcceleratorArn:
        Ref: Accelerator
      Protocol: TCP
      ClientAffinity: NONE
      PortRanges:
        - FromPort: 443
          ToPort: 443

  EndpointGroup1:
    Type: AWS::GlobalAccelerator::EndpointGroup
    Properties: 
      EndpointConfigurations: 
        - EndpointId: '${self:custom.ec2.instance1.id}'
          Weight: 1
      EndpointGroupRegion: '${self:custom.ec2.instance1.region}'
      HealthCheckIntervalSeconds: 30
      HealthCheckPath: '/health'
      HealthCheckPort: 80
      HealthCheckProtocol: 'HTTP'
      ListenerArn: !Ref 'Listener'
      ThresholdCount: 3

Detailed configuration

Stack configurations

Please configure the following values for the different stacks:

The target domain name where you want your reverse proxy to send the requests to (targetDomainName)
The email address to use for automatic certificate generation via LetsEncrypt (letsEncryptEmailAddress)
The domain name of the proxy service itself, which is then used by GlobalAccelerator (domain)
Optionally: The current IP address from which you want to use the EC2 instance(s) via SSH from (sshClientIPAddress). If you want to use SSH, you'll need to uncomment the respective SecurityGroup settings

Whitelisted domain configuration

You need to make sure that not everyone can use your reverse proxy with every domain. Therefore, you need to configure the whitelist of domains that you be used by Caddy's on-demand TLS feature.

This is done with the Domain Verifier Lambda function, which is deployed at a Function URL endpoint.

The configuration can be changed here before deploying the service.

HINT: To use this dynamically, as you'd probably wish in a production setting, you could rewrite the Lambda function to read the custom domains from a DynamoDB table, and have another Lambda function run recurrently to issue DNS checks for the CNAME entries the customers would need to make (see below).

DNS / Nameserver configurations

If you use an external domain provider, such as Namecheap or GoDaddy, make such that you point the DNS settings at your domain's configuration to those which are assigned to your HostedZone by Amazon. You can look these up in the AWS Console or via the AWS CLI.

CNAME configuration for proxying

You also need to add CNAME records to the domains you want to proxy for, e.g. if your proxy service domain is external.mygreatproxyservice.com, you need to add a CNAME record to your existing domain (e.g. test.myexistingdomain.com) to redirect to the proxy service domain:

CNAME test.myexistingdomain.com external.mygreatproxyservice.com

Passing options during deployment

When running sls deploy for each stack, you can specify the following options to customize the deployments:

--stage: This will configure the so-called stage, which is part of the stack name (default: prd)
--region: This will configure the AWS region where the stack is deployed (default: us-east-1)

Deployment

You need to follow a specific deployment order to be able to run the overall service:

Domain whitelisting service: cd domain-service-stack && sls deploy && cd ..
Proxy server(s): cd proxy-server-stack && sls deploy && cd ..
Global Accelerator & HostedZone / DNS : cd accelerator-stack && sls deploy && cd ..

Removal

To remove the individual stacks, you can run sls remove in the individual subfolders.

Wrapping up

We were able to build a POC for a (potentially) globally distributed reverse proxy service, with on-demand TLS support. We decided against using Fargate, and for using EC2 due to cost reasons. This prioritized costs higher, than running as Serverless as possible. It's possible that, in another setting / environment / experience, you might come to another conclusion, which is completely fine.

For a more production-like setup, you'd probably need to amend the Domain Verifier Lambda function, so that it dynamically looks up the custom domains that are configured e.g. by your customers via a UI, and stored in another DynamoDB table via another Lambda function. Deleting or updating those custom domains should probably be possible, too.

Furthermore, you should then write an additional Lambda function that recurrently checks each stored custom domain if:

The CNAME records point to your external.$YOUR_DOMAIN_NAME.tld, and updates the status accordingly
Performs a check via HTTPS whether an actual redirect from the custom domain to your domain is possible