DuckDB, the in-process DBMS specialized in OLAP workloads, had a very rapid growth during the last year, both in functionality, but also popularity amongst its users, but also with developers that contribute many projects to the Open Source DuckDB ecosystem.

DuckDB cannot "only" be run on a variety of Operating Systems and Architectures, there's also a DuckDB-WASM version, that allows running DuckDB in a browser. This opens up some very interesting use cases, and is also gaining a lot of traction in the last 12 months.

Use Case: Building a SQL Workbench with DuckDB-WASM

One of the first things that came to my mind once I learned about the existence of DuckDB-WASM was that it could be used to create an online SQL Workbench, where people could interactively run queries, show their results, but also visualize them. DuckDB-WASM sits at its core, providing the storage layer, query engine and many things more...

You can find the project at

It's built with the following core technologies / frameworks:

• React

• Vite

• DuckDB-WASM

• Perspective.js

It's hosted as a static website export / single page application on AWS using

• CloudFront as CDN

• S3 as file hosting service

• ACM for managing certificates

• Route53 for DNS

If you're interested in the hosting setup, you can have a look at https://github.com/tobilg/serverless-aws-static-websites which can deploy such static websites on AWS via IaC with minimum effort.

Using the SQL Workbench

There are many possibilities how you can use the SQL Workbench, some are described below

Overview

When you open sql-workbench.com for the first time, you can see that the workbench is divided in three different areas:

• On the left, there's the "Local Tables" area, that will display the created tables of you ran queries such as CREATE TABLE names (name VARCHAR), or used the drag-and-drop area on the lower left corner to drop any CSV, Parquet or Arrow file on it (details see below).

• The upper main editor area is the SQL editor, where you can type your SQL queries. You're already presented with some example queries for different types of data once the page is loaded.

• The lower main result area where the results of the ran queries will be shown, or alternatively, the visualizations of these results.

Running SQL queries

To run your first query, select the first line of SQL, either with your keyboard or with your mouse, and press the key combination CMD + Enter of you're on a Mac, or Ctrl + Enter if you're on a Windows or Linux machine.

The result of the query that was executed can then be found in the lower main area as a table:

Running multiple queries

You can also run multiple queries sequentially, e.g. to create a table, insert some records, and display the results:

CREATE TABLE first_names (name VARCHAR, birth_cnt integer);INSERT INTO first_names (name, birth_cnt) VALUES ('Liam', 20456);INSERT INTO first_names (name, birth_cnt) VALUES ('Noah', 18621);INSERT INTO first_names (name, birth_cnt) VALUES ('Oliver', 15076);INSERT INTO first_names (name, birth_cnt) VALUES ('James', 12028);INSERT INTO first_names (name, birth_cnt) VALUES ('Elijah', 11979);INSERT INTO first_names (name, birth_cnt) VALUES ('William', 11282);INSERT INTO first_names (name, birth_cnt) VALUES ('Henry', 11221);INSERT INTO first_names (name, birth_cnt) VALUES ('Lucas', 10909);INSERT INTO first_names (name, birth_cnt) VALUES ('Benjamin', 10842);INSERT INTO first_names (name, birth_cnt) VALUES ('Theodore', 10754);SELECT * FROM first_names;

When you copy & paste the above SQLs, select them and run them, the result looks like this:

You can see on the left-hand side the newly created table first_names, that can be reused for other queries without having to reload the data again.

If you want to open a new SQL Workbench and directly run the above query, please click on the image below:

Querying data you have on your machine

To try this, you can for example download a list of AWS Services as a CSV from

https://raw.githubusercontent.com/tobilg/aws-iam-data/main/data/csv/aws_services.csv

This file has four columns, service_id, name, prefix and reference_url. Once you downloaded the file, you can simply drag-and-drop from the folder you downloaded it to to the area in the lower left corner of the SQL Workbench:

A table called aws_services.csv has now been automatically created, which you can query via SQLs, for example:

SELECT name, prefix from 'aws_services.csv';

If you want to open a new SQL Workbench and directly run the above query, please click on the image below:

Querying and visualizing remote data

DuckDB-WASM supports the loading of compatible data in different formats (e.g. CSV, Parquet or Arrow) from remote http(s) sources. Other data formats that can be used include JSON, but this requires the loading of so-called DuckDB extensions.

In this example, we will use data about AWS CloudFront Edge Locations, that is available at tobilg/aws-edge-locations with this query:

SELECT * FROM 'https://raw.githubusercontent.com/tobilg/aws-edge-locations/main/data/aws-edge-locations.parquet';

The result will look like this:

We now want to create a bar chart of the data, showing the number of Edge Locations by country and city. This can be done by hovering over the result table, and clicking on the small "configure" button that looks like a wrench which subsequently appears on the upper right corner of the table:

You then see the overview of the available columns, and the current visualization type (Datagrid)

To get an overview of the possible visualization types click on the Datagrid icon:

Then select "Y Bar". This will give you an initial bar char:

But as we want to display the count of Edge Locations by country and city, we need to drag-and-drop the columns country and city to the "Group By" area:

We can now close the configuration menu to see the chart in it's full size:

There are many other visualization types from which you can choose from, such as Treemaps and Sunbursts:

Exporting visualizations and data

You can also export the visualizations, as well as the data. Just click on "Export" and type in a "Save as" name, and select the output format you want to download:

The data can be downloaded as CSV, JSON or Arrow file. Here's the CSV example:

"country (Group by 1)","count",517"Argentina",3"Australia",10"Austria",3"Bahrain",2"Belgium",1"Brazil",21"Bulgaria",3"Canada",8"Chile",6"China",8"Colombia",3"Croatia",1"Czech Republic",1"Denmark",3"Finland",4"France",17"Germany",37"Greece",1"Hungary",1"India",48"Indonesia",5"Ireland",2"Israel",2"Italy",16"Japan",27"Kenya",1"Korea",8"Malaysia",2"Mexico",4"Netherlands",5"New Zealand",2"Nigeria",1"Norway",2"Oman",1"Peru",2"Philippines",2"Poland",5"Portugal",1"Romania",1"Singapore",7"South Africa",2"Spain",12"Sweden",4"Switzerland",2"Taiwan",3"Thailand",2"UAE",4"UK",30"United States",179"Vietnam",2

And here's the exported PNG:

Exporting the HTML version will give you an interactive graph with hovering etc. Furthermore, you can also change the theme for the different visualizations:

This is also reflected in the exported graphs:

Using the schema browser

The schema browser can be found on the left-hand side. It's automatically updated after each executed query, so that all schema operations can be captured. On table-level, the columns, constraints and indexes are shown:

If you right-click on a table name, a context menu is shown that has different options:

• Generating scripts based on the table definition

• Truncating, deleting or summarizing the table

• Viewing table data (all records, first 10 and first 100)

Once clicked, those menu items will create a new tab (see below), and generate and execute the appropriate SQL statements:

Using query tabs

Another new feature is the possibility to have multiple query tabs. Those are either automatically created by context menu actions, or the user that clicks on the plus icon:

Each tab can be closed by clicking on the "x" icon next to the tab name.

Generating data models

If users have created some tables, it's then possible to create a data model from the schema metadata. If the tables also have foreign key relationships, those are also shown in the diagram. Just click on the "Data Model" menu entry on the lower left corner.

Under the hood, this feature generates Mermaid Entity Relationship Diagram code, that is dynamically rendered as a graph.

Using the query history

Each query that is issued in the current version of the SQL Workbench is recorded for the so-called query history. It can be accessed by clicking on the "Query History" menu entry in the lower left corner. Once clicked, there's an overlay on the right-hand side with the list of the issued queries.

The newest queries can be found on top of the list, and with each query listed, there's also an indication when the query was run, and how long it took to execute.

With the trash icon in the top-right corner, the complete query history can be truncated. Also, single query history entries can be deleted, as well as specific queries can be re-run in a new tab by clicking "Replay query" in the menu that's present for each query history entry.

Example Data Engineering pipeline

Dataset & Goals

A well-known dataset is the NYC TLC Trip Record dataset. It can be found freely available on the website of NYC Taxi and Limousine Commission website. It also comes with some explanations and additional lookup data. In this example, we focus on the yellow taxi data.

The goal of this example pipeline is to create a clean Data Mart from the given trip records and location data, being able to support some basic analysis of the data via OLAP patterns.

Source Data analysis

On the NYC TLC website, there's a PDF file explaining the structure and contents of the data. The table structure can be found below, the highlighted columns indicate dimensional values, for which we'll build dimension tables for in the later steps.

There's an additional CSV file for the so-called Taxi Zones, as well as a SHX shapefile containing the same info, but with an additional geo information. The structure is the following:

Target Data Model

The target data model is derived from the original trip record data, with extracted dimension tables plus a new date hierarchy dimension. Also, the naming schema gets unified and cleaned up.

It is modeled as a so-called Snowflake Schema (check the Mermaid source):

Loading & Transforming the data

The loading & transforming of the data is divided in multiple steps:

• Generating the dimensional tables and values from the given dataset information

• Generating a date hierarchy dimension

• Loading and transforming the trip data

  • Replacing values with dimension references

  • Clean the column naming

  • Unify values

Generating the dimensional tables

We use the given dataset information from the PDF file to manually create dimension tables and their values:

-- Install and load the spatial extensionINSTALL spatial;LOAD spatial;-- Create temporary tableCREATE TABLE tmp_service_zones AS (    SELECT         DISTINCT service_zone     FROM         'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv'     WHERE         service_zone != 'N/A'     ORDER BY         service_zone); -- Create dim_zone_type tableCREATE TABLE dim_zone_type (    zone_type_id INTEGER PRIMARY KEY,    name VARCHAR);-- Insert dim_zone_type tableINSERT INTO dim_zone_typeSELECT     -1 as zone_type_id,     'N/A' as name UNION ALLSELECT     (rowid + 1)::INTEGER as zone_type_id,     service_zone as name FROM     tmp_service_zones ; -- Drop table tmp_service_zonesDROP TABLE tmp_service_zones; -- Create temporary tableCREATE TABLE tmp_borough AS (    SELECT         DISTINCT borough     FROM         'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv'     WHERE        borough != 'Unknown'    ORDER BY         borough); -- Create dim_borough tableCREATE TABLE dim_borough (    borough_id INTEGER PRIMARY KEY,    name VARCHAR);-- Insert dim_borough tableINSERT INTO dim_boroughSELECT     -1 as borough_id,     'N/A' as name UNION ALLSELECT     (rowid + 1)::INTEGER as borough_id,     borough as name FROM     tmp_borough ; -- Drop temporary tableDROP TABLE tmp_borough;-- Create dim_zone tableCREATE TABLE dim_zone (    zone_id INTEGER PRIMARY KEY,    zone_type_id INTEGER,    borough_id INTEGER,    name VARCHAR,    geojson VARCHAR,    FOREIGN KEY (zone_type_id) REFERENCES dim_zone_type (zone_type_id),    FOREIGN KEY (borough_id) REFERENCES dim_borough (borough_id));-- Insert dim_zone tableINSERT INTO dim_zone SELECT DISTINCT    CASE        WHEN csv.LocationID IS NOT NULL THEN csv.LocationID::INT        ELSE raw.LocationID    END AS zone_id,     zt.zone_type_id,     CASE        WHEN b.borough_id IS NOT NULL THEN b.borough_id        ELSE -1    END AS borough_id,    CASE        WHEN csv.Zone IS NOT NULL THEN csv.Zone        ELSE raw.zone    END AS name,     raw.geojson FROM     (        SELECT             LocationID,             borough,            zone,              geojson         FROM             (                SELECT                     LocationID,                     borough,                     zone,                     rank() OVER (PARTITION BY LocationID ORDER BY Shape_Leng) AS ranked,                     ST_AsGeoJSON(ST_Transform(geom, 'ESRI:102718', 'EPSG:4326')) AS geojson                 FROM ST_Read('https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.shx')             ) sub        WHERE             sub.ranked = 1     ) raw FULL OUTER JOIN     (        SELECT DISTINCT             LocationID,            Zone,             service_zone         FROM             'https://data.quacking.cloud/nyc-taxi-metadata/taxi_zones.csv'     ) csv ON     csv.LocationId = raw.LocationId FULL OUTER JOIN     dim_zone_type zt ON     csv.service_zone = zt.name FULL OUTER JOIN     dim_borough b ON     b.name = raw.boroughWHERE    zone_id IS NOT NULLORDER BY    zone_id;-- Create dim_rate_code tableCREATE TABLE dim_rate_code (    rate_code_id INTEGER PRIMARY KEY,    name VARCHAR);INSERT INTO dim_rate_code (rate_code_id, name) VALUES (1, 'Standard rate');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (2, 'JFK');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (3, 'Newark');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (4, 'Nassau or Westchester');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (5, 'Negotiated fare');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (6, 'Group ride');INSERT INTO dim_rate_code (rate_code_id, name) VALUES (99, 'N/A');-- Create dim_payment_type tableCREATE TABLE dim_payment_type (    payment_type_id INTEGER PRIMARY KEY,    name VARCHAR);INSERT INTO dim_payment_type (payment_type_id, name) VALUES (1, 'Credit card');INSERT INTO dim_payment_type (payment_type_id, name) VALUES (2, 'Cash');INSERT INTO dim_payment_type (payment_type_id, name) VALUES (3, 'No charge');INSERT INTO dim_payment_type (payment_type_id, name) VALUES (4, 'Dispute');INSERT INTO dim_payment_type (payment_type_id, name) VALUES (5, 'Unknown');INSERT INTO dim_payment_type (payment_type_id, name) VALUES (6, 'Voided trip');-- Create dim_vendor tableCREATE TABLE dim_vendor (    vendor_id INTEGER PRIMARY KEY,    name VARCHAR);INSERT INTO dim_vendor (vendor_id, name) VALUES (1, 'Creative Mobile Technologies');INSERT INTO dim_vendor (vendor_id, name) VALUES (2, 'VeriFone Inc.');-- Create dim_stored_type tableCREATE TABLE dim_stored_type (    stored_type_id INTEGER PRIMARY KEY,    name VARCHAR);INSERT INTO dim_stored_type (stored_type_id, name) VALUES (1, 'Store and forward trip');INSERT INTO dim_stored_type (stored_type_id, name) VALUES (2, 'Not a store and forward trip');

Generating a date hierarchy dimension

-- Create dim_date tableCREATE TABLE dim_date (    day_dt DATE PRIMARY KEY,    day_name VARCHAR,    day_of_week INT,    day_of_month INT,    day_of_year INT,    week_of_year INT,    month_of_year INT,    month_name VARCHAR,    year INT);INSERT INTO dim_dateSELECT     date_key AS day_dt,    DAYNAME(date_key)::VARCHAR AS day_name,    ISODOW(date_key)::INT AS day_of_week,    DAYOFMONTH(date_key)::INT AS day_of_month,    DAYOFYEAR(date_key)::INT AS day_of_year,     WEEKOFYEAR(date_key)::INT AS week_of_year,    MONTH(date_key)::INT AS month_of_year,    MONTHNAME(date_key)::VARCHAR AS month_name,    YEAR(date_key)::INT AS yearFROM     (        SELECT             CAST(RANGE AS DATE) AS date_key         FROM             RANGE(DATE '2005-01-01', DATE '2030-12-31', INTERVAL 1 DAY)    ) generate_date;

Loading and transforming the trip data

-- Create sequence for generating trip_idsCREATE SEQUENCE trip_id_sequence START 1;-- Create fact_trip tableCREATE TABLE fact_trip (    trip_id INTEGER  DEFAULT nextval('trip_id_sequence') PRIMARY KEY,    pickup_zone_id INTEGER,    pickup_dt DATE,    pickup_ts TIMESTAMP,    dropoff_zone_id INTEGER,    dropoff_dt DATE,    dropoff_ts TIMESTAMP,    rate_code_id INTEGER,     stored_type_id INTEGER,     payment_type_id INTEGER,     vendor_id INTEGER,     passenger_count DOUBLE,    trip_distance_miles DOUBLE,    fare_amount DOUBLE,    extra_amount DOUBLE,    mta_tax_amount DOUBLE,    improvement_surcharge_amount DOUBLE,    tip_amount DOUBLE,    tolls_amount DOUBLE,    congestion_surcharge_amount DOUBLE,    airport_fee_amount DOUBLE,    total_amount DOUBLE);-- Deactivating FK relationships for now, due to performance issues when inserting 3 million records-- FOREIGN KEY (pickup_zone_id) REFERENCES dim_zone (zone_id),-- FOREIGN KEY (dropoff_zone_id) REFERENCES dim_zone (zone_id),-- FOREIGN KEY (pickup_dt) REFERENCES dim_date (day_dt),-- FOREIGN KEY (dropoff_dt) REFERENCES dim_date (day_dt),-- FOREIGN KEY (rate_code_id) REFERENCES dim_rate_code (rate_code_id),-- FOREIGN KEY (stored_type_id) REFERENCES dim_stored_type (stored_type_id),-- FOREIGN KEY (payment_type_id) REFERENCES dim_payment_type (payment_type_id),-- FOREIGN KEY (vendor_id) REFERENCES dim_vendor (vendor_id)-- Insert transformed fact dataINSERT INTO fact_tripSELECT     nextval('trip_id_sequence') AS trip_id,    PULocationID::INT as pickup_zone_id,    tpep_pickup_datetime::DATE as pickup_dt,    tpep_pickup_datetime AS pickup_ts,    DOLocationID::INT as dropoff_zone_id,    tpep_dropoff_datetime::DATE as dropoff_dt,    tpep_dropoff_datetime AS dropoff_ts,    RatecodeID::INT AS rate_code_id,    CASE        WHEN store_and_fwd_flag = 'Y' THEN 1        WHEN store_and_fwd_flag = 'N' THEN 2    END AS stored_type_id,    payment_type::INT AS payment_type_id,    VendorID::INT AS vendor_id,    passenger_count,    trip_distance AS trip_distance_miles,    fare_amount,    extra AS extra_amount,    mta_tax AS mta_tax_amount,    improvement_surcharge AS improvement_surcharge_amount,    tip_amount,    tolls_amount,    congestion_surcharge AS congestion_surcharge_amount,    airport_fee AS airport_fee_amount,    total_amountFROM     'https://data.quacking.cloud/nyc-taxi-data/yellow_tripdata_2023-01.parquet';

Data Analysis

The following analyses are just examples on how you could analyze the data set. Feel free to think about your own questions for the dataset, and try to build queries yourselves!

Preparation

DuckDB supports the SUMMARIZE command can help you understand the final data in the fact table before querying it. It launches a query that computes a number of aggregates over all columns, including min, max, avg, std and approx_unique:

The output already shows some "interesting" things, such as pickup_dt being far in the past, e.g. 2008-12-31, and the dropoff_dt with similar values (2009-01-01).

Most utilized trip locations

With this analysis, we want to have a look at the 20 most frequented trips from pickup zone to dropoff zone:

SELECT    pz.name || ' -> ' || dz.name AS trip_description,    count(DISTINCT ft.trip_id)::INT AS trip_count,    sum(ft.passenger_count)::INT AS passenger_count,    sum(ft.total_amount)::INT AS total_amount,    sum(ft.trip_distance_miles)::INT AS trip_distance_miles,    (sum(trip_distance_miles)/count(DISTINCT ft.trip_id))::DOUBLE AS trip_distance_miles_avg,    (sum(ft.total_amount)/count(DISTINCT ft.trip_id))::DOUBLE AS total_amount_avg,    (sum(ft.passenger_count)/count(DISTINCT ft.trip_id))::DOUBLE AS passenger_count_avgFROM    fact_trip ftINNER JOIN    dim_zone pzON    pz.zone_id = ft.pickup_zone_idINNER JOIN    dim_zone dzON    dz.zone_id = ft.dropoff_zone_idGROUP BY    trip_descriptionORDER BY    trip_count DESCLIMIT 20;

On a M2 Mac Mini with 16GB RAM, aggregating the 3 million trips takes around 900ms. The result looks like the following:

We can now also create a Y Bar chart showing the (total) trip count, passenger count and trip distance by top 20 most frequented trips:

End result:

Trip frequency by weekday and time of day

To inspect the traffic patterns, we want to analyze the trip frequency by weekday and time of day (aggregated on hourly level). Therefore, we make use of DuckDB's advanced timestamp/time handling functions:

SELECT    dd.day_name,    dd.day_of_week,    datepart('hour', time_bucket(INTERVAL '1 HOUR', ft.pickup_ts)) day_hour,    count(DISTINCT ft.trip_id)::INT AS trip_count,FROM    fact_trip ftINNER JOIN    dim_date ddON    dd.day_dt = ft.pickup_dtGROUP BY    dd.day_name,    dd.day_of_week,    day_hourORDER BY    dd.day_of_week,    day_hour;

Then, we can configure a Y Bar chart that can show us the number of trips by weekday and hour:

End result:

Sharing of Data Pipelines & Visualizations

With the latest version of sql-workbench.com it's possible to share both queries and the (customized) visualization of the last executed query. Therefore, you write your queries, run them to check whether they work, and then update the visualization configuration.

When you did that, you can then click on "Share queries" in the lower left corner of the SQL Workbench. The toggle will let you choose whether you want to copy the visualization configuration as well, or not.

If you want to run the complete pipeline to build our dimensional data model, you can click on the link below (this can take from 10 to 120 seconds depending on your machine and internet connection speed, as approximately 50MB of data will be downloaded):

Conclusion

With DuckDB-WASM and some common web frameworks, it's pretty easy and fast to create custom data applications that can handle datasets in the size of millions of records.

Those applications are able to provide a very lightweight approach to working with different types of data (such as Parquet, Iceberg, Arrow, CSV, JSON or spatial data formats), whether locally or remote (via HTTP(S) of S3-compatible storage services), thanks to the versatile DuckDB engine.

Users can interactively work with the data, create data pipelines by using raw SQL, and iterate until the final desired state has been achieved. The generated data pipeline queries can easily be shared with simple links to sql-workbench.com, so that other collaborators can continue to iterate on the existing work, or even create new solutions with it.

Once a data pipeline has been finalized, it could for example be deployed to DuckDB instances running in own cloud accounts of the users. A great example would be running DuckDB in AWS Lambda, e.g. for repartitioning Parquet data in S3 nightly, or automatically running reports based on aggregation pipelines etc.

The possibilities are nearly endless, so I'm very curious what you all build with this great technology! Thanks for reading this length article, I'm happy to answer any questions in the comments.

AWS Lambda@Edge is an extension of the traditional AWS Lambda service, but with a crucial twist it brings serverless computing capabilities closer to the end-users.

In essence, Lambda@Edge empowers developers to run custom code in response to specific CloudFront events. Whether it's tailoring content based on user location, device type, or handling real-time data processing at the edge, Lambda@Edge functions open up a world of possibilities for optimizing content delivery.

Lambda@Edge functions have the following features:

• They can access public internet

• They can be run before or after your cache

• They allow developers to view and modify not only the client request/response but also the origin request/response

• They can read the body of the request

• They can use a much bigger package size for code compared to CloudFront Functions (1MB for a client request/response trigger and 50MB for an origin request/response trigger)

• They allow up to 5 seconds for a client request/response trigger and up to 30 seconds for an origin request/response trigger.

How can you retrieve the Lambda@Edge logs?

Due to the fact that Lambda@Edge functions run in the CloudFront Regional Edge Caches, the CloudWatch logs cannot only be found in us-east-1, but potentially in all other AWS regions that support Regional Edge Caches.

The Lambda@Edge testing and debugging guide states

When you review CloudWatch log files or metrics when you're troubleshooting errors, be aware that they are displayed or stored in the Region closest to the location where the function executed. So, if you have a website or web application with users in the United Kingdom, and you have a Lambda function associated with your distribution, for example, you must change the Region to view the CloudWatch metrics or log files for the London AWS Region.

So, one simple solution to automatically get the logs from all regions would be to write a script that searches for relevant CloudWatch Logs groups in each region, and displays their respective LogStream entries.

This script could look like this

#!/bin/bashFUNCTION_NAME=$1for region in $(aws --output text ec2 describe-regions | cut -f 4) do  echo "Checking $region"  for loggroup in $(aws --output text logs describe-log-groups --log-group-prefix "/aws/lambda/us-east-1.$FUNCTION_NAME" --region $region --query 'logGroups[].logGroupName')  do    echo "Found '$loggroup' in region $region"    for logstream in $(aws --output text logs describe-log-streams --log-group-name $loggroup --region $region --query 'logStreams[].logStreamName')    do      aws --output text logs get-log-events --log-group-name $loggroup --region $region --log-stream-name $logstream | cat    done  donedone

If you save this script as cf-logs.sh, after giving it execution rights with chmod +x cf-logs.sh, can then be started with ./cf-logs.sh YOUR_FUNCTION_NAME, where YOUR_FUNCTION_NAME is the real Lambda@Edge function name.

This solution is great for quick log viewing while you're still developing your edge-enabled application, but surely is not a sustainable one when running in production.

Aggregating Lambda@Edge logs with Kinesis

For a production setup, you'd probably want to be able to aggregate and store the individual logs coming from the different regions in one place. A possible solution is to stream them to a Kinesis Firehose Delivery Stream, which then stores the logs in S3.

An example implementation can be found at https://gist.github.com/heitorlessa/5d2295655f9d76483969d215986e53b0

Please be aware that this will incur additional fixed and variable costs, so please review AWS' pricing of the used services.

• AWS Knowledge: Cloud Essentials

• AWS Knowledge: Architecting

• AWS Knowledge: Serverless

• AWS Knowledge: Object Storage

• AWS Knowledge: Block Storage

• AWS Knowledge: File Storage

• AWS Knowledge: Data Migration

• AWS Knowledge: Data Protection & Disaster Recovery

• AWS Knowledge: Networking Core

• AWS Knowledge: Compute

• AWS Knowledge: EKS

• AWS Knowledge: Events and Workflows

• AWS Knowledge: Braket

• AWS Knowledge: AWS for Games: Cloud Game Development

• AWS Knowledge: Media & Entertainment: Direct-to-Consumer and Broadcast Foundations

In today's data-driven world, interactive and visually appealing web-based maps have become an integral part of countless applications and services. Whether it's for navigation, location-based services, or data visualization, delivering seamless and lightning-fast user experiences is paramount. In this article, we explore how to host web-based maps on Amazon Web Services (AWS) in a serverless manner, exploring how this approach not only boosts speed but also brings significant cost advantages.

Web-Based Maps: A Key to Engaging User Experiences

The ubiquity of smartphones and the proliferation of internet-connected devices have transformed how we interact with the digital world. Whether we're exploring a new city, tracking our fitness activities, or monitoring real-time data, web-based maps provide a dynamic and immersive way to engage users. However, delivering smooth and responsive map experiences can be challenging, especially as data sizes and user demands continue to grow.

Traditionally, hosting web-based maps required dedicated servers or complex infrastructure setups, which often resulted in high operational costs and scalability limitations. Fortunately, the advent of serverless computing and the capabilities offered by AWS have strongly improved how we can deploy and manage web applications, including mapping services.

Serverless map hosting: A Game-Changer

AWS' serverless services allow developers to build and run applications without worrying about provisioning and managing servers. This approach not only simplifies development but also offers numerous advantages, particularly when it comes to hosting web-based maps.

1. Speed and Responsiveness: With serverless hosting via CloudFront and S3, the infrastructure caches data on the edge. This means that users will experience near-instantaneous map loading and seamless interactions, translating to heightened satisfaction and engagement.

2. Cost Efficiency: One of the most compelling aspects of the serverless approach is its cost-effectiveness. Traditional hosting methods often require paying for idle server resources, which can quickly add up and strain budgets. In contrast, serverless computing on AWS charges you only for the actual compute resources consumed during map requests. As a result, you can significantly reduce operational costs and allocate resources more efficiently.

In this comprehensive guide, we'll walk you through the steps of setting up a serverless web-based map hosting solution on AWS. From leveraging AWS Lambda for dynamic map tile querying to utilizing S3 for scalable and cost-effective data storage and using CloudFront for edge caching, you'll learn how to harness the full potential of AWS services for a top-notch mapping experience.

Choosing a map tile format

In the realm of digital maps, efficiency and performance are paramount. Whether you're navigating through city streets or exploring remote terrains, quick loading times and seamless zooming can make all the difference in delivering a superior user experience. Enter PMTiles, a cutting-edge map tile format that is redefining how we interact with digital maps.

What are PMTiles?

Traditional map tile formats, such as the popular PNG or JPEG images, can be bulky and slow to load, especially when dealing with intricate cartography or high-resolution imagery. PMTiles was specifically designed to address these limitations, offering a lightweight and performant solution for delivering map tiles.

According to the protomaps.com website

PMTiles is a single-file archive format for pyramids of tiled data. A PMTiles archive can be hosted on a storage platform like S3, and enables low-cost, zero-maintenance map applications.

Concepts include

• A general format for tiled data addressable by Z/X/Y coordinates, which can be cartographic basemap vector tiles, remote sensing observations, JPEG images, or more.

• Readers use HTTP Range Requests to fetch only the relevant tile or metadata inside a PMTiles archive on-demand.

• The arrangement of tiles and directories is designed to minimize the amount of overhead requests when panning and zooming.

The current specification of PMTiles v3 can be found on GitHub.

Why PMTiles matter?

1. Faster Loading Times: PMTiles leverages the power of vector data to store map tiles, which means that the tiles are smaller in size compared to raster image formats like PNG or JPEG. This smaller size translates to quicker loading times, enabling a seamless map browsing experience even in areas with limited network connectivity.

2. Efficient Storage: Due to their compact size, PMTiles require less storage space than traditional map tile formats. This advantage is particularly significant for applications with large map datasets, as it reduces the infrastructure costs associated with storing and serving the maps.

3. Dynamic Styling: PMTiles offer the flexibility of dynamic styling, allowing developers to modify the appearance of map tiles on the fly. By adjusting colors, labels, and map features in real-time, developers can create customized maps that cater to the specific needs of their users.

4. Offline Accessibility: One of the most impressive features of PMTiles is its ability to support offline access. By preloading PMTiles on a user's device, applications can ensure uninterrupted map access, even when an internet connection is not available. This functionality is invaluable for users in remote locations or areas with unstable network coverage. Also, usage of cache headers like Cache-Control, when used in a browser, can save bandwidth and rendering time.

How can PMTiles be used?

PMTiles can be used with common maps rendering engines like Leaflet, MapLibre GL or OpenLayers by using the protomaps.js library. Examples of how to integrate it can be found on the website, or the index.html and basemap.html examples in our repo.

Generating PMTiles from OpenStreetMap data

The first step for hosting your self-hosted maps can be downloading publically available map data from OpenStreetMap. This data is then to be transformed into a compatible basemap layer for use with PMTiles.

See the documentation about vector basemap layers to get an idea of how this works in general. The following is necessary to mention though (from protomaps.com):

The organization of features with layers and tags is specific to Protomaps services; this means that map styles are not directly portable with other systems such as OpenMapTiles or Mapzen tiles

This step can be automated to a high extent.

Solution

Enter ServerlessMaps! The project is outlined in the following paragraphs.

Architecture

In our example implementation, we will only rely on a very few services:

• AWS Lambda (proxying the map tile requests to the S3 origin)

• Amazon CloudFront (globally distributed CDN)

• Amazon S3 (storing the PMTiles files, and the map example website)

• Amazon CloudWatch (keeping the logs of the Lambda function)

The overall architecture looks like this:

Deployment

The deployment of ServerlessMaps must be done in multiple steps. From a high-level perspective, they are

• Setting up the project from GitHub locally

• Setting up the local environment to prepare the basemaps

• Build the desired basemaps as PMTiles file

• Deploy the serverless infrastructure on AWS

• Upload the basemaps and example websites to S3 (done automatically)

Setting up the project locally

To clone the project to your local machine, please use

git clone https://github.com/serverlessmaps/serverlessmaps.git

in your desired directory. Then, do a cd serverlessmaps in the directory you ran the above command.

Setting up the local environment

This step assumes that you're using MacOS as the operating system. If so, you can run

scripts/install_macos.sh

to install the dependencies needed to build the basemaps as PMTiles file. This will install the latest OpenJDK and Maven, which are both necessary for the basemaps build.

If your system is not using MacOS, you can also install those two dependencies manually.

The next step is to compile the Planetiler build profile, that later can generate the PMTiles file:

scripts/compile_basemaps_builder.sh

This will create a new directory builder that will contain the JAR with the runnable builder.

Build the desired basemaps

To build a basemap with the before compiled builder, run

scripts/build_pmtiles.sh

This will build a map with a default area of Hamburg / Germany. If you want to generate maps for other areas, have a look at the OSM (sub-)region names, e.g. on the https://download.geofabrik.de/ server.

For example, if you'd like to generate a map for the whole of Europe, you could run

scripts/build_pmtiles.sh europe

Please be aware that this will run for several hours depending on your machine, and will generate a PMTiles file of around 45GB. This file will take some time to upload to S3 in the next step as well. The recommendation is if you just want to try out this project, use the default hamburg sub-region, which is around 35MB.

Deploy the serverless infrastructure

This project assumes that you already have set up your AWS credentials locally so that the Serverless framework can use it accordingly.

To deploy the serverless AWS infrastructure, you can do a cd iac from the project's root directory, and then use

sls deploy

to deploy the necessary stack.

You can customize some parameters for the deployment:

• region: The AWS region you want to deploy your stack to (default: us-east-1)

• stage: The stage name (default: prd)

• cors: The allowed hostname for the CORS header (default: *)

The following will deploy the stack to the eu-central-1 region with the stage name dev and the allowed CORS hostname mymapservice.xyz:

sls deploy --region eu-central-1 --stage dev --cors mymapservice.xyz

Stack output

The deployment of the stack will generate an output like this on the console:

----------------------------------------------------------------------------------> The map can be viewed at https://d1b056iuztreqte.cloudfront.net/#13.54/53.54958/9.99286-> The basemap themes can be viewed at https://d1b056iuztreqte.cloudfront.net/basemap.html#13.54/53.54958/9.99286-> Please manually do a 'npm run website:sync' on your console to sync the static website assets if you changed them after the last deployment-> Afterwards, run 'npm run website:invalidate' to invalidate the website's CloudFront distribution cache---------------------------------------------------------------------------------

Automatically created files

There will be two automatically created files based on the setting you chose before:

• website/urlConfig.js: This contains the CloudFront Distribution hostname (variable tilesDistributionHostname) for the caching of the PMTiles. This is assigned by CloudFront during deployment.

• website/tilePath.js: This contains the needed tilePath variable, which depends on the area you chose for the basemap. This is generated by the scripts/build_pmtiles.sh script automatically.

Upload the basemaps and example websites

The basemap that was generated in the step before the deployment, and the two example websites are synched automatically to the website S3 bucket.

If you want to deploy your web application/website, you need to run the sync manually via npm run website:sync. After that, the CloudFront cache needs to be invalidated as well to show the new content. This can be done via npm run website:invalidate, both from the iac directory.

Result

If everything went well, you can access the URL (https://d1b056iuztreqte.cloudfront.net/#13.54/53.54958/9.99286 in the above example output) to view your basic map:

Conclusion

Hosting web-based maps on AWS in a serverless manner opens up a world of opportunities for delivering blazing-fast, interactive, and cost-efficient map services. By taking advantage of AWS's scalable infrastructure and pay-as-you-go pricing model, you can create an outstanding user experience without breaking the bank.

The goal of this article is to outline how this data can be unified, cleaned and made available on a platform that makes it easy for users to consume. Furthermore, some interesting statistics can be derived from those public datasets.

The data and the source code can be found at:

• https://github.com/tobilg/public-cloud-provider-ip-ranges

Data sources

The (incomplete) list of public cloud providers that are publishing their IPv4 CIDR blocks are:

• AWS

• Azure

• CloudFlare

• DigitalOcean

• Fastly

• Google Cloud

• Oracle Cloud

You can click on the individual links to view or download the data manually.

HINT: The published lists of IP address ranges don't represent the overall IP address space that each of these providers are possessing.

To get the complete IP range, data from organizations like ARIN would need to be added as well. For simplicity and brevity, only the publically downloadable info was used.

Retrieving, cleaning, storing and exporting data

The overall data engineering process is divided into multiple steps:

• Identify data sources (see above) and define the common schema

• Retrieving the raw data from public data sources (via HTTP)

• Cleaning the retrieved data (e.g. remove duplicates)

• Storing the data in a common schema, so that it can be aggregated and analyzed for different public cloud providers at once

• Exporting the stored data into different file formats, so that as many different types of clients can make use of it

Additionally, we'd like to keep the costs low, and the infrastructure as simple as possible. That's why DuckDB is chosen as the database layer, which offers a rich and advanced set of features to handle (read and write) different file formats, as well as it can read directly from remote data sources via HTTP, only by using SQL. That saves additional effort for out-of-band ETL.

Furthermore, to share the data, we chose GitHub, which is free to use for the scope of our use case. Most importantly, it allows us to store the exported data files in our repository. To run the overall process, GitHub Actions are used as they also offer a free usage tier, and have everything we need to create the described data pipeline.

Common data schema

After inspecting the data source files, the derived unified schema for all loaded data sources will look like this:

Creating the cloud provider tables

At first, we'll create a table in DuckDB for each of the public cloud providers. If DuckDB is installed (see docs) and in the PATH, we can execute SQL scripts like this:

# $DATA_PATH is the location of the DuckDB database file (this is important, because otherwise an in-memory table will be automatically created that will not be able to persist the data# $SCRIPT is the path to the SQL script that shall be executedduckdb $DATA_PATH < $SCRIPT.sql

Each table has different SQLs, as the data sources (contents and formats) of each provider are different.

Before starting, we need to make sure that the httpfs extension is installed and loaded (as we use remote datasets):

INSTALL httpfs;LOAD httpfs;

AWS table

CREATE TABLE aws_ip_data AS (  SELECT DISTINCT    prefixes.cidr_block,    prefixes.ip_address,    prefixes.ip_address_mask,    CAST(POW(2, 32-prefixes.ip_address_mask) AS INTEGER) AS ip_address_cnt,    prefixes.region  FROM (    SELECT      prefix_object.ip_prefix AS cidr_block,      STR_SPLIT(prefix_object.ip_prefix, '/')[1] AS ip_address,      CAST(STR_SPLIT(prefix_object.ip_prefix, '/')[2] AS INTEGER) AS ip_address_mask,      prefix_object.region    FROM (      SELECT UNNEST(prefixes) AS prefix_object FROM 'https://ip-ranges.amazonaws.com/ip-ranges.json'    )  ) prefixes);

Azure table

CREATE TABLE azure_ip_data AS (  SELECT DISTINCT    prefixes AS cidr_block,    STR_SPLIT(prefixes, '/')[1] AS ip_address,    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,    CASE      WHEN region = '' THEN 'No region'      ELSE region    END AS region  FROM (    SELECT DISTINCT      prop.region AS region,      UNNEST(prop.addressPrefixes) AS prefixes    FROM (      SELECT         values.properties AS prop      FROM (        SELECT           UNNEST(values) AS values        FROM          read_json_auto('https://download.microsoft.com/download/7/1/D/71D86715-5596-4529-9B13-DA13A5DE5B63/ServiceTags_Public_20230417.json', maximum_object_size=10000000)      )    )  )  WHERE    prefixes NOT LIKE '%::%');

CloudFlare table

CREATE TABLE cloudflare_ip_data AS (  SELECT DISTINCT    prefixes AS cidr_block,    STR_SPLIT(prefixes, '/')[1] AS ip_address,    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,    'No region' AS region  FROM (    SELECT      column0 AS prefixes    FROM      read_csv_auto('https://www.cloudflare.com/ips-v4')  ));

DigitalOcean table

CREATE TABLE digitalocean_ip_data AS (  SELECT DISTINCT    prefixes AS cidr_block,    STR_SPLIT(prefixes, '/')[1] AS ip_address,    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,    'No region' AS region  FROM (    SELECT      column0 AS prefixes    FROM      read_csv_auto('https://digitalocean.com/geo/google.csv')    WHERE      column0 NOT LIKE '%::%'  ));

Fastly table

CREATE TABLE fastly_ip_data AS (  SELECT DISTINCT    prefixes AS cidr_block,    STR_SPLIT(prefixes, '/')[1] AS ip_address,    CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER) AS ip_address_mask,    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,    'No region' AS region  FROM (    SELECT      UNNEST(addresses) AS prefixes    FROM      read_json_auto('https://api.fastly.com/public-ip-list')  ));

Google Cloud table

CREATE TABLE google_ip_data AS (  SELECT DISTINCT    prefixes.cidr_block,    prefixes.ip_address,    prefixes.ip_address_mask,    CAST(pow(2, 32-prefixes.ip_address_mask) AS INTEGER) AS ip_address_cnt,    prefixes.region  FROM (    SELECT      prefix_object.ipv4Prefix AS cidr_block,      str_split(prefix_object.ipv4Prefix, '/')[1] AS ip_address,      CAST(str_split(prefix_object.ipv4Prefix, '/')[2] AS INTEGER) AS ip_address_mask,      prefix_object.scope as region    FROM (      SELECT unnest(prefixes) AS prefix_object FROM 'https://www.gstatic.com/ipranges/cloud.json'    )    WHERE      prefix_object.ipv4Prefix IS NOT NULL  ) prefixes);

Oracle Cloud table

CREATE TABLE oracle_ip_data AS (  SELECT DISTINCT    prefixes.cidr AS cidr_block,    STR_SPLIT(prefixes.cidr, '/')[1] AS ip_address,    CAST(STR_SPLIT(prefixes.cidr, '/')[2] AS INTEGER) AS ip_address_mask,    CAST(POW(2, 32-CAST(STR_SPLIT(prefixes.cidr, '/')[2] AS INTEGER)) AS INTEGER) AS ip_address_cnt,    CASE      WHEN region = '' THEN 'No region'      ELSE region    END AS region  FROM (    SELECT DISTINCT      region,      UNNEST(cidrs) AS prefixes    FROM (      SELECT         regions.region AS region,        regions.cidrs AS cidrs      FROM (        SELECT           UNNEST(regions) AS regions        FROM          read_json_auto('https://docs.oracle.com/en-us/iaas/tools/public_ip_ranges.json', maximum_object_size=10000000)      )    )  ));

Create a combined view

The next step is to create a new view (ip_data) that combines our tables for the individual cloud providers. We can then use this view later to compare the different cloud providers.

The view definition looks like this:

CREATE VIEW ip_data AS (  SELECT 'AWS' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM aws_ip_data  UNION ALL  SELECT 'Azure' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM azure_ip_data  UNION ALL  SELECT 'CloudFlare' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM cloudflare_ip_data  UNION ALL  SELECT 'DigitalOcean' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM digitalocean_ip_data  UNION ALL  SELECT 'Fastly' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM fastly_ip_data  UNION ALL  SELECT 'Google Cloud' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM google_ip_data  UNION ALL  SELECT 'Oracle' as cloud_provider, cidr_block, ip_address, ip_address_mask, ip_address_cnt, region FROM oracle_ip_data);

Export the data

To be able to use the data with other tools, we need to export the data to different formats, in our case CSV and Parquet. You can review the executed queries in the repository.

-- Only an example, this needs to be done for all providers as well!-- Export complete data as CSVCOPY (SELECT * FROM ip_data ORDER BY cloud_provider, cidr_block) TO 'data/providers/all.csv' WITH (HEADER 1, DELIMITER ',');-- Export complete data as ParquetCOPY (SELECT * FROM ip_data ORDER BY cloud_provider, cidr_block) TO 'data/providers/all.parquet' (FORMAT 'parquet', COMPRESSION 'SNAPPY');

Analyze the data

As we now prepared our data, we can start analyzing it. Therefore, we'll use an ObservableHQ notebook, where we'll upload the all.csv file to.

Overall IP address counts

Total number and value of IP addresses

An astonishing insight is that both AWS and Azure have more than six times as many IP addresses available as their next competitor.

Also, the market values of their IP addresses are nearly four billion Dollars according to a market analysis.

CIDR masks distribution by public cloud provider

It's remarkable that, although AWS and Azure have similar absolute numbers of IP addresses, the type of CIDR blocks / IP ranges strongly differ: AWS owns very few very large IP ranges, whereas Azure owns very many rather small IP ranges, and just a few very large ones:

Another view is the filterable table for this data:

AWS CIDR masks

Azure CIDR masks

CloudFlare CIDR masks

DigitalOcean CIDR masks

Fastly CIDR masks

Google Cloud CIDR masks

Oracle Cloud CIDR masks

Conclusion

In this article, we described a simple and straightforward way to gather and transform data in different formats with DuckDB, as well as export it to a common schema as CSV and Parquet files.

Furthermore, we leveraged DuckDB on Observable to analyze and display the data in beautiful and interactive graphs.

By using GitHub Actions free tier as our "runtime" for the data processing via Bash and SQL scripts, and hosting our data in a GitHub repo (also covered by the free tier), we were able to show that data pipelines like covered use case can be built without accruing infrastructure costs. Also, the Observable pricing model supports our analyses for free.

For enterprises, this often means using SaaS solutions like Snowflake, Dremio, DataBricks or the like. Or, go all-in on the public cloud provider offerings from AWS, Azure and Google Cloud. But what if, as recent studies show, the data sizes aren't as big as commonly thought? Is it really necessary to spend so much money on usage and infrastructure?

In this blog post, we'll walk you through the steps to create a scalable, cost-effective data lake on AWS. Whether you're a startup, a small business, or a large enterprise, this guide will help you unlock the power of big data without breaking the bank (also see the excellent "Big data is dead" blog post by Jordan Tigani).

Modern Data Lake basics

The definition of what a Data Lake is, is probably slightly different depending on whom you're asking (see AWS, Google Cloud, Azure, DataBricks, IBM or Wikipedia). What is common to all these definitions and explanations is that it consists of different layers, such as ingestion, storage, processing and consumption. There can be several other layers as well, like cataloging and search, as well as a security and governance layer.

This is outlined in the excellent AWS article "AWS serverless data analytics pipeline reference architecture", which shall be the basis for this blog post:

Separation of storage & compute

Modern data lakes have revolutionized the way organizations handle big data. A data lake is a central repository that allows organizations to store all types of data, both structured and unstructured, at any scale. The flexibility and scalability of data lakes enable organizations to perform advanced analytics and gain insights that can drive business decisions. One of the key architectural patterns that modern data lakes follow is the separation of storage and compute.

Traditionally, data storage and processing were tightly coupled in data warehouses. However, in modern data lakes, data is stored in a separate layer from the computational layer that processes it. Data storage is handled by a data storage layer, while data processing is done by a compute layer. This approach allows organizations to scale storage and compute independently, enabling them to process vast amounts of data without incurring significant costs.

This has several advantages, which include:

1. Scalability: It allows organizations to scale each layer independently. The storage layer can be scaled up or down depending on the amount of data being stored, while the compute layer can be scaled up or down depending on the processing requirements.

2. Cost Savings: Decoupling storage and compute can significantly reduce costs. In traditional data warehouses, organizations must provision sufficient storage and processing power to handle peak loads. This results in underutilized resources during periods of low demand, leading to the wastage of resources and increased costs. In modern data lakes, organizations can store data cheaply and only provision the necessary compute resources when required, leading to significant cost savings.

3. Flexibility: Organizations can use a range of storage options, including object storage, file storage, and block storage, to store their data. This flexibility allows organizations to choose the most appropriate storage option for their data, depending on factors such as cost, performance, and durability.

4. Performance: In traditional data warehouses, data is moved from storage to processing, which can be slow and time-consuming, leading to performance issues. In modern data lakes, data is stored in a central repository, and processing is done where the data resides. This approach eliminates the need for data movement, leading to faster processing and improved performance.

Optimized file formats

As an example, Parquet is an open-source columnar storage format for data lakes that is widely used in modern data lakes. Parquet stores data in columns rather than rows, which enables it to perform selective queries faster and more efficiently than traditional row-based storage formats.

Additionally, Parquet supports compression, which reduces storage requirements and improves data processing performance. It's supported by many big data processing engines, including Apache Hadoop, Apache Spark, Apache Drill and many services of public cloud providers, such as Amazon Athena and AWS Glue.

Hive partitioning & query filter pushdown

The so-called "Hive partitioning" is a technique used in data lakes that involves dividing data into smaller, more manageable parts, called partitions, based on specific criteria such as date, time, or location.

Partitioning can help improve query performance and reduce data processing time by allowing users to select only the relevant partitions, rather than scanning the entire dataset.

Query filter pushdown is another optimization technique used in Apache Hive and other services that involves pushing down query filters into the storage layer, allowing it to eliminate irrelevant data before processing the query.

Combining Hive partitioning and query filter pushdown can result in significant performance gains in data processing, as the query filters can eliminate large amounts of irrelevant data at the partition level, reducing the amount of data that needs to be processed. Therefore, Hive partitioning and query filter pushdown are essential techniques for optimizing data processing performance in data lakes.

Repartitioning of data

Repartitioning Parquet data in data lakes is a useful technique that involves redistributing data across partitions based on specific criteria. This technique can help optimize query performance and reduce data shuffling during big data processing.

For instance, if a large amount of data is stored in a single partition, querying that data may take longer than if the data were spread across several partitions. Or, you could write aggregation queries whose output contains much less data, which could improve query speeds significantly.

The use case

Data privacy and GDPR are pretty talked-about topics in recent years. A lot of existing web tracking solutions were deemed as non-compliant, especially in the EU. Thus, individuals and companies had to eventually change their Web Analytics providers, which lead to a rise of new, data privacy-focussing companies in this space (e.g. Fathom Analytics, SimpleAnalytics, and Plausible just to name a few).

The pricing of those providers can get relatively steep quite fast if you have a higher amount of pageviews ($74/mo for 2m at Fathom, 99/mo for 1m at SimpleAnalytics, 89/mo for 2m at Plausible). Also, if you're using a provider, you're normally not owning your data.

So, let's try to build a web tracking and analytics service on AWS for the cheap, while owning our data, adhering to data privacy laws and using scalable serverless cloud services to avoid having to manage infrastructure by ourselves. And have some fun and learn a bit while doing it :-)

High-level architecture

The overall architecture for the outlined use case looks like this:

The details will be described further for each layer in the coming paragraphs. For brevity, the focus lies on the main data processing layers. Other layers, such as cataloging, consumption and security & governance, are eventually handled in other upcoming blog posts.

Serving layer

The serving layer is not a part of the data lake. Its main goal is to serve static assets, such as the tracking JavaScript libraries (those will be covered in more detail in another part of this series), and the 1x1 pixel GIF files that are used as endpoints that the tracking library can push its gathered data to. This is done by sending the JSON payload as URL-encoded query strings.

In our use case, we want to leverage existing AWS services and optimize our costs, while providing great response times. From an architectural perspective, there are many ways we could set up this data-gathering endpoint. Amazon CloudFront is a CDN that has currently over 90 edge locations worldwide, thus providing great latencies compared to classical webservers or APIs that are deployed in one or more regions.

It also has a very generous free tier (1TB outgoing traffic, and 10M requests), and with its real-time logs feature a great and very cost-effective way ($0.01 for every 1M log lines) to set up such an endpoint by just storing a 1x1px GIF with appropriate caching headers, to which the JavaScript tracking library will send its payload to as an encoded query string.

CloudFront can use S3 as a so-called origin (where the assets will be loaded from if they aren't yet in the edge caches), and that's where the static asset data will be located. Between the CloudFront distribution and the S3 bucket, an Origin Access Identity will be created, which enables secure communication between both services and avoids that the S3 bucket needs to be publicly accessible.

To configure CloudFront real-time logs that contain the necessary information, a RealtimeLogConfig needs to be created. This acts as "glue" between the CloudFront distribution and the Kinesis Data Stream that consumes the logs:

CFRealtimeLogsConfig:  Type: AWS::CloudFront::RealtimeLogConfig  Properties:     EndPoints:       - StreamType: Kinesis        KinesisStreamConfig:          RoleArn: !GetAtt 'AnalyticsKinesisDataRole.Arn'          StreamArn: !GetAtt 'AnalyticsKinesisStream.Arn'    Fields:       - timestamp      - c-ip      - sc-status      - cs-uri-stem      - cs-bytes      - x-edge-location      - time-taken      - cs-user-agent      - cs-referer      - cs-uri-query      - x-edge-result-type      - asn    Name: '${self:service}-cdn-realtime-log-config'    # IMPORTANT: This setting make sure we receive all the log lines, otherwise it's just sampled!    SamplingRate: 100

Ingestion layer

The ingestion layer mainly consists of two services: A Kinesis Data Stream, which is the consumer of the real-time logs feature of CloudFront, and a Kinesis Data Firehose Delivery Stream, which will back up the raw data in S3, and also store the data as partitioned parquet files in another S3 bucket. Both S3 buckets are part of the storage layer.

The Kinesis Data Stream (one shard in provisioned mode) provides an ingest capacity of 1 MB/second or 1,000 records/second, for a price of $0.015/hour in us-east-1, and $0.014 per 1M PUT payload units. It forwards the incoming data to the Kinesis Data Firehose Delivery Stream, whose pricing is more complex. The ingestion costs $0.029/GB, the format conversion $0.018/GB, and the dynamic partitioning $0.02/GB. That sums up to $0.067/GB ingested and written to S3, plus the S3 costs of $0.005/1k PUT object calls.

The Kinesis Data Firehose Delivery Stream uses data transformation and dynamic partitioning with a Lambda function, which cleans, transforms and enriches the data so that it can be stored in S3 as parquet files with appropriate Hive partitions.

The Delivery Stream has so-called BufferingHints, which either define from which size (from 1 to 128MB) or in which interval (between 60 to 900 seconds) the data is flushed to S3. The interval defines the minimum latency at which the data gets persisted in the data lake. The Lambda function is part of the processing layer and is discussed below.

The CloudFormation resource definition for the Kinesis Data Firehose Delivery Stream can be found below. It sources its variables from the serverless.yml:

AnalyticsKinesisFirehose:  Type: 'AWS::KinesisFirehose::DeliveryStream'  Properties:    DeliveryStreamName: ${self:custom.kinesis.delivery.name}    DeliveryStreamType: KinesisStreamAsSource    # Source configuration    KinesisStreamSourceConfiguration:      KinesisStreamARN: !GetAtt 'AnalyticsKinesisStream.Arn'      RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'    # Necessary configuration to transfrom and write data to S3 as parquet files    ExtendedS3DestinationConfiguration:      BucketARN: !GetAtt 'CleanedBucket.Arn'      BufferingHints:        IntervalInSeconds: ${self:custom.kinesis.delivery.limits.intervalInSeconds}        SizeInMBs: ${self:custom.kinesis.delivery.limits.sizeInMB}      # This enables logging to CloudWatch for better debugging possibilities      CloudWatchLoggingOptions:        Enabled: True        LogGroupName: ${self:custom.logs.groupName}        LogStreamName: ${self:custom.logs.streamName}      DataFormatConversionConfiguration:        Enabled: True        # Define the input format        InputFormatConfiguration:           Deserializer:             OpenXJsonSerDe:               CaseInsensitive: True        # Define the output format        OutputFormatConfiguration:           Serializer:             ParquetSerDe:               Compression: SNAPPY              WriterVersion: V1        # The schema configuration based on Glue tables        SchemaConfiguration:           RoleArn: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'          DatabaseName: '${self:custom.glue.database}'          TableName: 'incoming_events'      # Enable dynamic partitioning      DynamicPartitioningConfiguration:          Enabled: True      # Enable Lambda function for pre-processing the Kinesis records      ProcessingConfiguration:        Enabled: True        Processors:           - Type: Lambda            Parameters:               - ParameterName: NumberOfRetries                ParameterValue: 3              - ParameterName: BufferIntervalInSeconds                ParameterValue: 60              - ParameterName: BufferSizeInMBs                ParameterValue: 3              - ParameterName: LambdaArn                ParameterValue: !GetAtt 'ProcessKinesisRecordsLambdaFunction.Arn'      # Enable backups for the raw incoming data      S3BackupMode: Enabled      S3BackupConfiguration:        BucketARN: !GetAtt 'RawBucket.Arn'        BufferingHints:          IntervalInSeconds: ${self:custom.kinesis.delivery.limits.intervalInSeconds}          SizeInMBs: ${self:custom.kinesis.delivery.limits.sizeInMB}        # Disable logging to CloudWatch for raw data        CloudWatchLoggingOptions:          Enabled: false        CompressionFormat: GZIP        Prefix: '${self:custom.prefixes.raw}'        ErrorOutputPrefix: '${self:custom.prefixes.error}'        RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'      RoleARN: !GetAtt 'AnalyticsKinesisFirehoseRole.Arn'      # Define output S3 prefixes      Prefix: '${self:custom.prefixes.incoming}/domain_name=!{partitionKeyFromLambda:domain_name}/event_type=!{partitionKeyFromLambda:event_type}/event_date=!{partitionKeyFromLambda:event_date}/'      ErrorOutputPrefix: '${self:custom.prefixes.error}'

Processing layer

The processing layer consists of two parts, the Lambda function that is used for the dynamic partitioning of the incoming data, and a Lambda function that uses the COPY TO PARTITION BY feature of DuckDB to aggregate and repartition the ingested, enriched and stored page views data.

Data transformation & Dynamic partitioning Lambda

Data transformation is a Kinesis Data Firehose Delivery Stream feature that enables the cleaning, transformation and enrichment of incoming records in a batched manner. In combination with the dynamic partitioning feature, this provides powerful data handling capabilities with the data still being "on stream". When writing data to S3 as parquet files, a schema configuration in the form of a Glue Table needs to be defined as well to make it work (see "Cataloging & search layer" below).

It's necessary to define some buffer configuration for the Lambda function, meaning that you need to specify the time interval of 60 seconds (this will add a max delay of one minute to the stream data), the size in MB (between 0.2 and 3), and the number of retries (3 is the only usable default).

The input coming from the Kinesis Data Firehose Delivery Stream are a base64 encoded strings that contain the loglines coming from the CloudFront distribution:

MTY4MjA4NDI0MS40NjlcdDIwMDM6ZTE6YmYxZjo3YzAwOjhlYjoxOGY4OmExZmI6OWRhZFx0MzA0XHQvaGVsbG8uZ2lmP3Q9cHYmdHM9MTY4MjA4MzgwNDc2OCZ1PWh0dHBzJTI1M0ElMjUyRiUyNTJGbXlkb21haW4udGxkJTI1MkYmaG49bXlkb21haW4udGxkJnBhPSUyNTJGJnVhPU1vemlsbGElMjUyRjUuMCUyNTIwKE1hY2ludG9zaCUyNTNCJTI1MjBJbnRlbCUyNTIwTWFjJTI1MjBPUyUyNTIwWCUyNTIwMTBfMTVfNyklMjUyMEFwcGxlV2ViS2l0JTI1MkY1MzcuMzYlMjUyMChLSFRNTCUyNTJDJTI1MjBsaWtlJTI1MjBHZWNrbyklMjUyMENocm9tZSUyNTJGMTEyLjAuMC4wJTI1MjBTYWZhcmklMjUyRjUzNy4zNiZpdz0xMjkyJmloPTkyNiZ0aT1NeSUyNTIwRG9tYWluJnc9MzQ0MCZoPTE0NDAmZD0yNCZsPWRlLURFJnA9TWFjSW50ZWwmbT04JmM9OCZ0ej1FdXJvcGUlMjUyRkJlcmxpblx0Nzg5XHRIQU01MC1QMlx0MC4wMDFcdE1vemlsbGEvNS4wJTIwKE1hY2ludG9zaDslMjBJbnRlbCUyME1hYyUyME9TJTIwWCUyMDEwXzE1XzcpJTIwQXBwbGVXZWJLaXQvNTM3LjM2JTIwKEtIVE1MLCUyMGxpa2UlMjBHZWNrbyklMjBDaHJvbWUvMTEyLjAuMC4wJTIwU2FmYXJpLzUzNy4zNlx0LVx0dD1wdiZ0cz0xNjgyMDgzODA0NzY4JnU9aHR0cHMlMjUzQSUyNTJGJTI1MkZteWRvbWFpbi50bGQlMjUyRiZobj1teWRvbWFpbi50bGQmcGE9JTI1MkYmdWE9TW96aWxsYSUyNTJGNS4wJTI1MjAoTWFjaW50b3NoJTI1M0IlMjUyMEludGVsJTI1MjBNYWMlMjUyME9TJTI1MjBYJTI1MjAxMF8xNV83KSUyNTIwQXBwbGVXZWJLaXQlMjUyRjUzNy4zNiUyNTIwKEtIVE1MJTI1MkMlMjUyMGxpa2UlMjUyMEdlY2tvKSUyNTIwQ2hyb21lJTI1MkYxMTIuMC4wLjAlMjUyMFNhZmFyaSUyNTJGNTM3LjM2Jml3PTEyOTImaWg9OTI2JnRpPU15JTI1MjBEb21haW4mdz0zNDQwJmg9MTQ0MCZkPTI0Jmw9ZGUtREUmcD1NYWNJbnRlbCZtPTgmYz04JnR6PUV1cm9wZSUyNTJGQmVybGluXHRIaXRcdDMzMjBcbg==

After decoding, the logline is visible and contains the info from the real-time log fields, which are tab-separated and contain newlines:

1682084241.469\t2003:e1:bf1f:7c00:8eb:18f8:a1fb:9dad\t304\t/hello.gif?t=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\t789\tHAM50-P2\t0.001\tMozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/112.0.0.0%20Safari/537.36\t-\tt=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\tHit\t3320\n

During transformation and enrichment, the following steps are followed:

• Validating the source record

• Enriching the browser and device data from the user agent string

• Determine whether the record was generated by a bot (by user agent string)

• Add nearest geographical information based on edge locations

• Compute referer

• Derive requested URI

• Compute UTM data

• Get the event type (either a page view or a tracking event)

• Build the time hierarchy (year, month, day, event timestamp)

• Compute data arrival delays (data/process metrics)

• Generate hashes for page view, daily page view and daily visitor ids (later used to calculate page views and visits)

• Add metadata with the partition key values (in our case, the partition keys are domain_name, event_date, and event_type), to be able to use the dynamic partitioning feature

The generated JSON looks like this:

{  "result": "Ok",  "error": null,  "data": {    "event_year": 2023,    "event_month": 4,    "event_day": 21,    "event_timestamp": "2023-04-21T13:30:04.768Z",    "arrival_timestamp": "2023-04-21T13:37:21.000Z",    "arrival_delay_ms": -436232,    "edge_city": "Hamburg",    "edge_state": null,    "edge_country": "Germany",    "edge_country_code": "DE",    "edge_latitude": 53.630401611328,    "edge_longitude": 9.9882297515869,    "edge_id": "HAM",    "referer": null,    "referer_domain_name": "Direct / None",    "browser_name": "Chrome",    "browser_version": "112.0.0.0",    "browser_os_name": "Mac OS",    "browser_os_version": "10.15.7",    "browser_timezone": "Europe/Berlin",    "browser_language": "de-DE",    "device_type": "Desktop",    "device_vendor": "Apple",    "device_outer_resolution": "3440x1440",    "device_inner_resolution": "1292x926",    "device_color_depth": 24,    "device_platform": "MacIntel",    "device_memory": 8,    "device_cores": 8,    "utm_source": null,    "utm_campaign": null,    "utm_medium": null,    "utm_content": null,    "utm_term": null,    "request_url": "https://mydomain.tld/",    "request_path": "/",    "request_query_string": "t=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin\t789\tHAM50-P2\t0.001\tMozilla/5.0%20(Macintosh;%20Intel%20Mac%20OS%20X%2010_15_7)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/112.0.0.0%20Safari/537.36\t-\tt=pv&ts=1682083804768&u=https%253A%252F%252Fmydomain.tld%252F&hn=mydomain.tld&pa=%252F&ua=Mozilla%252F5.0%2520(Macintosh%253B%2520Intel%2520Mac%2520OS%2520X%252010_15_7)%2520AppleWebKit%252F537.36%2520(KHTML%252C%2520like%2520Gecko)%2520Chrome%252F112.0.0.0%2520Safari%252F537.36&iw=1292&ih=926&ti=My%2520Domain&w=3440&h=1440&d=24&l=de-DE&p=MacIntel&m=8&c=8&tz=Europe%252FBerlin",    "request_bytes": 789,    "request_status_code": 304,    "request_cache_status": "Hit",    "request_delivery_time_ms": 1,    "request_asn": 3320,    "request_is_bot": 0,    "event_name": null,    "event_data": null,    "page_view_id": "f4e1939bc259131659b00cd5f73e55a5bed04fbfa63f095b561fd87009d0a228",    "daily_page_view_id": "7c82d13036aa2cfe04720e0388bb8645eb90de084bd50cf69356fa8ec9d8b407",    "daily_visitor_id": "9f0ac3a2560cfa6d5c3494e1891d284225e15f088414390a40fece320021a658",    "domain_name": "mydomain.tld",    "event_date": "2023-04-21",    "event_type": "pageview"  },  "metadata": {    "partitionKeys": {      "domain_name": "mydomain.tld",      "event_date": "2023-04-21",      "event_type": "pageview"    }  }}

Then, the following steps are done by the Lambda function:

• Encode the JSON stringified records in base64 again

• Return them to the Kinesis Data Firehose Delivery Stream, which will then persist the data based on the defined prefix in the S3 bucket for incoming data.

Aggregation Lambda

As the ingested data contains information on a single request level, it makes sense to aggregate the data so that queries can be run optimally, and query response times are reduced.

The aggregation Lambda function is based on tobilg/serverless-parquet-repartitioner, which also has an accompanying blog post that explains in more detail how the DuckDB Lambda Layer can be used to repartition or aggregate existing data in S3.

The Lambda function is scheduled to run each night at 00:30AM, which makes sure that all the Kinesis Firehose Delivery Stream output files of the last day have been written to S3 (this is because the maximum buffer time is 15 minutes).

When it runs, it does three things:

• Create a session aggregation, that derives the session information and whether single requests were bounces or not

• Calculate the pageviews and visitor numbers, broken down by several dimensions which are later needed for querying (see stats table below)

• Store the extraction of the event data separately, newly partitioned by event_name to speed up queries

The queries can be inspected in the accompanying repository to get an idea about the sophisticated query patterns DuckDB supports.

Storage layer

The storage layer consists of three S3 buckets, where each conforms to a zone outlined in the reference architecture diagram (see above):

• A raw bucket, where the raw incoming data to the Kinesis Firehose Delivery Stream is backed up to (partitioned by event_date)

• A cleaned bucket, where the data is stored by the Kinesis Firehose Delivery Stream (partitioned by domain_name, event_date and event_type)

• A curated bucket, where the aggregated pageviews and visitors data are stored (partitioned by domain_name and event_date), as well as the aggregated and filtered events (partitioned by domain_name, event_date and event_name)

Cataloging & search layer

The Kinesis Data Firehose Delivery Stream needs a Glue table that holds the schema of the parquet files to be able to produce them (incoming_events table). The stats and the events tables are aggregated daily from the base incoming_events table via cron jobs scheduled by Amazon EventBridge Rules at 00:30 AM.

incoming_events table

This table stores the events that are the result of the data transformation and dynamic partitioning Lambda function. The schema for the table incoming_events looks like this:

stats table

The pageviews and visitor aggregation table. Its schema looks like this:

events table

The schema for the table events looks like this:

Consumption layer

The consumption layer will be part of another blog post in this series. Stay tuned! Until it's released, you can have a look at tobilg/serverless-duckdb to get an idea of how the data could potentially be queried in a serverless manner.

Wrapping up

In this article, you learned some basic principles of modern data lakes in the introduction. After that, it described how to build a serverless, near-realtime data pipeline leveraging AWS services and DuckDB on these principles, by the example of a web analytics application.

The example implementation of this article can be found on GitHub at ownstats/ownstats. Feel free to open an issue in case something doesn't work as expected, or if you'd like to add a feature request.

The next posts in this series will be

• Part II: Building a lightweight JavaScript library for the gathering of web analytics data

• Part III: Consuming the gathered web analytics data by building a serverless query layer

• Part IV: Building a frontend for web analytics data

Why not use existing AWS services?

If your data lake lives in AWS, a natural choice for ETL pipelines would be existing AWS services such as Amazon Athena. Unfortunately, Athena has pretty tight limits on the number of partitions that can be written by a single query (only up to 100). You, therefore, would need to create your workaround logic to be able to adhere to the limits, while still being able to do the operations you want. Additionally, Athena in some cases takes several 100ms to even start a query.

This is where DuckDB comes into play because it (theoretically) supports an unlimited amount of Hive partitions, and offers very fast queries on partitioned parquet files.

Use case

A common pattern to ingest streaming data and store it in S3 is to use Kinesis Data Firehose Delivery Streams, which can write the incoming stream data as batched parquet files to S3. You can use custom S3 prefixes with it when using Lambda processing functions, but by default, you can only partition the data by the timestamp (the timestamp the event reached the Kinesis Data Stream, not the event timestamp!).

So, a few common use cases for data repartitioning could include:

• Repartitioning the written data for the real event timestamp if it's included in the incoming data

• Repartitioning the data for other query patterns, e.g. to support query filter pushdown and optimize query speeds and costs

• Aggregation of raw or preprocessed data, and storing them in an optimized manner to support analytical queries

We want to be able to achieve this without having to manage our own infrastructure and built-upon services, which is the reason we want to be as serverless as possible.

Solution

As described before, Amazon Athena only has a partition limit of 100 partitions when writing data. DuckDB doesn't have this limitation, that's why we want to be able to use it for repartitioning.

This requires that we can deploy DuckDB in a serverless manner. The choice is to run it in Lambda functions, which can be provisioned with up to 10GB of memory (meaning 6 vCPUs), and a maximum runtime of 900 seconds / 15 minutes. This should be enough for most repartitioning needs, because the throughput from/to S3 is pretty fast. Also, we want to be able to run our repartition queries on flexible schedules, that's why we'll use EventBridge Rules with a schedule.

The project can be found at https://github.com/tobilg/serverless-parquet-repartitioner, and just needs to be configured and deployed.

Architecture overview

Configuration

Mandatory configuration settings

• S3 bucket name: You need to specify the S3 bucket where the data that you want to repartition resides (e.g. my-source-bucket)

• Custom repartitioning query: You can write flexible repartitioning queries in the DuckDB syntax. Have a look at the examples in the httpfs extension docs. You need to update this, as the template uses only example values!

Optional configuration settings

• S3 region: The AWS region your S3 bucket is deployed to (if different from the region the Lambda function is deployed to)

• The schedule: The actual schedule on why the Lambda function is run. Have a look at the Serverless Framework docs to find out what the potential settings are.

• DuckDB memory limit: The memory limit is influenced by the function memory setting (automatically)

• DuckDB threads count: Optionally set the max thread limit (on Lambda, this is set automatically by the amount of memory the functions has assigned), but with this setting, you can influence how many files are written per partition. If you set a lower thread count than available, this means that the computation will not use all available resources for the sake of being able to set the number of generated files! Ideally, rather align the amount of memory you assign to the Lambda function.

• Lambda timeout: The maximum time a Lambda function can run is currently 15min / 900sec. This means that if your query takes longer than that, it will be terminated by the underlying Firecracker engine.

Using different source/target S3 buckets

If you're planning to use different S3 buckets as sources and targets for the data repartitioning, you need to adapt the iamRoleStatements settings of the function.

Here's an example with minimal privileges:

iamRoleStatements:  # Source S3 bucket permissions  - Effect: Allow    Action:      - s3:ListBucket    Resource: 'arn:aws:s3:::my-source-bucket'  - Effect: Allow    Action:      - s3:GetObject    Resource: 'arn:aws:s3:::my-source-bucket/*'  # Target S3 bucket permissions  - Effect: Allow    Action:      - s3:ListBucket      - s3:AbortMultipartUpload      - s3:ListMultipartUploadParts      - s3:ListBucketMultipartUploads    Resource: 'arn:aws:s3:::my-target-bucket'  - Effect: Allow    Action:      - s3:GetObject      - s3:PutObject    Resource: 'arn:aws:s3:::my-target-bucket/*'

A query for this use case would look like this:

COPY (SELECT * FROM parquet_scan('s3://my-source-bucket/input/*.parquet', HIVE_PARTITIONING = 1)) TO 's3://my-starget-bucket/output' (FORMAT PARQUET, PARTITION_BY (column1, column2, column3), ALLOW_OVERWRITE TRUE);

Deployment

After you cloned this repository to your local machine and cd'ed in its directory, the application can be deployed like this (don't forget a npm i to install the dependencies!):

$ sls deploy

This will deploy the stack to the default AWS region us-east-1. In case you want to deploy the stack to a different region, you can specify a --region argument:

$ sls deploy --region eu-central-1

The deployment should take 2-3 minutes.

Checks and manual triggering

You can manually invoke the deployed Lambda function by running

$ sls invoke -f repartitionData

After that, you can check the generated CloudWatch logs by issuing

$ sls logs -f repartitionData

If you don't see any DUCKDB_NODEJS_ERROR in the logs, everything ran successfully, and you can have a look at your S3 bucket for the newly generated parquet files.

Costs

Using this repository will generate costs in your AWS account. Please refer to the AWS pricing docs for the respective services before deploying and running it.

Summary

We were able to show a possible serverless solution to repartition data that is stored in S3 as parquet files, without limitations imposed by certain AWS services. With the solution shown, we can use plain and simple SQL queries, instead of having to eventually use external libraries etc.

References

• Serverless parquet repartitioner repo: https://github.com/tobilg/serverless-parquet-repartitioner

DuckDB is an open-source in-process SQL OLAP database management system that has recently gained significant public interest due to its unique architecture and impressive performance benchmarks.

Unlike traditional databases that are designed to handle a wide variety of use cases, DuckDB is built specifically for analytical queries and is optimized to perform extremely well in these scenarios. This focus on analytics has allowed DuckDB to outperform traditional databases by several orders of magnitude, making it a popular choice for data scientists and analysts who need to process large datasets quickly and efficiently.

As DuckDB is designed to be a highly efficient and scalable database system, which makes it a perfect fit for serverless architectures that allow developers to build and run applications and services without having to manage infrastructure.

DuckDB's ability to handle large datasets in a memory-efficient manner makes it an ideal choice for serverless environments. Being able to read columnar storage formats like Parquet or Apache Arrow tables from local, S3 or HTTP sources, DuckDB can quickly scan and aggregate large amounts of data without having to load it all into memory, reducing the amount of memory required to perform complex analytical queries. This allows for cost savings, as serverless environments typically charge for both compute and memory resources.

Existing AWS services, such as Athena or RDS, don't provide the same functionalities, and also have different scaling and pricing models. That's why it makes sense to explore ways to run DuckDB as an analytical service on AWS.

How to run DuckDB in AWS Lambda?

The goal of this article is to use DuckDB on Node.js runtimes (12, 14, 16 and 18), thus it is necessary to find a way to make DuckDB usable with Lambda. The first idea was to simply use the existing DuckDB npm package and use the default packaging mechanisms when deploying Lambda functions. Unfortunately, this idea proved impossible due to downstream package problems and different build Operating Systems (Lambda needs statically linked binaries built on Amazon Linux).

Generally, there are several ways to package dependencies in AWS Lambda functions with Node.js runtimes. Using bundlers like WebPack or ESbuild is probably the most used option at the moment.

Another one is using AWS Lambda layers for distributing dependencies to Lambda functions, allowing developers to manage common components and libraries across multiple functions in a centralized manner. By creating a layer for the dependencies, developers can avoid having to include them in each functions deployment package. This helps reduce the size of the deployment package and makes it easier to manage updates to the dependencies. Moreover, using a Lambda layer can also improve the performance of Lambda functions.

Building DuckDB for AWS Lambda

So, how can we achieve to build a DuckDB version that can be used with Node.js runtimes on AWS Lambda?

• We have to use a compatible environment when compiling the DuckDB binary, to avoid GLIBC incompatibilities etc. This means that we have to use an Amazon Linux distribution to build DuckDB, and enable static linking.

• We have to solve the downstream package problems stated above, which make it impossible to use the default package.

• On a side note, we want to enable the current features like COPY TO PARTITIONED BY and improved Parquet file reading, thus requiring a build from the master branch of DuckDB.

I created https://github.com/tobilg/duckdb-nodejs-layer to achieve this. It uses GitHub Actions to automatically trigger a build of DuckDB from the current master, package it as AWS Lambda layer and automatically upload it to all AWS regions. Feel free to have a look at the source code, and open a GitHub issue in case you find some errors or ideas for improvement.

Using the DuckDB Lambda layer

Depending on your preferred framework, the methods to use a Lambda layer are different. You can find the respective docs of the most common frameworks below:

• Serverless Framework

• SAM

• CDK

• CloudFormation

The ARNs follow the following logic:

arn:aws:lambda:$REGION:041475135427:layer:duckdb-nodejs-layer:$VERSION

You can find the list of ARNs for all regions at https://github.com/tobilg/duckdb-nodejs-layer#usage.

I created an example repository with the Serverless Framework at https://github.com/tobilg/serverless-duckdb that uses API Gateway and Lambda to provide an endpoint to which SQL queries can be issued, which the built-in DuckDB then executes. Let's walk through it!

Requirements

You'll need a current v3 version installation of the Serverless Framework on the machine you're planning to deploy the application from.

Also, you'll have to set up your AWS credentials according to the Serverless docs.

Configuration

DuckDB is automatically configured to use the HTTPFS extension and uses the AWS credentials that are given to your Lambda function by its execution role. This means you can potentially query data that is available via HTTP(S) or in AWS S3 buckets.

If you want to also query data (e.g. Parquet files) that resides in one or more S3 buckets, you'll have to adjust the iamRoleStatements part of the function configuration in the serverless.yml file. Just replace the YOUR-S3-BUCKET-NAME with your actual S3 bucket name.

  query:    handler: src/functions/query.handler    memorySize: 10240    timeout: 30    iamRoleStatements:      - Effect: Allow        Action:          - s3:GetObject        Resource: 'arn:aws:s3:::YOUR-S3-BUCKET-NAME/*'      - Effect: Allow        Action:          - s3:ListBucket        Resource:          - 'arn:aws:s3:::YOUR-S3-BUCKET-NAME'    layers:      - 'arn:aws:lambda:${self:provider.region}:041475135427:layer:duckdb-nodejs-layer:3'    events:      - http:          path: ${self:custom.api.version}/query          method: post          cors: true          private: true

Deployment

After you cloned this repository to your local machine and cd'ed in its directory, the application can be deployed like this (don't forget a npm i to install the dependencies):

$ sls deploy

This will deploy the stack to the default AWS region us-east-1. In case you want to deploy the stack to a different region, you can specify a --region argument:

$ sls deploy --region eu-central-1

The deployment should take 2-3 minutes. Once the deployment is finished, you should find some output in your console that indicates the API Gateway endpoint URL and the API Key:

api keys:  DuckDBKey: REDACTEDendpoint: POST - https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query

Usage

You can now query your DuckDB endpoint via HTTP requests (don't forget to exchange REDACTED with your real URL and API Key), e.g.

curl --location --request POST 'https://REDACTED.execute-api.us-east-1.amazonaws.com/prd/v1/query' \--header 'x-api-key: REDACTED' \--header 'Content-Type: application/json' \--data-raw '{    "query": "SELECT avg(c_acctbal) FROM '\''https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet'\'';"}'

The query results will look to this:

[    {        "avg(c_acctbal)": 4454.577060000001    }]

Example queries

Remote Parquet Scans:  SELECT count(*) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/lineitem.parquet';  SELECT count(*) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet';  SELECT avg(c_acctbal) FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet';  SELECT * FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/orders.parquet' LIMIT 10;Remote Parquet/Parquet Join:  SELECT n_name, count(*)  FROM 'https://shell.duckdb.org/data/tpch/0_01/parquet/customer.parquet',       'https://shell.duckdb.org/data/tpch/0_01/parquet/nation.parquet'  WHERE c_nationkey = n_nationkey GROUP BY n_name;

Conclusion

We were able to show that it's possible to package and use DuckDB in a Lambda function, as well as to run performant queries on remote data with this setup.

But we need to keep in mind that this is just a showcase. The example application doesn't solve a lot of issues we'd have to solve if we'd want to run this in a distributed manner:

• A query planner and router, that scales DuckDB instances and redistributes the queries to the "query backend" functions, as well as unites the query results before passing them back to the query issuer

• "Query stickiness": The example is stateless, meaning that even if you'd load data into a memory table, you'd not be able to be sure that you'd reach the same function instance with a subsequent query due to Lambda's scaling/execution model

• Running DuckDB "only" in Lambda functions may not be the most performant way when AWS Fargate and very large EC2 instances exist

• The example application uses API Gateway as the event source for the Lambda function, which means the maximum runtime of the queries can be 30 seconds, which is unrealistic for large datasets or complicated queries. In real-world scenarios, the Lambda function would need to be triggered asynchronously, e.g. via SNS or SQS. This also means that the queries probably can't follow a strict request/response model.

References

• BoilingData

• STOIC

• Ismael Ghalimi's Twitter feed

Who needs a reverse proxy with on-demand SSL support? Well, think about services as Hashnode, which also runs this blog, or Fathom and SimpleAnalytics. A feature that all those services have in common? They all are enabling their customers to bring their own, custom domain names. The latter two services use them to bypass adblockers, with the intention that customers can track all their pageviews and events, which potentially wouldn't be possible otherwise because the service's own domain are prone to DNS block lists. Hashnode is using them to enable their customers to host their blogs under their own domain names.

Requirements

What are the functional & non-functional requirements to build such a system? Let's try to recap:

• We want to be able to use a custom ("external") domain, e.g. subdomain.customdomain.tld to redirect to another domain, such as targetdomain.tld via a CNAME DNS record

• The custom domains need to have SSL/TLS support

• The custom domains need to be configurable, without changing the underlying infrastructure

• We want to make sure that the service will only create and provide the certificates for whitelisted custom domains

• We want to optimize the latency of individual requests, thus need to support a scalable and distributed infrastructure

• We want to be as Serverless as possible

• We want to optimize for infrastructure costs (variable and fixed)

• We want to build this on AWS

• The service needs to be deployable/updateable/removable via Infrastructure as Code (IaC) in a repeatable manner

Architectural considerations

Looking at the above requirements, what are the implications from an architectural point of view? Which tools and services are already on the market? What does AWS as Public Cloud Provider offer for our use case?

For the main requirements of a reverse proxy server with automatic SSL/TLS support, Caddy seems to be an optimal candidate. As it is written in Go, it can run in Docker containers very well and can be used on numerous operating systems. This means we have the options to either run it on EC2 instances or ECS/Fargate if we decide to run it in containers. The latter would cater to the requirement to run as Serverless as possible. It has modules to store the generated SSL/TLS on-demand certificates in DynamoDB or S3.

Also, the whitelisting of custom domains is possible for those certificates, by providing an additional backend service, which Caddy can ask whether a requested custom domain is allowed to use or not.

A challenge is that none of those modules are contained in the official Caddy builds, meaning that we'd have to build a custom version of Caddy to be able to use those storage backends.

Regarding the requirement of global availability and short response times, AWS Global Accelerator is a viable option, as it can provide a global single static IP address endpoint for multiple, regionally distributed services. In our use case, those services would be our Caddy installations.

Running Caddy itself, as said before, is possible via Containers or EC2 instances / VMs. As the services will run the whole time and assumingly don't need a lot of resources if not under heavy load, we assume that 1 vCPU and 1 GB of memory should be enough.

When projecting this on the necessary infrastructure, the cost comparison between Containers and VMs looks like the following (for simplification, we just compare the fixed costs, ignore variable costs such as egress traffic, and assume the us-east-1 region is used):

Containers

• Fargate task with 1 vCPU and 1 GB of memory for each Caddy regional instance

  • Price: ($0.04048/vCPU hour + $0.004445/GB hour) * 720 hours (30 days) = $32.35 / 30 days

• ALB to make the Fargate tasks available to the outside world

  • Price: ($0.0225 per ALB-hour + $0.008 per LCU-hour) * 720 hours (30 days) = $21.96 / 30 days

In combination, it would cost $54.31 to run this setup for 30 days.

EC2 instances / VMs

• A t2.micro instance with 1 vCPU and 1 GB f memory for each Caddy regional instance

  • Price: $0.0116/hour on-demand * 720 hours (30 days) = $8.35

• There's no need for a Load Balancer in front of the EC2 instances, as Global Accelerator can directly use them

  • Price: $0

In combination, it would cost $8.35 to run this setup for 30 days.

Additional costs are:

• AWS Global Accelerator

  • Price: $0.025 / hour * 720 hours (30 days) = $18

• DynamoDB table

  • Price: (5000 reads/day * $0.25/million + 50 writes/day * $1.25/million) * 30 days = $0.0375 reads + 0.001875 writes = $0.04

• Lambda function (128 MB / 0.25 sec avg. duration / 5000 req./day)

  • Price: ($0.0000166667 / GB-second + $0.20 per 1M req.) = basically $0

Resulting architecture

Based on the calculated fixed costs, we decided to use EC2 instances instead of Fargate tasks, which will save us a decent amount of money even for one Caddy instance. The estimated costs for running this architecture for 30 days are $26.39.

As one of our requirements was that we can roll out this infrastructure potentially on a global scale, we need to be able to deploy the EC2 instances with the Caddy servers in different AWS regions, as well as having multiple instances in the same region.

Furthermore, we could use DynamoDB Global Tables to achieve a global distribution of the certificates to get faster response times, but deem it as out of scope for this article.

The final architecture:

Implementation

To implement the described architecture, we must take several steps. First of all, we must build a custom version of Caddy that includes the DynamoDB module, which then enables us to use DynamoDB as certificate store.

Custom Caddy build

This can be achieved via a custom build process leveraging Docker images of AmazonLinux 2, as found at tobilg/aws-caddy-build.

build.sh (parametrized custom build of Caddy via Docker)

#!/usr/bin/env bashset -e# Set OS (first script argument)OS=${1:-linux}# Set Caddy version (second script argument)CADDY_VERSION=${2:-v2.6.2}# Create release foldersmkdir -p $PWD/releases $PWD/temp_release# Run builddocker build --build-arg OS=$OS --build-arg CADDY_VERSION=$CADDY_VERSION -t custom-caddy-build .# Copy release from image to temporary folderdocker run -v $PWD/temp_release:/opt/mount --rm -ti custom-caddy-build bash -c "cp /tmp/caddy-build/* /opt/mount/"# Copy release to releasescp $PWD/temp_release/* $PWD/releases/# Cleanuprm -rf $PWD/temp_release

Dockerfile (will build Caddy with the DynamoDb and S3 modules)

FROM amazonlinux:2ARG CADDY_VERSION=v2.6.2ARG OS=linux# Install dependenciesRUN yum update -y && \  yum install golang -yRUN GOBIN=/usr/local/bin/ go install github.com/caddyserver/xcaddy/cmd/xcaddy@latestRUN mkdir -p /tmp/caddy-build && \  GOOS=${OS} xcaddy build ${CADDY_VERSION} --with github.com/ss098/certmagic-s3 --with github.com/silinternational/certmagic-storage-dynamodb/v3 --output /tmp/caddy-build/aws_caddy_${CADDY_VERSION}_${OS}

That's it for the custom Caddy build. You don't need to build this yourself, as the further steps use the release I built and uploaded to GitHub.

Reverse Proxy Service

The implementation of the reverse proxy service can be found at tobilg/global-reverse-proxy.

Just clone it via git clone https://github.com/tobilg/global-reverse-proxy.git to your local machine, and configure it as described below.

Prerequisites

Serverless Framework

You need to have a recent (>=3.1.2) version of the Serverless Framework installed globally on your machine. If you haven't, you can run npm i -g serverless to install it.

Valid AWS credentials

The Serverless Framework relies on already configured AWS credentials. Please refer to the docs to learn how to set them up on your local machine.

EC2 key already configured

If you want to interact with the deployed EC2 instance(s), you need to add your existing public SSH key or create a new one. Please have a look at the AWS docs to learn how you can do that.

Please also note the name you have given to the newly created key, as you will have to update the configuration of the proxy server(s) stack.

Infrastructure as Code overview

The infrastructure consists of three different stacks:

• A stack for the domain whitelisting service, and the certificate table in DynamoDB

• A stack for the proxy server(s) itself, which can be deployed multiple times if you want high (global) availability and fast latencies

• A stack for the Global Accelerator, and the according DNS records

Most important parts

The main functionality, the reverse proxy based on Caddy, is deployed via an EC2 instance. Its configuration, the so-called Caddyfile, is, together with the CloudFormation resource for the EC2 instance, the most important part.

Caddyfile

This configuration enables the reverse proxy, the on-demand TLS feature and DynamoDB storage for certificates. It's automatically parametrized via the generated /etc/caddy/environment file (see ec2.yml below). There's a systemctl service for Caddy generated, based on our configuration derived from the serverless.yml, as well.

{        admin off        on_demand_tls {                ask {$DOMAIN_SERVICE_ENDPOINT}        }        storage dynamodb {$TABLE_NAME} {                aws_region {$TABLE_REGION}        }}:80 {       respond /health "Im healthy" 200}:443 {        tls {$LETSENCRYPT_EMAIL_ADDRESS} {                on_demand        }        reverse_proxy https://{$TARGET_DOMAIN} {                header_up Host {$TARGET_DOMAIN}                header_up User-Custom-Domain {host}                header_up X-Forwarded-Port {server_port}                health_timeout 5s        }}

ec2.yml (extract)

The interesting part is the UserData script, which is run automatically when the EC2 instance starts. It does the following:

• Download the custom Caddy build with DynamoDB support

• Prepare a group and a user for Caddy

• Create the caddy.service file for systemctl

• Create the Caddyfile (as outlined above)

• Create the environment file (/etc/caddy/environment)

• Enable & reload the systemctl service

Resources:  EC2Instance:    Type: AWS::EC2::Instance    Properties:      InstanceType: '${self:custom.ec2.instanceType}'      KeyName: '${self:custom.ec2.keyName}'      SecurityGroups:         - !Ref 'InstanceSecurityGroup'      ImageId: 'ami-0b5eea76982371e91' # Amazon Linux 2 AMI      IamInstanceProfile: !Ref 'InstanceProfile'      UserData: !Base64         'Fn::Join':          - ''          - - |              #!/bin/bash -xe            - |              sudo wget -O /usr/bin/caddy "https://github.com/tobilg/aws-caddy-build/raw/main/releases/aws_caddy_v2.6.2_linux"            - |              sudo chmod +x /usr/bin/caddy            - |              sudo groupadd --system caddy            - |              sudo useradd --system --gid caddy --create-home --home-dir /var/lib/caddy --shell /usr/sbin/nologin --comment "Caddy web server" caddy            - |              sudo mkdir -p /etc/caddy            - |              sudo echo -e '${file(./configs.js):caddyService}' | sudo tee /etc/systemd/system/caddy.service            - |              sudo printf '${file(./configs.js):caddyFile}' | sudo tee /etc/caddy/Caddyfile            - |              sudo echo -e "TABLE_REGION=${self:custom.caddy.dynamoDBTableRegion}\nTABLE_NAME=${self:custom.caddy.dynamoDBTableName}\nDOMAIN_SERVICE_ENDPOINT=${self:custom.caddy.domainServiceEndpoint}\nLETSENCRYPT_EMAIL_ADDRESS=${self:custom.caddy.letsEncryptEmailAddress}\nTARGET_DOMAIN=${self:custom.caddy.targetDomainName}" | sudo tee /etc/caddy/environment            - |              sudo systemctl daemon-reload            - |              sudo systemctl enable caddy            - |              sudo systemctl start --now caddy

global-accelerator.yml

The Global Accelerator CloudFormation resource wires the EC2 instance(s) to its kind-of global load balancer. This is then referenced by the dns-record.yml, which assigns the configured domain name to the Global Accelerator.

Resources:  Accelerator:    Type: AWS::GlobalAccelerator::Accelerator    Properties:      Name: 'External-Accelerator'      Enabled: true  Listener:    Type: AWS::GlobalAccelerator::Listener    Properties:      AcceleratorArn:        Ref: Accelerator      Protocol: TCP      ClientAffinity: NONE      PortRanges:        - FromPort: 443          ToPort: 443  EndpointGroup1:    Type: AWS::GlobalAccelerator::EndpointGroup    Properties:       EndpointConfigurations:         - EndpointId: '${self:custom.ec2.instance1.id}'          Weight: 1      EndpointGroupRegion: '${self:custom.ec2.instance1.region}'      HealthCheckIntervalSeconds: 30      HealthCheckPath: '/health'      HealthCheckPort: 80      HealthCheckProtocol: 'HTTP'      ListenerArn: !Ref 'Listener'      ThresholdCount: 3

Detailed configuration

Stack configurations

Please configure the following values for the different stacks:

• The target domain name where you want your reverse proxy to send the requests to (targetDomainName)

• The email address to use for automatic certificate generation via LetsEncrypt (letsEncryptEmailAddress)

• The domain name of the proxy service itself, which is then used by GlobalAccelerator (domain)

• Optionally: The current IP address from which you want to use the EC2 instance(s) via SSH from (sshClientIPAddress). If you want to use SSH, you'll need to uncomment the respective SecurityGroup settings

Whitelisted domain configuration

You need to make sure that not everyone can use your reverse proxy with every domain. Therefore, you need to configure the whitelist of domains that you be used by Caddy's on-demand TLS feature.

This is done with the Domain Verifier Lambda function, which is deployed at a Function URL endpoint.

The configuration can be changed here before deploying the service.

HINT: To use this dynamically, as you'd probably wish in a production setting, you could rewrite the Lambda function to read the custom domains from a DynamoDB table, and have another Lambda function run recurrently to issue DNS checks for the CNAME entries the customers would need to make (see below).

DNS / Nameserver configurations

If you use an external domain provider, such as Namecheap or GoDaddy, make such that you point the DNS settings at your domain's configuration to those which are assigned to your HostedZone by Amazon. You can look these up in the AWS Console or via the AWS CLI.

CNAME configuration for proxying

You also need to add CNAME records to the domains you want to proxy for, e.g. if your proxy service domain is external.mygreatproxyservice.com, you need to add a CNAME record to your existing domain (e.g. test.myexistingdomain.com) to redirect to the proxy service domain:

CNAME test.myexistingdomain.com external.mygreatproxyservice.com

Passing options during deployment

When running sls deploy for each stack, you can specify the following options to customize the deployments:

• --stage: This will configure the so-called stage, which is part of the stack name (default: prd)

• --region: This will configure the AWS region where the stack is deployed (default: us-east-1)

Deployment

You need to follow a specific deployment order to be able to run the overall service:

1. Domain whitelisting service: cd domain-service-stack && sls deploy && cd ..

2. Proxy server(s): cd proxy-server-stack && sls deploy && cd ..

3. Global Accelerator & HostedZone / DNS : cd accelerator-stack && sls deploy && cd ..

Removal

To remove the individual stacks, you can run sls remove in the individual subfolders.

Wrapping up

We were able to build a POC for a (potentially) globally distributed reverse proxy service, with on-demand TLS support. We decided against using Fargate, and for using EC2 due to cost reasons. This prioritized costs higher, than running as Serverless as possible. It's possible that, in another setting / environment / experience, you might come to another conclusion, which is completely fine.

For a more production-like setup, you'd probably need to amend the Domain Verifier Lambda function, so that it dynamically looks up the custom domains that are configured e.g. by your customers via a UI, and stored in another DynamoDB table via another Lambda function. Deleting or updating those custom domains should probably be possible, too.

Furthermore, you should then write an additional Lambda function that recurrently checks each stored custom domain if:

• The CNAME records point to your external.$YOUR_DOMAIN_NAME.tld, and updates the status accordingly

• Performs a check via HTTPS whether an actual redirect from the custom domain to your domain is possible