loading
loading
DuckDB Ecosystem Newsletter : October 2026
· 8 min read
HEY, FRIEND 👋
I hope you're doing well. I'm Simon, and I am happy to share another monthly newsletter with highlights and the latest updates about DuckDB, delivered straight to your inbox.
In this October issue, I gathered the usual 10 updates, starting with Jev, the new "System One" model and how to use it with DuckDB and MotherDuck, the new PEG parser and faster S3 reads in DuckDB 2.0, a maintenance strategy for DuckLake, querying OneLake in the browser, and a native Mac app. Lots of great articles from the community as always.
Data Outpost is happening November 4-5 in SF, with speakers from DoorDash, Perplexity, Stripe, Canva, Vercel, and Fivetran + dbt Labs on agentic data infrastructure, the semantic layer, and data stacks after agents.
Mark Litwintschik
Mark is a Big Data, AI, GIS and networking consultant and the author of tech.marksblogg.com, which many of you will know from his 1.1 Billion Taxi Rides benchmarks. He started writing about DuckDB in March 2023 with Geospatial DuckDB, while looking for ways to speed up a data pipeline and cut down on engineering work. DuckDB has stayed in his toolbox ever since.
This month's links include his post on analysis-ready OpenStreetMap (see below). Thanks, Mark, for years of advocating for DuckDB in the geospatial world!
Connect with Mark on LinkedIn.
Introducing prompt_jev(): bringing Jev to DuckDB/MotherDuck SQL
TL;DR: Jev, TypeSafe AI's recently announced "System One" model, returns a typed answer (a yes/no probability, one label from a fixed list, or an ordinal score) instead of generated text and can now be called from SQL through the jev DuckDB community extension and MotherDuck's prompt_jev().
I combined different articles for this paragraph, in the first, Geertjan and Gábor walk through INSTALL jev FROM community; and the jev(), jev_prob(), jev_choice() and jev_score() functions. They note that a cheaper predicate such as age > 40 in the same WHERE clause runs first, so rows it rejects are never sent to the API. Then Hamilton, Till and Jacob report 89% accuracy on 100k AG News rows in 40 s for $0.50, against 83% in 17m49s for gpt-5-nano and 88% in about 32 minutes and $37.58 for GPT-5.6 Terra. Runs at 1M and 10M rows performed similarly, and the post includes SQL that reproduces the benchmark from hf://datasets/fancyzhx/ag_news/....
Mehdi classified 100k CFPB complaints in 82.2 s into a struct with request.choice and request.confidence fields. On high-confidence rows (≥ 0.8), Jev matched the label GPT-5 and Claude Opus agreed on 96.9% of the time. In Part 2, he swapped a remote-work regex (22 hand-tuned alternatives plus exclusion lists) for a single choice := question. On 2,000 postings it ran in 1.7 s, versus 37.0 s for prompt(). The regex was still faster at 0.2 s, but Jev only needs label definitions to maintain. There's also the video version with Hamilton.
DuckDB v2.0: Your Database Deserves a Better Parser
TL;DR: DuckDB v2.0 replaces its PostgreSQL-derived LALR parser with a PEG-based parser to simplify grammar evolution.
Daniël explains that shifting to a Parsing Expression Grammar (PEG) allows DuckDB to avoid traditional Bison shift/reduce conflicts by using ordered choices and packrat parsing. This memoization technique prevents exponential backtracking on malformed queries; for instance, parsing 19 unmatched parentheses improved from 10.6 seconds to 0.001 seconds.
The new architecture allows extensions to register custom grammar rules at runtime, such as BigQuery-style pipe syntax (|>), by hooking into extension points like select_atom_rule. This enables developers to reuse existing internal rules like GroupByClause or WhereClause within their own syntax transformations.
Kelsey Hightower using DuckDB
TL;DR: Even Kelsey Hightower (known for his work with Kubernetes & cloud computing) is using DuckDB 😉, and says:
I've been playing with DuckDB and I gotta say the docs are amazing. The built-in web shell is a really nice touch. You can play with DuckDB right inside your web browser. This might be my SQLite replacement.
DuckLake in Production: Maintenance Strategy
TL;DR: Hoyt details automated snapshot expiration and file cleanup routines to optimize storage and query performance.
Technical implementations focus on managing the DuckLake metadata catalog and S3 storage. DuckLake optimizes small writes by inlining data directly in the catalog, which may eventually require ducklake_flush_inlined_data to move rows to Parquet. After attaching the database, engineers should execute CALL ducklake_expire_snapshots(..., older_than => now() - INTERVAL '7 days') to enforce time-travel policies. This must be followed by ducklake_cleanup_old_files to purge physical data from S3 and ducklake_delete_orphaned_files to remove untracked Parquet files.
Analysis-Ready OpenStreetMap
TL;DR: Mark highlights how the Layercake project's spatially-sorted Parquet files enable high-performance OpenStreetMap analysis with DuckDB via metadata-based row-group skipping.
Mark demonstrates how DuckDB's Parquet reader efficiently processes the 39 GB buildings dataset. A count query with GROUP BY 2 on 657 million rows consumed only 2.2 MiB of bandwidth, showcasing projection pushdown and column-level statistics. The data is updated weekly and hosted on Cloudflare. He also details integrating these datasets into QGIS using the GeoParquet Downloader plugin, which utilizes an internal DuckDB environment to perform bounding box queries against remote files.
Falling Into Databases: The DuckDB Story with Hannes Mühleisen
TL;DR: Hannes explains how DuckDB grew from an accidental career at the CWI research lab into a from-scratch in-process analytical database.
The team first tried retrofitting MonetDB, but usability requirements leaked into engine design, so they restarted despite a roughly three-year setback. Adoption came from fixing concrete pain points: a robust CSV parser, Parquet reading without Hadoop, and SQL conveniences like trailing commas (SELECT a, b, FROM t). Another one is that operators spill to disk instead of crashing when intermediate results exceed memory. DuckLake later stored lake metadata in a database rather than Avro files. Quack is an HTTP-based protocol with DuckDB on both client and server, so queries can run locally, remotely, or split between them.
Serverless, in-browser SQL over your OneLake
TL;DR: The OneLake Studio project offers a SQL interface for querying OneLake data directly using DuckDB-WASM, eliminating the need for backend infrastructure or data copying.
The application is a static web app. It integrates with Azure Storage via delegated permissions (user_impersonation) to access data using an Entra ID token and queries data through OneLake's Iceberg REST catalog, processing tables via Parquet files. For efficient data retrieval, it utilizes HTTP range requests and implements a service worker (site/sw.js) for caching responses (up to 512MB).
Data files are registered as URLs, allowing DuckDB to fetch only required row groups and columns, with limitations including a 200,000-row cap for query results to prevent browser tab crashes. Other file formats like CSV, JSON, and SQLite are handled, with SQLite files being copied and converted into DuckDB tables. Web Demo
Bedevere-wise: In-browser SQL data viewer for SAS, SPSS, Stata, Parquet, Excel, and CSV files
TL;DR: bedevere-wise is a local-first viewer that utilizes DuckDB-WASM to query and visualize files like Parquet, SAS (statistical software packages), and SPSS without data leaving the device.
The technical implementation relies on a DuckDBService wrapper and a DataProvider interface to drive canvas-rendered grids and Vega-Lite charts. Integration is straightforward; developers can await duck.initialize() and then execute queries like CREATE OR REPLACE TABLE table_name AS SELECT * FROM read_parquet('file.parquet'). The project also features an EmbedSqlEditor with a PostgreSQL-extended dialect. Web app, Latest updates
DuckDB 2.0 and S3 File Read Performance
TL;DR: DuckDB 2.0 introduces native asynchronous I/O for file reads, delivering a ~67% performance improvement for querying large datasets on AWS S3.
Matt benchmarked the 2.0 Alpha against version 1.5.6, processing thousands of JSON files. By utilizing read_json_auto() combined with UNNEST, the test showed that the async engine reduces wait times on high-latency storage. For a workload of 10,000 files, the execution time decreased from approximately 297 seconds to 99 seconds.
I built a native Mac app for DuckDB
TL;DR: Haris developed Datapuddle, a DuckDB client optimized for columnar data rendering with minimal memory overhead.
Haris implemented a custom grid view utilizing CGContext and CTLineDraw to read cells directly from DuckDB's columnar chunks at draw time, bypassing row-object instantiation. This maintained flat memory usage while processing a 41-million-row Parquet file. For Mac users, this provides a low-latency environment for ad-hoc exploration of large datasets without the resource footprint of Chromium-based alternatives. AppStore (not free though), Reddit post.
Behind the Agent: How Hazel Built an AI Coworker for Consumer Brands
2026-10-21. h: 09:30 PT. Online
Hazel started out predicting customer lifetime value, then built an AI coworker that answers any question a consumer brand has about its business. Co-founder and CTO Patrik Devlin joins host Alex Monahan to talk through the data stack underneath it.
DuckLake Performance Optimization: Faster Queries From Your Lakehouse
2026-10-27. h: 10:00 PT. Online
Alex Monahan and Matt Martin, co-authors of O'Reilly's DuckLake: The Definitive Guide, walk through the book's new Performance Optimization chapter: how to get faster queries by skipping files and row groups and writing Parquet that suits your queries.
The Data Team's New Tools (MotherDuck at Helsinki Data Week)
2026-10-28. h: 15:00. Helsinki Bryggeri Brewhouse, Helsinki, Finland
MotherDuck is a Helsinki Data Week partner (Oct 26-30). Join us and dltHub for an afternoon of open discussion on how AI is reshaping data teams, their tools, and their jobs, and how to stay dressed for the new era rather than exposed like the emperor.
Data Outpost 2026
2026-11-05. h: 08:30. Convene 100 Stockton, San Francisco, CA
Editorial AI + data conference for builders at the forefront of AI + data. Two days on memory, agents, trust, interfaces, and data that can act. Small-format, high-context.
DuckDB 2.0: How to Speed Up Your Workloads
2026-11-10. h: 09:00 PT. Online
DuckDB 2.0 speeds up many workloads without changing a single query. Mehdi Ouazza walks through what changed under the hood, from async I/O on object storage to recursive CTEs up to 90x faster, and which workloads benefit most.
Table of contents
Read Nextdocument fetched 2026-10-24 11:26