Quack: The DuckDB Client-Server Protocol

DuckDB has officially introduced the Quack protocol, enabling its instances to communicate in a client-server setup, a significant expansion of its architecture capabilities. This development allows multiple concurrent writers and remote querying, addressing longstanding limitations of DuckDB’s in-process design.

The Quack protocol is implemented in DuckDB v1.5.2 and can be installed via the core_nightly repository. It allows two or more DuckDB instances to connect over network protocols, supporting data sharing and remote query execution. The protocol is designed to be simple, fast, and built on proven technologies like HTTP, making setup straightforward for users.

To use Quack, users install and load the extension in each DuckDB instance, then establish communication by calling specific functions such as quack_serve on one instance and attaching remote instances using the attach command. Once connected, users can perform data operations like copying tables or executing queries across instances, effectively creating a client-server environment.

Why It Matters

This development matters because it enables DuckDB, traditionally an in-process database optimized for data science and embedded use, to now support multi-process, networked environments. It broadens DuckDB’s applicability to scenarios like telemetry collection, concurrent data ingestion, and remote querying, which previously required workarounds or external systems.

By introducing a native, simple, and fast protocol, DuckDB aims to compete more directly with traditional client-server databases like PostgreSQL, while maintaining its lightweight and flexible design. This could lead to wider adoption in distributed data workflows and real-time applications.

Background

Before Quack, DuckDB primarily operated as an in-process database, excelling in interactive data analysis within a single process. While it supported embedding in applications and notebooks, multi-process access was limited due to internal state management challenges. Existing workarounds involved custom RPC solutions or retrofitting with protocols like Arrow Flight SQL, but these were often complex or limited.

The move to introduce a dedicated, purpose-built protocol reflects a recognition of user demand for networked access, driven by community-built workarounds. This is a significant shift, aligning DuckDB more closely with traditional client-server architectures without sacrificing its core simplicity and performance.

“We see DuckDB as a universal data wrangling tool. If this means having a client-server protocol in addition to the in-process capabilities – fine. If this ends up unlocking a vast new set of cases in which DuckDB can be useful – excellent!”

— DuckDB team

“Designing Quack from scratch allowed us to learn from existing protocols and optimize for simplicity and speed, ensuring it supports workloads from bulk operations to small transactions.”

— DuckDB developers

Klein Tools VDV226-110 Ratcheting Modular Data Cable Crimper / Wire Stripper / Wire Cutter for RJ11/RJ12 Standard, RJ45 Pass-Thru Connectors

EFFICIENT INSTALLATION: Modular crimp-connector tool with Pass-Thru RJ45 plugs for voice and data applications, streamlining installation process

AmazonView Latest Price

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

What Remains Unclear

It is not yet clear how widely adopted Quack will become or how it will perform in large-scale, real-world deployments. Details about security, scalability, and integration with existing network environments remain to be tested and documented.

What’s Next

Next steps include broader testing by the community, integration with various tools, and potential enhancements such as security features and support for more complex workflows. The DuckDB team is likely to monitor adoption and gather feedback to refine the protocol in future releases.

Key Questions

How does Quack compare to existing database protocols?

Quack is designed to be simple, fast, and built on proven technologies like HTTP, aiming to support a wide range of workloads with minimal overhead. It draws inspiration from protocols like Arrow Flight SQL but is tailored specifically for DuckDB’s architecture.

Can I use Quack to connect DuckDB instances over the internet?

Yes, theoretically, as long as network configurations allow, DuckDB instances can connect remotely using Quack. Security considerations such as encryption and authentication are still to be fully documented or implemented.

Will Quack replace existing workarounds or only complement them?

Quack is intended to provide a native, official solution for client-server communication, reducing reliance on custom RPCs or third-party protocols, and complementing DuckDB’s existing in-process capabilities.

Is Quack suitable for high-concurrency or large-scale data workloads?

Initial design aims for efficiency across workloads, but real-world performance and scalability at large scale are still to be validated through testing and deployment feedback.