120,000 distributed consistent writes per second with Calvin

As we prepare for the general availability release of FaunaDB, we’re happy to begin sharing performance data. I’m a big fan of ACID-compliant distributed transactions, so we’ll start there.

Our benchmarks show that FaunaDB can easily exceed 120,000 distributed, consistent writes per second, per logical database, on 15 machines.

FaunaDB can easily exceed 120,000 distributed, consistent writes per second, per logical database.

Unlike other distributed databases that rely on hardware clocks or multi-phase commits, FaunaDB’s transaction consistency algorithm is inspired by Calvin. Calvin is designed for high throughput regardless of network latency, and was the work of Alexander Thomson and others from Daniel Abadi’s lab at Yale.

Calvin’s primary trade-off is that it doesn’t support session transactions, so it’s not well suited for SQL. Instead, transactions must be submitted atomically. Session transactions in SQL were designed for analytics, specifically human beings sitting at a workstation. They are pure overhead in a high-throughput operational context.

Calvin is designed for high throughput regardless of network latency.

FaunaDB’s functional, relational query language is more expressive than SQL in an operational data context, and works perfectly with Calvin. We’ll explore the transaction model in detail in the upcoming FaunaDB white paper, but for now, let’s check out the benchmarks.

Hardware

For the FaunaDB cluster, we have fifteen c3.4xlarge AWS EC2 instances configured as three logical datacenters of five instances each.

These instances have 16 hardware threads, 30GB of RAM, and two ephemeral SSDs in RAID0. The entire cluster has 240 hardware threads, and each logical datacenter is in its own availability zone and replicates the entire dataset.

For the load generators, we have two c3.large instances per availability zone, for a total of 12 hardware threads.

Software

We are using our most recent FaunaDB build on Linux/JDK 8 with the G1 garbage collector, which has shown good adaptability to changing workloads for us. Although the maximum pause time can still be somewhat high with G1, FaunaDB uses strategies like redundant dispatch to minimize the impact of collection pauses and other temporary partition events.

FaunaDB uses strategies like redundant dispatch to minimize the impact of temporary partition events.

The JVM heap size is set to half of physical RAM; unlike many other database systems, FaunaDB does not require any further tuning, and there are no service or hardware dependencies.

Dataset

The schema is a single logical database, with a single class (table) in it, with four indexes defined.

Each request issues an authenticated query that inserts four instances (rows), with four random small strings in each instance. FaunaDB must issue each instance in the query a globally unique id, apply the four index functions to each instance, linearize and commit the transaction (maintaining the semantics of the entire batch), and replicate the effects to the appropriate replicas in every logical datacenter.

Each instance insert generates 9 write effects–one for the instance itself, and two for each index–for a total of 36 writes per transaction. The instances themselves have no partition contention because they all have unique ids, but the associated index writes may contend if any instances in the same Calvin epoch happen to share a value.

All transactions are fully replicated and persisted to disk before they are acknowledged to the client.

Results

Here are the results from running the benchmark for an hour:

  • Transactions per second: 3,330
  • Total transactions: 12,000,000
  • Writes per second: 120,000
  • Total writes: 432,000,000

Here are graphs of the various performance statistics over time:

Writes per second

Transactions per second

CPU utilization

This test shows great horizontal scalability across cores and machines. There is CPU to spare; CPU could be fully saturated if multiple logical databases were transacted at once. The theoretical throughput limit is 1 billion writes per second per logical database, so these results will only get better.

FaunaDB shows great horizontal scalability across cores and machines.

We have a lot more aspects of the system yet to demonstrate, including more reads, joins, and graph operations. Until then, happy transacting!

Get the Technical White Paper

To learn more about the inner workings of FaunaDB, download our white paper: "The Technical Architecture of FaunaDB, an adaptive operational database".

Get the Business White Paper

Discover how Fauna’s alternative approach, adaptive database infrastructure, can help businesses speed innovation while lowering costs and avoiding infrastructure lock-in.

Begin Today

FaunaDB is in production today serving tens of millions of users. Start your free trial of FaunaDB Serverless Cloud and experience the power of global consistency.