Entering the world of database surreality

extract

Entering the world of database surreality SurrealDB has implemented a distributed, multi-modal database that provides enormous scale and high-performance, while combining relational, full-text, document, time-series, graph and vector data type access to the same underlying data with its proprietary storage engine. How? Co-founder and CEO Tobie Morgan Hitchcock told me what sparked his interest in multi-modal databases: “It's the fact that you're using time series databases, document databases, graph databases, key value databases and related databases in every single app. Obviously nowadays you're using it even more, especially graph. But back then it was just like, how can we do something like Firebase, but actually make it a database that can scale to hundreds of terabytes or petabytes of data in a transactional way. That's always been my kind of interest.” He thinks graph is the best way of modelling data and said: “How do you do versioning of it? That was my dissertation at Oxford. It was in temporal versioning [of a] key value store for graph.” “It was just single node that was not distributed at that time, but it was the groundwork for distributed. So the dissertation looked at how it might do this in a distributed manner and back then Raft was the thing to look at. And then I built the multi-modal database on top of that. So it was always separated; storage from compute.” (From here in we’re assuming you’re familiar with Paxos and Raft fault-tolerant distributed data store protocols. Take a look at the bootnotes below to get a surface-level familiarity with them.) Initially, back in 2022, Tobie said: “ We launched [open-source] SurrealDB, which is the multi-modal database and it could run on top of different key value stores. We launched it and it could run on top of RocksDB. And then you could run it on TiKV. … That was our way of getting up quicker because building our own key-value store … would have been a challenge in that timeframe.” Things evolved. “We've now built our own distributed storage engine. So the design before … was a stateless layer sitting on top of TiKV, which is stateful and Raft- based and that didn't work very well for graph because [with] graph you have to multi-hop, and you get one set of records. You now have to go back and get the next set, and then you have to get the next set and it just gets larger and much larger. And when you're doing every single one of those queries over the network, because TiKV is stateful but networked, that becomes an issue.” “What we've done now is design our own distributed store. It's not based on Raft. It's based on something called TAPIR, Transactional Application Processing with Inconsistent Replication. We're the first distributed KV store in the world to implement on top of this.” TAPIR “comes from Microsoft Research Labs and the University of Washington.” (You can get an introduction to TAPIR in the bootnote at the end of this article.) Tobie said: “We've built TAPIR into the SurrealDB nodes. So SurrealDB is now stateful again, but it sits on top of object storage, which is stateless. You can scale the SurrealDB nodes out to 15 nodes, that's [the] limit for the moment, but those 15 nodes will be running on top of multi-petabyte scale object storage and we'll be pulling down the data as we need.” “Some queries might take a little bit longer, but generally speaking, we're caching that layer. … Our intention is to scale out that so you can have a hundred nodes, but right now we're at 15 nodes and we'll be extending that over time.” “Every single one of those 15 nodes can accept writes at the same time. And the intention is you can scale up to a thousand nodes sitting with all the object storage and they'll all be able to accept writes at the same time. So you never have two thirds of your cluster sitting there idly, just being used for out of date requeries or backups or quorum. “That's our new design. It's completely new. There's no one else. … It's very exciting because it reduces the amount of round trip time in a network. It increases the amount of nodes that can participate in write queries. It reduces the latency of writes as well. So it's a much more efficient consensus algorithm.” “It means we can scale up to much more data than previously could be done. And if you look at [Google] Spanner and TiKV and CockroachDB, they're all limited by cost effectively. You want to store a petabyte, it's a significant amount of cost to make that cluster run. Whereas with ours, you can store the petabyte in object storage and you can shrink the database down to three nodes.” With the SurrealDB cloud product “we run our workflows on EBS. It's not ideal. … If you're running on object storage, that node is ephemeral. If it goes down, it doesn't matter.” That’s because object storage has high data durability: “Without object storage, if you have three nodes, they can all accept writes and that's good. It's not like one goes down. But if [without object storage] one [database node] goes down and comes back up, it has to transfer the data from the other nodes. So if there's 10 terabytes in the cluster, it has to transfer 10 terabytes. Even if you shard that, it's still the same, because you want to have that redundancy.” Object storage such as S3 has built in redundancy, 11 nines (99.999999999 percent) of data durability. Tobie said: “So when a [SurrealDB] node goes down and comes back up again, it comes back up in a hundred milliseconds and then has access to all the dataset. … When a query is run on that node, it may have to go and fetch data behind the scenes, but then that's cache and you run it again. So if you've got 10 terabytes of cache there, it comes up instantly and stores the data as it's queried in its cache, and [is] able to handle it.” Transactionality He brought out his view of transactionality: “The traditional [database] use case is transactional, right? Every single thing, in my opinion, has to go in transactionally. If you're going to run an agent memory system for fraud detection or intelligence or even if it's just organisational-wide customer support, right? It's not even that complex or mission critical, but you still have to have the data. “This agent over here dealing with risk for insurance has to have a completely up-to-date database of what's happened for customer search or the customer conversation or sales. It can't be, "Oh, we'll process that later on. " It has to be real time. It has to be transactional and yet the transactional databases are limited to either three nodes. If you look at Neo4j and things like that, they're limited to three nodes and the vertical scaling of that node. … CockroachDB obviously can scale out, but you're limited to the amount that it can scale to and the cost of the devices. “One thing we're adding … is the ability to say this table is column now and it's stored on S3. And when you write to it, you get transactional writes, but it also means you can query in a column in a way very fast over analytics of a certain row or certain field. There's no hybrid in the middle. The problem is you need transactional for a lot of use cases and you need column for a lot of use cases. … There's no middle ground between them. One stores it in columns and one doesn't. One's very fast for analytics, but not for writes, and one's very fast for writing [but] not for analytics. Your hybrid is to effectively create a column index. So here's my analytical data, here's a column index.” A “but” here comes from agentic AI: “I think the biggest issue though, is every single one of these workloads is transactional for agent memory, for non-analytical use cases. And those workloads are scaling up massively … for AI, we want to store as much data as possible. We've got hundreds of thousands of agents writing back.” “We're writing back massive conversations of text that needs to be indexed and understood. What are our options? We can get rid of that and just summarise it into a few sentences. Over time, that's not going to be accurate. Or we can keep our entire accuracy across our organisation and store every single conversation an agent has ever had. That is significant scale and we want to do that on transactions.” “We have to have our transactional database work from object storage. So it's not just object storage from the common point of view, it's object storage from a transactional point of view.” “Now Databricks bought Neon and Neon enables this in a way. It's backed by object storage and it's relational database. But the problem with Neon is it's single writer. They don't really talk about it much. Single writer accepts one node writes and you can have a hundred read replicas, but that doesn't scale for an organisation. It's great to get going, but it doesn't scale. Our design will enable us to have 15 write nodes or 30 write nodes and then you'll be able to scale out and redirect to hundreds or thousands of those as well.” To indicate the scale involved, he said: “We’re working with an organisation who's spinning up 100,000 agents by the end of July and potentially to millions by the end of the year. ... Our concept is, or our belief is, to have the best accuracy and the best understanding, I have to store every conversation, every transaction. … That's why you need transactional over object storage, because, otherwise, the cost becomes unfathomable, let alone the token cost.” “If you have object storage drawing a petabyte and you're storing 10 percent of your most recently used data is in cache, that's a hundred terabytes sitting in front of EBS and that's significantly cheaper than storing a petabyte in EBS and on SSDs." Customers Who were the customers he had in mind when SurrealDB was founded? “From an idealistic point of view, for the customer base, there are very few customers in the world who need the ability to analyse massive data sets which are relational driven, so graph network-driven with support for time. Those are basically defence and government.” We suggested that then AI inferencing came along and dropped a nice big opportunity in SurrealDB’s lap with graph and vectors? He said: “Absolutely. I mean potentially more graph than vector because vector was just a way of searching at the time, but obviously now we know that it wasn't the be all and end all of searching. [We’re doing] graph with vector with full tech search, and actually we're adding a new piece of functionality in the next minor release, which is GREP indexing; the ability to have fuzzy search indexing over text in a similar way. So you could have on your text and your embeddings, you could have a vector search, you could have full text search, you could have grep search and then you could have graph for reranking as well.” SurrealDB has a set of solid globally prominent customers. Tobie said the product is being used instead of Palantir: “I've seen Andy Burnham's plans to rip up the NHS contracts that were planned, which is an interesting one because we're being used to replace Palantir in a number of cases. Obviously not the NHS because their contract was into the future.” He said: ‘we are working with defence organisations and financial services organisations. Financial services on the fraud side. We're talking with an organisation in the US who need a transactional graph database that can scale to where they can analyse the relationships being created and inferred in real time for fraud for financial services, but at the network level. … This company is looking to work with Google, Amazon, Cloudflare, and it's looking to take over one petabyte of data a day to analyse network requests and network traffic in order to build a centralised fraud detection system” before financial interactions even start. Memory vs storage He divides SurrealDB into a memory product and a storage product. The memory product is a kind “of layer on top that handles, that uses all the elements of the database, from full tech search to vector to graph search to graph to time series or temporal modelling, to deliver human-like memory.” And: “The memory layer is stateless, agnostic. We store everything in terms of relational, document, graph, vector, full set, search, everything like that … sits inside the database. There's no additional layer on top.” “The memory layer is the behaviour. We do things like analyse text that comes in and store it. That's the memory layer that analyses it. And then we have other background processes. We have background processes for consolidation, for reconciliation, for reflection, for elaboration, for calibration. So those are all the background process that come in and that's what that memory layer is doing.” “Ultimately, how we're storing that memory, we're mimicking the human brain by utilising the database and what it offers and then [have] the ability to scale out to petabytes of data.” This young startup already has a quite mature product that is positioned to provide contextual memory for large numbers of AI agents. Check it out. There’s a lot of documentation to explore on SurrealDB’s website if you want to learn more. Bootnotes - Paxos Algorithm Envisage a fault-tolerant, distributed set of nodes, such as a sharded database, with each node having a replica of some of the shards. Communications between the nodes is asynchronous. A node goes down and then comes back up to rejoin the set. The intervening updates referring to its replica shards must be applied. How does it know they are correct? The set of nodes collectively agrees on the status of updates by having a leader node. When an update message arrives at the set of nodes a receiving node gives it a unique number and sends it out to the other nodes as a proposed update. This is called a prepare message. If a majority of the receiving nodes accept it, as in each not having a conflicting message, than the proposed update is accepted and becomes the consensus view by the proposer now sending out an accept request message. To reduce inter-node message passing, a node can find out which proposals have been accepted by receiving this information from a so-called distinguished learner (leader) or subset of them. As the set of nodes agrees on an update message status (accepted or not) the Paxos algorithm is said to provide strong consistency. Overall, the distributed system behaves as if there is a single, correct, sequentially consistent copy of the data (ordered set log), even though it is replicated across nodes which can fail, i.e. are unreliable. Find out more from this PDF document. Raft Raft is an implementation of Paxos with a strong leader concept. There is only one leader at any time, with the other nodes called followers. All update requests come through the leader. Raft is easier to implement than Paxos. Read a paper about it here. TAPIR The TAPIR algorithm reduces the number of internode messages compared to Paxos and Raft. It is explained in a "Building Consistent Transactions with Inconsistent Replication” paper. The TAPIR paper’s abstract states: “Application programmers increasingly prefer distributed storage systems with strong consistency and distributed transactions (e.g., Google’s Spanner) for their strong guarantees and ease of use. Unfortunately, existing transactional storage systems are expensive to use – in part because they require costly replication protocols, like Paxos, for fault tolerance. In this paper, we present a new approach that makes transactional storage systems more affordable: we eliminate consistency from the replication protocol while still providing distributed transactions with strong consistency to applications. “We present TAPIR – the Transactional Application Protocol for Inconsistent Replication – the first transaction protocol to use a novel replication protocol, called inconsistent replication, that provides fault tolerance without consistency. By enforcing strong consistency only in the transaction protocol, TAPIR can commit transactions in a single round-trip and order distributed transactions without centralized coordination. We demonstrate the use of TAPIR in a transactional key-value store, TAPIR-KV. Compared to conventional systems, TAPIR-KV provides better latency and throughput.” The authors note “replication protocols with strong consistency, like Paxos, impose a high performance cost, while more efficient, weak consistency protocols fail to provide strong system guarantees.” They understand that “existing transactional storage systems waste work and performance by incorporating a distributed transaction protocol and a replication protocol that both enforce strong consistency. Instead, we show that it is possible to provide distributed transactions with better performance and the same transaction and consistency model using replication with no consistency.” They devized inconsistent replication (IR) “that provides fault tolerance without consistency. Rather than an ordered operation log, IR presents an unordered operation set to applications. Successful operations execute at a majority of the replicas and survive failures, but replicas can execute them in any order. Thus, IR needs no cross-replica coordination or designated leader for operation processing. … TAPIR efficiently leverages IR to distribute read-write transactions in a single round-trip and order transactions globally across partitions and replicas with no centralized coordination.” The authors state “Our experiments demonstrate that TAPIR-KV lowers commit latency by 50 percent and increases throughput by 3x relative to conventional transactional storage systems. In many cases, it matches the performance of weakly-consistent systems while providing much stronger guarantees." Read the paper for more details.