Hello, and happy morning!

Reflect is a new way to build multiplayer web apps like Figma, Notion, or Google Sheets.

Reflect is an evolution of Replicache, our existing client-side sync framework. It uses the same game-inspired sync engine as Replicache, but adds a fully managed server, making it possible to start building high-quality multiplayer apps in minutes.

Today, Reflect is publicly available for the first time. Visit reflect.net to learn more, or hello.reflect.net to get started.

Why Reflect?

Collaborative editing invariably involves conflicts.

This is just a matter of physics – information can only travel so fast. If you want instantaneously responsive UI, this means you can't wait for the server – changes have to happen locally, on the client. If the application is collaborative, then two users can be editing the same thing at the same time.

These conflicting edits must be synchronized somehow, so that all users see the same thing, and so that conflicts are resolved in a way that feels natural.

The heart of any multiplayer system is its approach to this problem. Like the engine of a car, the choice of sync engine determines just about everything else – the developer experience, the user experience, the performance that is possible, the types of apps that can be built, and more.

In the web ecosystem, CRDTs are a popular way to sync data. CRDTs (Conflict-Free Replicated Data Types) are a type of data structure that always converge to the same value, once all changes have been exchanged between collaborators. Yjs and Automerge are two popular open-source CRDT libraries.

But Reflect is not a CRDT. We call our approach Transactional Conflict Resolution. It's a twist on Server Reconciliation – a technique that has been popular in the video game industry for years.

All the unique benefits and differences of Reflect flow from this one core choice, so it helps to understand it if you want to know what Reflect's about. Let's dive in.

Consider the Counter

Imagine you're using the Yjs CRDT library and you need to implement a counter. You decide to store the count in a Yjs map entry:

const map = new Y.Map();

function increment() {
  const prev = map.get('count') ?? 0;
  const next = prev + 1;
  map.set('count', next);
}

You test your app and it seems to work, but in production you begin receiving reports of lost increments. You do some quick research and it leads you to this example from the Yjs docs:

// array of numbers which produce a sum
const yarray = ydoc.getArray('count');

// observe changes of the sum
yarray.observe((event) => {
  // print updates when the data changes
  console.log('new sum: ' + yarray.toArray().reduce((a, b) => a + b));
});

// add 1 to the sum
yarray.push([1]); // => "new sum: 1"

This is kind of surprising and awkward, not to mention inefficient. Why doesn't the obvious way above work?

Yjs is a sequence CRDT. It models a sequence of items. Sequences are great for working with lists, chunks of text, or maps — all tasks Yjs excels at. But a counter is not any of those things, so Yjs struggles to model it well.

Specifically, the merge algorithm for Yjs Map is last-write wins on a per-key basis. So when two collaborators increment concurrently, one or the other of their changes will be lost. LWW is the wrong merge algorithm for a counter, and with Yjs there's no easy way to provide the correct one.

This is a common problem with CRDTs. Most CRDTs are good for one particular problem, but if that's not the problem you have, they're hard to extend.

Transactional Conflict Resolution

Now let's look at how we would implement a counter in Reflect:

async function increment(tx, delta) {
  const prev = (await tx.get('count')) ?? 0;
  const next = prev + delta;
  await tx.put('count', next);
}

It's clear, simple, obvious code. But, importantly, it also works under concurrency. The secret sauce is Transactional Conflict Resolution. Here's how it works:

In Reflect, changes are implemented using special JavaScript functions called mutators. The increment function above is an example of a mutator. A copy of each mutator exists on each client and on the server.

When a user makes a change, Reflect creates a mutation – a record of a mutator being called. A mutation contains only the name of the mutator and its arguments (i.e., increment(delta: 1)), not the resulting change.

Reflect immediately applies the mutation locally, by running the mutator with those arguments. The UI updates and the user sees their change.

Mutations are constantly being added at each client, without waiting for the server. Here, client 2 adds an increment(2) mutation concurrently with client 1's increment(1) mutation.

Mutations are streamed to the server. The server linearizes the mutations by arrival time, then applies them to create the next authoritative state.

Notice how when mutation A ran on client 1, the result was count: 1. But when it ran on the server, the result was count: 3. The conflict was merged correctly, just by linearizing execution history. This happens even though the server knows nothing about what increment does, how it works, or how to merge it.

In fast-moving applications, mutations are often added while awaiting confirmation of earlier ones. Here, client 1 increments one more time while waiting for confirmation of the first increment.

Updates to the latest authoritative state are continuously streamed back to each client.

When a client learns that one of its pending mutation has been applied to the authoritative state, it removes that mutation from its local queue. Any remaining pending mutations are rebased atop the latest authoritative state by again re-running the mutator code.

This entire cycle happens up to 120 times per second, per client.

The Payoff

This is a fair amount of work to implement.

You need a fast datastore that can rewind, fork, and create branches. You need fast storage on the server-side to keep up with the incoming mutations. You need a way to keep the mutators in sync. You need to deal with either clients or servers crashing mid-sync, and recovering.

But the payoff is that it ✨generalizes✨. Linearization of arbitrary functions is a pretty good general-purposes sync strategy. Once you have it in place all kinds of things just work.

For example, any kind of arithmetic just works:

async function setHighScore(tx: WriteTransaction, candidate: number) {
  const prev = (await tx.get('high-score')) ?? 0;
  const next = Math.max(prev, candidate);
  await tx.put('high-score', next);
}

Most list operations just work:

async function append(tx: WriteTransaction, item: string) {
  const prev = (await tx.get('shopping')) ?? [];
  const next = [...prev, item];
  await tx.put('shopping', next);
}

async function insertAt(
  tx: WriteTransaction,
  { item, pos }: { item: string; pos: number },
) {
  const prev = (await tx.get('shopping')) ?? [];
  // splice() clamps pos for us
  const next = prev.toSpliced(pos, 0, item);
  await tx.put('shopping', next);
}

// Note: you need to pass the item (or a stable id), as the index can change.
async function remove(tx: WriteTransaction, item: string) {
  const prev = (await tx.get('shopping')) ?? [];
  const idx = list.indexOf(item);
  if (idx === -1) return;
  const next = prev.toSpliced(idx, 1);
  await tx.put('shopping', next);
}

You can even enforce high-level invariants, like ensuring that a child always has a back-pointer to its parent.

async function addChild(
  tx: WriteTransaction,
  parentID: string,
  childID: string,
) {
  const parent = await tx.get(parentID);
  const child = await tx.get(childID);
  if (!parent || !child) return;

  // Invariant: child.parentID and parent.childIDs must always be consistent.
  const nextParent = { ...parent, childIDs: [...parent.childIDs, childID] };
  const nextChild = { ...child, parentID };

  await tx.put(parentID, nextParent);
  await tx.put(childID, nextChild);
}

All of these examples just work, and merge reasonably without any special sync-aware code.

Server Authority

The benefits of Transactional Conflict Resolution extend further because of the way sync works.

In Reflect, the server is the authority. It doesn't matter what clients think or say the result of a change is – their opinion is not even shared with the server or other clients. All that is sent to the server is the mutation name and arguments. The server recomputes the result of a mutation for itself, and all clients see that result.

This means that the server can do whatever it wants to execute a mutation. It doesn't have to even be the same code as runs on the client. It can consult external services, or even roll dice.

One immediate result of this design is that you get fine-grained authorization for free.

For example, imagine you are implementing a collaborative design program and you want to allow users to share their designs and solicit feedback. Guests should be able to add comments, highlight the drawing, and so on, but not make any changes.

Implementing this would be quite difficult with a CRDT, because there is no place to put the logic that rejects an unauthorized change.

In Reflect, it's trivial:

async function updateShape(tx: WriteTransaction, shapeUpdate: Update<Shape>) {
  // Your authentication handler runs server-side and can set any fields on
  // tx.user. See: https://hello.reflect.net/authentication.
  if (tx.environment === 'server' && !tx.user.canEdit) {
    throw new Error('unauthorized');
  }
  // ...
}

Notice that the mutator actually executes different code on the server. This is fine in Reflect. The server is the authority, and it can make its decision however it wants.

Even More

There are even more benefits to this approach. For example, schema validation and migrations just sort of fall out of the design for free. Future blog posts will explore these topics in more detail.

For now, I'll end this where I started: the choice of sync strategy is the heart of any multiplayer system. It determines just about everything else. And while there are certainly benefits to other approaches, we find that the game industry has a lot to teach on this topic. Transactional Conflict Resolution "fits our brain" in a way other solutions don't. It's simple, flexible, and powerful.

If you're building a multiplayer application, you should try out Reflect and see if it fits your brain too.

And hey, if you've made it this far, you're our kind of person. We'd love to hear from you. Come find us at discord.reflect.net or @hello_reflect and say hi. We'd enjoy hearing about what you're building.