WAL: a write-ahead log, or writing it down before I do it

We were talking at the office this week about Cursor’s post on hosting Git at any scale, and one line stuck with me:

The core primitive behind it is a write-ahead log, which we store in S3-compatible object storage.

Hmmmm, a write-ahead log. I know Postgres has been doing this since forever. And now it holds up Git hosting for millions of repositories. So I spent a few evenings reading about it again. This post is my note to relearn it.

The one-line idea

Before changing stuff, write down what I’m about to do first.

That’s the whole idea. Simple as that.

Say I’m deploying a new app on my homelab: pull the image, run the db migration, repoint the proxy, restart. Halfway through, my SSH session drops. Did the migration run? I’m reconstructing what past-me did five minutes ago.

Instead, I write each step down before I run it, “running the migration now”, then run it. Session dies, I read my note, and I know exactly where I was.

That note is the write-ahead log. The “ahead” part is the whole point: the line goes down before the real change, not after.

flowchart LR
    R["Change request<br/>update row 42"] --> L["1 · Append to log<br/>flushed to disk first"]
    L --> D["2 · Update the data file<br/>whenever convenient"]
    L --> C["Crash? Replay the log on restart<br/>anything that never reached<br/>the data file gets redone"]

It’s not only about safety

The log is not just for safety. It also makes the database fast.

So now my data lives in pages scattered all over a file, and one transaction might touch a handful of them. Flushing all of that safely on every commit means many separate writes, right?

What if we use an append-only log? One file. Always write at the end. So a transaction commit becomes: append a small record, flush it, done.

Now it’s one fsync (the “really write to disk now” call) instead of many. 10 transactions committing around the same time can share a single flush. Postgres calls this group commit. So 10 commits cost about the same as 1. The data pages get updated later, in the background, in bulk.

Log now, apply later. That’s durability and speed, not just a safety net anymore.

Clever hah?

“What is the data page?”

A data page is the fixed-size block a database does all its I/O in. Postgres uses 8KB pages, and every row lives inside one. (Most rows, and oversized values get TOASTed out-of-line.)

Updating a row means reading that page, modifying it in memory, and writing the whole page back later. So a transaction touching rows on 5 pages means 5 scattered writes. That is exactly the cost the WAL collapses into 1 append.

The rule that makes it work

The log record has to physically hit the disk before the data page does.

If we break the order, the WAL is pointless. We get a half-baked data file, and no note explaining what was supposed to happen. That is worse than not having a log at all. Because now I think I’m safe, but I’m not.

The Cursor blog says one of the rules like this:

We never acknowledge a push until it has been fully persisted.

There is one more subtlety inside that rule. Replaying the log is called redo. And a page write can be torn:

flowchart LR
    A["Power fails<br/>in the middle of a write"] --> B["Only half of the<br/>8KB page lands on disk"]
    B --> C["WAL record says<br/>'add row to page 42'"]
    C --> D["But redo assumes page 42<br/>was in a good state already"]
    D --> E["Garbage, not recovery"]

So Postgres goes further with full_page_writes:

• Right after a checkpoint, the first change to every page writes the whole page image into the log.
• Later changes to the same page can be small diffs again.
• It makes the log fat right after a checkpoint. That is one reason checkpoint spacing is a tuning knob.
• But it is what makes “replay the log” actually sound.

And how does replay know what is already applied? Every WAL record and every page carry their position in the log. The LSN (Log Sequence Number). During redo, Postgres just compares the two:

flowchart LR
    R["Replay a WAL record"] --> Q{"Page LSN already at<br/>or past the record's?"}
    Q -->|yes| S["Already applied<br/>skip it"]
    Q -->|no| P["Apply the change"]

A position check, not a transaction-ID check. Remember this trick. It comes back later in this post.

The checkpoint

If I keep appending, the log grows forever. Not ideal. So once in a while, the database marks a point: “everything up to here is safely in the data file now, these log records are not needed anymore”. Then it truncates or recycles them.

That’s the checkpoint.

One thing worth mentioning: “recycled” only happens once nothing still needs the segment. And it’s not just the data files that need it. A lagging replica, or a failing archive_command, can pin the log in place:

“What is the replica?”

Two or more Postgres instances. One stays a live copy by replaying the log as it arrives. It’s for availability across zones, or just for read scaling. In this context, the replica marks its position in the log with a “replication slot”. A bookmark telling the main server: “don’t recycle the past yet, I’m still reading!”

If we kill the replica and forget to drop its slot:

flowchart LR
    A["Replica dies,<br/>slot forgotten"] --> B["Bookmark freezes"]
    B --> C["Postgres keeps its promise<br/>every segment since stays"]
    C --> D["pg_wal piles up<br/>until the disk is full"]
    D --> E["Database stops working"]

The checkpoint trims the head of the log. But anything pinning the tail wins. This is the classic “the log grew forever anyway” incident.

It’s the piece everybody skips when they build this pattern themselves. I haven’t had to handle this myself yet. But it’s worth learning anyway.

WAL is already everywhere

WAL is in our everyday life more than we notice.

• Postgres: the pg_wal/ directory. As we mentioned above, Postgres uses WAL to feed replicas and for point-in-time recovery. We can ask it to replay up to 3:45pm yesterday. Just like that.
• SQLite: PRAGMA journal_mode=WAL, and the -wal sidecar file next to the database. Most people flip this on because readers stop blocking writers, without ever asking why. So now we know!
• ext4, NTFS: I was surprised too when I learned this. They call it journaling, but it is more a close cousin than the same thing. By default they journal filesystem metadata, not file contents. So what we get is “the filesystem stays consistent”, not “our data survives”. (ext4’s default data=ordered does flush file data before the metadata commit, so we never get a half-garbage file but un-synced writes can still be lost. 🤯)
• Kafka: fundamentally a distributed write-ahead log. Consumers basically read the log directly.

How do we use WAL in daily life?

WAL is a concept we usually see at the library or database level, deep down. But that does not mean we cannot apply it ourselves, right?

And we might be surprised that this kind of pattern already exists in the normal application level. Same shape but different name. We might already use it too.

At the app level, the “crash” can be many things:

• A process dying.
• A pod getting evicted.
• A timeout.
• A 500 on the request.
• Something lost on the network.
• Or an OOM (out-of-memory) kill.

First, what “durable” means

Durable means: it survives the process dying right now. Not “I called the function”. The bytes are somewhere that survives a hard power loss.

The trap is all the layers that look durable and aren’t:

• A variable in memory → gone on crash.
• An INSERT sent, transaction not committed → gone.
• Written to a file, sitting in the OS page cache → still gone on power loss. write() returned, but the disk never saw it.
• COMMIT returned successfully → durable. The database did the fsync for us. (Pedantic, in Postgres: true as long as nobody turned synchronous_commit off.)

For application code: durable means the database transaction committed. Until await tx.commit() returns, we have nothing. That’s the D in ACID.

“ACID what?”

Atomicity, Consistency, Isolation, and Durability.

This is why the WAL rule is “the log entry must be durable before the side effect”. It’s not before in the order my code reads. It’s before in the order things land on disk.

The problem: dual writes

This is when 2 systems cannot share a transaction. (Two-phase commit exists. But it is rarely worth reaching for it, least of all against someone else’s API.)

await db.orders.update({ where: { id }, data: { status: "shipped" } }); // system A
await slack.chat.postMessage({
  channel: "#fulfillment",
  text: "Order shipped",
}); // system B  ← crash here

Two independent failure points, no atomicity. The order says shipped, but the #fulfillment channel never hears about it. Nobody packs the box, nobody ships it. Nobody finds out until the customer asks us where the order is.

We cannot reorder our way out of it either. Swap the two lines, and the bug just moves. Now we announce a shipment we never recorded. 😂

The fix: make it a single write

We can’t make 2 systems atomic but we can make 2 tables atomic.

await db.$transaction(async (tx) => {
  await tx.orders.update({ where: { id }, data: { status: "shipped" } }); // ← system A
  await tx.outbox.create({
    data: { channel: "#fulfillment", text: `Order ${id} shipped` },
  }); // ← push to the outbox
});

Both rows now live in the same database. One COMMIT makes them atomic.

// Somewhere: the relay worker, on a cron.
const rows = await db.$queryRaw`
  SELECT * FROM outbox
  WHERE published_at IS NULL
  ORDER BY created_at
  LIMIT 100
  FOR UPDATE SKIP LOCKED`; // ← keyword

for (const row of rows) {
  await slack.chat.postMessage({ channel: row.channel, text: row.text });
  await db.outbox.update({
    where: { id: row.id },
    data: { published_at: new Date() },
  });
}

• FOR UPDATE takes a row lock on everything it selected.
• SKIP LOCKED makes a worker that hits an already-locked row skip past it instead of waiting.
• Two workers polling at the same moment each get a different batch, no duplicates between concurrent workers. (Crash-retry duplicates are still possible tho if run the poll inside a transaction, or the locks release before we publish.)

One more trade-off worth naming. SKIP LOCKED gives us at-least-once delivery, not ordering:

• Two workers running in parallel can publish out of order. The ping for order 42 might land before order 41.
• For notifications, that’s fine.
• For anything order-sensitive (payment events, state transitions), serialize per entity, or rebuild the sequence on the consumer side.

We didn’t remove the risk, we moved it. The new failure mode is duplication, not loss. The relay might post, then die before marking the row published, and post again.

So the receiving end has to be idempotent. A Slack channel just sees the ping twice. The order here is deliberate. Mark first, and we trade duplicates for losses. And loss is the one failure we cannot recover from.

(And if polling a table offends us, there is another way. Tail Postgres’ own WAL with logical decoding. Debezium, essentially. Lower latency, but we couple our delivery pipeline to database internals. Polling is boring and portable. Start there.)

Now the harder case.

“Is this a queue? Why ‘outbox’, I never heard of it”

When reading online, I found a cool Reddit thread: “Unpopular opinion: Queues are just specialised databases, and the Outbox pattern IS using a database as a queue” and inside the Reddit post, top comment say “it’s just a database with a specific API on top”. And yes, I agreed. But we can call it whatever we like, as long as we’re on the same page about what problem we’re solving.

(The thread: r/dotnet)

When the 2nd system is a 3rd-party service

Take the Slack notification from the relay above. A duplicate ping is not a double charge. But if it gets too annoying, the channel gets muted. And a muted channel is where the real alert gets missed. We don’t build Slack notifications just to get them muted, right?

Okie, same shape of thought: LOG the intent, DO the post, APPLY the mark. We already have all 3 steps in the relay worker. The interesting part is what happens when the retry posts again. And that part is not ours to decide. It depends on whether the other side cooperates:

Slack does not cooperate. I use chat.postMessage in the Slack app that args: channel, text, blocks, metadata, and friends. No idempotency key, no dedupe, nothing. So a crash between the post and the mark will double-post eventually. All I can do is make the duplicate harmless:

• Keep the text deterministic, so a repeat at least says the same thing.
• Tag retries, so the channel knows it’s a resend.
• Or just accept it. Chat tolerates a repeat.

My current workaround approach: I write the text hash down first under a unique index and only then call Slack API. If the send fails and something retries, the retry hits the unique index and stops right there.

The same exact message never publishes twice. Closer to exactly-once, and I can accept the loss. But let’s be honest about what that loss is:

• Not just “a repeat does not happen”.
• Also “a genuine event with identical text gets dropped”. Two identical orders seconds apart? The second ping vanishes.
• Fine for chat. For anything near accounting, key on the event ID, not the text.

“What about payment? A double charge feels scarier than a duplicate ping”

Same 3 steps. But here the other side cooperates, if we ask properly:

• Idempotency key. We generate it on the LOG step, before the side effect. Stripe (and Square, Adyen, PayPal, all the same shape) saves the first result for that key. So a retry gets the original charge, not a second one.
• It’s my “I already replayed this entry” marker. Same job an LSN does in a real WAL. Asking if it “already applied?” but a blunter, it set membership, not the position check. So it closer LSN analogue but not the same.
• Keys expire. Stripe’s after about 24 hours, Adyen’s at least 7 days. So a retry that outlives the window charges twice. One more reason the retry cap matters.
• APPLY usually arrives as a webhook. The gateway calls us with payment_intent.succeeded. The retry worker is just the fallback for when webhooks are down. Which I find funny. The gateway’s event log is a WAL too, and our webhook handler is the replay.

Where the analogy gets thin

Now the part that took me longest to get straight.

• A database WAL is idempotent by construction. Postgres owns both the log and the data file. One system, one authority, no negotiation. Replaying twice is safe by definition.
• The outbox has no such luxury. The side effect lives in someone else’s system. So replay is only safe if they cooperate. Stripe cooperates if I remember to pass the key. Slack cannot cooperate at all.

The safety isn’t given to me by the pattern. It’s homework.

Same shape, different mechanism. And the difference is exactly the part that pages me at 3am.

Replay is the normal case, not the exception. Design for it up front, not after the first duplicate message.

Don’t forget the checkpoint

My outbox table needs the equivalent. And this is the part people skip. I ship the happy path, and six months later there’s a table with forty million rows in it and a handful of silent orphans.

The retry worker is not optional. Minimum viable version:

-- Find stuck notifications, safely, across multiple workers.
SELECT *
FROM outbox
WHERE published_at IS NULL
  AND created_at < now() - interval '2 minutes'
ORDER BY created_at
FOR UPDATE SKIP LOCKED
LIMIT 100;

(One caveat: a locking clause with ORDER BY at READ COMMITTED can return rows slightly out of order. Harmless for a retry worker. But don’t build anything that needs strict ordering on this query.)

Checklist for anything I build in this shape:

• Idempotency key on every entry, generated before the side effect, when the target supports one.
• A retry worker, with backoff and a cap on attempts.
• A unique index on the idempotency key, so two workers can’t publish the same entry twice.
• A terminal state: failed after N tries, so rows can’t retry forever.
• Cleanup of completed entries, or the table grows without limit.
• An alert on anything pending longer than it should be. This is the one that tells me the pattern is broken before a customer does.

And then Cursor takes it further

Aha, finally done with the WAL basics. Back to the Cursor blog.

In a normal database, the data file is the truth, and the log protects it. Cursor inverted that. Their WAL lives in S3, and that’s the source of truth. The Git repositories on local NVMe disks (fast SSDs), in their words, are “warm caches.”

Now the question I had to sit with. A WAL needs one strict, total order. But S3 has no append(). You cannot extend an object in place. Every write is a whole new object. So where does the order come from?

Cursor’s answer (the system is called Continuity):

flowchart LR
    P["git push"] --> O["1 · Write the packfile to disk and<br/>upload it to S3 simultaneously"]
    O --> I["2 · Rewrite the tiny<br/>WAL index file"]
    I --> W{"3 · Compare-and-swap<br/>on the index"}
    W -->|won the race| OK["Push is visible<br/>acknowledge the client"]
    W -->|lost the race| L["Re-read the index<br/>and retry"]

• Each push is an immutable object. The order lives in one tiny WAL index file.
• The rewrite is an atomic compare-and-swap (CAS): a conditional PUT with If-Match on the index object’s ETag (S3 conditional writes)
• The write fails if someone else won the race. Two servers cannot publish at the same position. The loser re-reads and retries.
• The CAS is the consensus.
• Replicas use the same trick for reads. A conditional GET with the ETag they expect (the object’s version tag). A 304 means “nothing changed, serve from cache”. A 200 means “catch up from the log first”.

So the strict order is not a property of the storage. It is enforced at one point, the index object, on top of storage that is just a pile of immutable bytes.

S3’s conditional writes quietly turned object storage into a consensus primitive: If-None-Match means “create only if absent”, If-Match means “overwrite only if it is still the version I saw” and that second one is what a rewritten index file needs.

Once the log is the truth, a pile of hard problems disappear. There’s no leader election, because there’s no special server to elect:

There’s no state and no consensus here. Any server can be the primary.

Any replica can be rebuilt by replaying the log. Losing a machine stops being an incident and starts being a cache miss. And we get history for free:

Since every push is in the WAL, we can look at every state a repository has ever been in.

That’s the same “replay it somewhere else, replay it up to yesterday” bonus Postgres gets from pg_wal/. Just at a different altitude.

Once the log is durable, ordered, and complete, the “real” data structure is just a projection of it. We can throw it away and rebuild it.

And this shape is escaping into the wild now. walgit, Tobi Lütke’s Rust implementation of the same architecture:

• One binary in front of an S3/GCS bucket.
• No database, no leader, no local state that matters.
• Even fresh clones get served as static bundle files straight from the bucket.
• In their words: “Every machine that runs walgit is a disposable cache; the bucket is the repository.”

Same idea, same WAL, all the way down.

Cool, right! 😆

What I’m taking away

I personally love that when a tech company writes a summary post about what they built. It’s a huge effort from many engineers, and it always sends me back to the fundamentals of systems we already have.

WAL (write-ahead log) is something I surely never learned at school or university (bechelor level). It was all pretty high-level there. this Cursor blog made me go through a deep hole. I read how Postgres works, asking Claude to teach me the concept over days, experimenting locally, digging into the log files.

Pretty valuable experience, IMHO. So yeah, few evenings well spent.

Nice to meet you guys who read til this point. if I missed something (I’m sure I did), please let me know and I’ll update the post.

Worth checking out.

Some blog I found nice to read along the way.

• Postgres (WAL Write-Ahead Log): https://www.postgresql.org/docs/current/wal-intro.html
• Object Storage as a Queue: https://deveshshetty.com/blog/object-storage-queue/
• Conditional Writes Without a Compare-and-Swap Store: https://www.seaweedfs.com/blog/conditional-writes/
• Simon’s blog about AWS S3 conditional writes https://simonwillison.net/2024/Nov/26/s3-conditional-writes/