When atomic isn't atomic

Intro

Have you ever received the same “you’re running low” alert twice, seconds apart? Annoying as a user, embarrassing when it’s your service sending them. I hit this bug recently in a usage-metering service I work on, and the root cause taught me something worth sharing: Redis transactions are atomic, but your logic around them usually isn’t.

The setup

The service is simple. Every time a user does a billable action, we decrement their credit balance in Redis. Users can configure alert thresholds (“warn me at 1,000 credits”, “warn me at 10%”) which we store as a Redis set. When the balance crosses a threshold, we send an alert and remove that threshold from the set, so it only fires once.

At least, that was the idea. In production, some users occasionally got the same alert twice. It never reproduced locally. If you’ve ever chased a bug that’s correct in isolation but wrong in production, you already know where this is going: concurrency.

The code that looked safe

Here’s the shape of the original logic (simplified):

// one Redis transaction: decrement + read thresholds
let (new_balance, thresholds) = redis::pipe()
    .atomic() // MULTI … EXEC
    .decr(&balance_key, used)
    .smembers(&alerts_key)
    .query_async(&mut con).await?;

// decide which thresholds we just crossed
let reached: Vec<i64> = thresholds
    .into_iter()
    .filter(|&t| new_balance <= t)
    .collect();

// claim them, so nobody alerts again
for t in &reached {
    con.srem(&alerts_key, t).await?;
}

send_alerts(reached);

Notice the .atomic(). That issues a real Redis MULTI/EXEC transaction, and the word “atomic” is right there in the code. So how do two alerts get out?

What MULTI actually guarantees

MULTI/EXEC guarantees that the commands inside the transaction run as one uninterrupted block. That part worked perfectly: the decrement and the threshold read always saw a consistent snapshot.

But look at what happens next. The result comes back to the application, we make a decision based on it, and then we send more commands: the SREM claims. The moment EXEC returns, the critical section is over. Any other request can sneak in between our transaction and our follow-up writes.

Think of two travel agents selling the last seat on a flight. Both check the seat map, both see 12A free, both tell their customer “it’s yours”, and only then do they update the map. Checking was safe. Deciding-then-writing was the race.

Here’s the exact interleaving, with a balance of 105 and one threshold at 100:

Both requests run their transaction before either runs its SREM. Both see the threshold still in the set, both decide “reached”, both alert. The SREM was meant to be a claim, but a claim only works if it happens in the same atomic step as the observation that justified it. Ours happened a full round-trip later.

The fix: move the decision into Redis

What we need is for observe → decide → claim to be one indivisible unit. Redis gives you two general tools for this: optimistic locking with WATCH (retry if the key changed under you), or a Lua script, which Redis runs atomically; nothing else executes while it does. I went with Lua: no retry loop on a hot path, one round-trip, and the logic lives right next to the data.

(There’s also a sneaky third option for this exact shape: SREM returns how many members it actually removed, so you could claim first and only alert when it returns 1. That fixes the duplicate alert, but it doesn’t compose. The moment you need anything else in the same atomic step, say clamping the balance at zero, you’re back where you started.)

local used = tonumber(ARGV[1])
local balance = redis.call('DECRBY', KEYS[1], used)

-- observe AND claim in the same step
local reached = {}
for _, member in ipairs(redis.call('SMEMBERS', KEYS[2])) do
    local threshold = tonumber(member)
    if threshold and balance <= threshold then
        redis.call('SREM', KEYS[2], member)
        table.insert(reached, threshold)
    end
end

return {balance, reached}

Now the decrement, the read, and the claims all happen while Redis is doing nothing else. Two concurrent requests serialize inside Redis: whichever script runs second finds the threshold already gone and gets back an empty list. Exactly one request wins the claim. Exactly one alert goes out.

And it’s cheap. Redis client libraries hash the script and invoke it with EVALSHA, so after the first call you’re sending a 40-byte hash per request, not the script body. The atomicity fix costs essentially nothing.

Takeaways

• Atomicity is a property of your logic, not your API call. The instant your code branches on a transaction’s result and writes again, you’ve built a read-modify-write across round-trips, and it will race.
• A claim must be atomic with the observation that justifies it. Any check-then-act split across a network boundary has this bug waiting inside it.
• Lua scripts are Redis’s answer to “I need to decide on the server.” One round-trip, fully serialized, effectively free with EVALSHA.

The next time you see .atomic() in a code review, ask one question: does anything after this depend on what comes back? If yes, the atomic part probably isn’t where the atomicity needs to be.