(id >> 22) % 10. That line decided which of my 10 workers handled each of 100,000 jobs, and it was the only coordination the system had.
On April Fools’ Day 2021 we changed the nickname of about 100,000 members of a Discord server, all in one go. It was for a long-term client’s event, and I can’t name them. A single bot runs into Discord’s rate limits quickly, so the work had to be spread across many. We made the plan with less than two days to spare, and I wrote the code in one day. As far as I remember, it was my first distributed system.
That line let me skip the database, the queue and the coordinator. It also has limits, and I’ll point out where the shortcut would have hurt if the job had run longer or the server had been bigger. The same trick applies to any job keyed by an ID, such as backfills and crawlers.
The job
The requirement: change the nickname of every member of one server, fast, and be able to undo it.
The constraint was Discord’s rate limits. They apply per bot token, so one bot is slow no matter how well you write it, and ten bots give you ten separate limits to spend. I had TypeScript, discord.js 12 and ten bot accounts. I also had one extra “boss” bot that listened for commands and reported progress. A caveat on the word “distributed”: all ten workers ran as separate Discord clients inside one Node process on one server. The split was across tokens, which is what the rate limits care about, and not across machines. The assignment rule would work the same on separate machines, because no worker needs to know about the others, but I never ran it that way.
New to this? What is a rate limit, and why do more bots help?
A rate limit is a cap on how many requests an account can make in a window of time. Go over it and the API tells you to wait. Discord applies these limits per token and per type of request, so a second bot with its own token gets its own allowance. Ten tokens let you make roughly ten times as many changes in the same time, as long as no two bots try to change the same person.
The textbook design is a queue. The boss reads the member list, pushes every ID onto a queue, and the workers pull from it. That needs somewhere for the queue to live, a way to mark jobs done, and a plan for when a worker dies halfway through a batch. I had a day, and I wanted something I could test on a small server before the real one.
So the problem got smaller: can every worker work out, by itself, which members belong to it?
The line
Discord already answers this for its own bots. A bot in many servers is split across gateway shards, and the documented formula is shard_id = (guild_id >> 22) % num_shards. I copied it, applied it to user IDs instead of server IDs, and used the number of bot tokens as the shard count.
const members = await guild.members.fetch();
this.members = members.filter(
(m) => BigInt(this.shardnumber) ==
(BigInt(m.id) >> BigInt(22)) % BigInt(constant.tokens.length)
);Each worker fetches the full member list when it starts, keeps the members whose number matches its own, and ignores the rest. Worker 3 never asks who should rename someone. It computes the answer, and every other worker computes the same answer for the same ID.
The shift is what makes the split even. The top bits of a Discord ID are a timestamp, the millisecond the account was created, so shifting right by 22 bits leaves just that timestamp. Taking it modulo 10 gives a number from 0 to 9. The last digit of a creation time is effectively random, so the ten groups should come out close to equal in size. I no longer have the exact counts from that day, so treat that as the expected result and not a measurement.
New to this? Snowflake IDs, bit shifts and modulo
A Discord ID is a 64-bit number called a snowflake. It packs several fields into one integer: a timestamp in the top bits, then a few bits for the machine that created it, then a counter. Shifting right by 22 bits throws away everything below the timestamp.
>> is a bit shift. x >> 22 is the same as dividing by 2^22 and dropping the remainder. % is modulo, the remainder after division, so n % 10 is always between 0 and 9.
JavaScript numbers lose precision above 2^53, and snowflakes are bigger than that, which is why the code uses BigInt.
Why there is no state
Nothing about who does what is stored anywhere. The boss sends one command. Each worker recomputes its slice from the live member list, then walks through it. No worker talks to another worker, and the boss never hands out a job.
The only progress that matters lives in Discord. Before each run, a worker filters out members who already have the target nickname, so the member’s own nickname is the done marker. If the process had crashed at hour five, I would have started it again and it would have skipped everyone it had finished.
// simplified: shardOf() is the formula from above
this.members = guild.members.cache.filter(
(m) => shardOf(m.id) === this.shardnumber && m.nickname != name
);Reversing it works the same way. The restore command runs the same loop, but picks members who currently have any nickname and sets it back to empty.
New to this? What does idempotent mean?
An operation is idempotent if doing it twice leaves things in the same state as doing it once. Setting a nickname to “x” is idempotent. After the first call the member has nickname “x”, and the second call changes nothing. That property is what lets you rerun a job after a crash without tracking which jobs finished.
I’d like to say stateless was a design principle. It started as a way to write less code. Every piece of state I added was something to test, and I had under two days to make this work on a real server. I rolled it out in three stages: first a small test server with a few members, then the real server limited to staff roles, then the full run on the day.
There is a little state. The boss keeps counters in memory for each worker: total, successful and failed. The status command turns them into an embed with progress, elapsed time, rate and an estimated time remaining, plus a combined total. If the boss restarts, the counters reset. Nothing breaks, because the counters were only for watching.
What was specific to Discord
Snowflakes are 64 bits: 42 bits of timestamp counted in milliseconds from 1 January 2015, then 5 bits for a worker, 5 for a process and 12 for a counter. Those last three fields add up to 22 bits, which is where the shift comes from.
Discord’s gateway formula splits by server ID so that one bot can spread its servers across connections. I reused the arithmetic for a different purpose, splitting members of a single server.
Rate limits are tracked per token and per route. discord.js 12 queues requests and waits out the reset, so I did not write retry logic. Each worker fires every setNickname call for its slice at once and lets the library space them. The rateLimit event logs each wait, which is how I could see the limits at work.
Fetching every member of a large server needs the privileged server members intent. These were private bots in a single server, so I could switch that intent on in the developer portal for all ten without going through Discord’s verification.
A bot can only rename members below its highest role, and it can never rename the server owner. Every member it can’t change shows up as a failed call. In the end fewer than five people stayed unchanged: the server owners and top staff, whose roles ranked above the bots.
Each change also carries a reason string that appears in the server’s audit log. Mine said who was behind it.
Running it
The boss answered a handful of commands: massnick to start, status for progress, restore to undo, stats for a count of how many members had the target nickname, and shards for each worker’s member count and ping. A worker only counted a rename as done when Discord confirmed it.
About 100,000 changes in 8 hours is roughly three and a half a second across the whole fleet. Divided across ten bots, each one made roughly one change every three seconds. I never measured why it was that slow, but I suspect most of that time was rate-limit waiting, which is why ten bots were worth having.
There is not much of a story in the 8 hours, and I count that as the result. The client and I started it, watched the status numbers climb for a while, and went to sleep. In the morning it had finished. I didn’t have to touch it overnight.
On the first full pass, fewer than 100 renames failed. I ran it a second time, and because everyone who was already done got skipped, it only touched the members who had failed and closed the gap in a few minutes. That second run is the idempotent filter doing the job a retry queue would have done.
Where it breaks
These are the weak spots I can see now, reading the code again. Some of them probably account for the failures on the first pass.
Each worker built its list from its member cache at the moment I sent the command. Anyone who joined after that was never assigned to anyone, which for a long job on a growing server is a real hole. Nothing checked whether a member was still in the server before the rename went out, so anyone who left during the run came back as a failed call, with no reason recorded.
The number of workers is baked into the formula. Change it and every member moves to a different worker. Because the jobs are idempotent, that only costs duplicate work, but it means the number of bots had to be decided before the first run.
Every worker fetched the whole member list and then threw away nine tenths of it. That is fine at 100,000 members and wasteful at ten million.
Firing every call at once puts thousands of pending requests in the library’s queue per worker. There is no backpressure, so a worker that falls behind just holds more in memory.
The counters live in the boss’s memory, and failed renames are not retried automatically. A retry is another run, which only touches the members who aren’t done. After a restart the status total counted the whole slice again, so the progress figure started low and understated how much was left.
What I’d do now
I would keep the core idea, because it held up. If every job has a stable ID, and the operation is safe to repeat, and you can ask the target whether the work is done, you can split the job across workers with one line and no shared storage.
I’d change what surrounds it. For anything that runs for days, grows while it runs, or can’t afford a duplicate, a real queue earns its keep. For a bounded one-off, I would skip the queue again. What took the longest was noticing that the ID already contained the answer.
I write up what I build and break. If you have split a job like this a different way, or you can see a hole in mine, I’d like to hear it. My email is just below.