At DPL I built the scheduler behind bulk push campaigns. An admin imports a file of MSISDNs in the back office and sets the notification's body, type, and send time. The scheduler then delivers it to Android (FCM) and Huawei devices: 700-800k notifications per run, in 6-8 minutes.
Sending one notification is easy. The interesting part is sending 800k without holding 800k of anything in memory, and without one bad device slowing everybody else down.
The naive version
Load every recipient, loop, send. It works in testing and falls over in production: the whole list sits in memory at once, one slow provider call holds up the rest, and a crash halfway means starting again with no idea who already got the message.
Two tables and a scheduler
The import writes the notification to one table and its MSISDNs to another, both marked unprocessed. The scheduler picks up unprocessed notifications, then works through the MSISDNs stored against each one. Keeping "what to send" and "who to send it to" apart is what makes the rest possible: the recipient list can be read piece by piece, and progress is recorded per recipient.
Read in chunks
Holding 800k+ records in memory at once is how a scheduler takes a server down with it. So recipients are read in fixed-size chunks, and only a chunk at a time is ever in flight per worker. A keyset query (WHERE Id > @lastId ORDER BY Id) is the simplest way to page through them; it stays fast however deep into the list you are, where OFFSET gets slower with every page.
A semaphore of two
Chunks run in parallel, but not unboundedly. A semaphore set to 2 means two workers each process one chunk at a time. That number is a deliberate ceiling: memory stays at two chunks, the database sees two readers instead of dozens, and the push providers see a steady rate rather than bursts that invite throttling.
A simplified sketch of the loop:
var gate = new SemaphoreSlim(2); // at most two chunks in flight
var sends = new List<Task>();
await foreach (var chunk in recipients.ReadChunksAsync(notification.Id, ChunkSize, ct))
{
await gate.WaitAsync(ct);
sends.Add(Task.Run(async () =>
{
try { await SendChunkAsync(notification, chunk, ct); }
finally { gate.Release(); }
}, ct));
}
await Task.WhenAll(sends);
await RetryFailedAsync(notification, ct); // one pass, after every chunk is deliveredLook up the device, then send
An MSISDN isn't an address a push service understands. For each chunk the worker looks up the device information, which says whether it's an FCM (Android) or Huawei device, and the notification content, then sends each group through the right provider.
async Task SendChunkAsync(Notification n, IReadOnlyList<Recipient> chunk, CancellationToken ct)
{
var devices = await deviceStore.GetAsync(chunk.Select(r => r.Msisdn), ct);
foreach (var platform in devices.GroupBy(d => d.Platform)) // Fcm or Huawei
{
var result = await senders[platform.Key].SendAsync(n.Content, platform, ct);
await recipients.MarkProcessedAsync(result.Delivered, ct); // failures stay unprocessed
}
}Mark as you go, retry at the end
- Sent records are marked processed straight away, so a crashed run resumes from what's left instead of starting over, and nobody gets the message twice.
- Failures stay unprocessed and are retried once every chunk has been delivered. Retrying inline would let a handful of bad devices slow down everyone else's delivery.
Check the arithmetic
800k messages in 8 minutes is about 1,700 per second, sustained. That number is what the chunk size and the parallelism have to deliver, and it's worth working out before choosing either. When a run gets slower, time each stage: reading recipients, looking up devices, or the push providers themselves. The fix is different for each.