At DPL I built a scheduler that dispatches 700-800k push notifications per run in 6-8 minutes, to Android and Huawei (HMS) devices, in multiple languages. Sending one notification is easy. Sending hundreds of thousands without falling over, and without anyone receiving the same message twice, is the interesting part.
Start from the deadline
800k messages in 8 minutes is roughly 1,700 per second, sustained. That one number drives most of the design: batch size, how many workers run in parallel, and how much headroom is left for retries. Doing the arithmetic first saves a lot of guessing later.
Split the audience before sending
- By platform. Android and HMS devices go through different push services with different payload shapes and limits, so each gets its own path.
- By language. Resolve each recipient's language up front and render one payload per language, instead of templating every message individually.
- Into batches. Page through recipients in fixed-size chunks rather than loading the whole audience into memory.
Keep workers boring
A worker takes a batch, sends it, records the outcome, and moves on. Stateless workers are easy to scale out and safe to restart mid-run.
await Parallel.ForEachAsync(
recipientBatches, // IAsyncEnumerable<Batch>
new ParallelOptions { MaxDegreeOfParallelism = workerCount },
async (batch, ct) =>
{
var payload = templates.For(batch.Platform, batch.Language);
var result = await senders[batch.Platform].SendAsync(batch.Tokens, payload, ct);
await outcomes.RecordAsync(batch.Id, result, ct);
});Plan for partial failure
- Record progress per batch, so a crashed run can resume where it stopped instead of starting over.
- Retry transient errors with backoff, and drop tokens the push service reports as invalid so the next run is faster.
- Respect provider limits. Throttling yourself is cheaper than being throttled.
Measure the run, not just the send
The numbers that matter are end to end: total duration, messages per second over time, failure rate by platform, and time spent on retries. When run time creeps up, they show whether the bottleneck is reading recipients, rendering payloads, or the push services themselves.