In my lab, changing a container’s CPU limit from two CPUs to one almost doubled the time needed to process 200 records: 42.1 seconds became 83.6 seconds. These are medians from five runs of each configuration. The two-CPU version processed two records at once; the one-CPU version processed one. Both wrote all 200 records.
I think this kind of slowdown is easy to miss: the batch still finishes and every record gets written. Without a baseline for how long it should take, the extra waiting can look like ordinary service latency.
Each record needed three enrichment calls followed by a write to a system of record. I used this familiar pattern:
await Parallel.ForEachAsync(records, cancellationToken, async (record, ct) =>
{
await ProcessAsync(record, ct);
});Parallel.ForEachAsync without ParallelOptions allows at most Environment.ProcessorCount concurrent operations. Passing new ParallelOptions { CancellationToken = ct } leaves MaxDegreeOfParallelism at its initial value, -1. For ForEachAsync, that means ProcessorCount again with the default scheduler. Setting -1 explicitly does not remove the limit.
The cancellation-token-only options are what I’d be most likely to overlook. An explicit options object looks reassuring, but here it still leaves concurrency to a CPU-based default.
On Linux and macOS, and on Windows since .NET 6, ProcessorCount returns the smallest applicable value among the machine’s logical processor count, the process’s CPU affinity count, and its CPU utilization limit rounded up to a whole number. A one-CPU limit therefore gives 1; a half-CPU limit also gives 1. This value is fixed when the runtime starts and stays unchanged for the process’s lifetime.
Go 1.25 uses the cgroup CPU limit, rather than the CPU request, when choosing its default GOMAXPROCS; the JVM has enabled container detection by default on supported Linux platforms. That does not mean Go or Java serializes I/O loops. The concern is code that turns a CPU-derived value into an I/O concurrency limit.
A record can spend most of its lifetime waiting for other services. During an asynchronous wait, it does not need a CPU to itself. How many records you need in flight depends on how fast you want to process them and how long each one waits. Little’s law relates these averages: items in flight equal throughput multiplied by time per item.
For illustration, suppose three enrichment calls take 360 ms altogether and the write takes another 30 ms. At 100 records per second, a 0.39-second record lifetime calls for about 39 records in flight. This is an illustrative calculation, not a lab measurement. The one-CPU default permits only one.
Setting a larger MaxDegreeOfParallelism lets more records make progress while others wait. But it also allows more calls to every dependency. An enrichment API might tolerate dozens of concurrent requests while the system of record, often an older component in the chain, can process only a few.
A single limit has no way to express that difference. If the write service becomes slow, records stay in its write stage longer. More writes overlap, and the service may start queuing or rejecting requests even though the enrichment API still has capacity.
Putting one SemaphoreSlim around all outbound calls limits their combined concurrency. It still leaves the distribution uncontrolled. A slow dependency holds permits longer and can occupy much of the shared pool. Thirty permits across all calls do not guarantee that a service with capacity for six sees no more than six.
Concurrency and rate also need separate controls. A semaphore caps calls in flight; it does not enforce an API’s per-minute quota, including an LLM provider’s quota. Little’s law helps estimate the concurrency needed for a target average rate, but enforcing a time-based quota requires a rate limiter.
I want the enrichment API and the system of record to have independent limits. The outer loop can then allow enough records in flight to keep both services busy. This helper gives each dependency its own semaphore:
public sealed class DependencyBudgets(int enrichmentLimit, int recordsLimit)
{
private readonly SemaphoreSlim _enrichment = new(enrichmentLimit, enrichmentLimit);
private readonly SemaphoreSlim _records = new(recordsLimit, recordsLimit);
public Task<T> CallEnrichmentAsync<T>(Func<CancellationToken, Task<T>> call, CancellationToken ct)
=> GateAsync(_enrichment, call, ct);
public Task<T> CallRecordsAsync<T>(Func<CancellationToken, Task<T>> call, CancellationToken ct)
=> GateAsync(_records, call, ct);
private static async Task<T> GateAsync<T>(SemaphoreSlim gate, Func<CancellationToken, Task<T>> call, CancellationToken ct)
{
await gate.WaitAsync(ct);
try { return await call(ct); }
finally { gate.Release(); }
}
}This is a trimmed version of PerDependencyStrategy in the lab’s src/Lab.Runner/ConcurrencyStrategy.cs (strategy C4), which also runs the outer loop.
Each wrapper acquires the relevant permit, invokes the supplied call, and releases the permit in finally. Keep one set of budgets shared by the work it is meant to constrain. Creating a new set for each record would give each record its own allowance.
Choose the limits from the services’ capacities. Ask their owners how many simultaneous requests they can handle. If that is unknown, increase concurrency in steps in a test environment and watch latency and rejections. A service whose latency starts climbing at a particular load needs a budget that respects that capacity.
Work out how many calls need to be in flight to reach your target rate, and check whether the service can handle that many. Use each dependency’s own service time in that calculation. If the desired concurrency exceeds its capacity, reduce the throughput target; capacity divided by service time gives a rough upper bound for that dependency.
Make these limits explicit configuration and validate them at startup. Monitor calls in flight for each service so you can compare the configured allowance with actual behavior.
I don’t want a container resize to quietly change how many calls I send to another service. If that limit matters to the service, I want it written down in the configuration.
When a service slows down, a fixed concurrency budget reduces the rate at which records get through it. The budget prevents the client from responding by adding more simultaneous calls to that service. Work waits upstream instead. That is backpressure, and the growing backlog needs monitoring too.
Use separate time limits for calling the service and waiting for permission to call it. Start the service-call timeout after acquiring the permit, so local queue time does not consume it. Also give permit acquisition its own deadline. The lab starts call timeouts after acquisition, but does not implement a deadline for the permit wait. The helper above accepts a cancellation token; it does not create either timeout.
I built a small lab (code and preregistered hypotheses). Its two simulated dependencies run in a separate container without a CPU limit. The enrichment API tolerates many concurrent calls. The system of record processes six requests at once, allows six more to wait, and returns HTTP 503 beyond that. A Docker runner limited to one CPU processes 200 records, each with three enrichment calls and one write.
- C1 uses the runtime default.
- C2 allows 32 records in flight, with no dependency limiter.
- C3 allows 32 records in flight and shares 30 permits across outbound calls.
- C4 allows 32 records in flight, with 24 enrichment permits and six write permits.
C3 and C4 have the same total allowance of 30 permits. I committed the hypotheses and thresholds to the repository before the first run, then ran each combination five times. One degraded scenario raises enrichment’s median latency to 1.2 seconds, with about 5% of calls hanging for eight seconds, beyond the client’s two-second timeout. The other raises write time from 30 ms to 150 ms.
C1 followed the CPU setting: enrichment never had more than one call in flight at one CPU or more than two at two CPUs. The opening timings come from this comparison.
With writes slowed from 30 ms to 150 ms, C2, C3 and C4 delivered similar useful throughput, but the rejection counts were very different. C2 accepted 34.35 records/s with 64 HTTP 503 rejections; C3 accepted 34.01 records/s with 57; C4 accepted 35.08 records/s with none. These are medians of five runs. Accepted records/s counts only records whose writes the system of record accepted.
Across these runs, both C2 and C3 offered up to 14 concurrent writes; their median peaks were 13. C4 never offered more than six, as configured, and the system of record accepted all 200 writes in each run. C4’s median completion time was about 5.7 seconds, compared with 4.2 seconds for C3, while its useful throughput was the same or slightly higher. If I had compared only completion times, I’d have picked C2 or C3 over C4. The rejected writes change my answer.
With both services healthy, C4 was the slowest of C2, C3 and C4: 58.38 accepted records/s, versus 65.32 for C3 and 67.86 for C2. The configuration reserves six permits for writes, leaving only 24 for enrichment, where healthy records spend most of their time. C2 and C3 each received a few 503 responses in two of five healthy runs; C4 received none.
I’d accept that roughly 11–14% reduction in healthy throughput when reliable writes matter more than finishing the batch sooner. If rejected writes are cheap to recover and latency is the priority, I’d want the measurements to justify that cost.
When only enrichment slowed down, C4 achieved 0.82 times C3’s accepted throughput. C3 could allocate all 30 permits to enrichment; C4 could use only 24. About a fifth of records were incomplete in each of C2, C3 and C4, consistent with enrichment timeouts. Those records were written and marked incomplete, never complete.
These are simulated services. Their timings and capacities describe the model, not a real system. My rule of thumb is to protect the dependency I can overload first, then see how much throughput I can get within that limit. I’d raise its budget only after measurements show the service can take it.
- Inspect the default concurrency before relying on it for I/O-heavy work.
- Find each dependency’s capacity and typical service time.
- Give each dependency its own budget, with enough records in flight to use it.
- Configure limits explicitly and validate them at startup.
- Monitor calls in flight per dependency alongside CPU and memory.
- Start call timeouts after permit acquisition and set a separate permit-wait deadline.
- Repeat your measurements under the CPU limit your containers will actually have.
AI use disclosure: I used Anthropic’s Claude for research, the outline, an initial draft of this article, and most of the lab code and the script that draws the two figures, written from my specification. I specified the experiments, directed and reviewed each stage of the lab, and ran it on my machine with Claude Code. I rewrote and edited the prose, including the evaluative passages, and checked every number about the lab runs, in the text and in both figures, against the experiment output with Claude Code.
