The idea in one minute. Good performance work starts with a workload and a bottleneck. Reducing allocations, memory traffic, redundant work, or contention often matters more than shortening an expression. A useful optimization preserves behavior, improves an end-to-end metric, and has a measured cost in complexity and maintenance.
Reading path. Start with Amdahl’s law and the measurement protocol. Then use the memory, allocation, and concurrency sections as a hypothesis checklist. The numerical examples are explicit models, not measurements of a particular machine.
1. Decide which metric matters
Latency, throughput, CPU time, peak memory, allocation count, and energy are different outcomes. A change can improve throughput while increasing tail latency through batching. A cache can lower average compute while increasing memory and invalidation complexity.
Specify a representative workload: input sizes and distributions, hot and cold cases, thread count, hardware, compiler, flags, and acceptable behavior. Distinguish a microbenchmark of one operation from an application benchmark. The Dean–Ghemawat Performance Hints offers useful engineering ideas; the derivations below turn several of them into measurable bounds and tradeoffs.
2. Amdahl’s law limits local wins
Suppose a fraction \(f\) of baseline execution time is spent in a component. Speeding up only that component by factor \(s\) gives an ideal total speedup
\[S=\frac1{(1-f)+f/s}.\]Derivation. Normalize original time to one. The unaffected portion stays \(1-f\) and the improved portion becomes \(f/s\). The inverse of the new time is speedup. This model assumes the workload, interactions, and other costs remain unchanged.
If \(f=0.2\) and \(s=5\), total speedup is \(1/(0.8+0.04)\approx1.19\), not \(5\). Even removing the component entirely gives at most \(1/(1-f)=1.25\). This explains why a dramatic microbenchmark can produce a small user-visible gain.
The formula is a planning tool, not a substitute for profiling. Removing an allocation might also reduce cache misses elsewhere, changing more than one component; batching can add work not present before. Measure the new total rather than enforcing the model mechanically.
3. A reproducible measurement protocol
Build equivalent programs with the same optimization mode and dependencies. Consume outputs so the compiler cannot discard the work. Separate initialization from repeated operation if the deployment does, and include initialization if users pay it. Measure warm and cold states independently.
For repeated times \(T_i\), report a distribution rather than only the smallest sample. Alternate or randomize baseline and candidate trials to reduce drift from temperature, background load, and CPU frequency. A paired difference \(D_i=T_i^{\mathrm{candidate}}-T_i^{\mathrm{baseline}}\) can remove shared conditions, though autocorrelation still requires care.
Use a CPU profiler to locate work, allocation profiling for churn, and hardware counters for memory and branch behavior. A sample attributed to a function is not proof that one line caused the cost. Change one causal hypothesis, remeasure, and inspect correctness on boundary cases before interpreting speed.
Performance claims should name the exact binary, input, host, repetitions, aggregation, and uncertainty. “Faster in release mode” has little meaning without a baseline also compiled in release mode.
4. Memory traffic and locality
Let \(F\) be required operations and \(B\) transferred bytes. If \(P\) and \(W\) are upper bounds on the corresponding processing and bandwidth rates, the accounting model gives
\[T\ge\max\left(\frac{F}{P},\frac{B}{W}\right),\]The bound follows separately from \(F\le PT\) and \(B\le WT\). This roofline-style model separates compute-heavy from bandwidth-heavy hypotheses. Actual time can be higher because of dependency stalls, cache hierarchy effects, and contention. With measured effective rates instead of guaranteed upper bounds, treat the expression as an estimate.
If a scan transfers \(B=1\) GB at an effective \(W=50\) GB/s, the bandwidth term is \(20\) ms. These are assumed values. Changing an arithmetic operation will not reduce this lower bound; reducing bytes transferred might.
For a contiguous array of \(n\) records with stride \(r\) bytes and cache-line size \(L\), a full scan touches roughly \(\lceil nr/L\rceil\) lines when accesses are dense. If only one field is used, a structure-of-arrays representation may reduce traffic. Whether it wins depends on which fields are used together, SIMD behavior, prefetching, and layout overhead.
Pointer-based structures add indirection and can scatter accesses. Indices into contiguous storage sometimes reduce pointer width and improve locality, but index bounds, handle lifetime, and relocation semantics become part of the API. A representation change should preserve these contracts.
5. Allocation and object construction
reserve(n) reserves vector capacity without creating \(n\) elements; resize(n) changes size and constructs new elements. Choose by semantics, not a rule that one is always faster.
std::vector<Result> out;
out.reserve(items.size());
for (const auto& item : items) {
out.push_back(compute(item));
}
With geometric capacity growth, appending has amortized constant reallocation work per element, but individual reallocations can still cause a latency spike. Reserve when a useful bound is known. Reserving repeatedly to exactly size()+1 can destroy geometric growth and introduce quadratic copying behavior.
Reusing a string or scratch vector can retain capacity and avoid repeated allocation:
std::string buffer;
for (const auto& item : items) {
buffer.clear();
encode_into(item, buffer);
consume(buffer);
}
This is safe only if consume does not retain a view after the next mutation. A std::string_view does not own storage; lifetime and invalidation errors are correctness failures regardless of benchmark speed. Similar care applies to iterators and references into growing vectors. C++ Core Guidelines.
6. Fast paths, caching, and avoiding work
A fast path should rely on a maintained invariant. If a flag says there is no pending work, every producer must update that flag correctly. Skipping a scan using a stale flag loses results. The performance technique creates an additional state consistency obligation.
Let cache hit rate be \(h\), hit cost \(c_h\), and miss cost including recomputation \(c_m\). Ignoring invalidation and memory effects, expected cost is
\[\mathbb E[C]=hc_h+(1-h)c_m.\]A useful cache also pays for hashing, keys, eviction, memory, and invalidation. Keys must include every input that affects the output, including configuration or data revision. Cached answers after a dependency changes can be fast and wrong.
Precompute only when reuse pays for storage and construction. For a sorted vector, binary search has \(O(\log n)\) comparisons and dense storage. A hash table has expected \(O(1)\) lookup under assumptions about hashing and load, with different locality, memory, and adversarial behavior. Asymptotic labels alone do not choose the best representation.
7. Concurrency and queueing
Adding threads helps only when useful independent work exceeds synchronization and scheduling costs. False sharing occurs when threads modify distinct variables located on the same cache line. Padding may help in a measured case but increases footprint and can hurt locality elsewhere.
Little’s law gives
\[L=\lambda W\]for a stable system with consistent averages, where \(L\) is average work in the system, \(\lambda\) arrival rate, and \(W\) average residence time. It does not by itself identify the source of delay. As utilization approaches capacity, queueing can dominate latency even if service time improves slightly.
For an idealized M/M/1 queue with Poisson arrivals, exponential service, and \(\lambda<\mu\), mean residence time is \(1/(\mu-\lambda)\). Real applications often violate those assumptions. The model illustrates sensitivity near saturation; it is not a production tail-latency prediction.
Batching amortizes fixed overhead but adds waiting. Benchmark at realistic offered load and report both p50 and p95/p99 when the sample budget supports them. A closed-loop benchmark that waits for each response may underrepresent queue buildup under external arrivals.
8. API choices and correctness costs
Bulk APIs can avoid repeated parsing, validation, and synchronization. They also change cancellation, partial failure, and ownership behavior. A shared buffer reduces copying only if callers understand how long it is valid. A move transfers ownership; it is not automatically free for every type or operation.
Before adopting a performance change, identify: preserved observable results, ordering guarantees, exception or error behavior, ownership lifetime, and concurrent access rules. Measure regressions on small inputs and infrequent paths as well as the hot case. A large gain with a subtle invalidation bug is not a completed optimization.
9. A practical decision record
Record the bottleneck hypothesis, baseline profile, change, representative results, uncertainty, and maintenance cost. Keep the smallest clear change that meets the workload’s goal. If an optimization changes only a tiny portion of total time, explain that limitation rather than presenting the microbenchmark ratio as application speedup.
Current tool versions and hardware rates can change. Recheck profiler and compiler documentation for the environment, and measure local constants instead of reusing famous latency tables as precise facts.
References
- Dean and Ghemawat, Performance Hints.
- C++ Core Guidelines.
- Google Benchmark: tooling for repeatable microbenchmarks.
- Abseil performance notes: current engineering discussions and measurement caveats.