Many developers treat Knuth's line, "Premature optimization is the root of all evil," as a license to ignore performance until something breaks. The full quote continues: "yet we should not pass up our opportunities in that critical 3%." This post is about that balance.

Scope: This article is for software with real performance requirements: production services, data pipelines, libraries, system components. It is not for one-off scripts, and not for the extreme end where hand-written assembly or compiler patches pay off.

Why Simple "Don't" Is Not Enough

"Don't optimize" warns against tuning code before you know the bottlenecks. Treated as a blanket ban, it produces systems nobody can rescue: the data model or the API contract makes fast code impossible.

Architectural decisions belong in design. Once a system ships, its data structures, memory layouts, API boundaries, and concurrency patterns are baked into every module that depends on them.

Retrofitting a different approach, say a linked list to a hash map or AoS to SoA, means rewriting every function that touches the data, updating every test, and migrating persisted state. The cost climbs with each dependent module, and the regression risk scares most teams away from trying.

Choosing the right structure during design costs no more than choosing a poor one, and it removes a whole class of problems before they spread. Performance-aware design is engineering discipline applied when change is cheapest.

Consider this C example:

// Storing user records in a linked list because "we'll optimize later"
typedef struct UserNode {
    User data;
    struct UserNode* next;
} UserNode;

UserNode* find_user(UserNode* head, uint64_t id) {
    while (head) {
        if (head->data.id == id) return head;
        head = head->next;
    }
    return NULL;
}

With 10 users, this is fine. With 10 million users in a hot path, replacing the linked list means changing every caller, every serialization path, and maybe your database schema.

The wrong data structure is a design problem, and design problems compound.

Why You Can't Ignore Optimization

Premature micro-optimization and performance-aware design are different things. Don't tune loop unrolling on day one. Do think about algorithmic complexity, memory layout, and I/O patterns from the start.

Algorithmic Complexity Is Not Optional

// O(n^2) - checking every pair
fn has_duplicate(items: &[u64]) -> bool {
    for i in 0..items.len() {
        for j in 0..i {
            if items[i] == items[j] {
                return true;
            }
        }
    }
    false
}

// O(n) - hash set with O(1) lookup
fn has_duplicate(items: &[u64]) -> bool {
    let mut seen = std::collections::HashSet::new();
    items.iter().any(|&item| !seen.insert(item))
}

With n = 100,000, the naive version does about 5 billion comparisons; the hash set does about 100,000 lookups. No amount of SIMD or cache tuning closes that gap.

Memory Layout Matters at Design Time

In C, Array of Structs (AoS) versus Struct of Arrays (SoA) is a design decision that affects every downstream function:

// Array of Structs - poor cache utilization when processing only positions
typedef struct {
    float x, y, z;   // 12 bytes
    float vx, vy, vz; // 12 bytes
    float mass;       // 4 bytes
    char name[32];    // 32 bytes - pollutes cache lines
} Particle_AoS;

// Struct of Arrays - process positions independently, cache-friendly
typedef struct {
    float* x;
    float* y;
    float* z;
    float* vx;
    float* vy;
    float* vz;
    float* mass;
    char** name;
    size_t count;
} Particles_SoA;

Retrofitting AoS to SoA in a large codebase is a multi-week refactor. Choosing SoA on day one costs almost nothing.

In Rust, soa-derive or my fork of it, layout, does this with a single derive attribute.

Profile Before You Optimize

A hot path is the code where your program spends most of its running time; a hotspot is the tightest point inside it. Measure to find them. You will guess wrong more often than right.

Tools

PlatformToolUse Case
Linuxperf, flamegraphCPU profiling, cache misses
Linuxvalgrind --tool=callgrindInstruction-level profiling
Rustcargo flamegraph, criterionBenchmarking, flame graphs
C/C++gprof, Instruments (macOS)Function-level timing
Allperf statHardware counter overview

Where to Optimize

Profiling tells you where the bottleneck is. The next question is what kind of fix it needs: a CPU-bound loop takes different treatment than a memory-hungry pipeline.

Memory Optimization

Memory pressure is common and hard to see. It shows up as heap churn, large RSS, cache thrashing, or GC pauses in managed runtimes.

Reduce allocations in hot paths. Every heap allocation must find a free block, initialize it, and later free it. In a tight loop that adds up fast.

// Bad: allocates a Vec on every call
fn process_batch(ids: &[u64]) -> Vec<u64> {
    ids.iter().filter(|&&x| x % 2 == 0).copied().collect()
}

// Better: reuse a pre-allocated buffer across calls (caller must out.clear() before calling)
fn process_batch_into(ids: &[u64], out: &mut Vec<u64>) {
    out.extend(ids.iter().filter(|&&x| x % 2 == 0).copied());
}

Pre-allocate with known capacity. When the output size is predictable, allocate once:

let mut result = Vec::with_capacity(input.len());

Prefer the stack for small, short-lived data. In C, avoid malloc for fixed-size scratch buffers:

// Bad: heap allocation for a small temporary buffer
char* buf = malloc(256);
// ... use buf ...
free(buf);

// Better: stack allocation
char buf[256];
// no free needed

In Rust, smallvec and arrayvec store elements inline on the stack up to a fixed capacity:

use smallvec::{SmallVec, smallvec};
use arrayvec::ArrayVec;

// Stored entirely on the stack if <= 8 elements
let mut v: SmallVec<[u64; 8]> = smallvec![1, 2, 3];

// Fixed capacity, never heap-allocates - panics or errors if capacity exceeded
let mut v: ArrayVec<u64, 8> = ArrayVec::new();
v.push(1);

Warning: SmallVec and ArrayVec are expensive to move. A move copies the entire inline buffer, used or not. Keep them in place: store them in a local or a struct field and pass them by reference. Avoid returning them by value or holding them in collections that relocate elements often.

Use memory pools for repeated same-size allocations. A slab or arena allocator removes per-object allocator overhead.

Minimize struct size. Smaller structs mean more fit in cache. Reorder fields to remove padding:

// 24 bytes due to padding
struct Bad {
    char a;      // 1 byte + 7 padding
    double b;    // 8 bytes
    char c;      // 1 byte + 7 padding
};

// 16 bytes due to 6 bytes of padding at the end
struct Good {
    double b;    // 8 bytes
    char a;      // 1 byte
    char c;      // 1 byte
};

Memory Alignment and Cache Lines

A value is naturally aligned when its address is a multiple of its size: a 4-byte uint32_t at an address divisible by 4, an 8-byte double at an address divisible by 8. Alignment matters because CPUs fetch whole cache lines (typically 64 bytes), not single bytes. Data that straddles a boundary needs two reads stitched together.

Unaligned Access Penalties

On x86-64, unaligned access works but is not free. A load that crosses a cache line costs two fetches; one that crosses a page may cost two TLB lookups. On ARM it depends on the core: older ones fault, modern ones take a penalty similar to x86.

// Unaligned: struct packed to 9 bytes, the double starts at offset 1
struct __attribute__((packed)) Unaligned {
    char flag;     // offset 0
    double value;  // offset 1 - NOT 8-byte aligned
};

// Aligned: compiler inserts padding automatically
struct Aligned {
    char flag;     // offset 0
    // 7 bytes padding inserted by compiler
    double value;  // offset 8 - naturally aligned
};

Iterate over an array of the packed struct and every value access risks a split load. In a hot loop the extra fetches add up. The aligned version wastes bytes per element and skips the penalty.

Alignment for SIMD

SIMD has the strictest requirements. Many SSE/AVX operations need 16-byte or 32-byte aligned data; violating that costs extra cycles or a hardware fault, depending on the instruction.

#include <immintrin.h>

// Bad: malloc guarantees only 8-byte (or 16-byte) alignment
float* data = malloc(count * sizeof(float));

// Good: aligned_alloc guarantees 32-byte alignment for AVX
float* data = aligned_alloc(32, count * sizeof(float));

// Now safe to use AVX 256-bit loads
__m256 vec = _mm256_load_ps(data);  // requires 32-byte alignment
// _mm256_loadu_ps works with unaligned data but may be slower on older CPUs

In Rust, the allocator already aligns every type to its natural alignment. For stricter alignment, use #[repr(align(N))]:

// Ensure a buffer is 32-byte aligned for SIMD operations
#[repr(C, align(32))]
struct AlignedBuffer {
    data: [f32; 256],
}

// Generic alignment helper
#[repr(align(64))]
struct CacheLineAligned<T>(pub T);

// Each counter sits on its own cache line - prevents false sharing (see below)
let counter_a = CacheLineAligned(0u64);
let counter_b = CacheLineAligned(0u64);
False Sharing

When two independent variables share a cache line, a write from one core invalidates that line in every other core's cache. Threads that never touch the same variable still fight over the same line, and the coherence traffic wrecks multi-threaded scaling.

// Bad: both counters likely share a cache line (64 bytes)
struct Counters {
    long a;  // updated by thread 1
    long b;  // updated by thread 2
};

// Good: each counter occupies its own cache line
struct alignas(64) AlignedCounter {
    long value;
    char padding[56];  // pad to fill 64 bytes
};

struct CountersFixed {
    struct AlignedCounter a;  // thread 1
    struct AlignedCounter b;  // thread 2
};

In Rust, use the CacheLineAligned wrapper above or CachePadded from crossbeam-utils.

The takeaway: let the compiler handle alignment, which it does unless you override it with packed. Pay attention in two places: buffers for SIMD, and data shared between threads.

Custom and Arena Allocators

General-purpose allocators balance fragmentation, latency, and overhead for any workload. When you know your access pattern, a custom allocator beats them.

Arena allocators (bump allocators) hand out memory from one large block. Each allocation is a pointer bump, and neighboring allocations sit together in cache. Freeing is one pointer reset: no fragmentation, no per-object free. Use arenas when a group of objects shares one lifetime, like an AST you discard after handling the request. In Rust, use the mature bumpalo crate.

Memory Pools and Object Recycling

Memory pools fit workloads that allocate and free the same size over and over. Freed objects go on a free list instead of back to the allocator; the next allocation pops one off.

Object recycling applies the same idea one level up: keep initialized objects (database connections, threads, parsed structures) ready for reuse and save the initialization cost too. In Rust, my opool crate reuses, recycles, and clears objects for you.

CPU Optimization

CPU optimization has three levels: algorithmic complexity, cache utilization, and low-level hardware behavior. Address them in that order. Algorithmic wins dwarf micro-optimizations.

Better Algorithms First

No amount of tuning saves an O(n²) algorithm at scale. Do not optimize the inner loop of an O(n^2) algorithm; replace the algorithm. Check the complexity of your hot path before touching anything else.

OperationNaiveBetter
Lookup in unsorted listO(n)O(1) with hash map
SortingO(n²) bubble sortO(n log n) introsort
Graph shortest pathO(V³) Floyd-WarshallO(E log V) Dijkstra
String dedupO(n²) nested scanO(n) with hash set

Cache Utilization

CPUs are fast; memory is slow. A cache miss can stall the core for 100-300 cycles, which makes cache-friendly code the largest win after the algorithm itself.

Access Memory Sequentially

CPUs prefetch cache lines ahead of sequential access. Random access defeats prefetching.

// Cache-friendly: sequential row-major access
for (int i = 0; i < N; i++)
    for (int j = 0; j < N; j++)
        sum += matrix[i][j];  // sequential

// Cache-hostile: column-major access on a row-major array
for (int j = 0; j < N; j++)
    for (int i = 0; i < N; i++)
        sum += matrix[i][j];  // strides across cache lines
Keep Hot Data Together

Use SoA layouts (shown earlier) so processing one field does not drag unrelated fields into cache.

Avoid Pointer Chasing

Linked lists, trees, and graphs follow pointers to random addresses, and each hop is a potential cache miss. That makes pointer chasing one of the worst access patterns for a cache. Replace these structures with flat arrays or index-based layouts where you can.

// Pointer-chasing tree traversal - random memory access
struct Node {
    value: u64,
    left: Option<Box<Node>>,
    right: Option<Box<Node>>,
}

// Cache-friendly flat representation (binary heap or arena-indexed tree)
struct FlatTree {
    values: Vec<u64>, // children of node i are at 2i+1 and 2i+2
}

Low-Level CPU Optimization

Apply these only after the algorithm and cache behavior are fixed, and only in measured hot paths.

Eliminate branch mispredictions. An unpredictable branch stalls the pipeline. When both sides are cheap, compute both and select. The compiler converts trivial cases like max(a, b) on its own; it cannot convert an if/else with two distinct computation paths:

// Branchy - two distinct paths, ~50/50 unpredictable
// The compiler cannot auto-convert this; it emits a real branch
unsigned collatz_sum(const unsigned* data, size_t n) {
    unsigned sum = 0;
    for (size_t i = 0; i < n; i++) {
        unsigned v = data[i];
        if (v & 1) {               // odd/even: unpredictable
            v = v * 3 + 1;         // 3n+1 step
        } else {
            v = v >> 1;            // halve step
        }
        sum += v;
    }
    return sum;
}

// Branchless - compute both paths, select with mask
unsigned collatz_sum_branchless(const unsigned* data, size_t n) {
    unsigned sum = 0;
    for (size_t i = 0; i < n; i++) {
        unsigned v = data[i];
        unsigned odd_path  = v * 3 + 1;
        unsigned even_path = v >> 1;
        unsigned mask = -(v & 1u); // all-ones if odd, 0 if even
        sum += (odd_path & mask) | (even_path & ~mask);
    }
    return sum;
}

Enable and guide auto-vectorization. Compilers emit SIMD on their own when loops are simple and data does not alias. Use plain index-based loops and restrict in C:

void add_arrays(float* restrict a, const float* restrict b, size_t n) {
    for (size_t i = 0; i < n; i++)
        a[i] += b[i];  // compiler can safely vectorize with restrict
}

In Rust, use iterators or explicit slice patterns so the compiler can drop bounds checks and vectorize.

Reach for intrinsics or SIMD libraries last, when auto-vectorization fails on a confirmed hot path.

Compiler Hints: likely, unlikely, assume, and Prefetch

Compilers accept hints about runtime behavior: branch probability (likely/unlikely), value assumptions (assume), and manual cache prefetch. These are the most misused tools in performance work. A correct hint removes a mispredicted branch or hides memory latency. A wrong one, the common case, blocks better optimizations, poisons code layout, or evicts hot data.

likely and unlikely: Almost Never Use Them

__builtin_expect in GCC/Clang, and std::intrinsics::likely/unlikely in Rust, tell the compiler which side of a branch to expect so it can lay out the hot path contiguously.

// GCC/Clang: hint that the condition is almost always true
#define likely(x)   __builtin_expect(!!(x), 1)
#define unlikely(x) __builtin_expect(!!(x), 0)

int process(int* data, size_t n) {
    for (size_t i = 0; i < n; i++) {
        if (unlikely(data[i] < 0)) {
            handle_error(data[i]);
            return -1;
        }
        process_normal(data[i]);
    }
    return 0;
}

// Rust nightly only - std::intrinsics::likely is unstable
// The idiomatic stable alternative is to restructure code so the hot path
// is the fall-through branch, which the compiler already prefers.
#![feature(core_intrinsics)]
use std::intrinsics::{likely, unlikely};

fn process(data: &[i32]) -> i32 {
    for &x in data {
        if unlikely(x < 0) {
            return -1;
        }
        process_normal(x);
    }
    0
}
Why to avoid them

  1. Hardware predictors learn. The CPU watches each branch over thousands of executions and models it better than any static hint. Your hint is fixed; the predictor adapts.

  2. Your benchmark is not production. A branch almost never taken in testing may fire 40% of the time on real data. A wrong hint misguides both code layout and the initial prediction, and mispredictions get more expensive.

  3. Hints constrain the optimizer. unlikely pushes that code into a cold section. If the assumption fails, you pay an unconditional jump plus an instruction cache miss, worse than untouched code.

  4. PGO does the same job with real data. Profile-guided optimization measures branch frequencies from real workloads and applies layout across the whole program, not the handful of branches you noticed. Use it instead.

The rare exception: error branches in hot loops that are structurally almost never taken, when PGO is impractical. Measure before and after even then.

assume: Dangerous

__builtin_unreachable() in C and std::hint::unreachable_unchecked() in Rust mark a path impossible. __builtin_assume(cond) in Clang treats cond as true without evaluating it. They can remove bounds checks and dead branches, and they are undefined behavior when violated.

// Tell the compiler n is always a multiple of 8 - enables vectorization
// of the loop without a scalar remainder epilogue
void process(float* data, size_t n) {
    __builtin_assume(n % 8 == 0);
    for (size_t i = 0; i < n; i++)
        data[i] *= 2.0f;
}

// Eliminate a bounds check the compiler cannot prove away on its own
unsafe fn get_unchecked_hint(data: &[u32], i: usize) -> u32 {
    // Caller guarantees i < data.len(); make that visible to the optimizer
    std::hint::assert_unchecked(i < data.len());
    *data.get_unchecked(i)
}

Why assume is dangerous:

  • If the assumption is ever false, the compiler may delete surrounding code or invert branches. There is no runtime check, no panic, no error.
  • Assumptions are invisible to maintainers. A refactor that breaks the invariant compiles clean and miscompiles silently.
  • Like likely, a wrong assumption narrows the input space the optimizer reasons about and blocks transformations it would otherwise apply.

Use assume only when a checked entry point enforces the invariant, the gain is measured, and a comment names the enforcing mechanism.

Software Prefetch with __builtin_prefetch

__builtin_prefetch in GCC/Clang tells the CPU to start loading a cache line before you need it, hiding memory latency behind computation.

// Prefetch for read (rw=0), high temporal locality (locality=3)
__builtin_prefetch(ptr, 0, 3);

// Prefetch for write (rw=1), no temporal locality - evict after use (locality=0)
__builtin_prefetch(ptr, 1, 0);

In Rust, std::arch exposes the x86 prefetch instructions:

#[cfg(target_arch = "x86_64")]
use std::arch::x86_64::{_mm_prefetch, _MM_HINT_T0, _MM_HINT_NTA};

unsafe fn prefetch_read(ptr: *const u8) {
    _mm_prefetch(ptr as *const i8, _MM_HINT_T0);  // prefetch into all cache levels
}

unsafe fn prefetch_write_no_temporal(ptr: *const u8) {
    _mm_prefetch(ptr as *const i8, _MM_HINT_NTA); // non-temporal: bypass L2/L3
}

The locality parameter picks the target cache levels:

ValueMeaning
0 / _MM_HINT_NTANon-temporal: minimize cache pollution
1 / _MM_HINT_T2Load into L3 and higher
2 / _MM_HINT_T1Load into L2 and higher
3 / _MM_HINT_T0Load into all cache levels

Why wrong prefetch is worse than none:

  1. Prefetching data you won't use evicts data you will. A wrong address, a prefetch issued too far ahead, or a branch not taken spends cache space and bandwidth for nothing, and may push out a hot line.

  2. Write prefetch is riskier. It can trigger a Read For Ownership request and coherence traffic across cores. If the write never happens, or other cores were using the line, you stalled them for nothing.

  3. Non-temporal stores bypass the cache. _MM_HINT_NTA and non-temporal stores (movntps, movntdq) write straight to memory without filling L1/L2/L3. Right for streaming writes you never read back. For data you read again soon, the next access is a full miss from main memory.

// Correct: streaming copy, prefetch one chunk ahead, output never read back
void copy_streaming(float* restrict dst, const float* restrict src, size_t n) {
    for (size_t i = 0; i + 16 <= n; i += 16) {
        __builtin_prefetch(src + i + 16, 0, 0); // next chunk: read, no reuse
        for (size_t j = i; j < i + 16; j++)
            dst[j] = src[j];
    }
    // remainder loop omitted
}

// Wrong: locality=0 for data the loop touches again
void wrong_prefetch(float* data, size_t n) {
    for (size_t i = 0; i < n; i++) {
        __builtin_prefetch(data + i + 8, 0, 0); // hints "no reuse", evicted early
        // the loop needs that line again 8 iterations later - full miss
        process(data[i]);
    }
}
  1. Prefetch distance must match your loop. One element ahead in a 2-cycle loop does nothing; 512 ahead wastes cache. The right distance depends on memory latency (200-400 cycles), loop throughput, and stride, and must be tuned per microarchitecture.

The practical rule: every modern CPU already prefetches sequential and strided patterns in hardware, and you will rarely beat it there. Software prefetch pays off only for:

  • Irregular or pointer-chasing access the hardware cannot predict.
  • Strides beyond the hardware prefetcher's detection range.
  • Cases where you know the next address far in advance, like a pointer held in the current node.

Anywhere else it adds noise, burns issue slots, and risks evictions. Measure with perf stat -e cache-misses,cache-references before and after. If the miss rate does not improve, remove the prefetch.

Where O(n) Beats O(1)

Big-O describes growth, not constants. For small fixed N, a linear scan over an array beats a hash map lookup, O(1) or not.

Why the hash map loses at small N:

  1. Hashing costs cycles. Computing the hash and reducing it to a bucket index takes dozens of cycles. Scanning 8 elements takes 8 to 16.

  2. Indirection costs cache misses. A HashMap lookup dereferences at least two pointers, each a potential miss. An array packed into one 64-byte cache line is a single L1 hit, about 4 cycles.

  3. Empty slots waste cache. A hash table keeps its load factor near 75%, so 16 entries spread over 3-4 cache lines with metadata. A dense array of 16 u32 keys fills exactly one.

Practical rule: if N is a compile-time constant up to about 64 items and the data fits in one or two cache lines, a flat array with linear search wins. Even more so on cold paths, where the hash table gets no warm-cache advantage.

// HashMap lookup: O(1) average but expensive per operation
use std::collections::HashMap;

fn lookup_hm(map: &HashMap<u32, u32>, key: u32) -> Option<u32> {
    // hash the key, pick the bucket, probe, compare
    // ~30-50 cycles even for a hit, plus a potential cache miss
    map.get(&key).copied()
}

// Linear scan: O(n) but cheap for small, cache-resident arrays
fn lookup_linear(keys: &[u32], values: &[u32], key: u32) -> Option<u32> {
    // 16 u32 keys fit in one cache line: sequential access, no hashing
    // ~4-8 cycles when the data is in L1
    keys.iter().position(|&k| k == key).map(|i| values[i])
}

The compiler can unroll a fixed-size linear scan, vectorize the comparisons, or replace it with conditional moves. A hash map lookup is an opaque library call it cannot touch.

Measure both, but for small fixed-size lookups, try the flat array first.

Eliminate Redundant Work

A large class of slow code does the same work more than once.

Remove Repeated Computation

// strlen called on every iteration - O(n^2) total
for (size_t i = 0; i < strlen(str); i++) { ... }

// Computed once - O(n) total
size_t len = strlen(str);
for (size_t i = 0; i < len; i++) { ... }

Cache Pure Results

A function with no side effects returns the same output for the same input, so you can cache (memoize) the result. This pays off for expensive computations called repeatedly with the same arguments.

For dense integer keys, a Vec beats a HashMap. Here 0 serves as the "not cached yet" sentinel, since every Fibonacci result for n > 1 is nonzero:

fn fib(n: usize, memo: &mut Vec<u64>) -> u64 {
    if n <= 1 {
        return n as u64;
    }

    let cached = *memo.get(n).unwrap_or(&0);
    if cached != 0 {
        return cached;
    }

    let result = fib(n - 1, memo) + fib(n - 2, memo);

    if n >= memo.len() {
        memo.resize(n + 1, 0);
    }
    memo[n] = result;

    result
}

Idempotent operations, where applying twice equals applying once, allow a similar trick: coalesce duplicate events before processing instead of handling each copy.

Don't Cache Cheap Results

Cache only what is expensive. A cached result is a memory read: an L1 hit costs a few cycles, a miss costs hundreds, while a few arithmetic operations cost one or two in registers. For cheap functions the lookup is slower than the work, and the table evicts hotter data. Recompute cheap values.

Popcount tables are the classic case. Before CPUs had a dedicated instruction, a 256-entry table was the fast way to count bits; today the instruction wins and the table only pollutes the cache:

// Bad: lookup table for a value the CPU computes in one instruction
static POPCOUNT: [u8; 256] = precomputed_table();

fn count_bits(x: u8) -> u32 {
    POPCOUNT[x as usize] as u32  // memory read, potential cache miss
}

// Better: compiles to a single popcnt instruction
fn count_bits(x: u8) -> u32 {
    x.count_ones()  // registers only, no memory traffic
}

Lazy Evaluation

Don't compute what you may not need. Defer expensive values that are only sometimes used, like error messages in hot paths:

// Eager: format! allocates a string on every call, even if the item is valid
let result = parse_item(data)
    .unwrap_or(format!("Failed to parse complex object ID: {}", data.id));

// Lazy: the closure and format! run only if parsing fails
let result = parse_item(data)
    .unwrap_or_else(|_| format!("Failed to parse complex object ID: {}", data.id));

Batch Work

Pay per-batch overhead instead of per-item overhead:

// N individual inserts - N round trips to the database
for record in records {
    db.insert(record)?;
}

// One batch insert - one round trip
db.insert_batch(&records)?;

Single-Pass Over Multiple Passes

Every extra pass over the data costs memory bandwidth. When several transformations can run in one scan, combine them.

// Three passes, three allocations - BAD
let step1: Vec<u64> = data.iter().filter(|&&x| x > 0).copied().collect();
let step2: Vec<u64> = step1.iter().map(|&x| x * 2).collect();
let step3: Vec<u64> = step2.iter().filter(|&&x| x < 1000).copied().collect();

// One pass, one allocation - Rust iterators are lazy and fused - GOOD
let result: Vec<u64> = data.iter()
    .filter(|&&x| x > 0)
    .map(|&x| x * 2)
    .filter(|&x| x < 1000)
    .collect();

In C, fuse loops that scan the same array for different purposes:

// Two passes - BAD
for (int i = 0; i < n; i++) sum += a[i];
for (int i = 0; i < n; i++) max = a[i] > max ? a[i] : max;

// One pass - GOOD
for (int i = 0; i < n; i++) {
    sum += a[i];
    if (a[i] > max) max = a[i];
}

Fewer Lines, Not Always Faster

Compilers optimize simple, straightforward code well. Fewer lines usually means less indirection, fewer branches, and more inlining. But fewer lines and faster code are different things, and chained high-level operations are the classic trap.

The Chained Replace Problem

Cleaning up a string by removing several substrings:

// Concise - but allocates a new String on every .replace() call
fn normalize(s: &str) -> String {
    s.replace("  ", " ")
     .replace("\t", " ")
     .replace("\r\n", "\n")
     .replace("\r", "\n")
}

Four lines that read well. Also four heap allocations and four full passes. For a 1MB input you allocate and scan megabytes.

// More lines, one pass, one allocation
fn normalize_fast(s: &str) -> String {
    let mut result = String::with_capacity(s.len());
    let mut chars = s.chars().peekable();
    while let Some(c) = chars.next() {
        match c {
            '\r' => {
                if chars.peek() == Some(&'\n') {
                    chars.next();
                }
                result.push('\n');
            }
            '\t' => result.push(' '),
            ' ' if chars.peek() == Some(&' ') => {}
            c => result.push(c),
        }
    }
    result
}

More lines: one allocation, one pass, no intermediate strings.

The Compiler Cannot Fix Algorithmic Waste

The compiler inlines, vectorizes, and removes dead stores. It cannot remove an allocation your API forces, and it cannot merge passes hidden inside library calls. Each .replace() returns an owned String the next call consumes; merging them would change semantics.

The rule: prefer fewer lines when the operations are equivalent at the machine level. Prefer explicit control when chaining hides repeated work.

Simple Code Ages Better

Hand-optimized code rots. Compilers improve, and the trick you wrote in 2015 to work around a missed optimization may now block a better one.

The Compiler Is a Moving Target

Bit tricks, manual unrolling, and register hints target the compiler you have today, not the one your users run in three years. GCC, Clang, MSVC, and rustc evolve independently: what GCC 9 needed may hurt on GCC 14 and produce different codegen on Clang.

// "Optimized" bit trick for checking power of two - written to avoid a branch
// that an older compiler wouldn't eliminate
int is_power_of_two(unsigned int n) {
    return n && !(n & (n - 1));
}

// Simple version - modern compilers generate identical or better output
int is_power_of_two_simple(unsigned int n) {
    if (n == 0) return 0;
    return (n & (n - 1)) == 0;
}

The bit trick mattered when compilers emitted branchy code for the simple form. Modern compilers produce the same instructions for both, and the simple form gives the optimizer more room to inline and combine with surrounding code.

Hand-Optimization Hurts Portability

Each compiler's vectorizer recognizes different patterns. A loop restructured to force vectorization on one compiler can block auto-vectorization on another. This bites hardest with:

  • Manual loop unrolling. Compilers have per-target unroll heuristics; explicit unrolling defeats them.
  • Explicit SIMD for simple reductions. Auto-vectorization handles sum, min, max, and dot products on all major compilers.
  • Bit tricks replacing arithmetic. Compilers already turn x / 2 into x >> 1; the explicit shift adds noise.

Future Optimizations Reward Simple Code

Optimization passes match patterns in plain code. Obscure the pattern and passes written years from now cannot fire. Real cases:

  • Compilers recognize a plain for loop that copies or fills a buffer and replace it with memcpy/memset or SIMD, so the loop and the library call cost the same. Loops "optimized" to copy two elements at a time are not recognized.
  • Vectorizers want clean induction variables and no aliasing. Loops restructured for an old compiler often carry exactly the aliasing that blocks a new one.
  • LTO and PGO work across function boundaries. Inline assembly and manual register tricks are opaque to them.

The Practical Rule

Write the clearest implementation first. Replace it only when:

  1. Profiling shows a genuine bottleneck.
  2. You measured the complex version faster on your toolchain and target.
  3. You documented why the simple version fell short.

Keep the simple version as a reference and a fallback. Compilers improve faster than codebases get refactored; the simple version may win again at your next compiler upgrade.

Establish Performance Budgets Early

Define the numbers before you write production code:

  • Latency: "p99 response time under 10ms."
  • Throughput: "1 million events per second."
  • Memory: "no more than 64MB RSS."
  • Target hardware: "meets its targets on a 2-core, 4GB VM, not just a 16-core workstation." Name the CPU architecture, cache sizes, memory bandwidth, and storage class of production. A benchmark that passes on NVMe may fail on network storage; code that vectorizes on AVX-512 may regress on ARM.

Enforce the budgets with benchmarks in CI: a commit that regresses fails the build like a failing test. In Rust, criterion with critcmp detects regressions across commits.

Don't Die by a Thousand Cuts

Small inefficiencies, each too minor to fix on its own, accumulate in hundreds of places until the total cost is severe.

The Compound Regression Problem

Regressions compound multiplicatively. A steady 1% loss per commit leaves you at 0.99^N of the original performance after N commits:

CommitsRemaining performance
10~90%
50~61%
70~50%
139~25%

No single commit looks like the problem; the total is catastrophic. Enforce budgets per commit so each regression is caught and justified when it lands, not discovered six months later.

Use LTO and PGO

Before any manual micro-optimization, take the free compiler wins. Link-Time Optimization and Profile-Guided Optimization can deliver double-digit improvements with zero code changes.

Link-Time Optimization (LTO)

Compilers normally optimize one translation unit (or crate) at a time. LTO delays optimization to link time so the compiler sees the whole program: cross-crate inlining, dead code elimination, better constant folding.

In Rust, enable it in Cargo.toml:

[profile.release]
lto = "fat"       # Enable full LTO across the dependency graph
codegen-units = 1 # Maximize optimization at the cost of slower compilation
panic = "abort"   # Remove panic unwinding overhead (if unwinding isn't needed)
strip = true      # Strip symbols to reduce binary size
opt-level = 3     # Maximum optimization (default for release)

For C/C++ (GCC/Clang), pass -flto at both compile and link time:

gcc -O3 -flto -o my_app main.c lib.c

Profile-Guided Optimization (PGO)

PGO replaces manual hints with measurement. You compile an instrumented build, run it on representative data, and recompile with the recorded profile.

The compiler uses the profile to:

  • Pack hot basic blocks together for the instruction cache.
  • Move cold error paths away from hot code.
  • Tell the inliner and vectorizer which loops and calls are worth the binary size.

In C/C++:

# 1. Compile with instrumentation
gcc -O3 -fprofile-generate -o app_instrumented source.c

# 2. Run your representative benchmark
./app_instrumented sample_workload.dat

# 3. Recompile with the recorded profile
gcc -O3 -fprofile-use -o app_optimized source.c

In Rust, the cargo-pgo plugin automates instrumentation and profile collection.

What This Article Doesn't Cover

Concurrency and parallelism: multi-threading, mutex contention, lock-free algorithms, CSP, work distribution. Those deserve their own post, and I will write one. Everything here applies no matter how many cores you add.

The Optimization Rules, Summarized

RuleMeaning
Don't optimize prematurelyDon't tune what you haven't measured.
Do design for performancePick the right algorithms and data structures from the start.
Profile before you optimizeMeasure where time goes; don't guess.
Optimize memory firstReduce allocations, pre-allocate, shrink structs, prefer the stack.
Optimize CPU in orderAlgorithm, then cache, then hardware. Never skip ahead.
Eliminate redundant workCache pure results, batch operations, evaluate lazily.
Prefer single-pass pipelinesFuse loops and transformations to save memory bandwidth.
Avoid compiler hintslikely/unlikely/assume/prefetch are wrong more often than right. Use PGO.
Benchmark every optimizationVerify the win is real and nothing else regressed.
Enforce budgets in CITreat performance regressions like test failures.
Watch for accumulated cutsPatterns that are cheap once are expensive in hundreds of places.

Performance is a property you preserve, commit by commit, for the life of the project.