Summary

A race condition in the buffer pool management causes data corruption when flashing large compressed images (typically 3.2GB+) using gzip and xz formats. The issue does not affect zip files due to their different decompression architecture.

Problem Description

• Data corruption during flashing of large compressed images
• Size-dependent: Affects large images (~3.2GB+) but not smaller ones
• Format-specific: Occurs with .gz and .xz files but not .zip files
• Intermittent: Race condition makes it difficult to reproduce consistently
• High severity: Results in corrupted flashed images that fail verification

How to Reproduce

# Create a large test file (3.5GB)
dd if=/dev/urandom of=/tmp/test-image.img bs=1M count=3500

# Compress with xz
xz /tmp/test-image.img

# Attempt to verify - will show data corruption on unfixed versions
TS_NODE_CACHE=false npx ts-node examples/multi-destination.ts /tmp/test-image.img.xz /dev/sda --verify

Note: The bug may be intermittent due to race conditions, but I can reproduce it everytime on my machine

Root Cause Analysis

The race condition occurs in the AlignedLockableBuffer buffer pool implementation:

1. Buffer Reuse Timing: Buffers are returned to the pool in a round-robin fashion without waiting for downstream consumers to release references
2. Shallow Slice References: Buffer.slice() creates shallow views that share underlying memory with the original buffer
3. Premature Reuse: New data overwrites buffer memory while old slice() references are still in use by downstream consumers
4. Data Corruption: Downstream consumers read corrupted data from their slice references

Problematic code (lib/aligned-lockable-buffer.ts):

public getCurrentBuffer(): AlignedLockableBuffer {
    // Buffer returned immediately without reference tracking
    this.currentBufferIndex = (this.currentBufferIndex + 1) % this.numBuffers;
    return buffer; 
}

Why GZ/XZ Are Affected But Not ZIP

GZ/XZ (Affected):

• Use streaming transforms (createGunzip(), createDecompressor())
• Decompressed data flows through BlockReadStream → shared buffer pool
• Vulnerable to race condition

ZIP (Not Affected):

• Use specialized libraries (unzip-stream, yauzl) with internal buffering
• Never touches the shared buffer pool
• Self-contained decompression with dedicated buffer management