Lambda isn't leaking memory, your metrics are lying to you (engineering.taktile.com)

A recent incident highlighted a critical misunderstanding regarding memory metrics in AWS Lambda. A customer running 40 ONNX machine learning models experienced unexpected out-of-memory (OOM) issues, prompting a series of adjustments intended to reduce memory usage. However, lowering the cache size from 16 to smaller values led to increased memory consumption due to a cycle of loading and unloading cache, revealing that mere adjustments to memory limits were ineffective. This problem culminated in a realization that the reported "@maxMemoryUsed" metric, traditionally viewed as a measure of memory consumed per invocation, actually reflects a high-water mark across the execution environment, misleading developers into believing there was a memory leak. Significantly, the investigation revealed that the underlying issue stemmed from the glibc allocator's behavior, which hoarded memory from previous allocations rather than reclaiming it effectively. By adjusting the allocation strategies and controlling the mmap threshold, the team achieved a notable reduction in memory footprint without severely impacting performance. This incident emphasizes the importance of understanding memory reporting mechanisms in serverless architectures like Lambda and suggests the need for more accurate profiling tools, as traditional metrics may mask underlying inefficiencies and lead to incorrect conclusions about application performance.