This is a long write-up about building a C# solver for the One Billion Row Challenge on Apple Silicon. I am going to start from the boring version, explain why it falls apart, then rebuild the system one layer at a time.
It’s 2:47 AM when the alerts start flooding in. Your e-commerce platform - handling Black Friday traffic - begins throwing OutOfMemoryErrors. Orders are failing, customers are abandoning carts, and your revenue is hemorrhaging by the minute. The CPU usage looks normal, disk I/O is fine, but something is silently consuming...