A Disappearing Service Processor (oxide.computer)

Oxide Computing encountered a significant issue with their next-generation Service Processor (SP) for the Oxide rack during initial tests, where the SP unexpectedly dropped off the management network, complicating debugging efforts. The incident highlighted potential software bugs in their custom operating system, Hubris, which manages tasks for system functions. Debugging revealed concerns over task starvation and potential stack overflows, particularly due to the complexity of managing separate tasks within the system. Efforts to diagnose the issue were aided by hardware adjustments and insights from documentation on the Cortex-M7 architecture. Ultimately, the root of the problem was traced back to memory access conflicts caused by a mismatch in memory attributes. The SP's interactions with an FPGA were leading to improper cache handling when switching between task and kernel mode. By adjusting the base address of the FPGA's interface to ensure compatibility with expected memory access properties, Oxide resolved the issue, marking a significant step in enhancing the reliability of their hardware systems. This incident highlights the importance of comprehensive documentation in hardware design and the need for robust debugging practices to ensure system stability in AI and data center environments.