We used an AI as a controlled probe of our alert documentation (glassmkr.com)

A team leveraging AI conducted an experiment to enhance their alert documentation by using a coding agent to resolve active alerts in their server fleet, constrained to only use internal resources without general Linux knowledge. This innovative approach aimed to pinpoint gaps in their documentation, ensuring that customers could autonomously address issues without needing additional external information. The results revealed different failure modes, highlighting inconsistencies between alert evidence and guidance, such as alerts retaining historic data without time windows and alerts lacking actionable fixes based on journal outputs. This experiment is significant for the AI/ML community as it demonstrates a practical application of AI in improving documentation and user autonomy in IT management. By restricting the AI from accessing broader knowledge, the team could effectively identify documentation shortfalls, resulting in rapid enhancements to both their alert evidence structure and documentation. Implementing this method not only streamlined operations but also provided valuable insights into the usability of existing materials, paving the way for future improvements and setting a precedent for organizations aiming to refine their operational guidance through controlled AI engagements.