Resetting RAG-based LLMs doesn't reset behavior (github.com)

A new dual-GPU workstation has been designed to support reproducible AI safety evaluations and research, specifically focused on deterministic execution and long-duration stability. Unlike conventional systems that prioritize raw performance, this hardware aims for isolation and reproducibility, making it suitable for assessing retrieval-augmented and multimodal language model (LLM) workflows. Key features include a desktop-class CPU, dual discrete GPUs for separating workloads, substantial system memory, and robust cooling, ensuring that UI activities do not interfere with research measurements. This initiative addresses significant challenges faced by current laptop-class setups, such as thermal throttling and scheduling conflicts, which compromise the confidence in audit-grade evaluations. By providing a controlled environment for long-term analytical workloads and human-in-the-loop studies, the workstation could enhance the reliability of AI assessments, ensuring that systems behave consistently across various evaluations. Ultimately, the platform signifies an important step toward establishing rigorous standards in AI safety research, paving the way for improved understanding and transparent evaluation of AI systems.