Inspect Your Model Training (github.com)

A new tool, tvis, has been introduced as a step-through debugger for PyTorch training, allowing developers to run their training scripts unmodified and gain deep insights into their models. By utilizing the command `tvis run`, users can capture the initial optimizer steps of their training, logging crucial details such as input and output shapes, data statistics, gradient information, and per-sample loss. The system compiles extensive debugging information, including the execution call order of every function and module, which can then be visualized in a web browser, providing a unique opportunity to analyze model performance and behavior remotely, even when training across clusters. This tool holds significant implications for the AI/ML community, particularly in enhancing model interpretability and debugging efficiency. As developers strive to optimize their algorithms, the real-time observations and statistics captured by tvis can pinpoint performance bottlenecks and areas for potential improvement. Notably, it also supports advanced functionalities like Grad-CAM for visualizing model attention and PCA for examining embeddings, making it a powerful resource for both researchers and practitioners aiming to refine their models comprehensively without interfering with the training process itself.