An independent evaluation of TabFM, Google's tabular foundation model (yashrajpandey.com)

Google Research's recent release of TabFM, a zero-shot foundation model designed for tabular data, has drawn attention in the AI/ML community. TabFM purportedly performs at or above the level of tuned gradient-boosted trees without the need for training or tuning—merely accepting tabular inputs for direct predictions. An independent evaluation of TabFM demonstrated that it outperformed an Optuna-tuned XGBoost on all ten matched datasets, with remarkable performances on various classification and regression tasks. Notably, in tasks like maternal health risk prediction, TabFM's accuracy increased impressively under zero-shot conditions, challenging traditional methods that have dominated tabular data analysis. The significance of TabFM lies in its potential to streamline predictions in tabular data, an area where deep learning has historically struggled compared to tree-based models. Technical insights revealed that TabFM employs a context-based learning approach, mirroring large language models, while efficiently managing memory with a fixed footprint of about 16.95 GB. This efficiency extends to the operational use of GPUs, revealing a speed advantage of up to 42 times faster compared to CPUs in ideal conditions. Additionally, the evaluation process uncovered a bug that affected performance on multi-GPU setups, demonstrating the importance of rigorous, reproducible testing in validating foundational models like TabFM.