HealthFormer: Transformers for irregular electronic health record events [pdf] (www.medrxiv.org)

A recent preprint introduces HealthFormer, a novel dual-level Transformer model designed to effectively handle irregular electronic health record (EHR) events. This framework addresses two key challenges in EHR modeling: preserving the intricate structures within clinical encounters and capturing the complex temporal dependencies between events. HealthFormer utilizes an Intra-Event Encoder to create event embeddings that account for multiple coding systems, while the Inter-Event Encoder applies continuous-time attention biases to consider long-term patient care trajectories. The model was pretrained on extensive Hungarian national health records and employs a multi-task self-supervised approach to refine its representations. The significance of HealthFormer lies in its ability to generate time-aware, event-centric representations that can be fine-tuned for various clinical prediction tasks without extensive redesign. Demonstrating impressive performance, the model achieved high average area under the curve (AUC) scores in predicting incident colorectal and prostate cancers, surpassing traditional logistic regression models. This innovation not only enhances the interpretability of learned diagnosis embeddings by aligning with classification hierarchies but also holds the potential for broader applications across diverse healthcare systems, given its adaptability to standard coding schemas like ICD-10 and ATC. Future work will focus on validating these findings across multiple clinical endpoints.