Predicting Success on Hacker News: Bert Models for Scoring HN Titles (philippdubach.com)

A recent study explored the ability to predict successful titles on Hacker News (HN) using advanced BERT models. The research revealed that while factors like timing and the submitter can significantly influence a post's success, the title itself also carries crucial signals. Initial models, including DistilBERT and RoBERTa, achieved AUC scores of around 0.654 and 0.692 respectively. However, these early results were affected by temporal leakage in the training data, prompting the researcher to implement a temporal split for validation and refine the model to focus more on trends rather than memorizing historical data. By optimizing the model's architecture and applying techniques like regularization and isotonic regression for better confidence calibration, the final solution improved overfitting significantly, dropping the train-test gap from 0.109 to 0.042 and achieving a test AUC of 0.685. This model now filters potential hit titles more effectively, with predictions in the top 10% yielding a hit rate of 62%, highlighting an ability to discern patterns that align with community preferences. The research suggests that these methodologies for avoiding temporal leakage and ensuring model calibration are broadly applicable across various fields beyond HN, where predictive accuracy is pivotal.