HackerRank open sourced its ATS. My resume scored 90/100. Oh wait 74. No – 88 (danunparsed.com)

HackerRank has open-sourced its Applicant Tracking System (ATS), sparking significant interest across platforms like LinkedIn and Reddit. The tool evaluates resumes using a language model (LLM) and assigns scores based on factors such as open-source contributions and work experience. However, users have reported highly inconsistent scores for the same resume, highlighting a fundamental issue with the model's ability to judge project complexity and relevance consistently. For instance, one user experienced scores ranging from 66 to 99, illustrating how this AI tool can act as a “luck filter” rather than a reliable assessment mechanism. The significance for the AI/ML community lies in the realization that while LLMs excel at parsing structured data and assessing technical skills, they struggle to evaluate subjective aspects like project quality and experience with precision. This inconsistency can lead to potentially overlooking highly qualified candidates due to the inherent design flaws in the scoring methodology—particularly when experience scoring offers no defined rubric compared to the more chaotic, project-based assessments. This raises concerns about reliance on AI for resume screening, emphasizing the need for careful consideration when implementing AI-driven hiring solutions in organizations.