Number of tokens shouldn't be the only metric (johnjwang.com)

A recent blog post critiques the growing trend of using token counts as the primary performance metric in AI engineering teams, labeling it a potentially harmful practice. While acknowledging tokens as an important measure of resource usage, the author argues that overemphasizing them can lead to inefficiencies, low-quality outputs, and an organization where employees are incentivized to spend tokens quickly rather than focusing on meaningful contributions. The concerns include rising costs associated with token use and the inability to differentiate between quality work and mere quantity in performance evaluations. To foster a productive AI environment, the author advocates for focusing on outcome-based metrics that truly reflect engineering effectiveness, such as product output per engineer, customer issue resolution speed, and team morale. Incentivizing genuine behavior change, the piece calls for empowerment and peer learning instead of quota-driven approaches. Ultimately, while token usage should still be monitored for insights into adoption rates and cost management, it should not overshadow more impactful measures of success in the AI/ML domain.