Effects of Algorithmic Flagging on Fairness: Evidence from Wikipedia (mako.cc)

A recent study published at the Conference on Computer-Supported Cooperative Work (CSCW) has examined the effects of algorithmic flagging on fairness within online platforms, specifically focusing on Wikipedia's RCFilters system. This system uses a machine learning algorithm, ORES, to flag potentially damaging edits, thus guiding moderators in maintaining content quality. The study reveals a significant finding: despite inherent biases within the ORES algorithm against unregistered editors, the use of algorithmic flags resulted in improved fairness across Wikipedia's editing landscape. The flags primarily reduce the bias moderators exhibit toward unregistered editors by enhancing the detection of damaging edits from both registered and unregistered users. This research is crucial for the AI/ML community as it highlights the complex interplay between algorithmic predictions and human moderation within sociotechnical systems. By utilizing regression discontinuity, the authors were able to demonstrate that slight shifts in edit scores could lead to disproportionately high moderation action on flagged edits, even when those edits were not significantly more damaging. Such insights have implications for designing fairer AI systems, suggesting that understanding sociotechnical contexts is just as vital as rectifying algorithmic biases. The study's findings call for a reevaluation of how algorithmic systems are implemented and monitored, ensuring that they promote equitable outcomes in collaborative environments like Wikipedia.