Expert Commentary:
In this article, the authors address the problem of “ability discovery” in a crowd-sourced setting. Unlike traditional crowd sourcing, where the focus is on finding the most appropriate label for a question, their problem is to determine a ranking of users based on their ability to answer questions. This problem is closely related to the well-studied problem of “truth discovery.”
To model items and their labels, the authors employ Item Response Theory (IRT), a widely accepted theory in standardized tests like SAT and GRE. They assume an idealized setting where the relative performance of users is consistent across items and better users choose better fitting labels for each item. Under this assumption, they observe that the response matrices in this setting obey the Consecutive Ones Property (C1P).
To tackle the problem at hand, the authors propose a novel variant of the HITS algorithm called “HITSNDIFFS” (HND). They prove that HND can recover the ideal C1P-permutation when it exists. Additionally, unlike other combinatorial algorithms, HND also returns an ordering even when such a permutation does not exist. As a result, it provides a principled heuristic for the ability discovery problem that guarantees correct answers in the ideal setting.
The authors validate their approach through experiments, comparing HND to state-of-the-art truth discovery methods. They demonstrate that HND produces user rankings with robustly high accuracy. Furthermore, they show that their novel variant of HITS scales better in terms of the number of users compared to prior spectral C1P reconstruction algorithms.
This research has significant implications for crowd sourcing platforms and their ability to identify users’ abilities accurately. By determining user rankings based on their answering abilities rather than just finding the most appropriate label, crowd sourcing platforms can leverage this information for tailored task assignments, quality control, and even expert identification. Future research in this area could explore the application of this approach to different domains and evaluate its effectiveness in real-world crowd sourcing scenarios.