סמינר בניהול טכנולוגיה ומערכות מידע

Dr. Tomer Geva, Coller Scholl of Management, Tel Aviv University.

The capacity to assess and continuously monitor the decision accuracy of expert workers as well as the ability to infer and continuously monitor the relative ranking of such experts is invaluable both for consumers of expert services, as well as for their management. In many such settings, ground truth—i.e., the correct decision for a given decision instance—is costly and otherwise difficult to acquire. Consequently, ground truth may not be revealed ex-post for most instances. Having limited ground truth undermines the inference of experts’ decision accuracies relative ranking in many settings, and in some settings, there are simply no existing methods to do so. In this work, we first propose three new business data science problems addressing the ability to reconstruct the correct ranking and assessment of expert workers by the accuracy of their decisions. This, solely based on historical data on past decisions, without any ground truth data or with only scarce ground truth data. We develop novel machine-learning-based algorithms for addressing these problems, and we demonstrate their performances over multiple domains and settings.