Proceedings of the Annual Meeting of the Cognitive Science Society

We study hybrid confirmation trees, a simple heuristic for producing hybrid intelligence in high-stakes classification tasks. Hybrid confirmation trees first elicit the decision of one human expert and one algorithm. Whenever the two agree a decision is immediately made. In case of disagreement, a second human expert is called in to break the tie. We apply this approach to data on deepfake detection, recidivism prediction and skin tumor diagnosis and investigate how it performs for experts of varying levels of skill. Our approach proves to be a powerful alternative to human-only confirmation trees in all data sets we test and for all skill levels as it performs similar, if not better, at reduced cost. In addition, for high-performing individuals it can outperform both human confirmation trees and algorithms, producing complementary human-algorithm performance. We show that this effect exists because skilled experts disagree with the algorithm on the right instances.