Artificial agents now perform on par with or better than experts on several challenging decision-making tasks. People, however, remain reluctant to allow algorithms to make decisions on their behalf and legal constraints may prevent it altogether. How can we harness artificial intelligence, while maintaining trustworthiness and accountability? We propose confirmation trees, a decision-tree strategy for hybrid intelligence that can improve accuracy while maintaining human control. First, decisions are elicited from a human expert and an artificial agent. If they agree, that decision is adopted. If they disagree, a second human expert is consulted to break the tie. Hence, a human expert always approves the final decision. Our approach outperforms human experts or algorithms alone at diagnosing malign skin lesions. Crucially, it performs better than a strong human baseline, using substantially fewer human ratings. Our results show the potential of this approach for medical diagnostics and beyond.

We study hybrid confirmation trees, a simple heuristic for producing hybrid intelligence in high-stakes classification tasks. Hybrid confirmation trees first elicit the decision of one human expert and one algorithm. Whenever the two agree a decision is immediately made. In case of disagreement, a second human expert is called in to break the tie. We apply this approach to data on deepfake detection, recidivism prediction and skin tumor diagnosis and investigate how it performs for experts of varying levels of skill. Our approach proves to be a powerful alternative to human-only confirmation trees in all data sets we test and for all skill levels as it performs similar, if not better, at reduced cost. In addition, for high-performing individuals it can outperform both human confirmation trees and algorithms, producing complementary human-algorithm performance. We show that this effect exists because skilled experts disagree with the algorithm on the right instances.