Skip to main content
Open Access Publications from the University of California

Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques


Expert feedback is valuable but hard to obtain for many de- signers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 facto- rial design. Student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated just as valuable as expert feedback. A follow-up analy- sis on writing style showed that student designers found feed- back most helpful when it was emotionally positive and spe- cific, and that providing a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other fa- vorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.

Main Content
For improved accessibility of PDF content, download the file to your device.
Current View