UC Irvine

Microtask crowdsourcing is a form of crowdsourcing in which work is decomposed into a set of small, self-contained tasks, which each can typically be completed in a matter of minutes. The approach has been used to address a number of different problems, ranging from labeling images to planning travel. To date, however, little is known about the potential of microtask crowdsourcing in software engineering.

This thesis explores microtask crowdsourcing as applied to software design work. We particularly conducted a large study with Amazon Mechanical Turk workers, who each provided one or more solution alternatives for a small, partial software design problem. We included two experimental conditions: (1) user interface design work versus internal code design work, and (2) workers operating independently versus workers being shown previous designs from other workers.

We report on various results concerning solution diversity, solution quality, and perceived task difficulty, across the different experimental conditions. Our primary findings show that: (1) it is feasible for a crowd to generate a broad range of solution alternatives for a software design problem, (2) solutions alternatives range all over the quality spectrum, and (3) many workers perceived the task as difficult.