Humans collaborate to improve productivity and collective outcomes, but people do not always exert maximal effort towards accomplishing collaborative goals. Instead, individuals often expend less effort in groups, a phenomenon known as social loafing that is traditionally viewed as detrimental to productivity. However, theories from distributed computer systems suggest that social loafing might be a rational response to the diminishing returns expected from division of labor when group size increases. Here, we examine how considerations of task efficiency affect the perceived acceptability of withholding effort during a collaborative task. We conducted experiments varying workload and group size across scenarios in which all group members except for one are actively contributing to a common goal. We then compare participant judgments to a model inspired by latency speed-up in distributed systems. We find that people are systematically influenced by task efficiency, in addition to social norms, when judging social loafing.