Skip to main content
eScholarship
Open Access Publications from the University of California

The Center for Information Technology Research in the Interest of Society (CITRIS), is a multi-campus, multi-disciplinary research institute of the University of California. Established in 2001 as one of four California Institutes for Science and Innovation, CITRIS bridges the gap between world-class laboratory research and the development of applications, platforms, companies, and even new industries. CITRIS facilitates partnerships among more than 300 affiliated faculty members, thousands of students, and researchers from over 60 corporations and institutions. Spanning four UC campuses, CITRIS leverages the research strengths of UC Berkeley, UC Davis, UC Merced and UC Santa Cruz, and operates within the greater ecosystem of the statewide University system and the innovative and entrepreneurial spirit of Silicon Valley.

Cover page of Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques

Almost an Expert: The Effects of Rubrics and Expertise on Perceived Value of Crowdsourced Design Critiques

(2016)

Expert feedback is valuable but hard to obtain for many de- signers. Online crowds can provide a source of fast and affordable feedback, but workers may lack relevant domain knowledge and experience. Can expert rubrics address this issue and help novices provide expert-level feedback? To evaluate this, we conducted an experiment with a 2x2 facto- rial design. Student designers received feedback on a visual design artifact from both experts and novices, who produced feedback using either an expert rubric or no rubric. We found that rubrics helped novice workers provide feedback that was rated just as valuable as expert feedback. A follow-up analy- sis on writing style showed that student designers found feed- back most helpful when it was emotionally positive and spe- cific, and that providing a rubric increased the occurrence of these characteristics in feedback. The analysis also found that expertise correlated with longer critiques, but not the other fa- vorable characteristics. An informal evaluation indicates that experts may instead have produced value by providing clearer justifications.

Cover page of SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries

SceneSkim: Searching and Browsing Movies Using Synchronized Captions, Scripts and Plot Summaries

(2015)

Searching for scenes in movies is a time-consuming but cru- cial task for film studies scholars, film professionals, and new media artists. In pilot interviews we have found that such users search for a wide variety of clips—e.g., actions, props, dialogue phrases, character performances, locations— and they return to particular scenes they have seen in the past. Today, these users find relevant clips by watching the entire movie, scrubbing the video timeline, or navigating via DVD chapter menus. Increasingly, users can also index films through transcripts—however, dialogue often lacks vi- sual context, character names, and high level event descrip- tions. We introduce SceneSkim, a tool for searching and browsing movies using synchronized captions, scripts and plot summaries. Our interface integrates information from such sources to allow expressive search at several levels of granularity: Captions provide access to accurate dialogue, scripts describe shot-by-shot actions and settings, and plot summaries contain high-level event descriptions. We propose new algorithms for finding word-level caption to script align- ments, parsing text scripts, and aligning plot summaries to scripts. Film studies graduate students evaluating SceneSkim expressed enthusiasm about the usability of the proposed system for their research and teaching.

Cover page of DevCAFE 1.0: A participatory platform for assessing development initiatives in the field

DevCAFE 1.0: A participatory platform for assessing development initiatives in the field

(2015)

The design and assessment of development initiatives is increasingly participatory, where decision makers consider feedback from affected populations. While digital data collection facilitates faster and more reliable analysis, existing data collection tools are not optimized for unstructured qualitative (textual) data and peer-to- peer participant collaboration. In this paper, we propose a system called the Development Collaborative Assessment and Feedback Engine version 1.0 (DevCAFE), a customizable participatory assessment platform that collects and integrates quantitative assessment, qualitative feedback and peer-to-peer collaborative filtering. DevCAFE incorporates a library of statistical analyses for researchers to quickly identify quantitative and qualitative trends while collecting field data. DevCAFE can run on any mobile device with a web-browser and can work with or without Internet connectivity. We present results from two pilot projects: (1) 137 participants evaluating family planning education trainings at three Nutrition Education Centers in rural Uganda, and (2) 4,518 participants evaluating policy priorities for elected leaders in the June 2015 Mexico mid-term elections. DevCAFE collected over 19,000 peer-to-peer ratings of 336 submitted ideas. Feedback gathered through DevCAFE enabled targeted reforms to the family planning efforts in Uganda and the need for increased government attention to public safety in Mexico. Case studies and interactive demos are available at: http://opinion.berkeley.edu/devcafe/

Cover page of M-CAFE 1.0: Motivating and Prioritizing Ongoing Student Feedback During MOOCs and Large on-Campus Courses using Collaborative Filtering

M-CAFE 1.0: Motivating and Prioritizing Ongoing Student Feedback During MOOCs and Large on-Campus Courses using Collaborative Filtering

(2015)

During MOOCs and large on-campus courses with limited face-toface interaction between students and instructors, assessing and improving teaching effectiveness is challenging. In a 2014 study on course-monitoring methods for MOOCs [30], qualitative (textual) input was found to be the most useful. Two challenges in collecting such input for ongoing course evaluation are insuring student confidentiality and developing a platform that incentivizes and manages input from many students. To collect and manage ongoing (“just-in-time”) student feedback while maintaining student confidentiality, we designed the MOOC Collaborative Assessment and Feedback Engine (M-CAFE 1.0). This mobile-friendly platform encourages students to check in weekly to numerically assess their own performance, provide textual ideas about how the course might be improved, and rate ideas suggested by other students. For instructors, M-CAFE 1.0 displays ongoing trends and highlights potentially valuable ideas based on collaborative filtering. We describe case studies with two EdX MOOCs and one on-campus undergraduate course. This report summarizes data and system performance on over 500 textual ideas with over 8000 ratings. Details at http://m-cafe.org.

Lamello: Passive Acoustic Sensing for Tangible Input Components

(2015)

We describe Lamello, an approach for creating tangible in- put components that recognize user interaction via passive acoustic sensing. Lamello employs comb-like structures with varying-length tines at interaction points (e.g., along slider paths). Moving a component generates tine strikes; a real- time audio processing pipeline analyzes the resultant sounds and emits high-level interaction events. Our main contributions are in the co-design of the tine structures, information encoding schemes, and audio analysis. We demonstrate 3D printed Lamello-powered buttons, sliders, and dials.

Fabryq: Using Phones as Gateways to Prototype Internet of Things Applications Using Web Scripting

(2015)

Ubiquitous computing devices are often size- and power-constrained, which prevents them from directly connecting to the Internet. An increasingly common pattern is therefore to interpose a smart phone as a network gateway, and to deliver GUIs for such devices. Implementing the pipeline from embedded device through a phone application to the Internet requires a complex and disjoint set of languages and APIs. We present fabryq, a platform that simplifies the prototyping and deployment of such applications. fabryq uses smartphones as bridges that connect devices using the short range wireless technology, Bluetooth Low Energy (BLE), to the Internet. Developers only write code in one language (Javascript) and one location (a server) to communicate with their device. We introduce aprotocol proxy programming model to control remote devices; and a capability-based hardware abstraction approach that supports scaling from a single prototype device to a deployment of multiple devices. To illustrate the utility of our platform, we show example applications implemented by authors and users, and describe μfabryq, a BLE prototyping API similar to Arduino, built with fabryq.

Structuring Interactions for Large-Scale Synchronous Peer Learning

(2015)

This research investigates how to introduce synchronous interactive peer learning into an online setting appropriate both for crowdworkers (learning new tasks) and students in massive online courses (learning course material). We present an interaction framework in which groups of learners are formed on demand and then proceed through a sequence of activities that include synchronous group discussion about learner-generated responses. Via controlled experiments with crowdworkers, we show that discussing challenging problems leads to better outcomes than working individually, and incentivizing people to help one another yields still better results. We then show that providing amini-lesson in which workers consider the principles underlying the tested concept and justify their answers leads to further improvements. Combining the mini-lesson with the discussion of the multiple-choice question leads to significant improvements on that question. We also find positive subjective responses to the peer interactions, suggesting that discussions can improve morale in remote work or learning settings.