UC San Diego

Programming tutorials often created with the goal of helping people perform complex software-based tasks in domains such as programming, data science, system administration, and computational research. However, it is tedious to create detailed step-by-step tutorials for tasks that span multiple interrelated Graphical User Interface (GUI) and command-line applications. Once these tutorials are created, it is also hard for tutorial creators to get fine-grained feedback about how learners are actually stepping through their tutorials and which parts lead to the most struggle.

We address these challenges by creating two prototypes:

- Torta: an end-to-end system that automatically generates step-by-step GUI and command-line app tutorials by demonstration, provides an editor to trim, organize, and add validation criteria to these tutorials, and provides a web-based viewer that can validate step-level progress and automatically run certain steps.

- Porta: a system that automatically tracks how users navigate through a tutorial webpage and what actions they take on their computer. Porta tracks running shell commands, invoking compilers, and logging into remote servers and surfaces this trace data in the form of profiling visualizations. The visualization augments the tutorial with heatmaps of activity hotspots and markers that expand to show event details, error messages, and embedded screencast videos of user actions.

The core technical insight that underpins both these systems is that Operating-system-wide activity tracing makes it possible to easily generate new tutorials and profile existing tutorials.

An exploratory study on 10 computer science teaching assistants (TAs) and 6 students found that they all preferred the experience and results of using Torta to record and consume tutorials respectively.

A user study of 3 tutorial creators and 12 students who followed their tutorials found that Porta enabled both the tutorial creators and the students to provide more specific, targeted, and actionable feedback about how to improve these tutorials.

These systems together open up possibilities of easing the use of creating quality step-by-step tutorials and user testing existing instructional material in a more systemic and scalable manner.