When attempting to accomplish unfamiliar tasks, people often look for tutorials to follow instructions. While it is easy to access online instructions shared by domain experts, navigating step-by-step guidance using existing tools remains inefficient. In addition, producing high-quality instructions that are easy to follow requires authoring expertise and a significant time investment in editing. This dissertation introduces video-based recording, editing, and playback tools optimized for creating and consuming tutorials from author demonstrations. Our interactive systems capture videos and high-level events that are important to a learner. Using video and audio analysis techniques, we develop algorithms that automatically produce high-quality instructions, which dramatically reduce the effort required for amateur creators. By introducing novel tutorial formats combined with video content, these designs in turn improve viewers’ learning experience.

We present a series of authoring tools that enable amateur authors to create effective tutorials: 1) MixT is a system that automatically generates step-by-step mixed media tutorials from software demonstrations. 2) DemoWiz is a tool that provides an increased awareness of upcoming events in a software demonstration video. 3) DemoCut is a semi-automatic video editing tool for physical tasks. 4) Kinectograph is a recording device that automatically follows an instructor for filming a physical demonstration. 5) DemoDraw is a multi-modal system to generate step-by-step motion illustrations from author’s body movements. Current authoring practices from professionals are encoded into automatic algorithms and interactive techniques. These systems are evaluated through a series of studies, which demonstrate that users can efficiently create and follow concise instructions using our tools.

While Deep Learning ("DL") techniques have enabled groundbreaking advances in many domains, non-expert DL users encounter significant usability challenges when attempting to develop, debug, and interpret DL applications. This work describes how techniques from program analysis and DL interpretability are drawn upon to build novel, interactive tools that support users in important stages of DL development. Key interactions of these tools facilitate pattern discovery through exploration and provide explanations that reveal underlying structure. At early stages, ACUMEN helps users find suitable templates to start their DL projects through exploring and annotating an interactive visualization of code embeddings and extracted attributes. UMLAUT helps users find and fix silent errors in DL programs during model training with an interface unifying visualizations, code, and error explanations. IMACS helps users explore and compare influential concepts extracted from image classification models during model evaluation. User studies reveal how these systems address usability gaps at different stages of the DL development process, as well as how these interaction techniques can generalize to other scenarios.

Collaboration and multi-user interactions are key aspects of many software tasks. In traditional desktop interfaces, such elements are well supported through built-in collaboration functions or general-purpose techniques such as screen and video sharing. Collaboration and guidance may also be required in Mixed Reality environments, where users carry out spatial actions in a three-dimensional space. However, not all users may have access to the same Mixed Reality interface. All of them may not have access to the same information, the same visual representation, or the same interaction affordances. Such asymmetries make communication and collaboration between users harder. To address these issues, we introduce Interactive Cross-Dimensional Media. In these media, the visual representation of information streams can be changed between 2D and 3D. Different representations can be chosen automatically based on context or through associated interaction techniques that give users control over exploring spatial, temporal, and dimensional levels of detail. This ensures that users understand and interpret any information or interaction across different dimensions, interfaces, and spaces. We have deployed these techniques in four different contexts: (1) Mixed Reality telepresence for remote instruction of physical tasks, (2) asynchronous video-based instruction of virtual tasks, (3) live asymmetric guidance of virtual tasks, and (4) live interactive spectating of virtual tasks. Through user studies of these systems, we show that Mixed Reality environments that provide users with interactive cross-dimensional media interfaces improve performance and user experience in multi-user and collaboration settings.

Video has become a primary medium for producing and consuming entertainment, educational content, and documented events. As the cost of recording, sharing, and storing video has decreased, the prevalence of video has increased. Yet, video remains challenging to use as an informative medium because it is difficult to search, browse and skim the underlying content.

Using a timeline-based video player, users must scrub back-and-forth through a video to gain an overview or locate content of interest. Some video viewing interfaces allow users to search and browse videos by text transcripts or caption files. However, transcribed and captioned speeches or conversations often contain disfluencies and redundancies typical of speech which make the transcripts time-consuming to read and difficult to skim. Further, transcripts and captions lack structure -- transcripts consist of long blocks of text, while captions have a series of short phrases. Without structured organization, it can be difficult for viewers to browse topics or get a high-level overview of the content.

This thesis explores new ways to search, browse and skim videos through structured text. We aim to create navigable representations of videos that let users ask questions of the video as efficiently and flexibly as possible, and facilitate low cost production. This thesis introduces systems that embody these goals, using structured-text documents aligned to video to enable efficient and flexible video searching, browsing and skimming across three domains: (1) informational lecture videos, (2) films, and (3) casually recorded feedback.

Traditional wood furniture often features decorative intrinsic joints and beautifully curved parts created with woodworking techniques. To craft such furniture, experienced woodworkers usually begin by drawing a visual design that describes not only the overall shape of the furniture but also the detailed appearances of the joints and curved parts. They then manually verify whether their furniture design satisfies all the constraints imposed by the intended design scenario (e.g., furniture assembled via joints must be stable when put together) and redesign if needed. Finally, they fabricate the furniture with conventional woodworking tools.

This traditional furniture crafting process demands experience in woodworking and has historically been inaccessible to amateur users. Encouragingly, the continuing advancement and increasing availability of CAD programs and CNC machines have lowered the entry to the visual design and physical fabrication steps of the furniture crafting process. But, neither CAD programs nor CNC machines help users verify whether their visual design of the furniture fulfills all the necessary constraints, even though such verification often requires advanced woodworking knowledge not commonly held by everyday users. As a result, the verification step of the crafting process presents a serious challenge that prevents average users from expressing their creativity in furniture design.

This thesis explores ways to assist users with verifying their furniture designs and guide users towards visual designs that satisfy all the requisite constraints. We aim to develop algorithms that automate the difficult verification step, allowing users to focus on producing the visual design of their furniture rather than analyzing the validity of their design. This thesis introduces interactive design tools that integrate algorithms informed by woodworking techniques to enable users to create (1) stable furniture models assembled via decorative intrinsic joints and (2) curved furniture parts formed through flexible kerf patterns.

This dissertation presents a new tool for exploratory text analysis that attempts to improve the experience of navigating and exploring text and its metadata. The design of the tool was motivated by the unmet need for text analysis tools in the humanities and social sciences. In these fields, it is common for scholars to have hundreds or thousands of text-based source documents of interest from which they extract evidence for complex arguments about society and culture. These collections are difficult to make sense of and navigate. Unlike numerical data, text cannot be condensed, overviewed, and summarized in an automated fashion without losing significant information. And the metadata that accompanies the documents -- often from library records -- does not capture the varied content of the text within.

Furthermore, adoption of computational tools remains low among these scholars despite such tools having existed for decades. A recent study found that the main culprits were poor user interfaces and lack of communication between tool builders and tool users. We therefore took an iterative, user-centered approach to the development of the tool. From reports of classroom usage, and interviews with scholars, we developed a descriptive model of the text analysis process, and extracted design guidelines for text analysis systems. These guidelines recommend showing overviews of both the content and metadata of a collection, allowing users to separate and compare subsets of data according to combinations of searches and metadata filters, allowing users to collect phrases, sentences, and documents into custom groups for analysis, making the usage context of words easy to see without interrupting the current activity, and making it easy to switch between different visualizations of the same data.

WordSeer, the system we implemented, supports highly flexible slicing and dicing, as well as easier transitions than in other tool between visual analyses, drill-downs, lateral explorations and overviews of slices in a text collection. The tool uses techniques from computational linguistics, information retrieval and data visualization.

The contributions of this dissertation are the following. First, the design and source code of WordSeer Version 3, an exploratory text analysis system. Unlike other current systems for this audience, WordSeer 3 supports collecting evidence, isolating and analyzing sub-sets of a collection, making comparisons based on collected items, and exploring a new idea without interrupting the current task. Second, we give a descriptive model of how humanities and social science scholars undertake exploratory text analysis during the course of their work. We also identify pain points in their current workflows and give suggestions on how systems can address these problems. Third, we describe a set of design principles for text analysis systems aimed at addressing these pain points. For validation, we contribute a set of three real-world examples of scholars using WordSeer 3, which was designed according to those principles. As a measure of success, we show how the scholars were able to conduct analyses yielding otherwise inaccessible results useful to their research.