While traditional computer interfaces based on the mouse and keyboard

are ubiquitous, they are ill suited to many common application

domains. This is particularly true in education, where recent research

suggests that students perform better when instructional interfaces

are more similar to work practice. Thus, the goal of our work is to

create computational techniques and user interface design principles

to enable natural, pen-based tutoring systems that scaffold students

in solving problems in the same way they would ordinarily solve them

with paper and pencil. In our work, we have focused on interfaces

suitable for a ``pentop computer,'' a writing instrument that is used

with special dot-patterned paper, and that has an integrated digitizer

and embedded processor. A pentop is capable of producing dynamic output

in the form of synthesized speech and recorded sound clips.

Accurate shape recognition is an essential foundation for developing

pen-based interfaces. We created a trainable, multi-stroke recognizer

that is insensitive to orientation, non-uniform scaling, and drawing

order. Symbols are represented internally as attributed relational

graphs describing both the geometry and topology of the symbols.

Symbol recognition is accomplished by finding the definition symbol

whose attributed relational graph best matches that of the unknown

symbol. We developed five efficient approximate matching techniques

to perform the graph matching.

To explore instructional and interface design issues, we created

Newton's Pen, a pentop-based statics tutor. This system, which is

intended for undergraduate education, scaffolds students in the

construction of free body diagrams and equilibrium equations.

Newton's Pen employs a finite state machine architecture that

effectively models the student's problem-solving progress, thus

serving as a convenient means for providing context-sensitive tutorial

feedback. User studies suggest that Newton's Pen is an effective

teaching tool, and that students are satisfied with the interface.

A key issue in the design of pentop interfaces is how to provide

effective feedback to the user. To explore this issue, we developed

PaperCAD, a system that enables users to query geometric information

from printed CAD drawings. PaperCAD employs two methods of feedback:

audio feedback with an adjustable level of conciseness, and a PDA that

provides a video display of the portion of the drawing near the pen

tip. This system also employs a novel technique that uses a hidden

Markov model to correct interpretation errors in hand-written

equations. Results of a user study suggest that users are highly

satisfied with the interface and prefer it to a traditional WIMP

interface.

Solving open-ended problems with paper and pencil is an important part of education and constitutes a large share of course workload, especially in science and engineering programs. However, grading this kind of work can be prohibitively expensive in large classes. Because of the complexity of handwritten free-form work, there are no existing computational methods capable of interpreting it. In this dissertation, we aim to work toward complete interpretation and semantic analysis of handwritten free-form solutions to homework and exam problems of the sort assigned in undergraduate engineering and science courses. In our study, students wrote their homework and exam solutions on dot patterned paper with Livescribe smartpens. Our work comprises three innovations. First, we developed methods to locate final answers. Our methods locate answers by identifying the boxes drawn around them. Experiments demonstrated that our methods are both accurate and efficient at recognizing answer boxes. Second, we developed a novel CNN-BLSTM-CRF network for semantic labeling of students' handwritten assignments. Semantic labeling is the task of classifying pen strokes according to the type of content they represent. Our method distinguishes between cross-out strokes, equation strokes, and free body diagram strokes. The input to our network is a set of pen strokes, which are sorted in chronological order, and the output is a sequence of labels for the strokes. Labeling strokes in this manner is an important step in enabling the semantic analysis of the writing. Our labeling approach outperforms existing methods. Finally, we developed a novel GRU-CRF network to locate complete mathematical equations. The network exploits the temporal context of consecutive strokes and simultaneously finds and groups equation strokes. Once our method has located the equations, they can be interpreted using existing techniques. Together, this work provides a significant step toward the automated grading of handwritten free-form course work.

This dissertation examines students’ homework behaviors and their relationship to academic achievement in introductory engineering courses. Much of the prior work examining the relationship between homework and achievement has relied on student self-reports of homework effort. Our results demonstrate that such self-reports are problematic. Instead, we avoid this methodological shortcoming by using smartpens to objectively measure students’ learning activities in an unobtrusive manner and with a high level of fidelity. This dissertation examines how much, how frequently, and when students work on their homework assignments, and if these factors are related to achievement. This dissertation also examines if informing students of their homework behavior influences them to change that behavior and improve achievement.

This work makes four major contributions. First, we developed quantitative measures of student homework behavior that are related to academic achievement. Second, we demonstrate that self-reported measures of student homework effort are problematic. Third, we show that measures of homework effort early in a course are nearly as effective at predicting achievement as measures from the entire course. This result suggests that student behavior does not change significantly over a course. Finally, we show that informing students of their homework behaviors, and providing suggestions for improving those behaviors, is an insufficient motivator to change behaviors and improve achievement. This result suggests a two-stage model of metacognition for study behaviors, requiring both monitoring (i.e., being aware of how one is studying) and regulation (i.e., adjusting how one studies based on feedback) to affect changes in behavior.

This work makes both applied and methodological contributions to educational research. In contrast to existing research, our results demonstrate a strong and consistent relationship between students’ homework behaviors and academic achievement. Additionally, this work shows that students’ homework behaviors are established early in a course, and tend to remain relatively constant throughout a course.

This work highlights the potential of educational data mining and smartpen technology for educational research. Our results confirm the unreliability of studies employing self-reports. Our studies also speak to the value of replication in education research.

Educational Data Mining is a nascent, but rapidly growing field which has typically applied data mining techniques to educational data extracted from digital systems such as Learning Content Management Systems and Intelligent Tutoring Systems.

The research thus far has been able to identify interesting patterns in the ways students learn when using these systems, but an analysis of students' ordinary problem-solving processes remains unexplored.

In this work we apply data mining and machine learning techniques to a digital data set of students' ordinary, handwritten coursework in the context of a Mechanical Engineering course. This work makes four major contributions. It is the first, to our knowledge, study in which data mining and machine learning techniques have been applied to students' problem-solving processes in their ordinary learning environment, using pen and paper at home or in the classroom; because the data set is unique, we provide an in-depth description of the large digital collection of students' handwritten course work we have collected. Second, we investigate novel, discrete and numerical representations of students' handwritten coursework which characterize different aspects of students' ordinary problem-solving processes. Third, we identify patterns in these representations as well as correlations between these representations and course performance using various machine learning and data mining techniques. Last and most importantly, we present interesting conclusions that may be drawn from these patterns and correlations, which allow instructors of the course which was investigated to improve future offerings.

We present a technique that examines handwritten equations from a student's solution to an engineering statics problem and estimates the correctness of the work. The solution is recorded with a smartpen that digitizes the writing with time stamps. Our technique first separates equation pen strokes from other content, such as diagrams. Then the equation pen strokes are grouped first into individual equations, then into individual characters. A character recognizer is used to recognize each character, and then a Hidden Markov Model is used to correct recognition errors. The equation text is characterized by a set of features. Some of these describe the frequency of various symbols and symbol combinations, such as a letter following a mathematical operator. One feature describes the frequency with which units of measure (e.g., "kg") appear, while another describes inter-character pauses. This set of features is used to construct SVM regression models to predict the correctness of the work. We tested our approach on a corpus of solutions to exam problems from an undergraduate statics course. The SVM models predicted the grade assigned by the instructor with an average coefficient of determination of 36%. We also combined our features with an existing set of features that describe the temporal and spatial organization of a handwritten solution a statics problem. Using both sets of features, the SVM models predicted the grade on the exam problems with an average coefficient of determination of 56%. This is a surprising result given that none of the features consider the semantic content of the writing, or even the correctness of a student's final answer.

We present an intelligent pen-based tutoring system for Statics - the sub-discipline of engineering mechanics

concerned with the analysis of mechanical systems in equilibrium under the action of forces. The system

scaffolds students in the construction of free-body diagrams and equilibrium equations for planar devices

comprised of one or more rigid bodies.

While there has been extensive research in intelligent tutoring systems, most existing systems rely

on traditional WIMP (Windows, Icons, Menus, Pointer) interfaces. With these systems, students typically

select the correct problem solution from among a set of predefined choices. With our pen-based interface, by

contrast, students are guided in constructing solutions from scratch, mirroring the way they solve problems

in ordinary practice, which recent research suggests is particularly important for effective instruction.

Our system embodies several innovations including a novel instructional technique that focuses

students' attention on a system boundary as a tool for constructing free body diagrams, a tutorial feedback

system based on "buggy rules" that attempt to diagnose and correct problem-solving errors typical of novice

students, and a hierarchical feedback system which promotes independent problem-solving skills. In winter

2010, the tutoring system was used by 100 students in an undergraduate Statics course at our university.

Results from pre- and posttests reveal measurable learning gains even after only a short exposure to the

system. In an attitudinal survey, students reported that, while there is room for improvement, the interface was preferable to a WIMP interface and the methodology implemented in the system was valuable for

learning Statics.

Engineers use sketches in the early phases of design because their expressiveness and ease of creation facilitate creativity and efficient communication. Our goal is to build software that leverages these strengths and enables natural sketch-based human-computer interaction. Specifically, our work is focused on creating algorithms that group hand-drawn strokes into individual objects so that they can be recognized. Grouping strokes is a difficult problem; many previous approaches have required the user to manually group the strokes. Others have used a search process, resulting in a computational cost that rises exponentially with the number of strokes in the sketch. In this dissertation we present a novel method for grouping strokes into objects based on a two-step algorithm that has a polynomial computational cost. In the first step, strokes are classified according to the type of object to which they belong, thus helping to create artificial separation between objects. In the second step, strokes of the same class are grouped into individual objects. Both steps rely on general machine learning techniques which seamlessly integrate spatial and temporal information, and which can be extended to new domains with no hand-coding. The single-stroke classifier used to separate strokes is the first in literature to perform multi-way classification, and it performs as well as or better than previous classifiers on text vs. non-text classification. Our grouping algorithm correctly groups between 84% and 91% of the ink in diagrams from four different domains, with between 61% and 82% of shapes being perfectly clustered. Our method runs in O(n²) time, where n is the number of points in the sketch. Real-world performance is improved with a conservative filter to eliminate consideration of distant strokes, and computation occurs incrementally as the sketch is constructed. Even without the filter, the computation for a large sketch containing over 700 strokes took less than 12% of the time required to draw the sketch. Experimental evaluation of our technique has shown it to be accurate and effective in four domains.

In education, many assessments boil down to getting the correct solution or necessary result to receive credit. This end goal mentality, in turn, influences how educators transfer knowledge to students. For example, some educators may present or walk through completed solutions. However, continually displaying and using worked-out solutions to teach can quickly become an obstruction to learning. In computer science, a significant amount of learning occurs while fixing the errors that litter the pathway from a blank page to a working solution. This dissertation establishes a methodology for teaching from student errors in computer science.

The first part of the dissertation establishes how we developed over fifty lightweight exercises to integrate into a ten-week course without content replacement. Using past research in computer science, education theory, and cognitive load theory, we developed and refined a standard exercise structure that incorporates student submissions containing erroneous code, past student solutions presented during student-instructor interactions, and instructor feedback. Collectively, our core exercises are known as "What's Wrong With My Code" exercises.

Next, we evaluated the "What's Wrong With My Code" exercises in three distinct ways. First, we performed a study to assess student improvement when using our exercises in place of current course activities (e.g., CodeLab). Second, we analyzed the differences in student error encounters for an entire term by comparing error counts in prior course offerings to offerings with our exercises integrated into the weekly course workload. Lastly, we evaluated student self-efficacy improvements over an entire term by comparing offerings with and without our exercises. In each of the studies, our exercises proved beneficial with increased student performance (with effect sizes of 0.56 and 0.42), increased self-efficacy (p-value < 0.05), and diminished student error encounter rates.

Finally, we used our methodology to implement additional exercises to demonstrate a pathway for use beyond common errors. Specifically, we developed exercises to teach programming style in an introductory C++ computer science course. We evaluated the style exercises alongside data from seven years of submissions, which spanned four different instructional methods of teaching programming style. Our research concludes that students showed increased use of proper programming style before receiving any assessment feedback in academic terms that utilized our exercises. Additionally, we discovered that using an automatic assessment tool with an assigned style grade significantly improves the use of proper programming style.

This dissertation creates a methodology for teaching from student errors in any computer science course, utilizes the methodology to provide multiple implementations and use case examples for an introductory computer science course in C++, and suggests concrete changes for computer science course instructors.

This thesis presents two intelligent tutoring systems for statics, Newton's Tablet and Newton's Pen, and the results from experiments examining their value as tools for undergraduate mechanical engineering education. The systems are unique in that they break down the problem-solving process in a manner which makes conceptual decisions explicit. This allows for focused feedback to correct conceptual errors that would otherwise lead to incorrect free-body diagrams and equilibrium equations. The intelligent tutoring systems were designed to help ease the intrinsic cognitive load associated with the first difficult engineering subject that mechanical engineering students encounter. Results from experiments indicate that the problem-solving method employed is beneficial, and that Newton's Tablet is more effective than Newton's Pen.