UC Riverside

Solving open-ended problems with paper and pencil is an important part of education and constitutes a large share of course workload, especially in science and engineering programs. However, grading this kind of work can be prohibitively expensive in large classes. Because of the complexity of handwritten free-form work, there are no existing computational methods capable of interpreting it. In this dissertation, we aim to work toward complete interpretation and semantic analysis of handwritten free-form solutions to homework and exam problems of the sort assigned in undergraduate engineering and science courses. In our study, students wrote their homework and exam solutions on dot patterned paper with Livescribe smartpens. Our work comprises three innovations. First, we developed methods to locate final answers. Our methods locate answers by identifying the boxes drawn around them. Experiments demonstrated that our methods are both accurate and efficient at recognizing answer boxes. Second, we developed a novel CNN-BLSTM-CRF network for semantic labeling of students' handwritten assignments. Semantic labeling is the task of classifying pen strokes according to the type of content they represent. Our method distinguishes between cross-out strokes, equation strokes, and free body diagram strokes. The input to our network is a set of pen strokes, which are sorted in chronological order, and the output is a sequence of labels for the strokes. Labeling strokes in this manner is an important step in enabling the semantic analysis of the writing. Our labeling approach outperforms existing methods. Finally, we developed a novel GRU-CRF network to locate complete mathematical equations. The network exploits the temporal context of consecutive strokes and simultaneously finds and groups equation strokes. Once our method has located the equations, they can be interpreted using existing techniques. Together, this work provides a significant step toward the automated grading of handwritten free-form course work.