Self-report is currently used as an indicator of professional practice in a variety of fields, including medicine and education. Important to consider, therefore, is the ability of self-report to accurately capture professional practice. This study investigated how well professionals' self-reports of behavior agreed with an expert observer's reports of those same behaviors. While this study explored self-report in the context of medical professionals, this topic is equally important to the measurement of teacher practices.

This study investigated agreement between: 1) medical student self-report and expert rater documentation of a clinical encounter; and 2) standardized patient (an actor highly-trained to portray a patient) and expert rater documentation of medical student performance. Additionally, this study investigated whether levels of agreement depended on the context and content of behaviors, features of the examination, or characteristics of the professional.

Performance data were analyzed from a stratified random sample of 75 fourth-year medical students who completed a clinical competence examination in 2012. Students rotated through a series of 15-minute encounters, called stations, interviewing a standardized patient in each. Medical students were instructed to: 1) obtain the patient's history; 2) conduct a physical examination; and 3) discuss potential diagnoses. Ratings of student performance were collected from the medical student self-reports, the SP checklists, and the expert rater's documentation of the encounters. Analyses focused on the 4-7 behavioral items in each of the three stations studied that were considered critical to patient care.

Comparison of the three sources of ratings revealed marked differences. Most importantly, medical students' self-reports did not agree highly with the expert's reports. Medical students both under-reported and over-reported a substantial number of critical action items with level of agreement varying by station and the nature of the behavior. Due to medical student tendency to under-report behaviors, use of self-report to score performance would result in a large number of students falsely identified as failing the examination.

This study discusses causes of medical student under- and over-report and recommends strategies for improvement. The study also addresses implications of findings for the use of self-report among teachers, citing specific examples.