International Society for CNS Clinical Trials and Methodology convened an expert working-group that assembled consistency/inconsistency flags for the Positive and Negative Syndrome Scale (PANSS). Twenty-four flags were identified and divided based on extent to which they represent error (Possibly, Probably, Very probably or definitely). The flags were applied to assessments derived from the NEWMEDS data repository and the CATIE clinical trial data. Almost 40% of ratings had at least one inconsistency flag raised and 10% had two. Application of flags to clinical rating can improve reliability and validity of trials.