The introduction of serum Prostate Specific Antigen (PSA) testing nearly 30 years ago has been associated with a significant shift towards localized disease and decreased deaths due to prostate cancer. Recognition that PSA testing has caused over diagnosis and over treatment of prostate cancer has generated considerable controversy over its value, and has spurred efforts to identify prognostic biomarkers to distinguish patients who need treatment from those that can be observed. Recent studies show that cancer is heterogeneous and forms a hierarchy of tumor cell populations. We developed a method of identifying prostate cancer differentiation states related to androgen signaling using Boolean logic. Using gene expression data, we identified two markers, CD38 and ARG2, that group prostate cancer into three differentiation states. Cancers with CD38-, ARG2- expression patterns, corresponding to an undifferentiated state, had significantly lower 10-year recurrence-free survival compared to the most differentiated group (CD38+ARG2+). We carried out immunohistochemical (IHC) staining for these two markers in a single institution (Stanford; n = 234) and multi-institution (Canary; n = 1326) cohorts. IHC staining for CD38 and ARG2 in the Stanford cohort demonstrated that combined expression of CD38 and ARG2 was prognostic. In the Canary cohort, low CD38 protein expression by IHC was significantly associated with recurrence-free survival (RFS), seminal vesicle invasion (SVI), extra-capsular extension (ECE) in univariable analysis. In multivariable analysis, ARG2 and CD38 IHC staining results were not independently associated with RFS, overall survival, or disease-specific survival after adjusting for other factors including SVI, ECE, Gleason score, pre-operative PSA, and surgical margins.