Photo Magnetic Imaging (PMI) is a novel laser-based optical imaging technique. PMI is used to identify diseased tissue based on its endogenous tissue contrast or using disease-targeting exogenous optical contrast agents. It was first developed at the Center for Onco Functional Imaging at the University of California, Irvine. Optical imaging techniques generally suffer from poor spatial resolution due to the high scattering of optical photons in tissue. PMI attempts to provide quantitatively accurate optical images with high spatial resolution. In this study, phantoms mimicking optical properties of tissue are utilized to explore the use of Machine learning techniques and Finite Element Methods to drive an AI-based reconstruction of PMI images. Our proposed methodology consists of hybrid deep learning and a machine learning pipeline that first identifies the inclusions in the phantoms representing cancerous lesions and finally delineates their boundaries. Our method was tested on a variety of inclusions’ sizes, locations, and absorption coefficients and demonstrated high precision (above 95%) and Intersection Over Union Accuracies (above 85%) for the realizable test cases.