We present a novel Material Point Method for simulating complex materials. The method achieves plasticity effects via the temporal evolution of the left elastic Cauchy-Green strain. We recast the upper-convected derivative of the strain in the Oldroyd-B constitutive model as a plastic flow and are able to simulate elastic and viscoelastic effects. Our model provides a volume-preserving rate-based description of plasticity that does not require singular value decompositions. Our semi-implicit discretization allows for high-resolution simulations. We also present novel discretizations of the temporal update of the left elastic Cauchy-Green strain for several constitutive models that preserve symmetry and positive-definiteness of the strain for use in the Material Point Method. A novel modification to a constitutive model is also presented that models material softening under plastic compression.