Autonomous dexterous manipulation (i.e., reorienting objects with the fingertips) remains beyond the grasp of robots. Dexterous manipulation (e.g., picking up a lemon to squeeze it) differs from grasping an object because it requires re-orienting the object with the fingertips without dropping it. Prior work has demonstrated autonomous learning tomanipulate objects, but dropping them is prevented by an upward-facing palm, a tabletop, or slowly introducing gravity. In this dissertation, I demonstrated autonomous learning of dexterous manipulation of a ball with a downward-facing palm against gravity. I use a reinforcement learning algorithm (proximal-policy optimization) to demonstrate that a simulated downward-facing three-fingered robotic hand can autonomously learn to reach for and manipulate a 5 gram ball while rotating it at the desired height. Importantly, only curricula that rewarded ball rotation from the start succeeded. This dynamic interaction with the ball in the absence of vision, like child’s play, is likely a form of active sensing necessary to build useful end-to-end models for dexterous manipulation against gravity. Tactile information was useful in interesting ways for this ball: Lacking tactile information hindered—but did not prevent—learning, while Binary-contact and Normal-force sensing performed comparably to 3D-force. This dynamical interplay among curricula, tactile information and learning trends illuminate features of human manipulation and provide a path towards autonomous reach and manipulation by robots.