The environment around an autonomously navigated vehicle can have an unpredictablenumber of other vehicles and stationary or moving obstacles that may or may not have harmful intentions. The safe navigation of the autonomous vehicle in the presence of other vehicles and obstacles can be formulated as a stochastic optimal control problem. While in theory one can write down the corresponding Hamilton-Jacobi-Bellman (HJB) equation for any state space control problem, practically solving the equation is computationally infeasible when the state space is large. Moreover, once it is accounted for a time varying number of obstacles and other vehicles, and the associated time varying dimension of the state space, it is clear that new approaches to the design of vehicle navigation have to be considered. This work addresses the problem of autonomous navigation by a scalable integration of stochastic optimal control solutions to problems such as vehicle-to-vehicle, vehicle-to-obstacle, or vehicle-to-goal problems. The scalable navigation means that the autonomous vehicle or team of vehicles can navigate toward their goals while coping with a large number of other vehicles, or obstacles in their proximity. The work is based on the Dubins nonholonomic vehicle model and is illustrated by multiple scenarios in simulations and with real robots.