A monolithic method for numerically solving fluid--structure interactions between viscous flow on an unbounded domain and rigid body system is presented. The vorticity form of incompressible Navier--Stokes equations is solved on a uniform Cartesian grid with immersed boundary projection method. The Lattice Green's function based integrating factor technique developed by Liska and Colonius is used to analytically evaluate the viscous diffusion term. Half-explicit Runge--Kutta method is applied for both fluid and rigid body system in time marching. The overall fluid--body saddle point system is designed to be arranged in a way to reveal an added-mass term for viscous flow in the solving procedure of block LU decomposition. The added-mass term is inherently embedded into the coupled system, compared to its explicit use in previous fluid--structure interaction studies. Because the added-mass augments inertia of the body system, the algorithm is proved to be stable for arbitrary small density ratios including zero mass case. Interaction force and boundary conditions on the interface are kept implicitly and solved as internal variables , and no iteration process or parameter tuning is needed. Both passive and active motions of rigid body system are allowed, with joint constraint and active motions enforced through Lagrange multipliers. With a unified treatment, the interaction force on the immersed surface is also enforced through Lagrange multipliers. Multiple applications are demonstrated, including cylinder rising/falling under gravity with different mass ratios, actively oscillating airfoil in a free stream, free swimming of an articulated fish, and self-excited oscillation of an articulated flag. Comparisons are made with previous studies and good agreement is found. Another application of a two-component linked plates model is studies for physical behaviour of bifurcation, hysteresis and energy harvesting. The algorithm is versatile in the sense of implementation. All aforementioned applications can be set up with very little change in the code, benefit from the use of rigid body dynamics in spatial vector form and numerical techniques in this work.