© 2015 American Physical Society. Here we present an efficient and numerically stable procedure for compressing a correlation matrix into a set of local unitary single-particle gates, which leads to a very efficient way of forming the matrix product state (MPS) approximation of a pure fermionic Gaussian state, such as the ground state of a quadratic Hamiltonian. The procedure involves successively diagonalizing subblocks of the correlation matrix to isolate local states which are purely occupied or unoccupied. A small number of nearest-neighbor unitary gates isolate each local state. The MPS of this state is formed by applying the many-body version of these gates to a product state. We treat the simple case of compressing the correlation matrix of spinless free fermions with definite particle number in detail, although the procedure is easily extended to fermions with spin and more general BCS states (utilizing the formalism of Majorana modes). We also present a density matrix renormalization group-like algorithm to obtain the compressed correlation matrix directly from a hopping Hamiltonian. In addition, we discuss a slight variation of the procedure which leads to a simple construction of the multiscale entanglement renormalization ansatz of a fermionic Gaussian state, and present a simple picture of orthogonal wavelet transforms in terms of the gate structure we present in this paper. As a simple demonstration, we analyze the Su-Schrieffer-Heeger model (free fermions on a one-dimensional lattice with staggered hopping amplitudes).