We present an effective field theory approach to the fracton phases. The approach is based on the notion of a multipole algebra. It is an extension of space(time) symmetries of a charge-conserving matter that includes global symmetries responsible for the conservation of various components of the multipole moments of the charge density. We explain how to construct field theories invariant under the action of the algebra. These field theories generally break rotational invariance and exhibit anisotropic scaling. We further explain how to partially gauge the multipole algebra. Such gauging makes the symmetries responsible for the conservation of multipole moments local, while keeping rotation and translations symmetries global. It is shown that upon such gauging one finds the symmetric tensor gauge theories, as well as the generalized gauge theories discussed recently in the literature. We refer to all such theories as multipole gauge theories. The outcome of the gauging procedure depends on the choice of the multipole algebra. In particular, we show how to construct an effective theory for the U(1) version of the Haah code based on the principles of symmetry and provide a two-dimensional example with operators supported on a Sierpinski triangle. We show that upon condensation of charged excitations, fracton phases of both types as well as various Symmetry-protected topological phases emerge. Finally, the relation between the present approach and the formalism based on polynomials over finite fields is discussed.