In this paper, we discuss the electromagnetic interaction between a point charge travelling inside a waveguide of elliptical cross section, and the waveguide itself. By using a convenient expansion of the Mathieu functions, useful in particular for treating a variety of problems in applied mathematics and physics with elliptic geometry, we first obtain the longitudinal electromagnetic field of a point charge (Green's function) in free space in terms of elliptical coordinates. This expression allows, then, to calculate the scattered field due to the boundary conditions in our geometry. By summing the contribution of the direct or primary field and the indirect field scattered by the boundary, after a careful choice of some expansion expressions, we derive a novel formula of the longitudinal electric field, in any transverse position of the elliptical cross section, generated by the charge moving along the longitudinal axis of the waveguide. The obtained expression is represented in a closed form, it can be differentiated and integrated, it can be used to fully describe the radiation process of a particle beam travelling inside a waveguide of elliptical cross section, and it is valid for any elliptic geometry. The equations are used to evaluate the coupling impedance due to indirect space charge in case of elliptical geometry. In addition, they are useful as preliminary studies for the determination of the coupling impedance in different cases involving elliptic vacuum chambers, as, for example, the effect of the finite conductivity of the beam pipe wall or the geometrical variation of the vacuum chamber due to elliptic step transitions existing in some accelerators.