Transmission in optical fiber offers long distance and high-speed, possibly tens of Terabits per second, optical communications. The utilization of the theoretically available bandwidth and/or the capacity is hindered by a number of impairments. Traditionally, these impairments are mitigated in the optical domain. The main subject of the dissertation is the investigation of performance and applicability of advanced equalization and coding methods in high-speed optical communications, which relies on processing in the electronic domain. In particular, the investigated approaches rely either on processing of the received signal in order to recover information distorted in transmission, or on encoding and/or shaping of the transmitted signal in a specific way that mitigates the signal degradations imposed by the channel. The specifics of the fiber-optic channel require that the well established approaches in other areas of communication theory, upon which the presented work leverages, be adjusted to the conditions of this communication channel (e.g. nonlinear effects in transmission, square-law detection and signal statistics). In the dissertation, firm theoretical foundation for the analysis of electronic equalization is established. Additionally, the proposed and theoretically analyzed techniques are validated experimentally, demonstrating feasibility of the proposed techniques and providing corroboration of the theoretical predictions