Cilia, organelles that move to execute functions like fertilization and signal to execute functions like photoreception and embryonic patterning, are composed of a core of nine-fold doublet microtubules overlain by a membrane. Distinct types of cilia display distinct membrane morphologies, ranging from simple domed cylinders to the highly ornate invaginations and membrane disks of photoreceptor outer segments. Critical for the ability of cilia to signal, both the protein and the lipid compositions of ciliary membranes are different from those of other cellular membranes. This specialization presents a unique challenge for the cell as, unlike membrane-bounded organelles, the ciliary membrane is contiguous with the surrounding plasma membrane. This distinct ciliary membrane is generated in concert with multiple membrane remodeling events that comprise the process of ciliogenesis. Once the cilium is formed, control of ciliary membrane composition relies on discrete molecular machines, including a barrier to membrane proteins entering the cilium at a specialized region of the base of the cilium called the transition zone and a trafficking adaptor that controls G protein-coupled receptor (GPCR) localization to the cilium called the BBSome. The ciliary membrane can be further remodeled by the removal of membrane proteins by the release of ciliary extracellular vesicles that may function in intercellular communication, removal of unneeded proteins or ciliary disassembly. Here, we review the structures and transport mechanisms that control ciliary membrane composition, and discuss how membrane specialization enables the cilium to function as the antenna of the cell.