© 2018. The American Astronomical Society. All rights reserved.. There has been speculation about a class of relativistic explosions with an initial Lorentz factor Γinit smaller than that of classical gamma-ray bursts (GRBs). These "dirty fireballs" would lack prompt GRB emission but could be pursued via their optical afterglow, appearing as transients that fade overnight. Here we report a search for such transients (that fade by 5-σ in magnitude overnight) in four years of archival photometric data from the intermediate Palomar Transient Factory (iPTF). Our search criteria yielded 50 candidates. Of these, two were afterglows to GRBs that had been found in dedicated follow-up observations to triggers from the Fermi GRB Monitor. Another (iPTF14yb) was a GRB afterglow discovered serendipitously. Eight were spurious artifacts of reference image subtraction, and one was an asteroid. The remaining 38 candidates have red stellar counterparts in external catalogs. The photometric and spectroscopic properties of the counterparts identify these transients as strong flares from M dwarfs of spectral type M3-M7 at distances of d ≈ 0.15-2.1 kpc; three counterparts were already spectroscopically classified as late-type M stars. With iPTF14yb as the only confirmed relativistic outflow discovered independently of a high-energy trigger, we constrain the all-sky rate of transients that peak at m = 18 and fade by Δm = 2 mag in Δt = 3 hr to be , with a 68% confidence interval of . This implies that the rate of visible dirty fireballs is at most comparable to that of the known population of long-duration GRBs.