A search for alpha and double beta decays of ytterbium isotopes was performed with an ultra low-background high purity germanium detector at Gran Sasso Underground Laboratory (Italy). A 194.7 g Yb2(C2O4)3 powder sample was measured for 11.3 days with a total Yb exposure of 1.25 kg× day. Half-life limits for α-decay modes of 168Y b, 170Y b, 171Y b, 172Y b, 173Y b, 174Y b and 176Y b into the first excited states have been obtained between 6 × 10 14 years and 2 × 10 16 years. These are the first experimental constraints of these decay modes. Double electron capture of 168Y b and double beta decay of 176Y b into the first excited 2+ and 0+ states could be excluded with limits between 1 × 10 14 years to 8 × 10 16 years. This improves the experimental information on some of the decay modes compared to previous constraints.