To allow wide-spread adoption of consumer robotics, robots must be able to adapt to their
environment by learning new skills and communicating with humans. Each chapter explains a
contribution to achieve this goal. Chapter One covers a stochastic And-Or knowledge
representation framework for robotic manipulations. Chapter Two further expands this
established system for robustly learning from perception. Chapter Three unifies perception with
natural language for a joint real-time processing of information. We've successfully tested the
generalizability and faithfulness of our robotic knowledge acquisition and inference pipeline. We
present proof of concepts in each of the three chapters.