UCLA

This dissertation rethinks the problem of robot perception from an embodied agent's perspective: While the classic view focuses on perceiving the semantics and geometry of objects (e.g., this piece of point cloud is a fridge), our new perspective emphasizes perceiving the fluent (a condition that can change over time) that provides actionable information for enabling an agent to reason about actions an object affords as well as the potential outcomes of actions for planning in daily scenes. We address this challenging problem by understanding (i) the geometry fluent that accounts for the changes in object pose, (ii) the topology fluent that accounts for the changes in object form, and (iii) the interconnection between the geometry and topology fluent. Considering the task of chopping garlic, one needs to transform whole garlic into minced and transport them from one place to another. An agent that only recognizes geometry and semantics can hardly accomplish such a task. Therefore, a scene reconstruction framework is proposed to reconstruct a functionally equivalent and interactive scene from RGB-D data streams to afford finer-grained interactions of geometry fluent. To further understand the interactions of topology fluent, a probabilistic framework is devised to induce an attributed stochastic grammar that models the space of object form changes. This learned grammar and its probability model serve as a new indication of object status regarding topology fluent and are useful for planning downstream tasks. Finally, we study the interconnection between the geometry and topology fluent via a tool-use example where we learn the essential physical properties contributing to the effects of a tool-use event. By understanding potential actions in a scene, this dissertation aims to enable a robot to perceive the geometry and topology fluent and to plan their actions in daily scenes.