A number of authors have proposed using live performances to drive computer animations. We strive to bring the expressiveness of these approaches to general purpose animation systems running on graphics workstations. Instead of relying on advanced motion capture devices, we exploit our fully 3D user-interface to control the animated environment at a higher level of abstraction. The guiding approach proposed in also seeks to provide better control of synthetic objects by raising the abstraction level of user interaction. That work concentrates on modeling complex behaviors in a discrete simulation framework, while we focus on providing intuitive user interfaces. A major limitation of current performance animation systems is the inability to build editable representations out of captured performances