Stefan Müller Arisona, research fellowship at UCSB, 2007 – 2008
Project Summary
The volume of digital media content has enormously grown in the recent years: Emerging internet-based applications, such as YouTube or MySpace, are media-rich, emphasise on content sharing, and attract huge communities. As the volume of content further expands, we see an increasing need for not only being able to annotate and retrieve content, but also to produce and edit content in a computer-assisted manner and by employing (semi-) automated methods. However, existing work so far does not satisfy these needs — both in terms of usable software instruments as well the underlying theoretical frameworks. This viewpoint is also acknowledged by upcoming European Union IST projects: The Framework Programme 7 (FP7) calls for projects addressing Intelligent Content and Semantics and targets at advanced authoring environments for interactive and expressive content [ICT Programme Committee, 2007, Objective ICT-2007.4.2].
This project aims at a systematic classification and implementation of novel content authoring techniques. By “mixed-media content authoring” we understand the interactive editing, or the ”composition”, of content for multiple modalities (e.g., auditory and visual). The project emphasises on applying computational methods in order to enable a high degree of automatisation of the authoring process. It further aims at techniques that are concerned with expressive human-computer interaction and those that need to operate in real-time, where needed.
Central to the project is the hypothesis that existing traditional composition techniques from individual domains (e.g., music, graphics, video) can be identified, classified, and then be generalised in a foundational framework of interactive media composition techniques that is applicable to automated authoring tasks. We claim that the implementation of these techniques will lead to more effective methods for simultaneous mixed-media authoring. The project starts out by establishing a catalogue of media composition techniques. This includes collecting and classifying existing techniques from specific modalities, identifying common methodologies, and also filling gaps and building bridges with new techniques where needed. Before actually implementing the composition techniques, an appropriate real-time content rendering infrastructure is required. Here, the project benefits from preliminary work carried out at the former Multimedia Laboratory of the University of Zurich and at the Computer Systems Institute of ETH Zurich. As its central part, the project implements relevant composition techniques as part of an interactive media authoring instrument. Special thoughts are given to a human-centred approach in order to prevent results that are stigmatised by software, and in order to provide the highest possible degree of expressiveness to content authors. At its final stage, the project empirically verifies the results by means of concrete application scenarios from media art and entertainment.
The project brings together elements from many different fields, primarily from software engineering, human-computer interaction, and the foundations of multimedia, but also from music composition, film editing theory, and aesthetics. The intellectual merit of the project is to provide systematic foundational work on mixed-media composition techniques by exploiting computational methods and to validate the practicability of the theory in terms of usable software instruments. Besides of dissemination of scientific findings, the broader impacts of the project are expected to be very significant for a wide range of applications that depend on computer-assisted and automated content creation techniques: Among others, example beneficiaries are the entertainment industry or commercial content providers who can benefit from novel expressive authoring instruments, or emerging services beyond IPTV, where automated content composition tasks help supporting whole communities. Collectively, the project aims at advancing the state of the art of interactive media content authoring to a new level.
Project: Post-doctoral research fellowship, fully funded by the Swiss National Science Foundation (“Fellowship for Advanced Researchers”)
Location: Media Arts & Technology, University of California, Santa Barbara
Period: 2007 – 2008