2016-016 – Event Labeling through Analytic Media Processing

Building on state-of-the-art audio and video processing, along with general language analysis techniques, E-LAMP is an interactive system for multimedia event detection.The invention describes a prototype capable of (1) analyzing massive numbers of digital multimedia files for the presence or absence of specific events, which are (2) stored in a representation that lends itself to efficient processing, supported by (3) an interface suitable for domain specialists, using “event kits”. Due to the large amounts of data to be searched, the original data is processed only once, and is stored in an efficient index structure, which includes semantic information, and general knowledge stored in shallow ontology stubs. The system extends on the state-of-the-art in audio processing (speech-to-text, acoustic scene analysis, meta-data extraction), video processing (video concept and activity analysis, optical character recognition), and general language analysis techniques. From the information extracted from the original video data, the system creates an indexing representation, which is semantically rich enough to support event kits (text queries which are optionally supported by multimedia examples) for multimedia events. The semantic representation is high-level, in that items are represented as meaningful concepts, so that queries about multimedia events can be answered quickly and with high recall, yet it is also detailed, guaranteeing precision and allowing generation of feedback in a probabilistic framework. Cindy Lou Chepanoske cchepano@andrew.cmu.edu 412-268-2979