IPSOM

From L²F

Indexing, Integration and Sound Retrieval in Multimedia Documents (2000-2004).

  • Sponsored by: FCT (POSI/PLP/34252/2000)
  • Start: November 2000
  • End: December 2004

Team

Project Leader: António Serralheiro (L²F, INESC-ID Lisboa)

Summary

The IPSOM project had as its goal improving the access to digital spoken books. In our country, the main users of digital spoken books are visually-impaired citizens. This is not the typical situation in other countries where spoken books are much more widely used (for instance, when driving). The development of tools that allow the easy detection and indexation of units (words, sentences, topics) may help to spread even further the usage of digital spoken books.

This project, which started in November 2000, had a planned duration of 3 years that was later extended to 4 years. It involved 3 different teams: the Spoken Language Systems Lab of INESC-ID, the LASIGE Group of the Faculty of Sciences of the University of Lisbon and the National Library. The participation of these three teams can be briefly summarized as follows: the National Library acted as a provider of the linguistic resources to be processed in this project - the spoken books; LASIGE had as its main task the development of multimedia interfaces for indexing and retrieval; and L2F had as its main task the alignment of the spoken and written versions of the books.

This project resulted in several software modules:

  • Automatic speech-text alignment systems with and without speaker adaptation (see demo);
  • Multimedia interface for indexing digital spoken books;
  • Plataform for the production of spoken books.

LIFAPOR is a follow-up project of IPSOM, in which L2F's main task is to test and adapt the alignment modules for spoken books read by Brazilian speakers.

Publications

Book Chapters

  • Luís Carriço, Carlos Duarte, Rui Lopes, Miguel Rodrigues, Nuno Guimarães. Building Rich User Interfaces for Digital Talking Books. Book Chapter in Computer-Aided Design of User Interfaces IV, Jacob, Robert; Limbourg, Quentin; Vanderdonckt, Jean (Eds.), Kluwer Academic Publishers, 2005 (accepted in 2003), XI, 374 p., Hardcover (ISBN 1-4020-3145-9).

Journals

  • Carlos Duarte, Luís Carriço, Hugo Simões, Teresa Chambel, Nuno Guimarães. «Avaliação de aspectos de sincronização de Livros Falados Digitais». Special issue with extended versions of the best papers presented at Coop-Media 2003 - Workshop de Sistemas de Informação Multimédia, Cooperativos e Distribuídos. APSI (accepted for publication).

Conference Proceedings

  • Carlos Duarte e Luís Carriço, «Usability Evaluation of Digital Talking Books», in Interacção'2004, 1st National Conference in Human-Computer Interaction. July 2004.
  • Carlos Duarte, Luís Carriço, Hugo Simões, «A Flexible Interface Architecture for Digital Talking Books», in Proceedings of the 6th International Conference on Enterprise Information Systems, ICEIS 2004, Porto, Portugal, April 2004.
  • Luís Carriço, Carlos Duarte, Nuno Guimarães, António Serralheiro e Isabel Trancoso, «Modular Production of Flexible Digital Talking Books», in Proceedings of the 6th International Conference on Enterprise Information Systems, ICEIS 2004, Porto, Portugal, April 2004.
  • Carlos Duarte, Luís Carriço, «Identifying Adaptation Dimensions in Digital Talking Books», in Proceedings of the 8th ACM International Conference on Intelligent User Interfaces, IUI 2004, Madeira, Portugal, January 2004.
  • Luís Carriço, Carlos Duarte, Rui Lopes, Miguel Rodrigues, Nuno Guimarães, «Building Rich User Interfaces for Digital Talking Books», in Proceedings of the Computer-Aided Design of User Interfaces IV, Madeira, Portugal, January 2004.
  • Carlos Duarte, Teresa Chambel, Luís. Carriço, Nuno Guimarães, Hugo Simões, «A Multimodal Interface for Digital Talking Books», in Proceedings of the IADIS International WWW/Internet 2003 Conference. Algarve, Portugal, November 2003.
  • Carlos Duarte, L. Carriço, Hugo Simões, Teresa Chambel, N. Guimarães, «Avaliação de aspectos de sincronização de Livros Falados Digitais» in Proceedings of the Coop-Media 2003, Workshop de Sistemas de Informação Multimédia, Cooperativos e Distribuídos, ISEP, Porto, October 2003.
  • A. Serralheiro, I. Trancoso, D. Caseiro, T. Chambel, L. Carriço e N. Guimarães, «Towards a Repository of Digital Talking Books», EUROSPEECH 2003 - 8th European Conference on Speech Communication and Technology (Interspeech'2003), Geneve, Switzerland, September 2003.
  • Carlos Duarte, Luís Carriço, Teresa Chambel, Nuno Guimarães, «Producing DTBs from Audio Tapes», in Proceedings of the 5th International Conference on Enterprise Information Systems. Angers, France, April 2003.
  • Luís Carriço, Nuno Guimarães, Carlos Duarte, Teresa Chambel, Hugo Simões, «Spoken Books: Multimodal Interaction and Information Repurposing», in Proceedings of the HCI International Conference, Lawrence Erlbaum Associates, Inc. Crete, Greece, June 2003.
  • A. Serralheiro, D. Caseiro, H. Meinedo, I. Trancoso, «Word Alignment in Digital Talking Books Using WFSTs», ECDL2002 - European Conference on Digital Libraries, Rome, Italy, September 2002, Lectures in Computer Science, Springer-Verlag GmbH & Co. KG. Editor: M. Agosti e C. Thanos, 2002.
  • D. Caseiro, H. Meinedo, A. Serralheiro, I. Trancoso, J. Neto, «Spoken Book Alignment Using WFSTs», HLT'2002 - Human Language Technology Conference, San Diego, California, March 2002.
  • A. Serralheiro, I. Trancoso, «Alinhamento de Livros Falados», XVII Encontro Nacional da Associação Portuguesa de Linguística, Lisboa, October 2001.

Pilot Corpus

In order to test the software developed in IPSOM, we have built a pilot corpus consisting of different types of aligned books:

  • Fiction:
    • O Senhor Ventura, by Miguel Torga, read by Isabel Bahia
  • Poetry:
    • Um Momento de Palavras, David Mourão Ferreira, read by himself
    • Manifesto Anti-Dantas, Almada Negreiros, read by Mário Viegas
    • Fado Falado, Aníbal Nazaré, read by João Villaret
  • Children's stories:
    • O Monge Desastrado, included in Histórias de Belém, Ana Maria Magalhães e Isabel Alçada, read by Jorge Moreira and Juva Batella (European and Brazilian Portuguese, respectively)
  • School books:
    • Chapter of a History book (2nd cycle), read by Jorge Moreira and Ana Bernardino

Demos

  • O Senhor Ventura, by Miguel Torga, read by Isabel Bahia [1]
  • O Monge Desastrado, included in Histórias de Belém, Ana Maria Magalhães e Isabel Alçada, read by Jorge Moreira [2] (European Portuguese) and Juva Batella [3] (Brazilian Portuguese)
  • O Monge Desastrado: [4] Parallel audio book (Audio in Portuguese / Text in English)