Dr. Stephanie Marie Schwartz

Dr. Stephanie Marie Schwartz

Grant Title

Exploiting Communicative Signals to Summarize Information Graphics

Faculty Involved

Sandra Carberry (University of Delaware, Principal Investigator), Stephanie Elzer (Senior Personnel)


May 1, 2006 - May 1, 2009

Funding Agency

National Science Foundation




Information is the key to knowledge and effective decision-making. But information is useful only if it is accessible in a form that can be easily assimilated. Unfortunately, the growing use of visual information displays, particularly information graphics (bar charts, line graphs, grouped bar charts, etc.), poses problems for individuals with sight impairments. Since current technologies that attempt to reproduce a graphic in an alternative medium (such as tactile or sound) have been unsuccessful at facilitating effective access, it is imperative that novel, innovative approaches be investigated that might have a dramatic impact on accessibility.

This project is a unique combination of research in 1) communication, natural language generation, and dialogue, and 2) universal access. The project will develop an innovative methodology for enabling effective access to, and assimilation of, information graphics via an interactive natural language system. Rather than translating a graphic into an alternative medium so that it can be "viewed" via a non-visual sense, the project will take the approach of providing the user with the knowledge that could be gleaned from the graphic. A prototype system, embedded in a web browser, will convey an initial summary of the graphic and will respond to followup questions for more detailed information. The system will be implemented and thoroughly evaluated, both in terms of its absolute success in conveying information graphics and in comparison with other alternative communication systems. In addition to taking an innovative approach that offers promise in significantly advancing access to information graphics, the proposed methodology has the advantage of not requiring expensive equipment. The project will use the insights of blind individuals, both as part of focus groups providing suggestions on system design and as part of user groups evaluating the prototype system.

The intellectual merit of the proposed project includes a better understanding of graphical communication, new developments in text organization and interactive dialogue, and most importantly, innovative advances in technology for providing effective access to information graphics for individuals with sight impairments. The project will have a broader impact beyond the immediate project goals. The technology developed under the project will provide a means of automatically generating "alt" text to accompany information graphics. More generally, the technology developed by this project will provide a means of accessing graphical information on devices with miniature viewing facilities, such as cellular telephones for accessing the Web. Moreover, the project's methodology will provide insights into the development of technology for accessing other kinds of visual displays. The project will contribute significantly to the development of human resources by providing research opportunities for both graduate and undergraduate students. Although participation on the project will be open to any qualified student, particular attention will be given to including minority students and students with disabilities.