Collaborative Research: ITR - [ASE+ECS] - [dmc+int]: DDDAS - Advances in recognition and interpretation of human motion: An Integrated Approach to

Award Number: 428231

Program(s): ITR FOR NATIONAL PRIORITIES

Start Date: 10/15/2004

Principal Investigator: Metaxas, Dimitris

Co-PI Name(s): Vladimir Pavlovic Ahmed Elgammal

PI Email Address: dnm@cs.rutgers.edu

Abstract: NSF ITR title: ITR -[ASE+ECS] - [dmc+int]:

DDDAS Advances in Recognition and Interpretation of Human Motion:

An Integrated Approach to ASL Recognition

This project is aimed at advancing the state of the art in the field of computer-based American Sign Language (ASL) recognition. To date, sign language recognition has focused primarily on detecting individual signs (words), articulated primarily with the arms and hands. This is a major limitation, given that critical linguistic information-including grammatical features such as negation, agreement, and question status-is conveyed through "non-manual" linguistic markings. These non-manual markings include facial expressions (such as raised or lowered eyebrows, varying gaze and aperture of the eyes, wrinkling of the nose, and mouth movements) and gestures or periodic movements of the head (such as tilts, nods, and shakes). No system for sign language recognition or generation can succeed without properly modeling the linguistic information produced both manually and non-manually.

The fact that these critical non-manual behaviors occur in parallel with manual signing, and that they are temporally aligned with phrases rather than with individual signs, greatly complicates the task. Further problems arise because of the difficulties of tracking the minute details of human facial movements from video, and the variations in the specific realizations (style) of manual signs and non-manual linguistic markings across different individuals, just as there is variation in the specific ways in which individuals produce a given spoken language. A comprehensive approach to ASL recognition thus requires the integration of information from multiple data sources with different spatial and temporal scales, the application of linguistic knowledge about both the manual and the non-manual aspects of ASL, and the modeling of interdependencies of activities in the manual and non-manual channels.

This collaborative project brings together the expertise of researchers in the fields of computer vision, linguistics, and recognition to achieve its goals. On the computer vision side, the principal investigators (PIs) will investigate the use of local free-form deformations and novel registration methods to enhance our existing face tracking software, so as to capture the minute details of the facial movements, and to improve robustness of the tracking. The tracking process results in a large number of facial parameters, which the researchers propose to reduce through nonlinear subspace manifold embedding. This embedding reduces the dimensionality of the parameter space, and more importantly, also results in a separation of style and content. Whereas style is specific to each signer, content captures the commonalities across all signers. Hence, by focusing on the content component, the PIs expect to be able to overcome the variations across signers and perform signer-independent recognition.

On the recognition side, the researhcers will combine the linguistic knowledge about facial microactions with computational clustering approaches to develop the necessary statistical models for recognition. Initially, these will be based on Hidden Markov Models from previous work by this research group and elsewhere; however, their power to describe the dynamical aspects of human movements is limited. To overcome these limitations, the PIs will research the use of Switching Linear Dynamic Systems, augmented by Coupled Dynamic Bayesian Networks to model and capture the interactions of the simultaneously occurring microactions. Linguists and computer scientists will collaborate in exploring the best ways to leverage information about the linguistic organization of ASL for improvement of recognition strategies.

This research will be performed on the existing linguistically annotated corpus of the National Center for Sign Language and Gesture Resources, as well as new data to be collected from 5-8 native ASL signers, which will also be annotated over the course of the project. The annotations will be used for the linguistic modeling; they also provide the "ground truth" for performing and validating the computer vision and recognition research.

Broader impact:

The computer-based techniques for ASL can be extended to more general systems for sign language recognition and generation, as well as for interpretation of other types of human movements, such as face gesture recognition for HCI, surveillance, verification of identity, interrogation, interviews and medical diagnosis applications. The materials to be distributed will benefit researchers in linguistics, computer science, and other domains. There are immediate applications for primary and secondary education of the deaf and training of sign language interpreters. Improvements in multimedia (linguistic) information technology promise to offer expanded employment possibilities for the deaf, as well as improved access to vocational and post-secondary education. Finally, this project itself will provide a huge boost in terms of education, awareness and encouragement of deaf students in enabling them to work on cutting-edge research that directly affects them and their community.

A Better Book Browser: How to Read a Million Books

Award Number: 414557

Program(s): INFORMATION and KNOWLEDGE MANAGE

Start Date: 10/1/2004

Principal Investigator: Lesk, Michael

Co-PI Name(s): Nina Wacholder

PI Email Address: lesk@acm.org

Abstract: A new interface to traditional monographs, introducing structure, summaries,

and personalized indexes, is being built. This enables students to search

and use digital libraries more effectively. The interface should be useful

in the context of the Million Book Project which will provide one million

out of copyright books in digital form, the equivalent of a major research

library. Key ingredients in this interface are selection of relevant

phrases, automatic summarization, and the ability to search and summarize

in the context of a particular information need.

The impact of this work includes access to the enormous contents of our

older libraries, improving our ability to use monographic material in

online teaching, and greater understanding of how students learn by

reading. In addition, we will employ the technology for teaching in our

university, helping educate our students in communication and library

studies.

Collaborative Research: Investigating the impact of microbial interactions with geologic media on geophysical properties: Implications for assessing g

Award Number: 433729

Program(s): BE-UF: BIOGEOSCIENCES, BE: NON-ANNOUNCEMENT RESEARCH

Start Date: 10/1/2004

Principal Investigator: Slater, Lee

Co-PI Name(s):

PI Email Address: lslater@andromeda.rutgers.edu

Abstract: Bacteria have been shown to play an important role in geologic processes, however, their role in altering geophysical properties of rocks is not well understood, nor has it been thoroughly investigated. This project is a three-year collaboration between researchers at the University of Missouri-Rolla, Rutgers University, and Western Michigan University. Its purpose is to understand and measure geophysical changes resulting from microbial interactions with geologic media.

Specific objectives of the project are to conduct laboratory and field studies to investigate: (1) the effect of increases in microbial cell concentrations and biofilm formation on soil and sediment electrical properties, (2) the effect of metabolic by-products of microbial activity, such as biosurfactants and organic acids, on geophysical electrical measurements, (3) potential changes in petrophysical properties (e.g., permeability, porosity, surface area) induced by microbial-mineral interactions, and (4) differences in the microbial communities and their structure, dynamics, and associations in sediments with anomalous geophysical signatures. The first phase of the work will involve measuring the electrical signatures of bacterial cells, biofilms, and organic acids in laboratory column reactors. Final reactor products will then be imaged with photomicrography to investigate biofilm distribution and changes in pore geometry. The second phase of the work will focus on measurements of reactor sediment physical properties (formation factor, surface area, porosity, permeability). Results of this work will help build a database to relate electrical, physical, and biochemical parameters that can be used in geophysical modeling of field input data. The final phase of the program will concentrate on field measurements and observations on how changes in sediment electrical conductivity can be related to microbial alteration of geologic materials. These analyses will be carried out on cores and at the field scale. This study will form the basis for the development of geophysics as a tool for investigating geomicrobiological processes.

Broader and potential societal benefits include development of geophysical techniques that can be used for monitoring and assessing biological colonization of groundwater aquifers and microbial mineralization of dissolved pollutants in the near subsurface. Educational and outreach initiatives will focus on student involvement and student-led promotion of biogeophysics to the wider Geophysics and Biogeosciences community at national mad international meetings.

Collaborative Research: ITR-(ASE+EVS)-(dmc+sim) Data Driven Simulation of the Subsurface: Optimization and Uncertainty Estimation

Award Number: 426354

Program(s): ITR FOR NATIONAL PRIORITIES

Start Date: 10/1/2004

Principal Investigator: Parashar, Manish

Co-PI Name(s):

PI Email Address: parashar@caip.rutgers.edu

Abstract: Intellectual Merit. Remote sensing is employed in science and engineering problems to infer material properties when these properties can not be directly sampled. To better understand and manage our environment for safety and economic reasons, much progress has been made in imaging the subsurface and estimating physical properties based on remote sensing data. Repeated observations over targets for environmental remediation and reservoir production has become a recognized diagnostic tool for assisting management decisions. In addition, improved optimization techniques capable of responding to large, multi-resolution, disparate, dynamic datasets in a fault tolerant and adaptive fashion are a fundamental requirement for effectively estimating and minimizing the uncertainty in any data driven application. The integrated and e_ective treatment of these issues motivates the present project. The assembled research team proposes to advance the mathematical, engineering and computational foundations necessary to enhance our understanding and extend the predictive capabilities of the physical processes that govern the subsurface phenomena at multiple temporal and spatial scales. Target applications include management of aquifers for water resources, optimizing oil and gas production, and monitoring environmental risks e.g., at waste containment sites or arising from natural hazards.

The intellectual merits of the project include: (1) development of the next generation of accurate, multi-scale, coupled chemical, uid, geomechanical, and geophysical simulators for modeling instrumented subsurface environments; (2) large scale optimization techniques (based on a hybridization of global and local approaches) to drive reliable decision-making and a dynamic symbiotic feedback between computation and data; (3) deployment of an autonomic Grid middleware for providing the adequate processing substrate and data management services for (1) and (2). The realization of the above contributions will result in the Data Driven Subsurface Simulation Framework (DDSSF).