Earlier this month, the FBI began rolling out a $1 billion update to the national fingerprinting database. Facial-recognition systems, DNA analysis, voice identification and iris scanning will all contribute to the government’s arsenal of Next Generation Identification (NGI) data. We asked Raymond Fu, a new assistant professor with joint appointments in the College of Engineering and the College of Computer and Information Science, to explain the science behind one of these new technologies: facial-recognition software.

How does facial recognition work, and where is the state of the art now?

Face-recognition research has been popular for more than two decades. Great advances have been made from researchers from a broad community, such as biometrics, computer vision and machine learning. The state-of-the-art techniques have been applied to real-world systems for applications in surveillance, security and forensics. Face recognition is a technology that requires high accuracy, especially when security and forensics factors are considered. The current challenges are scalability of databases; large variation factors in different environments; aging, makeup and pose factors of faces; and faces in social-media spaces.

Face-recognition systems start with face detection and tracking. Computational algorithms detect face positions and poses in an image and then extract them for processing and analysis. During this pipeline, a couple of major challenges create bottlenecks for the performance of real-world systems. Facial expressions, aging and makeup are key variations that cannot be easily removed. Techniques of 3-D morphable modeling and local features have been developed to mitigate such variations. Lighting variations can significantly affect the recognition accuracy especially when a system is used outside. Benchmark databases have been collected from well-controlled lighting sources for developing lighting insensitive feature extraction and analytical modeling for such purposes.

The increasing accessibility of the social-media space presents yet a new challenge to developing a large-scale identity database. Confusion of similar appearances, overload computations and multiple data sources bring up uncertainties in modern face recognition. Additionally, new trends of soft-biometrics, big data and multimodality face recognition have opened up new research thrusts.

What are the challenges and differences between identifying a single presented face and picking faces out of a crowd?

Face recognition and identification are two different problems. Face recognition is to match a person’s face against a set of known faces and identify who he or she is. For example, in a criminal investigation, a detective may want to ID a suspect from a face image captured on a surveillance camera.

Identification is to validate the match of a given face and the claimed ID. For example, if an employee wants to access a secured area in a classified department, she shows her ID card to the sensor while a camera captures her face to match it with the record retrieved from the ID card input. If the match passes, the door will open automatically.

How would you address concerns raised by privacy advocates?

Face recognition can be either passive or active. In the airport, for example, the surveillance cameras are taking videos in real time. Passengers’ faces are captured in a passive way. Online social-media spaces, like Facebook, provide public domains for users to share their photos in an active way. Both may involve privacy issues. How to balance the privacy issues and the public needs of security and human-computer interaction are new research topics in this era.

In my research group, we have been funded by Air Force Office of Scientific Research, IC Postdoc Fellowship and Google Research on these issues. Our research is mainly focused on understanding social status and networking of social-media users and their privacy concerns. We are working on new computational methodologies that could well analyze the visual content of social media and provide automatic solutions for human-computer interaction that could advance future social-network ecosystems.