Will Your Face Recognition System Still Work with Masks?

As the US and worldwide response to the COVID-19 pandemic continues, wearing of surgical and homemade masks in public settings has become ubiquitous. As face recognition systems regularly encounter masked individuals – a scenario that was not necessarily expected during design and testing of face systems – one question dominates all others.

How is the performance of face recognition algorithms and systems affected by this new reality? 

To answer this question, we must investigate the current status, capabilities, and limitations of the face recognition industry. Then we must ask how changes in industry focus – due to the pandemic – could affect the industry going forward. 

With this in mind, let’s consider the following issues. 

Subjects wearing face masks is a specific case of a known issue: occlusion

As long as there have been face recognition algorithms, there have been strategies to handle occlusions. Deep Convolutional Neural Networks (CNNs), which has transformed the accuracy of face recognition algorithms, perform well even with partial or limited amount of data.

Research continues to find the best methods to ignore occlusions in face images. For example, 2019 research from Tencent AI Labs and Tsinghua University describes techniques that improve occlusion handling in facial recognition, which is applicable when the subject is wearing sunglasses, a face mask, scarves, or when other random objects obscure the face. 

Face recognition performance will vary by algorithm 

To paraphrase from NISTIR 8280, Face Recognition Vendor Test (FRVT), Part 3: Demographic Effects, ‘the algorithm matters.’

Some algorithms will handle face-mask occlusion without any adjustments, some will need additional training before showing usable results, and still others may struggle. Therefore, any statement of face recognition performance as a generality should be taken with a grain of salt.

Since nearly all modern face recognition algorithms utilize CNNs, how they are trained can also have an enormous impact on their performance (more on this later). Another variable is the deployment scenario of the face recognition system. Is the system making 1:1 comparisons against a claimed identity (verification), or a 1:N identification? Does the user stop and pose for the capture, or are they captured naturally while passing through a marked zone, or are the images being captured surveillance-style? The quality of the source or enrollment image(s) to which these captures are compared will also affect the algorithm performance. 

In short, it would be recommended that organizations test their face recognition systems to see what the effects of masks are. One would expect some performance degradation – the issue is whether the degradation is small and manageable versus large and catastrophic. The latter would render a face recognition system unusable. 

Face identification performance will depend on gallery size 

For face recognition systems that identify subjects, the size of the gallery will impact algorithm performance. As SAFR points out in their blog: 

“…many vendors are excitedly sharing news that they are “mask immune” — able to recognize faces with the same accuracy as when people are not wearing masks. That may be true when their sample size is small. But what we know about human faces is that they are all different, and as your sample size (enrolled database) grows, the range of represented facial traits and likelihood that someone with somewhat similar-looking features will appear in the database increases.” 

It should be noted that organizations that can keep their galleries small may be less affected by masked subjects than other face recognition applications. However, one aspect of masked-subject identification that has been largely ignored by commentators so far is whether mask-wearing will increase the likelihood of false positives between genetically related family members. 

More stringent testing of algorithms will be necessary to determine their performance against larger galleries. It will be especially important for consumers of testing data involving face masks to evaluate the FMR (False Match Rate) parameters used when determining FNMRs (False Non-Match Rate) in their testing. 

While testing of algorithms against masked subjects will no doubt be a topic of interest to NIST, there is no expectation that will happen soon given the current COVID-19 operational environment.

Performance on smartphones is not an accurate representation of industry as a whole 

Many of the articles describing failures of face recognition reference smartphones. With stay-at-home orders affecting the vast majority of Americans, smartphones are the most common application of face recognition. Unfortunately, the performance of face recognition on smartphones has been poor when the owner is masked. This is partially caused by smartphones not using top of the line algorithms.

Smartphone algorithms are designed to perform 1:1 matches quickly, with a reasonable amount of security. They are unprepared for a significant change in usage, such as authenticating subjects wearing masks. In addition, this use case is quite different from most national security programs.

Smartphone algorithms may use settings that decrease the false reject rate, thereby slightly increasing the false accept rate. This caters to a more convenient user experience. In smartphone identification scenarios, security can be less stringent because ease of use is paramount. In national security identification scenarios, this would be unacceptable. False accepts have repercussions that could impact our security posture. 

While smartphone face recognition will receive a larger amount of media attention, it would be a mistake to evaluate face recognition algorithms collectively based on those running only in smartphones. 

Training of CNN algorithms will help 

CNNs require training data, which can be just as important as the development of the CNN itself. All algorithms, regardless of current performance detecting masked faces, will benefit from training data containing masked faces. The first step in creating such a training set will be to collect many images of subjects wearing masks. Such datasets did not exist pre-pandemic. 

The National Engineering Research Center for Multimedia Software, within Wuhan University’s School of Computer Science, has created 3 publicly available mask-related data sets – one for detecting masks on faces, another with synthetic masks added by a software algorithm, and one “real-world” data set. The Real-world Masked Face Recognition Dataset (RMFRD) contains 5,000 masked images and 90,000 unmasked images composed from 525 individuals.  

Given lesser privacy policies in many Asian countries, algorithm developers have a data advantage as compared to U.S. and Western-based algorithm developers. Having access to larger amounts of data earlier will allow for easier creation of proprietary datasets. This facilitates a quicker cycle of adding masked training data and possibly adjusting face recognition algorithms. One may expect that, in the short term, these algorithms will likely perform better sooner. As more datasets are produced, western algorithms will eventually catch up. 

Algorithms that are having issues recognizing masked faces will require extra training and redevelopment, but eventually, most of these will be able to recognize such faces. 

Algorithms that are currently showing promise and success recognizing masked faces, will still benefit from additional training, which can improve mask detection and matching rates. 

Potential issues 

There are two main situations where wearing a face mask can disrupt authentication, even for algorithms designed to work with masks. The first involves subjects that are wearing both a face mask and sunglasses. The cause is simple – as masks occlude a large amount of face data from the tip of the nose, mouth, chin, and lower jawline, algorithms rely on the bridge of the nose and eyes for the majority of face data. Sunglasses occlude this final area of usable face information. 

The second situation involves subjects wearing patterned masks. It is possible that the pattern printed on the mask could confuse face detection engines. This may be either a purposeful or an unwitting attack.

To facilitate purposeful attacks, niche clothing manufacturers currently produce items specifically designed to confuse face recognition systems. For example, a face mask may have eyes, noses, and/or mouths printed on them. Others may have particular patterns or dots, stripes, or coloring that is highly unusual. In these cases, it is obvious to human observers that the subject is purposely avoiding recognition, and failure to detect rates will be extremely high for most algorithms.

On the other hand, unwitting attacks may arise from individuals who have made their own masks, often from bandanas or other patterned, colorful material. In these cases, the failure to detect rate may be moderately increased, and likely will vary by algorithm. 

Enrollment best practices: remove the mask 

For face recognition systems that are continuing enrollments and/or new subject captures, the best practice for future performance will continue to be capturing and enrolling unmasked faces. This may require changes to current operations to provide subjects with a safe, hygienic experience. 

While researchers may examine the possibility to both enroll and authenticate using masked enrollments, there will certainly be a performance hit. The only question is by what magnitude. For most government applications, enrolling masked faces will be unacceptable. 

Industry progress 

In this section, we look at various vendors (alphabetically) in the news to see how their algorithms are working with masked subjects. 

Cortica: This provider of face recognition software for Israeli law enforcement and intelligence agencies will now be expanding usage to identify masked hospital staff, as described in this April 12th article from the Jerusalem Post. 

Hanvon (Hanwang Technology Ltd): This March 9th article documents Hanvon’s claim that they became the first Chinese company to successfully perform face identification on masked subjects. 

Herta: This March 11th Herta press-release, while short on technical details, claims that its deep learning algorithm is mask-capable. 

Innovatrics: This March 31st news post on the Innovatrics website discusses the accuracy of its face matching algorithm, as is. In part, it states, “In cases where the database of thousands of persons are used, the accuracy depends on the required security. With a security setting of 99.9 percent, the overall accuracy is 90 to 93 percent. Increasing it to 99.99 %, the accuracy is still over 80 percent – an acceptable tradeoff in high-security use cases.” Industry experts may not be in full agreement that over 80% accuracy at FAR of 1-in-10,000 is an acceptable tradeoff in high-security use cases. 

NEC: This April 7th article describes how NEC had to update security gates at their Tokyo corporate headquarters to accommodate masks. NEC is looking to roll out these updates to the market in six months.  

Neurotechnology: In their description of the Verilook SDK, Neurotechnology claims that “Partially occluded faces (i.e. face mask or respirator) can be recognized.” This is one of few references to face masks made before the breakout of the COVID-19 pandemic. It should be noted that the company recommends disabling face quality checks in masked environments. 

Paravision: Paravision (formerly known as Ever AI) released a video on their website touting capabilities developed in the first week of April. The video includes simultaneous face recognition with mask detection, mask detection in a crowd, and social distancing determination. 

SAFR: This SAFR blog post from April 7th discusses masked face recognition in general, and was previously quoted in this document. The inference here is that SAFR algorithms are not working out of the box with masks, however, there is one mitigating setting that might increase performance. SAFR also states that “We are currently emphasizing training for our algorithms to reduce the negative impact that occlusions such as face masks have on accuracy.” 

SenseTime: This February 11th article describes a Korean deployment of SenseTime technology to perform face recognition of masked subjects. SenseTime is a Chinese-based AI startup. 

Speech Technology Center: This Russian-based company discusses their ability to handle face recognition with masks by observing that their software was already designed to perform identification in cold climates where significant facial occlusion was expected. The news item was released on March 23rd. 

VisionLabs: This March 12th article discusses VisionLabs claim that they have been able to perform face recognition with masks since 2018. This is from a press release by Sberbank, a Russian bank and customer of VisionLabs. 

Yitu: Yitu is another Chinese startup, and this February 26 article discusses capabilities of Yitu, Hanwang, and SenseTime. 

The following algorithms provide cloud face recognition services and/or novel approaches, and for that reason have been grouped separately from the traditional face recognition vendors. 

AWS Rekognition: Dignari has performed a cursory investigation of AWS Rekognition’s performance by adding synthetic masks to images of its team members. While the dataset was extremely small, Rekognition matched all images – masked-to-unmasked and masked-to-masked. Obviously additional testing and analysis would be required to determine true impact to the cloud face recognition capability.

Alibaba: This article from March 27th describes how Alibaba cloud has unusually split its face recognition practices into two for mask handling. At enrollment, features are divided into a full-face model and an eye recognition model. If the recognition algorithm determines the subject is wearing a mask, then it compares against the eye model, otherwise, it uses the full-face model. 

Despite searches, as of the time of this writing, we were not able to find mask or COVID-19 specific public updates from some of the other players in the face recognition industry such as Cognitec, Gemalto, Idemia, RankOne, or Microsoft Azure.

Conclusion 

Few face recognition algorithms were prepared to authenticate subjects wearing masks when the COVID-19 pandemic began. Once algorithm vendors gathered enough data and invested time into retooling and retraining their algorithms, many more were able to perform authentication on masked subjects. These updated algorithms have already been deployed and are in use in systems around the world. 

The widespread use of Deep Convolutional Neural Networks (CNNs) has transformed face recognition. In the mid- to late 2010s, this transformation manifested itself in the form of greatly improved accuracy and greater tolerance of extreme pose angles. Today, the CNN transformation manifests itself in tolerance of significant occlusions and the ability to quickly train on a specific use case – the wearing of surgical and homemade face masks. 

Datasets and large-scale testing are still in their infancy. While the initial results are compelling, deeper technical analysis of face recognition algorithms is essential to determine if large scale limitations exist in updated face recognition algorithms.