WIXLIB

4Jukka Komulainen, Zinelabidine Boulkenafet and Zahid Akhtarspectral) reflectance [40, 42, 44, 55] properties. For instance, planar PAI detectionbecomes rather trivial if depth information is available [17], whereas near-infrared(NIR) or thermal cameras are efficient in display attack detection as most of thedisplays in consumer electronics emit only visible light. On the other hand, thesekinds of unconventional sensors are usually expensive and not compact, thus not(yet) available in personal devices, which prevents their wide deployment.It is rather appealing to perform face PAD by further analyzing only the same datathat is used for face recognition or additional data captured with the standard acquisition device (e.g., challenge-response approach). These kinds of software-basedmethods can be broadly divided into active (requiring user collaboration) and passive approaches. Additional user interaction can be very effectively used for facePAD because we humans tend to be interactive, whereas a photo or video-replayattack cannot respond to randomly specified action requirements. Furthermore, it isvery difficult to perform liveness detection or facial 3D structure estimation by relying only on spontaneous facial motion. Challenge-response based methods aim atperforming face PAD detection based on whether the required action (challenge), forexample facial expression [25, 36], mouth movement [11, 25] or head rotation (3Dstructure) [20, 34, 48], was observed within a predefined time window (response).Also, active software-based methods are able to generalize well across different acquisition conditions and attack scenarios but at the cost of usability due to increasedauthentication time and system complexity.Passive software-based methods are preferable for face PAD because they arefaster and less intrusive than active countermeasures. Due to the increasing numberof public benchmark databases, numerous passive software-based approaches havebeen proposed for face PAD. In general, passive methods are based on analyzingdifferent facial properties, such as frequency content [28, 46], texture [2, 12, 17, 27,32, 53] and quality [18, 21, 23], or motion cues, such as eye blinking [3, 38, 45, 47],facial expression changes [3, 25, 45, 47], mouth movements [3, 25, 45, 47], or evencolor variation due to blood circulation (pulse) [15, 29, 31], to discriminate faceartifacts from genuine ones. Passive software-based methods have shown impressiveresults on the publicly available datasets but the preliminary cross-database tests,such as [19, 48], revealed that the performance is likely to degrade drastically whenoperating in unknown conditions.Recently, the research focus on software-based face PAD has been graduallymoving towards assessing and improving the generalization capabilities of the proposed and existing methods in a cross-database setup instead of operating solelyon single databases. Among hand-crafted feature based approaches, colour textureanalysis [5, 6, 7, 8], image distortion analysis [21, 23, 49], combination of textureand image quality analysis with interpupillary distance (IPD) based reject option[39], dynamic spectral domain analysis [41] and pulse detection [29] have been applied in the context of generalized face PAD but with only moderate success.The initial studies using deep CNNs have resulted in excellent intra-test performance but the cross-database results have still been unsatisfactory [39, 52]. This isprobably due to the fact that the current publicly available datasets may not provideenough data for training well-known deep neural network architectures from scratch

Review of Face Presentation Attack Detection Competitions5or even for fine-tuning pre-trained networks. As a result, the CNN models have beensuffering from overfitting to specific data and learning database-specific informationinstead of generalized PAD related representations. In order to improve the generalization of CNNs with limited data, more compact feature representations or novelmethods for cross-domain adaptation are needed. In [33], deep dictionary learningbased formulation was proposed to mitigate the requirement of large amounts oftraining data with very promising intra-test results but the generalization capabilitywas again unsatisfying. In any case, the potential of application-specific learningneeds to be further explored when more comprehensive face PAD databases areavailable.3 First and second competitions on countermeasures to 2D facespoofing attacksIn this section, we recapitulate the first [10] and second [15] competitions on countermeasures to 2D face spoofing attacks, which were held in conjunction with International Joint Conference on Biometrics (IJCB) in 2011 and International Conference on Biometrics (ICB) in 2013, respectively. Both competitions focused onassessing the stand-alone PAD performance of the proposed algorithms in restrictedacquisition conditions, thus integration with actual face verification stage was notconsidered.In 2011, the research on software-based face PAD was still in its infancy mainlydue to lack of public datasets. Since there were no comparative studies on the effectiveness of different PAD methods under the same data and protocols, the goal of thefirst competition on countermeasures to 2D facial spoofing attacks [10] was to provide a common platform to compare software-based face PAD using a standardizedtesting protocol. The performance of different algorithms was evaluated under printattacks using a unique evaluation method. The used PRINT-ATTACK database [1]defines a precise protocol for fair and unbiased algorithm evaluation as it provides afixed development set to calibrate the countermeasures, while the actual test data isused solely for reporting the final results.While the first competition [10] provided an initial assessment of face PAD, the2013 edition of the competition on countermeasures to 2D face spoofing attacks[15] aimed at consolidating the recent advances and trends in the state of the artby evaluating the effectiveness of the proposed algorithms in detecting a variety ofattacks. The contest was carried out using the same protocol on the newly collectedvideo REPLAY-ATTACK database [12], introducing display attacks (digital photosand video-replays) in addition to print attacks.Both competitions were open to all academic and industrial institutions. A noticeable increase in the number of participants between the two competitions canbe seen. Particularly, six different competitors from universities participated in thefirst contest, while eight different teams participated in the second competition. The

6Jukka Komulainen, Zinelabidine Boulkenafet and Zahid AkhtarTable 1 Names and affiliations of the participating systems in the first competition on countermeasures to 2D facial spoofing attacksAlgorithm mbient Intelligence Laboratory, ItalyChinese Academy of Sciences, ChinaIdiap Research Institute, SwitzerlandUniversidad de Las Palmas de Gran Canaria, SpainUniversity of Campinas, BrazilUniversity of Oulu, FinlandTable 2 Names and affiliations of the participating systems in the second competition on countermeasures to 2D face spoofing attacksAlgorithm nameCASIAIGDMaskDownLNMIITMUVISPRA LabATVSUNICAMPAffiliationsChinese Academy of Sciences, ChinaFraunhofer Institute for Computer Graphics, GermanyIdiap Research Institute, SwitzerlandUniversity of Oulu, FinlandUniversity of Campinas, BrazilLNM Institute of Information Technology, IndiaTampere University of Technology, FinlandUniversity of Cagliari, ItalyUniversidad Autonoma de Madrid, SpainUniversity of Campinas, Brazilaffiliation and corresponding algorithm name of the participating teams for the twocompetitions are summarized in Table 1 and Table 2.In the following, we summarize the design and main results of the first and second competitions on countermeasures to 2D face spoofing attacks. The reader canrefer to [10] and [15] for more detailed information on the competitions.3.1 DatasetsThe first face PAD competition [10] utilized PRINT-ATTACK [1] database consisting of 50 different subjects. The real access and attack videos were captured with a320 240 pixels (QVGA) resolution camera of a MacBook laptop. The database includes 200 videos of real accesses and 200 videos of print attack attempts. The PAswere launched by presenting hard copies of high resolution photographs printed onA4 papers with a Triumph-Adler DCC 2520 color laser printer. The videos wererecorded under controlled (uniform background) and adverse (non-uniform background with day-light illumination) conditions.The second competition on face PAD [15] was conducted using an extensionof the PRINT-ATTACK database, named as REPLAY-ATTACK database [12]. Thedatabase consists of video recordings of real accesses and attack attempts corresponding to 50 clients. The videos were acquired using the built-in camera of a

Review of Face Presentation Attack Detection Competitions7Fig. 1 Sample images from the PRINT-ATTACK [1] and REPLAY-ATTACK [12] databases. Topand bottom rows correspond to controlled and adverse conditions, respectively. From left to rightcolumns: real accesses, print, mobile phone and tablet attacks.MacBook Air 13 inch laptop under controlled and adverse conditions. Under thesame conditions, high resolution pictures and videos were taken for each personusing a Canon PowerShot SX150 IS camera and an iPhone 3GS camera, later tobe used for generating the attacks. Three different attacks were considered: i) printattacks (i.e., high resolution pictures were printed on A4 paper and displayed tothe camera); ii) mobile attacks (i.e., attacks were performed by displaying picturesand videos on the iPhone 3GS screen); iii) high definition attacks (i.e., the picturesand the videos were displayed on an iPad screen with 1024 768 pixels resolution). Moreover, attacks were launched with hand-held and fixed support modes foreach PAI. Figure 1 shows sample images of real and fake faces from both PRINTATTACK and REPLAY-ATTACK databases.3.2 Performance evaluation protocol and metricsThe databases used in both competition editions are divided into train, developmentand test sets with no overlap between them (in terms of subjects or samples). Duringthe system development phase of the first competition, the participants were givenaccess to the labelled videos of the training and the development sets that were usedto train and calibrate the devised face PAD methods. In the evaluation phase, the performances of the developed systems were reported on anonymized and unlabelledtest video files. In the course of the second competition, the participants had access to all subsets because the competition was conducted on the publicly availableREPLAY-ATTACK database. The final test data consisted of anonymized videos of100 successive frames cut from the original test set videos starting from a randomtime.The first and second competitions considered a face PAD method to be proneto two types of errors: either a real access attempt is rejected (false rejection) ora PA is accepted (false acceptance). Both competitions employed Half Total Error

8Jukka Komulainen, Zinelabidine Boulkenafet and Zahid AkhtarTable 3 Overview and performance (in %) of the algorithms proposed in the first face PAD competition (F stands for feature-level and S for score-level AMPUOULUTexture, motion & livenessTexture, motionTextureMotionTexture, motion & livenessTextureSSF-DevelopmentFAR FRR HTER0.00 0.00 0.001.67 1.67 1.670.00 0.00 0.001.67 1.67 1.671.67 1.67 1.670.00 0.00 21.250.000.00HTER0.630.000.0010.630.630.00Rate (HTER) as principal performance measure metric, which is the average of falserejection rate (FRR) and false acceptance rate (FAR) at a given threshold τ :FAR(τ ) FRR(τ )(1)2For evaluating the proposed approaches, the participants were asked to providetwo files containing a score value for each video in the development and test sets,respectively. The HTER is measured on the test set using the threshold τ corresponding to the equal error rate (EER) operating point on the development set.HT ER(τ ) 3.3 Results and discussionThe algorithms proposed in the first competition on face PAD and the correspondingperformances are summarized in Table 3. The participated teams used either singleor multiple types of visual cues among motion, texture and liveness. Almost everysystem managed to obtain nearly perfect performance on both development and testsets of the PRINT-ATTACK database. The methods using facial texture analysisdominated because the photo attacks in the competition dataset suffered from obvious print quality defects. Particularly, two teams, IDIAP and UOULU, achievedzero percent error rates on both development and test sets relying solely on localbinary pattern (LBP) [37] based texture analysis, while CASIA achieved perfectclassification rates on the test set using combination of texture and motion analysis. Assuming that the attack videos usually are noisier than those of real videos, thetexture analysis component in CASIA’s system is based on estimating the differencein noise variance between the real and attack videos using first order Haar waveletdecomposition. Since the print attacks are launched with fixed and hand-held printouts with incorporated background (see Figure 1), the motion analysis componentmeasures the amount of non-rigid facial motion and face-background motion correlation.Table 4 gives an overview of the algorithms proposed within the second competition on face PAD and the corresponding performance figures for both development

Review of Face Presentation Attack Detection Competitions9Table 4 Overview and performance (in %) of the algorithms proposed in the second face PADcompetition (F stands for feature-level and S for score-level fusion)TeamFeaturesCASIAIGDMaskDownLNMIITMUVISPRA LabATVSUnicampTexture & motionLivenessTexture & motionTexture & lopmentTestFAR FRR HTER FAR FRR HTER0.00 0.00 0.000.00 0.000.005.00 8.33 6.67 17.00 1.259.131.00 0.00 0.500.00 5.002.500.00 0.00 0.000.00 0.000.000.00 0.00 0.000.00 2.501.250.00 0.00 0.000.00 2.501.251.67 0.00 0.832.75 21.25 12.0013.00 6.67 9.83 12.50 18.75 15.62and test sets. The participating teams developed face PAD methods based on texture,frequency, image quality, motion and liveness (pulse) features. Again, the use of texture was popular as seven out of eight teams adopted some sort of texture analysisin the proposed systems. More importantly, since the attack scenarios in the secondcompetition were more diverse and challenging, a common approach was combining several complementary concepts together (i.e., information fusion at feature orscore level). The category of the used features did not influence the choice of fusionstrategy. The best-performing systems were based on feature-level fusion but it ismore likely that the high level of robustness is largely based on the feature designrather than the used fusion approach.From Table 4, it can be seen that the two PAD techniques proposed by CASIA and LNMIIT achieved perfect discrimination between the real accesses and thespoofing attacks (i.e., 0.00% error rates on the development and test sets). Both ofthese top-performing algorithms employ a hybrid scheme combining the featuresof both texture and motion-based methods. Specifically, the used facial texture descriptions are based on LBP, while motion analysis components again measure theamount of non-rigid facial motion and face-background motion consistency as thenew display attacks are inherently similar to the ”scenic” print attacks of the previous competition (see Figure 1). The results on the competition dataset suggested thatface PAD methods relying on a single cue are not able to detect all types of attacks,and the generalizing capability of the hybrid approaches is higher but with highcomputational cost. On the other hand, MUVIS and PRA Lab managed to achieveexcellent performance on the development and test sets using solely texture analysis. However, it is worth pointing out that both systems compute the texture featuresover whole video frame (i.e., including background region), thus the methods areseverely overfitting to the scene context information that matches across the train,development and test data. All in all, the astonishing results also on the REPLAYATTACK dataset conclude that more challenging configurations are needed beforethe research on face PAD can reach the next level.

10Jukka Komulainen, Zinelabidine Boulkenafet and Zahid Akhtar4 Competition on generalized face presentation attack detectionin mobile scenariosThe vulnerabilities of face based biometric systems to PAs have been widely recognized but still we lack generalized software-based PAD methods performing robustly in practical (mobile) authentication scenarios. In recent years, many face PADmethods have been proposed and remarkable results have been reported on the existing benchmark datasets. For instance, as seen in Section 3, several methods achievedperfect error rates in the first [10] and second [15] face PAD competitions. More recent studies, such as [5, 8, 19, 49, 52], have revealed that the existing methods arenot able to generalize well in more realistic scenarios, thus software-based face PADis still an unsolved problem in unconstrained operating conditions.Focused large scale evaluations on the generalization of face PAD had not beenconducted or organized after the issue was first pointed out by de Freitas Pereiraet al. [19] in 2013. To address this issue, we organized a competition on mobileface PAD [4] in conjunction with IJCB 2017 to assess the generalization abilities ofstate-of-the-art algorithms under some real-world variations, including unseen inputsensors, PAIs, and illumination conditions. In the following, we will introduce thedesign and results of this competition in detail.4.1 ParticipantsThe competition was open to all academic and industrial institutions. The participants were required register for the competition and sign the end user license agreement (EULA) of the used OULU-NPU database [9] before obtaining the data for developing the PAD algorithms. Over 50 organizations registered for the competitionand 13 teams submitted their systems in the end for evaluation. The affiliation andcorresponding algorithm name of the participating teams are summarized in Table5. Compared with the previous competitions, the number of participants increasedsignificantly from six and eight in the first and second competitions, respectively.Moreover, in the previous competitions, all the participated teams were from academic institutes and universities, whereas in this competition, we had registered theparticipation of three companies as well, which highlights the importance of thetopic for both academia and industry.4.2 DatasetThe competition was carried out on the recently published1 OULU-NPU face presentation attack database [9]. The dataset and evaluation protocols were designed1The dataset was not yet released at the time of the competition.

Review of Face Presentation Attack Detection Competitions11Table 5 Names and affiliations of the participating systemsAlgorithm nameBaselineMBLPQPMLMassy URecodCPqDAffiliationsUniversity of Oulu, FinlandUniversity of Ouargla, AlgeriaUniversity of Biskra, AlgeriaUniversity of the Basque Country, SpainUniversity of Valenciennes, FranceChangsha University of Science and TechnologyHunan University, ChinaIndian Institute of Technology Indore, IndiaGalician Research and Development Centerin Advanced Telecommunications, SpainEcole Polytechnique Federale de LausanneIdiap Research Institute, SwitzerlandVologda State University, RussiaShenzhen University, ChinaFUJITSU laboratories LTD, JapanNorthwestern Polytechnical University, ChinaHong Kong Baptist University, ChinaUniversity of Campinas, BrazilCPqD, Brazilparticularly for evaluating the generalization of face PAD methods in more realisticmobile authentication scenarios by considering three covariates: unknown environmental conditions (namely illumination and background scene), PAIs and acquisition devices, separately and at once.The OULU-NPU database consists of 4950 short video sequences of real accessand attack attempts corresponding to 55 subjects (15 female and 40 male). The realaccess attempts were recorded in three different sessions separated by a time intervalof one week. During each session, a different illumination condition and backgroundscene were considered (see Figure 2): Session 1: The recordings were taken in an open-plan office where the electriclight was switched on, the windows blinds were open, and the windows werelocated behind the subjects. Session 2: The recordings were taken in a meeting room where the electric lightwas the only source of illumination. Session 3: The recordings were taken in a small office where the electronic lightwas switched on, the windows blinds were open, and the windows were locatedin front of the subjects.During each session, the subjects recorded the videos of themselves using thefront facing cameras of the mobile devices. In order to simulate realistic mobileauthentication scenarios, the video length was limited to five seconds. Furthermore,the subjects

Center for Machine Vision and Signal Analysis, University of Oulu, Finland e-mail: jukka.komulainen@iki.fi Zinelabidine Boulkenafet Center for Machine Vision and Signal Analysis, University of Oulu, Finland e-mail: zinelabidine.boulkenafet@oulu.fi Zahid Akhtar INRS-EMT, University of Quebec, Canada e-mail: zahid.akhtar.momin@emt.inrs.ca 1