"Degrees of Freedom" as Related to Biometric Device Performance

Recently at least 4 articles and papers [1-4] have proffered "degrees of freedom" as a figure of merit for biometric devices. The purpose of this article is to demonstrate that single measures, such as "degrees of freedom", cannot be generally used to compare biometric device performance. We will first, by use of a limiting case, establish that "degrees of freedom" is insufficient to determine performance. Then, we will show that the impostor distributions of at least two biometric devices cannot be characterized by "degrees of freedom". Finally, we will review other scalar performance measures to show that these also are not generally applicable as figures of merit for biometric devices.

A "probability density" is a mathematical function which allows us to compute the chances of a measure lying between two values. A "bell curve" is an example of such a probability density. If student test scores follow a bell curve, the probability of a student’s score lying between 50 and 60, for instance, is equal to the area under the curve between the score values of 50 and 60. Figure 1 illustrates this example.

Strictly speaking, the bell curve is only appropriate if the measures can take on fractional values. A more appropriate probability density curve can be the "binomial distribution", in the case that the measures can only be whole numbers (as usually occurs with test scores). The mathematical expression for the binomial distribution is

y(i)=N!/(i!(N-i)!) pⁱ (1-p)^N-i

where p is the probability of a particular outcome and N generally describes the number of independent trials or repetitions of a random experiment [7]. In the case where p is close to 0.5, this distribution resembles a "bell curve" for any value of N.

The exclamation mark, as in N!, has a special meaning in mathematics. It is called "factorial" and means that all integers from 1 to N are multiplied together. For instance, 3!=1x2x3=6. To make a graph of the above equation, we need to choose values for both N and p. References [1,2] call N the "degrees of freedom". Reference 5 claims that the binomial distribution is a good fit for observed impostor scores in iris scanning. References [1-3] indicate that the binomial distribution should be chosen with N= 266 and p= 0.499 [1,3]. We hypothesize that other distributions fitting the observed values are very possible. Figure 2 shows this curve with the i-axis (labeled "SCORE") normalized by division by N. The area under the curve between any two points on the "score" axis is the probability that an impostor score lies on that interval.

What is the performance of a device with such an impostor distribution? We can’t yet say. Device performance is related to both impostor and genuine distributions[6,7]. Without knowledge of the genuine distribution, no statements regarding "false match" and "false non-match" rates can be made, which are the undisputed measures of biometric system performance. Consider a biometric device where genuine and impostor distributions are both characterized by a binomial distribution with N=266 and p= 0.499. The genuine and impostor distributions are identical. We use this extreme case to demonstrate that the "degrees of freedom" is insufficient to characterize performance. The "false match" and "false non-match" error rates in this case are graphed as a "Receiver Operating Characteristic" curve in Figure 3. In this case, the equal error rate is 50%. Increasing the "degrees of freedom" for both densities does not increase the accuracy of the device. Figure 4 shows identically overlapping genuine and impostor distributions with 532 "degrees of freedom". The "Receiver Operating Characteristic" curve is absolutely unchanged and is still given by Figure 3. The equal error rate is still 50%. Clearly, in this limiting case, error rates are independent of "degrees of freedom".

We might seek to correct the assertion that "degrees of freedom" is a measure of device performance by incorporating the difference in the parameter p between the genuine and impostor binomial distributions. This difference is also important and that together N and the two p values characterize the performance of any device. Even this would not be generally correct because most devices do not have probability densities characterized by the binomial distribution. Figure 5 shows the impostor distribution developed in one test of fingerprinting [8]. Figure 6 shows the impostor distribution developed in an unpublished test of a hand-based biometric device. Neither of these densities is not adequately modeled by the binomial distribution, so the concept of "degrees of freedom" cannot be applied in a discussion of error rates for these devices.

Several related concepts in classical pattern recognition theory are well established for measuring the separation of two distributions of classes, when the classes are "bell" shaped. Let's first model the genuine distribution grossly with mean probability p₁ and variance s₁² and the impostor distribution grossly with mean probability p₂ and variance s₂². This gross modeling of the distributions, however, has been shown wrong in many large-scale, realistic applications.

The first of these separation measures is the Fisher ratio [10], which attempts to measure the overlap between two distributions. Overlap is based on two things: the separation between the distributions and the width of the distributions. Fisher’s discriminant is based on the ratio of inter-class separation over the average intra-class spread.

F = (p₁+ p₂)² /(s₁²+s₂²)

This is clearly related to the "decidability" index, defined originally in [11] as,

d’=|p₁- p₂|/s

where s₁² = s₂² = s². This concept was expanded by [12] to include cases where the distributions have different variances as

D^1/2 = |p₁- p₂| /((s₁²+s₂²)/2)^1/2

Information theory provides yet another dissimilarity measure between two classes, known as divergence, I. Divergence is the total average information for discriminating one class from another [13]. This measure has been successfully applied already in biometrics for speaker identification [14]. Interestingly, when the two distributions are bell curves with same widths, s₁² = s₂² = s², then the divergence measure, I, is similar to the previous measures, and is given by

I= (p₁- p₂)²/s²

Two biometric devices can be compared using these measures only if the genuine and impostor distributions for both devices have similar shapes. If the distributions of the two devices are dissimilar, none of these provides a comparative measure of device performance. Figures 5 and 6 show impostor distributions that depart significantly from the binomial distribution used to model the impostor distribution of iris scanning. In general, due to the difference in distributional shapes, biometric devices cannot be compared using any of these measures.

We conclude that the concept of "degrees of freedom" only applies to devices whose distributions are well modeled by the binomial. Even when distributions are binomial, devices with more "degrees of freedom" may not have lower error rates. Further, the concept of "degrees of freedom" is not applicable to most biometric devices . Other classical measures of distribution separation can be used to compare device performance only in the unusual case where the devices have distributions which are similarly shaped. Consequently, the relative performance of a biometric device cannot be expressed in any single number.

References:

[1] G.O. Williams, "Iris Recognition Technology", IEEE AES Systems Magazine, April 1997, pg. 23-29

[2] J. Daugman, "Continuing Debate on Issues and Strategies in Large-Scale Biometric Searches", Biometrics in Human Services User Group News Letter , volume 3, no.1 ,January 1999, downloadable from www.dss.state.ct.us/digital.html

[3] C. Seal, M. Gifford, D. McCartney, "Iris Recognition for User Validation", British Telecommunication Engineering, vol.16, July 1997

[4] C. Wu, "Private Eyes", Science News, Vol.153, No.14, April 4, 1998

[5] R.P. Wildes, "Iris Recognition: An Emerging Biometric Technology", Proc. IEEE, Vol. 85, No. 9, Sept. 1997

[6] J.L.Wayman, "Testing and Evaluating Biometric Technologies: What the Customer Needs to Know", Proc. CTST’97, pg. 329-348 , also on www.engr.sjsu.edu/biometrics

[7] J.L.Wayman, "Technical Testing and Evaluation of Biometric Devices", in A. Jain, etal (ibid)

[8] J.L Wayman, "Biometric Identification Standards Research -- Final Report: Vol. 1" (1997), report to the Federal Highway Administration, available for downloading at www.engr.sjsu.edu/biometrics/fhwa.htm

[9] Robert V. Hogg and Allen T Craig, "Mathematical Statistics", 4^th Edition, Macmillan Publishing Co, Inc., New York, 1978.

[10] Thomas Parsons, "Voice and Speech Processing", McGraw-Hill Book Co, New York, 1987.

[13] J. T. Tou and R.C. Gonzalez, "Pattern Recognition Principles", Addison-Wesley Publishing Co, Reading Mass, 1974.

[14] Joseph P. Campbell, "Features and Measures for Speaker Recognition", Oklahoma State University, Dec 1992.