False positives and false negatives: how to tell whether a vision system works

How to tell whether a machine vision system works: false positives and false negatives, the confusion matrix, sensitivity and acceptance criteria.

Por Oscar Serrano, CEO de i-mas 6 min de lectura

machine vision

When a machine vision system goes into service, the question that always comes up sooner or later is the same: does it work? And the usual answer, “it is 99% accurate”, means nothing.

An inspection system does not have one accuracy figure. It has two different errors that behave in opposite ways: rejecting good parts and letting defects through. Lowering one raises the other, and each one costs money in a different part of the plant.

This article explains how those two errors are measured, why the accuracy percentage is misleading and which acceptance criteria should be agreed before anything is installed.

False positive and false negative: fix the convention before arguing

By convention, a “positive” means the system detects a defect. The two errors follow from that:

False positive: the system rejects a good part. On the shop floor it is called a false alarm, a false call or, in electronics, overkill. It costs scrap, manual re-inspection and, if it keeps happening, operators who stop trusting the equipment.

False negative: a real defect passes inspection. This is the escape, and it is the expensive error: it reaches the customer.

Write it into the specification, because supplier and customer often use the terms the other way round (some people call the good part the “positive”) and end up agreeing on a criterion that each side reads in the opposite way. It is one of the common mistakes when implementing machine vision that only come to light at acceptance.

The four cells: the confusion matrix

Everything fits into a four-cell table that compares what the system decides with what the part really is:

  Defective part Good part
The system rejects True positive (TP) False positive (FP)
The system accepts False negative (FN) True negative (TN)

The three metrics that are actually used come from it:

Sensitivity = TP / (TP + FN). Of all the defects that were there, how many it caught. This is the one that protects the customer.

Specificity = TN / (TN + FP). Of all the good parts, how many it let through without trouble.

Reject precision = TP / (TP + FP). Of everything the system threw into the reject bin, how much was really defective. This is the one the operator sees.

Why “99% accurate” says nothing

Overall accuracy is misleading because defects are rare on an industrial line. An example with round numbers makes it clear.

Take a line with 1% defective parts and a system with 95% sensitivity and 98% specificity. Out of 10,000 parts:

Of the 100 defective parts, it detects 95 and misses 5. Of the 9,900 good ones, it accepts 9,702 and wrongly rejects 198. Result: overall accuracy of 97.97%, which sounds excellent, but of the 293 parts that end up in the reject bin only 95 are real defects. In other words, two out of three rejects are good parts.

That effect is not a fault of the system: it is arithmetic. The rarer the defect, the worse the reject precision gets, even when specificity is high. That is why an acceptance criterion that only says “95% accuracy” is an empty criterion.

Which error to prioritise: cost decides, not technology

No setting minimises both errors at once. Moving the decision threshold shifts the balance between them, and where to put it is a business decision.

For a safety component or a medical device, an escape is unacceptable and over-rejection is accepted. For a low-value part with cheap rework, the opposite may pay off. The question to answer before adjusting anything is how much an escape costs and how much a false reject costs.

There is also a second-order effect worth keeping in mind: if the false alarm rate is high, the operator who checks the rejects gets used to passing them, and then real defects slip past as well. Too many false positives end up creating false negatives.

How it is really measured: the agreement study

Measuring these figures requires a set of known samples (good parts and representative defects of each type) inspected several times. Validating with the easy product is not enough: the set has to include borderline defects, the ones right on the limit.

The standard method in industry is the attribute agreement analysis, which measures whether the system agrees with itself on repeat runs and whether it agrees with the reference standard. The usual indicator is the kappa statistic, which corrects agreement for chance.

As a reference, the Minitab documentation attributes to the AIAG MSA manual the criterion that a kappa of 0.75 or more is acceptable, with 0.90 or more preferred; other tools, such as SigmaXL, use slightly different bands and put the unacceptable limit at 0.70. The required value should be set in the specification, because there is no single universal threshold.

This work is the most laborious part of the project and the one that most shapes the result, whichever of the two approaches, traditional or deep learning, the system uses.

How regulated sectors handle it

Where inspection is regulated, the criterion is not a single metric but a comparison against the method being replaced.

In the visual inspection of injectables, the reference approach (the Knapp test, covered in USP chapter <1790>) starts by calculating the probability of detection of each unit in the challenge set from repeated inspections, and requires the automated system to detect at least as well as the panel of human inspectors.

The idea behind it applies to any sector: inspection is a probabilistic process, not a yes or no, and the reasonable bar is the real performance of the previous method, not perfection. It is the same criterion we explained for machine vision in regulated environments.

How we define acceptance criteria at I-MAS

At I-MAS we approach machine vision projects by agreeing the acceptance criteria before designing the system: what counts as a defect, which sample set will be used, which sensitivity and false reject rate are acceptable and how all of that will be measured at acceptance.

We then design the optics, lighting, algorithm and integration with the line around that goal, and choose the classic or deep learning approach depending on what has to be proven.

Want to know more about our services? Get in touch with us!