Introducing Performance and Accuracy SLAs

Introducing Performance and Accuracy SLAs Informed

Imagine that you bought a new car, and after the dealer tossed you the keys, they off-handedly said that the vehicle came with no driver’s manual or warning lights. Also, if the vehicle suffered mechanical problems, you were on your own. I bet you wouldn’t be happy. Buyers of new AI document intelligence software might have a similar feeling. They must fend for themselves in the wild, wild west of AI. As a best practice, document intelligence solutions should come with transparent performance and accuracy service level agreements (SLAs).

Why are accuracy SLAs so important?

Providing SLAs acknowledges that document intelligence doesn’t occur in a test environment. In the real world (production environments) it contends with the vagaries and changes of day to day business. 

For example – documents auto lenders collect from consumer applicants and dealers. Lenders regularly see new income documents, government documents, legal requirements, products, or new production technologies. Without accuracy SLAs, these unfamiliar documents could confuse the AI and interfere with automated processing of the loan, causing declines in accuracy. 

Consequences of accuracy and automation declines are far-reaching. In highly-scrutinized consumer lending, these declines lead to financial losses, reputational damage, and potentially severe regulatory and legal implications.  

Accuracy SLAs ensure that you avoid these costly issues. Lenders can use the SLA metrics to independently monitor vendor performance and raise questions at the first sight of AI deterioration. Accuracy metrics also have compliance benefits. These metrics provide a continuous, data-rich record that support model governance and internal audits. Your organization would otherwise be responsible for securing the budget to build out, monitor, and maintain your own audit trail. 

So how do accuracy SLAs work? 

There are three key metrics to look for: 

  1. Recall: Ensuring the Completeness of Information Capture: Recall measures how effective an AI system is in correctly classifying documents by type and/or extracting required information. For instance, in a document classification task, if 100 documents are classified and the AI correctly classifies 90 of them, the recall rate is 90%.
  1. Precision: Measuring Correctness: Precision assesses the correctness of the AI classifications. In the example above, say that four of the 100 documents were unreadable, and AI attempted to classify 96 documents. If it correctly classified 90 of 96 documents, its precision rate is 93.75%. 
  2. Concept Drift: Keeping up with changing data patterns. Concept Drift reflects the variance between the data the AI is trained on and the data the AI model encounters in the real world (production).

Together, these metrics form the backbone of effective AI SLAs.

You might wonder why ChatGPT and other horizontal document intelligence solutions do not commit to strict accuracy SLAs. The simple answer is that accuracy building SLAs are extremely difficult. Accuracy SLAs must be industry-specific and are incredibly challenging to implement. They demand significant investment in a vertical and a statistically-significant, up to date labeled dataset.

In assessing whether accuracy SLAs are sufficient, consider whether they feature:

  1. Ongoing monitoring of core accuracy metrics: recall, precision, and concept drift;
  2. Continuous expert labeling of real-world, representative data;
  3. Audit-ready insights for regulatory and compliance functions;
  4. Transparent AI with PII-redacted document viewing, and an easy-to-navigate client interface;

Conclusion

Accuracy SLAs are critical components of transparent, understandable, and reliable AI solutions. Accuracy They do more than provide insights; they build trust and confidence for key stakeholders, from operational teams to senior management and regulators. Accuracy SLAs enable lenders to embrace the power of AI with clarity and confidence. To learn more about accuracy SLAs, read this new white paper.

author avatar
Tom Oscherwitz VP of Legal
Tom Oscherwitz is Informed’s VP of Legal and Regulatory Advisor.  He has over 25 years of experience as a senior government regulator (CFPB, U.S. Senate) and as a fintech legal executive working at the intersection of consumer data, analytics, and regulatory policy.

Upcoming Webinar: Crossing the chasm into the new digital world: The impact of AI and automation in creating a fully digital auto ecosystem

X