Reporting & Compliance guide: AI Underwriting Vendor Due Diligence: The Questions to Ask About Explainability, Accuracy and Oversight

AI underwriting vendor due diligence comes down to one test: can the vendor show, on your own files, where every number came from, why every score or flag fired, what changed between versions and who signed the decision? Ask for evidence rather than assurances, run a blind proof of concept on decided files, and score the answers. Remember that the decision, and the accountability for it, stays with you.

This guide is the working playbook. It gives you a question bank grouped into ten areas, from data provenance to exit and portability, with what a good answer and an evasive answer sound like in each. It then covers how to run a proof of concept on your own historical files, a weighted scoring rubric, and whether an in-house tool or a vendor platform is easier to defend to a bank examiner. Where a topic already has its own guide on this site, such as explainable AI in underwriting or what a decision-ready credit memo contains, we link to it rather than repeat it.

01Why does AI underwriting need its own vendor due diligence?

Because an AI underwriting tool does not just store your data; it produces numbers, scores and draft language that end up in a credit decision. Standard software due diligence checks uptime and security. AI due diligence also has to check whether the outputs are traceable, testable and controlled, because a wrong output can travel straight into a memo.

The failure modes are quiet. A misread deposit total changes a holdback calculation without crashing anything. A fraud flag that cannot explain itself is either ignored or obeyed. A model update can shift how a whole category of files is summarized, and nobody will know unless the vendor says so and you re-test.

The regulatory vocabulary for this is well established, even if most of it was written for banks. In April 2026 the Federal Reserve, OCC and FDIC issued SR 26-2, revised guidance on model risk management, which supersedes the long-standing SR 11-7. The revised guidance says it is most relevant to banking organizations with over $30 billion in total assets, that it does not set enforceable standards, and that generative and agentic AI models are outside its scope, while its principles still apply to traditional statistical models and non-generative AI. On vendor products it is direct: banks may not receive a vendor's code, data or methodology, but the principles of model risk management remain applicable, and sound practice includes understanding the vendor model's conceptual soundness, design, development data and performance, then monitoring whether it stays fit for purpose.

Two other documents frame the rest. The June 2023 interagency guidance on third-party relationships from the Federal Reserve, FDIC and OCC sets out a life cycle of planning, due diligence and selection, contract negotiation, ongoing monitoring and termination. The NIST AI Risk Management Framework, released in January 2023 for voluntary use and organized around four functions (Govern, Map, Measure, Manage), gives non-bank lenders a neutral structure for the same conversation; NIST added a generative AI profile, NIST AI 600-1, in July 2024.

None of this binds an MCA funder or non-bank lender the way it binds a supervised bank, but it is the language bank partners, credit facility providers and auditors will use, so write your due diligence file in it.

What should lenders look for in a decision intelligence platform?

The same things this playbook tests: evidence attached to every output, a clear boundary between what the software prepares and what a person decides, policy that is configurable and versioned, and a record you can reconstruct later. Our guide to decision intelligence in underwriting explains the concept and its maturity levels; this guide is about proving a specific vendor delivers it. If you are still sorting vendors into categories, start with the four types of AI underwriting software, because a search for the best providers for integrating machine learning in credit underwriting often mixes extraction tools with decisioning platforms.

02What should an underwriting leader ask a vendor about the explainability of AI risk scores?

Ask the vendor to take one real score or flag on one of your files and walk you from the output back to the specific inputs, rules and documents that produced it, then show the same explanation in the record your auditor would see. If they can only describe the model in general terms, the score is not explainable in the sense your process needs.

Ask these against one of your own files:

1. For this flag on this file, which inputs drove it, and can I click from each input to the page it came from?
2. Is the reason text generated at the time of the decision and stored with the file, or reconstructed later when someone asks?
3. If the explanation is an approximation of a more complex model, how was that approximation tested against the model itself?
4. Which outputs are rules (deterministic checks against my policy) and which are statistical or generative? Show me the label in the interface.
5. Can the explanation be stated in terms an applicant would understand, or only in feature names?
6. When two signals conflict, does the output say so, or does it net them into one number?

The third question matters for adverse action. The CFPB's Circular 2022-03 says ECOA and Regulation B apply to credit decisions regardless of the technology used, that the statement of reasons must be specific and reflect the factors actually considered, and that a creditor cannot point to the complexity of its own algorithm as a defense. It also cautions that post-hoc explanation methods approximate a model rather than describe it exactly. Regulation B covers business credit too, with adjusted notice rules: under 12 CFR 1002.9(a)(3), applicants with gross revenues of $1 million or less generally receive notices much like consumers (with some allowance for oral statements), while larger businesses must be told the action taken within a reasonable time and can request the reasons in writing within 60 days. Whether a particular product, such as a merchant cash advance structured as a purchase of receivables, is credit under ECOA is a question for your counsel, and agency guidance changes, so confirm the current position before relying on it.

Whatever the legal answer for your product, a risk score that cannot say why is one your team will either over-trust or ignore. The design principles behind this are in our guide to explainable AI in underwriting compliance.

03How do lenders verify AI-extracted numbers before they reach a credit memo?

They check a defined sample of extracted values against the source page, confirm that totals tie out arithmetically, and log who checked what and when, before any of those numbers are quoted in the memo. The vendor's job is to make that possible: every extracted value needs a link to the page, line and region it came from.

A workable verification routine has four parts. Ask each vendor how its product supports every one of them, and ask to see it rather than hear about it.

• Source links on every value. Each extracted number opens the exact page and region it came from. Without this, verification means re-reading the whole statement, which is the work you bought the tool to remove. Our guide to source-linked extraction covers what good linking looks like.
• Arithmetic tie-outs. Opening balance plus credits minus debits should reach the closing balance for each statement period. Deposit totals should match the sum of the listed deposits. A tool that shows tie-out failures as exceptions turns a hidden extraction error into a visible one.
• A sampling rule. Decide in advance which values a person checks on every file (for example, monthly deposit totals and the figures used in pricing) and which are sampled. Make the sample larger for new document layouts, low-quality scans and files near a policy threshold.
• A logged check. The check should leave a record: who confirmed the value, when, and whether they corrected it. Corrections are data. If a vendor's extraction is corrected often on one bank's statements, you want to see that pattern.

Here is what the routine looks like on an illustrative file. Cedar & Stone LLC submits four months of bank statements. The tool extracts monthly deposits, flags that March's listed deposits sum to $1,140 less than the stated total, and links both figures to page 3. The analyst opens the page, finds a deposit line split across a page break that extraction missed, corrects it, and the correction is logged. The memo quotes the corrected figure with its source link. Nobody re-keyed a statement, and the one error that mattered was caught by arithmetic, not luck.

A number can also be extracted perfectly from a statement that was edited before it reached you. That is a separate check, covered in how to detect fake bank statements and edited PDFs, and it belongs with fraud signals in your evaluation.

04Does an AI-drafted credit memo still need a named human sign-off before committee?

Yes. AI can draft the memo; a named person signs it. The person who signs has read it, checked the figures that drive the decision, and accepts responsibility for the recommendation that goes to committee or to the final approver. An unsigned or system-signed memo leaves no one accountable for its content.

This is not a formality. A drafted memo can be fluent and wrong: it can describe a deposit trend that the numbers do not support, smooth over a verification conflict, or quote a figure from the wrong statement period. The signature is the control that forces a person to look. It also answers the first question anyone reviewing the file later will ask, which is who made this call and on what basis.

When you evaluate vendors, test the sign-off mechanics directly:

• Does the memo record the drafter (the system and its version) separately from the approver (a named user)?
• Can the approver edit the draft, and are edits recorded against the original text?
• Can a memo reach a decided state without a named person acting on it? If the answer is yes, find out who configured that and why.
• Does sign-off capture the evidence the approver relied on, or just a timestamp?
• When the underwriter disagrees with a flag or a policy result, is the override recorded with a reason? Our guide to policy exceptions and overrides covers what a good override record includes.

Be wary of any vendor that describes sign-off as a setting you can turn off once you trust the model. Our guide to human-in-the-loop AI underwriting explains where automation earns its place and where judgment has to stay.

05What does an AI pre-screen actually include compared to a full credit memo?

A pre-screen is a fast triage: it reads the submission, pulls headline figures and applies knockout rules so a team can decide whether a file is worth full review. A full credit memo is the decision record: verified financials, business verification, fraud review, policy results, exceptions, the underwriter's analysis and a recommendation with a named signature.

Pin down which one a vendor is demonstrating. A typical pre-screen covers:

• Document completeness: are the required statements, application and IDs present and for the right periods?
• Headline figures: average monthly deposits, average daily balance, count of negative-balance days, visible existing advances.
• Hard knockouts from your policy: minimum time in business, minimum revenue, restricted industries, states you do not fund.
• Obvious red flags worth a person's attention before anyone spends an hour on the file.

A full memo adds what a pre-screen skips: verification of the extracted figures, business verification and ownership, fraud and document-integrity review, evaluation against the full policy including exceptions, position and stacking analysis, the underwriter's narrative and a signed recommendation. Our guide to the AI credit memo lists the sections a decision-ready memo must contain.

The due diligence point is this: a pre-screen output should never be presented as a decision, and its figures should not flow into a memo unverified. Ask the vendor to show how a pre-screen result is labelled, whether its numbers are marked as unverified until checked, and whether a declined pre-screen can be reopened by a person. A pre-screen that silently declines files is an automated decision system, and it needs to be evaluated as one.

06The question bank, part one: data, extraction, explainability and model change

Send these questions in writing before the demo and ask for written answers. A vendor's written answer becomes part of your due diligence file; a verbal one does not. For each area, we list the questions, then what a good answer and an evasive answer tend to sound like.

1. Data provenance and source links

• Can every extracted value and every statement in the memo be traced to a source document, page and region?
• What data does the system use beyond what we upload (third-party data, public records, web data), and is each such item labelled with its source and retrieval date?
• Is our data used to train or improve models that serve other customers? Can we opt out, and is that in the contract?
• Where is our data processed and stored, and by which subprocessors?

Good answer: a live click from a memo sentence to the highlighted page, a list of external data sources with retrieval timestamps shown in the file, and a contractual position on training use. Evasive answer: "Our AI understands the documents holistically", or a training-data policy that exists only in a sales deck.

2. Extraction verification and sampling

• How does the product show extraction uncertainty: confidence markers, tie-out failures, unreadable regions?
• Which values does the product itself check arithmetically, and what happens when a check fails?
• How are reviewer corrections recorded, and can we report on correction rates by document type or bank?
• How do you measure extraction quality, on what documents, and will you measure it on ours?

Good answer: the vendor shows a failed tie-out on a real file, explains how its quality is measured and offers to measure it on your sample. Evasive answer: a single accuracy percentage with no description of the test set, the document types or who did the counting. Treat any headline accuracy figure as unverified until it has been reproduced on your files.

3. Explainability of scores and flags

• Use the six explainability questions earlier in this guide, asked against one of your own files.
• Is every score or flag labelled with what kind of logic produced it: a policy rule, a statistical model or a generative model?
• Are reasons stored with the decision record at the time they were produced?

Good answer: the reason, the inputs and the source pages appear together, and the logic type is visible. Evasive answer: "The model weighs hundreds of signals", which describes complexity, not a reason.

4. Model risk management and change control

• Which components are models, which are rules and which are third-party models (for example, a hosted language model)? Who provides each?
• How are model and prompt changes versioned, tested and released? Is the version that produced each output recorded on the file?
• How much notice do customers get before a change that can alter outputs, and can we test it on our files first?
• Can we pin a version for a period, or roll back, if a change misbehaves?
• What documentation can we have: model descriptions, intended use, known limitations, test results?

One nuance from SR 26-2: it defines a model as a complex quantitative method grounded in statistical, economic or financial theory, excludes deterministic rule-based processes, and leaves generative and agentic AI outside its scope while saying a bank's own risk management should still govern such tools. A modern underwriting tool can contain all three kinds of logic, and a good vendor can tell you which is which.

Good answer: a component inventory, version stamps on outputs, a written change-notice policy and a way to re-run your test files on a new version. Evasive answer: "We improve the model continuously", with no version history you can see.

07The question bank, part two: oversight, records, security and exit

The second half of the bank covers what happens around the outputs: who checks them, what is kept, how data is protected, what evidence the vendor gives you about itself, and how you leave.

5. Validation and ongoing monitoring

• What validation has been done on the components that matter for our use, by whom, and can we see a summary?
• What does the vendor monitor in production: drift in document types, rising correction rates, changes in flag rates?
• Will the vendor support our own outcomes analysis, for example by exporting outputs and reviewer actions so we can compare them with performance later?

The revised model risk guidance describes outcomes analysis as comparing model outputs with real-world outcomes, and expects ongoing monitoring of vendor models to assess whether they remain fit for purpose. You can only do that if you can get the outputs out. Good answer: a monitoring description plus a clean export. Evasive answer: "Our models are validated", with nothing you can read.

6. Human sign-off and overrides

• Can any file reach a final decision without a named person acting on it?
• Are overrides of flags and policy results recorded with a reason, the user and the time?
• Can we set role-based limits, for example exceptions above a size need a second approver?
• Can we report on override rates by underwriter, by rule and by flag type?

Good answer: a demonstration of an override with a mandatory reason and a report on override patterns. Evasive answer: "Users can always change anything", which is not the same as recording it.

7. Audit trail and record retention

• For a file decided last year, can we reproduce what the underwriter saw: the documents, extracted values, flags, policy version, memo draft and edits?
• Is the audit log append-only? Who can alter or delete records, and is that itself logged?
• How long are documents and decision records retained, and can we set retention to match our own obligations?
• Can we export a complete file record in a readable format without the vendor's help?

Good answer: the vendor reconstructs an old file in front of you. Evasive answer: "Everything is logged", without showing what a log entry contains. Our guide to underwriting audit trails describes what a reconstructable decision record needs.

8. Security and data handling

• Which independent assessments or audit reports exist today, and which are planned with dates? Ask for the report, not a logo.
• How is borrower data encrypted, segregated between customers and access-controlled internally?
• Which subprocessors, including AI model providers, receive borrower data, and do they retain or train on it?
• What is the incident notification commitment, in hours, in the contract?

Good answer: precise statements about what exists now and what does not yet, with documents. Evasive answer: "Bank-grade security", or certifications described as in progress with no date or scope.

9. Third-party risk management

• What are the vendor's ownership, financial position and runway, and who are its critical subcontractors?
• Will the contract give us audit rights, or at least specify which audit reports we receive and how often?
• Does the contract state that data generated from our files is ours?
• Where we are a bank or serve one, does the contract acknowledge that the vendor's work may be subject to regulatory examination, with access to relevant records?

These map to the contract topics in the interagency third-party guidance, which discusses the right to audit (including the types of audit reports a bank is entitled to receive), ownership of data generated by the third party, reliance on subcontractors, and regulatory supervision. Good answer: a vendor that has seen these clauses before and has a standard position. Evasive answer: reluctance to discuss its own subcontractors.

10. Exit and portability

• If we leave, what do we get back: original documents, extracted data, decision records and audit logs, in what format and within what time?
• Will the vendor confirm deletion of our data after transfer, in writing?
• Are our policy rules exportable in a form we can read, or do they live only inside the product?
• What notice period and termination costs apply?

The third-party guidance suggests contracts provide for orderly transition without prohibitive expense and for the timely return or destruction of the bank's data. Exit is the area most often skipped and most expensive to discover late. Good answer: a documented export, tested in the proof of concept. Evasive answer: "We have never had a customer leave."

08How do you run a proof of concept on your own historical files?

Pick a set of files you have already decided, hide the outcomes from the vendor, have the tool process them, and have your own underwriters score the outputs against what they know happened. Then test change control and exit while you still have leverage. A demo on the vendor's sample files tells you very little.

Choosing the files

Choose files that look like your real pipeline: funded files that performed and that went bad, declined files, files where you caught fraud, and awkward documents such as poor scans, unusual bank layouts and multiple accounts. Include files near your policy thresholds, where a misread number changes the answer. Before the run, write an answer sheet for each file: key figures, expected flags and policy exceptions.

Historical files contain borrower data, so sign an agreement first covering confidentiality, use limited to the evaluation, no training on your files and deletion at the end.

Running and scoring

Have the vendor process the files without knowing outcomes, then have two underwriters review each output independently: check sampled figures against source pages, record corrections, judge each flag as useful, noise or missed, and rate whether the draft memo was safe to sign. Time the review, because output your team must re-check line by line has saved nothing.

Useful measures are the share of sampled values that needed correction, flags judged useful versus noise, known issues the tool missed, and memo statements without a working source link. Keep the raw sheet as evidence and as a baseline for later monitoring.

Drills most evaluations skip

• Change drill. Ask the vendor to describe the last release that changed outputs and show the notice customers received. If possible, re-run five files on a newer version and compare.
• Reconstruction drill. Pick a file processed in week two and ask for the complete record as it stood then, including the versions used.
• Exit drill. Request a full export of everything from the proof of concept. Open it without the vendor's help. Then request deletion and the written confirmation.
• Override drill. Override a flag and a policy result, and check that both are recorded with your reason.

09How do you score AI underwriting vendors fairly?

Weight the ten areas by how much they matter to your process, score each from 0 to 3 based on the strength of evidence, and only award top marks for things demonstrated on your own files. A weighted rubric stops a polished demo from outscoring a vendor whose answers held up in writing and in testing.

The scale is about evidence, not enthusiasm. A 1 means a salesperson said it. A 2 means it is in writing, in documentation or the contract. A 3 means you watched it work on your files. The weights above are illustrative: a lender with a bank partner may weight model change control and third-party evidence higher; a high-volume MCA funder may weight extraction verification higher still.

Two rules keep the rubric honest. Set pass-or-fail minimums for security, data handling and human sign-off: a vendor that cannot keep a named person in the decision is out regardless of its total. And score before the commercial conversation, so price compares vendors that passed rather than rescuing one that did not.

10Is an in-house AI underwriting tool or a vendor platform easier to defend to a bank examiner?

Neither is automatically easier; the easier one to defend is the one with better evidence. An in-house tool gives you full access to code and data but makes you produce all the documentation, validation and change control yourself. A vendor platform can supply much of that material, but you must obtain it, review it and add your own testing. Responsibility stays with you either way.

The interagency third-party guidance is plain on the central point: a banking organization's use of third parties does not diminish its responsibility to operate safely and soundly and in compliance with law, to the same extent as if the activity were performed in-house. Model risk guidance takes the same line for vendor models, acknowledging that banks may not see proprietary code or data while expecting validation and monitoring anyway. So the question an examiner or a bank partner is really asking is whether you understand and control the tool, not who wrote it.

Each route has a typical weak spot. In-house builds tend to fail on documentation and independence: only the builders understand the tool and changes ship without a record. SR 26-2 describes effective challenge as critical analysis by people with the expertise, independence and standing to force changes, which is hard to arrange inside a small team. Vendor platforms tend to fail on access: no audit rights and no way to reproduce last year's outputs. The question bank is largely a list of ways to close that gap.

A practical test for either route: for one declined and one funded file from last quarter, can you show the documents, the figures and their sources, why each flag fired, the policy version, the overrides, the memo and the signer? If yes, your choice is defensible.

From here, this article is about Cevrynt

11How Cevrynt answers these questions

Cevrynt is an MCA platform for funders and alternative lenders: AI-assisted underwriting built around the questions in this playbook, plus a built-in MCA CRM for the deal pipeline, ISOs and brokers, offers and contracts, and funding and servicing, so the underwriting record and the deal record sit in one place. The platform structures borrower documents with source-linked extraction, analyzes bank statements, verification and fraud signals, evaluates files against each lender's own policy, and records reviewer notes, overrides and audit history. The underwriting report keeps each finding linked to the page it came from, so the person who signs can check the figures that matter.

Cevrynt is not a lender and does not make funding decisions. Fraud and policy outputs are flags and exceptions for a person to review, never automatic declines, and the lender's underwriter makes and signs the decision. You can read how we describe our approach to data protection on the security page.

If you are running a vendor evaluation, we expect to be tested the way this guide describes. A walkthrough is the place to start, and a scoped pilot on your own files is how to see the evidence for yourself.