An underwriting software pilot is a bounded test in which a lender runs files its own team has already reviewed, or is reviewing, through the software and compares the output with its underwriters' work against criteria agreed in advance. A useful pilot takes about 90 days: 30 to prepare, 30 to run side by side, and 30 to measure and decide.
This guide is the hands-on plan for that period, written for MCA funders and SMB lenders. It does not repeat how to shortlist vendors, which our buyer's guide to MCA underwriting software covers, or the case for keeping underwriters in control, which is in human-in-the-loop AI underwriting. It starts once you have a candidate tool and covers success criteria, historical versus live files, sample design, the data and security review, shadow review, the measures that matter, policy encoding, change management for underwriters, the go/no-go decision, and the ways pilots most often fail. A scorecard template you can copy sits near the end.
What is an underwriting software pilot, and how long should it take?
A pilot is a time-boxed evaluation on your own files, with a written question and a decision at the end; a demo is the vendor's files and a trial is open-ended use with no decision rule. Ninety days, split into prepare, run and decide, is long enough to gather evidence and short enough to keep attention.
A demo shows the product on files chosen to show it well; a trial shows whether people like using it. Only a pilot answers the question a credit or operations leader actually has to sign: does this tool make our underwriting better on our files, under our policy, with our people, by a margin worth the change? That question needs a sample, a comparison and a rule for deciding, which is why a pilot is closer to a small study than to a product tour.
Days 1-30: prepare
The first month produces no results and should not try to. It produces a signed one-page pilot charter (the question, scope, measures and decision rule), a pulled and labelled sample of files, a completed data and security review, and your credit policy written down in the form the software will apply. Gate 1 asks one thing: could we start running files tomorrow without changing any of these? If the answer is no, the run waits. Starting with an unsettled sample or a half-written policy is the most common way to burn the middle month.
Days 31-60: run side by side
The middle month is the run. Files go through in weekly batches, a named reviewer compares each output with the file's recorded review, and every difference is logged at field level. A short weekly calibration session settles disputed comparisons so that the rules for scoring stay consistent from the first batch to the last. Gate 2 asks whether you have enough comparisons, across enough of the sample, to read the measures with any confidence.
Days 61-90: decide
The last month is for an optional live shadow on incoming files, completing the scorecard, and making the call: go, extend with a narrower question, or stop. A stop still leaves a written record of where the tool fell short, which is worth keeping.
Ninety days is a default, not a rule. What should not shrink is the preparation. People searching for fast implementation underwriting software for merchant cash advance lenders usually want the run to start quickly, and the honest route to that is a narrow scope and historical files that need no integration, not skipping the charter.
What success criteria should you agree before the pilot starts?
Write down, before any file runs, the one question the pilot answers, the four to six measures that answer it, the baseline for each, the threshold that counts as good enough, and who makes the final call. Criteria written after results arrive tend to describe the results rather than test them.
Start with the question. "Is the tool good?" cannot be answered. "Can the tool prepare the financial section of a renewal file so that our analyst only checks it, without rebuilding it?" can. A good question names a stage, a file type, the expected change in reviewer work, and what is out of scope. For an underwriting tool, the credit decision itself should always be outside the comparison: approval rates move with your policy and appetite, not with how well a file was prepared.
Then pick measures that a reviewer can check on a single file. Time per file, field-level extraction errors, whether known issues were surfaced, whether flags were useful, and whether policy results matched your team's are all checkable. "Better risk assessment" is not, at least not within 90 days, because loss outcomes on MCA and SMB files take months to mature.
Set the baseline from your own process
A threshold means nothing without a baseline. Before the run, time a handful of files through your current process, from the moment a reviewer opens the submission to the moment the file is ready for a credit decision. Record how often your own team re-keys a figure wrongly, if you track it, and how often files come back for a second pass. If you do not have these numbers, say so in the charter and measure them on the pilot's control files rather than estimating them. Our piece on underwriting turnaround time shows where review time usually goes.
Write the decision rule down
Decide in advance what each outcome leads to. A common shape is three tiers: every must-have criterion met means go; a must-have missed for a reason the vendor can fix within a defined period means extend with a narrower question; a must-have missed for a structural reason means stop. Zero tolerance for misread figures on fields your policy depends on, such as deposits, ending balances and existing daily debits, is a reasonable must-have.
Should an MCA pilot use historical files or live deals?
Start an MCA pilot on historical files your team has already reviewed, because you already know the right answer for each one and nothing depends on the output. Add a live shadow on incoming deals only after the historical run passes, and keep your existing process deciding every live file.
Historical files
Historical files are the best evidence per hour spent. Each file comes with a recorded review: the figures your analyst keyed, the issues your reviewer raised, the policy exceptions recorded and the decision taken. That record is the answer key. You can choose files deliberately, including declined deals, files with confirmed tampering or stacking, and the awkward submissions that slow your team down. Because nothing is funded on the output, you avoid any risk that a pilot affects a real merchant.
The limits are real too. Historical files may be cleaner than today's submissions if your broker mix or document sources have changed. And the recorded review is not always right; part of the pilot's value is finding where your own record was incomplete, so plan for a way to settle those cases rather than scoring the tool against an imperfect key without question.
Live shadow
In a live shadow, incoming files go through your normal process and through the software in parallel, and only your normal process feeds the decision. It tests today's submissions under real turnaround pressure, but it costs more, and the right answer is not known until your own review finishes. A live shadow is a good final phase, not a good first one.
How many files does a pilot need, and which ones?
Most lenders get a usable read from roughly 40 to 100 historical files, chosen on purpose rather than at random, with the mix weighted toward the files that cost your team time. The right number depends on how many distinct file types you need to see, not on a statistical target.
This answers a question that comes up often in evaluations: what trial or pilot options exist for merchant underwriting datasets? The most useful dataset is almost never a vendor's sample set. It is a labelled slice of your own closed files, because only your files carry your broker mix, your merchants' banks and your policy's edge cases. A pilot that runs only on synthetic or vendor-supplied files tells you about the vendor's files.
Build the sample in strata
- Routine funded files: the baseline. If the tool cannot handle clean files well, nothing else matters.
- Funded files that later struggled: look for early-warning evidence in the original documents, such as rising NSF counts or new daily debits. See NSF patterns and overdrafts for what those signals look like.
- Declined files: check that the reasons your team declined for show up in the output, and that the tool does not make a declined file look clean.
- Edge cases: scanned or photographed statements, multiple operating accounts, statements from small regional banks, seasonal businesses, very new businesses with thin files, and mixed submissions with documents out of order.
- Known issues: a small set of files where your team confirmed an altered statement, an undisclosed position or an identity problem. These are your seeded tests, covered in detecting stacking and how to detect fake bank statements.
Pull files from the last six to twelve months so bank formats and broker habits match today's, label each with its stratum, issues found and policy outcome, and do not tell the vendor which files carry known issues.
Keep a handful of files back as a holdout. If the vendor adjusts configuration during the run, which is normal, the holdout tells you whether the adjustment generalised or only fixed the files everyone had already looked at.
What data handling and security review comes before the first file?
Before any borrower file leaves your control, settle who can access it, where it is processed and stored, how long it is kept, how it is removed, and what is logged. Treat the pilot as the first stage of a third-party relationship, not as an exception to your normal vendor review.
Pilot files are real bank statements, tax returns and identity documents, often for merchants who never became customers. The risk is the same whether the contract says pilot or production. A useful reference is the interagency guidance on third-party relationships issued in June 2023 by the FDIC, the Federal Reserve and the OCC, which describes a life cycle of planning, due diligence, contract negotiation, ongoing monitoring and termination (FDIC FIL-29-2023). It is written for banking organizations and, as the Federal Reserve's transmittal letter notes, describes risk management principles rather than imposing new requirements (SR 23-4). Non-bank MCA funders are not its direct audience, but if you fund through or alongside a bank, expect partner questions shaped by it, and its life cycle is a sensible structure regardless. State law and your own contracts may add requirements that vary, so involve counsel where they apply.
Map the life cycle onto the pilot: the charter is the plan, due diligence covers access, hosting, subprocessors and data use, the contract covers confidentiality, retention and deletion, monitoring means reviewing access logs, and termination means confirmed deletion whether or not you go ahead.
- Can we send redacted or minimised files, and which fields does the pilot actually need?
- Where are files processed and stored, and by which subprocessors?
- Who at the vendor can open a file, and is that access logged and reviewable by us?
- Is our data used to train or tune anything used by other customers?
- How long are files and derived data kept, and how is deletion confirmed?
A pilot comparison of underwriting output proves nothing about security, so keep the two reviews separate. Cevrynt's security page sets out how it approaches access, data handling and audit, and the pilot page treats data handling as a set of decisions settled before a file moves.
How does a side-by-side shadow review work?
In a shadow review, a named reviewer puts the software's output next to the file's recorded review and logs, signal by signal and field by field, whether the two agree, where the tool found something the team missed, where it raised noise, and where it got a value wrong.
Each comparison falls into one of five outcomes. Both agree. The tool raised something the team missed, and the reviewer confirms it is real. The tool raised something that is not a real issue. The team found something the tool missed. Or the tool read a value wrongly. Log every one, with the field or signal, the file, and the reviewer's note. The pattern across the log tells you far more than a single headline score.
Rules that keep the comparison fair
- Agree the comparison rules first. For example, a deposit total within a cent counts as a match; a balance on the wrong date does not.
- Blind where you can. Ideally the scorer is not the person who underwrote the file.
- Check the source, not the summary. When the tool and the record disagree, open the statement page. Source-linked extraction makes this fast; without it, comparisons slow to a crawl.
- Settle disputes weekly. One person, usually the credit lead, rules on contested comparisons and the ruling is written down so later batches are scored the same way.
- Never feed the decision. In a live shadow, the tool's output must not reach the underwriter deciding the live deal until that review is recorded.
When the tool finds something your record missed, confirm it on the source and log it as a tool-only confirmed finding. Do not quietly correct your answer key, or you lose the evidence that the tool added value.
What should a cash flow underwriting pilot measure?
A cash flow underwriting pilot should measure reviewer time per file, field-level errors on the figures your policy depends on, whether known issues were surfaced, how many flags were useful, and how often policy results agree with your team's. Each is checkable per file and none depends on loss outcomes that take months to arrive.
Time per file
Measure reviewer time, not processing time. The relevant clock runs from when a reviewer opens the file to when it is ready for a credit decision, and it includes checking the tool's output and fixing anything wrong. A tool that produces a report instantly but takes forty minutes to verify has not saved forty minutes. Report the median and the spread, and report edge cases separately, because averages hide the files where a tool struggles. Time early batches separately, because reviewers are slower while learning a new screen.
Field-level extraction errors
Score extraction per field, not per document. A statement can be 98 percent right and still be useless if the one wrong value is the ending balance your policy uses. List the critical fields before the run, typically deposits by month, ending and average balances, NSF and overdraft counts, recurring debits that look like existing positions, and account identifiers. Track each as correct, wrong, or missing, and note why: a poor scan, an unusual bank layout, a page out of order. The causes tell you whether the errors are fixable. For the underlying metrics, see cash flow underwriting for SMB loans and DSCR and average daily balance.
Flag usefulness
Raw flag counts reward noise. For each flag, ask the reviewer one question: did this change what you looked at or what you asked for? Classify flags as acted on, noted but not material, or noise. A tool that misses a seeded altered statement has failed a must-have, however many minor flags it raises. Fraud and policy signals are prompts for a person to look, so judge them by whether they direct attention well, not by whether they make the call.
Reviewer agreement and policy results
Where the tool applies your policy, compare each criterion's result with your team's recorded result: pass, fail or exception. Disagreements usually trace to one of three causes: a misread input, a rule encoded differently from how your team applies it, or a case where your team made a judgment the written policy does not describe. The third is valuable, because it shows where your policy and your practice have drifted apart. Measuring agreement between two of your own reviewers on a few files shows how much disagreement is normal.
A worked example, with illustrative numbers
Suppose, purely as an illustration, that a funder runs 60 historical files. On the routine files, reviewers spend noticeably less time because they check figures instead of keying them. On scanned statements, two ending balances are misread, which fails the critical-field must-have for that stratum. Of five seeded tampering files, four are surfaced and one is missed because the edit was in a scanned page. Most flags are acted on, but round-number transfers between the owner's accounts are flagged on almost every file and reviewers dismiss them. The point is the shape of the result, not the numbers. The decision rule turns that shape into an action: extend 30 days, fix scanned-statement handling and transfer-flag tuning, and rerun the scanned and seeded files plus the holdout.
How do you encode credit policy and exceptions during a pilot?
Write your policy as your team actually applies it, criterion by criterion, with thresholds, the data each criterion reads, the exceptions allowed and who may approve them. Then configure the tool to that document and version it, so every pilot result can be traced to the policy it was run against.
Most lenders discover during a pilot that their policy lives in three places: a written credit policy, a spreadsheet or rule set in their current system, and the heads of senior underwriters. The pilot forces them into one document. That is where pilots slip, because the underwriters who hold the unwritten rules are the busiest people in the building. Book their time in the first month.
- Criterion: for example, minimum average daily balance over the statement period.
- Threshold: the number, the window and whether it is inclusive.
- Input: exactly which extracted figure it reads, so a disagreement can be traced.
- Exception path: whether a miss is a hard stop under your policy or an exception, what evidence an exception needs and who can approve it.
- Version: the policy version and date, recorded with every result.
Do not let the pilot quietly change your policy. If the vendor suggests different thresholds, that is a separate credit conversation, not a configuration tweak. And keep exceptions as exceptions. A criterion your team regularly overrides with a documented reason should show up in the tool as an exception for a reviewer to settle, with the reasoning recorded, not as an automatic decline. Our guides to loan policy engines and policy exceptions, overrides and audit trails go deeper on both.
How do you bring underwriters along during a pilot?
Involve underwriters as evaluators from the first week, not as users at the end. Tell them plainly what the pilot measures and what it does not, keep their decision authority explicit, and treat their objections as data for the scorecard.
Underwriters reasonably read a software pilot as a question about their jobs. Framed as "can this replace review", it gives the people best placed to judge it every reason to find fault. Frame it instead as "can this take the keying and cross-checking off your desk so you spend your time on judgment", and say in writing that the credit decision stays with them. That is not only a message; it should be true of the pilot design, which keeps the decision outside the comparison.
- Pick a pilot reviewer group. Two or three underwriters with a mix of seniority, with time set aside for the run rather than squeezed in around a full pipeline.
- Give them the scoring rules. Reviewers who helped write the comparison rules trust the results.
- Collect friction, not just errors. Where did they leave the tool, rebuild a figure or hunt for a source?
- Ask a confidence question at the end. Would you want to work files this way? What would have to change first?
This is also a governance point. The NIST AI Risk Management Framework, a voluntary framework released in January 2023, organises AI risk work into four functions, Govern, Map, Measure and Manage (NIST AI RMF). Its Map function calls for processes for human oversight to be defined and documented. A pilot is a good time to write down who reviews what the tool produces, who can override it and how overrides are recorded.
How do you make the go/no-go decision and scale?
Fill in the scorecard agreed at the start, apply the written decision rule, and choose go, extend or stop. Go leads to a scaling plan that widens scope one step at a time; extend narrows the question; stop produces a written record of why.
NIST's AI RMF describes testing AI systems before deployment and regularly while in operation, and its Manage function includes a determination of whether a system meets its intended purpose and whether deployment should proceed (AI RMF core). The pilot is that pre-deployment test.
| Criterion | Measured how | Baseline | Threshold | Must-have? | Reading | Status |
|---|---|---|---|---|---|---|
| Reviewer time per file | Median and spread, file open to decision-ready | Your current process, timed | Set at kickoff | Yes | ||
| Critical-field errors | Wrong or missing values, per listed field | Your re-keying error rate, if known | Set at kickoff | Yes | ||
| Known issues surfaced | Seeded files where the confirmed issue was raised | All confirmed by your team | Set at kickoff | Yes | ||
| Flag usefulness | Flags acted on, noted, or noise, per reviewer | Not applicable | Set at kickoff | No | ||
| Policy result agreement | Criteria where pass, fail or exception matches the record | Agreement between two of your reviewers | Set at kickoff | Yes | ||
| Source traceability | Share of figures a reviewer could trace to a page | Not applicable | Set at kickoff | No | ||
| Underwriter confidence | End-of-pilot questions to pilot reviewers | Not applicable | Majority would keep using it | No | ||
| Data handling | Security review passed, deletion confirmed at exit | Your vendor standard | Pass | Yes |
Scaling without losing what the pilot proved
Underwriting software implementation after a pilot should widen one dimension at a time: more stages, more file types or more reviewers, not all three at once. Integration with your intake, LOS or CRM belongs here, scoped around the systems that already move your deals; Cevrynt's integrations page describes how that is approached. Keep a small, regular sample of files under side-by-side review after go-live, so drift in document formats or broker behaviour shows up as a change in the log rather than as a surprise in the portfolio.
Why do underwriting software pilots fail?
Most failed pilots fail on design rather than on the software: no written question, a sample of easy files, criteria invented after the results, reviewers with no time, or a security review that starts after the files have already been sent. Each failure is avoidable in the first 30 days.
- No question, so no answer. The pilot ends with impressions, and the decision goes to whoever argues hardest.
- A clean sample. Only routine files are run, the tool does well, and production brings the scans and multi-account files nobody tested.
- Moving goalposts. Thresholds are set after results arrive, either to justify a purchase already made or to reject a tool someone dislikes.
- Scoring against a flawed key. The recorded review is treated as perfect, so confirmed tool-only findings are scored as false alarms.
- No reviewer time. Underwriters do comparisons around a full pipeline, comparisons slip, and gate 2 arrives with a third of the sample done.
- Integration first. The pilot waits weeks on IT work before a single file runs, when historical files would have answered the question without it.
- Measuring the decision. Approval rate is used as a success measure, which tests your appetite rather than the tool's preparation of the file.
- Security as an afterthought. Files are shared before the data review, and the pilot is paused or abandoned when the review finally happens.
From here, this article is about Cevrynt
How Cevrynt approaches an underwriting pilot
Cevrynt's pilot follows the shape described in this guide. The Cevrynt team works with a lender to define a narrow workflow, representative files, review criteria and evaluation goals. A pilot is drawn around part of the workflow, commonly one stage such as documents, a run of stages from documents to verification, or the whole file from intake to report, and it runs on representative historical files the lender's team has already reviewed, so it does not need an integration first. The comparisons are agreed before any file runs: extracted values, financial calculations, source links, conflicts surfaced, policy results, missed flags and reviewer corrections.
Some things are outside every scope. Cevrynt is not a lender and issues no approval or decline; approval rate is deliberately left out of the comparison, and the credit decision stays with the lender's underwriters. Policy is configured from the lender's own criteria, thresholds and exception paths, and a policy break is an exception with reasoning for a reviewer to settle. Data handling is settled as its own review before a file moves, and Cevrynt publishes no accuracy figures to compare against, because every result depends on one lender's files and policy.
To see how the stages fit together, start with the platform overview, then the stage pages for document intelligence, bank statement analysis, fraud signals, the policy engine and the underwriting report. If you are scoping an evaluation, you can discuss a pilot with the team or email sales@cevrynt.com.