From AI pilot to production

Gartner expected at least 30% of generative AI projects to be abandoned after proof of concept by the end of 2025, citing poor data quality, inadequate risk controls, escalating costs and unclear business value (Gartner, July 2024). In June 2025 it forecast that over 40% of agentic AI projects will be cancelled by the end of 2027 for similar reasons, and noted that many use cases positioned as agentic do not require agentic implementations (Gartner, June 2025).

Most of these causes are settled before any model is built, by the choice of decision, the measure of success and the plan for running the result every day.

Choose a decision before choosing a technology: approving a payment, setting a credit limit, routing an application, choosing who receives an offer. Five questions describe it well enough to plan a pilot.

The answers point to the tool. Most processes use more than one, chained so that the fast, inexpensive step decides where the costlier one spends its time.

Their costs differ in shape. An agent is cheap to start and paid for on every case. A predictive model needs investment up front in data and training, then costs very little per decision, although keeping it healthy is a running cost. Prices are falling: Epoch AI measured the price of a given level of model performance falling by 9 to 900 times a year depending on the task, and noted that the fastest falls are recent and may not persist (Epoch AI, March 2025).

Before the pilot begins, write these down and have the owner of the decision sign them.

This gives the finance team a result it can check rather than a claim to take on trust.

Run the pilot on your own history first, then on live traffic beside the current system, in shadow: the new model scores every case and decides nothing. Customers see no change, and the team compares both systems on the same cases. Public datasets and vendor benchmarks rarely reflect your customers and fraud patterns.

Shadow running also shows how often inputs arrive late or missing, how long answers take at peak hours, and where the two systems disagree. Review a sample of those disagreements with the people who make the decision today.

Most of the work between a good pilot and a live system is engineering and governance. Start it while the pilot runs.

Supervisors expect this. The revised US model risk guidance from the Federal Reserve, the OCC and the FDIC (SR 26-2, April 2026, which replaced SR 11-7) says validation generally occurs before a model's first use, and covers outcomes analysis and ongoing monitoring. In the UAE, the Central Bank, SCA, DFSA and FSRA jointly issued guidelines in 2021 for financial institutions adopting enabling technologies, artificial intelligence among them. Bring compliance and model risk teams into the pilot itself.

A pilot budget usually covers licences and engineering. A production budget should be stated per decision and include inference, integration, monitoring, retraining and the time of the people who review flagged cases. A model that sends more cases to review can cost more than it saves.

Report the change in outcome per decision, multiplied by volume, less the full cost, measured against the control group.

Mantika pilots run on your own data beside your current systems, and nothing goes live until you approve it.