AI AgentsLast updated

How to move agentic AI from pilot to production

Evaluation sets, guardrails, approvals and monitoring: what changes when an AI agent leaves the pilot and takes on real work in your operations.

Joseph StalinChief Technology Officer6 min read
In short

An agent leaves the pilot when four things are true: it runs on a fixed evaluation set of real cases with recorded scores, every action that commits money or stock sits behind an approval, each tool has scoped permissions, and logs and monitoring are live before real users touch it. Production is an engineering milestone, not a bigger demo.

Key takeaways
  1. 01A pilot proves one path works; production needs a test set, an owner and a rollback.
  2. 02Score every change against real cases, so tuning is evidence rather than opinion.
  3. 03Approvals, scoped tools, cost limits and audit logs go in before the audience grows.
  4. 04Keep the old process running until the review and correction numbers are clear enough to widen access.

An agent leaves the pilot when four things are true: it runs on a fixed evaluation set of real cases with recorded scores, every action that commits money or stock sits behind an approval, each tool has scoped permissions, and logs and monitoring are live before real users touch it. Production is an engineering milestone, not a bigger demo.

Below: why pilots stall, what the evaluation set looks like, which guardrails go in first, how monitoring works, and how the rollout runs.

Why do agentic AI pilots stall?

Most pilots are built to show one path working. A sample of documents, one enthusiastic user, a demo on a good day. That proves the model can read a tender or draft a reply. It does not prove the agent survives a Monday morning queue, an empty field, a scanned attachment or a request in two languages.

The usual blockers are structural, not technical:

  • No test set, so nobody can say whether a change made the agent better or worse
  • Ad-hoc data access, granted by hand for the pilot and impossible to audit later
  • No defined failure path for the case where extraction is wrong
  • No owner after the pilot team moves on
  • The pilot never wrote to a system of record, so the hard part was never attempted

What does production mean for an AI agent?

An agent is in production when it handles real volume with a named owner, fails visibly, and can be changed without guesswork. In practice that means four properties: it runs on a schedule or trigger rather than on request, every run is logged, every change is checked against the evaluation set, and there is a rollback path.

It also means the surrounding work is finished. Someone monitors the queue. Approvals reach a person who can act. The infrastructure side covers alerting, monitoring and tested backups, which sits under AI integration, cloud and DevOps.

What evaluation set does an agent need before go-live?

Build it from real examples. Past RFQs, supplier quotes, service tickets, delivery notes, emails in Arabic and English, including the messy ones: scanned pages, mixed units, duplicate line items, a customer who changed the quantity mid-thread.

Each case needs an expected output or a written rubric, and a score you can compare over time. Then run the whole set on every prompt, retrieval or model change. Without that, tuning becomes opinion, and the agent drifts quietly between releases.

The evaluation set is a deliverable, not an afterthought. Our 2 to 3 week discovery sprint ends with a working prototype on your own data, the evaluation set and a costed plan for production, as described on AI agent development.

Which guardrails go in before real users touch it?

Put the controls in before the audience grows, because retrofitting permissions onto a live agent is how incidents happen.

  • Scoped credentials per agent, read-only first, then write tools one action at a time
  • Human approval for anything that commits money, stock or a customer promise
  • Limits on steps, cost per task and run time, so a loop cannot run all night
  • A log line for every tool call, input and output, tied to the person who approved it
  • An error queue for failed integration messages, so a failed write is visible and can be replayed
  • Role-based control over who may change an agent's scope, not only who may use it

Hosting is a guardrail too. Where data can leave your network we use OpenAI or Gemini under your own account; where it cannot, we run self-hosted open models through Ollama on your servers, in a cloud region you choose or on edge hardware. Teams running several agents with shared roles, secrets kept apart from workflow definitions, mandatory 2FA and audit logs can use Hivelock, our AI agents platform, currently in beta.

How do you monitor an agent after launch?

Watch the work, not only the uptime. Useful signals are the volume in the review queue, how often reviewers reject or correct an output, how many runs end in escalation, and which document types or suppliers produce the most corrections. Those numbers tell you where the agent is weak and whether widening its permissions is safe.

Keep the old process running alongside the agent until the numbers are clear. When they are, widen one permission or add one workflow, reusing the same tools and test set rather than starting again. Anything that reads and writes records in your systems should follow the rules in how to integrate AI into an existing ERP.

How does the move from pilot to production run?

Every engagement starts with a 2 to 3 week discovery sprint: choose the first workflow, collect sample data, map the systems involved and agree how success will be measured. It ends with a prototype on your data, the evaluation set and a costed plan.

Then the work is sequential:

  • Connect: link the agent to ERP, inbox, WhatsApp Cloud API or document stores with scoped credentials and read-only access first.
  • Build and test: run it against real cases, tune retrieval and prompts, and switch guardrails on before anyone outside the project team uses it.
  • Deploy and train: go live on private cloud, a cloud region you choose or edge, with approval steps on, then train users in Arabic and English.
  • Expand: widen permissions with care or add the next workflow.

Document-heavy workflows are the usual first candidate. Our RFQ automation build reads PDF and Excel tenders into editable quotes with a reviewer confirming every line, and it follows the same path from pilot to production. If you are still choosing which workflow to promote, the AI consulting team scores candidates on value, data readiness, integration effort and risk first.

About the authorJoseph StalinChief Technology Officer

Joseph leads engineering at ShichifukuTekx, building AI agents, quality and operations platforms, and the cloud infrastructure they run on.

Back to all insights

Related questions

How long does it take to move an agent from pilot to production?

Discovery is 2 to 3 weeks and ends with a working prototype, an evaluation set and a costed plan. Build time after that depends on how many systems the agent touches and how many approval steps you need, so we scope it per workflow rather than quoting a fixed duration in advance.

Do we need clean data before an agent goes live?

No. Most clients hold data across ERP systems, spreadsheets, emails and documents. The discovery sprint includes a data readiness review showing what is usable now, what needs cleaning and what is missing, and a document workflow can often start on the data you already have.

What breaks most often when an agent reaches real volume?

Messy input and failed writes. Scanned pages, mixed units and duplicate line items cause extraction errors, and a failed ERP call can leave a record half-updated. That is why reviewers confirm extracted lines and failed integration messages sit in an error queue that can be inspected and replayed.

Can the agent run on our own infrastructure?

Yes. We deploy on private cloud, in a cloud region you choose, on-premise or on edge hardware, and where data cannot leave your network we use self-hosted open models through Ollama. Hivelock can also run self-hosted with the same features as the hosted version. If you have an agent stuck in a pilot, book a 30-minute call with an engineer at https://cal.com/shichifukutekx/30min and we will scope production against your own data.

Tell us what you are building

Book a 30-minute call or send us a project brief.

Emailsales [at] shichifukutekx.aePhone and WhatsApp+971 58 505 7791
OfficeOffice C1-1F-SF10837, Ajman Free Zone C1 Building, Ajman
HoursSunday to Thursday, 9:00 to 18:00 GST
Message us on WhatsApp