Taking agents to production: our agentic AI approach

Without a crystal ball, it’s hard to make any firm predictions on who the eventual winners will be, but I’ve had the pleasure of spending my last six months developing and deploying agents into production with a couple of innovative organisations; here are some learnings on the challenges we faced taking agents from theory to production, and, ultimately, what worked.

Agentic frameworks: what underpins the ecosystem?

To be honest, I don’t think the framework itself makes a huge difference. An agent framework acts as a scaffold or a set of building blocks to quickly design agents, and I have seen successful agentic deployments underpinned by LangGraph, Strands, ADK, Crew AI and Agno. Personally, I’m more likely to subscribe to the 12 Factor approach, as it gives you far more flexibility for your specific agents’ needs, but I appreciate that most organisations aren’t in the business of coding bespoke agents, so a framework does make things easier. 

If you do decide to go down the AI framework route, I would suggest starting with your cloud vendor of choice (Strands for AWS, ADK for Google, etc.) as the hyperscalers are heavily investing in their own frameworks and have developed easy integrations with their other internal services and databases that you may not get as easily from third-party frameworks. 

Another thing to think about is how you will log your agents’ actions, undertake evaluations and provide relevant guardrails. The more common the framework, the more likely it will be natively supported by the leading observability and evaluation platforms (e.g. Langfuse, Arize Phoenix, MLFlow etc.) which makes this process much easier. 

To code or not to code

I think both of these solutions have their place, but it’s important to have a clear goal in mind when deciding how to build agents. If you are building a small number of agents to automate a complex process (or set of processes) and this involves multiple integrations with tools, databases, APIs etc, then building agents in code and deploying them into an Agent Runtime like Google’s Agent Engine, AWS Agent Core or Azure Foundry is a good move. Alternatively, if you are envisaging building a whole suite of agents that are all focusing on smaller, more discrete tasks involving one or two integrations, then using a no- or low-code interface like Bedrock Agent Flow, Google’s AgentSpace, or open source options like N8N could be a more scalable and easier way to build. 

Both options can work, but building agents through code and deploying them into production is very hard. If you opt to go down that route, it is important you have the right technical skillset in your organisation to navigate these challenges as they arise. Given the integration layers involved in agentic systems, you ideally want a multi-skilled technology team spanning Machine Learning, Software Engineering, Cloud Infrastructure and DevOps. This means you have dedicated skills to help build the agents, develop API integrations for tool calls, deploy into cloud environments and then actually monitor these agents in production.

Make your team agentic from the beginning

It is critical to involve the business in these projects from the beginning. All of the successful production deployments I’ve been across in the past few months had fully cross-functional teams, from the very beginning, including senior leadership, product, design, legal and compliance.

Are you ready for true autonomy?

One of the biggest realisations we encountered when building various parts of these agentic systems is that we didn’t actually need to use an agent at all. We’d start off with an open-ended agent that had a goal and a couple of tools. As we progressed with development, we constrained the scope, realised we wanted it to do certain things in a certain order, and added compliance and policy guardrails. At a certain point, we realised that what we were refining was more akin to a workflow using LLMs and tools, rather than a truly agentic system. 

By definition, agentic systems are ‘autonomous’ and work best for problems that have an open-ended decision space, but have an easily evaluatable outcome. Use cases like AI coding or claims processing are good agentic use cases, as they have multiple pathways to solve, but output is easily measurable (i.e. does the code run, was the claim approved?) whereas a simple onboarding flow that follows a logical step-by-step process is more suited to a deterministic workflow or LLM workflow instead. 

Navigating the token economy

Another challenge we faced when building these agentic systems was navigating the nebulous agentic costing model. Agents, like all LLM-backed systems, are almost always charged based on usage. More specifically, costs are incurred based on the token input to, and output from, the LLM used. 

What’s a token? Kind of like a word, but not really – hope that helps with the confusion. 

This costing model often means that the ‘value’ the agent brings by completing a task does not align with the ‘cost’ in tokens. For example, you might have a process that was being outsourced to a third-party tech solution for $2.00 a process. If, on average, the token cost for an agentic system to automate that process is $0.20, then there would be a clear ROI. 

However, a different scenario emerges if you want to build a smart assistant that helps your users find information and complete actions within your internet banking application. Forecasting this cost is far more complex, because a single power user could incur hundreds of dollars in costs through heavy use, whereas other users may not use it at all. While the power user clearly finds the tool valuable, would they find it valuable at $300 a week?

To try and address this issue, it is a very good idea to begin your agentic journey with a cost model and target ROI in mind. Furthermore, the second situation above may not be financially viable from an ROI perspective, but could still be valuable to the organisation as a loss leader, improving customer experience and acting as a differentiator in the market. All of these are valid considerations; it’s important to fully consider them. 

In terms of other practical advice, I would recommend working backwards from your cost model. Don’t choose the beefiest LLM available if you know you only have a few cents per process to spend. Optimise your prompts, tool calls, database calls, and memory usage, as this will reduce the token input and output from your agentic systems, thereby further reducing costs.

Complexity vs performance

Agent optimisation is also an important consideration when it comes to balancing the complexity of the agent against the agent’s performance. In my experience, it is more effective to build several smaller agents, each focused on a specific skill, task, or area of a process, rather than trying to build a super agent that attempts to solve every task but ends up struggling with any. There were times when we had to break individual agents into smaller subagents because they became too complex to keep them on task. 

Be very clear upfront as to what the agent is expected to do, and only give it access to the tools, data and memory that it absolutely needs to do its job. This will help prevent the agent from being confused about which tool to call or what data to leverage, making them faster and more effective at completing your desired task. 

As with costing, it is important to consider the latency requirements of the agentic system prior to building, as this will heavily impact the model choice, task complexity and agent architecture required to achieve the desired latency goal.