Getting Started with AI Agents (Part 2): Autonomy, Security, and Pitfalls

in ours first installmentwe have outlined key strategies for using AI agents to enhance enterprise performance. I explained how, unlike standalone AI models, agents iteratively refine tasks, using context and tools to enhance results, resembling code generation. I also discussed how multi-agent systems foster communication between departments, creating a unified user experience and increasing productivity, resiliency, and faster updates.

Success in building these systems depends on mapping roles and workflows, in addition to establishing safeguards resembling human oversight and error control to make sure protected operation. Let’s dive into these critical elements.

Security and autonomy

Agents imply autonomy, so various safeguards should be built into the agent in a multi-agent system to cut back errors, waste, legal risk, or damage when agents operate autonomously. Applying all of those safeguards to all agents could also be overkill and a resource challenge, but I strongly recommend that you concentrate on each agent in your system and make an informed decision about which of those safeguards they may need. An agent shouldn’t be allowed to operate autonomously if any of those conditions are met.

Clearly defined conditions for human intervention

Triggering any of a set of predefined rules determines the conditions under which a human must confirm certain agent behavior. These rules ought to be defined on a case-by-case basis and could be declared on the command line of the agent system – or in more critical cases, they could be enforced using deterministic code external to the agent. One such rule for a purchasing agent can be: “Every purchase should first be verified and confirmed by a human. Call ‘check_with_human’ and don’t proceed until it returns a value.”

Security agents

A security agent could be paired with an agent whose role is to ascertain for dangerous, unethical or non-compliant behavior. The agent may have to review all or some of his or her behavior towards the security agent and take no further motion unless the security agent consents.

Uncertainty

Our lab recently published a paper on a technique that may provide a measure of the uncertainty of what a large language model (LLM) generates. Given the LLM’s tendency to confabulate (commonly often known as hallucinations), preferring a particular final result can make the agent much more reliable. There are also costs to bear here. Assessing uncertainty requires generating multiple results for the same request so that we are able to rank them based on confidence and select the behavior with the least uncertainty. This can decelerate the system and increase costs, so this ought to be considered for more critical agents in the system.

Disconnect button

It may occur that we’d like to stop all autonomous agent-based processes. This could also be resulting from a need for consistency or a behavior detected in the system that should be stopped until it is determined what is unsuitable and fix it. For more critical workflows and processes, it is vital that this disabling does not cause all processes to stop or go fully manual, so it is really helpful to offer a deterministic failover mode.

Work orders generated by the agent

Not all agents in an agent network have to be fully integrated with applications and APIs. This may take some time and requires several iterations to get correct results. I like to recommend adding a generic alternative tool to agents (normally leaf nodes in the network) that will simply issue a report or work order containing suggested actions to be performed manually on the agent’s behalf. This is a great solution to efficiently launch and operationalize your agent network.

Testing

With LLM-based agents, we gain solidity at the expense of consistency. Furthermore, given the opaque nature of LLM, there are black box nodes in the workflow. This signifies that we’d like a different regime for testing agent systems than that used in traditional software. The excellent news, nevertheless, is that we are accustomed to testing such systems because we have been operating human-driven organizations and workflows since the dawn of industrialization.

Although the examples I showed above have a single entry point, all agents in a multi-agent system have LLM as their brain, so they’ll act as an entry point for the system. We should take a divide and conquer approach and first test subsets of the system, starting from different nodes in the hierarchy.

We also can apply generative AI to develop test cases that we are able to run on the network to investigate its behavior and trick it into revealing its vulnerabilities.

Finally, I’m a huge supporter of sandboxing. Such systems should first be launched on a smaller scale in a controlled and secure environment and then regularly deployed to exchange existing workflows.

Fine-tuning

A standard misconception about gene AI is that it gets higher the more you employ it. This is after all unsuitable. LLMs are pre-trained. Having said that, they could be fine-tuned to influence their behavior in various ways. Once we have developed a multi-agent system, we are able to improve its behavior by taking logs from each agent and marking our preferences to build a tuning corpus.

Traps

Collapse can occur in multi-agent systems, meaning that sometimes a query may never be accomplished and agents are continually talking to each other. This requires some type of timeout mechanism. For example, we are able to check the communication history for the same query, and if it becomes too large or we detect repetitive behavior, we are able to end the flow and start over.

Another problem that may occur is what I’ll call overload: expecting too much from a single agent. The current state of the art for LLM does not allow us to provide agents long and detailed instructions and expect them to follow them at all times. Did I mention that these systems could be inconsistent?

The solution in such situations is granulation: dividing agents into many connected agents. This reduces the workload on each agent and makes agents more consistent in their behavior and less more likely to go into a tailspin. (An interesting area of ​​research conducted by our laboratory is the automation of the granulation process.)

Another common problem with the way multi-agent systems are designed is the tendency to define a coordinating agent that calls different agents to perform a task. This introduces a single point of failure that may result in a quite complex set of roles and responsibilities. In such cases, I suggest treating the workflow as a pipeline, with one agent doing some work and then passing it on to the next one.

Multi-agent systems also are likely to pass context down the chain to other agents. This can overwhelm other agents, confuse them, and is often unnecessary. I suggest allowing agents to retain their very own context and resetting the context when we know we are dealing with a latest request (form of like how sessions work on web sites).

Finally, it ought to be noted that the bar for the possibilities of LLM used as the brain of agents is set relatively high. Smaller LLMs may require a lot of quick engineering or tuning to fulfill requests. The excellent news is that there are already some industrial and open source agents, albeit relatively large ones, that are breaking the bar.

This signifies that cost and speed should be vital considerations when building a large-scale multi-agent system. It is also to be expected that these systems, while faster than humans, is not going to be as fast as the software systems we are accustomed to.