Scientists from Stanford University‘S Scaling Intelligence Lab introduced a recent inference framework that can assist large language models (LLMs) process potential answers more quickly.
The Archon framework uses an inference-timed architecture search (ITAS) algorithm to improve LLM performance without additional training. It is model independent, open source, and designed to be plug-and-play for models large and small.
Archon could ideally help developers design AI model systems using multiple inference time techniques to reduce the variety of models needed to determine an answer. The Scaling Intelligence Lab said techniques like Archon would help reduce costs associated with model building and inference. As LLM development turns towards larger parameters or more advanced reasoning, costs may increase, whilst firms akin to OpenAI expect greater affordability.
According to the researchers, Archon mechanically designs architectures that improve task generalization, enabling models to perform tasks beyond those for which they were initially trained.
“Our Archon platform and ITAS algorithm draw inspiration from neural architectures and the search for neural architectures, respectively,” the researchers said in their paper. paper. “Archon is built with LLM layers, where models in the same layer run in parallel, but each subsequent layer runs sequentially.”
These layers perform various inference time techniques, “either transforming the number of potential answers through generation and fusion (like linear transformations) or reducing the number of potential answers to improve quality (like nonlinearity).”
Archon outperformed GPT-4o and Claude 3.5 Sonnet by 15.1 percentage points in benchmarks akin to MT-Bench, Arena-Hard-Auto, Alpaca-2.0 Eval, MixEval, MixEval Hard, MATH, and CodeContests. When Archon faced open source LLM solutions, it outperformed by 11.2 percentage points.
Archon Components
The ITAS algorithm consists of several LLM components and can perform time-based inference techniques.
The first component is the Generator, which creates possible answers for the model. The second component, Guser, will take these responses and mix them into one. An example could be when a query is asked to the model wanting to know the capital of France, the fixer will use the generated answers “the capital of France is Paris”, France is in Europe” and replace them with “the capital of France” France, a country in Europe, is Paris.
Archon then moves to the Ranking component, which ranks the best answers. The Critic component evaluates ranked responses to determine whether or not they are good or bad. The validator checks the logic and correctness before moving on to the unit test generator and evaluator, which perform small tests to confirm that the answer works and check the test results.
By building Archon this manner, the researchers say, the platform improves the quality of LLM responses faster and without additional tuning.
Archon Limitations
So far, the Archon framework works best with LLMs with parameters of 70B or greater, akin to Meta’s Code Llama 70B, which makes it difficult to goal most LLMs at this time. The researchers found that almost all of the challenges stem from the smaller model’s limited ability to execute instructions due to its smaller context windows.
“When we use the Archon architecture only with the open source 7B models, we experience a noticeable 16% performance penalty,” the article stated.
Smaller models using the Archon framework lagged behind single-turn models by 15.7%.
The Stanford lab also said Archon is “not ideal for tasks that prefer the latency of a single LLM connection,” akin to chatbots. The framework makes multiple calls to LLM due to the various operations it performs, so single Q&A queries won’t profit from its capabilities. Archon may perform higher for tasks that require complex instructions, akin to solving equations, programming, and even complex customer support issues.
Despite the limitations, the researchers behind Archon expressed hope that it could accelerate the development of efficient models without requiring more inference and training capital.