SambaNova Systems Wins Coolest Technology Award at VentureBeat Transform 2024

Palo Alto-based AI startup SambaNova Systems was awarded the “Coolest Technology” award at VentureBeat Transform 2024 in San Francisco. The company, founded in 2017, has raised $1.1 billion up to now and is currently valued at $5 billion.

We spoke with Kunle Olukotun, co-founder and chief technology officer of SambaNova, about the company’s technology and recent developments. Olukotun is also a professor of electrical engineering and computer science at Cadence Design Stanford University“I’m best known as one of the pioneers of this idea called multicore,” he said. “We designed the multiprocessor for the data center.” His previous enterprise, Afara Web Systems, commercialized multicore technology before being acquired by Sun Microsystems in 2002.

SambaNova, founded in 2017, emerged from a convergence of trends. Olukotun explained, “We saw an opportunity to mix insights from software [ML] models with efficient hardware implementations.” The genesis of the company was rooted in the realization that machine learning was becoming a recent paradigm in computing.

The company’s approach differs from traditional GPU makers. “Unlike NVIDIA GPUs, which were repurposed graphics and gaming chips, this one was designed specifically for AI computing,” Olukotun said. He added, “It focuses on optimizing data movement, which is a critical bottleneck for high-performance inference.”

The company’s latest chip, the SN40L, showcases their innovation. Unlike traditional AI-focused GPUs, the SambaNova chip is built from the ground up for AI computing. Olukotun explained their unique “reconfigurable dataflow” architecture: “Our innovation was to say, look. If all these models are expressed this way, why don’t we invent hardware where this is the native execution mode?”

This approach allows SambaNova to “streamline data movement and provide the lowest inference latency, the highest number of concurrent LLMs at the same time, and the lowest switch time between different LLMs,” based on Olukotun. The chip design focuses on optimizing data movement, which Olukotun identified as “a critical bottleneck for high-performance inference performance.”

Attracting customers

SambaNova systems are having fun with increasing interest among prestigious clients from various sectors.

In the financial services industry OTP Groupleading banking group in Central and Eastern Europe, has partnered with SambaNova to build Europe’s fastest AI supercomputer. This multi-tenant AI system, powered by SambaNova’s Dataflow-as-a-Service GPT, goals to remodel OTP services for 17 million customers in 11 countries.

In the public sector and in the field of scientific research SambaNova has made significant progress. The company has expanded its cooperation with Lawrence Livermore National Laboratory (LLNL), integrating its spatial dataflow accelerator with the LLNL Computer Center to reinforce cognitive simulation programs.

Similarly, Los Alamos National Laboratory has decided to scale up its current SambaNova deployment to expand its generative AI and LLM capabilities. This partnership is intended to support the lab’s mission goals in national security, science, energy, and environmental stewardship.

In the energy sector Saudi Aramcothe world’s largest energy company, implemented SambaNova hardware to power its internal LLM system called Metabrain, whose variety of users grew from several hundred to 1000’s in a matter of months.

How is SambaNova doing?

In the competitive AI chip market, SambaNova Systems is facing challenges from industry giants, cloud service providers, and several other well-known startups.

Nvidia currently dominates the market with its H100 GPU, backed by a robust ecosystem. Cloud giants like Google, Amazon, and Microsoft are developing their very own AI chips, optimized for their specific environments. Among startups, Cerebras Systems’ WSE-3 and Groq’s Tensor Streaming Processor offer formidable competition.

SambaNova’s approach focuses on optimizing data movement, which Olukotun identifies as “a critical bottleneck in high-performance inference performance.” The solution also stands out for its focus on supporting multiple LLMs concurrently and efficiently switching between them. “You can’t get that ability to have high speed, high number of models, the ability to switch between models instantly on any other platform,” Olukotun emphasized.

SambaNova Systems recently set a recent record for LLM inference performance, based on an independent benchmark conducted by Artificial Analysis. Their Samba-1 Turbo achieved 1084 output tokens per second on the Llama 3 Instruct (8B) Meta model, greater than eight times faster than the average output speed among vendors.

This performance was achieved using just 16 chips, while maintaining full precision and the ability to concurrently host as much as 1,000 Llama3 checkpoints on a single 16-socket SN40L node. This breakthrough, attributed to SambaNova’s Dataflow architecture, offers significant speed, efficiency, and cost-effectiveness advantages over GPU-based competitors. The company claims this performance translates to a 10x lower total cost of ownership, positioning SambaNova as a leader in fast, high-performance AI inference solutions for the enterprise.

Battle for programmers

While SambaNova’s technological innovations are impressive, the company’s success will depend not only on the performance of its chips but also on its ability to build a thriving developer ecosystem.

SambaNova is taking a variety of steps to draw developers. They recently introduced a recent Fast API that gives access to pre-trained models and chip capabilities, including “full precision, 8 billion, 70 billion Llama 3 models” running on the SN40L. The move is intended to lower the barrier to entry for developers and startups trying to leverage SambaNova’s technology.

In addition, SambaNova announced SambaVerse, a unique playground and API for developers. SambaVerse allows developers to check and compare a whole lot of accessible open-source LLMs from a single endpoint. This tool allows developers to directly compare model responses for any application, query multiple models concurrently, and view results in real time.

Olukotun believes the next generation of AI applications will focus on agent-based AI, or AI systems that may act autonomously to realize goals and often involve decision-making and interaction with the environment.

The SambaNova architecture can enable more dynamic and responsive agent AI systems, potentially enabling real-time adaptation to changing tasks or environments. This capability is vital because agent AI systems often need to make use of different models for different tasks or decision-making processes.

For example, agent-based AI might use one model to know natural language, one other to reason, and yet one other to generate responses or actions. Smaller models may be used for fast, specialized tasks, while larger models might handle more complex reasoning or generation tasks.

With hardware capabilities and development tools like Fast API and SambaVerse, developers using the SambaNova platform can gain a significant advantage in building advanced agent-based AI systems.

Future

SambaNova’s success will depend on its ability to deliver higher performance and efficiency for AI agent applications while developing a wealthy developer ecosystem. The introduction of Fast API and SambaVerse demonstrates SambaNova’s commitment to providing developers with the tools they should build and experiment with advanced AI systems.

SambaNova’s architecture, optimized for launching and quickly switching between multiple LLMs, is well-suited to the complex, dynamic processing needs of agent AI systems. As these applications change into more sophisticated, requiring seamless integration of various specialized models, SambaNova’s approach may change into increasingly useful.

Congratulations to SambaNova Systems for winning the “Coolest Technology” award at VentureBeat Transform 2024! This recognition highlights the company’s modern approach to AI computing and its potential to remodel the enterprise AI landscape.

Those interested in learning more about SambaNova technology, including the SN40L, Samba-1, and development tools, can visit their website at sambanova.aiDevelopers can even explore SambaVerse at sambaverse.sambanova.ai to see for themselves how the SambaNova platform can streamline AI development processes.