Simbian Announces Industry’s First Benchmark to Comprehensively Measure LLM Performance in Security Operations Centers

MOUNTAIN VIEW, Calif.--(BUSINESS WIRE)--Jun 12, 2025--

Simbian®, on a mission to solve security for businesses using AI, today announced the “ AI SOC LLM Leaderboard ” – the industry’s most comprehensive benchmark to measure LLM performance in Security Operations Centers (SOCs). The new benchmark compares LLMs across a diverse range of attacks and SOC tools in a realistic IT environment over all phases of alert investigation, from alert ingestion to disposition and reporting. It includes a public leaderboard to help professionals decide the best LLM for their SOC needs.

This press release features multimedia. View the full release here: https://www.businesswire.com/news/home/20250612762313/en/

“SOC analysts and vendors building tools for the SOC are rapidly embracing LLMs to scale their operations, increase accuracy, and reduce costs,” said Ambuj Kumar, Simbian CEO and Co-Founder. “Our industry-first benchmark enables SOC teams and vendors to pick the best LLM for this purpose. This benchmark is made possible by Simbian’s AI SOC Agent, a proven solution leading the industry in end-to-end alert investigation leveraging LLMs.”

Existing benchmarks compare LLMs over broad criteria such as language understanding, math, and reasoning. Some benchmarks exist for broad security tasks or very basic SOC tasks like alert summarization. But prior to today’s announcement, no benchmark existed to comprehensively measure LLMs on the primary role of SOCs, which is to investigate alerts end-to-end. This task involves diverse skills, including the ability to:

Simbian’s AI SOC LLM Leaderboard is the industry’s first and only benchmark that measures LLMs on autonomous end-to-end investigation of alerts, utilizing the above skills. To make the benchmark applicable across a range of SOC environments, it leverages 100 diverse full-kill chain scenarios that test all layers of defense. It is also the industry’s first benchmark to measure investigation performance in a lab environment mimicking an enterprise, with investigations autonomously retrieving data from live tools across the environment.

This first LLM benchmark tested today’s top-tier LLM models from Anthropic, OpenAI, Google, and DeepSeek. All tested models were able to complete over half (61%-67%) of the tasks involved in alert investigation, as long as there was a solid framework to break down an investigation into clearly defined tasks for LLMs. For this benchmark, that framework was provided by Simbian’s AI SOC Agent ( https://simbian.ai/products/ai-soc-agent ). See Simbian’s blog published today for details of the benchmark methodology at https://simbian.ai/blog/the-first-ai-soc-llm-benchmark.

The AI SOC LLM Leaderboard reveals that LLMs are more capable than commonly believed for autonomous alert investigation. Marginal difference was observed between standard LLMs and thinking LLMs for alert investigation. The results showed that the best LLM for cybersecurity is a generalist (like Sonnet 3.5) that knows how to code as well as how to perform logical reasoning, rather than a specialist that excels at code (Sonnet 4.0) or at logical reasoning (Opus 4). Finally, the benchmark highlighted that specialization such as SOC-specific training or a mix of LLMs yields higher performance than any single LLM.

Alert fatigue is common across SOCs and it is only getting worse with AI-powered attacks, requiring SOC teams to scale their capacity rapidly. AI offers a solution, and this benchmark guides the industry on the best LLM for the SOC. Simbian will update the measurement results periodically. Follow the AI SOC LLM Leaderboard page at https://simbian.ai/best-ai-for-cybersecurity.

The AI SOC LLM Leaderboard measures LLMs using Simbian’s AI SOC Agent, a proven framework for leveraging AI within the SOC. The AI SOC Agent is deployed at some of the largest SOCs in the world. Additionally, in a recent AI SOC Championship, the AI SOC Agent performed better than 95% of more than 100 analysts worldwide in correctly investigating alerts with supporting evidence.

About Simbian

Simbian is on a mission to solve security for businesses using AI. Simbian offers AI Agents that work like virtual employees and autonomously complete a variety of security tasks with increased precision and efficiency. The company is venture backed and headquartered in Mountain View, Calif. For more information, visit www.simbian.ai.

Simbian is a registered trademark of Simbian.

Simbian’s AI SOC Agent measured LLM performance for autonomous alert investigation, including tasks of diverse skills. All top-tier LLMs completed over 60% of the tasks but left a gap for improvement. This benchmark validates that using a mix of LLMs boosts performance, helps identify the best combination LLMs, and guides further training for optimal SOC performance.

BIRMINGHAM, Ala. (AP) — Former U.S. Sen. Doug Jones, the last Democrat to hold statewide office in Alabama, kicked off his campaign for governor Friday, saying voters deserve a choice and a leader who will put aside divisions to address the state's pressing needs.

“With your help we can finish what we began. We can build the Alabama we’ve always deserved,” Jones told a packed crowd at a Birmingham campaign rally featuring musician Jason Isbell.

He said the state has urgent economic, health care and educational issues that are not being addressed by those in public office.

The campaign kickoff came on the eighth anniversary of Jones' stunning 2017 win over Republican Roy Moore, and Jones said Alabama proved back then that it can defy “simplified labels of red and blue.”

“You stood up and you said something simple but powerful. We can do better,” Jones said. “You said with your votes that our values, Alabama values, are more important than any political party, any personality, any prepackaged ideology.”

His entry into the race sets up a possible rematch with Republican Sen. Tommy Tuberville, who defeated Jones by 20 points in 2020 and is also now running for governor. Both will have party primaries in May before the November election.

Before running for office, Jones, a lawyer and former U.S. attorney, was best known for prosecuting two Ku Klux Klansmen responsible for Birmingham’s infamous 1963 church bombing.

In an interview with The Associated Press, Jones said families are having a hard time with things like health care, energy bills and simply making ends meet.

“People are struggling,” he said. “They are hurting.”

Jones used part of his speech to describe his agenda if elected governor. He said it is time for Alabama to join most states in establishing a state lottery and expanding Medicaid. Expanding Medicaid, he said, will protect rural hospitals from closure and provide health care coverage to working families and others who need it.

He criticized Tuberville's opposition to extending Affordable Care Act subsidies. Jones said many Alabama families depend on those subsides to buy health insurance "to keep their families healthy."

Alabama has not elected a Democratic governor since Don Siegelman in 1998. In 2020, Tuberville held Jones to about 40% of the vote, which has been the ceiling for Alabama Democrats in recent statewide races.

Retired political science professor Jess Brown said Jones lost in 2020 despite being a well-funded incumbent, and that's a sign that he faces an uphill battle in 2026.

“Based on what I know today, at this juncture of the campaign, I would say that Doug Jones, who’s a very talented and bright man, is politically the walking dead,” Brown said.

Jones acknowledged being the underdog and said his decision to run stemmed in part from a desire for Tuberville not to coast into office unchallenged.

Jones pointed to recent Democratic victories in Georgia, Mississippi and other locations as cause for optimism.

Tuberville, who previously headed up the football program at Auburn University, had “no record except as a football coach” when he first ran, Jones said. And “now there are five years of being a United States senator. There are five years of embarrassing the state.”

Jones continued to question Tuberville’s residency, saying he “doesn’t even live in Alabama, and if he does, then prove me wrong.” Tuberville has a beach house in Walton County, Florida, but has repeatedly said Auburn is his home.

Tuberville's campaign did not immediately respond to a request for comment but has previously noted that he defeated Jones handily in 2020. Tuberville spent part of Friday with Defense Secretary Pete Hegseth in Huntsville to mark the official relocation of U.S. Space Command from Colorado to Alabama.

Jones' 2017 victory renewed the hopes, at least temporarily, of Democratic voters in the Deep South state. Those gathered to hear him Friday cheered his return to the political stage.

“I’m just glad that there’s somebody sensible getting in the race,” Angela Hornbuckle said. “He proved that he could do it as a senator.”

Former Sen. Doug Jones, D-Ala., and gubernatorial candidate waits to speak during an event Friday, Dec. 12, 2025, in Birmingham, Ala. (AP Photo/Brynn Anderson)

Former Sen. Doug Jones, D-Ala., and gubernatorial candidate speaks during an event Friday, Dec. 12, 2025, in Birmingham, Ala. (AP Photo/Brynn Anderson)

Former Sen. Doug Jones, D-Ala., and gubernatorial candidate speaks during an event Friday, Dec. 12, 2025, in Birmingham, Ala. (AP Photo/Brynn Anderson)

Gubernatorial candidate former Sen. Doug Jones, D-Ala., speaks during an event Friday, Dec. 12, 2025, in Birmingham, Ala. (AP Photo/Brynn Anderson)