Tracing history: The generative AI revolution in SIEM

This initial iteration of security log collection was defined as SEM (security event management) or SIM (security information management). It collected a combination of log data, or the digital records of system activity, along with event data. This was a game changer for analysts, as they now had a system within their control that contained the data needed to solve the digital crime. Basically, security teams now had their own data silo. This product revolution was mostly driven by a need to collect data in case something happened, like maintaining a forensic log and being able to demonstrate to auditors and investigators that indeed these logs were being collected. This compliance use case drove the adoption of central security event collection.

There were challenges with this new type of product. The SOC now needed security engineers to manage large amounts of data. They also needed the budget to collect and store this information, as they were copying data from numerous other systems into a monolithic, centralized system. But the benefits were clear: accelerating detection and remediation by slashing the time spent collecting and sorting data from across the business. Once notified of an attack, incident responders could get to work almost instantly.

The next advance was to apply detection logic at the centralized SIEM layer. A SIEM used to be a combination of the event data in a SEM and the information data in a SIM. The compliance and evidentiary collection power of SEM/SIMs was strong, but after almost a decade of just collecting and reviewing data, analysts realized they could do far more with centralized information. Instead of merely consolidating alerts from other systems and providing a central system of record for collected logs and events, SIEMs now enabled analysis across many data sources. Detection engineers could operate from a new perspective — spotting threats that may have been missed at a point-solution analyzing only one data source, like your anti-virus or network firewall. 

This evolution came with plenty of challenges. In addition to greater need for subject matter experts and prebuilt rules, SIEMs centrally collected alerts from numerous point solutions, each of which produced a lot of false positives on their own and exacerbated the issue. SIEM analysts had to review the collective network and desktop alerts. This resulted in an often-asked question by a SIEM analyst: “Where do I start?” paired with an entirely new set of detection alerts from the SIEM itself. Your SIEM now contains the sum of all other system alerts in the network plus the amount of alerts normally generated. Needless to say, this was incredibly overwhelming.

Machine learning (ML) promised to improve the detection of unknown threats with less required upkeep. The goal was to identify abnormal behavior, rather than depending on hard coded rules to find every threat.

Before ML, detection engineers had to analyze an attack that had already happened or one that could happen (courtesy of first party research) and write detections for that potential occurrence. For example, if there was an attack discovered that took advantage of some specific arguments sent to a Windows process, one could write a rule looking for those arguments to be invoked on execution. But the adversary could merely change the order of arguments or invoke them differently to avoid this brittle detection. And, if there were legitimate uses of those arguments, it would take days (maybe weeks) of tuning to remove these false positives from the detection logic. 

Machine learning promised to greatly reduce this challenge; specifically in two ways:

• “Unsupervised” ML-based anomaly detection: Analysts only had to decide in what area to look for unknown behaviors, such as in logins, in process executions, and in access to S3 buckets. Then the ML engine learned the NORMAL behavior for these areas and flagged what was unusual. SANS DFIR made a famous poster in 2014 that said "Know Abnormal…Find Evil."

• Trained or “supervised” ML models: Human analysts can see something, and their brains can connect the dots on how it looks somewhat similar to an attack previously observed. These experts are able to learn about how an attack took place and apply that knowledge to finding unknown attacks that follow a similar progression. Traditionally, they used this expertise in threat hunting to help find threats their security products may have missed. Now, with machine learning they were able to craft trained model detections with the ability to learn from previous attacks and find new ones that are similar in how they are attacking. Focusing on the behavior — and not only on atomic indicators like hashes, strings in files, and URLs — allows for detections with a longer shelf life and a higher rate of attack detection.

As we’ve seen, SIEMs used to be historical reports of problems rather than actual end-to-end solutions. SIEMs could alert you to an issue, but then you were on your own to clean it up. This changed with the entrance of SOAR: security orchestration, automation, and response. This new product line was created to fill a feature gap in SIEMs. They provided a place to collect and organize the steps an analyst wanted to perform for remediation as well as connectors to the rest of the ecosystem to initiate the response. In our police department analogy, SOARs were like traffic cops directing all the other systems to execute commands. They were the glue holding the discovery of the attack from the SIEM to the response actions of the other systems. 

Like UEBA, the ability to organize response plans and kick off actions from a central location has become an expectation of modern SIEMs. Now in the SIEM 2.0 lifecycle, it is expected that SIEMs can collect data at scale across the organization (.gen 0), detect new threats the point solutions may have missed, and correlate across disparate systems using both rule-based and machine learning-based technologies (SIEM 1.0), and allow for planning and execution of the response plans (2.0). In fact, a new acronym — TDIR (threat detection, investigation, and response) — was coined to capture the ability to handle the full scope of an attack.

SIEMs have become foundational in a SOC’s threat detection, triage, and investigation, despite failing to address a fundamental challenge: the massive skills shortage in cybersecurity. A March 2023 study commissioned by IBM and completed by Morning Consult found that SOC team members are “only getting to half of the alerts that they’re supposed to review within a typical workday.” That’s a 50% blind spot. Decades of incremental improvements to simplify workflows, automate routine steps, guide junior analysts, and more have helped — but not enough. With the advent of consumer-accessible generative artificial intelligence models with domain expertise in cybersecurity, this is changing quickly. 

SIEMs have traditionally been very dependent on the human behind the screen — alerting, dashboarding, and threat hunting are all human intensive operations. Even early AI efforts like AI co-pilots have depended on the ability of the analyst to use these co-pilots effectively. This revolution will happen when AI operates on behalf of the analyst, removing the requirement to “chat.” Imagine the system sifting through all the data, ignoring the irrelevant and identifying what's critical, discovering the specific attack, and crafting specific remediations — and, in turn, freeing up your experts to focus on stopping the impact to the business.

For the first time, technology is learning from senior analysts and transferring that knowledge to junior members automatically. Generative AI now helps security practitioners develop organization-specific remediation plans, prioritize threats, write and curate detections, debug issues, and tackle other routine and time-consuming tasks. Generative AI promises to automate the feedback loop back into the SOC, enabling continual improvement day after day. We can now close the OODA loop with this automated feedback and learning. 

Due to the nature of large language models (the science behind generative AI), we can finally leverage technology to reason across numerous data points, just like a human would — but with greater scale, higher speed, and broader understanding. Additionally, users can interact with large language models in natural language, rather than code or mathematics, and further reduce barriers to adoption. Never before has an analyst been able to ask questions in natural language, such as "Does my data contain any activity in any area that might represent a risk to my organization?" This is an unprecedented leap forward in the capabilities that can now be embedded in a SIEM for all members of a SOC. Generative AI has become a powerful and accurate digital SOC assistant. 

The products that harness the AI revolution in security operations workflows will deliver SIEM 3.0.

This blog post has reviewed the evolution of SIEM, from centrally collecting data to detecting threats at the organizational level and then to automation and orchestration for speeding up remediation. Now, in this third phase of SIEM technologies, we are finally tackling the massive skills shortage in cybersecurity. 

In part two of this series, we will discuss the evolution of Elastic Security from a TDIR to the world’s first and only AI-driven security analytics offering. In the meantime, you can learn more about how security professionals have reacted to the emergence of generative AI with this ebook: Generative AI for cybersecurity: An optimistic but uncertain future. Stay tuned for part two!