M-21-31 logging compliance: Where are we now?

Several federal agencies have been using Elastic’s Search AI Platform to meet M-21-31 requirements — taking a unified approach that includes both logging and threat response. Based on these agencies’ experience using Elastic for M-21-31, we recommend the following ways to overcome common M-21-31 challenges.

As federal agencies prioritize cost savings and interoperability, there is increasing scrutiny toward technology investments and budgets. Since M-21-31 was introduced in 2021, Elastic continues to offer even more opportunity for agencies to save on costs and store data affordably. 

• Affordable data tiering model: Elastic's data tiering approach optimizes data management by categorizing data into storage tiers based on access frequency and cost. For example, agencies can store longer term or historical data in Elastic’s frozen tier. Ideal for long-term retention, the frozen tier enables data to be stored in object storage for two years or longer. 

• Searchable snapshots: The unique searchable snapshots capability enables direct searches for data without any rehydration, maintaining fast search performance. (Elastic's cold and frozen tiers perform with search speeds that are comparable to our competitors' hot tiers.)

• Elasticsearch logsdb index mode: Elasticsearch logsdb index mode is designed to significantly reduce data storage costs by efficiently storing and searching essential log data. Logsdb index mode can cut data storage costs by up to 65%, making it an ideal choice for federal agencies aiming to optimize their data management budgets while complying with M-21-31.

Federal agencies are leveraging the AI and machine learning (ML) capabilities built into the Elastic Search AI Platform to automate time-consuming tasks and uncover data insights in real time. Instead of hiring more employees or re-skilling existing teams, agencies are benefiting from Elastic’s democratized approach to insights and the accessible capabilities built into the platform. A few of the functionalities helping agencies address the skills gap include: 

• Consolidated view into data: Align teams and roles around common datasets, providing a unified view of infrastructure performance and enriched by threat intelligence. This consolidated access makes it easier and faster to consume and act on data, no matter where it’s located.
• Drag-and-drop visualizations: Analyze logging and cybersecurity data through Elasticsearch’s intuitive, visual drag-and-drop dashboards. These dashboards surface insights generated from Elastic’s ML and AI capabilities, allowing everyone access to this information in real time — as opposed to having to wait on a data scientist with specialized knowledge or access.
• AI capabilities: Leverage the power of Elastic’s Search AI Platform to streamline tasks, reduce manual data correlation, and triage security alerts. The Elastic AI Assistant integrates generative AI to simplify tasks and help users find context and information for understanding anomalies and threats faster, speeding problem resolution. And Elastic’s Attack Discovery feature can automatically cut through hundreds of noisy alerts in order to surface the ones that matter most to your agency. As a result, your team can save time by quickly understanding the presented attacks, take immediate follow-up action, and more.

One of the roadblocks in logging compliance is not having access to all logging data. Without streamlined visibility into all data types and sources, the ability to accurately pinpoint threats and patterns is significantly limited. Many organizations are challenged with the high costs involved in managing and storing large quantities of disparate logging data. Elastic’s approach simplifies data ingest and analysis, while our resource-based pricing gives teams the flexibility to pay for what they need.

• Streamlined data ingest: Ingesting different types of data from different sources typically requires multiple tools and processes (and high costs). Using Elastic Agent to ingest all your logs, metrics, and traces can eliminate dependency on external plugins and integrations that may require you to give up control of sensitive data.

• Unified schema: To organize and make sense of all types of ingested data, Elastic uses an open source, community-driven schema known as the Elastic Common Schema, or ECS. This common data structure unifies all modes of analysis available in Elastic, including search, drill-down and pivoting, data visualization, ML-based anomaly detection, detection rules, and alerting. ECS is also part of OpenTelemetry, offering agencies yet more opportunity for building efficiencies and interoperability.

M-21-31 called for agencies to share logging data with one another, “as needed and appropriate, to accelerate incident response efforts.” Traditionally, sharing data outside an agency introduced significant risk for already-sensitive data, as well as potential costs and time required to copy data or move it to a central source. 

Using Elasticsearch, however, agencies can securely share data across agencies, teams, and projects. In fact, federal agencies are probably already familiar with the cyber intelligence data provided by CISA; Elasticsearch powers CISA’s CDM Dashboard, giving CISA centralized visibility into 100+ agencies’ cybersecurity data when needed. CISA, and other federal agencies, have been relying on Elasticsearch for its:

• Distributed approach: With Elastic cross-cluster search and cross-cluster replication capabilities, agencies can securely share their data outside their agency without moving it. In addition to reducing the risk, time, and costs involved with moving data, this approach enables each agency to retain control of their data in its original secure location. 
• Data privacy controls: Working hand in hand with cross-cluster search and replication, Elastic’s role and attribute-based access control (RBAC/ABAC) security lets you decide who at your agency can access what data — down to the document level. These security permissions are applied locally, where the data resides. This allows you to create secure dynamic data access policies for certain classification levels and functional areas.

Learn more about how Elastic can provide integrated, cost-effective support for M-21-31 compliance, from log storage, management, and cybersecurity capabilities within our unified AI-powered platform:

• White paper: M-21-31 industry brief

• Blog: Optimizing long-term storage costs for government compliance 

• Blog: It’s time for government to move beyond monitoring and into observability

• Contact the Elastic federal team

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.