Integrating the Elastic Stack with ArcSight SIEM - Part 1

Editor's Note: Like what you read? There's more where that came from. Part 2 continues the story with how to proactively monitor security data in Elasticsearch using X-Pack. Part 3 walks you through how to scale the architecture. Part 4 and Part 5 provide examples of setting up alerts for common security threats using the alerting features in X-Pack. Part 6 extends the alerting story with automated anomaly detection using machine learning.

In this blog series we will provide an overview of how to extend and complement the capabilities of your existing SIEM to create an effective security analytics solution for your organization. For the purposes of example, we will demonstrate the use of an X-Pack enabled Elastic Stack with one of the SIEM solutions...ArcSight.

The following demonstrates an example of Elasticsearch with the ArcSight SIEM. The existing ArcSight Smart Connector can be used to send data to Elasticsearch, with multiple possible approaches to configuration. The simplest solution is to add a CEF syslog destination to the ArcSight Smart Connector allowing it to send data to the Logstash. In Logstash we will use  Logstash ArcSight module to setup this integration. The module includes a Logstash configuration for ingesting and enriching CEF-formatted data from ArcSight Smart Connectors, while bundling a set of Kibana dashboards to view events from common sources.

Setup the Elastic Stack

1. Download and install Elasticsearch. In the Elasticsearch install directory, install X-Pack and start Elasticsearch:
   
   

   bin/elasticsearch-plugin install x-pack

   
   

   bin/elasticsearch

2. Download and install Kibana. In the Kibana install directory, install X-Pack and start Kibana:
   
   

   bin/kibana-plugin install x-pack

   
   

   bin/kibana

3. Download and install Logstash. In the Logstash install directory, install X-Pack and
   
   

   bin/logstash-plugin install x-pack

4. Start the Logstash ArcSight module by running the following command in the Logstash install directory:
   
   

   bin/logstash --modules arcsight --setup -M 
   "arcsight.var.inputs=smartconnector" -M
   "arcsight.var.input.smartconnector.port=5000" -M
   "arcsight.var.elasticsearch.username=elastic" -M
   "arcsight.var.elasticsearch.password=changeme" -M
   "arcsight.var.kibana.username=elastic" -M
   "arcsight.var.kibana.password=changeme"
   

   
   
   The `--modules arcsight` option spins up an ArcSight CEF-aware Logstash pipeline for ingestion. The `--setup` option creates an "arcsight-*" index pattern in Elasticsearch and imports Kibana dashboards and visualizations. On subsequent module runs or when scaling out the Logstash deployment, the `--setup` option should be omitted to avoid overwriting the existing Kibana dashboards.
   
   For a full list of ArcSight module settings, see the documentation on "ArcSight Module Configuration Options".

Setup the Elastic Stack with Docker

For a quick setup you can download an example docker-compose.yml definition to help you to install all the elastic stack with x-plugin ( step 1 to 5 ), then issue:

$ docker-compose up

But First, ensure that:

1. You have Docker Engine installed.
2. Your host meets the prerequisites.
3. If you are on Linux, that docker-compose is installed.

Configure ArcSight Smart Connectors

1. Configure ArcSight connectors to send data to Logstash
2. Run the command ..<installdir>\current\bin\arcsight agentsetup
3. Choose yes to start the ‘wizardmode’
4. Choose ‘I want to add/remove/modify ArcSight Manager destinations’
5. Choose ‘add new destination’
6. Choose ‘CEF syslog’
7. Add the information of the logstash host and port 5000 you prepared and choose the TCP protocol.

Explore your data with Kibana

1. Point your web browser at http://localhost:5601/ to open Kibana. You should be prompted to log in to Kibana. To log in, you can use the built-in ‘elastic’ user and the password ‘changeme’. NOTE: These are the same credentials used in the arcsight module installation. When you change them ensure you update your logstash configuration.
2. Open the "[ArcSight] Network Overview Dashboard"

For more information on exploring security data with the ArcSight module, see the documentation on "Exploring Your Security Data".

In this blog series, we will also cover using ArcSight with Kafka, and X-Pack Alerting to notify when security events occur.

If you want to check out more about security analytics presentations, we suggest Security Analytics @ USAA, Tapping Out Security with FireEye, Hunting the Hackers by Cisco's Talos, Tinder: Keeping Your Data From Getting Swiped Right Away , and Cyber Security Log Analytics with Decision Lab.

Interesting in learning more? Check out other posts in the Elasticsearch + ArcSight series.

• Part 1 - How to send Common Event Format data from ArcSight to Elasticsearch
• Part 2 - How to proactively monitor security data in Elasticsearch with X-Pack alerting features
• Part 3 - Scaling your ArcSight and Elasticsearch architecture
• Part 4 - Detecting Successful SSH Brute Force Attacks
• Part 5 - Detecting Unusual Process using Rules Based Alerts
• Part 6 - Using Machine Learning to Detect Rare (unusual) Processes on a Server