Example: Use standalone Elastic Agent with Elasticsearch Service to monitor nginx

edit

Example: Use standalone Elastic Agent with Elasticsearch Service to monitor nginx

edit

This guide walks you through a simple monitoring scenario so you can learn the basics of setting up standalone Elastic Agent, using it to work with Elasticsearch Service and an Elastic integration.

Following these steps, you’ll deploy the Elastic Stack, install a standalone Elastic Agent on a host to monitor an nginx web server instance, and access visualizations based on the collected logs.

Prerequisites
edit

To get started, you need:

  1. An internet connection and an email address for your Elastic Cloud trial.
  2. A Linux host machine on which you’ll install an nginx web server. The commands in this guide use an Ubuntu image but any Linux distribution should be fine.
Step 1: Install nginx
edit

To start, we’ll set up a basic nginx web server.

  1. Run the following command on an Ubuntu Linux host, or refer to the nginx install documentation for the command appropriate to your operating system.

    sudo apt install nginx
  2. Open a web browser and visit your host machine’s external URL, for example http://192.168.64.17/. You should see the nginx welcome message.

    Browser window showing Welcome to nginx!
Step 2: Create an Elastic Cloud deployment
edit

If you’ve already signed up for a trial deployment you can skip this step.

Now that your web server is running, let’s get set up to monitor it in Elastic Cloud. An Elastic Cloud Elasticsearch Service deployment offers you all of the features of the Elastic Stack as a hosted service. To test drive your first deployment, sign up for a free Elastic Cloud trial:

  1. Go to our Elastic Cloud Trial page.
  2. Enter your email address and a password.

    Start your free Elastic Cloud trial
  3. After you’ve logged in, select Create deployment and give your deployment a name. You can leave the default options or select a different cloud provider, region, hardware profile, or version.
  4. Select Create deployment.
  5. While the deployment sets up, make a note of your elastic superuser password and keep it in a safe place.
  6. Once the deployment is ready, select Continue. At this point, you access Kibana and a selection of setup guides.
Step 3: Create an Elastic Cloud API key
edit
  1. From the Kibana menu and go to Stack ManagementAPI keys.
  2. Select Create API key.
  3. Give the key a name, for example nginx example API key.
  4. Leave the other default options and select Create API key.
  5. In the Create API key confirmation dialog, change the dropdown menu setting from Encoded to Beats. This sets the API key format for communication between Elastic Agent (which is based on Beats) and Elasticsearch.
  6. Copy the generated API key and store it in a safe place. You’ll use it in a later step.
Step 4: Create an Elastic Agent policy
edit

Elastic Agent is a single, unified way to add monitoring for logs, metrics, and other types of data to a host. It can also protect hosts from security threats, query data from operating systems, and more. A single agent makes it easy and fast to deploy monitoring across your infrastructure. Each agent has a single policy (a collection of input settings) that you can update to add integrations for new data sources, security protections, and more.

  1. When your Elastic Cloud deployment is ready, open the Kibana menu and go to Fleet → Agent policies.

    Agent policies tab in Fleet
  2. Click Create agent policy.
  3. Give your policy a name. For this example we’ll call it nginx-policy.
  4. Leave Collect system logs and metrics selected.
  5. Click Create agent policy.

    Create agent policy UI
Step 5: Add the Nginx Integration
edit

Elastic integrations are a streamlined way to connect your data from popular services and platforms to the Elastic Stack, including nginx.

  1. From the Fleet → Agent policies tab, click the link for your new nginx-policy.

    The nginx-policy UI with integrations tab selected
  2. Note that the System integration (system-1) is included because you opted earlier to collect system logs and metrics.
  3. Click Add integration.
  4. On the Integrations page search for "nginx".

    Integrations page with nginx in the search bar
  5. Select the Nginx card.
  6. Click Add Nginx.
  7. Click the link to Add integration only (skip agent installation). You’ll install standalone Elastic Agent in a later step.
  8. Here, you can select options such as the paths to where your nginx logs are stored, whether or not to collect metrics data, and various other settings.

    For now, leave all of the default settings and click Save and continue to add the Nginx integration to your nginx-policy policy.

    Add Nginx Integration UI
  9. In the confirmation dialog, select to Add Elastic Agent later.

    Nginx Integration added confirmation UI with Add Elastic Agent later selected.
Step 6: Configure standalone Elastic Agent
edit

Rather than opt for Fleet to centrally manage Elastic Agent, you’ll configure an agent to run in standalone mode, so it will be managed by hand.

  1. In Fleet, open the Agents tab and click Add agent.
  2. For the What type of host are you adding? step, select nginx-policy from the drop-down menu if it’s not already selected.
  3. For the Enroll in Fleet? step, select Run standalone.

    Add agent UI with nginx-policy and Run-standalone selected.
  4. For the Configure the agent step, choose Download Policy. Save the elastic-agent.yml file to a directory on the host where you’ll install nginx for monitoring.

    Have a look inside the policy file and notice that it contains all of the input, output, and other settings for the Nginx and System integrations. If you already have a standalone agent installed on a host with an existing Elastic Agent policy, you can use the method described here to add a new integration. Just add the settings from the Configure the agent step to your existing elastic-agent.yml file.

  5. For the Install Elastic Agent on your host step, select the tab for your host operating system and run the commands on your host.

    Install Elastic Agent on your host step, showing tabs with the commands for different operating systems.

    Elastic Agent commands need to be run as root. You can prefix each agent command with sudo or you can start a new shell as root by running sudo su.

    If you’re prompted with Elastic Agent will be installed at {installation location} and will run as a service. Do you want to continue? answer Yes.

    If you’re prompted with Do you want to enroll this Agent into Fleet? answer no.

  6. You can run the status command to confirm that Elastic Agent is running.

    elastic-agent status
    
    ┌─ fleet
    │  └─ status: (STOPPED) Not enrolled into Fleet
    └─ elastic-agent
       └─ status: (HEALTHY) Running

    Since you’re running the agent in standalone mode the Not enrolled into Fleet message is expected.

  7. Open the elastic-agent.yml policy file that you saved.
  8. Near the top of the file, replace:

        username: '${ES_USERNAME}'
        password: '${ES_PASSWORD}'

    with:

        api_key: '<your-api-key>'

    where your-api-key is the API key that you generated in Step 3: Create an Elastic Cloud API key.

  9. Find the location of the default elastic-agent.yml policy file that is included in your Elastic Agent install. Install directories for each platform are described in Installation layout. In our example Ubuntu image the default policy file can be found in /etc/elastic-agent/elastic-agent.yml.
  10. Replace the default policy file with the version that you downloaded and updated. For example:

    cp /home/ubuntu/homedir/downloads/elastic-agent.yml /etc/elastic-agent/elastic-agent.yml

    You may need to prefix the cp command with sudo for the permission required to replace the default file.

    By default, Elastic Agent monitors the configuration file and reloads the configuration automatically when elastic-agent.yml is updated.

  11. Run the status command again, this time with the --output yaml option which provides structured and much more detailed output. See the elastic-agent status command documentation for more details.

    elastic-agent status --output yaml

    The results show you the agent status together with details about the running components, which correspond to the inputs and outputs defined for the integrations that have been added to the Elastic Agent policy, in this case the System and Nginx Integrations.

  12. At the top of the command output, the info section contains details about the agent instance. Make a note of the agent ID. In this example the ID is 4779b439-1130-4841-a878-e3d7d1a457d0. You’ll use that ID in the next section.

    elastic-agent status --output yaml
    
    info:
      id: 4779b439-1130-4841-a878-e3d7d1a457d0
      version: 8.9.1
      commit: 5640f50143410fe33b292c9f8b584117c7c8f188
      build_time: 2023-08-10 17:04:04 +0000 UTC
      snapshot: false
    state: 2
    message: Running
Step 7: Confirm that your Elastic Agent data is flowing
edit

Now that Elastic Agent is running, it’s time to confirm that the agent data is flowing into Elasticsearch.

  1. Check that Elastic Agent logs are flowing.

    1. Open the Kibana menu and go to Analytics → Discover.
    2. In the KQL query bar, enter the query agent.id : "{agent-id}" where {agent-id} is the ID you retrieved from the elastic-agent status --output yaml command. For example: agent.id : "4779b439-1130-4841-a878-e3d7d1a457d0".

      If Elastic Agent has connected successfully with your Elastic Cloud deployment, the agent logs should be flowing into Elasticsearch and visible in Kibana Discover.

      Kibana Discover shows agent logs are flowing into Elasticsearch.
  2. Check that Elastic Agent metrics are flowing.

    1. Open the Kibana menu and go to Analytics → Dashboard.
    2. In the search field, search for Elastic Agent and select [Elastic Agent] Agent metrics in the results.

      like the agent logs, the agent metrics should be flowing into Elasticsearch and visible in Kibana Dashboard. You can view metrics on CPU usage, memory usage, open handles, events rate, and more.

      Kibana Dashboard shows agent metrics are flowing into Elasticsearch.
Step 8: View your system data
edit

In the step to create an Elastic Agent policy you chose to collect system logs and metrics, so you can access those now.

  1. View your system logs.

    1. Open the Kibana menu and go to Management → Integrations → Installed integrations.
    2. Select the System card and open the Assets tab. This is a quick way to access all of the dashboards, saved searches, and visualizations that come with each integration.
    3. Select [Logs System] Syslog dashboard.
    4. Select the calandar icon and change the time setting to Today. The Kibana Dashboard shows visualizations of Syslog events, hostnames and processes, and more.
  2. View your system metrics.

    1. Return to Management → Integrations → Installed integrations.
    2. Select the System card and open the Assets tab.
    3. This time, select [Metrics System] Host overview.
    4. Select the calandar icon and change the time setting to Today. The Kibana Dashboard shows visualizations of host metrics including CPU usage, memory usage, running processes, and others.

      The System metrics host overview showing CPU usage, memory usage, and other visualizations
Step 9: View your nginx logging data
edit

Now let’s view your nginx logging data.

  1. Open the Kibana menu and go to Management → Integrations → Installed integrations.
  2. Select the Nginx card and open the Assets tab.
  3. Select [Logs Nginx] Overview. The Kibana Dashboard opens with geographical log details, response codes and errors over time, top pages, and more.

    The nginx logs dashboard shows various visualizations on the nginx logs.
  4. Refresh your nginx web page several times to update the logging data. You can also try accessing the nginx page from different web browsers. After a minute or so, the Browsers breakdown visualization shows the respective volume of requests from the different browser types.

    Kibana Dashboard shows agent metrics are flowing into Elasticsearch.

Congratulations! You have successfully set up monitoring for nginx using standalone Elastic Agent and an Elastic Cloud deployment.

What’s next?
edit