Switch from APM to Integrations Server payload

This example shows how to use the Elasticsearch Service RESTful API to switch from using APM & Fleet Server to Integrations Server.

Requirements

edit

Given a deployment that is using an APM & Fleet Server with Elastic Stack version 8.0 or later, it is possible to start using Integrations Server instead by updating the deployment with an IntegrationsServerPayload. Switching from APM & Fleet Server to Integrations Server in this way ensures that the endpoints and credentials currently used by APM Server and Fleet Server remain the same after the switch.

In order to start using the IntegrationsServerPayload, you first need to enable the APM integration for Elastic Agent by following the steps in Switch to the Elastic APM integration.

API request example

edit

The example shows how to use the API to create a deployment with APM with version 8.0 and update the deployment to switch to Integrations Server.

Create a deployment with APM

edit

When creating a deployment with version 8.0 using the ApmPayload, the APM integration for Elastic Agent is enabled by default.

The following creates a deployment that uses the gcp-storage-optimized deployment template in the gcp-us-east4 region

curl -XPOST \
-H 'Content-Type: application/json' \
-H "Authorization: ApiKey $EC_API_KEY" \
"https://api.elastic-cloud.com/api/v1/deployments" \
-d '
{
  "resources": {
    "apm": [
      {
        "elasticsearch_cluster_ref_id": "main-elasticsearch",
        "region": "gcp-us-east4",
        "plan": {
          "cluster_topology": [
            {
              "instance_configuration_id": "gcp.apm.n2.68x32x45",
              "zone_count": 1,
              "size": {
                "resource": "memory",
                "value": 1024
              }
            }
          ],
          "apm": {
            "version": "8.0.0"
          }
        },
        "ref_id": "main-apm"
      }
    ],
    "elasticsearch": [
      {
        "region": "gcp-us-east4", 
        "settings": {
          "dedicated_masters_threshold": 6
        },
        "plan": {
          "autoscaling_enabled": false,
          "cluster_topology": [
            {
              "zone_count": 2,
              "elasticsearch": {
                "node_attributes": {
                  "data": "hot"
                },
                "enabled_built_in_plugins": []
              },
              "instance_configuration_id": "gcp.es.datahot.n2.68x10x45",
              "node_roles": [
                "master",
                "ingest",
                "transform",
                "data_hot",
                "remote_cluster_client",
                "data_content"
              ],
              "id": "hot_content",
              "size": {
                "resource": "memory",
                "value": 8192
              }
            },
            {
              "zone_count": 2,
              "elasticsearch": {
                "node_attributes": {
                  "data": "warm"
                },
                "enabled_built_in_plugins": []
              },
              "instance_configuration_id": "gcp.es.datawarm.n2.68x10x190",
              "node_roles": [
                "data_warm",
                "remote_cluster_client"
              ],
              "id": "warm",
              "size": {
                "resource": "memory",
                "value": 0
              }
            },
            {
              "zone_count": 1,
              "elasticsearch": {
                "node_attributes": {
                  "data": "cold"
                },
                "enabled_built_in_plugins": []
              },
              "instance_configuration_id": "gcp.es.datacold.n2.68x10x190",
              "node_roles": [
                "data_cold",
                "remote_cluster_client"
              ],
              "id": "cold",
              "size": {
                "resource": "memory",
                "value": 0
              }
            },
            {
              "zone_count": 1,
              "elasticsearch": {
                "node_attributes": {
                  "data": "frozen"
                },
                "enabled_built_in_plugins": []
              },
              "instance_configuration_id": "gcp.es.datafrozen.n2.68x10x95",
              "node_roles": [
                "data_frozen"
              ],
              "id": "frozen",
              "size": {
                "resource": "memory",
                "value": 0
              }
            },
            {
              "zone_count": 3,
              "instance_configuration_id": "gcp.es.master.n2.68x32x45",
              "node_roles": [
                "master",
                "remote_cluster_client"
              ],
              "id": "master",
              "size": {
                "resource": "memory",
                "value": 0
              },
              "elasticsearch": {
                "enabled_built_in_plugins": []
              }
            },
            {
              "zone_count": 2,
              "instance_configuration_id": "gcp.es.coordinating.n2.68x16x45",
              "node_roles": [
                "ingest",
                "remote_cluster_client"
              ],
              "id": "coordinating",
              "size": {
                "resource": "memory",
                "value": 0
              },
              "elasticsearch": {
                "enabled_built_in_plugins": []
              }
            },
            {
              "zone_count": 1,
              "instance_configuration_id": "gcp.es.ml.n2.68x16x45",
              "node_roles": [
                "ml",
                "remote_cluster_client"
              ],
              "id": "ml",
              "size": {
                "resource": "memory",
                "value": 0
              },
              "elasticsearch": {
                "enabled_built_in_plugins": []
              }
            }
          ],
          "elasticsearch": {
            "version": "8.0.0"
          },
          "deployment_template": {
            "id": "gcp-storage-optimized" 
          }
        },
        "ref_id": "main-elasticsearch"
      }
    ],
    "enterprise_search": [],
    "kibana": [
      {
        "elasticsearch_cluster_ref_id": "main-elasticsearch",
        "region": "gcp-us-east4",
        "plan": {
          "cluster_topology": [
            {
              "instance_configuration_id": "gcp.kibana.n2.68x32x45",
              "zone_count": 1,
              "size": {
                "resource": "memory",
                "value": 1024
              }
            }
          ],
          "kibana": {
            "version": "8.0.0"
          }
        },
        "ref_id": "main-kibana"
      }
    ]
  },
  "name": "switch-to-integrations-server",
  "metadata": {
    "system_owned": false
  }
}
'

	The region where the deployment is created
	The deployment template used by the deployment

Identify the instance configuration to use for Integrations Server

edit

Once the deployment is created, find the instance_configuration_id for IntegrationsServerPayload supported by the deployment template used by the deployment created in the previous step.

In the example above, the deployment was created using the gcp-storage-optimized deployment template in the gcp-us-east4 region.

To find the instance_configuration_id, fetch the deployment template using the template ID, the region, and the version used by the deployment (Integrations Server is supported on version 8.0 and higher).

curl -XGET \
-H 'Content-Type: application/json' \
-H "Authorization: ApiKey $EC_API_KEY" \
"https://api.elastic-cloud.com/api/v1/deployments/templates/gcp-storage-optimized?region=gcp-us-east4&show_instance_configurations=false&stack_version=8.0.0"

This returns a deployment template like the following:

{
    "description": "Good for most ingestion use cases with 7-10 days of data available for fast access. Also good for light search use cases without heavy indexing or CPU needs.",
    "name": "Storage optimized",
    "template_category_id": "storage-optimized",
    "id": "gcp-storage-optimized",
    "deployment_template": {
        "resources": {
            "integrations_server": [
                {
                    "elasticsearch_cluster_ref_id": "es-ref-id",
                    "region": "gcp-us-east4",
                    "plan": {
                        "cluster_topology": [
                            {
                                "instance_configuration_id": "gcp.integrationsserver.n2.68x32x45", 
                                "zone_count": 1,
                                "size": {
                                    "resource": "memory",
                                    "value": 1024
                                }
                            }
                        ],
                        "integrations_server": {}
                    },
                    "ref_id": "integrations_server-ref-id"
                }
            ],
            "elasticsearch": [
                ...
            ],
            "enterprise_search": [
                ...
            ],
            "kibana": [
                ...
            ],
            "apm": [
                ...
            ]
        }
    },
    "order": 1,
    "system_owned": true,
    "metadata": [
        ...
    ]
}

The instance configuration ID to use in IntegrationsServerPayload in the next step.

Update deployment using IntegrationsServerPayload

edit

Finally, to switch to Integrations Server, update the deployment using the IntegrationsServerPayload, setting instance_configuration_id to the value identified in the previous step.

curl -XPUT \
-H 'Content-Type: application/json' \
-H "Authorization: ApiKey $EC_API_KEY" \
"https://api.elastic-cloud.com/api/v1/deployments/<deployment-id>" \
-d '
{
  "name": "switch-to-integrations-server",
  "alias": "switch-to-integrations-server",
  "prune_orphans": false, 
  "metadata": {
    "system_owned": false,
    "hidden": false
  },
  "resources": {
    "integrations_server": [
      {
        "region": "gcp-us-east4",
        "ref_id": "main-integrations_server",
        "elasticsearch_cluster_ref_id": "main-elasticsearch",
        "plan": {
          "cluster_topology": [
            {
              "instance_configuration_id": "gcp.integrationsserver.n2.68x32x45", 
              "zone_count": 1,
              "size": {
                "resource": "memory",
                "value": 1024
              }
            }
          ],
          "integrations_server": {
            "version": "8.0.0"
          },
          "transient": {
            "strategy": {
              "autodetect": {}
            }
          }
        }
      }
    ]
  }
}
'

	Make sure to use the `instance_configuration_id` for Integrations Server from the deployment template.
	Make sure `prune_orphans` is set to `false`. `prune_orphans` is an important parameter. It specifies how resources not included in the body of this PUT request should be handled. If `false`, those resources not included are kept intact.

Manage your APM & Fleet Server

edit

Beginning with Elastic Stack version 8.0, Integrations Server is replacing APM & Fleet Server. New deployments with version 8.0 will use Integrations Server automatically. Existing deployments using APM & Fleet Server will continue to use APM & Fleet Server after upgrading to version 8.0.

You have the option to add a combined Application Performance Monitoring (APM) Server and Fleet Server to your deployment. APM allows you to monitor software services and applications in real time, turning that data into documents stored in the Elasticsearch cluster. Fleet allows you to centrally manage Elastic Agents on many hosts.

As part of provisioning, the APM Server and Fleet Server are already configured to work with Elasticsearch and Kibana. At the end of provisioning, you are shown the secret token to configure communication between the APM Server and the backend APM Agents. The APM Agents get deployed within your services and applications.

From the deployment APM & Fleet page you can also:

Get the URL to complete the APM agent configuration.
Use the elastic credentials to go to the APM area of Kibana. Step by step instructions to configure a variety of agents are available right in Kibana. After that, you can use the pre-built, dedicated dashboards and the APM tab to visualize the data that is sent back from the APM Agents.
Use the elastic credentials to go to the Fleet area of Kibana. Step by step instructions to download and install Elastic Agent on your hosts are available right in Kibana. After that, you can manage enrolled Elastic Agents on the Agents tab, and the data shipped back from those Elastic Agents on the Data streams tab.
Reset the APM secret token.

Resetting the token disrupts your APM service and restarts the server. When the server restarts, you’ll need to update all of your agents with the new token.
Access the APM & Fleet logs and metrics.
Stop and restart your APM & Fleet Server.
Upgrade your APM & Fleet Server version if it is out of sync with your Elasticsearch cluster.
Fully remove the APM & Fleet Server, delete it from the disk, and stop the charges.

Upgrading to Elastic Stack 8.0

edit

The following APM settings have been removed in Elastic Stack version 8.0. This change is only relevant to users upgrading a standalone (legacy) deployment of APM Server to Elastic Stack version 8.0. Check Add APM user settings for more details.

apm-server.api_key.enabled
apm-server.api_key.limit
apm-server.ilm.*
apm-server.frontend.*
apm-server.register.*
apm-server.rum.allow_service_names
apm-server.rum.event_rate.lru_size
apm-server.rum.event_rate.limit
apm-server.rum.rate_limit
output.elasticsearch.bulk_max_size
output.elasticsearch.index
output.elasticsearch.indices
output.elasticsearch.pipeline
output.elasticsearch.pipelines
output.elasticsearch.worker
setup.*
queue.*

« Manage your Integrations Server Enable Enterprise Search »