Getting started with the Azure Marketplaceedit

The Azure Marketplace and Resource Manager (ARM) template has reached its end of life. Instead, customers are strongly advised to use our newer Elastic Cloud (Elasticsearch Service) offering in the Azure Marketplace. To learn more, refer to the Native Azure integration documentation.

Navigate to the Azure Marketplace offering, and click the "Get it Now" button:

marketplace

This will take you straight to the solution from the Marketplace, within the Azure portal. If you’re not already logged into the portal, you’ll be prompted to do so first.

Once you’re ready to proceed, click the Create button to walk through the steps.

Marketplace stepsedit

Each step of the process is presented in its own blade within the portal, which we’ll walk through.

Basicsedit

The first step involves setting up the details for the username and authentication mechanism you’ll use to access the virtual machines deployed by the solution:

marketplace step1

The username must be a valid Ubuntu username and for authentication mechanism, choose between password or SSH key. When choosing SSH key, password authentication will be disabled on the virtual machines.

Next, choose the subscription, resource group and location into which to deploy the solution. When choosing an existing resource group, there a couple of points to bear in mind

  1. A resource group is a container that holds related resources. As such, those resources share the same lifecycle, permissions and policies
  2. The solution template deploys in incremental mode; If a previous solution deployment has been performed into the target resource group, the resources that exist in the resource group but are not in the template are left unchanged. All resources that are specified by the solution will be deployed, and for those resources that already exist and whose settings are unchanged, no change will be made. For those resources whose settings are changed however, the resource is provisioned with those new settings. What this means in practice is that you should carefully consider how the settings specified within the template being deployed may affect the resources that could be currently running Elasticsearch.

More often than not, you’ll want to deploy the solution into a new resource group. Providing access to Elasticsearch to existing Azure resources such as Azure App Services can be achieved by connecting the deployed virtual machines to an existing virtual network, which can be specified in the next step.

Cluster settingsedit

the Cluster settings blade provides the option to pick the version of Elasticsearch to deploy, and give the cluster a name:

marketplace step2

Additionally, the solution can deploy a new virtual network or use an existing virtual network within the same subscription and location as the target resource group. This is particularly useful in situations where resources may already be deployed in another resource group, and the cluster should be accessible to those resources on the same network, and possibly same subnet.

The Azure Marketplace constrains the available existing subnets to those that have a prefix size of /25 i.e. 128 addresses or more. The prefix size must be large enough to be able to accommodate the largest size cluster/deployment that the Marketplace offering can deploy, and the prefix size chosen is the smallest that meets this requirement. The ability to dynamically adjust the prefix size based on the cluster topology to be deployed is not possible.

Fortunately, this is only a Marketplace UI constraint and not an ARM template constraint! It is possible to deploy a cluster using the ARM template from our GitHub repository directly, and select the existing virtual network and subnet that you want to use, using the networking parameters. Be sure to target a specific GitHub tag release for repeatable deployments.

Nodes configurationedit

The Nodes configuration blade controls the number of each type of node to deploy for the cluster:

marketplace step3

Elasticsearch nodes resolve other nodes on the network by their hostname, which Azure DNS supports. It is therefore critical that when deploying a cluster to an existing virtual network, the hostnames for Elasticsearch nodes are unique. The Hostname prefix input will be applied to the naming convention for master, data and coordinating nodes, allowing for unique hostnames when deploying multiple clusters to the same virtual network.

For data nodes, the number of managed disks, in addition to size and type of disks, can be specified. When the number of managed disks is two or more, a RAID 0 array is created across the disks to improve overall operations. When no disks are specified, the node will use the temp storage of the VM for the data directory.

Temp storage, with filesystem /dev/sdb1 mounted on /mnt in Ubuntu, is present on the physical machine hosting the VM. It is ephemeral in nature and not persistent; A VM can move to a different host at any point in time for various reasons, including hardware failures. When this happens, the VM will be created on the new host using the OS disk from the storage account, and new temp storage will be created on the new host.

Using temp storage can be a cost effective way of running an Elasticsearch cluster on Azure with decent performance, so long as you understand the tradeoffs in doing so, by snapshotting frequently and ensuring adequate data redundancy through sufficient replica shards.

Kibana, Logstash and External Accessedit

The Kibana & Logstash blade determines whether to deploy Kibana and/or Logstash in addition to Elasticsearch, as well as how the cluster can be accessed from outside of the virtual network:

marketplace step4

When installing Kibana, a separate VM will be provisioned onto which Kibana will be deployed and a public IP address will be attached to the Kibana VM along with a Network Security group, allowing TCP access over ports 5601 and 22. The former is used to connect to Kibana through the browser, whilst the latter is used to connect to the Kibana VM via SSH.

marketplace step4 2

When installing Logstash, separate VMs will be provisioned onto which Logstash will be deployed. A Logstash configuration file can be specified, as well as additional Logstash plugins to install. The Logstash VMs are placed in an Availability set.

A jumpbox VM can optionally be deployed, which can be connected to via SSH in order to gain access to the Elasticsearch VMs.

Either an internal or external load balancer can be deployed from the Marketplace; When choosing external load balancer, both an internal and external load balancer will be deployed. Kibana will use the internal load balancer to communicate with Elasticsearch, and the external load balancer is configured with a public IP address , allowing access to Elasticsearch from outside of Azure over port 9200.

User configurationedit

An Elasticsearch cluster deployed through the Azure Marketplace is always deployed with a 30 day trial license that enables all of the platinum features of the Elastic Stack. Security features allow Basic Authentication configuration for the cluster, setting up six built-in user accounts

  1. The elastic user, a built-in superuser account
  2. The kibana user (kibana_system in 7.8.0+), a built-in account that Kibana uses to connect and communicate with Elasticsearch
  3. The logstash_system user, a built-in account that can be used by Logstash when storing monitoring information in Elasticsearch.
  4. The beats_system user, a built-in account that can be used by Beats when storing monitoring information in Elasticsearch.
  5. The apm_system user, a built-in account that can be used by APM server when storing APM information in Elasticsearch.
  6. The remote_monitoring_user user, a built-in account that can be used by Metricbeat when collecting and storing monitoring information in Elasticsearch.

marketplace step5

Passwords must be specified for all of these accounts.

Certificatesedit

Transport Layer Security (TLS) can be configured for both Elasticsearch and Kibana

marketplace step6

TLS can be configured for

Elasticsearch HTTP layer

exposes Elasticsearch APIs over HTTP. Kibana and Logstash communicate with Elasticsearch using the HTTP layer through the internal load balancer, as does an external load balancer through its public IP address, if deployed.

Either a certificate and key can be provided in a PKCS#12 archive (.p12/.pfx) which will be used to secure the HTTP layer of all Elasticsearch nodes, or a Certificate Authority (CA) certificate and in a PKCS#12 archive can be provided, which will be used to generate a certificate for each Elasticsearch node to secure the HTTP layer.

Elasticsearch Transport layer

used for internal communication between Elasticsearch nodes within the cluster.

A Certificate Authority (CA) certificate and in a PKCS#12 archive can be provided which will be used to generate a certificate for each Elasticsearch node to secure the Transport layer.

Kibana

communication between the browser and Kibana, through the public IP address of Kibana.

A certificate and private key can be provided in PEM format to secure communication between the browser and Kibana.

Configuring Transport Layer Security (TLS) within the Marketplace UI is optional, to keep the getting started experience simple for those wishing to try the offering out

marketplace step6 2

It is highly recommended that you configure TLS, to encrypt communications.

TLS is required for the Elasticsearch Transport layer with the Security features available at the basic license tier, so if you are intending on using the basic license with Security, providing certificates as part of a Marketplace deployment will be easier than not doing so, and then needing to connect to each Elasticsearch node after deployment to configure TLS, when activating a basic license.

elasticsearch-certutil command tool bundled with Elasticsearch can be used to simplify the creation of certificates for use with TLS.

Summaryedit

The last steps of configuring a deployment allow you to confirm the values provided for all inputs in the previous steps, and to read the terms and conditions before deployment:

marketplace step7

The Azure infrastructure also performs some final validation steps. Once you’re happy to proceed and the OK button is clicked, a new deployment will be started and within minutes, you’ll have Elasticsearch running on Azure!