Elastic Stack 7.2.0 is now available on the Azure Marketplace

We’re pleased to announce that Elastic Stack 7.2.0 is available on the Azure Marketplace through our Azure Resource Manager (ARM) solution template. With this release, users running the Elastic Stack in Azure will be able to take advantage of all the many features that Elastic Stack 7.2.0 has to offer, including Elastic SIEM.

Leveraging Azure features

Through our close partnership with Microsoft, our ARM template has continued to benefit from the evolving capabilities of the Azure platform. We’d like to take this opportunity to highlight some of the new features recently introduced into the ARM template.

8-, 16-, and 32TiB managed disks

With the general availability of Standard HDD, Standard SSD, and Premium SSD disks up to 32TiB, it is now possible to deploy Elasticsearch clusters with large amounts of storage space on managed disks of varying capability, to suit a variety of use cases.

For example, deploying a three master-eligible node cluster of the memory optimized Standard_E16s_v3 virtual machines (VMs), where each VM has 32 managed disks attached (each 32TiB in size), results in total space for data of 3072TiB (or roughly 3 pebibytes), configured as a RAID 0 array on each VM to maximize IOPS throughput:

template_version=7.2.0
template_base_uri="https://raw.githubusercontent.com/elastic/azure-marketplace/$template_version/src/"
resource_group="<resource group>"
location="<location>"
az group create --name "$resource_group" --location "$location"
az group deployment create \
    --resource-group "$resource_group" \
    --template-uri ${template_base_uri}mainTemplate.json \
    --parameters _artifactsLocation=$template_base_uri \
        esClusterName=elasticsearch \
        adminUsername=russ \
        authenticationType=password \
        adminPassword=Password1234 \
        dataNodesAreMasterEligible=Yes \
        vmSizeDataNodes=Standard_E16s_v3 \
        vmDataNodeCount=3 \
        vmDataDiskCount=32 \
        vmDataDiskSize=32TiB \
        securityBootstrapPassword=BootstrapPassword123 \
        securityAdminPassword=AdminPassword123 \
        securityKibanaPassword=KibanaPassword123 \
        securityLogstashPassword=LogstashPassword123 \
        securityBeatsPassword=BeatsPassword123 \
        securityApmPassword=ApmPassword123 \
        securityRemoteMonitoringPassword=RemoteMonitoringPassword123

Accelerated networking

Accelerated networking has been generally available for some time for certain VM sizes. This moves much of Azure’s software-defined networking stack off the CPUs and into FPGA-based SmartNICs, reclaiming compute cycles and reducing the load on VMs, which improves network latency consistency in the process. For those VMs that support accelerated networking, it’s enabled by default within the ARM template.

New VM SKUs

The template continues to be updated to support the new virtual machine sizes on Azure, such as the Lsv2-series.

The Lsv2-series VMs have directly mapped local NVMe storage of 1.92TB per 8 vCPUs, making them an attractive choice for Elasticsearch nodes. The ARM template has been able to utilize local temp storage mounted on VMs for some time, as a cost-effective way of running an Elasticsearch cluster with good IOPS characteristics, with the caveats that come with the ephemeral nature of such storage. Expect an update to the ARM template in the future to be able to use local NVMe storage over local temp storage, where present.

Shard allocation awareness with update and fault domains

When data, master, and coordinating nodes are provisioned, they are added to separate availability sets and assigned an update and fault domain. The ARM template specifies the maximum number of 20 update domains for each availability set, as well as the maximum number of fault domains available within the target Azure region.

The fault and update domain values are added as node attributes to the elasticsearch.yml configuration file of each Elasticsearch node, and the values are used to configure shard allocation awareness on the node. This allows Elasticsearch to take the fault and update domain values into account for shard allocation.

Azure Marketplace Elastic Stack features

In addition to leveraging new features available in Azure, the ARM template has continued to evolve its own feature set, from SAML-based Single Sign-On with Azure Active Directory, to Transport Layer Security (TLS) configuration, to Application Gateway for load balancing. Some of the more recent features include multiple Logstash instances, shared storage for snapshots, and configuration of all built-in users.

Multiple Logstash instances

The Azure Marketplace and ARM template now has the ability to deploy multiple instances of Logstash alongside Elasticsearch and Kibana. The VMs running Logstash are placed into their own availability set, ensuring the VMs run across multiple physical servers, compute racks, storage units, and network switches.

Shared storage for snapshots

In addition to being able to specify an existing storage account to configure for snapshotting cluster data with the Azure Repository plugin, the shared storage account resource deployed with the template can be configured as the account to use for snapshots, keeping them together with the rest of the Stack resources within the resource group.

Configured with built-in users

Simple things should be simple! When the Elastic Security features are enabled, the cluster configures all of the built-in users needed to start using it to receive data from Beats, APM Server, and Logstash, making it simple to get started with the Elastic Stack in Azure for a plethora of use cases.

Coming soon: Elasticsearch Service on Azure

We trust you’ll continue to get real value from using our Azure Marketplace and ARM template offering to easily deploy the Elastic Stack to Azure; go try out the new release now, and check out the documentation to get started!

Elasticsearch Service on Elastic Cloud — that’s hosted Elasticsearch and Kibana — is coming to Microsoft Azure. With this new offering, users will be able to launch a hosted Elasticsearch and Kibana cluster on Microsoft Azure, offering the latest versions of the Elastic Stack as they’re released, one-click upgrades and scaleups, and access to Elastic support. Sign up for updates about this offering as news develops.