Migrating from self-managed Elastic Stack to Elastic Cloud using snapshot and restore with Amazon S3

I had the pleasure of talking with many interesting people at AWS re:Invent 2023 in Las Vegas while I was helping out at Elastic®’s booth in the Expo hall. I spoke to many SREs, developers, solution architects, and other professionals who were running a self-managed Elastic Stack and had questions about the process of migrating their Elastic stack to Elastic Cloud. 

To help provide guidance on questions about Elastic Cloud migration, this blog post provides a walkthrough of how to set up a self-managed Elastic stack and then migrate that Elastic stack’s contents to Elastic Cloud using a snapshot and restore migration method. For the storage location of the snapshot, we’ll use Amazon S3, which is a perfect service for saving and accessing files with fine grain control provided by AWS Identity and Access Management (IAM).

This blog post covers four main steps:

1. Set up and run a self-managed Elastic stack using Docker.

2. Create a simple search index in the self-managed Elastic stack.

3. Create an Elastic Cloud deployment.

4. Migrate the self-managed Elastic stack to Elastic Cloud.

In your terminal, download two files, .env and docker-compose.yml, into the elastic-self-managed directory by running the following two CURL commands:

Edit the .env file in your terminal. 

• On a macOS or Linux based computer, open the file for editing by typing “nano .env”. Then after you’ve made the necessary edits, close the file and save the changes by typing “Ctrl + x”. Press the “y” key and then the “Enter” key to save the changes.

• On a Windows based computer, type “notepad .env” to open the file for editing. This will open the Notepad text editor. Use the Notepad’s File menu to save and close the file after you’ve made the necessary edits.

Make the following changes to the .env file:

• Specify a password for the ELASTIC_PASSWORD and KIBANA_PASSWORD variables.

• Set STACK_VERSION to the current Elastic stack version, which is 8.11.1 at the time this blog post was published.

• Set ES_PORT to 127.0.0.1:9200, which ensures Elasticsearch® is only accessible from the computer where the Elastic stack is running.

After you made the changes, the .env file should look something like this:

Run the following command in your terminal to start the Elastic stack:

Go back to Docker Desktop, where you should now see the self-managed Elastic stack running.

Now that you’ve got an Elastic deployment running on your local computer, you’re ready to create a search index. Open the locally running deployment by going to http://localhost:5601 in a web browser to access Kibana® (which is the Elastic stack’s user interface). You’ll be prompted for a Username and Password. Enter “elastic” as the Username and enter the Password that you specified in the .env file. 

You should see the Elastic Home page. Select Search.

>> Related article: What is an Elasticsearch index?

On the Search page, select Indices.

Click Create a new index.

On the Select an ingestion method page, click Use the API.

On the New search index page, enter an Index name like search-example-index and click Create index.

On the overview page for the newly created index, click the Try Console now link.

A new browser tab will open Dev Tools. By default, an example query is provided to insert six documents into the newly created index. Click the run button to execute the example query.

You should see a 200 - OK response confirming that the documents have been added to the index.

You now have a search index with six documents. This index is what we’ll use to confirm that the migration process is successful.

Go to Elastic Cloud and create a new deployment. There’s a 14-day free trial if you’re a new user of Elastic Cloud.

The release and timing of any features or functionality described in this post remain at Elastic's sole discretion. Any features or functionality not currently available may not be delivered on time or at all.