05 October 2021
Last month, AWS announced that the Amazon Elasticsearch Service has become Amazon OpenSearch Service. This change has effectively stopped any further updates to the Elasticsearch product line within the service due to changes to Elastic’s licensing model, and the forked OpenSearch will become the only product line to receive updates in the future.
In this post, we’ll walk through how to migrate from an Elasticsearch domain to an OpenSearch domain if you already have your domain defined within CloudFormation. If you don’t have your domain defined in CloudFormation but would like to, we’ll also cover that.
Let’s go through the changes in the service. The service name itself has changed from Amazon Elasticsearch Service to Amazon OpenSearch Service or, annoyingly, its current full canonical name “Amazon OpenSearch Service (successor to Amazon Elasticsearch Service)”. The Kibana equivilent is also now known as OpenSearch Dashboards.
The 18 Elasticsearch versions currently supported in the service will be all the Elasticsearch versions there will ever be, with only OpenSearch versions being included in the future. This also includes the possibility for the addition of features exclusive to OpenSearch, though Elastic could easily incorporate OpenSearch features into Elasticsearch if they wanted, despite the reverse no longer being an option. OpenSearch has added 3 of these exclusive features in their first version: Transforms, Data Streams, and Notebooks.
Though AWS has provided an easy upgrade path from Elasticsearch to OpenSearch within the console, the same cannot be said about CloudFormation which has created a new resource type for OpenSearch. You must not simply update the CloudFormation type in your template, as this will lead to the deletion of your domain and all data within it.
We will be migrating from one stack from another in this case (which is helpful as many of you may have used “elasticsearch” or “es” in your stack name), although you could follow a similar approach entirely within an existing stack. Despite the CloudFormation docs indicating OpenSearch resources are not supported for this operation (as of the time this was published), almost all resources created from about the end of 2019 should have CloudFormation import support, as is the case here.
Before beginning, you should first confirm that your configuration won’t be affected by the minor breaking changes and you should take a manual snapshot for safety, as the process is irreversable. This is a dangerous process if you don’t take care so please follow all steps and precautions carefully if your data is critical.
To begin, we will prepare the existing stack to be deprecated. If you do not currently have your domain within a CloudFormation stack (but would like to), you can instead generate a template for your new domain using Former2.
My existing template looks like this:
Resources: ... MyElasticsearchDomain: Type: AWS::Elasticsearch::Domain Properties: DomainName: mydomain ElasticsearchClusterConfig: InstanceCount: 3 InstanceType: t3.medium.elasticsearch DedicatedMasterEnabled: true DedicatedMasterType: t3.medium.elasticsearch DedicatedMasterCount: 3 ZoneAwarenessEnabled: true ZoneAwarenessConfig: AvailabilityZoneCount: 3 EBSOptions: EBSEnabled: true VolumeSize: 20 VolumeType: gp2 ElasticsearchVersion: "7.10" ...
Your template may have a number of different properties and other resources not shown here. Take a copy of the contents of your template now, and save this for later.
Now, we’re going to add the
UpdateReplacePolicy attributes to the resource with the value
Retain, like so:
Resources: ... MyElasticsearchDomain: DeletionPolicy: Retain UpdateReplacePolicy: Retain Type: AWS::Elasticsearch::Domain Properties: DomainName: mydomain ElasticsearchClusterConfig: InstanceCount: 3 InstanceType: t3.medium.elasticsearch DedicatedMasterEnabled: true DedicatedMasterType: t3.medium.elasticsearch DedicatedMasterCount: 3 ZoneAwarenessEnabled: true ZoneAwarenessConfig: AvailabilityZoneCount: 3 EBSOptions: EBSEnabled: true VolumeSize: 20 VolumeType: gp2 ElasticsearchVersion: "7.10" ...
By doing this, we are telling CloudFormation to not touch the connected resource (the domain) when this resource is deleted from the stack. In my case, I also had a number of CloudWatch alarms and even some custom resources for index and search template creation defined in my stack. These aren’t important to me during this short migration exercise, so I simply deleted them outright from my template as they will be recreated based on the template copy we took previously. My template now consists of only the
AWS::Elasticsearch::Domain resource, as well as any parameters and outputs that existed previously.
Update your stack now with the new content. As expected, the supporting resources such as CloudWatch Alarms will be deleted during this process (if any), however the Elasticsearch domains remain untouched.
Next up, we’ll use the defined process to upgrade the domain in-place. Note this can produce a minor downtime so plan around this. My upgrade with a small amount of documents took approximately 30 minutes.
Open the AWS Management Console, choose the domain that you want to upgrade, choose Actions, and then select Upgrade.
Choose OpenSearch 1.0 as the version to upgrade to, and I highly recommend selecting the Enable compatibility mode option to reduce the risk of any incompatibility issues. Check upgradeability and once verified, you can select the Upgrade operation.
You can prepare the next steps whilst waiting for the upgrade to complete.
We’ll now prepare our new template for our new OpenSearch-specific stack. If you previously used Former2 to define your stack, you’ll only need to make the
DeletionPolicy change from the steps below.
Using the copy of your original template, carefully make the following adjustments:
UpdateReplacePolicyattributes to the resource, as previously performed
EngineVersionand set its value to
ClusterConfig, if set
DedicatedMasterType), change the
ClusterConfig, if you have specified
ColdStorageOptionsyou must remove it as it is not currently supported
!GetAtt MyDomain.DomainArn), change these to instead use
Descriptionand any comments to remove Elasticsearch references, if needed
My new template looks like this:
Resources: ... MyOpenSearchDomain: DeletionPolicy: Retain UpdateReplacePolicy: Retain Type: AWS::OpenSearchService::Domain Properties: DomainName: mydomain ClusterConfig: InstanceCount: 3 InstanceType: t3.medium.search DedicatedMasterEnabled: true DedicatedMasterType: t3.medium.search DedicatedMasterCount: 3 ZoneAwarenessEnabled: true ZoneAwarenessConfig: AvailabilityZoneCount: 3 EBSOptions: EBSEnabled: true VolumeSize: 20 VolumeType: gp2 EngineVersion: "OpenSearch_1.0" ...
Once you have confirmed the in-place upgrade has completed, and prepared your new stack for import, we can import the new stack. You can check the Logs tab in your domain to confirm the upgrade.
Open the CloudFormation console and use the Create stack > With existing resources (import resources) option. Upload your newly prepared template with the
AWS::OpenSearchService::Domain resource within it.
You’ll then be prompted for the domain name of the existing domain. Carefully copy this name from the OpenSearch console domain listing, or from the template if hardcoded. Proceed to give your new stack a name, which should be different than the one currently existing, then finalize the stack creation. As your domain is only being imported, this process should only take a moment.
After confirming the stack creation has been successful, uncomment any related resources from the stack such as CloudWatch alarms and update your stack in-place to ensure these are put back in. I recommend also adding a tag to the stack at this point to make doubly sure that CloudFormation is aware of the resource and is targetting it correctly. You can optionally also remove the
UpdateReplacePolicy attributes at this time, however they are a good safety feature for preventing against accidental deletions so I do recommend you leave them in place.
We now have both the old and the new stack in place, so let’s get rid of the old one. You can ignore this part entirely if you did not have an existing stack prior.
If you have any stacks that are dependant on a CloudFormation output export from the original, you can now update those stacks to point to the new export name you defined.
Once you have updated all of the export references, you may delete the older Elasticsearch stack. Before doing this, double check the template for the stack you are deleting only contains a single
AWS::Elasticsearch::Domain resource, and that it has at least
DeletionPolicy: Retain at the same level as
Properties. If you have missed any export values, the stack events will inform you of this during the stack deletion, which you can remediate and reattempt the deletion.
Once the stack is deleted you’re done.
I hope this has been a helpful resource for you to upgrade your domain. Reach out to me on Twitter at @iann0036 if you liked this post. Happy searching!