AWS Data Migration

When a business with on-premises or inhouse data-center plans for cloud migration, one of their most prominent concerns is how their applications which rely on databases at backend will be migrated to cloud without any downtime. Cloud service providers have brought forth number of tools to address these requirements and make migration process as smooth as possible. In this article we will discuss details about the AWS Data Migration Service which simplifies the data migration process for all types of data stores- relational, data warehouses or non-relational (No SQL).

AWS Data Migration Service (shortly knows as AWS DMS) is a web service which runs using two end points one for source data store and another for target data store with one of them necessarily be on AWS. In addition to the two end points, it needs a replication instance and a replication task. It supports both homogenous (between same database platform like oracle to oracle) and heterogenous migrations (between different database platforms like Oracle to Microsoft SQL Server). Consolidation of multiple databases to a single data warehouse can be carried out if the business use case calls for a switch from a single tenant to multi-tenant model after the migration.

Advantages of AWS Data Migration Service-

1. Easy to use – AWS Data Migration service is very lucid as there is no need to install any external drivers or applications and for homogenous migrations majority of the time no schema change is required. Migration process can be initiated by the user using few clicks and the DMS service itself manages the complexities of the migration process.

2. Minimal Downtime – The way Data Migration Service operates is that after a successful connection and privileges test, it downloads and stores the schema information and uses it for tasks configuration. The source database remains operational during the migration process. Once initial replication is complete target database remains synchronised with the source as per the timeline chosen by user. Hence, technical team can switchover at a convenient time with minimal disruption.

3. Supports majority of widely used databases – With AWS Data Migration Service, you can migrate data to and from most of the open source as well as commercial databases provided either the source or target is in AWS. It supports both homogenous and heterogenous migrations.

4. Cost Efficient – As you have to pay only for any additional log storage or compute resources used in the migration, it is less expensive than the traditional migration processes. Some use case studies report migration of a terabyte-size database took less than $5.

5. Ongoing replication – AWS Data Migration Service can also be used to achieve HA by using ongoing replication instead of one-time migration. The source and target will remain in sync with minimal latency and switchover can be made easily in case of an incident.

6. Reliability – AWS data migration service is both resilient and self-healing. It proactively monitors source, target, network and replication instance and automatically restarts the stopped migration process in case of an interruption. Multi-AZ option can be used for high availability for migration and replication by enabling redundant replication instances.

High Level Operational Overview- The main high-level steps performed while doing migration using AWS Data Migration service are:

• Creation of a replication server
• Creation of source and target endpoints with data stores connection information.
• Creation of one or more migration tasks to migrate data between source and target datastores.

Three major phases of a task consist of:

• Full Load of existing data
• Application of cached changes
• Ongoing replication

By default, AWS Data Migration Service (DMS) takes a minimalistic approach means while creating the target schema it only creates the objects necessary to do the migration. Secondary Indexes, non-primary key constraints or data defaults are omitted. If your business use case needs complete schema, then you can use either of the 2 approaches which fits your scenario. In case of homogenous migration, you can use your database engines native tools to export and import the schema itself without any data.  In case of heterogenous migration, you can use AWS Schema Conversion Tool (AWS SCT) to generate a complete target schema. In case a performance issue is experienced during migration phase disabling or removing secondary indexes helps.