GBase Heterogeneous OLTP Database Migration Solution
Currently, mainstream international database products such as Oracle and DB2 have been widely deployed in core business systems across numerous industries in China for many years. When enterprises look to replace or modernize these systems, they must address challenges such as data migration, application compatibility, and new system stability during the entire transition from heterogeneous databases, in order to mitigate the risks of migrating the database management system architecture.
Recently, in the database product application migration capability evaluation organized by the National Industrial Information Security Development Research Center of the Ministry of Industry and Information Technology, General Data Technology's shared-storage database cluster GBase 8s successfully passed the evaluation, demonstrating outstanding performance in areas such as heterogeneous database compatibility, migration assessment, migration implementation, hardware/software compatibility, migration efficiency, performance optimization, and data validation—comprehensively verifying its ability to handle real application migration scenarios.
Below is a brief introduction to the migration solution for heterogeneous OLTP databases to GBase 8s:
Project Migration Process
The migration process serves as the guiding framework for the entire migration effort, enabling structured and orderly execution. The main process steps are as follows:
Assessment & Solution Design
This phase focuses on pre-migration assessment and the formulation of a plan to guide migration execution. The contents include:
1. Pre-Migration Survey: Content and Key Points
The focus of the survey is to understand customer requirements and current pain points so that targeted improvement plans can be provided after migration. The pre-migration survey is a critical step in the migration project and must not be omitted.
1.1 Goals of Pre-Migration Survey
Understand the scope of the migration project
Assess migration workload, implementation timeline, staffing needs, and skill requirements
Assess technical challenges and primary project risks
1.2 Content of Pre-Migration Survey
Current system status: topology of the original database system architecture, related configurations, upstream/downstream dependencies, and the original database version
System operational status: application scenarios, criticality assessment, application deployment architecture, business types, approximate proportion of CRUD operations in business scenarios, concurrency levels, and total number of application system users
System performance requirements: response latency requirements during peak business hours, batch processing concurrency, data volume, processing time, and business data lifecycle
Interface details: application development language, ORM framework version information in the development environment, database interface protocols, connection pool configuration, persistent vs. short connections, and third-party tool support requirements
2. Migration Solution Design
The migration solution should be based on the pre-migration survey results and tailored to the customer’s usage requirements and pain points, to formulate the replacement and transformation implementation plan. It mainly includes:
Hardware configuration and deployment plan for the target system;
Targeted solutions addressing the customer’s requirements and pain points;
Steps and tasks for migration execution;
Implementation plan and milestones.
3. Hardware Resource Assessment
The hardware sizing for a GBase 8s cluster uses a reverse-engineering approach: first obtain detailed configuration information of the original database, then determine the configuration of each GBase 8s server and the number of servers based on total CPU cores, memory capacity, disk capacity, and projected data growth. In principle, the total CPU cores, memory capacity, disk capacity, and number of disk spindles across all servers in the GBase 8s cluster should not be less than those of the original system. For scenarios where the reverse-engineering approach is not suitable, the recommended hardware configurations for GBase 8s clusters can also be referenced.
Data and Business Migration
GBase 8s offers a seamless migration solution to ensure the overall risk of the migration is manageable and system operations remain smooth during the transition. The process is divided into three stages: In the first stage, after completing a full backup and restore, real-time data synchronization from the current primary source database to the standby GBase 8s database is achieved using data synchronization tools. After the first stage runs stably for a period, the primary and standby roles are switched, entering the second stage: the GBase 8s cluster takes over business operations, and business data is synchronized back to the source database in real time. In the final stage, the original database system is cut over and decommissioned, completing the smooth migration.
Next, technical challenges such as full and incremental data migration and data comparison before and after migration must be planned in advance.
Full Data Migration Strategy: Assessment and Key Points
For full data migration, the migration time window is largely determined by whether the migration is done in one batch or in phases. The following key factors must be considered:
Amount of data to be migrated from the source database [requires actual measurement to assess the data compression ratio within the source database];
Allowable downtime window for the business [during which the source database must be read-only and under light load—this is often a hard constraint];
Data export performance of the source database [requires actual measurement to determine capability];
Number of migration tool servers, I/O performance, and network bandwidth to the GBase 8s cluster [environmental constraints of the migration];
Loading performance of GBase 8s cluster nodes [comprehensive consideration of I/O performance and network bandwidth].
The migration time window can be estimated using the following formula:
Total migration time = Source database export time + GBase 8s loading time + Data validation time
Additional considerations for data migration:
The data export performance of the source database must be verified through actual testing;
When estimating the utilization of 10 Gigabit Ethernet bandwidth, a margin of redundancy should be allowed—typically a factor of 1/2 is applied in calculations;
The overall time window should be evaluated assuming serial export and loading to ensure a safety margin for unexpected situations, while the actual execution can be designed with multiple concurrent operations to optimize the migration.
Data Validation
Data validation is performed by comparing the processing results of the same data on the original system and the migrated system, and inferring the correctness of the migration process based on the consistency of the results.
Common data validation methods:
Total-level validation: compare count, sum, and avg values;
Dimension-level validation: group by various dimensions within a table and verify a metric value using count, sum, avg;
Error comparison: when inconsistencies are found in tables, bring those tables into a common environment for detailed analysis, such as performing a MINUS operation to identify and further analyze the differing data;
Validation report: the report is the output of the validation work, documenting and analyzing data consistency issues, tracing previously resolved problems, and building a knowledge base of common data validation issues.