Teradata to GBase 8a Migration Solution

Recently, Teradata announced its withdrawal from the Chinese market, a move that will undoubtedly create new opportunities for the domestic data warehouse sector. GBASE (General Data Technology Co., Ltd.), a leading provider of data warehouses in China, has developed the GBase 8a MPP (GBase UP LDW) distributed logical data warehouse. It has already completed the migration and replacement of Teradata and other foreign databases for over 100 users. Through numerous project POCs and subsequent implementation contracts, GBase 8a has accumulated extensive hands-on experience in migrating from Teradata. This has evolved into a complete migration methodology that can be rapidly replicated and scaled, enabling a high-precision transition from Teradata to GBase 8a.

Below, we explain the migration solution based on GBase 8a's proven experience, from top to bottom:

I. Let's first look at the post-migration results from an actual project:

• Deployed two clusters, A (7 nodes) and B (14 nodes), to host major business workloads

• Total data volume across all layers (SGA, ODS, F, DW, DM) exceeded 200+TB

• Completed migration of 1000+ programs

• Daily incremental base data loading is completed before 7:00 AM each day

• Monthly reports are published by the 3rd day of each month

• Overall performance improved by a factor of 2 to 10

II. The migration project process is the guiding framework that effectively organizes the structured execution of migration work. The main process steps are as follows:

III. Pre-migration research and migration solution design mainly involve conducting pre-migration investigation and assessment, and formulating a migration solution to guide execution on the ground. It includes the following points:

1. Pre-migration research content and key points

The focus of the research is to understand the customer's needs and current pain points, and to provide targeted improvement solutions after the migration!

• Migration implementation research

Migration implementation research is an essential work stage of the migration project; this part cannot be omitted during project implementation.

Migration implementation research may be conducted before or after contract signing, depending on the progress of the pre-sales phase.

• Objectives of migration implementation research

Understand the scope of the project migration implementation.

Assess migration workload, implementation duration, personnel quantity, and competency requirements.

Evaluate migration technical difficulties and primary project risks.

• Contents of migration implementation research

Current state of the source system: architecture, configuration, upstream/downstream dependencies, logical warehouse design.

System operation status: application scenarios, job types, workload profile, overall ETL processing flow, number of jobs, concurrency, processing time.

System performance requirements: requirements for different job types, such as time window requirements for batch processing, response time requirements for ad-hoc queries, and concurrency capability requirements.

Interfaces: upstream data ingestion methods, downstream data provisioning methods, requirements for third-party tool support.

2. Migration solution design

The migration solution is the project implementation plan formulated based on the migration research findings and aimed at addressing the customer's current needs and pain points. It primarily includes:

• Hardware configuration and deployment plan for the target system

• Targeted solutions addressing customer needs and pain points

• Steps and work content for the migration implementation

• Project work plan

3. Hardware resource sizing

The hardware sizing for GBase 8a clusters to replace Teradata uses a reverse calculation approach. First, obtain the detailed configuration information of the Teradata system. Then, derive the per-server configuration and the number of servers required for the GBase 8a cluster based on the total CPU cores, memory capacity, and disk capacity of the Teradata appliance. As a rule, the total CPU cores, total memory capacity, total disk capacity, and total number of disks across all servers in the GBase 8a cluster should be no less than twice that of the Teradata appliance.

IV. Data and business migration mainly involves pre-planning for technical challenges such as full and incremental data migration, ETL catch-up processing, and pre- and post-migration data comparison methods, as well as the data flow migration process.

1. Full data migration approach assessment and key points

The time window required to migrate full data largely determines whether the full migration is done in one shot or in batches. The following factors have a major influence and need careful consideration:

• Source database migration data volume [requires actual measurement to estimate the data compression ratio in the source system]

• Business-allowed downtime window [during this period the source system needs to be in read-only mode and under light load; this is often a hard constraint]

• Data export performance of the source database [needs actual measurement to confirm capability]

• Number of loading file servers, I/O performance, and network bandwidth to the GBase 8a cluster [migration environment constraints]

• Loading performance of GBase 8a cluster nodes [considering both I/O performance and network bandwidth]

• Type of incremental business: append only, IDU (Insert/Delete/Update), zipper table, or full refresh each time [determines the incremental catch-up method and whether batch migration is feasible]

• Whether the warehouse design supports layering and data processing idempotency [determines whether migration can be batched vertically by business or horizontally by layer]

To estimate the migration time window, the following formula should be used:

• Total migration time = Teradata export time + GBase 8a loading time

• Teradata export time = Teradata stored data volume (in GB) / Teradata parallel export performance (GB/hour)

• GBase 8a loading time = Teradata exported data volume (in GB) / GBase 8a parallel loading performance (GB/hour)

• GBase 8a parallel loading performance = Number of loading servers × 1000 MB/s × 1/2 × 3600

Points to note for data migration:

• The stored data volume on Teradata must be converted to the volume of data in flat files outside the database, so the compression ratio must be evaluated.

• Teradata parallel export performance needs to be empirically measured, and the workload on Teradata during export must be taken into account.

• GBase 8a parallel loading performance needs to be empirically measured; when estimating, refer to formula 4.

• When evaluating the utilization rate of 10 Gigabit bandwidth, a certain redundancy should be maintained; generally, a factor of 1/2 is applied in calculations.

• The total time window is estimated assuming a serial process of export followed by loading, rather than a pipelined approach, to ensure some redundancy for handling unexpected situations. However, the actual execution can be designed as a pipeline.

Here are several full data migration approaches of varying complexity that have been designed:

1) Simple migration approach

2) Vertical migration by business

3) Horizontal migration by warehouse layer

4) Hybrid vertical-horizontal migration approach

2. Incremental data migration

Incremental data typically originates from upstream sources. The most common method is the timestamp-based approach, where incremental data is identified by timestamps, synchronizing iteratively the data modified since the last timestamp to achieve incremental synchronization.

Timestamp-based approach (applicable to various databases): This is a snapshot-based change data capture method. A timestamp column is added to the source table, and whenever the system updates or modifies table data, the value of the timestamp column is also updated. During incremental data extraction, the system compares the system time with the timestamp column to decide which data to extract.

Advantages: Similar to the trigger-based approach, the timestamp-based method delivers good performance. ETL system design is clear, source data extraction is relatively straightforward, and incremental data loading can be achieved.

Disadvantages: Timestamp maintenance must be handled by the business system, making it highly invasive to the business system (adding an extra timestamp column) and requiring certain modifications to the table model design in the business system.

V. The final step is data accuracy and data reconciliation.

The ultimate goal of data reconciliation is to verify the correctness of migration scripts and programs. The method employed is to compare the processing results of the same data between the original system and the target system, and infer whether the migration process was correct based on result consistency.

Data accuracy does not mean identical results. It means that the compared data results fall within an acceptable margin of error. In a migration project, it is sometimes impossible to achieve 100% identical computation results. Possible reasons include: different business times for extracting front-end data, different rounding and truncation rules among databases, inconsistent ordering of rows with identical sort key values across databases, and different mechanisms for handling a small number of dirty records in loading files.

1. Common reconciliation approach

Top-down reconciliation order: If upper-level metrics pass reconciliation, lower-level inconsistencies can be temporarily deferred.

Bottom-up troubleshooting order: For inconsistent upper-level metrics, use a data lineage diagram to start troubleshooting from the bottom layer.

2. Common data reconciliation methods

Aggregate reconciliation: Comparing count, sum, and avg values.

Dimensional reconciliation: Applying GROUP BY to various dimensions in a table, then checking count, sum, and avg of a key metric for each dimension.

Error comparison: When an inconsistency is found in a table, pull both the original and migrated tables into the same environment for detailed analysis, such as performing a MINUS operation, to identify differing rows and further analyze them.

Reconciliation report: The reconciliation report is the output of the reconciliation work. It documents and analyzes data inconsistency issues, traces back previously resolved problems, and builds a knowledge base of common data reconciliation scenarios.

General Data Technology Co., Ltd. (GBASE), headquartered in Tianjin, China, was founded in 2004 and currently has 650 employees. Since its inception, the company has remained committed to independent innovation, with independently developed and fully controlled core technologies and underlying code across its products. GBASE has built a full-stack data product portfolio and service solutions covering the entire data management lifecycle, including analytical, transactional, distributed transactional, and cloud-native data warehouse databases. Today, GBASE is one of the few independent database vendors in China that focuses on database R&D and has achieved large-scale deployment in the financial and telecommunications industries. Its self-developed GBase series database products and services are available across all 32 provincial-level administrative regions in China, serving more than 10,000 customers in over 100 industries—including finance, telecommunications, government, energy, transportation, and national defense. With over 30,000 deployed nodes and managing a total data volume exceeding 300 petabytes, GBASE has also expanded its reach to more than 30 countries and regions, including the United States, Brazil, and Mexico.

GBase 8a is a high-performance distributed logical data warehouse independently developed by GBase (General Data Technology Co., Ltd.) for massive data query and analysis workloads. It addresses the growing data storage, management, and processing demands of data-intensive industries — including data querying, statistics, analytics, mining, and backup. The product serves as the database backbone for data warehouse systems, BI platforms, and decision support systems. GBase 8a is primarily deployed in sectors such as finance, telecommunications, government and enterprise, energy, and security, where vast amounts of business data are generated.