Teradata to GBase 8a Migration Solution

Published on 2023-02-22

Recently, Teradata announced its withdrawal from the Chinese market, which is bound to create opportunities for the domestic data warehouse market. As a leading enterprise in China's data warehouse sector, GBASE General Data Technology Co., Ltd. has independently developed GBase 8a MPP (GBase UP LDW), a distributed logical data warehouse that has completed replacement and migration for over 100 users from foreign databases such as Teradata. Through numerous project PoCs and subsequent signed implementations, GBase 8a has accumulated extensive experience in replacing Teradata, forming a complete migration implementation plan that can be quickly replicated and promoted to achieve highly accurate migration from Teradata to GBase 8a.
 

Based on GBase 8a migration experience, the migration plan is elaborated from top to bottom as follows:

I. Post-Migration Implementation Effects

  • Two clusters, A (7 nodes) and B (14 nodes), were built to carry the main business

  • Total data volume across SGA, ODS, F, DW, and DM layers exceeds 200 TB

  • Completed migration of over 1,000 programs

  • Daily incremental basic data is completed before 7:00 AM each day

  • Monthly reports are published by the 3rd of each month

  • Overall performance improved by 2 to 10 times


 

II. Project Migration Process
 

The migration process serves as the guiding framework for the entire migration effort, effectively organizing the orderly execution of migration activities. The main process steps are:

III. Pre-migration Research and Solution Design

This phase mainly involves pre-migration assessment and the formulation of a migration solution to guide migration execution. It includes the following points:

1. Pre-migration Research Content and Key Points

The focus of research is to understand customer requirements and current pain points, so that targeted improvement solutions can be provided after migration!

Migration Implementation Research

  • Migration implementation research is a critical step in the migration project and cannot be omitted

  • Depending on the progress of the pre-sales phase, migration implementation research may be conducted before or after contract signing

Objectives of Migration Implementation Research

  • Understand the scope of the project migration

  • Assess migration workload, implementation period, number of personnel, and skill requirements

  • Evaluate technical difficulties and major project risks

Content of Migration Implementation Research

  • Current state of the system to be migrated: architecture, configuration, upstream/downstream dependencies, logical warehouse design

  • System operation status: application scenarios, job types, load conditions, overall ETL processing flow, number of jobs, concurrency, processing time

  • System metric requirements: requirements for different types of jobs, such as batch processing time windows, response time for ad-hoc queries, concurrency requirements, etc.

  • Interface conditions: upstream data ingestion methods, downstream data delivery methods, third-party tool support requirements

2. Migration Solution Design

The migration solution is a project implementation plan formulated based on the migration research results and tailored to the customer’s current needs and pain points. It mainly includes:

  • Hardware configuration and deployment plan for the migration system

  • Targeted solutions addressing customer needs and pain points

  • Steps and tasks for migration implementation

  • Work schedule

3. Hardware Resource Assessment

The hardware configuration for the GBase 8a cluster replacing TD is evaluated using a backward approach. First, obtain detailed configuration information from TD, then work backward to determine the configuration of a single GBase 8a server and the number of servers required based on TD’s total CPU cores, memory capacity, and disk capacity. In principle, the total CPU cores, memory capacity, disk capacity, and number of disks across all GBase 8a cluster servers should be no less than twice that of the TD appliance.

IV. Data and Business Migration

Next, it is necessary to plan in advance for technical challenges such as full and incremental data migration, ETL catch-up, and data comparison before and after migration, as well as the data flow migration process.

1. Full Data Migration Scheme Assessment and Key Points

For full data migration, the migration time window largely determines whether it will be a one-time or batch migration. The following factors must be considered:

  • Amount of data to be migrated from the source database [actual measurement is needed to assess the compression ratio of data in the source database]

  • Allowed business downtime window [the source database must be in read-only mode with light load during this process, often a hard constraint]

  • Data export performance from the source database [actual measurement needed to determine capability]

  • Number of file servers for loading, I/O performance, and network bandwidth to the 8a cluster [environmental constraints]

  • Loading performance of 8a cluster nodes [consider both I/O performance and network bandwidth]

  • Type of incremental business: append only, IDU, zipper table, full refresh each time [determine how incremental catch-up is performed and whether batch migration is supported]

  • Warehouse design support for layering and data processing idempotence [determines whether migration can be split vertically by business or horizontally by layer]

The migration time window should be estimated using the following formula:

  • Total migration time = TD export time + GBase 8a loading time

  • TD export time = TD stored data volume (in GB) / TD parallel export performance (GB/hour)

  • GBase 8a loading time = TD exported data volume (in GB) / GBase 8a parallel loading performance (GB/hour)

  • GBase 8a parallel loading performance = number of loader machines * 1000 MB/s * 1/2 * 3600

Points to note for data migration:

  • TD stored data volume must be converted to the data volume of exported out-of-database flat files, so compression ratio must be assessed

  • TD parallel export performance needs to be measured, taking into account the load on TD during export

  • GBase 8a parallel loading performance needs to be measured; reference formula 4 when estimating

  • When evaluating 10 Gigabit bandwidth utilization, allow some redundancy; typically a 1/2 factor is used in calculations

  • The overall time window is assessed assuming serial execution of export and loading, rather than pipelined, to ensure a certain level of redundancy for unexpected situations. However, the actual migration execution can be designed as a pipeline.

The following three full data migration schemes of different complexity levels are designed:

1) Simple Migration Scheme

2) Vertical Migration Scheme by Business

3) Horizontal Migration Scheme by Warehouse Layer

4) Hybrid Vertical-Horizontal Migration Scheme

2. Incremental Data Migration

Incremental data usually originates from upstream sources, most commonly identified by a timestamp. By using the timestamp to differentiate incremental data, each synchronization only processes data with timestamps updated since the last sync, achieving incremental synchronization.

Timestamp-Based Approach (applicable to various databases): This is a change data capture method based on snapshot comparison. A timestamp field is added to the source table, and whenever table data is updated, the timestamp field is also modified. When extracting incremental data, the system compares the system time with the timestamp field value to decide which data to extract.

Advantages: Like the trigger method, the timestamp approach provides good performance, clear ETL system design, and relatively straightforward source data extraction, enabling incremental data loading.

Disadvantages: Timestamp maintenance must be handled by the business system, which introduces significant intrusiveness (adding an extra timestamp field) and may require modifications to the data model design of the business system tables.

V. Data Auditing

The ultimate goal of data auditing is to verify the correctness of migrated scripts and programs. The method involves comparing the processing results of the same data between the original system and the migrated system; consistent results infer the correctness of the migration process.

Data accuracy does not mean exactly identical results; it means the compared data results fall within an acceptable expected error range. It is sometimes impossible to achieve 100% identical computation results in a migration project. Possible reasons include: different business times for extracting front-end data, differences in rounding and truncation rules between databases, inconsistent sorting of rows with the same sort key value, and different handling mechanisms for a small amount of dirty data in loaded files.

1. Common Auditing Approaches

Top-Down Auditing Sequence: If upper-level metrics pass verification, inconsistencies at lower levels can be temporarily disregarded.

Bottom-Up Troubleshooting Sequence: For inconsistent upper-level metrics, use lineage diagrams to start investigation from the bottom layer.

2. Common Data Auditing Methods

Aggregate Auditing: Compare COUNT, SUM, AVG values.

Dimensional Auditing: Perform GROUP BY on various dimensions within a table, then verify COUNT, SUM, AVG of a metric for each dimension.

Discrepancy Comparison: When inconsistent tables are found, bring the compared tables into a single environment for detailed analysis, such as performing a MINUS operation to identify differing records for further analysis.

Audit Report: The audit report is the deliverable of the auditing work. It records and analyzes data consistency issues, traces previously resolved issues, and builds a knowledge base of common data auditing problems.

About GBASE

Tianjin General Data Technology Co., Ltd. (abbreviated as GBASE) was founded in 2004 and currently has 650 employees. Since its establishment, the company has adhered to independent innovation with fully self-controllable core technologies and underlying code. It has built a data product and service solution portfolio covering the full data management lifecycle, including analytical, transactional, distributed transactional, and cloud-native data warehouse technologies. Currently, GBASE has become one of the few independent database service providers in China focused on database product R&D with large-scale applications in the financial and telecommunications industries. Its independently developed GBase series database products and services cover 32 provincial administrative regions nationwide, serving over tens of thousands of users in more than a hundred industries such as finance, telecommunications, government, energy, transportation, and national defense, with over 30,000 nodes established and a total managed data volume exceeding 300 PB. Its products are also exported to over thirty countries and regions, including the United States, Brazil, and Mexico.

About GBase 8a

GBase 8a is a high-performance, domestically developed distributed logical data warehouse independently developed by GBASE, designed for massive data query and analysis applications. It meets the growing data storage, management, and processing needs—such as data querying, statistics, analysis, mining, and backup—across various data-intensive industries. It can serve as the underlying database for data warehouse systems, BI systems, and decision support systems. The product is primarily used in industries with massive business data, such as finance, telecommunications, government and enterprises, energy, and security.