【GBASE Case Study】Seamless Integration of Historical and Incremental Data – Real-time Sync System GBase RTSync

This project adopted the GBase 8a MPP massive parallel distributed database cluster, building a 14-node cluster (5 cluster nodes + 9 data nodes) to store integrated detailed data and lightly aggregated data.

● Near real-time synchronization, accurate and efficient: GBase RTSync enables near real-time data synchronization with incremental extraction and incremental loading, achieving second-level synchronization efficiency and ensuring data consistency;

● High-speed initialization for massive data: Achieves high-speed data loading while delivering high compression ratios for ingestion to boost performance, massive storage capacity, and integration of multi-source business data, with the ability to dynamically scale out online as needed;

● Full initialization with zero downtime: Adding new business tables to the source database of State Grid’s unified full-service data center requires no downtime or interruption to existing operations. GBase RTSync seamlessly combines historical and incremental data, ensuring efficient loading into the integrated database while the source system remains fully operational.

The GBase RTSync and GBase 8a MPP combination: Set up GBase RTSync front-end servers to connect the front-end staging historical databases with the detailed data layer and lightly aggregated layer of the back-end enterprise-wide data model. GBase RTSync handles historical data import and zero-downtime incremental data ingestion from the source, supporting peak incremental sync of up to 1.5 TB/day of archived data from the source database. This deeply optimized and tightly integrated solution of GBase RTSync and GBase 8a MPP cluster databases meets the functional and performance requirements for massive data storage, large-scale parallel computing, and near-real-time incremental data flow.

General Data Technology's Real-Time Synchronization System (GBase RTSync) is an independently developed real-time incremental data synchronization product for both heterogeneous and homogeneous databases. It delivers real-time performance, consistency, accuracy, easy scalability, and seamless integration. GBase RTSync is designed for scenarios where OLTP and OLAP databases jointly support data management and analytics in application systems, synchronizing data from OLTP to OLAP databases in real time. This provides the foundation for real-time analytics, solves incremental data synchronization challenges, and significantly improves the efficiency and timeliness of data analysis for data warehouses, BI systems, and decision support systems.

GBase RTSync Core Components

● Capture: Acquires and parses incremental data logical logs from the data source, encapsulates them based on a specific protocol, and sends them to the message queue;

● Delivery: Retrieves protocol data from the message queue, then organizes and optimizes the data based on the target database type before writing it into the target database;

● MQ: Supports Kafka and RabbitMQ, both of which ensure high availability;

● Management Module: Starts, stops, and monitors the Capture and Delivery components, and validates the synchronization configuration between source and target.

GBase RTSync Key Technologies

● Minimal impact on the source database with incremental synchronization based on logical log parsing.

● Online takeover, reducing unplanned downtime of the source database.

● Distributed architecture supports horizontal scaling.

● Compatible with mainstream message middleware such as Kafka and RabbitMQ.

● High availability mode support.

A provincial subsidiary of State Grid needed to focus on personnel, finance, materials, customers, assets, projects, and more, covering 10 business subject area models for data integration. The integration task involved over 3,000 physical models and over 3,000 source tables. Despite the high complexity of numerous models, tables, and relationships, the project required fast and accurate integration with near-real-time incremental data synchronization to the data warehouse.