GBase 8a Cluster Active-Active Disaster Recovery Solution

The Necessity of Disaster Recovery for MPP Analytical Clusters
 

Disaster Recovery Requirements

The disaster recovery of MPP analytical clusters differs significantly from that of online OLTP transaction databases in both technical implementation and disaster recovery level requirements. For example, transaction databases have comprehensive WAL (write ahead logging) transaction logs, which enable backups, active-active replication, and remote disaster recovery through transaction logs. MPP analytical databases, in pursuit of high throughput performance, abandon transaction log support and therefore cannot adopt disaster recovery solutions based on transaction logs.

With the release of the 14th Five-Year Plan for Big Data Industry Development, big data is increasingly becoming an important pillar of the digital economy. Comprehensive disaster recovery for MPP clusters that host data warehouses and are responsible for high-value-density big data storage and analysis has become especially critical.

In terms of industry requirements for database disaster recovery, the primary reference is the national standard GB/T 20988-2007, "Information security technology—Disaster recovery specifications for information systems", which defines the disaster recovery levels and requirements for industries and enterprises.

Disaster Recovery Levels

According to GB/T 20988-2007 "Information security technology—Disaster recovery specifications for information systems", enterprise information system disaster backup technology levels are classified into the following six levels:

Level 6  Zero data loss and remote cluster support

Level 5  Real-time data transmission and complete equipment support

Level 4  Electronic transmission and complete equipment support

Level 3  Electronic transmission and partial equipment support

Level 2  Alternate site support

Level 1  Basic support

Each level corresponds to different Recovery Time Objective (RTO) and Recovery Point Objective (RPO) requirements, leading to different disaster recovery solutions. For example, in the industry standard "Management specification for disaster recovery of banking information systems", the corresponding requirements for disaster recovery levels with RTO and RPO are as follows:

The disaster recovery of MPP cluster databases also follows the above requirements. For example, the disaster recovery requirements for an MPP cluster database carrying important data and services in a certain industry are as follows:

Active-Active Clusters

An active-active system is a higher level of disaster recovery. For disaster recovery levels 4 and above, active-active systems can be built to meet data security and business continuity requirements.

MPP analytical databases do not support transaction logs and cannot build active-active clusters based on mature transaction log replication solutions. Therefore, the only available active-active cluster solution for MPP analytical databases is based on data shard redundant storage (data file replication). Real-time synchronization of multiple data replicas within the same cluster is only suitable for deployment within the same data center and cannot meet the disaster recovery requirements across data centers or cities. To achieve remote disaster recovery, data file replication between two different MPP clusters is necessary.

GBase 8a Asynchronous Active-Active Cluster

The GBase 8a asynchronous active-active cluster adopts a master-slave form. The primary cluster is writable and handles data processing such as statistical analysis. After completing data processing, important result data is synchronized to the backup cluster. The standby cluster can share the external query service workload of the primary cluster, reducing resource contention pressure from read and write operations on the primary cluster.

The data synchronization process is implemented based on the inter-cluster synchronization tool (RSyncTools). RSyncTools provides core data synchronization functions and exception handling, with low coupling between functional points, enabling applications to flexibly call interfaces to achieve intra-city or remote disaster recovery.

The schematic diagram of asynchronous active-active synchronization is as follows:

1. Set up two GBase 8a clusters of the same scale and data distribution in two data centers located in the same city or different cities.

2. Business applications connect to the primary cluster (the cluster in Data Center 1 in the diagram above) for batch job execution.

3. Inter-cluster data synchronization is triggered through script-based scheduled tasks or by application schedulers.

The diagram above shows that an application scheduler triggers the synchronization of result tables immediately after completing a job execution.

For script-based scheduled tasks, for example, after daily batch processing is completed at 5 a.m., a scheduled task triggers the data synchronization job.

4. Data synchronization is completed by executing the RSyncTools tool. RSyncTools performs synchronization at the table level and provides a synchronization function call interface. The main parameters passed include the database and table names to be synchronized, the IP address of the primary cluster management node, and the IP address of the standby cluster management node. Based on the database and table names, RSyncTools connects to the primary and standby clusters to query synchronization table metadata information. If metadata differences are found, it performs differential synchronization of data files. The synchronization process uses parallel node-to-node mode between the primary and standby clusters, and the synchronized data files are compressed data storage files.

The inter-cluster synchronization tool has the following characteristics:

• It provides table-level incremental data synchronization capability and supports user-defined synchronization tasks, allowing priority to be assigned to synchronize important table data first.
• It is atomic, meaning that if a table encounters a failure during data synchronization, the inter-cluster synchronization tool can automatically roll back, ensuring data consistency between the primary and standby clusters.
• It has incremental data capture capability, automatically identifying and synchronizing changed data.
• It features efficient data synchronization, synchronizing only primary replica data; secondary replica data is synchronized internally within the standby cluster. Additionally, it directly uses compressed binary files during synchronization, which effectively improves synchronization efficiency.

5. The standby cluster in Data Center 2 can be used to handle read-only workloads, achieving read-write separation—the primary cluster handles batch statistical analysis and computation, while the standby cluster handles query responses.

GBase 8a Real-Time Active-Active Cluster

GBase 8a V9 supports virtual clusters. Within a virtual cluster, a mirror relationship can be established between two sub-clusters of the same scale and data distribution policy, which is called a mirror cluster. As the name implies, the tables and data in the two mirror clusters are consistent.

The mirror cluster enables real-time active-active for the computing cluster, supporting real-time active-active scenarios within the same city. The active-active schematic diagram is as follows:

1. Establish a mirror relationship between two sub-clusters of the same scale and configuration in the GBase 8a V9 virtual cluster. The mirror relationship can be created at the table level or the database level.

2. The two sub-clusters in a mirror relationship within the virtual cluster can be deployed within the same data center or in different data centers within the same city, with certain requirements for network quality and bandwidth between the two sub-clusters. When the two sub-clusters are located in different data centers, it forms an intra-city remote disaster recovery setup. In an intra-city remote deployment, one coordinator node can be deployed in the backup data center to achieve data disaster recovery for the management cluster.

3. Business applications directly connect to one of the sub-clusters for operations, issuing DDL, DML, DQL, etc. For DDL, it is simultaneously issued to both mirror sub-clusters for execution. For DML and DQL operations, they are directly issued to the default sub-cluster of the current application for execution. The DML execution results are transferred to the other sub-cluster via a chain forwarding method. Data modification operations in the mirror cluster must be uniformly committed across both sub-clusters before returning the result to the user.

4. If a computing cluster fails, business can be switched by modifying the default sub-cluster for the business. In the diagram above, VC1 and VC2 are mirror clusters. The business executes directly on VC1 by default. When VC1 experiences an overall failure, changing the business’s default connection to VC2 allows the business to execute on VC2.

5. If all nodes of the management cluster in the primary data center fail, the configuration of the cluster management nodes must be manually modified to set the management node in the backup data center as the sole management node. After that, business can issue SQL tasks through the management node in the backup data center.

GBase 8a Two-Site Three-Center Disaster Recovery Solution

For scenarios with higher disaster recovery requirements, the asynchronous active-active and real-time active-active capabilities of GBase 8a can be combined with additional data backup protection. For example, in an environment where an asynchronous active-active cluster has been established, while synchronizing between the primary and standby clusters daily, periodic backups can be performed on the primary cluster, such as a full backup once a week and incremental backups daily. Furthermore, combining the asynchronous active-active and real-time active-active features of GBase 8a can form a two-site three-center disaster recovery solution, as follows:

The primary data center uses virtual cluster mirroring technology to achieve strongly consistent real-time active-active between two virtual sub-clusters within the same data center.

The standby data center and the primary data center achieve near-real-time active-active disaster recovery through asynchronous data synchronization using the inter-cluster synchronization tool.

Differences Between GBase 8a Asynchronous and Real-Time Active-Active Clusters

The GBase 8a asynchronous active-active and real-time active-active clusters have the following differences: