How to use the GVR visual tool to achieve active-active cluster disaster recovery for GBase databases?
GBase 8a is one of the few analytical MPP databases in China with two-site three-center disaster recovery capabilities. Leveraging its RsyncTool synchronization tool, it enables near-real-time data synchronization between clusters in the same city or across regions, and has been deployed in many large banks, insurance companies, and other financial institutions.This article introduces a GBase 8a cluster active-active disaster recovery solution based on the visual cluster active-active synchronization tool GVR (GBase Visual RsyncTool). It covers the resource requirements, applicable scenarios, and a brief introduction to the GVR tool features.
Solution Overview
GBase has introduced an enhanced version of the RsyncTool synchronization tool, GVR (GBase Visual RsyncTool), which provides GUI-based operations for tool configuration management, synchronization task orchestration, and synchronization task monitoring. This greatly enhances the manageability, maintainability, and usability of inter-cluster data synchronization and disaster recovery. At the same time, optimizations have been made to improve incremental synchronization performance and reduce table lock granularity during synchronization. By comparing the metadata information between the primary and standby clusters in parallel, tables without data changes are excluded from synchronization tasks, thereby improving synchronization performance. The lock granularity on the primary cluster for tables being synchronized has been reduced: during the metadata snapshot saving phase, the table is read-only, while during the data synchronization phase, DML operations are supported, significantly reducing the impact of synchronization on business operations.
Active-Active Synchronization Solution Overview
The GBase 8a cluster active-active disaster recovery solution is implemented through the visual data synchronization tool GVR (GBase Visual RsyncTool). GVR supports table-level incremental data synchronization between two clusters deployed in the same city or across different regions, forming an active-active disaster recovery setup with mutual data backup between two independent clusters.
It is recommended to use one cluster as the primary cluster, which provides the data write entry point (and can also handle query tasks), while the other cluster acts as a read-only standby cluster.
Under normal conditions, the primary cluster handles write operations and uses the synchronization tool to synchronize incremental data from tables in the primary cluster to the standby cluster at the table granularity. Both clusters can serve queries externally.
In the event of a complete failure of the primary cluster, services can be switched to the standby cluster, which then takes over write operations. After the primary cluster is restored, the synchronization tool can be used to perform a reverse synchronization at the table granularity, bringing the incremental data generated after the switch back to the primary cluster until the data is consistent. Once data is caught up, services can either switch back to the primary cluster or, after a primary-standby switchover, let the former standby cluster assume the primary role while the original primary cluster is downgraded to standby.
Resource Requirements
3.1 Primary-Standby Cluster Scale Requirements for Active-Active DR
1) The logical architecture between the primary and standby clusters must be identical:
The active-active solution requires the same number of primary shards in both clusters. For example, if the primary cluster has 10 gnode computing nodes (or gnode instances in multi-instance deployments) with one primary shard per node, the primary shard count is 10. The disaster recovery standby cluster must also have 10 primary shards, matching the primary cluster.
To ensure that the standby cluster can provide equivalent computing performance after a primary-standby switchover, it is recommended to build the standby cluster with the same number of nodes and identical server configurations as the primary cluster.
2) The data distribution rules between the primary and standby clusters must be identical:
The active-active solution requires that the hash distribution rules be the same across the primary and standby clusters, meaning the hash bucket rules inside the gbase.nodedatamap system table must be consistent between the two clusters.
3.2 Primary-Standby Cluster Scale Requirements for Active-Active DR
1) Network connectivity requirements between nodes:
The active-active synchronization solution involves three components: the primary cluster, the standby cluster, and the GVR synchronization tool. The network between these three components must be open, meaning that all node servers in both clusters (management nodes gcluster, computing nodes gnode) and the server hosting the GVR synchronization tool must have network connectivity with each other.
The clusters support a dual-plane network deployment, where the internal cluster communication uses one network plane (private network), and external services use another network plane (business network). For geographically separated active-active clusters, the business networks of the primary and standby clusters can be connected without connecting their internal private networks. GVR supports IP Mapping to enable data synchronization based on the business network.
Regarding network ports, the database service access ports (default 5258, 5050) and the data synchronization service ports (default 5288) must be opened between all nodes of the primary and standby clusters, as well as between the two clusters and the GVR synchronization tool.
2) Bandwidth requirements between nodes:
Cluster active-active synchronization uses a node-to-node (shard-to-shard) data synchronization approach, ensuring data consistency between shard n1 of a table in the primary cluster and shard n1 of the corresponding table in the standby cluster.
During synchronization, parallel synchronization between corresponding nodes of the primary and standby clusters is employed, and multiple tables can be synchronized in parallel. The network bandwidth requirement between the primary and standby clusters should be assessed based on the disaster recovery RTO requirement and the size of incremental data. That is, the aggregate bandwidth between all corresponding nodes of the primary and standby clusters > daily incremental data volume / synchronization window time.
3.3 Resource Requirements for GVR Synchronization Tool Deployment
It is recommended to deploy the GVR tool on a dedicated server with at least 8 cores, 32 GB RAM, and 1 TB storage. This server will be used for tool deployment, storing synchronization logs, and saving synchronization configuration data.
Applicable Scenarios
4.1 T+1 Synchronization Scenario
Solution Description
T+1 asynchronous active-active synchronization is suitable for scenarios with moderate timeliness requirements and a relatively long time window for data synchronization, such as batch processing jobs. Daily batch processing runs overnight, and once completed, the cluster data is in a static state with no further changes. At this point, cluster active-active synchronization can be initiated to perform incremental data synchronization. During data synchronization, tables being synchronized on the primary cluster support queries and Appendonly-type DML operations (insert select, insert values, load, delete, and update/merge operations when fast_update mode is enabled), but do not support DDL or update/merge operations without fast_update. Tables being synchronized on the standby cluster cannot provide external services.
The T+1 asynchronous synchronization approach has low coupling with business operations, as different time windows avoid slice consistency issues across tables during data synchronization. Architecture:
① Batch processing jobs in application processing run according to the original logic on the primary cluster.
② Tables modified by the application processing jobs are added to the list of pending synchronization tasks.
③ After batch jobs complete, cluster active-active synchronization is triggered, or it can be scheduled based on a time split—for example, if batch processing finishes at 8:00 AM, start synchronization at 9:00 AM.
④ After all pending tables are synchronized, the data in the standby cluster becomes consistent with the primary cluster, completing the active-active synchronization. At this point, the standby cluster can provide external data query services.
Users can also choose to synchronize all tables within a database or provide a larger list of tables to be synchronized, pre-defining the databases or tables to synchronize, thus eliminating step ②. During active-active synchronization, the GVR synchronization tool automatically determines whether tables are consistent between the primary and standby clusters, and skips tables that have not been modified during the current batch cycle.
Incremental Synchronization Time Estimation
T+1 asynchronous active-active synchronization requires completing the synchronization of incremental data within a fixed time window, which can take a considerable amount of time. The required time can be estimated based on the changed data volume divided by network transmission bandwidth. Because the database cluster active-active synchronization uses shard-to-shard parallel network transfers and supports parallel synchronization of multiple tables, network bandwidth can be fully utilized.
For scenarios with insufficient synchronization windows, data can be synchronized in batches according to business characteristics. For example, if the batch window is from 0:00 to 8:00, with daily incremental loading from 0:00 to 2:00, common data processing from 2:00 to 3:00, summary layer data processing from 3:00 to 5:00, and indicator computation from 5:00 to 8:00, synchronization batches can be organized as follows:
At 3:00, synchronize the daily incremental data tables in the ODS layer.
At 5:00, synchronize the common processing data tables.
At 9:00, synchronize the remaining unsynchronized tables.
DR Indicator Evaluation:
Under the T+1 asynchronous active-active synchronization scheme, the DR indicators depend on the time required to re-run the daily batch processing.
Since the raw data of an OLAP system comes from external systems and generally has idempotent processing, data recovery can be achieved by reloading and re-running batch operations, resulting in an RPO of 0. For data that cannot be re-computed through re-running, the worst-case RPO is from the time of the last synchronization completion to the time of the disaster, i.e., RPO < 24 hours. For such business data, increasing the data synchronization frequency can reduce the RPO—for example, synchronizing this type of data every hour will shorten the RPO to within 1 hour.
For the RTO recovery target, the recovery time is the time from the completion of the last data synchronization plus the time required to re-run the batch processing. For instance, if re-running the daily batch takes 8 hours, then RTO < 8 hours.
4.2 Job-Level Near-Real-Time Synchronization Scenario
Solution Description
Job-level near-real-time synchronization integrates table data synchronization tasks into the application batch processing job scheduler. After a batch job task completes, an active-active synchronization for the tables involved in that job is triggered. GVR provides a user synchronization task scheduling interface; a synchronization task can be added as the final step of a batch job. The job is only considered complete after the synchronization task finishes successfully.
In this scheme, active-active synchronization has a certain level of coupling with business operations. By integrating synchronization with job scheduling, data between the primary and standby clusters is synchronized in a timely manner, reducing RPO and RTO.
Architecture diagram:
① Modify business program scripts to add an active-active synchronization task schedule at the end of each batch job.
② Pre-set or generate a list of tables to synchronize when the job executes. After the job completes, data synchronization is performed based on the pre-defined list or the list of tables that had data changes during this job execution.
③ The batch job is not formally completed until the table synchronization tasks are finished. If a data synchronization task fails, the job scheduler should support retrying the synchronization task without re-running the entire job.
④ To reduce the impact of data synchronization on batch job performance, the final synchronization task can be executed asynchronously. That is, after the batch job finishes, subsequent batch tasks can proceed while data synchronization runs asynchronously, with the synchronization result returned later.
Incremental Data Synchronization Time Estimation
Job-level near-real-time data synchronization distributes the data synchronization tasks between primary and standby clusters into individual job tasks. The amount of data per single job synchronization is relatively small, resulting in shorter synchronization times.
Synchronization performance can still be estimated based on the synchronized data volume and network bandwidth. For multiple serial batch job chains with high performance requirements, table synchronization tasks can be run asynchronously to minimize impact on the job chain performance.
DR Indicator Evaluation
Similar to T+1 asynchronous active-active synchronization, for businesses that can regenerate data through re-running batch processing, RPO = 0. For data that cannot be recalculated via batch reruns, RPO = active-active switchover time + job batch runtime, which can be reduced to the minute level.
The RTO time is close to the time needed to re-run the failed job. For example, if a job that normally takes 1 hour is running when the primary cluster experiences a catastrophic failure requiring a switch to the standby cluster, RTO < 1 hour. Only the most recent job needs to be re-executed on the standby cluster based on the table state from the last synchronization.
4.3 Impact on Business and Functional Limitations of Active-Active Synchronization
1) Impact on Business: Active-active synchronization operates at the table level, so the business impact is at the granularity of tables being synchronized.
Primary cluster: Tables being synchronized support reads and Appendonly-type DML operations (insert select, insert values, load, delete, and update/merge operations with fast_update mode enabled). DDL and update/merge operations without fast_update mode are not supported.
Standby cluster: Tables being synchronized do not support reads, DML, or DDL operations.
2) Synchronization Scope:
The GVR tool synchronizes DML operations on synchronized tables; DDL operations on synchronized tables are not supported.
GVR supports metadata synchronization between primary and standby clusters, including synchronization of tables, stored procedures, views, and user-defined functions.
Introduction to GVR Active-Active Synchronization Tool
GVR (GBase_Visual_RsyncTool) encapsulates the underlying details of data synchronization, simplifies active-active synchronization configuration through a graphical interface, supports metadata synchronization (such as stored procedures), and provides monitoring and O&M functions for active-active synchronization.
The architecture of the GVR synchronization tool is shown above. The responsibilities of each module are as follows:
Frontend service: Visual interface, SNMP/RESTful APIs
Backend service: Receives frontend requests, handles user and permission authentication, synchronization task scheduling, etc.
Underlying tools: Responsible for synchronization execution, encapsulated internally and transparent to users
Logging system: Backend service and underlying tool modules output and record operation logs
Configuration repository: Persists active-active synchronization configuration information and records synchronization task execution status
Conclusion
GVR supports a wide range of features, including visual configuration of primary-standby cluster synchronization, metadata synchronization, dynamic progress bars showing synchronization progress, viewing historical synchronization tasks, visual editing of scheduled tasks, synchronization task alarms, data consistency verification between primary and standby clusters, data source management, synchronization task management, synchronization scheduling policy configuration, synchronization task monitoring, and operation audit trails. Click to learn more about GVR.