DTC 2023 Review | A Multi-Write Multi-Read Database Cluster Solution Based on Data Replication Technology
At the recently concluded 2023 DTC Data Technology Carnival China Database Innovation Forum, Guo Zhuo, product manager of GBase 8s at GBASE (General Data Technology), shared GBase 8s "A Multi-Write Multi-Read Database Cluster Solution Based on Data Replication Technology". Today, let's review this presentation.
The storage and computation of massive data is a critical challenge facing the industry today, and many solutions have been proposed. This time, GBase 8s introduces a "centralized high-availability technology combined with data replication technology" to explore a comprehensive cluster solution supporting multi-read and multi-write that breaks through the limitations of traditional centralized clusters.
Centralized Database
Centralized computing, centralized storage
Adheres to ACID, emphasizes transaction processing performance and strong data consistency
Suitable for transaction-intensive scenarios and core industry transaction systems
Distributed Database
Distributed computing, distributed storage
Adheres to CAP, emphasizes availability and scalability
Suitable for data-intensive scenarios and big data application domains
Compared with centralized databases, distributed databases have inherent advantages in availability. However, centralized databases are primarily used in core transaction scenarios across industries. Therefore, the ability to break through traditional cluster technology limitations while retaining benefits such as strong consistency is a key capability that centralized databases must focus on.
GBase 8s's data replication technology, GBase 8s Replication (GR), provides exactly the boost needed to address this issue.
Data replication creates multiple copies of data at one or more locations to achieve data sharing.
GBase 8s Replication (GR) is a log-based asynchronous tool used to replicate data between GBase 8s database servers. On the source server, GR reads the logical log, stores transactions, and reliably transmits each transaction as replication data to the target server.
GR data replication consists of three stages: data capture, data transmission, and applying replicated data.
1. The client application executes a transaction in the database that defines replication.
2. The transaction is written to the logical log.
3. The log capture component reads the logical log and passes logical records to the grouping component.
4. The grouping component calculates the logical logs that need to be replicated and groups them into messages describing the operations of the original transaction.
5. The grouping component adds the message to the send queue. In certain situations, the send queue temporarily stores the message to disk.
6. The send queue transmits the replication message to the target server over the GR network.
7. The replication message is added to the receive queue of the target server.
8. The data synchronization component applies the transaction to the target database. If necessary, the data synchronization component also performs conflict resolution.
9. A message indicating that the message has been successfully applied is placed in the acknowledgment queue.
10. The acknowledgment message is sent back to the source server.
Advantages of Data Replication Technology
Real-world Deployment Cases of Multi-Write Multi-Read
Smart Network of a Telecom Operator in Shanxi Province
This project went live in 2017, using GBase 8s replication cluster technology for the IVPN service of a telecom operator in Shanxi province, with a 4-node GR cluster.
Customer Value
1. Enabled real-time data replication;
2. Average SQL response time was less than 5ms, far lower than that of traditional databases;
3. The 4 nodes achieve load balancing, enabling multi-node writes and multi-node queries.
A Supermarket Group in Beijing
This project went live in 2017, used for the group's corporate business management and store settlement operations. In 2018, a 68-node GR replication cluster was deployed, enabling real-time exchange and synchronization of data between store and headquarters.
Customer Value
1. The GR replication cluster included up to 68 nodes;
2. With a network bandwidth of 2 Mbps, the cluster synchronization had low bandwidth requirements;
3. Databases at both stores and headquarters were both writable and readable, with real-time data synchronization;
4. If a network anomaly occurred, after network recovery the backlog data would automatically synchronize to the peer;
5. If the host or database encountered an anomaly, after the failure was resolved, data on the peer would also be automatically synchronized.
A Substation Monitoring System
The business system is a substation monitoring system. The equipment is located in remote areas for long periods, the server operating environment is poor, and the maintenance staff have relatively weak IT skills. The user wanted the database to have excellent self-maintenance capabilities.
Customer Value
1. Business system switchover during a failure can be completed within 1 second;
2. After failure recovery, data in the database can automatically catch up without manual intervention;
3. The database remains in a state of automatic recovery for a long time, requiring no dedicated DBA; customers only need to maintain the database servers.