[Tech Outlook] Distributed Databases — The Mainstream Database for Future Industry Applications

With the advancement of cloud computing and big data technologies, traditional IT systems and applications have been profoundly disrupted. Databases, as fundamental software, are also encountering new challenges and opportunities. Going forward, database selection for information applications across industries will increasingly lean toward distributed databases. This article analyzes the demand for distributed databases in future industrial informatization based on typical business application types.

1. Distributed Databases Are Needed to Support New Technologies and Industry Application Development

A distributed database can be considered as a collection of multiple computing nodes located at different sites connected through a network, jointly forming a complete, global, logically centralized but physically distributed database. In the future, traditional single-server databases will be replaced by distributed databases in more and more scenarios, and the prospects for distributed databases will continue to broaden.

The development of distributed databases is necessary for at least the following reasons:

• Standalone mainframes are gradually being replaced by clusters of PC servers. Modern data centers increasingly adopt clusters rather than mainframes to boost compute and storage capacity, a trend that inevitably pushes databases to adapt to distributed clusters instead of mainframe infrastructure;
• Future data centers will increasingly deliver services based on cloud infrastructure. The fundamental characteristic of a cloud computing environment is the on-demand allocation of compute and storage resources, and these cloud resources reside on distributed nodes. Naturally, databases in a cloud environment also need to be adapted to a distributed paradigm;
• A single server cannot manage databases with ever-growing data volumes. Obviously, if a database system does not move beyond the confines of a single server, it is difficult to cope with continuously expanding data scales while ensuring performance. In contrast, a cross-node, horizontally scalable database can well address the compute and storage requirements of massive data;
• A single server cannot provide high-availability services for a big data platform. In the era of big data, a limited number of two or a few nodes cannot meet the demands of a future big data environment. Distributing data resources across multiple nodes not only guarantees high data availability but also accommodates massive data growth; moreover, multi-node distribution delivers obvious advantages in large-scale data processing speed.

Gartner predicts that the cloud database market will maintain a compound annual growth rate of 31% from 2018 to 2023, and that by 2021 cloud databases will account for 50% of the entire database market, a share that will reach 75% by 2023. Related surveys show that enterprises' willingness to adopt cloud has surged to 84%. As enterprises become increasingly accustomed to cloud-based IT architectures, distributed database technologies suited for cloud deployment will also become the development trend.

2. Trends of Distributed Databases in OLTP Applications

Distributed databases are being used in more and more OLTP scenarios. Especially in recent years, with the rapid growth of emerging Internet-based online applications, distributed OLTP databases have gained even wider adoption. Since these applications do not demand extremely strong consistency, many responsive, large-scale applications follow the BASE theory to enhance throughput and performance of distributed databases. Compared with the traditional mainframe-dependent model, these distributed OLTP deployments have significant advantages, at least in terms of cost and elastic scalability.

For industries that require strong consistency, especially core systems in the financial sector, distributed database adoption is also advancing. However, because these special industries impose strict consistency requirements, the deep and widespread application of distributed OLTP databases still needs further technical refinement. A key characteristic of distributed databases is the use of a multi-replica mechanism to ensure system security and reliability. Once a database spans multiple nodes, failures at the node, network, disk, or composite level introduce a series of complex problems with multi-replica consistency that are difficult to resolve. Currently, in addressing multi-replica consistency, the Paxos and Raft consensus protocols provide solid theoretical support. In particular, the Raft protocol is a consensus algorithm that is easy to understand and implement. However, the Raft algorithm also has two performance bottlenecks: first, each log write must be flushed to disk before returning success, and disk flushing is a relatively time-consuming operation; second, due to the algorithm's design, all requests are handled by the Leader, making it hard for all nodes to provide services. Therefore, in practical implementations, the Raft algorithm often requires optimization.

Of course, there are still quite a few technical challenges in distributed OLTP that need further improvement, such as the global clock service. Since the global clock service is constrained by the computing power of individual nodes, it can easily become a bottleneck for overall system scalability. Many technical difficulties in OLTP remain to be solved, but various countermeasures also exist. Currently, there are numerous open-source and self-developed distributed OLTP products both domestically and internationally. These products have already been applied in industries such as government services, retail, telecommunications, manufacturing, logistics, and meteorology, and their adoption in the financial sector is also deepening. The further maturation of distributed OLTP databases will also be essential for replacing traditional databases like Oracle with locally-developed alternatives.

The following is a classic architecture of a distributed OLTP database:

3. Trends of Distributed Databases in OLAP Applications

Distributed databases are already mature in OLAP applications, where they offer clear advantages and have become the trend. Almost all OLAP scenarios today face massive data, and it is obvious that a single server cannot cope with the rapid growth of such data. A decade ago, OLAP scenarios in China still heavily relied on foreign distributed databases like Teradata and Sybase. With the development of domestic distributed database products and their ecosystems, replacing foreign products is no longer a problem, and domestic solutions offer notable advantages in terms of cost and service.

Well-known domestic distributed OLAP products include GBase 8a by GBASE, Alibaba Cloud AnalyticDB, and Huawei GaussDB 200. Currently, domestic OLAP products have been widely adopted in data warehouses and big data platforms. Even the financial industry, which imposes particularly stringent data storage and processing requirements, has already deployed distributed OLAP databases on a large scale. A classic OLAP use case is the data warehouse at Agricultural Bank of China, where data sources encompass over 120 business systems including core banking, credit, finance, investment, and e-banking. The cluster consists of a total of 1,156 nodes, with 112*2 nodes for the main database and 8 data marts covering individual customer, corporate customer, finance, regulatory, risk, operational risk control, audit and internal control, and branch marts. The total data volume exceeds 20 PB, with raw data in a single database surpassing 5 PB. Daily incremental data reaches 4.7 TB, with 380,000 tables, and the largest table exceeds 2.5 trillion rows. More than 15,000 intra-database processing jobs run every day, and the daily data processing window is 8 hours. The deployment scale has already surpassed the historical scale of foreign products used in China.

From a technical perspective, OLAP does not require strong consistency, and the read-to-write ratio is far higher than that of OLTP, so the data scale supported by OLAP can be much larger than that of OLTP. Key technical features of OLAP databases typically include columnar storage and data sharding. Columnar storage allows reading only the required columns, skipping useless data; homogeneous data can be stored more compactly and is friendly to encoding compression. These advantages reduce I/O. Data sharding enables database scalability, fault tolerance, and read performance. At the same time, sharding makes operations more automated and horizontal scaling more flexible. Well-designed sharding techniques can also avoid I/O hot spots on some nodes while leaving others underutilized, achieving load balancing.

Beyond these common key technologies, the query and retrieval efficiency and ecosystem adaptability of OLAP are also very important. After years of technological development, domestic OLAP distributed databases have achieved breakthroughs in multiple critical areas, including integration with data lakes, adaptation to cloud environments, and compatibility with domestic operating systems and chips. As big data continues to evolve, distributed OLAP databases still hold huge potential for future development.

The following is a classic architecture of a distributed OLAP database:

 
4. Trends of Distributed Databases in HTAP Applications

In information applications, there are many business scenarios where the majority of operations involve data reading and analysis with only a small portion being data updates. Many other scenarios simultaneously include transaction processing and real-time analysis of the latest historical data. These business types are neither purely OLTP nor purely OLAP, but rather a mix of both. For example, in online sales across various industries, the online order management is an OLTP workload, while sales statistics and analysis are OLAP. However, hot-selling recommendations and promotional price adjustments are neither OLTP nor OLAP. This business type requires that, at the moment a transaction happens, the system can also quickly calculate which items are trending and make minor price adjustments according to promotions. Such workloads belong to HTAP (Hybrid Transactional/Analytical Processing). HTAP is a term coined by Gartner, referring to a database that simultaneously supports both OLTP and OLAP capabilities.  
   
To address various needs, OLTP and OLAP have different technical characteristics. Traditionally, OLAP data often comes from OLTP systems, and there can be a delay of minutes or even hours between OLAP and OLTP systems. This makes it difficult to realize business scenarios that require high real-time analytics. Furthermore, because enterprises need to maintain separate databases and support two different types of workloads, maintenance costs increase. Therefore, a database that can uniformly support transaction processing and business statistical analysis becomes a need. An HTAP database can avoid the massive data exchange between online and offline databases in traditional architectures. HTAP is also based on a distributed architecture, supporting elastic scaling, and can scale throughput or storage on demand, coping with high concurrency and massive data scenarios.   

Currently, there are generally two approaches to realizing HTAP. The first is to use separate compute and interpretation engines for OLTP and OLAP workloads, but with unified management of data resources. Different engines execute different workload types, and OLTP and OLAP use different copies of the same data, better achieving load balancing and isolation. The second approach keeps OLTP and OLAP relatively independent, each with its own set of data, and achieves rapid synchronization from OLTP to OLAP through replication tools. Obviously, in the first mode, data redundancy is lower and consistency is better, but the throughput and performance of OLAP are more limited. The second mode basically preserves the characteristics of both OLTP and OLAP, allowing each to leverage its strengths.

The advantages of distributed databases in OLTP and OLAP are also the advantages of HTAP. Although HTAP is a relatively new concept, many scenarios that combine OLTP and OLAP have long existed. HTAP can also be realized through the convergence of OLTP and OLAP technologies. The best integrated solution should be closely related to actual business requirements and can adopt an appropriate approach based on the specific situation.

The following is a classic architecture diagram of a distributed HTAP database: