GBASE's Focus on User Experience, Part 1
GBASE focuses on user experience
"Providing world-class domestic databases for Chinese users" has always been the mission of GBASE, and GBASE has been striving for it!
More than ten years of focus have also brought us gratifying market feedback. The following is an excerpt from the December 2015 issue of "Financial Informatization" magazine, fromthe Agricultural Bank of ChinaSoftware Development Center Director Zhao Weiping's article
Research and Practice of Domestic MPP Architecture Database in
Financial Big Data Computing Environment
Article Agricultural Bank of China Software Development Center Zhao Weiping
With the construction of the information system of the Agricultural Bank in recent years, the data volume has increased sharply, and the demand for big data processing, data analysis, and mining has been continuously expanding. Looking to the future, in order to better utilize emerging technologies in the field of big data to build a big data platform, respond to market changes, drive business model innovation, and seek new technological directions has become an inevitable choice. At the same time, information security issues have increasingly attracted the attention of the Chinese government and enterprises in recent years, and the localization of information systems in key areas is gradually being carried out. Based on considerations of information security, technological innovation, and other aspects, firmly taking the path of localization replacement, and choosing domestic software in the basic architecture and core areas is one of the key factors in our technical selection. Therefore, our bank has increased research efforts on localization replacement, conducted in-depth research and selection of domestic MPP database products, and finally chose the domestic MPP architecture database—GBase 8a MPP Cluster. Through practical application, research, and analysis, improvements and enhancements have been made to the GBase 8a MPP Cluster database in many aspects such as architecture design, resource management, and function implementation, ultimately forming a database product suitable for PB-level big data computing in the financial industry, basically meeting our bank's data statistical analysis and application needs.
One
Overall Architecture
During the implementation of the project, based on our bank's years of use, research, and testing comparison of various databases, starting from actual needs, our bank redesigned and optimized the GBase 8a MPP Cluster database to form a big data computing environment solution that can support PB-level data processing. Its overall architecture is shown in Figure 1.
Figure 1 Overall Architecture Diagram
The entire solution consists of a 56-node GBase 8a MPP Cluster database system and 8 ETL servers, using Huawei's RH2288 PC servers for hardware. Each database node server is configured with: 2x10-core CPU, 3Tx12 hard drives at 7200 RPM, and 256GB of memory. The system has PB-level raw data storage capacity. In project testing and verification, more than 60,000 data processing jobs and interactive data query tasks are run daily, processing 52 source systems with 1287 source tables totaling 1.4TB of interface data, and at the end of the month processing 3.6TB of data with the largest table record count reaching 200 billion rows. Data processing tasks include a large number of complex time zipper and statistical summary aggregation operations. Practice has proven that the PB-level big data computing environment developed by our bank can meet the requirements of the financial industry and has the feasibility of being promoted to other fields.
Two
Technological Innovation
During the research and practice of the project, our bank optimized and improved the domestic MPP architecture database product, retaining its own technical characteristics such as column storage, coarse-grained indexing, intelligent compression, and flat architecture, achieving innovative breakthroughs in multiple key technologies. In terms of architecture design, resource scheduling, execution efficiency, etc., hundreds of technical issues were researched, designed, and improved, bringing the domestic database product to a basic level of commercial application. The innovations in improving and optimizing the domestic MPP product are as follows.
1. Cluster Architecture
Improvements to the cluster architecture include two aspects: grouping of functional modules for each node and the mechanism for session establishment and execution.
(1) Functional Grouping MPP Architecture. Based on research and practice of the current master-slave MPP architecture and non-master-slave MPP architecture in the market, our bank proposed an improvement plan for establishing a functional grouping MPP architecture: deploying the three major modules GCluster, GNode, and GCWare on different nodes (as shown in Figure 2), forming three types of functional grouping clusters. The three types of functional grouping clusters can be expanded separately, completely breaking the bottleneck of one aspect becoming a limitation on the performance and concurrent access capability of the entire system, improving system efficiency and stability.
Figure 2 Functional Grouping MPP Architecture
(2) Asynchronous SQL Session Establishment and Execution Mechanism. Due to the serious deficiencies in architecture and performance design when GCluster sends commands to GNode using multi-threaded synchronous API in the GBase architecture, SQL concurrency is limited. To address this issue, the project team proposed establishing an asynchronous SQL session and execution mechanism. Executing SQL asynchronously will not occupy too many network SOCKETs, and GBase does not need to enable too many threads, saving connection numbers, expanding channel capacity and cluster scale. This improves the number of concurrent accesses and better solves the problem of severe system resource consumption and performance degradation when SQL concurrency is large.
Execution performance degradation and other issues.
2. Resource Management
Resource management mainly includes optimization of the mechanism for external customer access and internal resource scheduling.
(1) Optimization and improvement of concurrent access scheduling mechanism. In a mixed load and concurrent environment, when the MPP database cluster executes LOAD, DDL, DML, and SELECT concurrently, it relies on the concurrency mechanism of GCluster. The concurrency control of GCluster is divided into two levels: external concurrency control and internal concurrency control. Our bank innovatively proposed adjusting the external transaction concurrency control in GCluster to be managed by two services, namely distributed lock service and distributed priority service. Internal concurrency control is implemented through some specific mechanisms. Through optimization and improvement of the access scheduling mechanism, concurrency is expanded, and the overall performance and availability of the system are improved.
(2) Comprehensive resource management technology. Generally, MPP database systems manage resources using a hierarchical approach, with different restrictions and dynamic adjustments at different levels. In the final optimization design of GBase 8a MPP Cluster, the project considered using three-level management technology to achieve effective resource management: at the connection layer, controlling the number of connections entering the cluster to ensure the system runs tasks according to the optimal resource mode; at the session layer, all users are divided into three roles: system administrator, DBA, and User, achieving priority management; at the execution layer, according to user priority, obtaining the resource control strategy set by the system administrator,
and executing tasks according to the strategy. Through three-level management technology, comprehensive resource management is effectively achieved.
3. Execution Efficiency
Optimization of execution efficiency mainly focuses on algorithm and parallel submission mechanism.
(1) Algorithm optimization based on operator classification. The focus of distributed query optimization in MPP databases is to parse and optimize SQL, trying to distribute the parsed SQL to each node in the optimal way for execution by each node, thereby avoiding frequent movement of table data between nodes. Based on the characteristics of MPP database SQL parsing, the project's algorithm optimization ideas mainly distinguish two types of cluster operators, including distributed JOIN and aggregation operations, adopting differentiated optimization strategies for different cluster operators, reducing a large amount of data movement between nodes, and improving SQL execution efficiency.
(2) Parallel submission mechanism. GBase 8a MPP Cluster uses a master-replica mechanism to achieve SQL transaction integrity, previously using two Commits to achieve the goal of ensuring SQL transaction integrity. In this project, based on analysis of some issues in testing, the original two Commit mechanism was changed to parallel submission, improving performance.
4. Data Interaction
Technological innovations in data interaction include: data interaction between GBase 8a MPP Cluster and the same type of database, and interaction between MISP cluster and Hadoop cluster.
(1) Efficient data transmission between the same type of databases.Data interaction between GBase 8a MPP Cluster and the same type of database needs to focus on solving the problem of tight time window resources when synchronizing large amounts of data between data warehouses and data marts. In this project, the DB_Link connection method between data warehouses is adopted, and the advantages of GBase 8a MPP Cluster's balanced data distribution and parallelism are fully utilized, directly transmitting data between clusters without intermediate landing, improving transmission efficiency and facilitating data synchronization.
(2) Data backup based on Hadoop. Since the data in GBase 8a MPP Cluster and Hadoop cluster are stored in shards, the project team utilizes direct communication between GBase 8a MPP Cluster computing nodes and Hadoop DataNode. By installing Hadoop clients and backup recovery programs on each GBase node, using Hadoop read-write tools to perform parallel backup and recovery of database binary storage files on each node, fully leveraging Hadoop's high read-write throughput advantages, solving the problem of massive data backup and recovery.
Three
Application Results
After more than two years of research, this project used data from 52 systems in core business areas such as deposits, credit, and investment across the bank for actual verification, forming a feasible plan for building a PB-level big data platform in the financial industry, achieving the expected goals in improving domestic MPP database software and breaking the monopoly of foreign basic software in this field, fulfilling our bank's social responsibility in ensuring domestic information security.
First, breaking the monopoly of foreign basic software in China. This project is the first to achieve a breakthrough in the PB-level big data computing environment field in the financial sector based on domestic MPP databases and PC servers. It has opened a gap in the monopoly market of foreign MPP database products in the domestic big data computing field, providing a new choice for domestic finance and even other industries in the construction of big data computing environments.
Second, conducting beneficial exploration in ensuring national information security. For a long time, the position of IOE in the construction of core systems in the financial sector has been unshakable, and in the large-scale MPP data warehouse construction that began around 2000 and subsequent big data construction, there was no choice but foreign products for processing massive structured data. Through this project, our bank actively explored and practiced in the fields of big data processing and analysis, breaking the monopoly situation where only foreign products could support PB-level structured data computing. The results prove that domestic MPP databases can also complete PB-level data computing, and the success of this project is a typical event and strong evidence of ensuring national information security through information localization.
Third, improving our bank's technical level in the field of big data. Although our bank has been continuously tracking cutting-edge technology for many years, due to the confidentiality of foreign manufacturers, the understanding of MPP database technology was limited to the application level. In this project, we were open and honest with domestic MPP database manufacturers, deeply collaborating, conducting deep research on domestic MPP database products as a "white box," and in the process of solving GBase MPP product problems, our bank thoroughly understood and mastered MPP architecture databases from deep principles, improving our bank's technical level in the field of big data.
Fourth, saving the cost of building a big data platform. The success of the research and practice of this project provides an economical choice for the financial industry in constructing a PB-level big data computing environment. Based on domestic MPP databases, the cost of building our bank's big data platform has been reduced from the billion-level to the ten-million-level or even million-level, making PB-level data warehouses and big data platforms a computing environment that most enterprises can afford.
Four
Conclusion
The Agricultural Bank took the lead in responding to the government's call for information localization, with a strong sense of mission, not fearing risks, daring to bear pressure, carefully planning, and cautiously implementing. From a comprehensive understanding of the MPP database industry, considering comprehensively, selecting domestic MPP databases as the initial product, choosing Huawei PC servers and SuSE Linux ll SP2 as supporting facilities. With the support of manufacturers, through testing with large data volumes, multiple scenarios, high loads, and high concurrency, discovering problems, analyzing problems, delving into principles, redesigning, developing corrections, testing and verification, continuously improving and enhancing domestic software. Ultimately, through the joint efforts of both parties, the domestic MPP database cluster product filled the domestic gap in PB-level data processing. At the same time, it provides a low-cost choice for the domestic financial industry and even other industries. It can force foreign MPP database manufacturers to lower prices, and also allow enterprises that have stopped trying due to price reasons to dare to try, helping to expand the application range of domestic big data computing environments and promote the pace of development in the domestic big data field.
(Research and Practice of Domestic MPP Architecture Database in Financial Big Data Computing Environment won the second prize of the "Bank Technology Development Award" in 2014)