Global Distributed Cloud Conference: GBase GCDW Cloud-Native Logical Data Warehouse Empowers Industries to Meet New Challenges of Digital Transformation
Recently, the Global Distributed Cloud Conference with the theme "Leading Distributed Cloud Transformation, Empowering the Bay Area Digital Economy" was grandly held in Shenzhen. This year's conference was co-organized by the Global Distributed Cloud Alliance, Shenzhen Science and Technology Exchange Service Center, Shenzhen Communications Society, and VisionTech. The organizing committee joined hands with GBase, Alibaba Cloud, Tencent Cloud, Google Cloud, Huawei Cloud, Ant Group, Inspur Cloud, Kingsoft Cloud and other top cloud computing teams and distributed cloud pioneers from home and abroad to inject distributed cloud momentum into the digital economy development of the Guangdong-Hong Kong-Macao Greater Bay Area, and further push China's distributed cloud computing development to a new height!
At the Distributed Data Forum, Zhang Shaoyong, General Manager of the Data Intelligence Product Management Department of Tianjin General Data Technology Co., Ltd. (GBase), delivered a brilliant speech titled "GBase GCDW Cloud-Native Logical Data Warehouse Empowers Industries to Meet New Challenges of Digital Transformation".
Six Major Challenges Facing Traditional Enterprise Databases
1. Traditional databases, with standalone, distributed, and integrated storage-compute architectures, all face the challenge of resource elasticity. Hardware resources become bottlenecks during peak business hours, yet are wasted during off-peak periods.
2. As data volumes grow, the three traditional architectures lack sufficient data carrying capacity.
3. The demand for real-time data processing capabilities is increasing.
4. Data types are becoming increasingly complex. Traditional data warehouses primarily handle structured data, but now there are more and more scenarios involving structured, semi-structured, and unstructured data.
5. Users' demand for mining data value is rising, and traditional databases are weak in supporting in-depth mining algorithms.
6. Security requirements are increasingly stringent, especially with the release of regulations such as the Regulations on the Security Protection of Critical Information Infrastructure. Traditional databases urgently need to strengthen security in many aspects.
To address the above challenges, GBase calls the new-generation solution the "Next-Generation Enterprise Data Warehouse". The next-generation enterprise data warehouse should evolve in two directions. The first is cloud-native, distributed, converged, intelligent, real-time, and secure—these six aspects correspond to the six pain points above. Cloud-native solves resource elasticity, distributed solves data capacity, convergence solves multi-modal issues, intelligence solves data value mining, real-time transitions from T+1 to on-time processing, and security addresses encryption, authorization, and other aspects.
These six directions are interdependent. The foundational architecture is based on the traditional MPP database’s distributed, converged, intelligent, and real-time capabilities as the cornerstone, on which data value mining capabilities are built. On top of the resource pooling provided by cloud computing, the distributed, converged, intelligent, secure, and real-time foundational products are architected on the cloud. At the core is big data processing capability to solve the most fundamental problems. On this basis, Zhang Shaoyong introduced the focus of his speech—how to migrate to the cloud.
Key Capabilities of the Next-Generation Enterprise Data Warehouse
Zhang Shaoyong summarized four key capabilities of the data warehouse from the six evolution directions:
First is data real-timeness. There are three aspects: first, how to synchronize data from traditional relational databases into the next-generation data warehouse; second, stream data processing capability; third, how to process faster after data is migrated into the data warehouse lake.
Second is multi-modal capability. It is addressed through data virtualization. Data is stored in multiple engines at the bottom layer, but users do not need to care about the data location; a unified data access capability must be provided.
Third is data democratization. Data should be more visual and can be used through self-service automatic configuration.
Fourth is data collaboration. The system faces multiple types of users; multi-tenancy management and division of labor collaboration within a single system are very important.
Zhang Shaoyong stated that the key capability of the next-generation enterprise data warehouse is convergence and unification. Built upon convergence and unification are data lakes and data warehouses. The next-generation enterprise data warehouse fully supports the capabilities of data lakes and data warehouses. A data lake has larger storage capacity and lower cost, keeps data in its original format, offers more flexible data usage, and serves diverse users. The data warehouse model pre-processes and patterns data.
Cloud-Native Logical Data Warehouse
What is a logical data warehouse?
Gartner defines it as no longer limited to structured data, but including unstructured data. Logically, it is a large data warehouse with various data sources at the bottom layer for correlation processing.
From Gartner's definition, we can summarize the following points:
First, it processes not only structured data;
Second, logically it is a large data warehouse;
Third, although logically unified, data between users or multiple engines need to be related, and multiple data sets need to be associated and exchanged.
Goals of the Logical Data Warehouse
Run Anywhere It can be deployed on the cloud, on physical machines, or virtual machines, and can run regardless of the environment.
Store Anywhere Users do not need to care where the data is. Structured data is stored in traditional databases and MPP, while unstructured data can be stored in Hadoop engines. This is the capability of data virtualization.
Use Anywhere The capability of data federation. When used, it can handle algorithms for structured data, algorithms for unstructured data, and support mixed scenarios.
Design Philosophy of the Logical Data Warehouse
Traditional data warehouses involve a data collection process, where data from sources is collected into the pipeline via ETL. The logical data warehouse is based on heavy correlation, requiring no collection; it only needs the data (whether on-premises, in the cloud, or anywhere else) and performs correlation processing when used. Zhang Shaoyong added that this scenario is not absolute. Some data that needs processing must first be centralized, actually still going through ETR. But some data, due to quality or self-developed requirements, may be in other data sources and can be used directly without collection. Both scenarios have their practical business use cases.
Based on this design philosophy, Gartner proposed the concept of the next-generation enterprise data platform in 2016: it is a comprehensive analytics scenario covering both batch and real-time work modes. To address these scenarios, it provides data virtualization and data federation capabilities. Data virtualization primarily manifests as the ability to implement a unified interface for data, while data federation enables cross-data access and computation.
The core technologies of the logical data warehouse are data virtualization and data federation. The data virtualization capability solves the problem of multi-data fusion, enabling the fusion and management of structured, semi-structured, and unstructured data, and achieving store-anywhere through a unified entry point. The data federation capability integrates multiple computing models—such as online analysis, correlation analysis, and graph computing—to enable the fusion of various algorithms, thereby allowing data to be used anywhere. In addition, the federation capability can connect data among multiple underlying engines for real-time flow.
Next-Generation Big Data Platform Architecture
In the middle data management layer, the entire architecture consists of data collection, data management, data analysis and mining, and data services computing. Traditional platform architectures follow this structure. The logical data warehouse, at the data processing layer, integrates all scenarios except the real-time processing of the IoT platform into a unified logical processing warehouse module for unified processing. From the architecture of the next-generation big data platform, we can see that the logical data warehouse carries a very important function in the overall architecture.
The cloud-based logical data warehouse architecture consists of three layers. The first layer is the unified interface layer. Due to the widespread use of the Hadoop platform, GBase GCDW supports standard SQL interfaces. The middle layer implements multi-engine management, unified users, unified metadata, and cross-engine query planning. The bottom layer implements unified data exchange between multiple engines, consistency management, transparent real-time data synchronization, and data management assurance.
Data virtualization capability is achieved through a unified SQL interface, transparent cross-engine access, full data type fusion processing, and cross-domain and multi-center management. Data federation capability is achieved through full algorithm fusion, connecting data associations between computing engines, and data lifecycle management.
The middle unified logical data warehouse supports multiple storage engines and computing engines, providing a unified data virtualization layer externally. Zhang Shaoyong introduced that overall, GBase GCDW caters different storage engines to different scenarios, shielding the underlying layer from users. Users only need to access different engines through the unified data virtualization layer, which is very convenient.
With the logical processing warehouse capability, it can perfectly support data lakes, data warehouses, and data marts. The traditional architecture of a data lake is basically supported by Hadoop + MPP. Through the logical data warehouse, multiple engines of Hadoop + MPP are unified and managed to support data lake construction. Traditional data warehouses and data marts, which mainly support structured data processing, can also be perfectly supported by the logical data warehouse. Such a logical data warehouse product can unify and simplify the architecture, enabling a single technology architecture to support data lakes, data warehouses, and data marts simultaneously.
Realizing Data Intelligence
Mining Algorithms
Previously, mining algorithms were applied by importing data into front-end tools for mining and learning using Python before the algorithm was executed inside the database. By integrating algorithms inside the MPP cluster, distributed computing is achieved, thereby improving data analysis performance and reducing data movement.
Data Security
Data storage encryption, and it must be transparent encryption. Encrypt the columns of interest. Only the private key file is needed; user usage is completely transparent, with no need to invoke encryption algorithms.
If storage is not encrypted, sensitive data also needs to be masked. GBase's dynamic data masking technology mainly has three methods: specify masking for concerned data fields, which can be set as default masking, random masking, or custom masking. After the masking method is specified, it is bound to user permissions, so that only authorized users can see plaintext data, while unauthorized users see masked data. Ciphertext has multiple representation methods. Through dynamic data masking, the data storage itself does not have plaintext or encryption, but the front end controls different data presentation methods based on different user permissions.
Data Integration
The next-generation big data platform faces multiple data sources and outputs, so data integration and consolidation are necessary. To accommodate real-time processing, one method is to add data via streaming, the second is through CDC, the third is through batch data loading, and in addition, data can be loaded via DBlink.
After data is ingested into the logical data warehouse through various methods, unified processing of online and offline databases can be performed. The processed results can be provided to applications in standard development result formats, to downstream systems as files, or to visitors. Regardless of input or output, GBase supports multiple data integration methods.
Fast Algorithms After Data Ingestion
The cloud-based logical data warehouse supports a new-generation vectorized computation engine. Traditional row-by-row processing faces significant function call overhead and cannot fully utilize modern CPU SIMD instruction sets. The new vectorized computation engine performs block-wise computation, thereby improving data processing capability. With the advancement of IT application innovation, GBase now adopts NUMA architecture, leveraging the advantages of multiple architectures to provide multi-instance deployment. Originally, one instance was deployed; now multiple instances are bound according to NUMA, avoiding the performance penalty of cross-NUMA memory access and improving computational performance on multi-NUMA architectures.
By the end of 2020, GBase achieved a performance value of 8.94 million with only 8 physical machines. The core of database products is to fully leverage hardware capabilities, which is also the continuous effort required for foundational software.
Cluster scale is also a goal GBase has always pursued. The problem that distributed systems need to solve is how to expand data capacity, from GB to PB levels. GBase conducted two types of tests: one for basic data capabilities and one for performance. The basic capability test covered various aspects, and the performance test was conducted in two scenarios: finance and telecom. To date, GBase has the largest MPP scale in China, with a single MPP cluster reaching 4,096 nodes, and it has passed performance tests based on both finance and telecom models, being the first in China to achieve this.
How to Migrate a Data Warehouse to the Cloud
Migrating to the cloud involves two aspects: one is migrating to a private cloud, and the other is migrating to a public cloud. GBase supports both methods—one is block and object storage on a private cloud, and the other is object storage on a public cloud. The cloud-based logical data warehouse can be fully deployed on public and private clouds, and there are already practical cases. The core of data ingestion and operation management capabilities lies in the virtual warehouse layer and compute layer of the middle data warehouse, which divides data into different virtual warehouses according to different logics. The bottom storage layer implements both block and object storage methods; on public clouds it is mainly object storage, and on private clouds it can be block and object storage.
The middle layer, i.e., the virtual warehouse layer, logically presents to users unified scheduling management through a unified access layer at the top. In the middle layer, users can establish independent virtual warehouses according to different business needs to support different concerned businesses. The underlying data is shared among different virtual warehouses, and the top unified scheduling achieves unified access. Fault isolation and resource isolation are realized between different virtual warehouses, and each independent virtual warehouse can be scaled independently, making user operations simpler.
The lakehouse architecture has three core characteristics. First, since it integrates lake and warehouse, its storage is shared. Second, data among multiple engines in the lakehouse must be integrated and connected, which is the data federation capability for data association operations. Third, a single dataset serves different scenarios, requiring support from multiple computing engines, which demands multi-modal capabilities.
The cloud-based logical data warehouse can support the lakehouse with data sharing and separation of compute and storage, meeting the first point; the logical data warehouse supports multiple computing engines, meeting the second point; the logical data warehouse itself connects multi-engine data, association computation, and data exchange, meeting the third point.
The storage layer implements one-stop data storage. The middle elastic compute layer supports virtual machines, containers, and bare metal. The top layer implements one-stop data governance to data assetization. Therefore, the capabilities of the cloud-based logical data warehouse can perfectly support the construction of a next-generation enterprise data warehouse as a lakehouse. The platform is deployed on the cloud; all data storage integration, data management, and platform management belong to PaaS. On this basis, SaaS-layer data services are provided. The entire infrastructure plus services are delivered externally in a DaaS model.
Zhang Shaoyong summarized: To address the six major challenges facing traditional data warehouses, we propose the main evolution directions for the next-generation enterprise data warehouse—cloud-native, distributed, real-time, intelligent, and secure. The logical data warehouse provides distributed capabilities, converged processing, real-time, and security capabilities, thereby achieving the fusion of all data types and solving the entire problem of data storage and processing. Through the integration of machine learning in Database AI, data value mining is realized. Migrating the entire logical data warehouse to the cloud and providing services in a DaaS model achieves the integration of DaaS, IaaS, and PaaS, thus realizing cloud migration capability.
In summary, the cloud-based logical data warehouse provides a fulcrum for informatization in the digital industry, solving the six challenges faced by traditional data warehouses and the six aspects that the next-generation enterprise big data warehouse should address. GBase can perfectly support the next-generation enterprise big data warehouse.
GBase supports multi-cloud and hybrid cloud, including public and private clouds both domestically and internationally. Through a unified interface layer, it achieves unified access to block and object storage data. Through the computing engine, it implements buffer management; since the performance of object storage is relatively low, multi-level buffer management is needed to accelerate execution performance. It provides users with an access scheduling layer for unified resource scheduling. With the above layers of capabilities, it can provide users with the entire ordering, service entry, and unified management of cloud services.
About GBase
GBase has been established for 17 years, focusing on the research and development of database and big data products. It is currently the only database vendor in China that has achieved large-scale application in the finance and telecommunications sectors, with a large customer base in these fields and three mainstream big data products.
GBase was included in Gartner's Magic Quadrant for Data Management Solutions for Analytics in 2017 and 2018. In 2019, in Gartner's ranking of 19 data warehouse products worldwide, GBase GCDW ranked seventh globally, the only product from China to make the list.
GBase has achieved great success in the Chinese market, deploying MPP warehouses in finance and telecom, providing logical data warehouse solutions, and holding a leading position in the industry. It also covers various other industries beyond telecom and finance, with a total node count exceeding 30,000.
GBase GCDW has assisted the head office of Agricultural Bank of China in building a centralized big data platform, with a total of over 2,000 nodes, and has been running stably for more than 2,000 days.
GBase GCDW is also applied in major state-owned and joint-stock banks such as Bank of China, People's Bank of China, China Merchants Bank, and also covers a large number of city commercial banks and rural credit cooperatives.
In the telecom industry, GBase has assisted the headquarters of China Mobile Group in deploying a centralized business analysis system, also with over a thousand nodes, covering 22 of the 31 provinces of China Mobile nationwide, with a coverage rate exceeding 80%.
In other industries such as insurance, electric power, government affairs, and party-government bodies, GBase has also been widely applied. Whether on-premises or on the cloud, GBase GCDW can support enterprises in building their next-generation data warehouses and big data platforms.
In the future, GBase will continue to focus on database software products and services, striving to become the most trusted database product service provider for users.