DTC 2023 Recap | GBase GCDW: Cloud-Native Real-Time Logical Data Warehouse Enabling the Lakehouse
At the 2023 DTC Data Technology Carnival, Lakehouse Innovation Forum, Zhang Shaoyong, Chief Engineer of GBase 8a at GBASE, delivered a speech titled "GBase GCDW: Cloud-Native Real-Time Logical Data Warehouse Enabling the Lakehouse". This article provides an in-depth look at how GBase GCDW, a cloud-native data warehouse, supports and implements the lakehouse architecture.
Data warehouses differ from databases and data lakes. Databases primarily serve operational systems, while data lakes store all raw data. The emergence of data warehouses was driven by enterprises' need to mine data value. Traditional OLTP transaction-oriented databases struggle with cross-departmental data integration, often creating data silos. A data warehouse aggregates data from diverse sources into a centralized and consistent storage system, overcoming the limitations of cross-database operations. Combined with data mining, artificial intelligence (AI), and machine learning, it extracts data value to support business decision-making. It also serves as the technological foundation for BI, meeting customers' needs for data analysis and decision support.
According to Zhang Shaoyong, traditional enterprise data warehouses face six major challenges as application scenarios rapidly expand and data volume and variety soar.
Data Warehouse Technology Evolution Trends
Data warehouse technology has a long history. Since standalone products emerged in the 1980s, it has evolved through several stages.
Cloud-Native Real-Time Logical Data Warehouse: A New Fulcrum for the Lakehouse
While data warehouses have limitations such as lacking support for unstructured data, high costs, and inflexibility, data lakes suffer from poor query performance, low real-time capabilities, and limited reliability. The convergence of the two — the lakehouse architecture — combines their strengths and has gradually emerged. GBase GCDW was born to meet this need.
GBase Cloud Data Warehouse (GCDW) is a cloud-native elastic data warehouse independently developed by GBASE. It features a hybrid row-column storage architecture, massive distributed parallel processing, and elastic scalability. It is designed to meet the demands of enterprise-grade elastic data warehouse systems. GCDW has two core features.
GCDW supports both on-premises deployment (with compute-storage separation in a private cloud) and cloud deployment. It offers elastic resource scaling, allowing users to elastically expand compute or storage units as needed.
GCDW provides SaaS capabilities in the cloud, delivering an enterprise-grade elastic data warehouse system that enables users to set up and operate with ease in the cloud.
GCDW's virtual clusters (physical isolation) and resource management (logical isolation) deliver elastic resources and mixed workloads. It supports multi-source, real-time, and efficient data integration, and provides vectorized computation and hardware acceleration to meet real-time data processing demands, enabling businesses to shift from offline batch processing to real-time operations.
Its next-generation vectorized computing engine combines the classic volcano model with block processing to fully leverage the CPU, boost cache utilization, and minimize unnecessary storage and memory access. Additionally, built on 4th Gen Intel® Xeon® Scalable processors, it significantly enhances performance and compression rates, delivering higher compression ratios and improved performance per core.
Use Case
GCDW Data Mart Application at a Bank
The project piloted three business scenarios: risk data mart, regulatory reporting, and historical data rerun.
Leveraging GCDW's elastic resources and multi-tenancy capabilities, a sub-tenant was created for each data mart application, achieving resource isolation, elastic scaling, rapid resource provisioning, and readiness for the future lakehouse architecture.
Risk data mart: Over 50 daily jobs, 1 TB of daily incremental data, 4–8 node warehouse (WH), performance on par with the GBase 8a cluster;
Ad hoc regulatory reporting: WH compute resources can be instantly provisioned through the interface; data is already in the warehouse, enabling immediate development and testing;
Historical data rerun: Archived data in the data lake does not need to be 'warmed up'; the lakehouse architecture allows data to be read directly from the lake for reruns. With multi-tenancy, each tenant can perform operations independently without relying on the primary database.
As a key component of a big data platform with comprehensive processing capabilities, GBase GCDW provides multi-tenant data openness and cross-center operation capabilities. Leveraging the complexity and convergence of modern big data applications, it integrates and manages diverse platforms, incorporating intelligent computing through machine learning, unifying relational and non-relational computing, and enabling real-time data ingestion and processing. It also integrates with data lakes, supports secure data transmission and unified storage, and utilizes a compute-storage separation architecture to deliver DaaS (Data as a Service) deployment and openness that bridges private cloud and public cloud in the analytics domain. With these capabilities, it builds the lakehouse architecture in the cloud.