GBASE Financial Application Guide 2 | GBase 8a MPP Cluster Logical Architecture
To help financial institutions select distributed analytical database products and promote successful deployment experiences, GBASE, under the guidance of the Beijing Financial Technology Industry Alliance, has authored the GBase 8a Financial Application Guide. The guide provides an in-depth look at the entire deployment process for distributed analytical databases, from selection planning, development and design standards, and data security and high availability, to operations optimization. It also introduces representative deployment cases of GBase 8a MPP Cluster at China's policy banks and large state-owned commercial banks.
GBASE will publish a series of articles to share and interpret the guide's content, aiming to provide financial users with a reference for database selection and help drive the efficient implementation and high-quality development of technology finance.
This is the second article in the series, following the introduction of distributed analytical database characteristics and financial industry application scenarios, it further details the system architecture of distributed analytical databases used in the financial industry, along with the corresponding GBase 8a MPP Cluster product architecture.
Distributed Analytical Database Logical Architecture
1. System Architecture
Common system architecture in the financial industry.
Common system architecture in the financial industry
Data Exchange Layer: Used to aggregate various data from front-end production systems, avoiding cross-system operations that impact business operations, and ensuring the authenticity, integrity, and uniqueness of data in core systems; provides multiple data integration methods to meet batch and real-time data ingestion requirements.
Data Processing Layer: Typically adopts a Hadoop + MPP data warehouse and data lake architecture.
– Hadoop is usually used as an ODS database and data lake, for raw data collection and storage, data standardization, accumulating full data snapshots, retaining historical data assets, and processing unstructured data.
– MPP typically serves as the data warehouse, responsible for high-performance statistical analysis of structured data. Within MPP, business data is logically layered and processed according to data warehouse models. The analyzed results are provided to upper-layer services through data export, DBLink, virtual cluster cross-business access, etc.
Data Mart Layer: Usually stores pre-calculated data for specific users, i.e., the analysis results from the data processing layer are provided to the data mart for secondary processing and analysis combined with local data to meet specific thematic needs, such as report query services.
Data Service Interface Layer and Data Application Layer: Provide data access interfaces and query services to front-end businesses.
GBase 8a MPP Cluster is primarily used in the data processing layer and data mart layer within the financial industry system architecture.
2. GBase 8a Product Architecture
GBase 8a MPP Cluster Product Architecture Diagram
GBase 8a MPP Cluster Product Architecture Diagram
GBase 8a MPP Cluster adopts an MPP + Shared Nothing distributed federated architecture, comprising three core components: the distributed management cluster GCWare, the distributed scheduling cluster GCluster, and the distributed storage and computing cluster GNode. Key features include:
• High Performance: Columnar storage, hybrid row-column storage, HASH indexes
– Uses columnar storage, compression, and intelligent indexing; in data warehouse scenarios, moderate compression algorithms typically offer compression ratios from 1:3 to 1:5;
– Uses hybrid row-column storage to reduce I/O in select * scenarios;
– Uses HASH indexes and data sorting to boost equi-join query performance;
– Leveraging MPP multi-node parallelism, performance scales linearly with cluster size;
– Supports vectorized computation to increase calculation speed.
• Cross-Business Cluster Management and Interconnectivity: Virtual Clusters, DBLink
– Virtual clusters provide unified management, a single access entry, and unified user permission control for multiple compute clusters through a single management cluster, while ensuring resource isolation between clusters. They support cross-cluster data access and computation. When data relevance between multiple business systems is high, virtual cluster management can be adopted;
– DBLink enables cross-cluster data access, facilitating data flow between clusters and supporting queries on remote databases, as well as join operations between remote and local data.
• High Availability: Federated architecture with no single point of failure, backup and recovery, active-active clusters, online node replacement
– No single point of failure: All components are deployed in Active-Active mode, eliminating single-point performance bottlenecks and failures;
– Backup and recovery: Supports instance-level, database-level, and table-level backup and recovery, including full and incremental backups;
– Active-active clusters: Supports real-time and asynchronous active-active schemes. With virtual cluster mirroring technology, real-time table-level data replication between two compute clusters is supported. Synchronization tools enable asynchronous data consistency between clusters. Point-to-point incremental sync based on binary logs delivers high synchronization performance. Supports geo-redundant active-active deployment and two-site three-center architectures;
– Online node replacement: Allows node replacement online without service interruption; during replacement of a faulty node, the cluster supports DQL/DML/DDL operations.
• High Scalability: Online expansion
– Cluster expansion requires no downtime; the process can be monitored, paused, resumed, or cancelled. This meets continuous business availability requirements and minimizes performance impact during scaling.
• Data Integration: High-performance bulk loading, real-time data ingestion, Hadoop integration
– High-performance bulk loading: Policy-based data loading with replica chain forwarding, P2P multi-point transfer, and parallel loading agents can achieve up to 30TB/hour. Supports various data sources and protocols such as HTTP, FTP/SFTP, HDFS, Kafka, S3, and file formats including text, ORC, Parquet, gzip, snappy, lzo, etc. Enables loading from and exporting to HDFS for seamless data interaction with Hadoop clusters;
– Supports external tables, enabling direct querying of data files in open formats from HDFS, S3, FTP, etc.;
– Real-time data ingestion: Built-in Kafka Consumer consumes real-time Kafka messages; the GBaseRTSync tool provides real-time data synchronization from OLTP databases to MPP.
• Security: Permission management, user security, transparent storage encryption, data masking
– Comprehensive user authentication and permission management;
– Data is stored encrypted in the file system, with table-level and column-level encryption, and supports the SM4 Chinese national encryption algorithm to meet data security requirements;
– Dynamic data masking restricts the visibility of raw data through permission controls, simplifying security design and coding at the application layer.
• Machine Learning-Based Data Mining
– Provides a machine learning extension library plugin to integrate ML algorithms for in-depth data analysis and mining;
– Supports Python and C/C++ UDF/UDAF to extend algorithm functions, meeting custom data mining and analysis needs.
• Ease of Use: Visual monitoring
– Offers a visualized O&M management tool, GDOM, that supports installation, upgrade, expansion, node replacement, and service start/stop of GBase 8a MPP Cluster clusters. It monitors the running status, resource utilization, and SQL execution of single or multiple clusters, and provides real-time alerts and trend displays.