Unveiling the Technology Behind ABC's Big Data Platform: 1,000 Days of Stable Operation
GBase 8a MPP analytical database cluster has powered the Agricultural Bank of China's big data platform for over 1,000 days of stable operation, supporting 600+ nodes and managing 20 PB of data. With complex business demands and rapid data growth, the high reliability and availability of this Chinese-developed database have been robustly demonstrated.
This article will provide an in-depth look at the technology behind this achievement.
Background
To meet the growing demands for in-house data analytics and regulatory reporting, ABC initiated the construction of a fully autonomous and controllable big data platform in 2013. General Data Technology's GBase 8a MPP Cluster stood out among many alternatives and was selected as the foundational data management software for the platform's core component, the Enterprise Data Warehouse (EDW) and data marts.
During platform construction, ABC and General Data Technology jointly conducted in-depth research and tuning of the MPP database, completing hundreds of optimizations and improvements covering underlying architecture, high reliability, high availability, and performance.
Objectives
Build an enterprise-level big data platform across the bank to achieve unified management of data resources, comprehensively enhance data service capabilities, fully unlock data value, meet the needs of customer marketing, risk management, operations management, and external regulatory requirements, drive data governance, and improve the management level of data resources and the comprehensive utilization of data assets. Empower the business to "speak with data," provide endless internal momentum for business growth, and continuously drive innovation in business, marketing, services, and management.
Achieve unified management of internal business data; acquire external data through third-party partnerships to enable full lifecycle management of data resources; deliver comprehensive data products to all levels and business lines across the bank; establish a robust data service management system; strategically build data marts across eight key domains—personal customers, corporate customers, operational risk control, risk management, performance management, audit and internal control, regulatory statistics, and branch data marts. Business support spans customer marketing, risk control, operational analysis, external regulatory compliance, asset-liability management, performance management, etc. Leverage both internal and external data to conduct analyses across diverse areas and hot topics, deeply mining data value.
Challenges
ABC faced the following challenges in building its big data platform:
1. Achieving a Seamless Integration Between the MPP Database and the Hadoop Platform
MPP databases excel at high-density structured computing, while the Hadoop platform shines in unstructured data processing and scalability. Thus, it was essential to evaluate which scenarios were best suited for MPP versus Hadoop, how to enable data exchange between them, and how to ensure the two architectures complemented each other's strengths.
2. Smooth Transition in System Development
Migrating from traditional databases to MPP and Hadoop platforms: how to leverage new infrastructure features while swiftly migrating existing data models and developing new ones.
3. Unified Planning, Deployment, Management, and Monitoring of Large-Scale Cluster Environments
The big data platform involved dozens of clusters and nearly a thousand servers, imposing stringent requirements on data center facilities and networking that demanded meticulous planning in advance. Managing OS and database installation, deployment, upgrades, and administration at this scale required unified processes and operational methods. Multi-cluster monitoring, alerting, and health checks also needed effective procedures and system support.
4. Meeting Diverse Data Demands and Response Times for Upper-Layer Applications and Online Services
The platform supported applications across different business areas such as regulatory reporting, auditing, and retail, each with distinct data interaction patterns and varying latency requirements. A unified interface approach with configurable management was needed.
5. Ensuring High Reliability, High Availability, and Disaster Recovery
As the platform underpinned multiple business domains, its criticality in the IT landscape meant that any failure could have immeasurable impacts. Thus, stability and high availability had to be guaranteed at both the database and application levels. Additionally, the challenge of backing up petabytes of data required in-depth research; a multi-layered disaster recovery strategy was implemented, including active-active clusters and data backups to Hadoop clusters, to ensure data security.
Solution
The platform was built on a hybrid architecture of General Data Technology's GBase 8a MPP Cluster and Hadoop. The GBase clusters comprised a total of 636 data nodes: the primary data warehouse had 112 nodes configured as two clusters forming an active-active primary database, plus 8 data mart environments and 5 peripheral application clusters. The Hadoop side included 263 nodes, with 172 nodes dedicated to the ODS Hadoop cluster and the remainder to stream processing (Spark) for data analysis and mining.
Overall Architecture Diagram
GBase 8a MPP clusters served as the core components of the big data platform: Enterprise Data Warehouse (EDW), data marts (DW), data mining, data extraction, and regulatory reporting applications.
Hadoop clusters handled Operational Data Store (ODS) processing, historical data backup, stream computing, and some analytical and mining tasks.
The big data platform also included systems for unified scheduling, unified monitoring, unified ETL development tools, unified metadata management, unified data quality management, and a unified visualization platform.
Key Technology 1: Hybrid Architecture
Leveraging Hadoop's strengths in unstructured data processing, ODS tasks such as data cleansing, transformation, automatic character encoding recognition and conversion, and deduplication were decomposed into distributed, parallel MapReduce jobs. At the same time, full data was processed into incremental data, reducing the data volume and significantly boosting ETL performance.
For the primary database, GBase 8a MPP loading tools directly read and loaded LZO files from Hadoop, markedly improving data ingestion efficiency while reducing network overhead. After incremental data was loaded, the primary MPP cluster performed high-value, large-volume complex computations like base model processing and indicator aggregation. The data mart MPP clusters were mainly responsible for creating wide tables for various domains, multidimensional analysis (CUBE), and some report generation.
Key Technology 2: MPP Active-Active Cluster
Active-Active Architecture Diagram
Utilizing GBase 8a's inter-cluster synchronization tools, incremental data from the primary cluster (in Data Cell units) was identified and transmitted point-to-point to the standby cluster, ensuring data consistency. Combined with the platform's batch processing scheduler and monitoring system, an active-active plan was developed. This enabled: 1) daily incremental data backup; 2) a workload distribution model where the primary cluster handled batch processing while the standby cluster managed online queries (Active-Query for Asymmetric Workload); and 3) rapid failover of batch processing to the standby cluster in the event of a primary cluster anomaly.
The active-active cluster mechanism addressed the challenge of backing up petabyte-scale data in big data environments, enhanced the platform's business availability and stability, ensured continuity of batch operations, and boosted the platform's overall service capacity.
Key Technology 3: Inter-Cluster Data Exchange
After the primary database completed base model processing and indicator aggregation, each data mart retrieved full or incremental interface data from the primary database based on its business domain. Daily interface data volume could reach hundreds of terabytes. Using traditional file exchange methods would be too inefficient to meet the demands of the applications supported by the data marts and could also cause system imbalances and performance bottlenecks. The transparent gateway mechanism (DBLink) provided by the GBase 8a MPP database effectively solved this problem. With DBLink, the eight data marts now obtain daily interface data from the primary database in approximately one hour in total, while ensuring high availability of data transmission.
DBLink Transmission Diagram
Currently, the big data platform has integrated source data from over 90 business systems. It provides data services to more than 20 applications, including asset-liability decision support, pricing management, fund and FTP management, credit card customer value analysis, credit card data platform, payment information statistical analysis, e-banking reports, customer relationship analysis, unified customer view, information sharing platform, and data information reporting system. It supports 130+ analytical scenarios across 33 business lines within the bank.
Value
As the core architecture of ABC's big data platform, GBase 8a MPP Cluster delivers massively parallel complex data processing capabilities with excellent price-performance, enabling the bank to build a unified view of business data at the petabyte scale and providing timely, efficient analytical results.
Its high performance and highly scalable architecture ensure the platform can incorporate a broader range of business data, meeting analytical demands for marketing, internal management, and regulatory compliance.
Its high compression ratio enables efficient storage and management of massive data sets in a compressed state, further reducing the total cost of building a data warehouse.
Its high reliability and availability, including the world's first active-active cluster at petabyte scale, have enabled ABC's big data platform to operate continuously and stably for over 1,000 days.