Agricultural Bank of China Data Warehouse Project – Massive Data Complex Computation and Processing

As business analytics demands grew and required broader data scope and longer cycles, the existing statistical analysis and other systems at the Agricultural Bank of China (ABC) headquarters reached their performance limits. Their scalability constraints hindered support for ever-expanding requirements. Processing performance also limited the ability to integrate data from multiple business systems, further compromising cross-business data analysis and business processing. As the bank with the largest depositor base in China, ABC needed to build a cost-effective, reliably stable big data integrated business platform with an architecture capable of sustaining extreme future data growth.

As a typical data-intensive organization, Agricultural Bank of China (ABC) depends heavily on data. First, data is the core of its digital infrastructure, critical to ensuring the bank’s daily operations, placing stringent requirements on the stability and security of database systems. Second, data is a valuable asset, with volume growing rapidly; the new data warehouse must have the ability to process, compute and manage new data in a timely manner. Its key requirements are outlined as follows:

(1)    Address the challenge of fast computation and management of massive data:

l  A parallel database product capable of supporting petabyte-scale data, database tables with over a trillion rows, and handling more than 4,000 complex jobs per day (Note: The performance specifications listed here reflect the original requirements at the project’s inception in 2014. The data warehouse now deployed far exceeds these benchmarks).

(2)    The system must be flexible and scalable:

l  The system should offer continuous linear scalability and a high data compression ratio, ensuring that it can scale linearly as data grows.

(3)    Security and stability:

l  The data warehouse must be secure and stable, with 24/7 uninterrupted service to ensure the stable operation of upper-layer business systems.

l  The MPP database must provide robust backup and disaster recovery capabilities, guaranteeing data safety and eliminating any risk of data loss due to failures.

(4)    Open and reliable:

l  The solution must be built on open x86 and Linux platforms.

l  The vendor must have a strong local support team capable of providing timely and comprehensive services.

With a forward-looking technology roadmap, the bank strategically adopted a converged architecture combining GBase 8a MPP Cluster and Hadoop. Built upon China-developed and open-source technologies, this setup establishes a full-stack data service architecture, achieving independent innovation, security, and full control over data applications.

Agricultural Bank of China's Head Office big data platform was initiated in August 2013, with the main database going live in 2015. By 2019, the platform was completed, forming a main database coupled with eight data marts. The platform's roadmap envisions a next-generation big data capability system — reinforcing the data foundation, building operational systems, and enabling comprehensive data empowerment across the organization.

The system includes the BDS/GDS main databases and the eight data marts, which collectively form the bank's big data foundation. It provides data support for the data middle platform's service layer, AI, and BI platforms. GBase databases serve as the critical MPP infrastructure, primarily used for structured data model storage and processing. The big data platform hosts over 70% of the bank's data. The main GBase database in the platform manages nearly 100 PB of data across more than 3,000 nodes, while the Hadoop cluster runs over 1,500 nodes.

Beyond the main big data platform, the largest-scale GBase deployment is for branch-level data marts, featuring 10 GBase clusters with hundreds of nodes and a capacity exceeding 10 PB.

The Regulatory Reporting Data Service Cloud System, integrated with the GBase Cloud Data Warehouse, achieves effective isolation and elastic scaling of storage and compute resources, boosting online concurrent processing capabilities.

1. The project broke the monopoly of foreign integrated hardware-software data warehouse appliances.

2. Thousands of nodes have been deployed in clusters, with over a hundred production clusters managing more than 25 PB of raw data. The solution fully meets high-performance requirements, and the dual-active cluster technology ensures financial-grade high availability. It is the largest data warehouse and big data platform in China's financial industry.

3. In 2018, the big data platform won the First Prize of the Banking Technology Development Award from the People's Bank of China. The project’s success strongly demonstrates that domestically developed technology can safeguard national financial information security.

4. By using fully homegrown technologies, the customer’s construction cost was reduced from hundreds of millions to tens of millions of RMB.