Shandong Mobile Hybrid Architecture Big Data Platform Transformation Project

1 Project Background

With the continuous expansion and deepening of China Mobile's various businesses, the construction of business analysis systems and big data thematic analysis platforms will provide more comprehensive, in-depth, and efficient data for operational decision-making. In this context, operators are continuously promoting the integration of data across BSS, OSS, and MSS domains. Traditional business analysis systems and big data platforms thus need to handle larger data volumes and workloads.

This project addresses the mounting pressure on Shandong Mobile's business analysis data and various thematic analysis scenarios. It implemented a cloud-based transformation of the big data platform based on x86 PC servers. Through the hybrid integration of a Hadoop-based ETL platform and an MPP data warehouse platform, it successfully supports the storage of historical massive data on Shandong Mobile's big data platform and meets the needs of various big data thematic analysis processing.

2 Requirements Analysis

(1) Large and Rapidly Growing Data Scale

72 million active users; the system needs to collect traffic analysis-related data. Average daily data volume exceeds 2.1 TB;
Total data scale is growing rapidly, approaching the petabyte (PB) level;
As a future trend, the system's computing and storage capabilities face continuous scaling demands.

(2) Improving Existing System Response Speed

High concurrency and heavy load require an urgent improvement in response capability under highly concurrent conditions;
Improve join computation capabilities between wide tables, such as those containing user tags;
Shorten the data production time from source data extraction to report layer generation.

(3) High Data Availability Requirements

For various business analysis processes such as daily, monthly, BA, reports, and data mart processing, delays caused by system failures must not exceed one working day;
For the platform architecture based on x86 PC servers, the system must demonstrate high fault tolerance and self-healing capabilities.

3 Solution

The overall system architecture adopts a hybrid structure combining a Hadoop ETL platform with an MPP primary business analysis database, processing and storing cross-domain data from the BSS/OSS/MSS domains.

The MPP cluster environment in this project is divided into three parts based on functional requirements: a 26-node "Original BA Database", a 72-node "New BA Database", and a 40-node "Big Data Platform". Data is first structured, cleansed, and lightly aggregated on the Hadoop ETL platform. Then, through the data distribution tool of GBase 8a MPP Cluster, the output from Hadoop ETL is loaded into the MPP distributed database. As the foundational data platform for the business analysis system, GBase 8a MPP Cluster handles data processing from the data layer to the shared layer and transformations between layers. Finally, the highly summarized results in the primary BA database are imported into an Oracle traditional database, which serves the data to existing report and graphical application interfaces.

4 Value Delivered

Achieve deep and refined business analysis: Efficient data analysis capabilities help customers tackle scenarios with high complexity, high efficiency, and real-time requirements, effectively manage massive data, and conduct multi-dimensional in-depth analysis to accurately mine data value. This enables themed applications such as social circle and re-subscriber identification, WLAN near-real-time marketing, and CI self-service analysis;
Hybrid architecture supports massive data: Through distributed computing and storage and the Hadoop + MPP hybrid structure, massive data is effectively supported;
Low investment, high efficiency: GBase 8a MPP Cluster runs on low-cost x86 PC servers. The new system's performance is comparable to the original system (both have an execution time of approximately 10 hours), while the overall cost is reduced to one-tenth of the original.