GBase 8a MPP Cluster and Hadoop Hybrid Solution
Yang Weiwei: Good morning, distinguished guests. It is a great pleasure to have the opportunity to discuss the application of big data in the telecommunications industry with you today, and I look forward to your insights.
Yang Weiwei, Deputy General Manager of Product Marketing, GBase
My topic today is a big data platform built on a hybrid architecture of MPP database and Hadoop. If the strategic data assets held by China's three major telecom operators are a precious gold mine, then a hybrid-architecture big data platform serves as the mining machine that helps enterprises rapidly uncover the intrinsic value of their data. I will divide my presentation into three parts. The first part explains the characteristics of various technical architectures in the big data era. The second part introduces the solution that combines GBase 8a MPP Cluster with Hadoop, covering platform architecture, data flow, and unified management. The final part uses real-world cases to demonstrate how the hybrid solution is applied in projects, what problems it solves, and ultimately what value it brings to enterprise customers.
Characteristics of Technical Architectures in the Big Data Era
From a macro perspective, big data is divided into big transaction data and big interaction data. Big transaction data primarily serves enterprise users, while big interaction data mainly serves mobile and internet end-users. These two types of data are not isolated—they are interrelated. Big data inevitably requires integrating and fusing both types to form a unified processing and analytics platform. From an application standpoint, big data is applied across industries: detecting financial irregularities in banking, identifying social circles in telecommunications, and recognizing online customers in internet marketing, among others. A big data platform must be able to analyze the data generated by these applications using certain technologies.
What technologies do big data processing platforms employ to analyze the massive data from applications? The answer boils down to MPP databases and Hadoop. MPP databases have inherent advantages: they support the relational model, use standard SQL, and are well-suited for big data analytics, often deployed in data warehousing and large-scale data analytics scenarios. Hadoop, on the other hand, based on key-value storage and MapReduce/RDD algorithms, excels at storing semi-structured and unstructured data and is mainly used for simple, batch query workloads. More specifically, the technical strengths of MPP databases lie in supporting complex associative queries and stored procedures, making them suitable for building cost-effective data warehouse platforms. They run the same SQL as traditional databases, handle large data volumes, and deliver fast response times. However, MPP databases are not strong in certain areas, such as processing unstructured data or implementing functions that require flexible, custom-developed algorithms. Hadoop's advantages reside in massive-scale, simple batch query jobs and the ability to customize algorithms through MapReduce, offering great flexibility. Yet Hadoop also has its limitations. While it enables flexible algorithm development, the resulting plug-ins are often highly customized and lack generality. Furthermore, trying to superficially simulate SQL for complex associative queries and real-time jobs on a Hadoop platform often yields half the result with twice the effort. As many of you know, in the telecommunications and financial sectors, over 90% of database applications are still developed using stored procedures. Therefore, a single traditional architecture cannot meet the demands of the big data era. The future big data platform will undoubtedly adopt a mix of technologies, each playing to its strengths, to deliver a holistic big data solution to customers.
A big data platform must embrace a divide-and-conquer data processing strategy, integrating traditional transactional databases, new MPP databases, and Hadoop, leveraging the strengths of each architecture to form a unified platform that ultimately serves enterprise users.
GBase 8a MPP Database and Hadoop Hybrid Solution
This section introduces the solution through its system architecture, data flows, and unified access management.
First, let us look at the system architecture. The hybrid solution consists of three layers: ETL layer, data processing layer, and unified access management layer. The ETL layer handles various types of data and can use either traditional ETL tools or cloud-based ETL tools built on Hadoop. The data processing layer stores the data preprocessed by the ETL layer and performs deep data processing. In this layer, both a data warehouse built on MPP databases and a co-processing library built on Hadoop can be employed to support the overall data processing.
Data must be exchangeable between the MPP database and Hadoop through certain mechanisms. The unified access management layer coordinates job scheduling between the MPP database and Hadoop, while providing standard, unified access interfaces for front-end applications. From a data organization perspective in the hybrid big data platform, the bottom-layer data sources supply all kinds of data—structured, semi-structured, and unstructured. Typical examples are data from the B, O, and M domains in telecommunications. The ETL layer cleanses and further aggregates this diverse data. The preprocessed data must be rapidly loaded into the data processing layer, which performs complex statistical analysis, associative queries, and data mining to deeply process the ETL output, serving front-end applications such as reporting, analysis, querying, and log-based applications. Simultaneously, the data processing layer must support mutual data backup and recovery with the backup repository built on Hadoop.
How does data exchange between the MPP database and Hadoop occur in the data processing layer? A high-speed interface channel enables data transfer between them. This involves several categories of data. First, for online data, a standard interface must allow data exchange between the MPP database and Hadoop. For offline data, the highly efficient parallel loading tools native to each platform can be used. Additionally, many third-party tools can facilitate data exchange, such as Sqoop from the Hadoop ecosystem, which can enable real-time data interchange. A hybrid-architecture big data platform must provide standard access interfaces for front-end applications. GBase's hybrid big data platform uses unified access management to coordinate job scheduling between the MPP database and Hadoop while offering a unified external interface for the application layer. Applications can decide, based on their characteristics and data association requirements, which tasks to send to the MPP database and which to the Hadoop platform.
Let me give an example to illustrate how the hybrid big data platform processes application-generated data, taking user location trajectory data in telecommunications as an example. We know that telecom networks generate tens of terabytes or more of signaling data daily. The data first lands in the Hadoop platform, which leverages its powerful text processing capabilities for aggregation, grouping, and sorting. The processed data is then rapidly loaded into the MPP database, where it is associatively queried with stored user profile information to ultimately generate customer-tag data for location trajectories. The entire application workflow can be written using standard interfaces.
This is the hybrid MPP and Hadoop big data platform I have introduced. From the above, we can conclude that the hybrid platform leverages the strengths of MPP databases—flexible SQL writing and the ability to handle complex statistical analysis logic—while incorporating Hadoop's advantage in fast querying. The combined strengths broaden the scope of supported applications and significantly boost the overall data processing performance of the big data platform. Because the hybrid platform fuses two distributed systems, it offers flexible system design and excellent scalability.
Application Cases of the Hybrid Architecture
How is the hybrid technical architecture applied in real-world projects? The first example is a cloud-based business intelligence system for a provincial telecom operator. Let us review the problems the system faced: data volume was exploding—the online user count reached 72 million, daily data volume averaged nearly 3 TB, and total data volume had exceeded 500 TB and was growing rapidly. In addition, query responses were slow, and traffic analysis support was not granular enough to achieve event-level response, making it impossible to meet the goal of refined support. Based on the analysis of these challenges, we proposed building a clustered cloud architecture on an open x86 platform, using a big data processing platform built on Hadoop and MPP database to solve the project's problems. The system architecture is quite generic and comprises several layers. The key highlight is that the underlying data processing platform uses an ETL tool built on Hadoop for data validation, cleansing, associative queries, and further processing. The MPP database handles the preprocessed data, so the vast amount of structured data is processed there. The processed data then supports applications such as KPI reports, first-tier and second-tier business analytics, and tag libraries. For implementation, the system deployed 62 open x86 servers, with 26 cluster nodes for the MPP database. It ingests up to 500 TB of data and executes nearly a thousand business jobs daily. The Hadoop platform in the data center uses 36 cluster nodes, processing 3.5 TB of data per day. In terms of hardware cost alone, the entire platform investment is in the order of millions (RMB), while the legacy system relied on minicomputers. The overall hybrid big data platform reduced hardware costs to one-tenth of the original system, while maintaining roughly the same overall product performance.
The second case concerns the data warehouse platform of a major state-owned bank. This system faced similar challenges: first, analytical applications encountered performance bottlenecks; second, as data volumes grew, the demands from front-end applications and business units could not be met in a timely manner; and third, growing data meant that valuable historical data and current data were underutilized. Through system analysis, we also adopted a hybrid MPP database plus Hadoop architecture. The overall logical architecture is fairly standard. The data loading layer uses a data exchange platform built on Hadoop for data cleansing, quality checks, processing, and aggregation. The storage layer uses the MPP database to perform overall data processing; within this layer, the MPP database builds both a data warehouse and topic-oriented data marts, effectively supporting applications such as credit card analytics, internal auditing, and electronic banking. For implementation, nearly 100 x86 PC servers were used. The storage management layer currently employs 28 MPP database nodes, 4 loading machines, and 2 10GbE switches. The system processes over 4,000 complex jobs daily, with actual stored data reaching 460 TB. The largest table exceeds 100 billion rows and grows by 2 billion rows each day. Data extraction uses a Hadoop-based platform with 56 nodes and 4 10GbE switches for interconnection, handling both underlying processing and data backup. The system went live in January 2014.
From these two representative cases, we can deduce that the MPP database and Hadoop hybrid architecture big data platform delivers the following values:
• First, high performance: In the hybrid solution, MPP database and Hadoop each play to their strengths—Hadoop handles the ETL process, while the MPP database takes on the heavy lifting of structured data and deep, complex analytics, thereby significantly improving the overall platform response efficiency.
• Second, low cost: Both cases were built on low-cost, open x86 PC servers, effectively saving hardware expenses.
• Third, scalability: The hybrid solution provides excellent system scalability by shifting from the traditional vertical scale-up model to a horizontal scale-out model, ensuring business continuity during expansion.
Through this presentation, I hope to engage in discussions with all the guests and experts present. Facing the big data demands in the telecommunications industry, the MPP database and Hadoop hybrid platform will play a significant role.
That concludes my presentation. Thank you.