GBASE Focuses on User Experience - Part 2: Building a Big Data Analytics and Mining System
GBASE focuses on user experience
"Providing world-class domestic databases for Chinese users" has always been GBASE's mission, and GBASE has been striving for it!
Over ten years of focus has also brought us gratifying market feedback. The following is reprinted from the January 2016 issue of "Financial Informatization" magazine, fromChina Agricultural Bank Software Development Center Zhao Weiping Zhao Cun Chao's article.
Big Data Analysis and Mining System Construction
Written by China Agricultural Bank Software Development Center Zhao Weiping Zhao Cun Chao
In the era of big data, complex human behavior becomes traceable, old production relationships and lifestyles become obsolete, and new industrial ecosystems and game rules emerge. In the face of this "data earthquake," how to effectively master the methods of collecting, analyzing, and utilizing data, turning "data" into treasure, and transforming data into productivity in business development and management has become a new topic for banks. Data analysis and mining is the "key" to unlocking the data treasure.
Data analysis and mining have four levels: answering "what was done" through reports, answering "what is happening" through customer behavior analysis, answering "what should be done" through data-driven product creation, and answering "what will happen in the future" through data prediction. How to fully utilize internal and external data wealth, awaken the value of data, and leverage data as an engine for product innovation, precise marketing, and risk management? Agricultural Bank is keeping pace with the times, boldly innovating, and exploring practices.
Development Ideas:
Four Principles, Seven Key Points
Agricultural Bank proposes a "big data strategy led by technology to drive business development, aiming to build an internally and externally refined 'smart bank,'" optimizing bank processes through big data, efficiently allocating financial resources, keenly understanding customer needs, and creating the best service experience. Through in-depth analysis of massive data, comprehensively adjust product structure and marketing models, fundamentally improve risk management, cost performance management, asset-liability management, and customer relationship management levels. The construction of Agricultural Bank's big data analysis and mining system is based on "fully leveraging the comprehensive value of big data to add a technological engine for business development and management," adhering to four principles and seven key points.
1
Four Principles
Principle One: Integration of Business and Technology with Organic Linkage between Headquarters and Branches. Data analysis and mining require a combination of business-driven and data-driven approaches, with a clear business perspective and the ability to mine potential patterns from data. The deep integration of business and technology is the primary prerequisite for the value of data to take root in bank business management. At the same time, it is necessary to promptly follow up on the urgent issues that need to be addressed in the grassroots management of branches, making data analysis and mining more targeted.
Principle Two: Mutual Promotion of Platform Construction and Exploration Practice. On one hand, through exploratory applications focused on hot issues and key directions in business areas, problem-oriented, urgent needs first, rapid application, and quick results; on the other hand, gradually explore the complete process of analysis and mining, accumulate experience, cultivate talent, train teams, promote platform construction, and continuously improve the data analysis and mining system.
Principle Three: Deep Integration of Technological Innovation and Value Creation. Innovation is the soul of big data, and technological innovation must be deeply integrated with value creation, with value creation as the goal and technological innovation as the means.
Principle Four: Virtuous Cycle of Talent Cultivation and Capability Enhancement. On one hand, cultivate data analysts who understand business operations, possess big data technical capabilities, and have strong innovation abilities, and establish a talent ladder covering business analysis, data analysis, platform construction, data services, and application development. On the other hand, strengthen capability building in management, operation, evaluation, and application of data analysis and mining, achieving scientific data mining processes, efficient data mining execution, and reasonable data mining evaluation.
2
Seven Key Points
Key Point One: Based on the Bank-wide Big Data Platform. The big data platform collects, processes, and models data from the entire bank, providing a unified customer view, product view, and transaction view for the entire bank, offering strong foundational support for data analysis and mining. At the same time, data analysis and mining, as a part of the bank-wide big data system, achieve organic integration with other applications and data services through the big data platform's transfer station.
Key Point Two: Centered on the Big Data Analysis and Mining Platform. The big data analysis and mining platform is the core technical platform of the bank-wide data analysis and mining system, integrating resources for data analysis and mining across the bank, building a data analysis cloud environment, and providing unified platform support for data analysis and mining across the bank.
Key Point Three: Providing Comprehensive Analysis and Mining Data Services. Mainly includes: multi-dimensional analysis services—business-oriented self-service "drag and drop" services; massive data mining services—deep insights, precise judgment, and development prediction services; decision support services—deep analysis of business hotspots and intelligent decision information services; customized data services—providing "tailor-made" data services for internal and external customers.
Key Point Four: Supporting Multiple Application Areas of Business Analysis. Mainly divided into four categories: customer management, operation management, risk management, and financial management, strongly supporting refined management and intelligent development in various business areas across the bank.
Key Point Five: Building a Complete Analysis and Mining Process System. Data analysis and mining is a systematic, process-oriented, scientific project, requiring the construction of a cyclical, iterative, and continuously optimized process system from data integration, analysis and mining, application implementation to evaluation and enhancement, achieving process-oriented, professional, and standardized analysis and mining.
Key Point Six: Establishing Comprehensive Management Norms and Systems. Including bank-wide unified data analysis and mining management systems, operational methods for analysis and mining platforms, evaluation standards for data analysis and mining, work guidance for the application of analysis and mining results, targeted innovation project management methods, talent assessment and incentive systems, etc.
Key Point Seven: Building a Professional Data Analysis and Mining Team. Through the cultivation of professional talent and data analysis teams, continuously enhance the comprehensive utilization capability of data value across the bank.
Figure Big Data Analysis and Mining Platform Diagram
Value Engine:
Big Data Analysis and Mining Platform
In the entire data analysis and mining ecosystem, the big data analysis and mining platform is the core, the engine that transforms data value into productivity. The platform consists of 7 layers: the data source layer obtains source data, with the big data platform being the main data source; the data processing layer performs targeted processing on various types of data to obtain the data information needed for analysis and mining; the analysis and mining layer establishes models for processed data to conduct data analysis and mining, including model selection, training, testing, and validation; the decision support layer solidifies the analysis results and runs them regularly, providing decision support information for related business applications; the analysis application layer applies the results of data analysis and feeds the results back to the decision center and mining center; the unified presentation layer provides visualization services for the analysis and mining process, and offers a unified data display platform for the results of analysis and mining and related analytical applications; the data management layer provides data access engines, data management engines, and unified analysis and mining process management and platform support services.
The big data analysis and mining platform can be briefly summarized as"Five Centers, Three Data Zones, Two Management Engines, One Process Platform, Unified Intelligent Display".
1. Five Centers
These are the "heart" of the mining platform, namely the information extraction center, unstructured data processing center, real-time stream computing center, analysis and mining center, and intelligent decision center. The first three belong to the data processing layer.
Information Extraction Center: Completes the acquisition and targeted processing of structured data, generating wide tables for analysis and mining used in data analysis and mining. Processing mainly includes three types of operations: first, summarizing, joining, and integrating data to generate indicators needed for analysis and mining; second, performing targeted transformations on data, including discretizing continuous attributes, binarizing categorical data, and transforming continuous data attributes; third, smoothing outliers that do not meet data quality requirements, including noise reduction, attribute correction, outlier analysis, and smoothing.
Unstructured Data Processing Center: Uses Hadoop technology architecture to process unstructured data inside and outside the bank, including public web information, social media information, and purchased information, supporting subsequent customer profiling, sentiment analysis, and event marketing analysis.
Real-time Stream Computing Center: Based on the storm technology framework, performs real-time stream computing processing, supporting real-time fraud prevention, real-time marketing, and other business application scenarios.
Analysis and Mining Center: Is the experimental environment for data analysis and mining. Supports analysis and mining of various types of data such as massive data, unstructured data, real-time stream data, and mixed scenario processing, supporting exploration of data models in multiple fields, algorithm selection, development testing, model training, comparative analysis, data validation, and result evaluation processes, and deploying models that meet requirements to the intelligent decision center; at the same time, iteratively optimize models in business applications, continuously improving the accuracy and effectiveness of data analysis.
In the analysis and mining center, multiple experimental scenarios are supported for multiple application themes, achieving flexible and efficient experimental window management, enabling reasonable allocation and management of data and computing resources, and maximizing the provision of more mining services. At the same time, it provides an efficient and stable computing environment for analysis and mining personnel, integrated mining algorithms and tools, common models that enhance productivity, a good user interface, and convenient operation styles.
Intelligent Decision Center: Is the operational environment for data analysis and mining. Models trained and tested in the analysis and mining environment are transformed into business rule processing here, achieving daily operation of data analysis and mining models, ultimately providing data services to headquarters and branch business personnel and related customers, and feeding analysis results back to business systems for use in production and management activities. It can be said that the intelligent decision center is the point of convergence between data analysis and mining and conventional system construction, a key link in applying the data analysis and mining process to production and management.
At the same time, the intelligent decision center must promptly collect feedback information from actual applications, thereby conducting effective post-evaluation and continuous monitoring, based on which to optimize and enhance models and retire outdated and ineffective models.
2. Three Data Zones
Analysis Data Zone: Mainly stores data from the analysis and mining preprocessing environment, providing basic data for subsequent data analysis and mining; it can also be directly provided to business personnel for multi-dimensional analysis services.
Common Model Zone: Mainly stores common method models explored and accumulated by the data mining center, building a unified, flexible, and efficient model library, and providing them to analysis and mining in a componentized manner, enhancing efficiency and accuracy.
Business Knowledge Zone: Mainly stores result information from data analysis and mining, converting it into business rules and support information, providing data services for business personnel and corresponding production systems.
3. Two Management Engines
Data Access Engine: Provides comprehensive data access services, including database access, data file access, Web Service service interfaces, etc., while efficiently achieving rapid replication and transfer of massive data, providing technical support for data analysis and mining.
Data Management Engine: Provides metadata management, data permission management, data resource allocation management, data quality management, data desensitization services, laying the foundation for data management in the data analysis and mining process.
4. One Process Platform
Refers to providing unified process management and supporting tools for data analysis and mining, including three aspects: first, unified process management, solidifying the data analysis and mining process and providing flexible and convenient workflow services; second, providing ETL tools, job scheduling, execution monitoring, and other supporting tools; third, providing user permission management, resource management, and task management services.
5. Unified Intelligent Display
Refers to the intelligent resource window, providing users with a unified, intelligent, and customized display platform, including analysis reports, ad-hoc queries, multi-dimensional analysis, and other functions, while integrating various resources.
Technological Innovation:
The Tool for Utilizing Data Value
The purpose of big data analysis and mining is to transform data into productivity, and innovation is the incubator and accelerator of this transformation process. Agricultural Bank, in constructing the big data analysis and mining system, emphasizes technological innovation, fully absorbing the latest technological development achievements, and drawing on advanced practices in the industry, especially in open service clouds, integrated architecture systems, Agricultural Bank-specific mining model libraries, rapid replication/transfer of massive data, intelligent resource allocation, dynamic mixed scheduling mechanisms, memory analysis and grid computing, application of domestic databases, and many other areas, making pioneering explorations and practices.
The biggest technical feature of the big data analysis and mining platform is "heterogeneous integration." In the data preprocessing zone, it integrates MPP architecture databases, Hadoop, stream computing Storm environments, and other technical platforms; in the data mining center, it integrates SAS environments,
based on Hadoop's Spark environment, R language, and other pioneering experimental environments; in the intelligent decision center, it integrates MPP architecture databases, Oracle databases, IQ databases, Hadoop platforms, ODM rule engines, and other technical platforms, fully reflecting the latest development trends of the internet and big data technology, fully drawing on the concept of "cloud," achieving logical centralization, physical dispersion, unified service, and intelligent management.
The big data analysis and mining platform lays a solid technical foundation for the deep application of big data value, providing scientific and comprehensive data analysis and mining services for business applications. Multi-dimensional analysis services can help business personnel achieve multi-dimensional, multi-view, multi-level analysis, and through operations such as drilling down, drilling up, slicing, and rotating, provide more dynamic and intelligent data analysis, discovering the patterns behind the data. For example, conducting product profitability analysis can perform comprehensive analysis from multiple dimensions such as institutions, time, customers, product types, channels, marketing activities, etc., effectively optimizing and innovating products. Based on the data analysis and mining platform, flexible, precise, and efficient mining services can be achieved for multiple business themes, such as customer insights and precise marketing, credit evaluation and risk assessment, sentiment analysis and customer emotion management, etc. Keeping up with market development dynamics, directly facing business hotspots and difficulties, fully mining the huge value of big data, providing deeper customer insights and strong support for business development and management decisions.