GBASE Financial Application Guide 6 | Data Integration
To help financial institutions select distributed analytical database products and promote successful deployment of such databases in the financial industry, GBASE, under the guidance of the Beijing FinTech Industry Alliance, has compiled the GBase 8a Financial Application Guide. The guide thoroughly covers the entire deployment process of a distributed analytical database—from selection planning, development and design standards, data security and high availability, to operation and maintenance optimization—and presents representative deployment cases of GBase 8a MPP Cluster at national policy banks and large state-owned banks.
GBASE will release a series of articles sharing and interpreting the guide, hoping to provide reference for financial users’ database selection and to drive the efficient implementation and high-quality development of technology finance. This is the sixth installment, which introduces system data integration.
Data integration is a critical step in building a data warehouse. It involves continuously centralizing historical and real-time data that needs to be processed into the distributed analytical database, and its performance directly impacts the overall performance of the database system.
1. Data Loading
Data loading in GBase 8a MPP Cluster is achieved by sending a load task via SQL statements and directly transferring underlying data files. This makes it easy to use and highly performant. It enables efficient input of data files exported from homogeneous or heterogeneous databases into the GBase 8a cluster.
The loading principle of the GBase 8a cluster is shown below.
Figure 1: Loading Principle of the GBase 8a Cluster
Data files to be loaded into GBase 8a MPP Cluster can be in plain text, Avro/ORC, or compressed formats (gzip, snappy, lzo). It is recommended to place them on a separate server as the data source. GBase 8a pulls data from the source, which can be configured with one or more services such as Kafka, HDFS, FTP, SFTP, HTTP, S3, or the GBase-specific gbfs service.
There is no significant difference in loading performance among these services. Choose a suitable data source service based on security, convenience, and reusability within your existing environment:
• Kafka and HDFS are cluster-based, handle large scale, store data files in shards with backups, support Kerberos authentication, and offer higher data file security;
• FTP, SFTP, and HTTP services are easy to set up; SFTP is more secure than FTP;
• S3 is typically used in cloud environments;
• gbfs is a dedicated service provided by GBase, designed with security in mind and avoiding the use of public services.
Details:
Figure 2: Loading Feature Overview
2. Data Export
When data from GBase 8a MPP Cluster needs to be sent to an application or migrated elsewhere, the data export feature can be used. The export feature corresponds to the loading feature. Based on the supported functionalities, you can choose the appropriate export mode according to business needs.
Figure 3: Export Feature Overview
3. Streaming and Real-Time Data Loading Solution
With the widespread use of online and mobile banking, real-time monitoring of bank accounting, credit, and other business data, along with OLAP analysis, has become an essential requirement. A real-time data warehouse processes and analyzes data in real time, ensuring the data is always up-to-date and accurate while responding to user queries and analysis demands instantaneously. Compared with traditional data warehouses, a real-time data warehouse places greater emphasis on data freshness and the ability to respond to business in real time.
Streaming data includes real-time requirements; it refers to data that flows in continuously, such as user behavior data streams and machine data streams. Streaming data must be ingested in real time. If processing efficiency falls behind the data inflow rate, data backlogs occur, leading to degraded system performance.
GBase 8a products provide solutions for real-time and streaming data:
Real-Time Data Warehouse Construction Solution
GBase 8a's real-time data warehouse application solution leverages self-developed ETL tools or third-party ETL tools (open-source/commercial) to build real-time and streaming data pipelines.
Figure 4: GBase 8a Real-Time Data Warehouse Deployment Architecture
You can select different ETL tools based on real-time requirements. The ETL tools supported by GBase 8a products are shown in Table 1.
Table 1: ETL Tool Classification
For reading real-time and streaming data, GBase 8a MPP Cluster provides the following two tools:
1) The built-in Kafka Consumer for the GBase 8a cluster, which receives data in real time from Kafka clusters;
2) The GBLoad Server tool, which receives streaming data transmitted via API (JDBC).
Real-Time Data Ingestion via Kafka Consumer
GBase 8a products have a built-in Kafka consumer whose main function is to synchronize data from Kafka to the 8a cluster.
Figure 5: Kafka Consumer Feature Overview
GBLoad_Server Tool for Streaming Data Batch Ingestion
GBLoad Server receives INSERT VALUES data via JDBC, accumulates them into batches and writes them as text files, then uses the loading feature to ingest data via the FTP protocol. Specifics:
Figure 6: GBLoad_Server Feature Overview
4. DBLINK Cross-Cluster Data Access
Applications of the GBase 8a cluster can access external data sources via DBLINK. DBLINK can be used in the following scenarios:
• Data from homogeneous/heterogeneous sources needs to be extracted into the 8a cluster using business SQL;
• Data within the 8a cluster needs to be pushed to other homogeneous/heterogeneous clusters using business SQL;
• Data from homogeneous/heterogeneous sources requires inter-access and associative operations via business SQL.
DBLink feature overview:
Figure 7: DBLink Feature Overview