Data Loading Features and Tuning for GBase 8a (Part 1)
This article will explain the loading function module of the GBase 8a database cluster from both principles and architecture across three chapters, providing loading performance parameter tuning solutions and Kafka data source loading and exporting solutions, to offer a reference for users to understand and use the GBase 8a loading functionality.
Features
GBase 8a cluster loading supports common data sources such as local files, FTP/SFTP, HTTP/HTTPS, HDFS, Kafka, S3; supports loading using plain text, Avro/ORC (not released) file formats; supports compressed formats gzip, snappy, lzo; supports normal text, fixed-length, and lax loading modes; supports Kerberos authentication, multi-Namenode high availability; supports multi-table, single-table parallel loading, and parallel loading from multiple data sources; supports multi-point transmission, multi-node parallel data parsing, and single-node multi-thread parallelism.
Architecture and Principles
The architecture diagram of the GBase 8a loading function is as follows:
1: The data loading SQL is dispatched; gcluster receives the SQL task.
2: gcluster parses the URL to generate a specific list of source data files; gcluster evenly distributes the source data files among the gnode loading nodes. For example, for a 15GB data file distributed to 3 loading nodes, gcluster performs data splitting, assigning each loading node a 5GB loading task; after splitting, gcluster dispatches the loading SQL tasks to the gnode loading nodes.
3: After receiving the loading SQL task, gnode reads the specified data from the file server; gnode parses the loaded data; distributes valid data into the corresponding table shard DCs based on table and hash column; gnode forwards the DC compressed files to the corresponding primary shard node; the primary shard gnode assembles the received DC files and forwards them to the replica nodes in real time.
Loading Performance Tuning and Related Parameters
Loading Performance Tuning
GBase 8a uses multi-point transmission technology; every node in the cluster participates in data parsing and loading, and loading performance can scale linearly with the number of cluster nodes. Through loading performance tuning, a speed of 33TB/hour can be achieved using only 8 servers.
Loading Performance Related Parameters
gcluster_loader_max_data_processors
Description: Number of parallel loading machines for a single load task.
Default: 16
Tuning value: In scenarios with high loading concurrency and many cluster nodes, it is recommended to configure 4~8.
gbase_loader_parallel_degree
Description: Parallelism of a single load task executed on the data node.
Default: 0, which uses half of the CPU cores; if set larger than the thread pool size, the thread pool size is used.
Tuning value: Configure as 4~6. Can be set via SET command, or specified with PARALLEL in the loading statement.
_gbase_dc_sync_size
Description: Size of data file written to disk.
Default: 1TB
Tuning value: When the data node disk I/O is busy, loading performance is slow, and CPU utilization is low, this parameter can be reduced; recommended 10M.
gbase_parallel_max_thread_in_pool
Description: Total number of threads in the thread pool, used to configure the thread pool size.
Default: 2 times the number of CPU cores.
Tuning value: When deploying 1 gnode per server, it is recommended to configure this as 4~8 times the CPU cores.
gcluster_enable_serial_load、gcluster_serial_exec_query
Description: Limit task concurrency.
Tuning value: After enabling the gcluster_enable_serial_load parameter, each gcluster node can dispatch up to gcluster_serial_exec_query SQL tasks (select, insert select, load, etc.), mainly controlling the number of SQLs sent to gnode during concurrency; exceeding tasks will be queued.
gbase_loader_max_line_length
Description: Maximum line length.
Default: 4M
Tuning value: When there are lines larger than 4M in the data file, the loading task will abort with an error; increasing this value allows the task to skip the line and continue loading, with data exceeding 4M saved in errdata.
gbase_loader_read_timeout
Description: Data file read timeout, timeout setting for reading FTP/HTTP/SFTP files.
Default: 300 seconds; 0 means unlimited.
Tuning value: When the cluster load is high or the data source I/O or network is poor, increasing this parameter can avoid timeout errors while reading from the data source.
Data source server parameters
Such as FTP/SFTP concurrent access count, port number, etc., analyzed on a case-by-case basis.
Data source concurrency evaluation
When the maximum number of concurrent loading tasks in the cluster is N, and the maximum number of loading machines per load task (max_data_processors) is M, the recommended concurrency of the data source is not less than M*N.
Chunked Loading
Chunked Loading Principle:
gcluster file splitting represents multiple logical chunks via URLs, which are loaded in parallel by multiple data nodes; the file server does not actually split the file.
Loading is processed line by line; each data node loads its assigned logical block data line by line, excluding data at the beginning that belongs to the previous node (before the first newline), and reading complete data lines at the end (up to the first newline of the next node).
Loaded Data Volume:
The amount of data that can be read and processed in a single load is related to the following parameters:
gbase_loader_buffer_count
Description: Number of memory blocks used for loading. The data node internally maintains a memory linked list.
Default: 16
gbase_loader_line_length
Description: Size of each memory block. The data node reads 4M of data content from the data source at a time and places it in the memory linked list for subsequent processing.
Default: 4M
Chunked Loading Use Scenarios:
For different usage scenarios, chunked loading configuration can be flexibly adjusted:
For scenarios with few loading nodes and a large total data volume, gcluster evenly distributes the total data of a load task among the loading nodes for parallel execution, significantly improving loading performance.
For scenarios with many loading nodes and a small total data volume, to avoid too fine granularity of data chunking that degrades loading performance, tuning can be done with the following parameter:
gcluster_loader_min_chunk_size
Description: Sets the minimum granularity for large file chunking, range 4M–128M, equivalent to the loading SQL parameter MIN_CHUNK_SIZE.
Default: 64M, meaning files smaller than 128M are not split.
In the following scenarios, data files are not split during loading:
1. Specifying the NOSPLIT parameter in the load statement disables chunked parallelism;
2. The load statement includes having lines separator or format 5;
3. The loaded file is a .gz or .snappy compressed file; it is recommended to package smaller files.
After disabling chunked parallelism, each data node processes one file, as shown.