Meteorological Radar Data Sharing Platform Real-time Transmission System Project

Project Background

In accordance with the Meteorological Radar Development Special Plan (2017-2020) and its Implementation Plan, the National Meteorological Information Center is building an efficient distributed real-time historical integrated data storage and management system at both national and provincial levels to meet the demands of massive data access and analysis timeliness. The real-time data transmission system is a critical component of the radar data sharing platform, effectively advancing the informatization and intensification of national and provincial meteorological radar data sharing and meteorological operations. The project must be launched as soon as possible.

Business Requirements

High Availability: Meteorological data is continuously collected, requiring the distributed database cluster to operate 24/7 with an annual cumulative downtime of less than 3 hours and each individual outage not exceeding 30 minutes.

Compatibility: Migrate historical data from existing national and provincial environments (mainly Oracle and MySQL databases) to an analytical database, while supporting application migration and integration.

Construction Requirements

The National Meteorological Information Center and 31 provincial nodes will deploy mature, cost-effective distributed databases to store long-term historical data, supporting climate monitoring, prediction, and decision-making services with data access and analysis.

Complete the procurement, installation, deployment, joint tuning, and application integration support for the national and provincial distributed database software.

The meteorological big data platform realizes unified management and service of complete data sets, providing platform computing services for processing and mining analysis of all meteorological data products except numerical forecasting. It directly supports applications and drives the comprehensive development of the "cloud + endpoint" business model, shaping an intensive, standardized, and open new meteorological landscape.

Based on the GBase 8a MPP Cluster database from General Data Technology Co., Ltd., the project synchronizes existing meteorological observation data, geographic information data, industry and social data, and other basic data in real time into the GBase 8a database via Kafka, providing long-term historical data storage and multi-dimensional analysis and computation. It offers a unified database access interface, enabling transparent access to heterogeneous database engines, which facilitates unified data management as well as application migration and development.

The deployment scale consists of three clusters with 51 nodes at the national administration, and a total of 286 nodes across 31 provincial administration clusters. Specifically, the national main system has 26 nodes, 25.4 TB of total data, 2,410 tables in the database, a daily data increment of approximately 30 GB, and a maximum of 170 concurrent sessions. The backup database has 17 nodes, 36.2 TB of total data, 3,610 tables, a daily incremental data volume of about 40 GB, and a maximum of 157 concurrent sessions.

Figure 1: Overall Architecture of the Meteorological Radar Data Sharing Platform

This project consolidates meteorological data and algorithm resources, moving algorithms closer to the data. This eliminates the redundancy caused by repeated calls, storage, and transmission across legacy systems. It serves as a key driver for meteorological agencies to embrace the big data era, modernize operational systems, and optimize end-to-end workflows. As the supporting database, GBase 8a provides robust backing for upper-layer data services.

Unified Full-Data Management and Services: Achieve unified management of all meteorological data types, delivering platform-based computing services for data processing and mining analysis beyond numerical weather prediction.

Real-Time Data Synchronization: Meteorological data enters transactional databases to support real-time operational queries. Via Kafka, incremental data from the transactional store is synchronized to the GBase 8a analytical database in near real time. Leveraging GBase 8a’s storage, service, and analysis capabilities maximizes data value, safeguarding weather forecasting, disaster prevention, and mitigation operations.

High-Performance Analysis: Harness the high-performance, distributed parallel computing power of the GBase 8a cluster to dramatically boost performance for complex analytical workloads in meteorology.