Design of Meteorological Big Data Storage System
Recently, the 25th China International Software Expo & China Database Industry Summit, hosted by GBASE (General Data Technology Co., Ltd.), was held in Tianjin. At the summit, Xu Yongjun, Chief Expert of Meteorological Science and Technology at the National Meteorological Information Center, delivered a keynote speech titled "Design of Meteorological Big Data Storage System".
Background
To meet the needs of meteorological operations such as weather forecasting, climate prediction, decision-making services, and public services for massive meteorological data storage and services, and to achieve the goal of intensification of meteorological operations, the National Meteorological Information Center designed a meteorological big data storage system based on the characteristics of business data applications and combined with various distributed storage technologies.
By standardizing the data storage structure and utilizing key technologies in data synchronization, tiered storage, and data service interfaces, the problem of unified management and service of meteorological data among distributed heterogeneous storage technologies was solved. Application practice has shown that the meteorological big data storage system has a storage capacity of 30PB, an efficient data tiered management process, and a unified data service response time within 1 second, capable of directly supporting meteorological operations.
Design of Meteorological Big Data Storage System
Xu Yongjun said that the development history of China's meteorological industry is also a history of informatization construction of the meteorological department. Every major progress in meteorological operations has been driven and supported by information technology. In recent years, with more precise monitoring and the wide convergence of multi-source data, meteorological data has continued to explode. By the end of 2022, meteorological data covered 21 industries with 2,348 categories; the data volume increased to nearly 100 PB, with a daily increment of 41 TB.
To solve the problems of data and business intensification and effectively support meteorological business systems and various data applications, at the end of 2021, according to the business layout of meteorological information network deployment at the national and provincial levels and application at the national, provincial, municipal, and county levels, the Meteorological Big Data Cloud Platform (referred to as "Tianqing") was put into operation at the national and provincial levels, opening a new "cloud + terminal" meteorological business operation model.
The core capabilities of the Tianqing system are to manage observation, forecast, service, industry, and social data, and other complete and authoritative Earth system big data in a coordinated manner, and to provide platform-based services that unify "data, computing power, and algorithms". It integrates various distributed storage technologies to achieve long-sequence online, high-concurrency, and fast access to massive data, providing access timeliness of milliseconds to seconds for meteorological applications. Based on computing frameworks such as distributed, GPU, and stream processing, it builds a "data-computing integration" processing pipeline, supporting rapid integration of business algorithms and scheduling operation according to the natural business sequence.
The Tianqing system is built on the distributed logical data warehouse GBase 8a MPP Cluster from General Data Technology (GBase). The national meteorological department has deployed nearly 100 nodes, and the provincial meteorological departments have deployed nearly 300 nodes. In the national "two sites, three centers" Tianqing system, there are 23 data tables with more than 2 billion records in a single cluster, and the largest single table has reached 126.8 billion records.
Case 1: Location-based Real-time Data Public Service System
The GBase 8a historical analysis database provides data interfaces supporting high-concurrency access. It performs statistical calculations such as maximum, minimum, and average values for 5-minute rolling 24-hour intervals, involving 128 billion records of China surface minute precipitation data with a daily increase of 80 million records; and 6.9 billion records of China surface hourly data with a daily increase of 1.68 million records. Leveraging GBase 8a's columnar storage, efficient query, and high-concurrency access capabilities, the statistical performance achieved a response within 2 seconds.
Case 2: Public Service Center Traffic Management Weather Risk Control Platform
The platform uses the analysis and statistics functions of the GBase 8a logical data warehouse to complete nationwide risk statistics at each time interval.
Currently, the "Public Service" database has a data scale of 28 billion records. Based on GBase 8a's high compression feature, the data storage only occupies 3T, meeting the requirements for efficient compressed storage. The raw data file size is about 5.8T, with a compression ratio of about 1:4. At this scale, it is estimated that it can support nearly 70 years of data storage and access.