Environmental Big Data Platform
Case Background
This case comes from real user applications in a large environmental regulation and economic management department. After years of centralized environmental data management, its data management center has achieved centralized and unified management of over 300TB of environmental data. However, the databases of different specialized business domains vary in construction progress and standards, and are not interconnected. There is an urgent need to integrate specialized environmental databases and consolidate them with the comprehensive environmental database, unify data dictionaries, service interfaces, query and retrieval, and collaborative presentation, to form a comprehensive database covering all environmental business domains. This enables integrated management, end-to-end control, and efficient collaborative applications of environmental data from various sources, enhancing the overall service effectiveness of environmental data.
Case Overview
The Environmental Big Data Platform enables efficient convergence of environmental business data resources, integration of the comprehensive environmental database, and analysis and sharing based on the comprehensive database. It features a "three horizontal and two vertical" overall architecture, relying on the environmental data standard specification system, and carries out platform construction at three levels: access and convergence, resource integration, and data services. This streamlines the entire "convergence - integration - service" process for various environmental data resources, and establishes a comprehensive supervision and real-time monitoring mechanism covering the full data lifecycle to ensure the quality and healthy operation of data resources across the entire system.
Environmental Big Data Platform Architecture Diagram
Environmental Basic Data Convergence Platform
The data convergence layer is responsible for the access and convergence of structured data, unstructured data, and spatial data from various environmental business systems, and provides unified supervision over the convergence process status.
Environmental Big Data Resource Platform
The data storage and integration layer uses mirroring and synchronization to store all converged data. It performs intelligent analysis and integration based on business requirements and data application scenarios, establishing correlations between structured and unstructured data to build a comprehensive database covering all environmental business domains.
Data Comprehensive Supervision Platform
This platform comprehensively supervises data convergence and integration. It achieves full lifecycle management of various environmental data resources within the system, ensuring traceable data provenance, monitorable data convergence, and assessable data quality, while providing data support for higher-level big data analysis and visualization platforms.
Environmental Comprehensive Data Service Platform
The data service layer provides external data users with various data access methods and manages the full lifecycle of data access services uniformly, ensuring data services are secure, reliable, and efficient.
Implementation Results
83.6 billion data records generated in the database;
80 computing nodes deployed;
Nearly 70 billion rows in a single table;
Total data processed: 360.2TB;
45 types of raw data files processed by raw data file templates;
155 types of standard data files loaded;
286 ETL scheduling tasks, processing 112,000 rows in real time per day;
A comprehensive environmental data standard dataset, comprehensive dataset, and feature dataset have been basically established.
Case Value
The implementation of this case established a unified, open, and compatible comprehensive environmental database, enabling dynamic collection, processing, management, analysis, and sharing of various environmental information data. It created an efficient ecological chain of "data-information-knowledge-value" integration and utilization for environmental information resources, significantly enhancing the processing, management, analytical mining, and open sharing service levels of environmental information resources. This holds strategic significance for improving scientific environmental management and decision-making capabilities for major environmental matters, expanding the capacity to safeguard environmental economic interests and the security of related areas, strengthening environmental forecasting and disaster prevention and mitigation capabilities, and advancing environmental information technology innovation.