How Big Data Drives IT Architecture Transformation – A Recap of the Massive Big Data Workshop on Big Data Storage

On August 14, the 12th event of the Massive Big Data Research Club, the Big Data Storage Session, was held at the Tianjin Hebei District Science and Technology Investment Promotion Service Center. GBASE was invited to participate, and Huang Dongxu, pre-sales manager of GBASE's analytical database GBase 8a, served as one of the keynote speakers. Mr. Huang is familiar with in-memory database technology and MPP database technology, and has extensive project experience in finance, government, security, and other industries. He delivered a keynote speech titled "Big Data Drives IT Architecture Transformation."

With the advent of the big data era, traditional data architectures can no longer meet the demands of building big data platforms. Therefore, big data is driving the transformation of data architecture, shifting from a single data management model to a tripartite structure: traditional databases (OldSQL), new databases (NewSQL), and Hadoop (NoSQL). 
 

Traditional technology architectures, based on SMP technology, have limited scalability, and row-based storage performs poorly when processing big data, failing to adequately meet the 4V characteristics of big data (Volume, Velocity, Variety, Veracity). Therefore, they are not suitable for processing big data.  

New databases (NewSQL), on the other hand, adopt an MPP architecture, greatly expanding data management capabilities. Hadoop is mainly oriented toward Internet big data, managing semi-structured and unstructured data, and excels at batch processing and complex MapReduce tasks.  

Mr. Huang's speech, set against the backdrop of new MPP databases, introduced some new technologies to adapt computing performance in big data environments, such as columnar storage, compression, intelligent indexing, scalability, distributed high availability, data distribution, MPP parallelism, and HashJoin, and compared them with corresponding functions in traditional databases. In particular, he emphasized the differences between new and traditional databases on key database application issues, such as normalization design, distributed transactions, and data models. He also shared his views on some paradoxes encountered when building big data platforms, such as the use of primary keys, data synchronization issues, concurrency issues, and data distribution issues.  

Finally, Mr. Huang predicted the future trend of big data IT architecture: a hybrid architecture. In a single data platform, multiple data architecture technologies are integrated, with each architecture technology managing its own portion of data and handling the applications it excels at, based on differences in data characteristics and application features. Such a hybrid big data platform composed of different architectures can address diverse big data analysis scenarios, manage all data, and support all applications.