Transcript | Decoding the Big Data Fight Against COVID-19: Finding Close Contacts in Milliseconds
The novel coronavirus is highly contagious and has a long incubation period. Quickly identifying the movements of infected individuals to cut off the source of infection and break the chain of transmission became a top priority in epidemic control. To support epidemic prevention and the resumption of work and production, health QR codes and other application platforms were rapidly developed and deployed. However, these applications all rely on powerful databases as their foundation.
Tianjin-based database developer General Data Technology Co., Ltd. (GBASE) is the unsung hero behind the scenes. Together with software partners, its database products can perform calculations on hundreds of billions or even trillions of records in an extremely short time, locating information on close contacts in just seconds, or even milliseconds. How is such massive computation achieved? In a database field long dominated by foreign giants, how did GBASE emerge as a dark horse to achieve domestic substitution?
▲ Exterior view of General Data Technology Co., Ltd. (GBASE)
Big Data Empowers Precise Epidemic Control
Finding Close Contacts Within Milliseconds
During the epidemic, every confirmed case involved close contacts, and tracing them was a highly complex task—especially for those with travel history. If we relied solely on public notices and passenger self-identification, information could easily be missed, and some might hesitate to come forward due to personal concerns, hindering epidemic control efforts. Therefore, quickly generating a list of close contacts from confirmed cases—who they were, which train they took, which seat—allowed health authorities to swiftly locate them and initiate epidemiological investigations. This placed extremely high demands on data completeness and the accuracy and speed of data retrieval.
▲ The powerful GBASE database system behind the 12306 ticketing system
Founded in 2004, General Data Technology Co., Ltd. (GBASE) is one of the “Four Golden Flowers” of China’s domestic database industry and a key player in advancing the independent and controllable development of national basic software. In 2017, the China Academy of Railway Sciences adopted GBASE’s GBase 8a MPP database to support the upgrade of the new-generation ticketing and electronic payment platform for the 12306 system, which handles hundreds of TBs (1TB = one trillion bytes) of data. The outbreak of COVID-19 posed new demands on the database system. Yuan Mengjia, General Manager of GBASE’s Marketing Department, said: “The COVID-19 outbreak coincided with the Spring Festival travel rush, when railway traffic peaks. As the railway’s ‘data warehouse,’ we process hundreds of terabytes of business data daily, with clicks reaching hundreds of billions during peak times. Under such conditions, providing queries on COVID-19 patients and close contacts placed enormous demands on the database.”
The database GBASE provides to 12306 stores 20 years of passenger railway travel data since 2000, amounting to hundreds of terabytes. Yuan Mengjia offered a vivid analogy: “You can think of a data warehouse as a large wardrobe with many clothes. If you want to quickly find the clothes you want to wear, they need to be neatly organized and arranged cleverly, with your frequently worn favorites placed in front. That way, you can find them fast. In this data warehouse, however, it’s not clothes but tables and data. The ticketing data comprises around 1,000 tables, with the largest table containing about 23 billion records. That’s a huge amount of tabular data. Previously, the data warehouse we designed for 12306 was aimed at managing tickets. Now, with COVID-19 tracing, the data management target has shifted to people, focusing on passengers boarding and alighting in each carriage, their travel trajectories, and so on. The management goal has changed. To understand that, imagine your previous wardrobe was organized by season—summer clothes in front, winter clothes inside. Now it’s organized by person: only your clothes, regardless of season, are brought to the front. So the clothes need to be rearranged.”
In early February, after receiving requirements from the China Academy of Railway Sciences responsible for 12306 R&D, GBASE immediately started development. Zhang Shaoyong, General Manager of GBASE’s Data Intelligence Product Division, said: “Fighting COVID-19 is a battle! We must defend our position at all costs and ensure the stability of the database, the accuracy of the data, and the timeliness of queries.”
It was the height of the epidemic, and many researchers were stuck in their hometowns and couldn’t return to Tianjin. GBASE broke down departmental barriers, urgently drawing over ten technical staff in Tianjin, including Zhang Jian, to form a task force led personally by Zhang Shaoyong.
“Due to the emergency secondment, many colleagues were not initially familiar with the product, so we learned while developing. The railway system stores enormous amounts of data. Using previous data analysis methods required querying multiple databases, and tracing the trajectories of confirmed COVID-19 patients was slow—taking several minutes or even over ten minutes. Now, to meet the specific epidemic analysis and query needs, we built a dedicated epidemic trajectory data mart. This data mart contains only epidemic-related trajectory information, so the data volume is greatly reduced, making queries and data retrieval very fast. Going back to the wardrobe analogy, it’s like taking out all your clothes and putting them into a small suitcase. There are far fewer clothes in the suitcase than in the wardrobe, so finding your clothes becomes much faster. You no longer have to search through a large wardrobe full of everyone’s clothes, pulling out items one by one. As a result, query times have been reduced from minutes or even over ten minutes to just seconds or even milliseconds, with overall query and analysis performance improved by at least ten times.”
▲ Zhang Shaoyong, General Manager of the Data Intelligence Product Division, leading R&D
After the new version was upgraded and launched online, its stability had to be ensured. To this end, over a dozen engineers at the company took turns working 24/7.
“To race against the epidemic, we launched this new version rapidly. For many tasks that couldn’t be automated at the time, we did them manually, such as manually cleaning disks and checking for vulnerabilities. Every day, at least two engineers were on duty at any given minute,” Zhang Shaoyong explained.
Born to Fight, Breaking Foreign Monopolies
Not just during the epidemic—GBASE has been fighting since its founding 16 years ago.
Databases are not a new industry. Over its more than 40-year evolution, it has gone through multiple eras, but Chinese companies were rarely involved in the first three decades, let alone building a world-class database. For a long time, Chinese companies and technology were constrained by others in critical areas such as chips, databases, and operating systems. Traditional commercial databases represented by Oracle have always dominated the market. Almost every large enterprise had to purchase database software from abroad, with high prices, steep learning curves, and difficult maintenance—but they had no other choice. As one of the most complex and critical foundational software systems in modern IT, the database is an extremely crucial link. Sectors like banking, telecommunications, and transportation urgently needed professional domestic databases to replace foreign products and ensure the security of vast amounts of data. GBASE seized this opportunity.
“In 2011, Ye Yousheng, then Chief Engineer of the Technology Department at Agricultural Bank of China, came to us with two project options: a small audit system and a large data warehouse. We were bold and chose the big data warehouse project,” said Yuan Mengjia. Agricultural Bank of China had previously used the international database giant Sybase IQ, mainly for statistical reporting. As data volumes grew and more systems were connected, Sybase IQ’s performance limitations made it difficult to complete data statistical analysis within the specified time window and prevented further integration of additional business system data. It simply could not meet the bank’s internal data analysis needs or the regulatory data requirements. “We were like a newborn calf unafraid of the tiger. Before this, we had only done projects worth less than a million. Once we started, we realized how complex banking systems are—tables with trillions of rows were commonplace, and the relationships between tables were intricate. Each bank has over a hundred information systems with different data standards and software formats. At the peak, we faced over 200 issues a day. This lasted for more than a year to two years. Now, it has been five years without any problems, and 78 of ABC’s systems run on our business platform.” The Agricultural Bank of China’s big data platform project was subsequently awarded the First Prize for Scientific and Technological Development by the People’s Bank of China.
▲ R&D team
Hard work pays off. Through the team’s efforts, GBASE has now launched multiple world-class domestic database products, filling a gap in China and breaking the monopoly of foreign vendors in technology and market. Last year, they also began the domestic replacement of a large insurance company’s data warehouse. Zhang Shaoyong said: “Previously, this company used an all-in-one foreign database combining hardware and software. The more applications built on it, the stronger the user dependency and the more difficult the replacement, akin to a heart transplant or replanting a tree by the roots. Foreign products often use this approach to lock in users and charge high maintenance fees. We migrated the overall data warehouse business in phases based on business priority. The first phase of migration is now largely complete. After the successful replacement, the incremental execution time dropped from 18 hours to 7 hours—a 2.5x improvement; monthly report execution speed improved 10 times; and overall performance improved 10 times. While ensuring security, our product outperforms the foreign all-in-one database in functionality, performance, price, and stability.”
Even a Domestic Database Leader Faced Loan Difficulties During the Epidemic
During the epidemic, to defend its position, GBASE spared no expense. Originally planned projects were put on hold, and some orders were even cancelled. Yuan Mengjia said: “Projects couldn’t be launched, so we had only expenses and no revenue, straining our cash flow. We thought about loans, but as an IT company with light assets and no collateral, borrowing was very difficult. During the epidemic, leaders from the Binhai New Area and the High-tech Zone visited frequently. They not only helped us resume work in an orderly manner but also learned about our loan difficulties during their surveys. Through the Tianjin SME Credit Guarantee Center (‘Zhongxiaodan’), they provided us with a guarantee and connected us with Binhai Rural Commercial Bank for a loan, solving our urgent need.”
The “Zhongxiaodan” Yuan Mengjia referred to is the Tianjin SME Credit Guarantee Center. It is a public institution jointly funded by the Tianjin Municipal Bureau of Industry and Information Technology and the Tianjin Municipal Finance Bureau, established by the Tianjin municipal government to address the financing difficulties of SMEs and support their healthy development. By the end of 2019, the Center had provided guarantees to over 2,700 enterprises with more than 10,000 financing guarantee transactions, totaling nearly 15 billion yuan. Among these, over 70% of the guaranteed enterprises were credit-guaranteed loans without physical collateral, just like GBASE.
“For a tech company in the foundational software industry like ours, R&D investment often amounts to over a billion yuan. Relying solely on the enterprise itself to raise that money brings enormous pressure. Many private companies have had to leave the field regretfully because their funds dried up,” said Zhao Wei, Senior Vice President of GBASE. “And once we achieve technological breakthroughs and industrialize our products, foreign giants will start a price war and encircle us, so our cash flow is constantly tight. Financial support is especially crucial. In fact, the municipal government has always supported us—not just during the epidemic. In 2018, the company’s founder and largest shareholder suddenly passed away, and operations were once on the brink of collapse. At that critical juncture, the Tianjin Municipal Tax Service and the Binhai High-tech Zone Tax Service formed an expert team to provide us with ‘made-to-order’ targeted services, helping us sort out key policy points and resolve tax-related difficulties.”
Wang Zhuxia, CFO of GBASE, did the math: in 2018, R&D investment reached nearly 100 million yuan, a year-on-year increase of 53.85%. The super-deduction ratio for R&D expenses rose from 50% to 75%, resulting in a super-deduction amount of 49.455 million yuan. In addition, the company also enjoys the VAT “immediate collection and refund” policy for software products, receiving tens of millions of yuan in VAT refunds in 2019 alone. This year, as part of the deepening VAT reform, the VAT rate was reduced from 16% to 13%, saving the company several million yuan in taxes. “Tianjin’s implementation of the national tax and fee reduction policies is really timely,” remarked Zhao Wei.
Facing Death to Survive, Betting Everything
In the GBASE conference room hangs a calligraphy scroll with the words: “Survive by facing death; go all out with everything you have.” Chairman Ding Mingfeng said with deep feeling: “Since its establishment, after many fierce battles with foreign database giants, we have achieved today’s results, with products competitive in all aspects—performance, price, and others—against foreign counterparts. After more than 30 years of development, China’s ICT industry has entered a stage of deep advancement and requires more core independent and controllable capabilities. The transformations brought by 5G, AI, and cloud computing hide future opportunities. GBASE has reached an inflection point, and we must bet everything.”
▲ Survive by facing death; go all out with everything you have
Massive data is the means of production of the future, containing unlimited value; cooperation leads to win-win outcomes. Ding Mingfeng said that GBASE has formulated a development strategy centered on partners to build a domestic foundational software ecosystem. “We are fully committed to working with partners to build a healthy and stable domestic ecosystem, because databases and big data platforms serve as the foundation for this round of data-driven economic growth. We must embrace the entire industry with a more open mindset and fully cooperate with upstream and downstream players. Only by achieving the overall development and growth of the domestic ecosystem can China’s high-tech products step out of the niche market left by international giants and truly stand on the world stage. In an era of increasingly refined specialization, focus requires knowing what to do and what not to do. On the road to independent and controllable foundational software, we need more pioneers like GBASE to question and explore.”