Metro CLC System in Crisis: Database CPU Spikes to 100% — GBASE Emergency Rescue Record

Database CPU at 100%???

Staring at the abnormally skyrocketing CPU utilization, Xiao Ming, an O&M engineer for a top-5 metro customer’s core business system CLC (Centralized Line Control System), let out a cry of alarm.

The transaction settlement system page went blank and sluggish, generating numerous timeout errors. A quick check of the monitoring dashboard revealed that the database CPU utilization had hit 100%! Xiao Ming was horrified. This project had been running for over two years since its 2022 launch, connecting nearly 400 stations—with close to 100 existing line stations still waiting to be integrated, plus another 100+ planned for the future. How could it already be at 100%?

Xiao Ming immediately called GBase’s expert support team. Upon receiving the request, the team deployed expert Xiao An to the site to coordinate with Xiao Ming on the spot.

Database CPU Soars to 100%, GBASE Experts to the Rescue

Xiao An quickly gathered the facts: the production server had the database bound to 64 CPU cores out of a total 128 cores, with the GEM monitoring agent running on the same server. Total allocated tablespace was 32 TB, but only about 21 TB was in use. Total memory was 754 GB, with utilization around 16%. Storage consisted of hyper-converged disks with SSD caching—when full, some I/O degradation could occur, but still within acceptable limits.

Clearly, the CPU was the culprit! Xiao An immediately began investigating:

With 64 cores assigned to the database, normal peak should be around 50%. The jump to 90% was likely due to CPU spikes caused by the GEM agent. So, Xiao An stopped the GEM agent and observed. The spikes disappeared, and CPU usage smoothed out—proving the spikes had not affected database performance.

Next, Xiao An suspected that SQL execution was driving CPU usage up and started analyzing the running queries. He ran the command “onstat -g ses 0” every 5 minutes on the primary node, saving the output. The logs revealed a large number of repeated SQL statements.

Xiao An guessed that the highlighted part was not using an index, or the index was very inefficient, so he examined the execution plan.

It turned out the SQL was using an index, but the index efficiency was extremely low—no wonder it was so slow. Xiao An immediately
added a composite index, placing the low-cardinality columns first.

The result was immediate: after the change, index efficiency improved by 1,000 times.

After a few days of monitoring, CPU utilization dropped from 50% to 20%.

Xiao Ming gave Xiao An a thumbs-up, but then asked: Could other SQL statements have the same problem? Did this mean CPU utilization could drop even further? He sought more help from Xiao An.

Having already identified the root cause in the first round, Xiao An performed a second round of optimization. Using the same approach, he analyzed all high-CPU SQL statements and tuned those without indexes or with inefficient ones. After the SQL optimization, CPU utilization fell from 20% to single digits.

The database performance issue was completely resolved.

In subsequent operations, they also found that a certain business report system that previously took half an hour on Oracle returned results in just a few minutes after switching to GBase 8s. The customer was delighted and publicly commended Xiao An.

Troubleshooting Tips

Ø Database design is a discipline. Sometimes the database isn’t “not good enough”—it’s a planning issue. Only by properly designing databases, tables, and indexes based on business requirements and database features can you unlock the database’s true potential.

Ø From a business access perspective, with the current 397 connected stations and 128-core CPUs (using 80% as the bottleneck line), the database would need to handle 3,176 stations (397 × 8) to hit a performance bottleneck. With 64-core CPUs, the future planned full-line connections total fewer than 800 stations—keeping 64-core CPU usage under 40%. With full optimization, CPU usage is estimated to be around 20% or even lower, though this requires joint effort from the application side.

Ø This case demonstrates that the GBase 8s V8.8 SSC shared-storage cluster can handle massive workloads with lower cluster and maintenance costs. Therefore, when selecting a database, avoid stereotypes. The best choice is the database product and cluster technology that truly meets your business needs.