缩容操作异常处理

发表于2026-04-28 18:05:0230次浏览3个评论

项目背景介绍：已有一套6台服务器的集群（3gc+6gn，集群版本v9.5.2.46_patch1），服务器资源相对老旧，需要更换成新的6台服务器，采取的方案：先扩容，表重分布到新distribution，缩容旧节点，替换ip；因replace.py脚本只能一个一个节点来替换，操作繁琐，故采取此方案来替换。

下面就只讲遇到的问题，以及解决办法：

问题出现：

变更操作，扩容，rebalance重分布六千多张表、删除旧distribution一切正常，直到将数据节点移除集群时，执行完gcadmin rmnodes gcRm.xml后就出现问题了；

[gbase@p11 gcinstall]$ cat gcRm.xml
<?xml version="1.0" encoding="utf-8"?>
<servers>
<rack>
<node ip="192.168.1.241"/>
<node ip="192.168.1.242"/>
<node ip="192.168.1.243"/>
<node ip="192.168.1.244"/>
<node ip="192.168.1.245"/>
<node ip="192.168.1.246"/>
</rack>
</servers>

[gbase@p11 gcinstall]$ gcadmin

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | OPEN | OPEN | 0 |
------------------------------------------------------------------
=========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
=========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
| node1 | 192.168.1.241 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node2 | 192.168.1.242 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node3 | 192.168.1.243 | | OPEN | OPEN | 0 |
--------------------------------------------------------------------------------------------------------
| node4 | 192.168.1.244 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node5 | 192.168.1.245 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node6 | 192.168.1.246 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------

[gbase@p11 gcinstall]$ gcadmin rmnodes gcRm.xml
gcadmin remove nodes ...

gcadmin rmnodes from cluster success

[gbase@p11 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | OPEN | OPEN | 0 |
------------------------------------------------------------------
===========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
===========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
-----------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------

就是执行命令后，gcadmin查看集群状态时，多了一些offline的记录，且NodeName和 IpAddress等信息栏都是空的，同时也验证了数据库的读写功能，发现是不行的，报错内容：:Get vcid failed.

日志报错内容：

2026-xx-xx xx:xx:xx.885 [METADATA][ERROR][S:12][Q:4]<GetVcIdByDataNodeId|6711>:get vcid by nodeId fail, nodeId = 40 :GC_AIS_ERR_NODEID_NOT_IN_VC
2026-xx-xx xx:xx:xx.885 [DDL][WARN ][S:12][Q:4]<AssignWorkers|4892>:vcid is empty by node(40).
2026-xx-xx xx:xx:xx.885 [DDL][ERROR][S:12][Q:4]<Execute|7875>:Get vcid failed.

目前已无法继续往下走了，担心往下走卸载操作会对集群数据有影响，所以此时就需要先恢复集群，将集群恢复成原来状态：

恢复问题：

当前是做了rmnodes操作才出现这种现象，就先加回去，使用addnodes将节点加回集群，

[gbase@p11 gcinstall]$ gcadmin addnodes gcRm.xml
gcadmin add nodes ...

gcadmin addnodes from cluster success

此时查看集群状态是这样的：

[gbase@p11 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | OPEN | OPEN | 0 |
------------------------------------------------------------------
===========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
===========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
-----------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| \; | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| | | | OFFLINE | | |
-----------------------------------------------------------------------------------------------------------
| node13 | 192.168.1.241 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node14 | 192.168.1.242 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node15 | 192.168.1.243 | | OPEN | OPEN | 0 |
--------------------------------------------------------------------------------------------------------
| node16 | 192.168.1.244 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node17 | 192.168.1.245 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node18 | 192.168.1.246 | | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------

此时数据库状态是可以正常读写了的，算是一个好消息，但是gcadmin看到中间还是有一些不正常的内容，为消除中间不正常的内容，就手动清除

清除步骤：

停止集群所有服务（gcluster_services all stop）

备份整个gcware（cp /opt/gcware /opt/gcware_normal ）

单独开启gcware（gclustes_services gcware start）

刷gcware快照（python -c ‘import gcware;gcware.flush_statemachine();’）

确保所有gcware节点的快照是一直的（ls /opt/gcware/data/gcware*;确认所有文件的后缀数字一致即可）

关闭gcware（gclustes_services gcware stop）

修改快照文件（cd /opt/gcware/data/gcware/SNAPSHOT-*/statemachine/vc00001/;vim DATASERVER;vim DATACLUSTERSTATE;）(快照文件只清除异常的内容，其他的不动，且dataserverlist_entries值要修改正确的数据节点数量)

启动gcware（gclustes_services gcware start）（确认集群信息都正常后，再开启其他服务）

启动其他服务（gclustes_services all start）

上面操作下来后，集群就恢复到删除distribution后的状态了：

[gbase@p11 gcinstall]$ gcadmin

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | OPEN | OPEN | 0 |
------------------------------------------------------------------
=========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
=========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node13 | 192.168.1.241 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node14 | 192.168.1.242 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node15 | 192.168.1.243 | | OPEN | OPEN | 0 |
--------------------------------------------------------------------------------------------------------
| node16 | 192.168.1.244 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node17 | 192.168.1.245 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node18 | 192.168.1.246 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------

卸载操作之前必须要做rmnodes操作，后面解决办法：

因为第一次移除的节点中有复合data节点和纯data节点，所以第二次采取将这两种分开来移除集群

[gbase@lx-bd-gb-p11 gcinstall]$ cat gcRm.xml
<?xml version="1.0" encoding="utf-8"?>
<servers>
<rack>
<node ip="192.168.1.244"/>
<node ip="192.168.1.245"/>
<node ip="192.168.1.246"/>
</rack>
</servers>

[gbase@lx-bd-gb-p11 gcinstall]$ gcadmin rmnodes gcRm.xml
gcadmin remove nodes …
gcadmin rmnodes from cluster success

[gbase@lx-bd-gb-p11 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
===========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
===========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
-----------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node13 | 192.168.1.241 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node14 | 192.168.1.242 | | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node15 | 192.168.1.243 | | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------

状态正常，继续移除复合节点：

[gbase@lx-bd-gb-p11 gcinstall]$ cat gcRm.xml
<?xml version="1.0" encoding="utf-8"?>
<servers>
<rack>
<node ip="192.168.1.241"/>
<node ip="192.168.1.242"/>
<node ip="192.168.1.243"/>
</rack>
</servers>

[gbase@lx-bd-gb-p11 gcinstall]$ gcadmin rmnodes gcRm.xml
gcadmin remove nodes …
gcadmin rmnodes from cluster success

[gbase@lx-bd-gb-p11 gcinstall]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL

==================================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
==================================================================
| NodeName | IpAddress | gcware | gcluster | DataState |
------------------------------------------------------------------
| coordinator1 | 192.168.1.241 | OPEN | OPEN | 0 |
------------------------------------------------------------------
| coordinator2 | 192.168.1.242 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator3 | 192.168.1.243 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator4 | 192.168.1.251 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator5 | 192.168.1.252 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
| coordinator6 | 192.168.1.253 | CLOSE | OPEN | 0 |
------------------------------------------------------------------
===========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
===========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
-----------------------------------------------------------------------------------------------------------
| node7 | 192.168.1.251 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node8 | 192.168.1.252 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node9 | 192.168.1.253 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node10 | 192.168.1.254 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node11 | 192.168.1.255 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
| node12 | 192.168.1.256 | 3 | OPEN | OPEN | 0 |
-----------------------------------------------------------------------------------------------------------
状态也正常，移除集群操作完成。问题解决。

后面卸载操作和替换ip操作一切都正常，就不展示了。

缩容操作异常处理

评论

热门帖子