GBase8a手工恢复表分片数据方案及步骤
问题背景：
用户在对表tdb.t1表进行DQL操作时，报“表不存在，分片表不存在，分片表文件损坏”等错误；进行DML操作时报“分片表不存在，分片表文件损坏，主备分片元数据不一致，主备分片数据不一致”等错误。
集群环境：
[gbase@GBASEMA01 ~]$ gcadmin
CLUSTER STATE: ACTIVE
VIRTUAL CLUSTER MODE: NORMAL
======================================
| GBASE GCWARE CLUSTER INFORMATION |
======================================
| NodeName | IpAddress | gcware |
--------------------------------------
| gcware1 | 192.168.195.10 | OPEN |
--------------------------------------
========================================================
| GBASE COORDINATOR CLUSTER INFORMATION |
========================================================
| NodeName | IpAddress | gcluster | DataState |
--------------------------------------------------------
| coordinator1 | 192.168.195.10 | OPEN | 0 |
--------------------------------------------------------
=========================================================================================================
| GBASE DATA CLUSTER INFORMATION |
=========================================================================================================
| NodeName | IpAddress | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
| node1 | 192.168.195.103 | 1 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node2 | 192.168.195.101 | 1 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
| node3 | 192.168.195.105 | 1 | OPEN | OPEN | 0 |
---------------------------------------------------------------------------------------------------------
[gbase@GBASEMA01 ~]$ gcadmin showdistribution
Distribution ID: 1 | State: new | Total segment num: 3
Primary Segment Node IP Segment ID Duplicate Segment node IP
========================================================================================================================
| 192.168.195.103 | 1 | 192.168.195.101 |
------------------------------------------------------------------------------------------------------------------------
| 192.168.195.101 | 2 | 192.168.195.105 |
------------------------------------------------------------------------------------------------------------------------
| 192.168.195.105 | 3 | 192.168.195.103 |
========================================================================================================================
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "show nodes"
+------------+-----------------+-------+--------------+----------------+--------+-----------+
| Id | ip | name | primary part | duplicate part | status | datastate |
+------------+-----------------+-------+--------------+----------------+--------+-----------+
| 1740875968 | 192.168.195.103 | node1 | n1 | n3 | online | 0 |
| 1707321536 | 192.168.195.101 | node2 | n2 | n1 | online | 0 |
| 1774430400 | 192.168.195.105 | node3 | n3 | n2 | online | 0 |
+------------+-----------------+-------+--------------+----------------+--------+-----------+
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "show vcs"
+---------+--------------+---------+
| id | name | default |
+---------+--------------+---------+
| vc00001 | vcname000001 | Y |
+---------+--------------+---------+
故障节点：
192.168.195.105节点上tdb.t1表故障，192.168.195.105的主分片在192.168.195.101节点n2上，备份分片在192.168.195.103节点n1上
应急处理：
1）故障节点隔离：gcadmin setnodestate 192.168.195.105 failure
2）故障处理后恢复：gcadmin setnodestate 192.168.195.105 normal
问题处理方法：
*********************************************** S1 重建表 *************************************************************
方案一：服务器无问题，且故障表不多时，并且集群层还能查询正确数据，通过“重建表”恢复表数据
参考网址：https://www.gbase8.cn/174
1）复制表结构
create table tdb.t1_1 like tdb.t1;
2）复制表数据
insert into tdb.t1_1 select * from tdb.t1;
3）修改表名
alter table tdb.t1 rename tdb.t1_0;
alter table tdb.t1_1 rename tdb.t1;
4）根据情况考虑删除（存储空间足够可不删）
drop table tdb.t1_0;
************************************************** S2 工具同步 **********************************************************
方案二：服务器无问题，且故障表不多时，通过工具同步正常节点的表分片数据，手工恢复表分片数据。
参考网址：https://www.gbase8.cn/176
1）检查故障节点分片（Coor）
#105节点上分片t1_n2文件为0，但101节点上分片t1_n2文件大小不为0，说明105节点上该分片数据异常
[gbase@GBASEMA01 ~]$ cexec data: 'du -sh /opt/192.*/gnode/userdata/gbase/tdb/sys_tablespace/t1_n*'
************************* data *************************
--------- 192.168.195.101---------
16K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
28K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
--------- 192.168.195.103---------
16K /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
0 /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
--------- 192.168.195.105---------
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
2）同步分片数据(Gnode)：
# 如果在105节点上没有t1_n2分片目录，需手工创建mkdir t1_n2，否则工具不能正常同步
[gbase@GBASEDATA01 ~]$ source /opt/192.168.195.105/gbase_profile
[gbase@GBASEDATA01 ~]$ /opt/192.168.195.105/gnode/server/bin/gc_sync_client 192.168.195.101 vcname000001 tdb t1_n2 192.168.195.10 5258
参数：正确分片的节点IP VC名 库名 表分片名 管理节点IP 管理节点端口
3）刷新表分片(Gnode)：
[gbase@GBASEDATA01 ~]$ gncli -ugbase -pX@gbase2024 -e "refresh table tdb.t1_n2"
4）核对主备分片(Coor)：
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.101 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.105 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ cexec data: 'du -sh /opt/192.*/gnode/userdata/gbase/tdb/sys_tablespace/t1_n*'
************************* data *************************
--------- 192.168.195.101---------
16K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
28K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
--------- 192.168.195.103---------
16K /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
0 /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
--------- 192.168.195.105---------
12K /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
************************************************** S3 自动恢复 **********************************************************
方案三：服务器无问题，且故障表不多时，通过设置不一致标志，由GBas的同步机制自动恢复表分片数据。
参考网址：https://www.gbase8.cn/5799
1）检查故障节点分片（Coor）
#105节点上分片t1_n2文件为0，但101节点上分片t1_n2文件大小不为0，说明105节点上该分片数据异常
[gbase@GBASEMA01 ~]$ cexec data: 'du -sh /opt/192.*/gnode/userdata/gbase/tdb/sys_tablespace/t1_n*'
************************* data *************************
--------- 192.168.195.101---------
16K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
36K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
--------- 192.168.195.103---------
16K /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
0 /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
--------- 192.168.195.105---------
36K /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
2）同步分片数据(Coor)：
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "set self gcluster_node_status_list='vc00001.tdb.t1:n2:1774430400:0:1'"
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "set self gcluster_node_status_list='vc00001.tdb.t1:n3:1774430400:0:1'"
参数：vcid.库名.表名:分片号:主机nodeid:状态:SCN号，如果是复制表分片号为空，例如：vc00001.tdb.t1::1774430400:0:1
3）核对主备分片(Coor)：
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.101 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.105 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ cexec data: 'du -sh /opt/192.*/gnode/userdata/gbase/tdb/sys_tablespace/t1_n*'
************************* data *************************
--------- 192.168.195.101---------
16K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
36K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
--------- 192.168.195.103---------
16K /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
0 /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
--------- 192.168.195.105---------
36K /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
************************************************** S4 手工复制 **********************************************************
方案四：服务器无问题，且故障表不多时，故障表处理期间无DML DDL操作，通过手工复制正常节点的表分片数据，手工恢复表分片数据。
参考网址：https://www.gbase8.cn/7619
1）对故障表进行锁写表处理(Coor)
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "lock table tdb.t1 write; select sleep(100000)" &
[1] 78149
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "show processlist"
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 11738 | Waiting for next activation | NULL |
| 1469 | gbase | localhost | NULL | Sleep | 4058 | | NULL |
| 2005 | gbase | localhost | NULL | Query | 19 | checking permissions | select sleep(100000) |
| 2009 | gbase | localhost | NULL | Query | 0 | NULL | show processlist |
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
[gbase@GBASEMA01 ~]$ gcadmin showlock
+======================================================================================+
| GCLUSTER LOCK |
+======================================================================================+
+----------------------------+--------------+---------------+--------------+------+----+
| Lock name | owner | content | create time |locked|type|
+----------------------------+--------------+---------------+--------------+------+----+
| gc-event-lock |192.168.195.10| global master |20241029063626| TRUE | E |
+----------------------------+--------------+---------------+--------------+------+----+
| vc00001.hashmap_lock |192.168.195.10|2005(LWP:78150)|20241029095146| TRUE | S |
+----------------------------+--------------+---------------+--------------+------+----+
| vc00001.tdb.db_lock |192.168.195.10|2005(LWP:78150)|20241029095146| TRUE | S |
+----------------------------+--------------+---------------+--------------+------+----+
|vc00001.tdb.t1.execute_lock |192.168.195.10|2005(LWP:78150)|20241029095146| TRUE | E |
+----------------------------+--------------+---------------+--------------+------+----+
| vc00001.tdb.t1.meta_lock |192.168.195.10|2005(LWP:78150)|20241029095146| TRUE | S |
+----------------------------+--------------+---------------+--------------+------+----+
|vc00001.tdb.table_space_lock|192.168.195.10|2005(LWP:78150)|20241029095146| TRUE | S |
+----------------------------+--------------+---------------+--------------+------+----+
2）备份故障表及数据(Coor)
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "create table tdb.t1_1 like tdb.t1"
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "insert into tdb.t1_1 select * from tdb.t1"
3）手工清理故障表EVENT(Coor)
[gbase@GBASEMA01 ~]$ gcadmin showddlevent |grep "tdb.t1" |wc -l; gcadmin showdmlevent| grep "tdb.t1"|wc -l; gcadmin showdmlstorageevent |grep "tdb.t1" |wc -l
[gbase@GBASEMA01 ~]$ python
>>> import gcware
>>> gcware.clearddlfevent('vcname000001','^tdb.t1$')
>>> gcware.cleardmlfevent('vcname000001','^tdb.t1$')
>>> gcware.cleardmlstoragefevent('vcname000001','^tdb.t1$')
>>> exit()
[gbase@GBASEMA01 ~]$ gcadmin showddlevent |grep "tdb.t1" |wc -l; gcadmin showdmlevent| grep "tdb.t1"|wc -l; gcadmin showdmlstorageevent |grep "tdb.t1" |wc -l
4）移除故障表文件(Coor)
[gbase@GBASEMA01 ~]$ ssh -n 192.168.195.105 "mkdir -p /opt/gbase/20241029/metadata; mkdir -p /opt/gbase/20241029/sys_tablespace"
[gbase@GBASEMA01 ~]$ ssh -n 192.168.195.105 "mv /opt/192.168.195.105/gnode/userdata/gbase/tdb/metadata/t1_n2.* /opt/gbase/20241029/metadata"
[gbase@GBASEMA01 ~]$ ssh -n 192.168.195.105 "mv /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2 /opt/gbase/20241029/sys_tablespace"
5）从正常节点复制文件(GNode)
[gbase@GBASEDATA01 ~]$ scp -r 192.168.195.101:/opt/192.168.195.101/gnode/userdata/gbase/tdb/metadata/t1_n2.* /opt/192.168.195.105/gnode/userdata/gbase/tdb/metadata/
[gbase@GBASEDATA01 ~]$ scp -r 192.168.195.101:/opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/
6）刷新表分片(GNode)：
[gbase@GBASEDATA01 ~]$ source /opt/192.168.195.105/gbase_profile
[gbase@GBASEDATA01 ~]$ gncli -ugbase -pX@gbase2024 -e "refresh table tdb.t1_n2"
7）核对主备分片(Coor)：
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.101 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -h192.168.195.105 -P5050 -Ns -e "select count(*) from tdb.t1_n2"
22
[gbase@GBASEMA01 ~]$ cexec data: 'du -sh /opt/192.*/gnode/userdata/gbase/tdb/sys_tablespace/t1_n*'
************************* data *************************
--------- 192.168.195.101---------
16K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
36K /opt/192.168.195.101/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
--------- 192.168.195.103---------
16K /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n1
0 /opt/192.168.195.103/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
--------- 192.168.195.105---------
36K /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n2
0 /opt/192.168.195.105/gnode/userdata/gbase/tdb/sys_tablespace/t1_n3
8）关闭锁写表进程(Coor)
[gbase@GBASEMA01 ~]$ ps -ef|grep gccli
[gbase@GBASEMA01 ~]$ kill -9 78149
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "show processlist"
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
| Id | User | Host | db | Command | Time | State | Info |
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
| 1 | event_scheduler | localhost | NULL | Daemon | 11738 | Waiting for next activation | NULL |
| 1469 | gbase | localhost | NULL | Sleep | 4058 | | NULL |
| 2005 | gbase | localhost | NULL | Query | 19 | checking permissions | select sleep(100000) |
| 2009 | gbase | localhost | NULL | Query | 0 | NULL | show processlist |
+------+-----------------+-----------+------+---------+-------+-----------------------------+----------------------+
[gbase@GBASEMA01 ~]$ gccli -ugbase -pX@gbase2024 -e "kill 2005"
[gbase@GBASEMA01 ~]$ gcadmin showlock
+=====================================================================+
| GCLUSTER LOCK |
+=====================================================================+
+-------------+--------------+-------------+--------------+------+----+
| Lock name | owner | content | create time |locked|type|
+-------------+--------------+-------------+--------------+------+----+
|gc-event-lock|192.168.195.10|global master|20241029063626| TRUE | E |
+-------------+--------------+-------------+--------------+------+----+
************************************************** S5 节点替换 **********************************************************
方案五：服务器文件系统故障，或故障表很多，修复耗时太长，建议设置为unavaliable, 做节点替换
评论
热门帖子
- 12025-12-01浏览数：182759
- 22023-05-09浏览数：25052
- 42023-09-25浏览数：18521
- 52020-05-11浏览数：17526