GBase 8a数据库集群，数据不一致dmlevent故障模拟和恢复过程分析

老紫竹 老紫竹 博客等级 墨衍会员 · AI 创作全网分发 墨衍智能分发 本文由作者通过墨衍一键同步至各平台 1分发平台 1.8k累计阅读 1.8k平均单平台 已同步平台 CSDN 微信公众号 微博 知乎 掘金 百家号 博客园 抖音 小红书 你的文章也可以这样分发 墨衍会员 ¥399起/年，写一次发全网 我也要 码龄19年
关注
原创 最新推荐文章于 2023-10-31 17:05:00 发布 1.8k 阅读 AI 写同款· GEO 优化 ›
GBase 8a 扩容操作意外的处理方案 GBase 8a 数据库扩容时，可能发生各种意外情况，本文针对扩容每个操作步骤进行分析，考虑发生的各种意外，以及人工处理方法。 阅读详情

GBase 8a数据库集群，通过副本来保证数据高可用，当某些服务或节点故障时，就会产生不一致，比如dmlevent。本文在测试环境模拟故障，并分析其恢复过程。

原文 http://www.gbase8.cn/855

环境

3节点集群，关闭1个节点的数据库服务。V95版本。

[gbase@localhost gcluster]$ gcadmin
CLUSTER STATE:         ACTIVE
VIRTUAL CLUSTER MODE:  NORMAL

=============================================================
|           GBASE COORDINATOR CLUSTER INFORMATION           |
=============================================================
|   NodeName   | IpAddress  | gcware | gcluster | DataState |
-------------------------------------------------------------
| coordinator1 | 10.0.2.102 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator2 | 10.0.2.202 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator3 | 10.0.2.203 | CLOSE  |  CLOSE   |     0     |
-------------------------------------------------------------
=========================================================================================================
|                                    GBASE DATA CLUSTER INFORMATION                                     |
=========================================================================================================
| NodeName |                IpAddress                 | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
|  node1   |                10.0.2.102                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node2   |                10.0.2.202                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node3   |                10.0.2.203                |       3        | CLOSE |   CLOSE    |     0     |
---------------------------------------------------------------------------------------------------------

模拟故障，insert一些数据

[gbase@localhost gcluster]$ gccli testdb

GBase client 9.5.2.17.115980. Copyright (c) 2004-2020, GBase.  All Rights Reserved.

gbase> create table t12(id int) distributed by('id');
Query OK, 0 rows affected (Elapsed: 00:00:01.02)

gbase> insert into t12 values(1);
Query OK, 1 row affected (Elapsed: 00:00:00.12)

gbase> insert into t12 values(2);
Query OK, 1 row affected (Elapsed: 00:00:00.19)

gbase> insert into t12 values(3);
Query OK, 1 row affected (Elapsed: 00:00:00.33)

gbase> ^CAborted
[gbase@localhost gcluster]$ gcadmin showdmlevent
Vc event count:0
[gbase@localhost gcluster]$ gccli testdb

GBase client 9.5.2.17.115980. Copyright (c) 2004-2020, GBase.  All Rights Reserved.

gbase> insert into t12 values(4);
Query OK, 1 row affected (Elapsed: 00:00:00.61)

gbase> insert into t12 values(5);
Query OK, 1 row affected (Elapsed: 00:00:00.48)

gbase> insert into t12 values(6);
Query OK, 1 row affected (Elapsed: 00:00:00.15)

gbase> show create table t12;\
+-------+-----------------------------------------------------------------------------------------------------------------------------------------+
| Table | Create Table                                                                                                                            |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------+
| t12   | CREATE TABLE "t12" (
  "id" int(11) DEFAULT NULL
) ENGINE=EXPRESS DISTRIBUTED BY('id') DEFAULT CHARSET=utf8 TABLESPACE='sys_tablespace' |
+-------+-----------------------------------------------------------------------------------------------------------------------------------------+
1 row in set (Elapsed: 00:00:00.00)

gbase> ^CAborted
[gbase@localhost gcluster]$ gcadmin
CLUSTER STATE:         ACTIVE
VIRTUAL CLUSTER MODE:  NORMAL

=============================================================
|           GBASE COORDINATOR CLUSTER INFORMATION           |
=============================================================
|   NodeName   | IpAddress  | gcware | gcluster | DataState |
-------------------------------------------------------------
| coordinator1 | 10.0.2.102 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator2 | 10.0.2.202 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator3 | 10.0.2.203 | CLOSE  |  CLOSE   |     0     |
-------------------------------------------------------------
=========================================================================================================
|                                    GBASE DATA CLUSTER INFORMATION                                     |
=========================================================================================================
| NodeName |                IpAddress                 | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
|  node1   |                10.0.2.102                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node2   |                10.0.2.202                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node3   |                10.0.2.203                |       3        | CLOSE |   CLOSE    |     1     |
---------------------------------------------------------------------------------------------------------

[gbase@localhost gcluster]$ gcadmin showdmlevent
Vc event count:1
Event ID:    5
ObjectName: testdb.t12

Fail Data Copy:
------------------------------------------------------
SegName: n2     SCN: 9222       NodeIP: 10.0.2.203      FAILURE
SegName: n3     SCN: 9224       NodeIP: 10.0.2.203      FAILURE


[gbase@localhost gcluster]$

如上可以看到，只有数据影响到了故障节点，才会别设置故障标记。

查看当前管理节点的express.log

关注设置event的部分，如下event id = 5，和event的信息一致。

2020-08-15 19:29:15.022 [DEF  ] [S:49][Q:60][insert] Set Node(10.0.2.203)testdb.t12[n2][scn:9222] data state
2020-08-15 19:29:15.076 [DEF  ] [S:49][Q:60]Successfully set data(testdb.t12:normal) LOCKED, event id is 5

2020-08-15 19:29:18.371 [DEF  ] [S:49][Q:61][insert] Set Node(10.0.2.203)testdb.t12[n2][scn:9223] data state
2020-08-15 19:29:18.399 [DEF  ] [S:49][Q:61]Successfully set data(testdb.t12:normal) LOCKED, event id is 5

2020-08-15 19:29:20.470 [DEF  ] [S:49][Q:62][insert] Set Node(10.0.2.203)testdb.t12[n3][scn:9224] data state
2020-08-15 19:29:20.510 [DEF  ] [S:49][Q:62]Successfully set data(testdb.t12:normal) LOCKED, event id is 5

恢复故障节点

[gbase@localhost gcluster]$ gcadmin
CLUSTER STATE:         ACTIVE
VIRTUAL CLUSTER MODE:  NORMAL

=============================================================
|           GBASE COORDINATOR CLUSTER INFORMATION           |
=============================================================
|   NodeName   | IpAddress  | gcware | gcluster | DataState |
-------------------------------------------------------------
| coordinator1 | 10.0.2.102 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator2 | 10.0.2.202 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
| coordinator3 | 10.0.2.203 |  OPEN  |   OPEN   |     0     |
-------------------------------------------------------------
=========================================================================================================
|                                    GBASE DATA CLUSTER INFORMATION                                     |
=========================================================================================================
| NodeName |                IpAddress                 | DistributionId | gnode | syncserver | DataState |
---------------------------------------------------------------------------------------------------------
|  node1   |                10.0.2.102                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node2   |                10.0.2.202                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------
|  node3   |                10.0.2.203                |       3        | OPEN  |    OPEN    |     0     |
---------------------------------------------------------------------------------------------------------

查看gc_recover.log，看看恢复调度信息

2020-08-15 19:35:13.750 [INFO ] <session:7>: Start dml recover .,tid 357, eventid 5
2020-08-15 19:35:13.763 [INFO ] <session:7>: source node is 0xca02000a, table t12, suffix n2
2020-08-15 19:35:13.849 [INFO ] <session:7>: DealDMLRecoverLock vc00001.testdb.t12 success
2020-08-15 19:35:13.858 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: recoverinfo gn dml: eventid:5 DB(testdb), TABLE(t12), SLICE(2), TID(357), src:3389128714(10.0.2.202) dst:3405905930(10.0.2.203)
2020-08-15 19:35:13.924 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: PreWriteDMLSBeforeSync eventid 6
2020-08-15 19:35:13.924 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5258, do "select table_id from information_schema.tables where table_schema='testdb' and table_name='t12' and TABLE_VC='vcname000001'"
2020-08-15 19:35:13.926 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:357
2020-08-15 19:35:13.926 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n2'"
2020-08-15 19:35:13.928 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9220
2020-08-15 19:35:13.928 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "FLUSH ROLLBACK "testdb"."t12_n2""
2020-08-15 19:35:13.929 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "flush transaction_log"
2020-08-15 19:35:13.929 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.202,port 5050, do "flush transaction_log"
2020-08-15 19:35:13.930 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "start 'gc_sync_client' '10.0.2.202 vcname000001 testdb t12_n2 10.0.2.102 5258 2 0'"
2020-08-15 19:35:14.206 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: ExecSyncQuery, sync table returned with (0)
2020-08-15 19:35:14.206 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: check drop sql thread quit.
2020-08-15 19:35:14.206 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.207 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.202,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.207 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CALL_SYNC, Sync client executed successfully
2020-08-15 19:35:14.207 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "refresh table "testdb"."t12_n2""
2020-08-15 19:35:14.208 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: Sync client returned errcode:[0],errmsg:[success]
2020-08-15 19:35:14.208 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CALL_SYNC, Sync end, return 0
2020-08-15 19:35:14.212 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,Checking whether SYNC is succeeded{DB: 'testdb', Table: 't12', slice: 'n2'}
2020-08-15 19:35:14.212 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n2'"
2020-08-15 19:35:14.213 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9223
2020-08-15 19:35:14.213 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.202,port 5050, do "refresh table "testdb"."t12_n2""
2020-08-15 19:35:14.213 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.202,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n2'"
2020-08-15 19:35:14.214 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9223
2020-08-15 19:35:14.214 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,        recover GNode: 9223
2020-08-15 19:35:14.214 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,      source GNode[1]: 9223
2020-08-15 19:35:14.214 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: delete dml event, table testdb.t12,event id 5 suffix n2
2020-08-15 19:35:14.265 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: SET_ONLINE, Succeeded in setting local node table 'testdb.t12_n2' online, eventinfo(eventid:5,segId:2)
2020-08-15 19:35:14.265 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: CheckToClearOrKeepPreWriteDMLS delete eventid 6
2020-08-15 19:35:14.279 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: delete dmlstorage event, event id: 6
2020-08-15 19:35:14.312 [INFO ] <session:7>: DealDMLRecoverUnLock vc00001.testdb.t12
2020-08-15 19:35:14.524 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n2,et:DML,eid:5,en:node,ip:10.0.2.203>: GetSourceNodeForDMl for nodeId(3405905930), the node is inValid
2020-08-15 19:35:14.524 [INFO ] <session:7>: source node is 0x6602000a, table t12, suffix n3
2020-08-15 19:35:14.554 [INFO ] <session:7>: DealDMLRecoverLock vc00001.testdb.t12 success
2020-08-15 19:35:14.561 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: recoverinfo gn dml: eventid:5 DB(testdb), TABLE(t12), SLICE(3), TID(357), src:1711407114(10.0.2.102) dst:3405905930(10.0.2.203)
2020-08-15 19:35:14.566 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: PreWriteDMLSBeforeSync eventid 7
2020-08-15 19:35:14.566 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5258, do "select table_id from information_schema.tables where table_schema='testdb' and table_name='t12' and TABLE_VC='vcname000001'"
2020-08-15 19:35:14.567 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:357
2020-08-15 19:35:14.567 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n3'"
2020-08-15 19:35:14.569 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9221
2020-08-15 19:35:14.569 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "FLUSH ROLLBACK "testdb"."t12_n3""
2020-08-15 19:35:14.570 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.570 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.571 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "start 'gc_sync_client' '10.0.2.102 vcname000001 testdb t12_n3 10.0.2.102 5258 2 0'"
2020-08-15 19:35:14.921 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: ExecSyncQuery, sync table returned with (0)
2020-08-15 19:35:14.921 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: check drop sql thread quit.
2020-08-15 19:35:14.922 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.922 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5050, do "flush transaction_log"
2020-08-15 19:35:14.923 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CALL_SYNC, Sync client executed successfully
2020-08-15 19:35:14.923 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "refresh table "testdb"."t12_n3""
2020-08-15 19:35:14.924 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: Sync client returned errcode:[0],errmsg:[success]
2020-08-15 19:35:14.924 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CALL_SYNC, Sync end, return 0
2020-08-15 19:35:14.929 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,Checking whether SYNC is succeeded{DB: 'testdb', Table: 't12', slice: 'n3'}
2020-08-15 19:35:14.929 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.203,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n3'"
2020-08-15 19:35:14.930 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9224
2020-08-15 19:35:14.930 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5050, do "refresh table "testdb"."t12_n3""
2020-08-15 19:35:14.931 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: QueryExecute ip 10.0.2.102,port 5050, do "select scn from information_schema.tables where table_schema='testdb' and table_name='t12_n3'"
2020-08-15 19:35:14.932 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: MatchJustOneResult:9224
2020-08-15 19:35:14.932 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,        recover GNode: 9224
2020-08-15 19:35:14.932 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CHECK_SYNC_RESULT,      source GNode[1]: 9224
2020-08-15 19:35:14.932 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: delete dml event, table testdb.t12,event id 5 suffix n3
2020-08-15 19:35:14.952 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: SET_ONLINE, Succeeded in setting local node table 'testdb.t12_n3' online, eventinfo(eventid:5,segId:3)
2020-08-15 19:35:14.952 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: CheckToClearOrKeepPreWriteDMLS delete eventid 7
2020-08-15 19:35:14.962 [INFO ] <session:7, tid:357, db:testdb,tb:t12,nx:n3,et:DML,eid:5,en:node,ip:10.0.2.203>: delete dmlstorage event, event id: 7
2020-08-15 19:35:14.973 [INFO ] <session:7>: DealDMLRecoverUnLock vc00001.testdb.t12
2020-08-15 19:35:15.175 [INFO ] <session:7>: End dml recover .,tid 357

如上能看到调用了 sync_client程序进行恢复，然后调用了refresh刷新，最后设置节点状态为online。
分别处理了n2和n3分片

查看同步日志

在故障节点gnode的日志目录下，也是n2和n3两个日志，如下只贴一个。

[gbase@localhost gbase]$ cat syncclient_vcname000001_testdb_t12_n2_2020-08-15.log
2020-08-15 19:35:13.494 6377 INFO: read from config file:
2020-08-15 19:35:13.494 6377 INFO: log path is /opt/gbase/gnode/log/gbase
2020-08-15 19:35:13.494 6377 INFO: log level is 3
2020-08-15 19:35:13.494 6377 INFO: sync level is 1
2020-08-15 19:35:13.494 6377 INFO: server parallel is 4
2020-08-15 19:35:13.494 6377 INFO: server port is 5288
2020-08-15 19:35:13.494 6377 INFO: block size is 512
2020-08-15 19:35:13.494 6377 INFO: read from parameter:
2020-08-15 19:35:13.494 6377 INFO: server ip address is 10.0.2.202
2020-08-15 19:35:13.494 6377 INFO: table vc is vcname000001
2020-08-15 19:35:13.494 6377 INFO: sync database name is testdb
2020-08-15 19:35:13.494 6377 INFO: sync table name is t12_n2
2020-08-15 19:35:13.494 6377 INFO: gcluster ip address is 10.0.2.102
2020-08-15 19:35:13.494 6377 INFO: gcluster port is 5258
2020-08-15 19:35:13.494 6377 INFO: check method is 2
2020-08-15 19:35:13.494 6377 INFO: gcluster lock is 0
2020-08-15 19:35:13.494 6377 INFO: gnode lock is 1
2020-08-15 19:35:13.494 6377 INFO: double check is 0
2020-08-15 19:35:13.494 6377 INFO: truncate space is 0
2020-08-15 19:35:13.494 6377 INFO: TaskID is
2020-08-15 19:35:13.495 6377 INFO: Socket.cpp:45 10.0.2.202 is not a valid ipv6 address!
2020-08-15 19:35:13.495 6377 INFO: Start Sync Table vcname000001.testdb.t12_n2 from 10.0.2.202
2020-08-15 19:35:13.495 6377 INFO: =============InitTableInfo=================
2020-08-15 19:35:13.512 6377 CRITICAL: TableInfo.cpp:2526 can't find "default-character-set" key in config file
2020-08-15 19:35:13.512 6377 WARNING: TableInfo.cpp:645 can't get charset from config file /opt/gbase/gnode/config/gbase_8a_gbase.cnf.
2020-08-15 19:35:13.513 6377 INFO: =============Sync Table Info And Table Struct=================
2020-08-15 19:35:13.520 6377 INFO: Connect to gnode server ok

2020-08-15 19:35:13.520 6377 INFO: TableInfo.cpp:2116 GN Lock table testdb.t12_n2
2020-08-15 19:35:13.520 6377 INFO: Lock table testdb.t12_n2 start
2020-08-15 19:35:13.520 6377 INFO: TableInfo.cpp:2182 Lock table testdb.t12_n2 successfully
2020-08-15 19:35:13.520 6377 WARNING: FileOperation.cpp:228 access file /opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.par failure! errno:2 errmsg:No such file or directory
2020-08-15 19:35:13.520 6377 WARNING: TableInfo.cpp:2552 file /opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.par not exist.
2020-08-15 19:35:13.520 6377 WARNING: TableInfo.cpp:1502 No par file /opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.par
2020-08-15 19:35:13.520 6377 INFO: Local Table t12_n2 DataVersion:1

2020-08-15 19:35:13.520 6377 INFO: Local Table t12_n2 State:1

2020-08-15 19:35:13.520 6377 INFO: Local Table t12_n2 Des:0

2020-08-15 19:35:13.520 6377 INFO: Local Table t12_n2 Del:1

2020-08-15 19:35:13.520 6377 INFO: Local Table t12_n2 Ctl:1

2020-08-15 19:35:13.520 6377 INFO: Table t12_n2 max scn is  9220
2020-08-15 19:35:13.520 6377 WARNING: FileOperation.cpp:228 access file /opt/gbase/gnode/userdata/gbase/testdb/tablespace.cnf failure! errno:2 errmsg:No such file or directory
2020-08-15 19:35:13.520 6377 INFO: TableInfo.cpp:2790 tablespace.cnf file not exist. path: /opt/gbase/gnode/userdata/gbase/testdb/tablespace.cnf
2020-08-15 19:35:13.520 6377 INFO: TableInfo.cpp:1154 col 0 m_CurrentReadLoc = 0 m_CurrentReadVersion=0
2020-08-15 19:35:13.521 6377 INFO:
===  table information  ===
CREATE TABLE "t12_n2" (
  "id" int(11) DEFAULT NULL
) ENGINE=EXPRESS DEFAULT CHARSET=utf8 TABLESPACE='sys_tablespace' COLUMN_IDS(0)
===========================
2020-08-15 19:35:13.521 6377 INFO: TableInfo.cpp:2202 GN Unlock table testdb.t12_n2
2020-08-15 19:35:13.522 6377 INFO: TableInfo.cpp:2240 Unlock table testdb.t12_n2 successfully
2020-08-15 19:35:13.526 6377 INFO: Server Table t12_n2 DataVersion:1

2020-08-15 19:35:13.526 6377 INFO: Server Table t12_n2 Info:1

2020-08-15 19:35:13.526 6377 INFO: Server Table t12_n2 Map:0

2020-08-15 19:35:13.526 6377 INFO: Server Table t12_n2 Del:1

2020-08-15 19:35:13.526 6377 INFO: Server Table t12_n2 Ctl:1

2020-08-15 19:35:13.526 6377 INFO: =============Sync Table=================
2020-08-15 19:35:13.537 6377 CRITICAL: TableSyncClient.cpp:1551 normal table, des or frm file not exist.
2020-08-15 19:35:13.537 6377 WARNING: FileOperation.cpp:228 access file /opt/gbase/gnode/userdata/gbase/testdb/tablespace.cnf failure! errno:2 errmsg:No such file or directory
2020-08-15 19:35:13.537 6377 INFO: SyncTable column file NoCols: 1
2020-08-15 19:35:13.537 6377 INFO: Sync Col DC Diff Info 1
2020-08-15 19:35:13.537 6377 INFO: ============Sync column DC Diff start. column:1============
2020-08-15 19:35:13.537 6377 INFO: ============Sync Col Mount Info============
2020-08-15 19:35:13.537 6377 INFO: ============Sync map file content============
2020-08-15 19:35:13.537 6377 INFO: ============Sync Column different DCInfo============
2020-08-15 19:35:13.537 6377 INFO: TableSyncClient.cpp:1859 Diff DC number is 1. first different DC is 0.
2020-08-15 19:35:13.537 6377 INFO: SyncColFile 1
2020-08-15 19:35:13.537 6377 INFO: ============Sync column file start. column:1============
2020-08-15 19:35:13.537 6377 INFO: ============Sync column file end. column::1============
2020-08-15 19:35:13.537 6377 INFO: RecvType
2020-08-15 19:35:13.607 6377 INFO: Calculate Number of RecvDC
2020-08-15 19:35:13.607 6377 INFO: RecvColDCFile
2020-08-15 19:35:13.688 6377 INFO: SyncTableFile
2020-08-15 19:35:13.688 6377 INFO: TableSyncClient.cpp:1563 normal table, par file not exist.
2020-08-15 19:35:13.688 6377 WARNING: FileOperation.cpp:228 access file /opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.GED/table.delete.B failure! errno:2 errmsg:No such file or directory
2020-08-15 19:35:13.688 6377 INFO: TableSyncClient.cpp:1617 partition table, logic table, table.delete file not exist.
2020-08-15 19:35:13.689 6377 INFO: TableSyncClient.cpp:2781 File sync:/opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.GED/table.info.B.
2020-08-15 19:35:13.713 6377 INFO: TableSyncClient.cpp:2781 File sync:/opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.GED/table.map.A.
2020-08-15 19:35:13.730 6377 INFO: TableSyncClient.cpp:2781 File sync:/opt/gbase/gnode/userdata/gbase/testdb/metadata/t12_n2.frm.
2020-08-15 19:35:13.741 6377 INFO: TableSyncClient.cpp:2781 File sync:/opt/gbase/gnode/userdata/gbase/testdb/sys_tablespace/t12_n2/C00000.seg.
2020-08-15 19:35:13.767 6377 INFO: =============Sync Table vcname000001.testdb.t12_n2 Success========
收录于
数据库

当前文章被收录于：

数据库
91 篇文章 4 人学习
订阅专栏 查看详情
本内容遵循 CC 4.0 BY-SA 版权协议
版权声明：本文为博主原创文章，遵循 CC 4.0 BY-SA 版权协议，转载请附上原文出处链接和本声明。
GBase 8a MPP Cluster DDL之建库建表语法 阅读详情

相关推荐

GBase8a MPP Cluster查看集群数据不一致

什么是event？ 如主备节点出现不一致，或者数据读取故障，则设置该表的这个分片状态为1即产生了event。根据不同的错误类型，分为 dmlevent 数据内容不一致 ddlevent 元数据不一致，或者表结构不一致 dmlstorageevent 数据存储异常，一般是磁盘物理损坏或者数据文件checksum错误。 这些event，由系统的gcrecover进行处理，绝大部分event可以由集群内部自动恢复，部分特殊场景下，比如单节点集群、节点磁盘损坏等，可以通过手工命令强行清理。 如何查看even

Mr_dar的博客 1785

GBase 8a Mpp Cluster集群产品特性之节点替换篇

GBase 8a Mpp Cluster集群产品特性之节点替换篇： 当节点发生故障时，在节点替换过程中，集群支持查询、DML及DDL操作，不影响业务运行。 产品特性的重要性：增强运维能力，因为x86机器不可靠，当集群中的节点出现故障而需要进行在线的替换，不能影响业务是非常重要的。 另，还有 GCMonit 组件，用于定期监测 GBase 8a MPP Cluster 服务程序的运行状态, 一旦发现某个服务程序的进程状态发生变化， 就会根据配置文件中的内容来执行相应的服务启停脚本命令，从而保证服务程 序健康运

zhu1981hui的博客 521

GBase 常见网络问题及排查方法

网卡降速 集群部分节点性能差，查看 nmon 等看到外发和收取速度明显低于其它节点。再查看 ethtools 网卡， 看到网速不是正常的千兆或万兆。比如万兆网，显示千兆，甚至百兆。这种情况一般是网线或者网卡不稳定。 解决方案 可以尝试重启网卡或者换其它正常的网口以及维修或更换网卡、检测网线。 SSH 不通 集群部分节点性能差，查看 nmon 等看到外发和收取速度明显低于其它节点。再查看 ethtools 网卡， 看到网速不是正常的千兆或万兆。比如万兆网，显示千兆，甚至百兆。这种情况一般是网线或者网

Mr_dar的博客 1155

GBase8a MPP Cluster查看故障转换状态showfailover

Failover 释义： 当一个操作在发起端gclusterd出现意外时，为了确保影响到的数据的一致性，设置failover, 由其它gclusterd进程负责检查故障操作设计的表的一致性。 如发现不一致，比如在commit阶段，部分分片成功了，那么根据主备比例，采取对不成功的分片设置1进行同步，或者对少量已经commit的分片进行强制回滚。 最终确保所有节点全部成功或者全部失败。 gcamdin showfailover 功能 显示当前保留在 gcware 中的所有 failover 信息。 ..

Mr_dar的博客 675

GBase 8a查看和清理故障恢复状态Failover的方法

GBase 8a在执行dml,ddl等数据变动业务时，为了避免发起节点出现故障，提供了failover机制来清理残余信息，保证集群一致性。针对一些特殊情况，特别是早期的版本，可能存在某些情况需要强行清理的情况。结合强行释放锁的操作，可以清理指定SQL占用的资源。本文提供的方案请慎重使用。 查看failover方法 在数据库操作系统dbauser下(一般是gbase)执行 gcadmin showfailover [gbase@rh6-1 ubas]$ gcadmin showfailover +=+ |

jingjing1068的博客 625

GBase 8a 高可用特性-故障检测切换

GBase 8a MPP Cluster的采用联邦架构，协调节点和计算存储节点均构成高可用集群，在单个服务器节点，无论是协调节点还是计算存储节点，磁盘故障或服务器节点故障时，集群会自动故障检测切换，不需人工干预,元数据和数据自动进行恢复，实现业务不中断，不影响对外服务，保障节点故障场景下无单点失败风险。 在集群中某一协调节点发生节点故障时，GCware首先会检测到该节点上的故障，并将这一故障状态通知给集群内其他协调节点，此时GCware会协调这个节点从集群成员中离开并调整集群构成的元数据信息更新。当节点故

qq_22310167的博客 389

GBase 8 字符集不一致导致的主备不一致报错案例分析

问题描述： 集群从8节点扩至16节点，扩容后出现一个报错（主副分片不一致），可以正常创建与原库一样的表，但在表中插入数据就会报主备分片不一致的报错 报错信息： ERROR 1705（HY000）：gcluster DML error (IP:5050)(GBA-02AD-0005)Failed to query in gnode; DETAIL: (GBA-01EX-700) Ggase general error: (gns_host：IP) source table and destinatio

Mr_dar的博客 4274

gbase8a服务器堆使用率已超过MemoryLimit，经常导致sql执行失败

gbase8a服务器堆使用率已超过MemoryLimit，经常导致sql执行失败

唐可盐的专栏 1654

gbase数据库建表

CREATE TABLE "dwd_o_cust_person_day" (  "MONTH_ID" varchar(20) DEFAULT NULL COMMENT '月份',  "DAY_ID" varchar(40) DEFAULT NULL COMMENT '日期',  "AREA_NO" varchar(10) DEFAULT NULL COMMENT '地市',  "CITY_NO" ...

weixin_34718910的博客 1万+

GBase 8a EVENT事件

CREATE [DEFINER = { user | CURRENT_USER }] EVENT [IF NOT EXISTS] event_name ON SCHEDULE schedule [ON COMPLETION [NOT] PRESERVE] [ENABLE | DISABLE] [COMMENT ‘comment’] DO event_body; 参数说明： event_name ：创建的 event 名字（唯一确定的）。 ON SCHEDULE：计划任务。 schedule: 决定 ev.

qq_37004539的博客 297

Gbase数据库中文排序异常原因探究

数据库字段排序Gbase 中文排序错乱根源是字符集二进制编码，UTF8、GBK 下汉字编码数值不同会导致排序顺序相反，且字段字符集优先级高于表默认编码，仅修改表字符集无法修正排序，文中附完整建表测试案例。方式追溯排序的编码方式，说明了查看编码的方式和更改编码的方式

weixin_42968761的博客 853

GBase 8a 对主副本不一致时的处理方案测试

GBase 8a 对主副本不一致时的处理方案测试

qq_22310167的博客 614

GBase 8a 创建event

GBase 8a event创建

m0_37936059的博客 279

GBase 8a管理员常用命令---showddlevent和showdmlevent

GBase 8a管理员常用命令---showddlevent和showdmlevent

weixin_34421618的博客 1911

gbase8aV86集群进行数据分片手动同步

对V86集群自动恢复无法完成需要人工干预场景，进行数据分片手动全同步 需要手动全同步的情况，主要分为如下两种： 1、集群中表的某一个分片因某种原因自动恢复无法完成，此时使用正常的分片去覆盖异常的分片进行手动同步。 2、集群中表的某一个分片因某种原因主备片同时不可用（损坏或者同时都记录event），此时主备片的数据都是不可信的， 只能人工从主备分片中选择一个分片作为有效分片进行手动同步。 场景举例 三节点集群，集群IP：192.168.92.129，192.168.92.130，192.168.92.

pengrander的博客 1756

GBase 8a 管理员常用命令---showdmlstorageevent和showcluster

GBase 8a 管理员常用命令---showdmlstorageevent和showcluster

weixin_34421618的博客 2185

gbase报错总结(持续更新)

gbase报错总结文件入库 文件入库 执行load语句时，报I/O 错误 gbase@gbase01:~$ gccli -uroot -Dssbm -vvv -e "load data infile 'sftp://gbase:gbase123@172.16.227.110//opt/ssbm/data/lineorder.tbl' into table lineorder data_format 3 fields terminated by '|';" -------------- load data i

weixin_36815898的博客 5833

GBase 8a 集群扩容

GBase 8a 集群扩容

GNAIXGNAHZ的博客 834
上一篇： GBase 8a数据库加载LOAD报错信息分析和解决文章汇总
下一篇： GBase 8a数据库集群新手使用入门
点赞 0
评论 0
书签 书签 1
老紫竹
老紫竹
博客等级 码龄19年 1万+粉丝 1252原创
+ 关注 专栏目录 订阅专栏
评论
成就一亿技术人!
拼手气红包6.0元
还能输入1000个字符
 
表情包 插入表情
表情包 代码片
 条评论被折叠 查看
添加红包

请填写红包祝福语或标题

个

红包个数最小为10个

元

红包金额最低5元

当前余额3.43元 前往充值 >
需支付：10.00元
成就一亿技术人!
领取后你会自动成为博主和红包主的粉丝 规则
hope_wisdom
发出的红包
实付元
使用余额支付
点击重新获取
扫码支付
钱包余额 0

抵扣说明：

1.余额是钱包充值的虚拟货币，按照1:1的比例进行支付金额的抵扣。
2.余额无法直接购买下载，可以购买VIP、付费专栏及课程。

余额充值