Commit graph

52 commits

Author SHA1 Message Date
Kaili Xu
13ae5350b7
feat: rollup sma (#33063) 2025-09-26 17:32:32 +08:00
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
Hongze Cheng
b26166b48d support meta_only option 2025-02-25 17:57:25 +08:00
Shengliang Guan
adf7ea29cb feat: alloc dnode from input dnode list 2024-10-29 14:40:40 +08:00
Shengliang Guan
02d56519a2 feat: parse dnodeId from dnodeList 2024-10-29 11:23:58 +08:00
dmchen
55efe1b92b fix/restore-2-replica 2024-09-12 10:34:32 +00:00
Shungang Li
3a656c3595 feat: support 2 replica with arbitrator 2024-03-19 09:30:35 +08:00
dm chen
b899a8b7df
Merge branch '3.0' into feat/TD-22970 2023-08-22 10:12:30 +08:00
Benguang Zhao
f01a26f68a enh: refactor func names of new Vg prepare actions 2023-08-18 14:26:58 +08:00
dmchen
8062d603b7 restore old change replica 2023-07-19 15:35:37 +08:00
Benguang Zhao
4a6d81855b enh: add mndTransValidate for prepare actions 2023-05-30 18:50:00 +08:00
Benguang Zhao
3c698b2461 enh: add mndAddPrepareNewVgAction 2023-05-29 14:13:11 +08:00
dmchen
9ff56a4e2b unsafe command 2023-05-16 14:02:54 +08:00
cadem
843d606058 feat/restore dnode 2023-05-09 17:15:37 +08:00
Benguang Zhao
72c6292ab4 enh: declare mndSplitVgroup in mndVgroup.h 2023-05-06 10:03:03 +08:00
Hongze Cheng
89281ee488 compact with time range 2023-03-07 21:46:34 +08:00
Shengliang Guan
7a3079e79a enh: send compact msg from mnode to vnode 2022-12-27 16:12:50 +08:00
Shengliang Guan
7e9b7f4e15 enh: show more info while drop dnode 2022-11-30 14:06:22 +08:00
Shengliang Guan
7ddde569a0 enh: force drop dnode 2022-10-31 17:29:24 +08:00
Shengliang Guan
4d318b32da refact: alter db replica 2022-10-20 16:47:03 +08:00
Shengliang Guan
dbae9a47da fix: check memory while alter db buffer 2022-10-11 16:23:23 +08:00
Shengliang Guan
630eebecd5 enh: check available memory of dnode while creating db 2022-07-04 17:50:35 +08:00
Shengliang Guan
344ef7e1bc fix: drop tsma 2022-07-01 17:09:44 +08:00
Shengliang Guan
4cc6b1fb62 feat: redistribute vgroups 2022-06-10 18:56:11 +08:00
Shengliang Guan
e85e323863 feat: split vgroup 2022-06-09 20:46:15 +08:00
Shengliang Guan
5838956831 refactor: dnode online 2022-06-06 14:45:53 +08:00
Shengliang Guan
980e289cdf refactor: add alter-confirm while alter db 2022-06-05 14:59:53 +08:00
Shengliang Guan
7fcfd45a03 feat: increase the number of database replicas 2022-06-01 17:26:36 +08:00
Shengliang Guan
381ff01f52 feat: add vgroup for tsma 2022-05-26 13:18:39 +08:00
Shengliang Guan
f94eaa6730 fix: only send message to one vnode in the vgroup 2022-05-18 20:41:21 +08:00
Shengliang Guan
0886d23e02 fix: alter db replications 2022-05-06 22:17:56 +08:00
Shengliang Guan
ae23dd2382 fix: alter db replications 2022-05-06 17:28:58 +08:00
Shengliang Guan
2383b316d7 refactor: adjust db codes 2022-05-05 21:52:18 +08:00
Shengliang Guan
149619b172 refactor: return vgroup global verison 2022-04-26 10:28:23 +08:00
dapan1121
8cfcd6d4f4 feature/qnode 2022-04-22 17:48:12 +08:00
Liu Jicong
70cddceb60 assign task to vg 2022-03-21 16:35:27 +08:00
Shengliang Guan
7ce10a868b serialize create vnode msg 2022-02-16 14:57:47 +08:00
Shengliang Guan
2b73e4f424 serialize drop vnode req 2022-02-16 14:10:09 +08:00
Shengliang Guan
155978b883 minor changes 2022-01-10 00:31:50 -08:00
Shengliang Guan
33a60b6044 minor changes 2022-01-07 23:20:18 -08:00
Shengliang Guan
c9f8a1109f fix show dnodes error 2021-12-28 23:10:50 -08:00
Shengliang Guan
293ad0bb06 TD-10431 process stable create msg in mnode 2021-12-21 16:16:12 +08:00
Shengliang Guan
8bba00f247 TD-10431 build create vnode msg 2021-12-18 20:56:05 +08:00
Shengliang Guan
190a798530 TD-10431 save vgroup 2021-12-16 13:35:51 +08:00
Shengliang Guan
9dbefbe543 TD-10431 alloc vgroup 2021-12-16 11:37:41 +08:00
Shengliang Guan
58d819ee86 TD-10431 add numOfVgroup to db object 2021-12-15 18:37:28 +08:00
Shengliang Guan
8d77b0a7bb TD-10431 vgroup manage 2021-12-09 16:47:23 +08:00
Shengliang Guan
23da7d4230 TD-10431 create dnode 2021-12-08 18:50:52 +08:00
Shengliang Guan
35b0c69333 TD-10431 rename some funcs 2021-11-29 13:29:13 +08:00
Shengliang Guan
6ee8832271 TD-10431 fix compile errors in mnode 2021-11-29 12:09:18 +08:00