Commit graph

118 commits

Author SHA1 Message Date
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
Simon Guan
e777dc5e9f
fix: the REPLICA parameter supports plural forms when used to create and alter a database (#30732) 2025-04-10 16:48:06 +08:00
Shengliang Guan
fb4486cedb Merge branch 'main' into merge/mainto3.0 2024-11-28 19:21:25 +08:00
factosea
d50ee39dab fix: basic1.sim case 2024-11-26 10:01:09 +08:00
Haojun Liao
4dd4f217c1 fix(test): update test cases. 2024-11-14 17:43:00 +08:00
Shengliang Guan
689cdd4e44 test: add dnodes cases 2024-10-29 21:33:36 +08:00
Minglei Jin
cd470aa2a6 s3/test: default duration to 100 with replicas 2024-10-23 17:00:56 +08:00
lyh250-666
cf35e31289 modify test case 2024-09-11 09:43:55 +00:00
lyh250-666
25c0b92885 feat:Optimising the duration and keep 2024-09-11 01:11:56 +00:00
Minglei Jin
11b26d80b7 wal_level 0: update unit & tsim test cases 2024-06-05 18:22:23 +08:00
Minglei Jin
a491e250fb cos/multi-write: fix compilation on mac 2024-04-08 13:24:47 +08:00
Shungang Li
3a656c3595 feat: support 2 replica with arbitrator 2024-03-19 09:30:35 +08:00
Benguang Zhao
bad2bb52b5 enh: update test cases for show vnodes 2023-08-16 20:17:11 +08:00
Benguang Zhao
d776abb634 fix: adjust test cases for default WAL_RETENTION_PERIOD as 3600 2023-07-26 14:20:50 +08:00
Ganlin Zhao
ef242ba8fa fix test cases 2023-07-11 11:11:57 +08:00
Alex Duan
b925342ad0 test: fix tsim/db/error1.sim random failed 2023-05-06 15:08:33 +08:00
Alex Duan
fc799bcd19 test: fix tsim/db/error1.sim random failed 2023-05-06 15:06:53 +08:00
dapan1121
877b6e7053 fix: add table_prefix/table_suffix cases 2023-04-21 14:58:58 +08:00
dapan1121
948a603513 fix: fix minrows/maxrows case issue 2023-03-16 14:49:27 +08:00
Hongze Cheng
b86c23b963 change test case 2023-03-09 18:07:38 +08:00
Benguang Zhao
96f5249782 enh: flush database in testcase alter_replica_13.sim to trigger snapshot repl 2023-01-29 15:58:23 +08:00
Xiaoyu Wang
9dff4371ec enh: shielding database option 'strict' 2022-12-13 12:20:48 +08:00
Shengliang Guan
1dd1e497f9 test: update case 2022-12-07 23:57:51 +08:00
Shengliang Guan
66f4ee9a41 enh: alter db replica return quickly, operation executed asynchronously 2022-12-07 21:57:49 +08:00
Shengliang Guan
6d94afe48f
Revert "refact: adjust some assert cases" 2022-12-07 21:51:17 +08:00
Shengliang Guan
64e39cf2d4 enh: alter db replica return quickly, operation executed asynchronously 2022-12-07 20:16:22 +08:00
Shengliang Guan
a33799fe26 test: adjust alter replica case 2022-11-28 19:34:25 +08:00
Shengliang Guan
9c426540e7 fix: memory leaks while check mnode state 2022-11-18 20:15:13 +08:00
Cary Xu
69c1465142 Merge branch '3.0' into feat/TS-2007 2022-11-10 15:49:15 +08:00
Cary Xu
8205dad2a1 enh: adjust test case to support wider range of future timestamp 2022-11-10 15:48:31 +08:00
Hongze Cheng
683e2cff7c change test cases 2022-11-09 15:49:39 +08:00
Shengliang Guan
ff428696d5 enh: drop snode 2022-10-27 15:03:18 +08:00
Minghao Li
913517f05a fix(sync): modify case alter_option.sim, default value 2022-10-26 19:49:12 +08:00
Shengliang Guan
6d1bf00cee fix: use sync cfg from vnode 2022-10-24 12:42:44 +08:00
Shengliang Guan
bbb0475d69 fix: alter replica one by one 2022-10-24 10:05:57 +08:00
Shengliang Guan
6c3cd3e19e test: adjust log 2022-10-21 10:39:29 +08:00
Xiaoyu Wang
ccaf6f8a78 feat: add database option 'table_prefix' and 'table_suffix' 2022-09-03 11:22:36 +08:00
Shengliang Guan
ae54c4a50b enh: let show vnodes works 2022-09-02 14:24:45 +08:00
Shengliang Guan
8f6aaf0a60 fix: donot retry if error code not match retry code 2022-08-24 13:47:27 +08:00
Ganlin Zhao
234b89767d fix test case 2022-08-18 19:33:45 +08:00
Shengliang Guan
e73a311a08 test: add case for dupliate supertable name 2022-08-16 19:30:32 +08:00
Minghao Li
e8d63b7228 refactor(sync): update DEFAULT_DB_WAL_RETENTION_PERIOD 2022-08-12 17:35:41 +08:00
Xiaoyu Wang
44df51f98f enh: show command optimize 2022-08-11 20:56:31 +08:00
Xiaoyu Wang
7176bb9198 merge 3.0 2022-08-11 20:30:02 +08:00
Xiaoyu Wang
9a9aef985e enh: show command optimize 2022-08-11 20:26:40 +08:00
Shengliang Guan
29402da45f test: adjust case 2022-08-09 20:46:37 +08:00
Xiaoyu Wang
04a2e8e541 fix: the default value of wal-related database parameters is incorrect 2022-08-09 11:20:02 +08:00
Shengliang Guan
8cebad4739 test: restore case 2022-08-01 11:54:18 +08:00
Shengliang Guan
9187231615 test: valgrind case 2022-08-01 11:48:55 +08:00
Shengliang Guan
f491de4fdc test: adjust case 2022-07-31 22:35:42 +08:00