Commit graph

55 commits

Author SHA1 Message Date
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
xiao-77
bc95f82373 add ut . 2024-12-30 17:45:43 +08:00
xiao-77
a6e1af6b42 Merge branch '3.0' into enh/3.0/TS-5007 2024-12-13 15:04:36 +08:00
xiao-77
c50781d089 Send alter msg sync. 2024-12-11 15:46:44 +08:00
kailixu
eaf5978b2c Merge branch '3.0' into feat/TD-30268 2024-12-10 18:49:15 +08:00
Zhixiao Bao
405d924a6f
Merge branch '3.0' into enh/3.0/TS-5007 2024-12-10 14:32:43 +08:00
dapan1121
011ff04234 fix: scheduler dead lock issue 2024-12-05 18:07:37 +08:00
xiao-77
eb202e556a Fix sdb lost val while kill taosd. 2024-12-04 19:32:21 +08:00
xiao-77
d778ada8f9 Add show 2024-11-26 16:08:16 +08:00
xiao-77
5d2bc35569 Add more log info. 2024-11-25 16:23:34 +08:00
xiao-77
a579abd0f0 Add dyn lazy type. 2024-11-22 17:52:17 +08:00
xiao-77
db67245af0 Add more dyn configs and fix vesion in mnd bug. 2024-11-22 15:57:18 +08:00
xiao-77
f46037870d Add version control and persist. 2024-11-21 15:34:32 +08:00
xiao-77
4b8feab900 fix mnd config persist. 2024-11-18 19:52:41 +08:00
wangmm0220
7aef49638e feat:[TD-32642] add timezone logic 2024-11-15 19:44:03 +08:00
xiao-77
cc3f4504bf Add mnd init cfg and mnd cfg trans. 2024-11-14 17:20:50 +08:00
xiao-77
748bb000b0 fix some error and rename some functions 2024-11-13 16:51:26 +08:00
xiao-77
07de144000 serialize and persist local config 2024-11-04 16:16:24 +08:00
xiao-77
bfceaae124 fix CFG_DYN_ENT_READONLY declare error 2024-11-01 16:29:47 +08:00
xiao-77
9f6504cdf4 Add category to config item and rename array to loaclArray. 2024-11-01 10:24:26 +08:00
xiao-77
cda8bd8047 Add category in configuration's struct 2024-10-21 13:51:33 +08:00
wangmm0220
a2e7e18440 fix:[TD-31696] change fmin/fmax in SConfigItem from double to float 2024-08-28 10:13:25 +08:00
Shungang Li
4281289eef feat: (errcode) tconfig.c 2024-07-24 15:16:46 +08:00
Haojun Liao
1f6b7bbcff fix(util): fix dead lock. 2024-04-27 10:08:15 +08:00
Haojun Liao
faa87e13b0 fix(util): fix race condition. 2024-04-26 19:12:19 +08:00
Haojun Liao
c7806ebca2 fix(util): fix race condition. 2024-04-25 13:58:02 +08:00
Minglei Jin
44b9785853 cos/multi-write: include headers part 2024-04-08 10:33:02 +08:00
Shungang Li
de631e20a5 fix: refactor logdebugs 2024-01-03 19:57:22 +08:00
Shungang Li
c6f5e2a9bd enh: add dynamic configuration 2023-11-03 14:28:02 +08:00
Haojun Liao
4f1a5423cb fix(config): remove the dval from struct SConfigItem. 2023-10-27 09:54:07 +08:00
Alex Duan
ea1fb3a195 fix: cfgAddFloat with float params 2023-09-23 12:03:01 +08:00
lx1zhong
6da4088012 add TSZ and ADT-FSE algorithm 2023-09-15 14:30:23 +08:00
Ganlin Zhao
707be3a6fe unify scope for client/system/server configs 2023-07-17 15:53:08 +08:00
Ganlin Zhao
cdb3453812 add show local variables scope 2023-07-14 14:38:35 +08:00
Hongze Cheng
4cc94968ee refact: format header file 2022-10-13 11:09:43 +08:00
dapan1121
33452da7bd feat: alter local command 2022-06-22 19:45:41 +08:00
dapan1121
66cfe3a3fc feat: show dnode variables 2022-06-22 17:17:18 +08:00
afwerar
f966242047 feature(dnode): mul parameter start. 2022-04-25 13:57:55 +08:00
Shengliang Guan
f6597c37f4 fix: make taos_options work 2022-04-19 17:54:57 +08:00
Shengliang Guan
c6fb4e928a Also start taosd when the config file does not exis 2022-03-01 17:34:55 +08:00
Shengliang Guan
9a9e87ec34 config from hash to array 2022-02-28 16:29:25 +08:00
Shengliang Guan
68f3de6111 remove tfile 2022-02-28 13:49:38 +08:00
Shengliang Guan
f5e08cdf74 tfs cfg 2022-02-25 16:07:39 +08:00
Shengliang Guan
0a38853fc3 tfs config 2022-02-25 15:49:22 +08:00
Shengliang Guan
6f92389ad0 arg list 2022-02-25 10:40:53 +08:00
Shengliang Guan
546d3e84d5 daemon config 2022-02-24 19:06:39 +08:00
Shengliang Guan
e321edbeb2 config 2022-02-24 18:16:16 +08:00
Shengliang Guan
4278439666 refact config 2022-02-24 18:13:57 +08:00
Shengliang Guan
18634533a5 config 2022-02-22 13:36:26 +08:00
Haojun Liao
04057a39e2 [td-10564] connect to server success. 2021-12-13 19:31:00 +08:00