Commit graph

96 commits

Author SHA1 Message Date
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
Haojun Liao
6f1662c2be fix(test): wait for a little bit more time. 2024-06-13 17:38:39 +08:00
chenhaoran
ad182673fb test: configure debug flag to 135 or 131 in dnodes 2024-04-16 14:27:06 +08:00
Haojun Liao
3de1413974 test(stream): add some test cases to inc coverage. 2024-03-08 13:38:51 +08:00
wangmm0220
c7d115d8aa fix:add excluded msg for delete in tmq 2024-03-04 14:58:13 +08:00
Haojun Liao
d87db2f829 fix(stream): fix the pause failure bug. 2023-10-09 14:48:11 +08:00
dapan1121
cc17848b78 fix: merge 3.0 to main 2023-09-01 13:24:47 +08:00
Haojun Liao
57fcd553f8 refactor: do some internal refactor. 2023-06-28 13:13:13 +08:00
Shuduo Sang
ea5d925c50
fix: alpine support (#19353)
* fix: alpine support

* fix: ostimer thread_id

* fix: taosPrintTrace and tsStreamMax

* fix: make_install.sh and dnodes.py for alpine

* fix: cmake.platform and cut option for alpine

* fix: timer sigev and string convert

* fix: test case for alpine

* fix: fix test script for alpine

* fix: fix test script for alpine

* fix: fix taosLogTrace for Alpine

* fix: fix taosSort

* fix: vnode and mnode compare functions and test cases

* fix: sorting algorithms and unit tests

* fix: libtaosws.so build error on Alpine

* fix: libtaosws.so build error on Alpine

---------

Co-authored-by: t_max <1172915550@qq.com>
2023-02-10 13:00:55 +08:00
Shengliang Guan
90e44ced3e test: add logs 2022-12-23 11:15:45 +08:00
Shengliang Guan
8a7d34eae7 fix: adjust log 2022-11-04 19:41:11 +08:00
Cary Xu
b07670a680 enh: improve test case coverage of tsma 2022-10-25 18:03:09 +08:00
Shengliang Guan
8764cf9adb refact: remove multi process codes 2022-10-19 11:08:42 +08:00
Shengliang Guan
724d2e0f5f test: valgrind case 2022-07-14 10:51:09 +08:00
Shengliang Guan
9fe73acc86 refactor: adjust logs 2022-07-10 18:53:52 +08:00
Shengliang Guan
6e12d8e707 refactor: adjust log 2022-07-08 15:20:15 +08:00
Shengliang Guan
0feee09804 test: cases with 100 dnodes 2022-07-04 20:13:39 +08:00
Shengliang Guan
76e426490c fix: drop sma 2022-07-01 20:26:48 +08:00
Shengliang Guan
9b273e1766 feat: create mnode 2022-05-24 15:05:01 +08:00
Liu Jicong
7b892a0f28 fix: client memory leak 2022-04-26 21:50:36 +08:00
Shengliang Guan
eedd733230 enh: add cb function for transaction 2022-04-26 14:14:28 +08:00
afwerar
97a373fa17 fix(build): exec output build bin. 2022-04-16 15:12:11 +08:00
afwerar
09be9e4d9d fix(test): enterprise test. 2022-04-14 17:49:48 +08:00
Shengliang Guan
5a2a34f1bd add multiprocess mode to CI 2022-04-01 19:07:05 +08:00
Shengliang Guan
b2405a0098 minor changes 2022-03-29 20:04:03 +08:00
Shuduo Sang
43feda0595
[TD-13408]<test>: move tests in for3.0 (#10598)
* restore .gitmodules

* Revert "[TD-13408]<test>: move tests out"

This reverts commit f80a4ca49f.

* revert f80a4ca49f

* immigrate file change from stand-alone repo to TDengine

for 3.0

* remove tests repository for Jenkinsfile2

Co-authored-by: tangfangzhi <fztang@taosdata.com>
2022-03-07 19:25:29 +08:00
Shuduo Sang
bf5a3e106b
[TD-13408]<test>: move tests out (#10176)
* [TD-13408]<test>: move tests out

for 3.0 branch

* add Jenkinsfile2

Co-authored-by: tangfangzhi <fztang@taosdata.com>
2022-02-10 15:25:50 +08:00
Shengliang
2b135f6eef minor changes 2022-01-23 20:27:00 +08:00
Shengliang Guan
bf7c5baa89 minor changes 2022-01-22 12:03:25 +08:00
Shengliang Guan
e3ee88244e TD-12868 2022-01-10 23:42:51 -08:00
Shengliang Guan
b2cf6b3496 add support vnodes config 2021-12-29 19:35:46 -08:00
Shengliang Guan
6f7eecb215 add dnodes sim 2021-12-27 22:57:20 -08:00
Shengliang Guan
147f5af9b2 adjust sim script 2021-12-17 18:23:13 +08:00
Ganlin Zhao
e98b8355c3 [TD-5352]<fix>:
Unable to run tsim test script on MacOS
2021-07-16 19:56:23 +08:00
Shengliang Guan
f82f11e26d scripts 2020-12-22 18:15:07 +08:00
Shengliang Guan
2db5df4c40 TD-2498 2020-12-20 19:07:50 +08:00
Shengliang Guan
418f15e2d1 T-2270 change monitordebugflag 2020-12-03 18:04:18 +08:00
Shengliang Guan
007af68057 TD-2046 2020-11-20 14:35:27 +08:00
Shengliang Guan
566b749cc5 TD-2046 2020-11-20 12:32:42 +08:00
Shengliang Guan
97e41cdf8d TD-2046 2020-11-18 15:39:57 +08:00
Shengliang Guan
ccb8ab9328 TD-1670 2020-10-11 07:51:17 +00:00
Shengliang Guan
1a5272a49f TD-1520 2020-09-21 04:34:29 +00:00
Shengliang Guan
25c0b8cbe9 [TD-1293 ]<fix>: monitor uses too many resources and may cause auth failure 2020-09-05 09:50:15 +00:00
Shengliang Guan
9152898418 minor changes 2020-08-08 09:38:01 +08:00
Bomin Zhang
ca9c0ad282 TD-1033: enable telemetry reporting by default
disable telemetry reporting in test.
fix other telemetry related bugs.
2020-08-03 11:31:14 +08:00
Shengliang Guan
1fe3dbebfb set telemetryReporting default val to false 2020-08-02 11:48:42 +00:00
Hui Li
0bc21b2849 [modify install] 2020-07-30 11:08:29 +08:00
Shengliang Guan
fd7b5e6e86 scripts 2020-07-23 13:29:14 +08:00
Shengliang Guan
d896fc19d6 [TD-913] 2020-07-17 14:39:47 +00:00