Commit graph

260 commits

Author SHA1 Message Date
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
xinsheng Ren
789f36edda
docs: udf function, extract_avg (#30244) 2025-03-18 17:52:21 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
freemine
958cf30e25 unify among LINUX and MacOSX 2025-01-04 10:52:05 +08:00
freemine
9c7278f642 typo correction, and add MacOSX support 2024-12-27 17:21:25 +08:00
freemine
0c69fff849 fix ps/grep/kill flaws found in test.script 2024-12-27 17:09:40 +08:00
Haojun Liao
47634f4f18 test: add a new test case for TDgpt. 2024-11-26 22:48:30 +08:00
chenhaoran
01ad6a685b tetst:sleep 2 after stop taosd in windows ci 2024-09-03 19:37:22 +08:00
chenhaoran
639ec1211d tetst:check stream task status in common.py 2024-08-28 20:54:42 +08:00
chenhaoran
8a5cf9cebc tetst:add memory leak detection for TD in psiminfo to the CI 2024-08-26 10:44:23 +08:00
Shengliang Guan
7be4d5b592 enh: adjust udf example codes 2024-08-08 19:19:23 +08:00
Shengliang Guan
09f6411f66 fix: format udf example codes 2024-08-08 16:22:49 +08:00
Shengliang Guan
05be996342 enh: adjust bit_add example codes 2024-08-08 16:18:45 +08:00
Haojun Liao
6f1662c2be fix(test): wait for a little bit more time. 2024-06-13 17:38:39 +08:00
Yihao Deng
0c25b71add change test case 2024-05-14 20:11:18 +08:00
chenhaoran
ad182673fb test: configure debug flag to 135 or 131 in dnodes 2024-04-16 14:27:06 +08:00
Haojun Liao
993a340f3d
Merge pull request #25044 from taosdata/coverage/replace_kill_signal
replace kill signal
2024-03-08 17:19:23 +08:00
Ping Xiao
ecfc5ec439 replace kill signal 2024-03-08 16:12:27 +08:00
Haojun Liao
3de1413974 test(stream): add some test cases to inc coverage. 2024-03-08 13:38:51 +08:00
wangmm0220
c7d115d8aa fix:add excluded msg for delete in tmq 2024-03-04 14:58:13 +08:00
charles
8b50f346aa udpate checkAsan.sh 2024-02-04 18:09:21 +08:00
charles
57148a7731 udpate checkAsan.sh to skip asan_malloc_linux.cpp error 2024-02-02 10:16:14 +08:00
chenhaoran
3a1d8f2e8f test: replace timestamp now with var ts 2024-01-15 19:14:24 +08:00
Haojun Liao
d87db2f829 fix(stream): fix the pause failure bug. 2023-10-09 14:48:11 +08:00
dapan1121
cc17848b78 fix: merge 3.0 to main 2023-09-01 13:24:47 +08:00
yihaoDeng
4372ebdab5 change asan leak 2023-08-30 21:48:16 +08:00
yihaoDeng
4b1e73860f change asan leak 2023-08-30 21:25:17 +08:00
yihaoDeng
2ed576748a fix merge error 2023-08-30 18:18:21 +08:00
kailixu
f70709be3c fix: run sim cases on windows 2023-07-28 14:54:03 +08:00
Haojun Liao
57fcd553f8 refactor: do some internal refactor. 2023-06-28 13:13:13 +08:00
shenglian zhou
90f542857d enhance: improve document 2023-05-25 11:13:05 +08:00
Haojun Liao
99148d6719 other: merge 3.0 2023-05-13 20:02:48 +08:00
chenhaoran
57d67faabe test:add testcases of learner in election 2023-05-11 19:05:21 +08:00
yihaoDeng
5708fcb5cb Merge branch 'enh/3.0' into enh/rocksdbSstateMerge 2023-05-10 08:02:16 +00:00
Minglei Jin
3822a5858b sim/valgrind: suppress rocks reachables 2023-04-26 16:03:37 +08:00
yihaoDeng
4687430a65 remove reachable mem check 2023-04-18 20:03:51 +08:00
Alex Duan
ad6dd54a00 test: add udf function return with varchar 2023-04-07 18:14:02 +08:00
slzhou
ba82f05d68 fix: keep original udf file name when rename file error 2023-03-17 20:19:46 +08:00
slzhou
3bf9ea9e59 fix: prepare_pyudf.sh is not executable 2023-03-17 15:22:06 +08:00
slzhou
7360011a9e fix: move from run_case.sh to each case 2023-03-17 14:08:18 +08:00
slzhou
e71df1aca6 enhance: add taospyudf test and run it in CI 2023-03-16 19:06:18 +08:00
Xuefeng Tan
449f7115e4
fix: windows test (#20256)
* fix: windows sim test

* fix: windows sim test

* fix: random on windows

* fix: taosd online check

* fix: windows path in python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: windows python test

* fix: make_install.bat

* fix: python test

* fix: jdbc test

* fix: develop test

* fix: develop test

* fix:add log

* test:add win case file

* test:add win case file

* test:add win case file

* fix: python test get taosBenchmark path

* test:add win case file

* test:comment some failed cases in win

* test:add win case file

* test:add win case file

* test:comment some win case file

* test:add some win case file

---------

Co-authored-by: wangmm0220 <wangmm0220@gmail.com>
Co-authored-by: chenhaoran <haoran920c@163.com>
2023-03-16 10:18:06 +08:00
Xiaoyu Wang
961107d1ca merge main 2023-02-24 16:55:50 +08:00
Xiaoyu Wang
1ab144609e merge main 2023-02-23 14:58:22 +08:00
Ganlin Zhao
a185561405 fix: fix asan error 2023-02-22 18:46:59 +08:00
chenhaoran
650147ccb8 fix: ingore filter.c error found by asan 2023-02-22 14:46:19 +08:00
chenhaoran
96389577d7 fix: ingore filter.c error found by asan 2023-02-22 14:42:17 +08:00
Shuduo Sang
ea5d925c50
fix: alpine support (#19353)
* fix: alpine support

* fix: ostimer thread_id

* fix: taosPrintTrace and tsStreamMax

* fix: make_install.sh and dnodes.py for alpine

* fix: cmake.platform and cut option for alpine

* fix: timer sigev and string convert

* fix: test case for alpine

* fix: fix test script for alpine

* fix: fix test script for alpine

* fix: fix taosLogTrace for Alpine

* fix: fix taosSort

* fix: vnode and mnode compare functions and test cases

* fix: sorting algorithms and unit tests

* fix: libtaosws.so build error on Alpine

* fix: libtaosws.so build error on Alpine

---------

Co-authored-by: t_max <1172915550@qq.com>
2023-02-10 13:00:55 +08:00
chenhaoran
ca1e1c1694 test:modify failed cases in ci 2023-01-13 10:30:43 +08:00
Shengliang Guan
90e44ced3e test: add logs 2022-12-23 11:15:45 +08:00