Commit graph

123 commits

Author SHA1 Message Date
Pan Wei
bcc25e56bc
feat: new stream (#31678)
* fix: windows compile issue

* test: add vtable cases (#31829)

* fix: windows compile issues

* test:add test cases

* fix: windows compile issue

* case: em-4 stream case submit

* test: stream4_sub1 found bug2

* test: submit test_scene_meters_bug2.py

* add stream parameters example

* feat: [TS-6100] Do not translate const value as column.

* Feat/ts 6100 3.0 zlv (#31747)

* modify asan exampel

* modify asan exampel

* add example

* add example

* modify case example

---------

Co-authored-by: zelv01 <1101510017@qq.com>

* feat(stream): fix memory leak

* modify sliding example

* test: update test case.

* feat(stream): fix conflicts

* fix: add offset case 10a 10s 10m 10h 10d

* feat(stream): fix conflicts

* chore(stream): rename case name #TS-6100

* add case

* modify example

* fix: windows compile issues

* fix: data null check

* feat: [TS-6100] Forbid where when using %%trows (#31827)

* feat: [TS-6100] Forbid where when using %%trows

* test: update cases

* feat: [TS-6100] Fix leaks.

---------

Co-authored-by: Simon Guan <guanshengliang@qq.com>

* test: reproduce bugs

* test: update test case.

* test: update test case.

* feat: [TS-6100] Fix leaks.

* test: add cases

* Feat/ts 6100 3.0.pw10 (#31841)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* test: reproduce bugs

* fix: add sliding interval combine case

* test: add cases

* test: add recalc test.

* test: reproduce bugs

* case : add vt ts is null check

* modify case

* bug: submit test_idmp_meters_bug3.py

* test: add test for recalc.

* test: add cases

* fix: error code check

* test: add cases

* fix(stream): scan wal with schema in that version

* add case

* test: add cases

* test: update test case.

* fix: windows compile issues

* add case

* test: add cases (#31845)

* modify case

* fix: reset interpPrev

* test: add test_idmp_meters bug4 and bug3

* add case

* fix(stream): opti wal interface

* fix: remove test_idmp_meters_bug5.py

* test: add cases

* fix(stream): fix ts data fetch for virtual tables

* cancel asan case

* test: update test case.

* test: update test case.

* add case

* test: add cases

* test: add cases

* test: add case test_idmp_meters_bug5.py

* test: update test case.

* fix(stream): tmq error

* test: add cases

* feat: [TS-6100] Restore deleted code in mndSma.c since they are still in use.

* fix(stream): optimize val scan logic

* test: add test_recalc_expired_time.py  to ci.

* test: update test case.

* test: update test case.

* feat: [TS-6100] Fix fill range check

* fix(stream): optimize val scan logic

* add case

* test: modify for partition by %%1

* test: add fun case stream4_sub7

* fix(stream): optimize val scan logic

* add case

* feat: [TS-6100] Rename OPTIONS to STREAM_OPTIONS.

* test: add test for recalc.

* test: use stream_options.

* fix: some cases error.

* test: remove recalc from ci.

* fix: ci case issues (#31880)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

* fix: external window idx issue

* fix: ci issues

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* fix(stream): fix compilation error

* fix(stream): optimize val scan logic

* test:add test cases

* test: modify case

* fix: external agg error

* test(stream): tobacco scene testing #TD-36514

* test: add stream cases (#31885)

* fix: windows compile issue

* fix: calc timerange

* fix: windows compile issue

* modify case

* fix(stream): compile error

* test: remove one debug test case file

* test: modify

* test: add test cases

* test: reproduce bugs

* test: reproduce bugs

* feat: [TS-6100] Placeholder function should only appera in SELECT and… (#31868)

* feat: [TS-6100] Placeholder function should only appera in SELECT and WHERE and FROM.

* test: update case

---------

Co-authored-by: Simon Guan <guanshengliang@qq.com>

* add example

* add example

* modify case example

* modify case

* test:alter sql

* test: add stream5 case

* fix(stream): get schema error with version

* test: add delete recalc test py.

* test: remove bug cases

* test: stream5 case test passed

* test: add state cases (#31893)

* fix(stream): compile error

* test: modify case

* test: add cases

* test: add test.

* test: update test case.

* chore(test): fix case err

* test: update test case.

* fix: align data get

* fix(stream): fix row index of datablock written into data cache

* fix: put align data

* test: update test case.

* test: add test cases for virtual table

* chore(test): fix case err #TD-36514

* add case

* test: add test for water mark.

* test: add meters bug6 for stream5

* test: add cases (#31903)

* test: add test for recalc.

* feat: [TS-6100] %%trows can only be used when event type is window close.

* test: add precision of database for ms/us/ns

* modify case

* add case

* add case

* test: add test to ci.

* modify case

* fix: ci case issues (#31904)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

* fix: external window idx issue

* fix: ci issues

* fix: ci case issues

* fix: drop dnode issue

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* fix(stream): ci error

* test: update test case.

* feat: [TS-6100] Disable some failed UT.

* feat: [TS-6100] Fix virtual table

* test: add bug 5.

* test: add test delete recalc to ci.

* test: add bug 6.

* test(stream): tobacco scene #TD-36514

* fix: reqCids,reqCols memory leak in SSTriggerRealtimeContext

Co-authored-by: Tony Zhang <tonyzhang@taosdata.com>

* test: add case stream6

* fix(stream): implement some pending features in trigger task

* modify case

* modify case

* fix: case issues

* modify case

* test: add recalc for warter mark.

* fix(stream): fix count window trigger of virtual tables

* fix(stream): memory leak

* test: fix run err.

* test: add stream6 bug7

* fix: adjust format

* test(stream): tobacco scene testing #TD-36514

* test: change bug7 with update window1 and 2

* test: add test bug 7.

* case: restore write 3 window

* fix: windows compile issue

* fix: notify

* test: add cases

* modify case

* test: update test case.

* test(stream): toobacco scene testing #TD-36514

---------

Co-authored-by: Simon Guan <slguan@taosdata.com>
Co-authored-by: plum-lihui <huili@taosdata.com>
Co-authored-by: Alex Duan <417921451@qq.com>
Co-authored-by: zelv01 <1101510017@qq.com>
Co-authored-by: Jing Sima <simondominic9997@outlook.com>
Co-authored-by: xiangyang guo <66111494+happyguoxy@users.noreply.github.com>
Co-authored-by: wangmm0220 <wangmm0220@gmail.com>
Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: zyyang90 <zyyang@taosdata.com>
Co-authored-by: Alex Duan <51781608+DuanKuanJun@users.noreply.github.com>
Co-authored-by: facetosea <285808407@qq.com>
Co-authored-by: Simon Guan <guanshengliang@qq.com>
Co-authored-by: huohong <sallyhuo@taosdata.com>
Co-authored-by: Li Hui <52318143+plum-lihui@users.noreply.github.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: xiao-77 <berylbao@taosdata.com>
Co-authored-by: Zhixiao Bao <62235797+xiao-77@users.noreply.github.com>
Co-authored-by: happyguoxy <happy_guoxy@163.com>
Co-authored-by: Tony Zhang <34825804+Tony2h@users.noreply.github.com>
Co-authored-by: Tony Zhang <tonyzhang@taosdata.com>
2025-07-16 14:42:16 +08:00
WANG MINGMING
ddd32e23d9
fix(tmq): init tmq handle error if snapshotVer is out of range (#30859) 2025-04-24 09:50:51 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
xiao-77
9096312b41 Replace unsafe memory functions with safe versions in wal&sync. 2024-12-13 09:21:43 +08:00
wangjiaming0909
65dffbda0c fix tsma drop ctb 2024-11-08 09:47:04 +08:00
xiao-77
ca7f490e6d fix invaild snapshotVer while repair wal meta file 2024-10-15 16:53:15 +08:00
Hongze Cheng
645160d251 ehn: remove void 2024-09-24 10:06:30 +08:00
xiao-77
b49d9f3691 change mutex lock in SWal to rwlock 2024-09-11 19:48:50 +08:00
dmchen
04764316b1 fix/TD-30849-fix-unit-test-compile 2024-08-15 06:07:15 +00:00
dmchen
b85dba328c fix/TD-30849 2024-08-15 03:55:20 +00:00
Minglei Jin
38689df0b9 use new return macros for wal read 2024-07-22 15:28:04 +08:00
Minglei Jin
fcce9ebafa return wal fsync error code 2024-07-19 15:20:38 +08:00
Minglei Jin
180411434a wal/ref: use new return macros for ref 2024-07-18 14:14:51 +08:00
Minglei Jin
964397ea63 move log macros into internal header 2024-07-18 10:29:25 +08:00
Minglei Jin
e6c86f5586 fix(wal/write): remove walWrite which is only used in unit testing 2024-07-17 14:45:23 +08:00
Minglei Jin
568799a6a3 fix(wal/level): enable alter replica from fake walLevel 1 2024-06-12 15:16:58 +08:00
Minglei Jin
d6b2bcba68 enh/wal_level: new level 0 to disable wal writing 2024-06-05 15:19:55 +08:00
dmchen
5f31575624 refactor 2024-04-09 06:55:01 +00:00
dmchen
b9deb71fc5 basic function 2024-03-29 02:48:49 +00:00
Benguang Zhao
c74c4fe208 enh: clean WAL logs based on size if one follower is offline. 2023-12-13 10:10:02 +08:00
Haojun Liao
aa8909b267 other: add some logs. 2023-09-09 15:18:54 +08:00
wangmm0220
f1baed79b4 opti:rm useless code 2023-08-11 20:22:51 +08:00
wangmm0220
7b22d37f12 fix:lost data when consumer db 2023-08-11 17:25:43 +08:00
dapan1121
4dae9f4eff merge main to 3.0 2023-06-26 18:43:00 +08:00
wangmm0220
c5dfe169a6 fix:error 2023-06-16 19:33:03 +08:00
wangmm0220
8a81278024 fix:load pHandle if necessary wher tq restore 2023-06-13 15:14:11 +08:00
Haojun Liao
5301c5756a other: merge main into 3.0. 2023-06-01 09:29:18 +08:00
Haojun Liao
b38e253019 fix(stream): set the skip offset when resume stream task. 2023-05-26 22:51:58 +08:00
wangmm0220
a3e214b9e8 fix:put poll to push manager if wal not exist when offset is latest 2023-05-26 19:17:04 +08:00
Haojun Liao
5d5b2bb16a refactor: do some internal refactor. 2023-05-18 12:59:19 +08:00
Haojun Liao
041ffa91bd other: merge main. 2023-05-17 15:42:15 +08:00
Haojun Liao
366cd1871a
Merge pull request #21314 from taosdata/fix/liaohj_main
fix(stream):extract delete msg from wal.
2023-05-17 14:52:49 +08:00
Haojun Liao
30e7cb5860 fix(tmq): avoid return delete msg for table subscription. 2023-05-16 10:42:02 +08:00
Haojun Liao
d585f34ea9 fix(tmq): avoid seek to previous position. 2023-05-15 18:48:50 +08:00
wangmm0220
36454c73ed fix:add lock to pHandle for safety 2023-05-15 18:38:14 +08:00
Haojun Liao
170c603995 other: merge 3.0 2023-05-13 20:16:55 +08:00
Haojun Liao
7ef16e1922 other: merge main. 2023-05-12 11:49:53 +08:00
yihaoDeng
5708fcb5cb Merge branch 'enh/3.0' into enh/rocksdbSstateMerge 2023-05-10 08:02:16 +00:00
yihaoDeng
8356533eb1 Merge branch 'enh/standVer' into enh/rocksdbSstateMerge 2023-05-09 12:13:16 +00:00
wangmm0220
f9142c0ddb fix:[TD24010] lost data if apply ver is small than commit ver 2023-05-08 16:49:56 +08:00
Haojun Liao
6b305bf745 refactor: do some internal refactor. 2023-05-06 13:35:18 +08:00
Haojun Liao
da6212bab0 other: merge main. 2023-05-06 12:51:38 +08:00
Haojun Liao
09a04052ec refactor: do some internal refactor. 2023-05-04 16:15:14 +08:00
Haojun Liao
ace193ca9b fix(stream): set the seek version if the reader's version is earlier than the first version in wal. 2023-04-29 23:34:49 +08:00
Haojun Liao
c654f1145d enh(stream): add new msg for seek, and do some internal refactor. 2023-04-20 10:59:16 +08:00
Haojun Liao
70b45a4cf0 fix(stream): all data should be extracted from wal. 2023-04-13 23:22:09 +08:00
Haojun Liao
7419c0bfe9 enh(stream): the stream will start after vnode restore being completed. 2023-04-10 16:56:14 +08:00
Haojun Liao
f5b52749c2 refactor: do some internal refactor. 2023-04-09 01:39:09 +08:00
wangmm0220
7616a283e3 fix:error in TD-23218 & remove useless logic 2023-03-18 18:23:38 +08:00
Xiaoyu Wang
d04d610a85
Merge pull request #20061 from taosdata/FIX/TD-22611-main
enh: refactor syncBeginSnapshot and walBeginSnapshot for logRetention
2023-02-24 13:12:06 +08:00