WANG MINGMING
585324ceec
Merge pull request #31806 from taosdata/feat/TS-6100-3.0-mm
...
Feat/ts 6100 3.0 mm
2025-07-11 17:33:25 +08:00
wangmm0220
f85fc43c8f
feat(stream): ts condition error in vtable scan & remove code of old stream
2025-07-11 16:50:27 +08:00
Pan Wei
2e3cbf4aa9
fix: placeholder function issues ( #31789 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
* fix: memory issues
* fix: memory leak issues
* fix: recalculate time range issue
* fix: add debug log
* fix: memory issues
* fix: enable case asan
* Update streamlist_for_ci.task
* fix: case asan issue
* fix: stream name issue
* fix: external window compile issues
* fix: deploy memory issue
* fix: ahandle issue
* fix: ahandle issue
* fix: ahandle issue
* fix: virtual table reader list issue
* fix: log info
* fix: msg error
* fix: virtual table addr list issue
* fix: memory issues
* fix: memory leak issue
* fix: memory issues
* fix: memory free issues
* fix: memory issues
* fix: snode deploy issue
* fix: mnode reader issue
* fix: memory issues
* fix: add debug test
* enh: add ignore nodata trigger
* fix: memory leaks
* fix: configuration issue
* fix: memory issue
* fix: external window issue
* fix: external window issues
* fix: external window placeholder issue
* fix: placeholder function init issues
* fix: memory leak issue
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-11 13:55:15 +08:00
wangmm0220
25dd06bc50
feat(stream): fix tmq error
2025-07-10 14:20:35 +08:00
Pan Wei
6cea090c86
fix: memory issues and case issues ( #31743 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
* fix: memory issues
* fix: memory leak issues
* fix: recalculate time range issue
* fix: add debug log
* fix: memory issues
* fix: enable case asan
* Update streamlist_for_ci.task
* fix: case asan issue
* fix: stream name issue
* fix: external window compile issues
* fix: deploy memory issue
* fix: ahandle issue
* fix: ahandle issue
* fix: ahandle issue
* fix: virtual table reader list issue
* fix: log info
* fix: msg error
* fix: virtual table addr list issue
* fix: memory issues
* fix: memory leak issue
* fix: memory issues
* fix: memory free issues
* fix: memory issues
* fix: snode deploy issue
* fix: mnode reader issue
* fix: memory issues
* fix: add debug test
* enh: add ignore nodata trigger
* fix: memory leaks
* fix: configuration issue
* fix: memory issue
* fix: external window issue
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-09 18:57:01 +08:00
facetosea
2059de7d7a
fix: _rowts in external window
2025-07-08 16:32:49 +08:00
Pan Wei
f6ab877e05
fix: case memory issues ( #31621 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
* fix: memory issues
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-03 13:26:22 +08:00
Pan Wei
4270f35fb9
fix: crash and memory issues ( #31588 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-02 10:14:55 +08:00
wangmm0220
564d585619
feat(stream): fix err that no rtInfo in stream fetch
2025-06-28 11:47:31 +08:00
dapan1121
feabdc7be9
fix: crash issues
2025-06-27 18:42:21 +08:00
facetosea
8462a1055f
feat: get vginfo for exec
2025-06-26 18:34:18 +08:00
Jing Sima
a177e75195
feat: [TS-6100] Improve external window condition.
2025-06-25 16:33:02 +08:00
Jing Sima
6696b73042
feat: [TS-6100] Parse create stream sql: remove table list info in virtual scan operator.
2025-06-23 16:42:02 +08:00
Jing Sima
40a714a020
feat: [TS-6100] Parse create stream sql: change virtual table plan
2025-06-21 17:29:36 +08:00
wangmm0220
395ee23fe8
fix(stream): add logic of resetting operator
2025-06-12 16:56:21 +08:00
wangmm0220
4572c0243d
fix(stream): add logic for %%tbname
2025-06-05 17:26:09 +08:00
wangmm0220
3aa8c175f5
feat(stream): reset agg operator & reset tsdb reader in fetch request
2025-05-26 14:52:02 +08:00
wangmm0220
a716fabbbc
feat(stream): set tag value in trigger cols
2025-05-22 15:45:57 +08:00
wangmm0220
2d717c3a4f
feat(stream): set tag value in trigger cols
2025-05-22 15:38:32 +08:00
wangjiaming0909
0c402fbb38
new-stream: runner impl external window and force output
2025-05-16 08:29:34 +00:00
wangmm0220
9ffe95755b
feat(stream): add interface of getting cols value of groupid
2025-05-15 10:03:57 +08:00
wangjiaming0909
3db03588bd
impl runner external window operator
2025-05-14 09:55:32 +00:00
wangjiaming0909
bfdad60c2e
new-stream: external window operator impl
2025-05-12 17:36:10 +08:00
dapan1121
a4869f5263
fix: compile issues
2025-05-12 17:14:35 +08:00
dapan1121
009e8aa41e
Merge remote-tracking branch 'origin/3.0' into feat/TS-6100-3.0
2025-05-12 16:24:22 +08:00
wangmm0220
df42d12b06
feat(stream): return calc cols and trigger cols from stream reader
2025-05-10 21:22:11 +08:00
wangmm0220
f1155c54ad
feat(stream): return calc cols and trigger cols from stream reader
2025-05-10 18:34:01 +08:00
Jing Sima
84016904ff
fix: [TS-6445] Fix virtual tables not detecting schema changes. ( #30999 )
...
Ensure virtual tables invalidate cached schema metadata and reload it when a schema change (e.g., ALTER VTABLE) is detected. This prevents stale schema references and query errors after structural updates.
2025-05-10 10:34:54 +08:00
factosea
f3027e938e
feat: stream data insert
2025-05-08 09:48:53 +08:00
wangjiaming0909
e95c5a9d85
new-stream: runner impl
2025-05-07 18:14:17 +08:00
wangjiaming0909
1ccc6c69a8
new-stream: runner impl
2025-04-29 15:31:17 +08:00
Jing Sima
9d3ded13d0
fix: [TS-6100] Remove useless code and fix bugs
2025-04-24 18:48:28 +08:00
Jing Sima
26d4c933a2
fix: [TS-6100] Fix compile on mac and add toJson/jsonTo func.
2025-04-24 10:16:29 +08:00
dapan1121
330cf9982d
fix: compile issues
2025-04-24 09:11:54 +08:00
WANG MINGMING
ca724865b9
fix(stream): parName is null ( #30592 )
2025-04-19 10:26:32 +08:00
wangjiaming0909
163dd33a0a
new-stream: impl stream runner
2025-04-16 18:16:55 +08:00
wangjiaming
b3b6c3d3ff
fix(fill): fix fill next/prev got different result with asc/desc order
2025-04-03 21:24:39 +08:00
Jing Sima
7cffc7aff2
fix: [TD-34074] Forbid virtual table in tq and stmt.
2025-03-20 22:42:27 +08:00
Jing Sima
818bea5568
fix: [TD-34074] Forbid virtual table in tq and stmt and forbid supertable query's origin table from different databases.
...
fix: [TD-34074] Forbid virtual table in tq and stmt and forbid supertable query's origin table from different databases.
2025-03-20 22:42:10 +08:00
Simon Guan
eff763dc91
fix(stream): fix stream processing for virtual tables
2025-03-20 22:31:29 +08:00
Simon Guan
e5a1539c6d
Merge pull request #30307 from taosdata/fix/TS-5467-3.0
...
fix(stream): fix stream processing for virtual tables
2025-03-20 22:24:45 +08:00
Jinqing Kuang
4d97d691cf
fix(stream): add test cases for vtable stream processing
2025-03-20 18:57:58 +08:00
liuyao
fac36002c1
ci(stream): add ci and fix issue ( #30270 )
...
* ci(stream): add ci and fix issue
* ci: adj test
* ci: add test and fix issue
* ci: adj test
* ci(stream): add test for fill operator and fix issue
* fix: fix issue
* fix: fix issue
* fix: adjust defalut rec value
2025-03-20 10:34:31 +08:00
Jinqing Kuang
c82398e175
fix(stream): fix stream processing for virtual tables
...
- Fixed status check in virtual table stream processing
- Fixed multi-way merge logic for super virtual tables
- Fixed generation of target subtable names during merge
2025-03-20 01:15:54 +08:00
Jing Sima
ccaa0b91ae
feat: [TS-4897] Add more test and fix bugs. ( #30208 )
...
* feat: [TS-4897] Fix state window wrong res when all state col is NULL.
* fix: [TD-34074] Forbid decimal type in virtual table.
2025-03-19 17:15:17 +08:00
Haojun Liao
dfe32225de
chore: merge main.
2025-03-17 14:51:11 +08:00
Haojun Liao
ab92886820
fix(stream): reduce the consensus checkpoint id trans. ( #30105 )
...
* fix(stream): reduce the consensus checkpoint id trans.
* refactor(stream): add some logs.
* refactor(stream): set the max checkpoint exec time 30min.
* refactor(stream): add checkpoint-consensus trans conflict check.
* refactor(stream): remove unused local variables.
* fix(stream): fix syntax error.
* fix(stream): 1. fix free memory error 2. continue if put result into dst hashmap failed.
* fix issue
* fix issue
* fix(mnd): follower mnode not processes the timer event.
* fix(stream): print correct error msg.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): truncate long subtable name
* fix(stream): add buffer len.
* refactor(stream): update some logs.
* fix issue
* refactor(stream): update some logs.
* refactor(stream): update some logs.
* fix(stream): check return value.
* fix(stream): fix syntax error.
* fix(stream): check return value.
* fix(stream): update the timer check in mnode.
* fix(stream): add restart stage tracking.
* fix(stream): track the start task stage for meta.
* fix(stream): fix error in log.
* refactor(stream): adjust log info.
* fix mem issue
* fix(stream): check the number of required tasks for consensus checkpointId.
* fix(stream): lock the whole start procedure.
* fix(stream): add lock during start all tasks.
* fix(stream): update logs.
* fix(stream): update logs.
* fix(stream): update logs.
* fix(stream): fix dead-lock.
* fix(stream): fix syntax error.
* fix(stream): not drop the scan-history task.
* fix(stream): fix syntax error.
* fix(stream): wait for executor stop before restarting.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): add some logs.
* fix(stream): disable some logs.
* fix(stream): reset the start info if no task left.
---------
Co-authored-by: 54liuyao <54liuyao@163.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
2025-03-17 10:20:17 +08:00
Jinqing Kuang
ac23a4fc55
feat(stream): support stream processing for virtual tables ( #30144 )
...
* enh: add client processing
* enh: add mnode vtables processing
* enh: add mnode vtable processing
* enh: add normal child vtable support
* fix: compile issues
* fix: compile issues
* fix: create stream issues
* fix: multi stream scan issue
* fix: remove debug info
* fix: agg task and task level issues
* fix: correct task output type
* fix: split vtablescan from agg
* fix: memory leak issues
* fix: add limitations
* Update 09-error-code.md
* Update 09-error-code.md
* fix: remove usless case
* feat(stream): extract original table data in source scan task
Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.
* feat(stream): multi-way merge using loser tree in virtual merge task
Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing. Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.
* fix(test): remove useless cases
---------
Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
Jing Sima
410324746b
feat:[TS-4897] virtual table ( #30098 )
...
* feat: [TS-4897] Support create/drop/alter/show/describe vtable
* feat: [TS-4897] Support vtable's query
* feat: [TS-4897] Support create virtual supertable
* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.
* feat: [TS-4897] Add create test and fix bugs
* feat: [TS-4897] Add alter/drop test and fix bugs
* feat: [TS-4897] Add describe/show test and fix bugs
* feat: [TS-4897] Add auth test and fix bugs
* feat: [TS-4897] Fix meta/catalog/cache bugs
* feat: [TS-4897] Support select tag from virtual child table
* feat: [TS-4897] Add select test and fix plenty of bugs
* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.
* feat: [TS-4897] Fix 'schema is old'
* feat: [TS-4897] Support virtual stable query
* feat: [TS-4897] Add tests and Fix bugs
* feat: [TS-4897] resolve conflict.
2025-03-15 14:10:46 +08:00
liuyao
2de6736160
feat(stream): add new trigger continuous_window_close ( #30125 )
...
* opt stream build twa result
* opt force window close memory
* feat(stream):optimize new interval and scan operator
* adj log
* opt code
* opt code
* fill history
* fix issue for fill history
* add ci
* feat(stream): add new stream nonblock interval operator
* adjust code
* use new scan operator
* use new scan operator
* add log
* fix issue
* recover stream scan next function
* fix issue
* fix issue
* fix issue
* ignore disorder data
* fix issue
* fix issue for interval sliding
* fix issue
* fix ci issue
* fix ci issue
* fix ci issue
* add semi && final nonblock interval operator
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* refactor(stream): track the msgId for each upstream tasks.
* fix(stream): fix race condition.
* fix(stream): update the task last msgId when putting into input queue succ.
* fix issue
* fix issue
* put recalculate data to rocksdb
* fix issue
* fix issue
* enh(query)[TD-33071]: add support for saving and restoring tsdbReader scan progress
- Implement functionality to save scan progress during tsdbReader operations
- Enable resuming scans from the last saved position
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* add rec interval check
* enh(stream):add recalculate tasks.
* enh(stream): support the re-calculate the tasks.
* fix issue && do refactor
* do refactor
* fix issue
* fix issue
* update backend opt
* add new interface
* add new session operator
* support blob
* add new session operator
* fix issue
* add rec state for rec task
* fix invalid read
* add new session window
* enh(stream): update the stream tasks backend.
* new session operator
* add pull data
* fix(stream): fix error in expand stream backend.
* fix issue
* fix issue
* fix issue
* merge code
* fix issue
* fix(stream): check for null ptr.
* fix(stream): add more check.
* fix issue
* fix issue
* fix issue
* add debug code
* fix issue
* fix issue
* fix issue
* set rec end flag
* fix(stream): opt re-calculate stream tasks.
* fix issue
* fix issue
* add new operator
* enh(stream): dispatch recalculate block to agg tasks.
* fix issue
* fix issue
* fix(stream): adjust filter.
* fix issue
* refactor
* refactor(stream): adjust the recalculate end block.
* fix issue
* fix(stream): set correct create trigger block.
* fix issue
* fix(stream): fix error in build recalculate end block.
* fix(stream): check null ptr.
* add stream client && fix issue
* fix mem leak
* fix(stream): free msg.
* add stream client
* fix(stream): fix error.
* add stream client && fix issue
* add stream client
* refactor(stream): set the recalculate task complete.
* add wend and group_id for session window dest table
* feat(stream): refactor and set the recalcul agg task complete.
* add cfg for adapter
* fix issue
* add state && event operator
* feat(stream): support fill-history task.
* add new fill operator
* fix(stream): set correct backend when updating fill-history task to recalculate task.
* add new fill operator
* fix(stream): s2025-03-06 11:10:31.272
et ops always open in scan tsdb
* fix(stream):set the correct taskType for sink task.
* new fill operator
* adj stream fill operator
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* fix issue
* adj test
* fix issue
* fix(stream): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(steam): fix issue
* fix(stream): fix issue
* fix(stream): fix issue
* fix(stream): fix issue
* fix: ut com error
* fix(stream): fix mem leak and adjust operator type check rule
* fix(stream): fix mem leak and adjust test case
* refactor code
* fix(stream): free items.
* fix(stream): free fix memory leak.
* fix(stream): fix syntax error.
* fix: ignore unexpect block
* fix: adjust op type
---------
Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 20:14:01 +08:00
