Commit graph

398 commits

Author SHA1 Message Date
Jing Sima
f84f04398d
fix: [TS-4897] Support query stables cross databases and fix potential deadlock. (#30507) 2025-03-28 10:07:08 +08:00
Haojun Liao
af06bea439
fix(stream): add start/stop message. (#30400) 2025-03-24 19:38:00 +08:00
Jinqing Kuang
ac23a4fc55
feat(stream): support stream processing for virtual tables (#30144)
* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
Jing Sima
410324746b
feat:[TS-4897] virtual table (#30098)
* feat: [TS-4897] Support create/drop/alter/show/describe vtable

* feat: [TS-4897] Support vtable's query

* feat: [TS-4897] Support create virtual supertable

* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.

* feat: [TS-4897] Add create test and fix bugs

* feat: [TS-4897] Add alter/drop test and fix bugs

* feat: [TS-4897] Add describe/show test and fix bugs

* feat: [TS-4897] Add auth test and fix bugs

* feat: [TS-4897] Fix meta/catalog/cache bugs

* feat: [TS-4897] Support select tag from virtual child table

* feat: [TS-4897] Add select test and fix plenty of bugs

* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.

* feat: [TS-4897] Fix 'schema is old'

* feat: [TS-4897] Support virtual stable query

* feat: [TS-4897] Add tests and Fix bugs

* feat: [TS-4897] resolve conflict.
2025-03-15 14:10:46 +08:00
WANG MINGMING
3d053e2c9b
feat(stream): optimize client logic of creating stream if with history (#30059)
* feat:[TS-5617]use last_row cache mode if creating stream in fill_history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* fix: heap use after free

* feat: add log

* fix: ci case error

* fix: compile error in windows

* fix: ci case error

* fix: heap user after free

* fix: memory leak

* fix: ci case error

* fix: ci case error

* fix: ci case error

---------

Co-authored-by: yihaoDeng <yhdeng@taosdata.com>
2025-03-14 13:55:52 +08:00
Haojun Liao
d24bc139a5 chore: merge 3.0 2025-03-10 18:31:56 +08:00
Haojun Liao
a71f414d97 refactor(stream): stop all tasks before drop vnodes. 2025-03-02 02:06:14 +08:00
Haojun Liao
6766e6db53 refactor(stream): add checkpoint queue to handle chkpt. 2025-02-20 16:13:33 +08:00
dmchen
b422e2b2f2 feat/TS-5805-force-assign-leader-msg-type 2025-02-12 06:29:24 +00:00
dmchen
9608344833 feat/TS-5805-force-assign-leader-add-sql 2025-01-24 03:47:45 +00:00
dmchen
d9401fd75d feat/TS-5805-add-sql-command 2025-01-23 09:37:07 +00:00
Shengliang Guan
c62e189496 Merge branch 'main' into merge/mainto3.0 2025-01-09 13:38:37 +08:00
xiao-77
3068226c3c Fix tmsg ut failed. 2025-01-08 11:11:21 +08:00
xiao-77
e597baa345 Fix mnode = 3 , trans stuck. 2025-01-07 22:01:33 +08:00
54liuyao
0bf9fb1e31 add reset stream 2024-12-27 15:11:19 +08:00
xiao-77
df2331d5e7 Add config req and config thread. 2024-11-07 15:29:33 +08:00
dongming chen
26485b235d
Merge branch '3.0' into feat/TS-5484-audit-delete 2024-11-01 13:48:38 +08:00
Shengliang Guan
4527e96d2b
Merge pull request #27907 from taosdata/enh/opt-transport
Enh: RPC transport optimization
2024-10-22 09:56:48 +08:00
dmchen
f5c959da6f feat/TS-5484-audit-delete 2024-10-17 03:34:57 +00:00
yihaoDeng
c4a8248f14 Merge remote-tracking branch 'origin/3.0' into enh/opt-transport 2024-10-15 14:24:12 +08:00
Shengliang Guan
3003e7286e feat: data analysis 2024-10-15 10:00:38 +08:00
Shengliang Guan
9f05ca5ee8 revert: data analysis 2024-10-14 23:27:11 +08:00
yihaoDeng
486b559167 Merge remote-tracking branch 'origin/3.0' into enh/opt-transport 2024-10-10 18:13:10 +08:00
Shengliang Guan
4dc0c2f47e feat: data analysis 2024-10-09 17:09:43 +08:00
yihaoDeng
e12741ed18 Merge remote-tracking branch 'origin/3.0' into enh/opt-transport 2024-09-23 17:16:45 +08:00
kailixu
d7fcf21f8b enh: support drop table with uid 2024-09-17 15:41:10 +08:00
yihaoDeng
94891d5bff opt transport 2024-09-07 17:29:35 +08:00
Haojun Liao
91dcf63933 other: merge 3.0 2024-07-31 15:01:13 +08:00
Haojun Liao
52b3e1be6f fix(stream): drop orphan tasks/reset task in trans done by write-queue. 2024-07-31 14:08:22 +08:00
kailixu
ce54abd94d fix: trans conflict during update dnode info 2024-07-30 16:54:39 +08:00
Haojun Liao
647f9f47ef refactor: do some internal refactor. 2024-07-08 10:32:23 +08:00
Haojun Liao
8e6bb176c2 fix(stream): use trans to set the consensus-checkpoint id 2024-07-05 16:17:20 +08:00
Haojun Liao
653f7a1a43 fix(stream): refactor the checkpoint consensus policy. 2024-07-04 13:55:50 +08:00
Haojun Liao
48bf5eb4d8 other: merge 3.0 2024-06-27 19:23:06 +08:00
Haojun Liao
1818edcb2b fix(stream): consensus the start checkpoint id, and extract the streamhb related functions into a new file. 2024-06-26 10:18:32 +08:00
kailixu
431c0cf331 fix: compatibility for msg and error code 2024-06-22 23:04:56 +08:00
kailixu
edabeb3a98 fix: compatibility for msg and error code 2024-06-21 11:22:32 +08:00
kailixu
79f8ab4596 fix: compatibility for msg and error code 2024-06-21 08:07:18 +08:00
Haojun Liao
623a73f57b
Merge pull request #26089 from taosdata/fix/3_liaohj
fix(stream): init the node list and task list when starting mnode.
2024-06-12 11:34:10 +08:00
Haojun Liao
00eb621825 fix(stream): do scan checkpoint-report in write queue. 2024-06-11 14:09:50 +08:00
kailixu
ed33493521 enh: support get msg string and msg number 2024-06-10 22:56:44 +08:00
Haojun Liao
20bade24c3
Merge branch '3.0' into fix/3_liaohj 2024-06-10 01:34:55 +08:00
Haojun Liao
cbf8b363fc fix(stream): update checkpoint info by using trans. 2024-06-10 00:46:01 +08:00
wangjiaming0909
46351a6ba9 fix defining tsma messages messing vnode commit messages 2024-06-06 16:13:02 +08:00
Haojun Liao
be107b204c enh(stream): support retrieve checkpoint data remotely when start stream tasks in vnodes. 2024-05-28 09:54:22 +08:00
Haojun Liao
89fb56e77d other: merge 3.0 2024-05-24 11:41:51 +08:00
Shungang Li
a75a06f12c fix: TDMT_MND_ARB_UPDATE_GROUP placeholder 2024-05-22 15:49:20 +08:00
Shungang Li
307ddc632f enh: batch update arbgroup in trans 2024-05-22 10:44:46 +08:00
Haojun Liao
7f93ec2c53 fix(stream): update the checkpoint info for followers, and some internal refactor. 2024-05-11 09:37:02 +08:00
kailixu
74afb8deae chore: solve conflicts 2024-04-17 10:52:34 +08:00