Commit graph

85 commits

Author SHA1 Message Date
Simon Guan
58bdc656c8 Merge branch '3.3.6' into merge/3.3.6tomain 2025-11-05 14:35:39 +08:00
dongming chen
2e1be8f777
enh(TD-38474): modify sdb data (#33457) 2025-11-04 09:26:24 +08:00
Kaili Xu
13ae5350b7
feat: rollup sma (#33063) 2025-09-26 17:32:32 +08:00
Hongze Cheng
e61fc3c3a6
feat: impl vnode scan (#32942)
* feat: support SQL commands

* feat: impl `scan database` client part code

* feat: impl `scan db.vgroups` client part code

* feat: impl `show scans` client codes

* feat: impl `show scan <id>` client code

* feat: impl `kill scan <id>` client code

* feat: impl mnode part code

* more code

* feat: impl mndScanDb

* feat: Impl all command mnode code

* feat: impl vnode part code

* fix: ci return code check

* fix: return code ci

* doc: change document

* fix: forever loop

* fix: fix some test cases

* add more codes

* fix: compile error

* fix: CI cases

* fix: some scan issues
2025-09-19 13:41:41 +08:00
Pan Wei
bcc25e56bc
feat: new stream (#31678)
* fix: windows compile issue

* test: add vtable cases (#31829)

* fix: windows compile issues

* test:add test cases

* fix: windows compile issue

* case: em-4 stream case submit

* test: stream4_sub1 found bug2

* test: submit test_scene_meters_bug2.py

* add stream parameters example

* feat: [TS-6100] Do not translate const value as column.

* Feat/ts 6100 3.0 zlv (#31747)

* modify asan exampel

* modify asan exampel

* add example

* add example

* modify case example

---------

Co-authored-by: zelv01 <1101510017@qq.com>

* feat(stream): fix memory leak

* modify sliding example

* test: update test case.

* feat(stream): fix conflicts

* fix: add offset case 10a 10s 10m 10h 10d

* feat(stream): fix conflicts

* chore(stream): rename case name #TS-6100

* add case

* modify example

* fix: windows compile issues

* fix: data null check

* feat: [TS-6100] Forbid where when using %%trows (#31827)

* feat: [TS-6100] Forbid where when using %%trows

* test: update cases

* feat: [TS-6100] Fix leaks.

---------

Co-authored-by: Simon Guan <guanshengliang@qq.com>

* test: reproduce bugs

* test: update test case.

* test: update test case.

* feat: [TS-6100] Fix leaks.

* test: add cases

* Feat/ts 6100 3.0.pw10 (#31841)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* test: reproduce bugs

* fix: add sliding interval combine case

* test: add cases

* test: add recalc test.

* test: reproduce bugs

* case : add vt ts is null check

* modify case

* bug: submit test_idmp_meters_bug3.py

* test: add test for recalc.

* test: add cases

* fix: error code check

* test: add cases

* fix(stream): scan wal with schema in that version

* add case

* test: add cases

* test: update test case.

* fix: windows compile issues

* add case

* test: add cases (#31845)

* modify case

* fix: reset interpPrev

* test: add test_idmp_meters bug4 and bug3

* add case

* fix(stream): opti wal interface

* fix: remove test_idmp_meters_bug5.py

* test: add cases

* fix(stream): fix ts data fetch for virtual tables

* cancel asan case

* test: update test case.

* test: update test case.

* add case

* test: add cases

* test: add cases

* test: add case test_idmp_meters_bug5.py

* test: update test case.

* fix(stream): tmq error

* test: add cases

* feat: [TS-6100] Restore deleted code in mndSma.c since they are still in use.

* fix(stream): optimize val scan logic

* test: add test_recalc_expired_time.py  to ci.

* test: update test case.

* test: update test case.

* feat: [TS-6100] Fix fill range check

* fix(stream): optimize val scan logic

* add case

* test: modify for partition by %%1

* test: add fun case stream4_sub7

* fix(stream): optimize val scan logic

* add case

* feat: [TS-6100] Rename OPTIONS to STREAM_OPTIONS.

* test: add test for recalc.

* test: use stream_options.

* fix: some cases error.

* test: remove recalc from ci.

* fix: ci case issues (#31880)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

* fix: external window idx issue

* fix: ci issues

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* fix(stream): fix compilation error

* fix(stream): optimize val scan logic

* test:add test cases

* test: modify case

* fix: external agg error

* test(stream): tobacco scene testing #TD-36514

* test: add stream cases (#31885)

* fix: windows compile issue

* fix: calc timerange

* fix: windows compile issue

* modify case

* fix(stream): compile error

* test: remove one debug test case file

* test: modify

* test: add test cases

* test: reproduce bugs

* test: reproduce bugs

* feat: [TS-6100] Placeholder function should only appera in SELECT and… (#31868)

* feat: [TS-6100] Placeholder function should only appera in SELECT and WHERE and FROM.

* test: update case

---------

Co-authored-by: Simon Guan <guanshengliang@qq.com>

* add example

* add example

* modify case example

* modify case

* test:alter sql

* test: add stream5 case

* fix(stream): get schema error with version

* test: add delete recalc test py.

* test: remove bug cases

* test: stream5 case test passed

* test: add state cases (#31893)

* fix(stream): compile error

* test: modify case

* test: add cases

* test: add test.

* test: update test case.

* chore(test): fix case err

* test: update test case.

* fix: align data get

* fix(stream): fix row index of datablock written into data cache

* fix: put align data

* test: update test case.

* test: add test cases for virtual table

* chore(test): fix case err #TD-36514

* add case

* test: add test for water mark.

* test: add meters bug6 for stream5

* test: add cases (#31903)

* test: add test for recalc.

* feat: [TS-6100] %%trows can only be used when event type is window close.

* test: add precision of database for ms/us/ns

* modify case

* add case

* add case

* test: add test to ci.

* modify case

* fix: ci case issues (#31904)

* enh: add operator reset func

* fix: merge join reset issue

* fix: memory issues

* fix: add debug assert

* fix: memory issues

* fix: memory leak

* fix: memory issues

* fix taos log miss

* fix: case issue

* fix: case issue

* fix: case issues

* fix: drop dnode issue

* fix: memory issues

* fix: memory issues

* fix: memory leak issues

* fix: recalculate time range issue

* fix: add debug log

* fix: memory issues

* fix: enable case asan

* Update streamlist_for_ci.task

* fix: case asan issue

* fix: stream name issue

* fix: external window compile issues

* fix: deploy memory issue

* fix: ahandle issue

* fix: ahandle issue

* fix: ahandle issue

* fix: virtual table reader list issue

* fix: log info

* fix: msg error

* fix: virtual table addr list issue

* fix: memory issues

* fix: memory leak issue

* fix: memory issues

* fix: memory free issues

* fix: memory issues

* fix: snode deploy issue

* fix: mnode reader issue

* fix: memory issues

* fix: add debug test

* enh: add ignore nodata trigger

* fix: memory leaks

* fix: configuration issue

* fix: memory issue

* fix: external window issue

* fix: external window issues

* fix: external window placeholder issue

* fix: placeholder function init issues

* fix: memory leak issue

* fix: add debug log

* fix: compile issues

* fix: double free issue

* fix: runner addr update issue

* fix: msg rsp issue

* fix: external window reset issue

* fix: configuration issue

* fix: deploy msg issue

* fix: compile issue

* fix: external window idx issue

* fix: ci issues

* fix: ci case issues

* fix: drop dnode issue

---------

Co-authored-by: huohong <sallyhuo@taosdata.com>

* fix(stream): ci error

* test: update test case.

* feat: [TS-6100] Disable some failed UT.

* feat: [TS-6100] Fix virtual table

* test: add bug 5.

* test: add test delete recalc to ci.

* test: add bug 6.

* test(stream): tobacco scene #TD-36514

* fix: reqCids,reqCols memory leak in SSTriggerRealtimeContext

Co-authored-by: Tony Zhang <tonyzhang@taosdata.com>

* test: add case stream6

* fix(stream): implement some pending features in trigger task

* modify case

* modify case

* fix: case issues

* modify case

* test: add recalc for warter mark.

* fix(stream): fix count window trigger of virtual tables

* fix(stream): memory leak

* test: fix run err.

* test: add stream6 bug7

* fix: adjust format

* test(stream): tobacco scene testing #TD-36514

* test: change bug7 with update window1 and 2

* test: add test bug 7.

* case: restore write 3 window

* fix: windows compile issue

* fix: notify

* test: add cases

* modify case

* test: update test case.

* test(stream): toobacco scene testing #TD-36514

---------

Co-authored-by: Simon Guan <slguan@taosdata.com>
Co-authored-by: plum-lihui <huili@taosdata.com>
Co-authored-by: Alex Duan <417921451@qq.com>
Co-authored-by: zelv01 <1101510017@qq.com>
Co-authored-by: Jing Sima <simondominic9997@outlook.com>
Co-authored-by: xiangyang guo <66111494+happyguoxy@users.noreply.github.com>
Co-authored-by: wangmm0220 <wangmm0220@gmail.com>
Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: zyyang90 <zyyang@taosdata.com>
Co-authored-by: Alex Duan <51781608+DuanKuanJun@users.noreply.github.com>
Co-authored-by: facetosea <285808407@qq.com>
Co-authored-by: Simon Guan <guanshengliang@qq.com>
Co-authored-by: huohong <sallyhuo@taosdata.com>
Co-authored-by: Li Hui <52318143+plum-lihui@users.noreply.github.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: xiao-77 <berylbao@taosdata.com>
Co-authored-by: Zhixiao Bao <62235797+xiao-77@users.noreply.github.com>
Co-authored-by: happyguoxy <happy_guoxy@163.com>
Co-authored-by: Tony Zhang <34825804+Tony2h@users.noreply.github.com>
Co-authored-by: Tony Zhang <tonyzhang@taosdata.com>
2025-07-16 14:42:16 +08:00
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage (#31552)
* add API to use s3 as shared storage

* support using local file system as shared storage

* upload file to shared storage

* support read, compact and drop

* finish basic mnode & vnode msg processing

* follower sync migration state

* implement mnode transaction, and improve log

* send migration progress msg to dnode to avoid deadlock

* implement following migration

* remove mcount

* avoid redo migration on startup

* avoid follower deadlock when leader is down

* trigger migrate by timer,  avoid compact after migration

* comment out the usage of 'tcs' functions in stream

* change config item prefix from s3 to ss

* change db option prefix from s3 to ss

* rename s3 data struct, function, file to ss

* rename s3 macro to ss

* update s3 sql to ss

* rename remaining s3 items to ss

* check ss configruation, improve s3 retry

* grant object storage -> shared storage,  check ssEnabled

* fix memory leaks

* update build options

* omit sensitive information when dump config

* fix backward compatibility issue

* fix issues found in ci-checks

* fix some failed test cases

* avoid follower timeout and improve log

* fix: follower timeout because migration status not updated

* refuse migration if there's an in progress one

* fix ss test case

* remove garbage files and other minor improvement

* fix failed test cases

* update unit test

* fix failed test case

* fix failed test case

* update document

* update document and fix failed test cases

* fix minor issues in code, test and document

* check new commit after migration task is scheduled

* fix several issus

1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case

* add shared storage upgrade tool

* fix compile error
2025-07-14 16:33:53 +08:00
Kaili Xu
dc2b021778
feat: support mount data path (#31677) 2025-07-12 15:39:29 +08:00
Minglei Jin
66c4ef3e90
feat(mqtt): mqtt subscription (#30127)
* feat(mqtt): Initial commit for mqtt

* chore(xnode/mnd): xnode message handlers for mnode

* chore(mnd/xnode): mnode part for xnode

* chore(xnode/translater): fix show commands

* fix(ast/creater): fix xnode create option

* fix(xnode/ci): fix ci & doc's error codes

* chore(xnode/sql): make create/drop/show work properly

* fix(xnode/sql): commit new files

* fix(xnode/sql): commit cmake files

* fix: fix testing cases

* fix(xnode/tsc): fix tokens

* fix(ast/anode): fix anode update decl.

* fix(xnode/error): fix xnode error codes

* fix: xnode make/destroy

* chore: xnode with option & dnode id

* chore: use taosmqtt for xnode

* chore: new error code for xnode launching

* chore(xnode): new error code

* chore: header for _xnode_mgmt_mqtt

* chore: source for _xnode_mgmt_mqtt

* chore: remove test directory from cmake

* chore: remove taosmqtt for ci to compile

* chore: remove taosudf header from xnode

* chore: new window macro

* chore: remove xnode mgmt mqtt for windows compilation

* Revert "chore: remove xnode mgmt mqtt for windows compilation"

This reverts commit 197e1640c7.

* chore: cleanup code

* chore: xnode mgmt comment windows part out

* chore: mgmt/mqtt, move uv head toppest

* xnode/mnode: create xnode once per dnode

* fix(xnode/systable/test): fix column count

* xnode/sdb: renumber sdb type for xnode to make start/stop order correct

* xnode/mqtt: new param mqttPort

* fix SXnode's struct type

* transfer dnode id to mqtt subscription

* tmqtt: remove uv_a linking

* tmqtt/tools: sources for tools

* tools: fix windows compilation

* tools/producer: fix windows sleep param

* tools/producer: fix uninited var rc

* make tools only for linux

* test/mnodes: wail 1 or 2 seconds for offline to be leader

* update topic producer tool for geometry data type testing

* format tool sql statements

* show xnodes' ep

* make shell auto complete xnodes

* use usleep instead of sleep

* mqtt/proto: first version mqtt protocol

* remove assert styles

* build with linux only

* fix libuv for taosmqtt building

* fix log printing

* mem: use ttq_ prefix instead of tmqtt

* xnode/parser/proto: protocol option for xnode

* xnode/translater/option: translate xnode option proto

* xnode/translator: translate proto param

* xnode/tmsg: encode/decode proto param

* xnode/mnode: proto parma for mndXnode

* xnode/proto: protocol param for xnode

* xnode/mqtt: save/load proto from xnode json

* rename tmqtt proto header

* rename head directories

* rename header name styles

* restyle func names

* update

* update

* use camel db

* use camel for cxt

* update count of information_schema

* fix exceptional cases

* fix w.r.t reviews

* fixes w.r.t reviews

* use ttq_free

* append xnode msgs

* update information schema count

* support tmq meta data sub

* success typo fix

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix error line var w.r.t. suggestions from copilot

* denote mqtt node with bnode instead of xnode

* smoke testing for mqtt

* soak testing

* new package for test framework: paho 2.1.0

* import mqtt into util

* fix soak testing

* test/soak: user 5 topics per client

* test/soak: cover qos

* update docker image references to tdengine-ci:0.1 in CI scripts and common.py

* refactor: rename bnode msg to backup node

* refactor: rename xnode to bnode 1

* refactor: rename xnode to bnode 2

* refactor: rename xnode to bnode 3

* refactor: rename xnode to bnode 4

* refactor: rename xnode to bnode 4

* refactor: rename xnode to bnode 5

* refactor: rename xnode to bnode 6

* refactor: rename some files 1

* refactor: rename some files 2

* refactor: rename some files 3

* refactor: rename some files 4

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: chenhaoran <haoran920c@163.com>
Co-authored-by: Simon Guan <guanshengliang@qq.com>
2025-06-25 14:58:51 +08:00
Simon Guan
b91dd10846 enh: add log level to each log entry 2025-02-27 09:55:11 +08:00
xiao-77
474113da4e Move rebuild config sdb to after restored. 2024-12-17 11:22:38 +08:00
xiao-77
90af4457ef Add func sdb set&get float. 2024-11-27 10:50:04 +08:00
xiao-77
4a03483683 add prepare func. 2024-11-15 14:39:10 +08:00
xiao-77
cc3f4504bf Add mnd init cfg and mnd cfg trans. 2024-11-14 17:20:50 +08:00
Shengliang Guan
3003e7286e feat: data analysis 2024-10-15 10:00:38 +08:00
Shengliang Guan
9f05ca5ee8 revert: data analysis 2024-10-14 23:27:11 +08:00
Shengliang Guan
a5b5bd61f6 feat: anode 2024-10-09 17:20:13 +08:00
dmchen
0da625bf55 dmchen/delete-trans-tool 2024-09-03 02:21:31 +00:00
dmchen
fee9a14b69 fix/TD-31622-avoid-refrence-prow 2024-08-29 03:39:10 +00:00
kailixu
280c5d4b3d enh: return error code 2024-07-22 16:33:42 +08:00
kailixu
0ef8d3510a fix: use unsigned type for union bit operation 2024-06-14 15:46:11 +08:00
wangjiaming0909
3566cac206 add max tsma num opt 2024-04-16 14:23:26 +08:00
Shungang Li
3a656c3595 feat: support 2 replica with arbitrator 2024-03-19 09:30:35 +08:00
Benguang Zhao
dfef26866a enh: decode sdb raw object in SdbValidateFp instead 2024-02-23 10:39:50 +08:00
kailixu
76558da849 feat: support uniq grant 2024-01-18 15:23:38 +08:00
Hongze Cheng
04e84ba152 Merge branch '3.0' of https://github.com/taosdata/TDengine into feat/TS-4229 2023-12-06 16:43:00 +08:00
dmchen
4e1a355052 mnode for compact 2023-11-23 04:43:28 +00:00
yihaoDeng
08e3448498 add seq 2023-11-08 19:54:57 +08:00
dapan1121
14e8db289d feat: support view 2023-09-19 14:19:54 +08:00
Benguang Zhao
9f1c8cc7b8 enh: unify validation of prepare actions with cb validateFp 2023-08-18 18:21:30 +08:00
Benguang Zhao
3c698b2461 enh: add mndAddPrepareNewVgAction 2023-05-29 14:13:11 +08:00
Benguang Zhao
d27f392beb enh: control frequency of mnode sdb flushing with mndSdbWriteDelta 2023-03-03 17:30:23 +08:00
Benguang Zhao
caaff5c546 enh: rotate wal of mnode properly 2023-03-01 10:02:33 +08:00
yihaoDeng
a05eb4b110 refactor tag index 2023-02-03 22:19:01 +08:00
Shengliang Guan
f02f3e7a07 fix: allow mnode start even dnode in dropping state 2023-01-11 10:12:54 +08:00
Shengliang Guan
9c8b194c34 refact: set FpSnapshotStopRead return type to void 2022-12-20 15:07:27 +08:00
Minghao Li
a35ad361d0 refacotr(sync): call syncBeginSnapshot instead of walBeginSnapshot 2022-10-20 18:01:43 +08:00
Shengliang Guan
3a7d89a6bb refact: remove bnode codes 2022-10-18 17:34:58 +08:00
Shengliang Guan
00080750fa fix: mnode may deadlock while drop another mnode if it crashed 2022-10-17 20:40:11 +08:00
Shengliang Guan
78fb1911d4 enh: refactor the code to create and delete mnodes 2022-10-14 17:59:09 +08:00
Shengliang Guan
390264db3f enh: assert the term of the snapshot as same as the memory 2022-07-19 17:11:15 +08:00
Liu Jicong
af0f398121
Merge pull request #14960 from taosdata/feature/stream
feat(stream): recover from failure
2022-07-15 18:32:48 +08:00
Liu Jicong
9394e3380f feat(stream): recover from failure 2022-07-15 17:58:53 +08:00
wangmm0220
e711f56027 fix:remove code for resolve the core of amotic_load_n 2022-07-14 16:33:11 +08:00
wangmm0220
2b442621e8 feat:get child table name in tmq meta 2022-07-12 18:58:33 +08:00
Shengliang Guan
ab0d7d31bc refactor: get sdb snapshot in atomic way 2022-06-20 18:37:46 +08:00
Shengliang Guan
96434756de refactor: save sdb file on needed 2022-06-20 16:23:04 +08:00
Shengliang Guan
db4e02c7c7 fix: add log to debug mnode sync 2022-06-17 15:23:17 +08:00
Shengliang Guan
ad5f9555fb refactor: sdb commit index 2022-06-17 14:48:30 +08:00
Shengliang Guan
4da408afe6 enh: call walEndSnapshot while sdb writefile 2022-06-09 17:44:39 +08:00
Shengliang Guan
17f8ccc901 enh: record config index 2022-06-09 14:12:52 +08:00