Commit graph

349 commits

Author SHA1 Message Date
WANG MINGMING
3e0e339985
feat(stream): fix thread run out for runner (#31559) 2025-07-01 08:43:11 +08:00
dapan1121
0e69cb3886 Merge remote-tracking branch 'origin/3.0' into feat/TS-6100-3.0 2025-06-27 14:53:03 +08:00
Minglei Jin
66c4ef3e90
feat(mqtt): mqtt subscription (#30127)
* feat(mqtt): Initial commit for mqtt

* chore(xnode/mnd): xnode message handlers for mnode

* chore(mnd/xnode): mnode part for xnode

* chore(xnode/translater): fix show commands

* fix(ast/creater): fix xnode create option

* fix(xnode/ci): fix ci & doc's error codes

* chore(xnode/sql): make create/drop/show work properly

* fix(xnode/sql): commit new files

* fix(xnode/sql): commit cmake files

* fix: fix testing cases

* fix(xnode/tsc): fix tokens

* fix(ast/anode): fix anode update decl.

* fix(xnode/error): fix xnode error codes

* fix: xnode make/destroy

* chore: xnode with option & dnode id

* chore: use taosmqtt for xnode

* chore: new error code for xnode launching

* chore(xnode): new error code

* chore: header for _xnode_mgmt_mqtt

* chore: source for _xnode_mgmt_mqtt

* chore: remove test directory from cmake

* chore: remove taosmqtt for ci to compile

* chore: remove taosudf header from xnode

* chore: new window macro

* chore: remove xnode mgmt mqtt for windows compilation

* Revert "chore: remove xnode mgmt mqtt for windows compilation"

This reverts commit 197e1640c7.

* chore: cleanup code

* chore: xnode mgmt comment windows part out

* chore: mgmt/mqtt, move uv head toppest

* xnode/mnode: create xnode once per dnode

* fix(xnode/systable/test): fix column count

* xnode/sdb: renumber sdb type for xnode to make start/stop order correct

* xnode/mqtt: new param mqttPort

* fix SXnode's struct type

* transfer dnode id to mqtt subscription

* tmqtt: remove uv_a linking

* tmqtt/tools: sources for tools

* tools: fix windows compilation

* tools/producer: fix windows sleep param

* tools/producer: fix uninited var rc

* make tools only for linux

* test/mnodes: wail 1 or 2 seconds for offline to be leader

* update topic producer tool for geometry data type testing

* format tool sql statements

* show xnodes' ep

* make shell auto complete xnodes

* use usleep instead of sleep

* mqtt/proto: first version mqtt protocol

* remove assert styles

* build with linux only

* fix libuv for taosmqtt building

* fix log printing

* mem: use ttq_ prefix instead of tmqtt

* xnode/parser/proto: protocol option for xnode

* xnode/translater/option: translate xnode option proto

* xnode/translator: translate proto param

* xnode/tmsg: encode/decode proto param

* xnode/mnode: proto parma for mndXnode

* xnode/proto: protocol param for xnode

* xnode/mqtt: save/load proto from xnode json

* rename tmqtt proto header

* rename head directories

* rename header name styles

* restyle func names

* update

* update

* use camel db

* use camel for cxt

* update count of information_schema

* fix exceptional cases

* fix w.r.t reviews

* fixes w.r.t reviews

* use ttq_free

* append xnode msgs

* update information schema count

* support tmq meta data sub

* success typo fix

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>

* fix error line var w.r.t. suggestions from copilot

* denote mqtt node with bnode instead of xnode

* smoke testing for mqtt

* soak testing

* new package for test framework: paho 2.1.0

* import mqtt into util

* fix soak testing

* test/soak: user 5 topics per client

* test/soak: cover qos

* update docker image references to tdengine-ci:0.1 in CI scripts and common.py

* refactor: rename bnode msg to backup node

* refactor: rename xnode to bnode 1

* refactor: rename xnode to bnode 2

* refactor: rename xnode to bnode 3

* refactor: rename xnode to bnode 4

* refactor: rename xnode to bnode 4

* refactor: rename xnode to bnode 5

* refactor: rename xnode to bnode 6

* refactor: rename some files 1

* refactor: rename some files 2

* refactor: rename some files 3

* refactor: rename some files 4

---------

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: chenhaoran <haoran920c@163.com>
Co-authored-by: Simon Guan <guanshengliang@qq.com>
2025-06-25 14:58:51 +08:00
Haojun Liao
4743cc3683 refactor(stream): check for write queue size for sink tasks. 2025-05-22 17:55:47 +08:00
dapan1121
330cf9982d fix: compile issues 2025-04-24 09:11:54 +08:00
dapan1121
ae0e1ded7d enh: add hb processing 2025-04-22 17:46:17 +08:00
Simon Guan
b74b038f4f Merge branch '3.0' into merge/mainto3.0 2025-03-04 11:32:29 +08:00
Haojun Liao
a71f414d97 refactor(stream): stop all tasks before drop vnodes. 2025-03-02 02:06:14 +08:00
Haojun Liao
3e55f8edfe fix(stream): follower nodes not restart tasks. 2025-02-28 13:54:26 +08:00
xiao-77
5d2bc35569 Add more log info. 2024-11-25 16:23:34 +08:00
xiao-77
07de144000 serialize and persist local config 2024-11-04 16:16:24 +08:00
dmchen
5934fe8857 fix/TD-31891-remove-void-mnode4 2024-09-20 06:17:33 +00:00
dmchen
0da625bf55 dmchen/delete-trans-tool 2024-09-03 02:21:31 +00:00
sima
32faa6fccc enh:[TD-31063] Handling return value in qnode.c 2024-07-23 15:02:54 +08:00
Haojun Liao
acde36a25b Merge branch '3.0' into fix/3_liaohj 2024-07-09 14:43:35 +08:00
Haojun Liao
8e6bb176c2 fix(stream): use trans to set the consensus-checkpoint id 2024-07-05 16:17:20 +08:00
Haojun Liao
420b59fea5
Merge pull request #26344 from taosdata/fix/3_liaohj
fix(stream): validate the stream hb msg, and discard the invalid hb msg.
2024-07-02 09:07:31 +08:00
wangjiaming0909
5040ff5bd9 mnode/qnode support QueryAutoQWorkerPool 2024-07-01 11:08:40 +08:00
Haojun Liao
1818edcb2b fix(stream): consensus the start checkpoint id, and extract the streamhb related functions into a new file. 2024-06-26 10:18:32 +08:00
Haojun Liao
2cb9644471 fix(stream): update the checkpoint info only when the status is normal. 2024-06-21 09:23:29 +08:00
Haojun Liao
42e0dff589 fix(stream): move the expansion of the stream task related fill-history task to the place where the fill-history task must have been deoployed already. 2024-06-14 17:03:04 +08:00
Haojun Liao
cbf8b363fc fix(stream): update checkpoint info by using trans. 2024-06-10 00:46:01 +08:00
Haojun Liao
bef1953b24
Merge branch '3.0' into fix/3_liaohj 2024-05-31 14:27:46 +08:00
Haojun Liao
d1d868f239 refactor: do some internal refactor and set the pActiveInfo for rsma. 2024-05-29 00:08:29 +08:00
Shungang Li
a8383369ba enh: arb check roletime before check sync 2024-05-28 15:13:58 +08:00
Haojun Liao
bd514fe4d3 other: merge rows. 2024-05-28 11:13:09 +08:00
Haojun Liao
be107b204c enh(stream): support retrieve checkpoint data remotely when start stream tasks in vnodes. 2024-05-28 09:54:22 +08:00
wangjiaming0909
22aad41810 fix smaTest failed caused by mnode not inited 2024-05-24 11:42:01 +08:00
Haojun Liao
6898eba6ed fix(stream): expand stream tasks are divided into two phase, the first is to build stream task and then expand stream task before start stream tasks. 2024-05-14 14:34:31 +08:00
Haojun Liao
7f93ec2c53 fix(stream): update the checkpoint info for followers, and some internal refactor. 2024-05-11 09:37:02 +08:00
Haojun Liao
7ad7ef7cfc refactor: do some internal refactor. 2024-05-06 17:03:41 +08:00
Haojun Liao
ea527b19b8 fix(stream): set correct start version if no checkpoint exist. 2024-04-19 19:12:23 +08:00
Haojun Liao
d799212fb2 refactor: do some internal refactor. 2024-04-12 10:02:18 +08:00
Shungang Li
3a656c3595 feat: support 2 replica with arbitrator 2024-03-19 09:30:35 +08:00
Haojun Liao
aee0460daf enh(stream):rsp the checkpoint ready msg. 2024-02-27 11:15:09 +08:00
Haojun Liao
dafac591f7 enh(stream): add rsp for req-checkpoint 2024-02-27 10:53:25 +08:00
Haojun Liao
b361eb4a7c fix(stream): add hb rsp handle function for vnode and snode. 2024-02-21 15:16:07 +08:00
Haojun Liao
cb2ea4a721 fix(stream): take ver snapshot for all vgroups before launching stream with fill-history option opened. 2024-02-01 11:37:57 +08:00
Haojun Liao
ac08f659c3 refactor: remove invalid procedure in fill-history. 2024-01-26 16:20:43 +08:00
Haojun Liao
80793c94c2 fix(stream): reset the tasks status before start all tasks, and do some other internal refactor. 2024-01-16 14:32:45 +08:00
Haojun Liao
bff0ad89a4 other: merge 3.0 2024-01-11 09:51:12 +08:00
wangmm0220
df69a89da2 fix:[TD-28185]add pause & resume logic for snode 2024-01-10 18:46:10 +08:00
Haojun Liao
29219ba9f4 fix(stream): add task number check. 2024-01-07 01:32:50 +08:00
Haojun Liao
0c4b91dc72 enh(stream): start task async. 2024-01-05 17:55:39 +08:00
Haojun Liao
b175a4b7b3 refactor(stream): async stop tasks, and do some internal refactor. 2023-12-22 17:56:16 +08:00
Haojun Liao
92045f5485 fix(stream): add snode check, and handle the task status reset from checkpoint when it is in the snode. 2023-12-19 18:34:30 +08:00
Haojun Liao
fee198f9ad refactor: do some internal refactor. 2023-12-19 16:21:11 +08:00
Haojun Liao
ea6e78cbaa fix(stream): add async call restart, instead of sync wait. 2023-12-18 16:33:44 +08:00
wangmm0220
70c5d36103 opti:extract common logic from snode & vnode to tqCommon 2023-11-27 17:49:09 +08:00
wangmm0220
9a09e318ca opti:common message process logic in vnode and snode 2023-11-22 16:53:33 +08:00