Commit graph

2714 commits

Author SHA1 Message Date
Simon Guan
2c27c10fa8 Merge branch 'main' into merge/mainto3.0 2025-03-20 14:05:45 +08:00
Haojun Liao
db57fc4ba8
fix(stream): fix race condition in send msg. (#30277)
Co-authored-by: 54liuyao <54liuyao@163.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: wangmm0220 <wangmm0220@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-19 23:15:15 +08:00
Simon Guan
d02196a231
refactor: adjust some logs (#30245)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* docs: update keywords

* fix: compile errors in windows

* docs: minor changes

* docs: minor changes

* refactor: minor changes

* docs: fminro changes

* docs: format

* refactr: remove uncessory logs

* docs: format doc
2025-03-18 17:55:36 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
Jinqing Kuang
ac23a4fc55
feat(stream): support stream processing for virtual tables (#30144)
* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
Jing Sima
410324746b
feat:[TS-4897] virtual table (#30098)
* feat: [TS-4897] Support create/drop/alter/show/describe vtable

* feat: [TS-4897] Support vtable's query

* feat: [TS-4897] Support create virtual supertable

* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.

* feat: [TS-4897] Add create test and fix bugs

* feat: [TS-4897] Add alter/drop test and fix bugs

* feat: [TS-4897] Add describe/show test and fix bugs

* feat: [TS-4897] Add auth test and fix bugs

* feat: [TS-4897] Fix meta/catalog/cache bugs

* feat: [TS-4897] Support select tag from virtual child table

* feat: [TS-4897] Add select test and fix plenty of bugs

* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.

* feat: [TS-4897] Fix 'schema is old'

* feat: [TS-4897] Support virtual stable query

* feat: [TS-4897] Add tests and Fix bugs

* feat: [TS-4897] resolve conflict.
2025-03-15 14:10:46 +08:00
WANG MINGMING
3d053e2c9b
feat(stream): optimize client logic of creating stream if with history (#30059)
* feat:[TS-5617]use last_row cache mode if creating stream in fill_history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* fix: heap use after free

* feat: add log

* fix: ci case error

* fix: compile error in windows

* fix: ci case error

* fix: heap user after free

* fix: memory leak

* fix: ci case error

* fix: ci case error

* fix: ci case error

---------

Co-authored-by: yihaoDeng <yhdeng@taosdata.com>
2025-03-14 13:55:52 +08:00
Kaili Xu
13f9bddf3d
feat: support customized taos/taosd (#29736)
* feat: support TDAcoreOS

* chore: cmake options for TD_ACORE

* chore: disable lemon for TD_ACORE

* chore: add lzma2 and msvcregex

* chore: cmake for lzma2

* chore: adapt for TD_ACORE

* chore: adapt strcasecmp for TD_ACORE

* chore: adapt for geos/threadName

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE termio

* chore: refact transComm.h for TD_ACORE

* chore: refact transportInt.h for TD_ACORE

* chore: refact trans.c for TD_ACORE

* chore: refact trpc.h for TD_ACORE

* chore: refact transCli.c/transComm.c/transSvr.c for TD_ACORE

* chore: refact uv.h for TD_ACORE

* chore: refact geosWrapper.h for TD_ACORE

* chore: refact token/builtins/udf for TD_ACORE

* chore: refact rocks for TD_ACORE

* chore: refact tsdbCache.c for TD_ACORE, use LRU cache for last/last_row, not use rocksdb

* chore: refact FAIL to _ERR to solve conflicts for TD_ACORE

* chore: restore lemon.c/lempar.c

* chore: support build lemon for TD_ACORE

* chore: refact trpc and siginfo_t for TD_ACORE

* chore: refact timezone for TD_ACORE

* chore: refact lz4 for TD_ACORE

* chore: refact TD_ACORE to make compile pass

* chore: code optimization for TD_ASTRA

* feat: support run taos with taosd integrated

* feat: support invoke taos shell

* feat: support invoke taos shell

* feat: support invoke taos shell

* chore: code optimization

* chore: fix undefined reference problem os TD_ASTRA

* chore: resolve compile problem for TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix getpid

* chore: fix typo

* chore: set stack size and ajust min pack size for TD_ASTRA

* chore: fix pthread create parameters

* chore: chmod adapt for TD_ASTRA

* chore: fix trans compile problem

* chore: adapt chmod for TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: conditional compile option

* chore: adapt for TD_ASTRA

* chore: adjust taosPId and msvcregex for TD_ASTRA

* chore: log dir separator for wal build name

* chore: fix type of pointer parameter

* chore: fix compile problem of tsdbGetS3Size

* enh: get last ver from wal log for TD_ASTRA

* enh: refact wal meta ver

* enh: refact wal meta ver

* fix: typo of taosUcs4Compare

* enh: process return value of CI

* chore: more code for TD_ASTRA adaption

* chore: return value of taosCloseFile in walMeta.c

* chore: fix compile problem

* chore: fix compile problem of TD_ASTRA

* fix: update macro for tq and stream task

* chore: code optimization for TD_ASTRA

* chore: restore create log and init cfg interface

* chore: restore strncasecmp and strcasecmp

* fix: adjust the field position of SDataBlockInfo

* fix: pragma pack min size

* fix: pragma pack min size

* chore: more code for TD_ASTRA adaption

* fix: type of parameters

* chore: adapt strncasecmp and strcasecmp for TD_ASTRA

* chore: restore interface of init log

* enh: pack push optimization

* fix: taos init cfg

* add astra support

* fix: fetch the value of suid

* chore: switch of  build with udf

* add temp code

* chore: more code for TD_ASTRA adaption

* chore: add macro ERRNO to replace errno

* chore: bytes align for TD_ASTRA

* fix: remove obsolete codes

* enh: support USE_UDF macro

* fix compile error

* fix: resolve redefinition problem

* fix: compile problem of log.cpp

* fix: compile problem of osTimezone

* fix: resolve compile problem of udf

* fix: pragma definition on windows

* fix: ucs4 and stpncpy for TD_ASTRA

* fix: memory align problem for TD_ASTRA

* enh: solve memory leak for TD_ASTRA_RPC

* fix: compile problem of taosSetInt64Aligned

* fix: restore mndSubscribe.c

* fix: scalar for udf

* chore: code adaption for TD_ASTRA

* chore: code optimization for TD_ASTRA

* fix: typo of add definition

* fix: typo of macro in tudf.h

* chore: remove void to make CI pass

* enh: move macro from cmake.platform to cmake.options

* enh: byte align for hash node and error code

* chore: restore the size for lru cache

* enh: restore some code about pack push

* chore: restore the pack push in tmsg.h

* fix: add macro of pack pop for windows

---------

Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 13:32:13 +08:00
Hongze Cheng
4c6098e290 feat: revert taosd -r option 2025-03-12 11:25:17 +08:00
Haojun Liao
90aa27ee23 fix(stream): handle TDMT_STREAM_CHKPT_EXEC in stream queue. 2025-03-11 13:57:31 +08:00
Haojun Liao
d24bc139a5 chore: merge 3.0 2025-03-10 18:31:56 +08:00
Simon Guan
8fa8fc2a0c
Merge pull request #29934 from taosdata/enh/TS-5650/showLike
enh: show variables like
2025-03-07 16:37:19 +08:00
xinsheng Ren
4edddd6038
Merge branch '3.0' into enh/TS-5650/showLike 2025-03-07 10:29:35 +08:00
Simon Guan
609ff730c4 refactor: jemalloc options 2025-03-07 10:03:43 +08:00
Simon Guan
6250adb3b1 refactor: rename udfd to taosudf 2025-03-06 22:20:52 +08:00
Haojun Liao
95e75196e2 chore: merge main. 2025-03-06 18:45:21 +08:00
Simon Guan
34120a38c5 Merge branch '3.0' into merge/mainto3.0 2025-03-05 18:12:55 +08:00
Simon Guan
292b0cb317 Merge branch 'main' into merge/mainto3.0 2025-03-05 18:10:31 +08:00
Simon Guan
1a9f75e663 Merge branch '3.0' into fix/internal 2025-03-04 13:47:03 +08:00
Simon Guan
b74b038f4f Merge branch '3.0' into merge/mainto3.0 2025-03-04 11:32:29 +08:00
Simon Guan
65d5fd0578
Merge pull request #29975 from taosdata/enh/stopall
refactor(stream): stop all tasks before drop vnodes.
2025-03-03 20:01:56 +08:00
Haojun Liao
f563a35b3f fix(stream):check the maximum allowed forecast rows, add one test case, and do some internal refactor. 2025-03-03 17:27:05 +08:00
Haojun Liao
f59d25f1a5 fix(stream): fix syntax error. 2025-03-03 16:27:54 +08:00
Haojun Liao
ac667164ed refactor: do some internal refactor. 2025-03-03 16:20:58 +08:00
Simon Guan
7326e3d8bc Merge branch '3.0' into merge/mainto3.0 2025-03-03 14:01:51 +08:00
Simon Guan
6826edf5e0 Merge branch 'main' into merge/mainto3.0 2025-03-03 14:01:43 +08:00
Simon Guan
f88c249fa4
Merge pull request #29946 from taosdata/fix/remove-sync-heartbeat-lock
fix: remove sync heartbeat lock
2025-03-03 13:52:16 +08:00
Simon Guan
eee9976a44
Merge pull request #29958 from taosdata/fix/TD-33872-improve-status-sync-heartbeat
fix: [td-33872] improve status sync heartbeat
2025-03-03 13:51:44 +08:00
Simon Guan
d7207861bb Merge remote-tracking branch 'origin/3.0' into fix/internal 2025-03-03 10:43:07 +08:00
factosea
e91014f807 feat: show variables like 2025-03-02 22:48:05 +08:00
Haojun Liao
a71f414d97 refactor(stream): stop all tasks before drop vnodes. 2025-03-02 02:06:14 +08:00
dmchen
b0f62f4f91 fix:statusseq 2025-03-01 12:51:09 +08:00
dmchen
3fb2476b81 fix:statusseq 2025-03-01 12:50:28 +08:00
dmchen
ccd4b19c5f fix:remove-send-status-lock 2025-03-01 00:53:04 +08:00
dmchen
092e8e4488 fix:remove-send-status-lock 2025-03-01 00:52:11 +08:00
Simon Guan
43ca6d5f9e refactor: adjust some logs 2025-02-28 16:49:46 +08:00
dmchen
45cdd871db fix/remove-get-queue-size-when-print-heartbeat-log 2025-02-28 14:15:20 +08:00
dmchen
aa7cb880bb fix/remove-get-queue-size-when-print-heartbeat-log 2025-02-28 11:53:18 +08:00
dmchen
1237e762c1 fix/remove-get-queue-size-when-print-heartbeat-log 2025-02-28 11:52:27 +08:00
dmchen
2d4697d32f fix:decouple get mload 2025-02-27 19:17:54 +08:00
dmchen
4dd0622594 fix:decouple get mload 2025-02-27 19:07:33 +08:00
dmchen
f6a472ae68 fix:decouple get mload 2025-02-27 19:05:06 +08:00
Hongze Cheng
0d2ca243b1 Merge branch '3.0' of https://github.com/taosdata/TDengine into enh/TS-5445-3.0 2025-02-27 17:18:33 +08:00
dmchen
d435bbae1d fix:add-status-timestamp 2025-02-27 15:54:07 +08:00
dmchen
55f58d743f fix/move-status-to-new-queue 2025-02-27 15:54:06 +08:00
dmchen
7eeb800213 fix:add-status-timestamp 2025-02-27 15:29:52 +08:00
dmchen
1f6c4df422 fix/move-status-to-new-queue 2025-02-27 15:26:06 +08:00
dmchen
93992f3d6e fix:add-status-timestamp 2025-02-27 15:22:33 +08:00
dmchen
031f176aa1 fix/move-status-to-new-queue 2025-02-27 11:02:18 +08:00
Simon Guan
76ad364b93 enh: adjust some logs 2025-02-27 09:55:11 +08:00