Commit graph

2741 commits

Author SHA1 Message Date
dapan1121
ee09c9643c Merge remote-tracking branch 'origin/feat/TS-6100-3.0' into feat/TS-6100-3.0.pw5 2025-05-09 11:32:10 +08:00
dapan1121
7aded97b75 fix: add more fetch types 2025-05-07 18:30:05 +08:00
wangjiaming0909
e95c5a9d85 new-stream: runner impl 2025-05-07 18:14:17 +08:00
Jinqing Kuang
87da875fdf fix(stream): fix calc request send of stream trigger task 2025-05-06 15:22:05 +08:00
Jinqing Kuang
437a6fc4fd fix(stream): fix message process of stream trigger task 2025-05-06 09:20:37 +08:00
wangmm0220
145a535e12 feat(stream): remove useless code 2025-04-30 09:12:20 +08:00
wangmm0220
7dc6557b46 fix(stream): message error 2025-04-29 21:10:12 +08:00
wangmm0220
19a878cc79 feat(stream): add thread worker for stream reader 2025-04-29 18:51:21 +08:00
wangjiaming0909
1ccc6c69a8 new-stream: runner impl 2025-04-29 15:31:17 +08:00
Jinqing Kuang
2a796e17ef feat(stream): implement stream trigger realtime calc 2025-04-29 09:23:05 +08:00
dapan1121
ef1e51e9b9 enh: add stream obj name 2025-04-29 08:38:03 +08:00
wangmm0220
7158d4c76f Merge remote-tracking branch 'origin/feat/TS-6100-3.0' into feat/TS-6100-marks-new 2025-04-28 14:37:28 +08:00
wangmm0220
a78b09577a feat(stream): add thread worker for stream reader 2025-04-28 14:37:05 +08:00
Jing Sima
22ebaece7c feat: [TS-6100] Process SCMCreateStreamReq 2025-04-28 11:28:33 +08:00
wangmm0220
11b625c63e feat(stream): add thread worker for stream reader 2025-04-25 18:47:25 +08:00
wangmm0220
eee6456cc4 feat(stream): add thread worker for stream reader 2025-04-25 18:18:17 +08:00
dapan1121
cc025dbd59 enh: add task plan update 2025-04-25 14:59:41 +08:00
dapan1121
330cf9982d fix: compile issues 2025-04-24 09:11:54 +08:00
dapan1121
ae0e1ded7d enh: add hb processing 2025-04-22 17:46:17 +08:00
dapan1121
279595240e enh: add deploy processing 2025-04-18 19:28:43 +08:00
dapan1121
7c714f7d5f Merge remote-tracking branch 'origin/feat/TS-6100-3.0.pw1' into feat/TS-6100-3.0 2025-04-17 10:48:44 +08:00
dapan1121
38bdafd329 enh: add deploy processing 2025-04-17 10:45:06 +08:00
wangjiaming0909
ceaf69184c new-stream: create new-stream threads 2025-04-16 18:16:09 +08:00
dapan1121
675790a63c fix: remove codes 2025-04-10 09:09:30 +08:00
Haojun Liao
af06bea439
fix(stream): add start/stop message. (#30400) 2025-03-24 19:38:00 +08:00
Haojun Liao
82ffeb82a9
fix(gpt): fix the bug for algo name case sensitive (#30362) 2025-03-23 13:20:24 +08:00
Alex Duan
c133489917
new-libtaos-with-ws (#29971)
* enh: rename libtaos.so to libtaosinternal.so

* enh: let python system-test work

* enh: shell for libinternal.so

* enh: refact script

* enh: wrapper for libtaosinternal

* enh: rename some files

* enh: let shell support internal driver

* enh: minor changes

* enh: minor changes

* add intenal.h

* enh: remove unused codes

* enh: minor changs

* feat: mac os issues

* enh: let python test use internal connection

* enh: let nettest work

* enh: let taosc -C work

* enh: test shell in mac os

* enh: minor changes

* enh: let libtaosinternal.so work in linux

* enh: update install script

* enh: for ci

* enh: for windows compile

* enh: minor changes

* enh: compile in windows

* enh: minor changes

* enh: taosinternal work in windows

* enh: add taos_internal_static

* enh: rollback os_for_wrapper

* fix: conflicts

* fix: conflicts

* fix: compile errors

* fix: conflicts

* enh: rename taosinternal to taosnative

* enh: rename taosinternal to taosnative

* fix: compile error

* enh: remove taosnative.h

* fix: compile errors

* enh: set default shell options

* fix: compile errors

* fix: compile errors

* debug: switch taosws branch to feat/new-libtaos-with-ws

* enh: taosBenchmark remove WEBSOCKET finished

* fix: CMakeLists.txt remove WEBSOCKET

* fix: compile errors

* fix: taosBenchmark delete restful code

* .gitignore remove taos-tools folder

* fix: remove taosdump WEBSOCKET MACRO

* fix: build error for taosdump

* fix: build project passed

* fix: support -Z for connect mode

* fix: build error

* fix: build error fixed

* fix: add -Z options

* fix: native and websocket with string

* fix: -Z option core

* fix: build unit test

* fix: build error

* fix: add colon for array

* fix: unit test can not include fun

* fix: CTest pointer null is nullptr

* fix: include pub.h

* fix: taosdump add -Z --driver options

* fix: taosdump support -Z with pub.h pub.c

* fix: toolsGetTimeDay move to pub.c

* fix: restore pub.c only simple fun

* fix: add error tips

* fix: tmfree remove from taosdump

* fix: remove double declare taos variant

* fix: taos support -Z option

* enh: configDir set with main fun

* fix: add g_arguments.configDir

* fix: show tips add cfgdir

* fix: g_argument is pointer

* fix: configDir is global var

* fix: declare g_configDir in benchMain.c

* fix: taos version not show on websocket

* stmt2 websocket prepare with supertable

* stmt2 websocket prepare build

* fix: stmt and stmt2 prepare add db name

* fix: remove -R --restful test case

* fix: taosBenchmark adjust priority cmd > json > evn

* fix: add connect mode test case connMode.py

* fix: clear evn and add host port check

* fix: army/test.py modify start taosAdapter is default

* change: system-test/test.py default start taosAdapter

* fix: add taosAdapter variant to system-test/test.py

* fix: -y must put behind other options

* fix: remove rest sml and add -N normal case

* fix: add -c work cfg test case

* del: queryMain.py remove rest query

* fix: remove groups have a blank append bug

* fix: caseBase.py add db in sql query

* fix: query_json.py remove rest test

* fix: taosBenchmark uniform dsn describe

* fix: add pub.h pub.c to tools public

* fix: case remove restful interface

* fix: build error

* fix: taosDriverCleanup no arg call

* fix: taosdump add test conn mode case taosdumpCommandline.py

* fix: modify insertFullType.json db name

* fix: taosdump connMode test case taosDumpCommandline.py passed

* enh: priority cmd > env > json

* fix: taosCli remove is_native variant

* fix: password is not pointer

* fix: comment -o test case

* fix: solve fun return code check

* fix: windows unresolved strcasecmp

* fix: remove coverHostToServAddr

* fix: forbid check ODR violation

* ci: trigger pr run

* fix: build error not del >>>>>3.0

* fix: support json dbinfo->vgroups have blank space

* fix: solve conflict with merge

* fix: with TrimCaseCmp fun to compare vgroups

* fix: del vgroups support blank code

* feat: wrapper add taos_fetch_fields_e

* fix: move jni from libtaosnative.so to libtaos.so

* fix: move clientTmqConnector.c for jni

* fix: fix assert check with del comment code

* fix: taosBenchmark add prepare stmt debug log

* fix: stmt2 already have debug log on preapare

* fix:  (double)LLONG_MAX force covert

* fix: stmt insert normal table with websocket

* fix: normal table with stmt too many ?

* fix: taosdump stmt_prepare with child tbname

* fix: build error add int code

* feat: stmtPrepare with child table to do

* taos -a options move to native test case

* fix:unit test benchmarkTest passed

* fix: query_json-with-sqlfile.py case no check results

* fix: queryMain.py case lost, put in tasks again

---------

Co-authored-by: Shengliang Guan <slguan@taosdata.com>
2025-03-20 22:54:29 +08:00
Simon Guan
2c27c10fa8 Merge branch 'main' into merge/mainto3.0 2025-03-20 14:05:45 +08:00
Haojun Liao
db57fc4ba8
fix(stream): fix race condition in send msg. (#30277)
Co-authored-by: 54liuyao <54liuyao@163.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: wangmm0220 <wangmm0220@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-19 23:15:15 +08:00
Simon Guan
d02196a231
refactor: adjust some logs (#30245)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* docs: update keywords

* fix: compile errors in windows

* docs: minor changes

* docs: minor changes

* refactor: minor changes

* docs: fminro changes

* docs: format

* refactr: remove uncessory logs

* docs: format doc
2025-03-18 17:55:36 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
Jinqing Kuang
ac23a4fc55
feat(stream): support stream processing for virtual tables (#30144)
* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
Jing Sima
410324746b
feat:[TS-4897] virtual table (#30098)
* feat: [TS-4897] Support create/drop/alter/show/describe vtable

* feat: [TS-4897] Support vtable's query

* feat: [TS-4897] Support create virtual supertable

* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.

* feat: [TS-4897] Add create test and fix bugs

* feat: [TS-4897] Add alter/drop test and fix bugs

* feat: [TS-4897] Add describe/show test and fix bugs

* feat: [TS-4897] Add auth test and fix bugs

* feat: [TS-4897] Fix meta/catalog/cache bugs

* feat: [TS-4897] Support select tag from virtual child table

* feat: [TS-4897] Add select test and fix plenty of bugs

* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.

* feat: [TS-4897] Fix 'schema is old'

* feat: [TS-4897] Support virtual stable query

* feat: [TS-4897] Add tests and Fix bugs

* feat: [TS-4897] resolve conflict.
2025-03-15 14:10:46 +08:00
WANG MINGMING
3d053e2c9b
feat(stream): optimize client logic of creating stream if with history (#30059)
* feat:[TS-5617]use last_row cache mode if creating stream in fill_history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* feat(stream): optimize client logic of creating stream if with history

* fix: heap use after free

* feat: add log

* fix: ci case error

* fix: compile error in windows

* fix: ci case error

* fix: heap user after free

* fix: memory leak

* fix: ci case error

* fix: ci case error

* fix: ci case error

---------

Co-authored-by: yihaoDeng <yhdeng@taosdata.com>
2025-03-14 13:55:52 +08:00
Kaili Xu
13f9bddf3d
feat: support customized taos/taosd (#29736)
* feat: support TDAcoreOS

* chore: cmake options for TD_ACORE

* chore: disable lemon for TD_ACORE

* chore: add lzma2 and msvcregex

* chore: cmake for lzma2

* chore: adapt for TD_ACORE

* chore: adapt strcasecmp for TD_ACORE

* chore: adapt for geos/threadName

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE

* chore: build adapt for TD_ACORE termio

* chore: refact transComm.h for TD_ACORE

* chore: refact transportInt.h for TD_ACORE

* chore: refact trans.c for TD_ACORE

* chore: refact trpc.h for TD_ACORE

* chore: refact transCli.c/transComm.c/transSvr.c for TD_ACORE

* chore: refact uv.h for TD_ACORE

* chore: refact geosWrapper.h for TD_ACORE

* chore: refact token/builtins/udf for TD_ACORE

* chore: refact rocks for TD_ACORE

* chore: refact tsdbCache.c for TD_ACORE, use LRU cache for last/last_row, not use rocksdb

* chore: refact FAIL to _ERR to solve conflicts for TD_ACORE

* chore: restore lemon.c/lempar.c

* chore: support build lemon for TD_ACORE

* chore: refact trpc and siginfo_t for TD_ACORE

* chore: refact timezone for TD_ACORE

* chore: refact lz4 for TD_ACORE

* chore: refact TD_ACORE to make compile pass

* chore: code optimization for TD_ASTRA

* feat: support run taos with taosd integrated

* feat: support invoke taos shell

* feat: support invoke taos shell

* feat: support invoke taos shell

* chore: code optimization

* chore: fix undefined reference problem os TD_ASTRA

* chore: resolve compile problem for TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix undefined reference problem os TD_ASTRA

* chore: fix getpid

* chore: fix typo

* chore: set stack size and ajust min pack size for TD_ASTRA

* chore: fix pthread create parameters

* chore: chmod adapt for TD_ASTRA

* chore: fix trans compile problem

* chore: adapt chmod for TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: more code for adaption of TD_ASTRA

* chore: byte alignment for TD_ASTRA

* chore: conditional compile option

* chore: adapt for TD_ASTRA

* chore: adjust taosPId and msvcregex for TD_ASTRA

* chore: log dir separator for wal build name

* chore: fix type of pointer parameter

* chore: fix compile problem of tsdbGetS3Size

* enh: get last ver from wal log for TD_ASTRA

* enh: refact wal meta ver

* enh: refact wal meta ver

* fix: typo of taosUcs4Compare

* enh: process return value of CI

* chore: more code for TD_ASTRA adaption

* chore: return value of taosCloseFile in walMeta.c

* chore: fix compile problem

* chore: fix compile problem of TD_ASTRA

* fix: update macro for tq and stream task

* chore: code optimization for TD_ASTRA

* chore: restore create log and init cfg interface

* chore: restore strncasecmp and strcasecmp

* fix: adjust the field position of SDataBlockInfo

* fix: pragma pack min size

* fix: pragma pack min size

* chore: more code for TD_ASTRA adaption

* fix: type of parameters

* chore: adapt strncasecmp and strcasecmp for TD_ASTRA

* chore: restore interface of init log

* enh: pack push optimization

* fix: taos init cfg

* add astra support

* fix: fetch the value of suid

* chore: switch of  build with udf

* add temp code

* chore: more code for TD_ASTRA adaption

* chore: add macro ERRNO to replace errno

* chore: bytes align for TD_ASTRA

* fix: remove obsolete codes

* enh: support USE_UDF macro

* fix compile error

* fix: resolve redefinition problem

* fix: compile problem of log.cpp

* fix: compile problem of osTimezone

* fix: resolve compile problem of udf

* fix: pragma definition on windows

* fix: ucs4 and stpncpy for TD_ASTRA

* fix: memory align problem for TD_ASTRA

* enh: solve memory leak for TD_ASTRA_RPC

* fix: compile problem of taosSetInt64Aligned

* fix: restore mndSubscribe.c

* fix: scalar for udf

* chore: code adaption for TD_ASTRA

* chore: code optimization for TD_ASTRA

* fix: typo of add definition

* fix: typo of macro in tudf.h

* chore: remove void to make CI pass

* enh: move macro from cmake.platform to cmake.options

* enh: byte align for hash node and error code

* chore: restore the size for lru cache

* enh: restore some code about pack push

* chore: restore the pack push in tmsg.h

* fix: add macro of pack pop for windows

---------

Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 13:32:13 +08:00
Hongze Cheng
4c6098e290 feat: revert taosd -r option 2025-03-12 11:25:17 +08:00
Haojun Liao
90aa27ee23 fix(stream): handle TDMT_STREAM_CHKPT_EXEC in stream queue. 2025-03-11 13:57:31 +08:00
Haojun Liao
d24bc139a5 chore: merge 3.0 2025-03-10 18:31:56 +08:00
Simon Guan
8fa8fc2a0c
Merge pull request #29934 from taosdata/enh/TS-5650/showLike
enh: show variables like
2025-03-07 16:37:19 +08:00
xinsheng Ren
4edddd6038
Merge branch '3.0' into enh/TS-5650/showLike 2025-03-07 10:29:35 +08:00
Simon Guan
609ff730c4 refactor: jemalloc options 2025-03-07 10:03:43 +08:00
Simon Guan
6250adb3b1 refactor: rename udfd to taosudf 2025-03-06 22:20:52 +08:00
Haojun Liao
95e75196e2 chore: merge main. 2025-03-06 18:45:21 +08:00
Simon Guan
34120a38c5 Merge branch '3.0' into merge/mainto3.0 2025-03-05 18:12:55 +08:00
Simon Guan
292b0cb317 Merge branch 'main' into merge/mainto3.0 2025-03-05 18:10:31 +08:00
Simon Guan
1a9f75e663 Merge branch '3.0' into fix/internal 2025-03-04 13:47:03 +08:00
Simon Guan
b74b038f4f Merge branch '3.0' into merge/mainto3.0 2025-03-04 11:32:29 +08:00
Simon Guan
65d5fd0578
Merge pull request #29975 from taosdata/enh/stopall
refactor(stream): stop all tasks before drop vnodes.
2025-03-03 20:01:56 +08:00
Haojun Liao
f563a35b3f fix(stream):check the maximum allowed forecast rows, add one test case, and do some internal refactor. 2025-03-03 17:27:05 +08:00
Haojun Liao
f59d25f1a5 fix(stream): fix syntax error. 2025-03-03 16:27:54 +08:00