Commit graph

5966 commits

Author SHA1 Message Date
Haojun Liao
8bb4979e53 refactor: remove some comments. 2025-05-22 09:55:56 +08:00
Haojun Liao
231d8b416e fix(stream): fix memory leak. 2025-05-22 09:55:18 +08:00
Haojun Liao
a05b023c3e fix(stream): fix memory leak. 2025-05-21 19:13:23 +08:00
Haojun Liao
e04f81385a fix(stream): collect involved tasks firstly. 2025-05-21 18:41:47 +08:00
WANG MINGMING
40216c6143
opti(tmq): save tmq offset in vnode commit time & change log level (#30993) 2025-05-15 18:15:01 +08:00
Yihao Deng
c61c087e98
feat: Refresh user IP white list when drop-dnode finished (#31089) 2025-05-15 16:03:41 +08:00
Haojun Liao
d02b17be0e
fix(stream): extends the node update check interval. (#31048) 2025-05-14 10:15:17 +08:00
Haojun Liao
e439569455
test(stream): update the test cases. (#31044) 2025-05-12 11:25:07 +08:00
Zhixiao Bao
338072e262
enh: add more feat for stb keep. (#31019) 2025-05-10 15:17:22 +08:00
Jing Sima
84016904ff
fix: [TS-6445] Fix virtual tables not detecting schema changes. (#30999)
Ensure virtual tables invalidate cached schema metadata and reload it when a schema change (e.g., ALTER VTABLE) is detected. This prevents stale schema references and query errors after structural updates.
2025-05-10 10:34:54 +08:00
Kaili Xu
a65dd404ee
enh: number of dnodes with vgroups not exceed 2 when dnode limit is 2 and dualRepliaHA is specified (#30959) 2025-05-06 16:00:59 +08:00
Haojun Liao
704686ece6
fix(stream): fix the double release stream object in mnode. (#30942) 2025-05-06 15:56:24 +08:00
Kaili Xu
f1e480ef53
enh: support revoke sign in show grants logs (#30929) 2025-04-29 14:46:27 +08:00
wangmm0220
ffffefb802 fix(tmq): add log 2025-04-24 11:45:37 +08:00
freemine
6267763e85
Freemine.cmake.3.0 (#30663)
* chore(ci): with cmake verbose, temporarily

* chore(cmake): prefer Threads::Threads

* fix(cmake): debug building gtest on ubuntu 20.04 for the moment

* fix(cmake): try adding Threads::Threads

* fix(cmake): debug building gtest on ubuntu 20.04 for the moment

* fix(cmake): revert taosd-ci-build.yml

* fix(cmake): debug building on ubuntu 20.04 for the moment

* fix(cmake): debug building on ubuntu 20.04 for the moment

* fix(cmake): debug building on ubuntu 20.04 for the moment, adding dl right after DEP_ext_libuv

* fix(cmake): debug building on ubuntu 20.04 for the moment, try full run

* fix(cmake): fix building on ubuntu 20.04

* fix(cmake): dl is only valid in non-windows platform

* fix(cmake): dl order

* refactor(cmake): adjust option/cache variables for external dependencies

* refactor(cmake): mv patch files from contrib/ to cmake/

* refactor(cmake): typo

* refactor(cmake): adding cache on ci

* refactor(cmake):
1. ext_lzma2 is only used by Linux
2. fix bug in taosd-ci-build.yml

* enh: adding functionalities of build.sh

* fix(ci): `cmake -B` shall maintain consistency

* refactor(cmake): LOCAL_REPO/LOCAL_URL as cache variable

* refactor(cmake): taosadapter to be built in main build directory rather than .externals

* refactor(cmake): adding rpath for taosadapter, linux now

* refactor(cmake): adding rpath for taosadapter, macos now

* chore(cmake): remove some comments

* debug(ci): bypass TD_EXTERNALS_USE_ONLY

* debug(ci): bypass .externals caching for the moment

* fix(cmake): temporarily use taosadapter/main branch because this branch(freemine.cmake.3.0) has not include taos_fetch_fields_e from upstream

* refactor(cmake): clean GIT_TAG/GIT_SHALLOW pairs

* enh(ci): use build.sh rather than hard-coded in taosd-ci-build.yml, linux for now

* debug(ci): undefined __isoc23_strtoull

* debug(ci): benchmarkTest

* debug(ci): what if use_only = off

* debug(ci): remove .externals to make a refresh build

* debug(ci): back to cache, and ls files of gtest

* debug(ci): remove find coz {} not escaped

* debug(ci): cache key seems missing "matrix.os"

* refactor(ci): fixed cache key and reopen macos/windows

* chore(ci): literally -j4

* chore(ci): Trigger GitHub Actions re-run

* chore(ci): Trigger GitHub Actions re-run

* fix(ci): typo

* chore(ci): JOBS calculated by build.sh itself

* refactor(ci): enable WEBSOCKET in build.sh by default

* refactor(ci): enable WEBSOCKET in build.sh by default, macOS

* chore(cmake): remove obsoletes

* refactor(ci): stop service before running taosd

* chore(ci): `./build.sh install` just install TDengine, you will have to start taosd by `./build.sh start`

* refactor(cmake): adding source code found on http://emfisis.physics.uiowa.edu/Software/C/libargp/

* refactor(cmake): moving jansson/snappy/avro to .externals, and porting BUILD_TEST/WEBSOCKET on Windows

* refactor(cmake): porting linux

* refactor(cmake): porting macos
1. check argp.h
2. re-enable building taosdump

* refactor(cmake): porting linux
1. adjust CMAKE_C/CXX_FLAGS

* refactor(cmake): porting windows, adjust CMAKE_C/CXX_FLAGS

* refactor(cmake): use cmake_parse_arguments in macro INIT_EXT

* refactor(cmake): format ci cache directives

* refactor(cmake): adding comments for ci externals-cache directives

* refactor(cmake): adding check_env.cmake to checking libraries installed on the system

* refactor(cmake): porting linux, make homebrew-installed-libs available

* refactor(cmake): remove BYPRODUCTS from INIT_EXT macro

* refactor(cmake): INC_DIR is oneValueArg

* refactor(cmake): taosdump depends on ext_xz(LZMA)

* refactor(cmake): add copy_pthreadVC3 target on windows

* refactor(cmake): mv taosd-ci-build.yml to tdengine-build.yml

* chore(ci): try to fix yml syntax error

* refactor(cmake): use brew --prefix to get HomeBrew's install path

* refactor(cmake): typo

* debug(ci): build and cache externals first

* debug(ci): adding -Wno-stringop-overread for the moment

* debug(ci): make Threads::Threads public from os target

* debug(ci): make os public

* debug(ci): re-enable start and test

* debug(ci): let it go even if taosBenchmark failed

* debug(ci): copy logic from TDengine/main

* debug(ci): try ./build.sh start/test

* debug(ci): revert to original simple test logic in ci, taosBenchark/taos

* chore(cmake): code-convention and ci-logic

* debug(ci): show versions

* debug(ci): forcefully use specific GIT_TAG by commit-id for taos-connector-rust.git and taosadapter.git

* refactor(cmake): remove empty lines

* debug(ci): typo

* debug(ci): set supportVnodes to 256 before starting taosd

* debug(ci): fix: sudo and pipe

* debug(ci): cat /etc/taos/taos.cfg to check if supportVnodes effective

* debug(ci): cat forceReadConfig to taos.cfg

* debug(ci): adding taos -s "show dnodes"

* debug(ci): print more info

* debug(ci):
1. git log .... in an independant step
2. sleep 3 deliberately

* build: release ver-3.3.6.0

* chore(ci): comment and remove unnecessary code

* refactor(ci): cleanup tools/CMakeLists.txt
1. remove unnecessary or uneffective codes
2. better approach to pass compile flags to go

* chore(cmake): remove unnecessary code

* merge(cmake): porting linux

* merge(cmake): porting macOS

* merge(cmake): porting windows

* merge(cmake): pitfall fix: use literal format string instead

* merge(cmake): ver-3.3.6.0 for both taosadatper and taosws

* merge(cmake): adjust ci for debug purposes

* merge(cmake): add dependenices to make sure native built

* merge(main): link os

* merge(main): revert to main

* chore(cmake): remove freemine'd comments

* chore(cmake): make github ci to run with empty change

* refactor(cmake): remove unnecessaries

* refactor(cmake): remove unnecessaries

* chore(cmake):
1. remove taosudf in build.sh::do_purge
2. packaging taosBenchmark

* chore(cmake):
1. typo corrections
2. add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to bypass compatibility check imposed by latest cmake since 4.0

* chore(cmake): show cmake version

* chore(cmake): add -DCMAKE_POLICY_VERSION_MINIMUM=3.5 to bypass compatibility check imposed by latest cmake since 4.0

* chore(cmake): typo

* chore(cmake): remove unnecessaries

* cmake: debug enterprise compilation

* refactor: preparatory work before merging the freemin PR

* refactor(cmake): revert to d91205594a

* refactor(cmake): optimize by adjusting cache key

* chore(cmake): make github ci to run with empty change

* refactor(cmake): move libs3 and azure to external.cmake

* refactor(cmake): porting macOS

* refactor(cmake): move cos to external.cmake

* refactor(cmake): bypass s3/azure/cos on windows for the moment

* refactor(cmake): try find_package(Threads)

* refactor(cmake): add dl/pthread in libs3

* refactor(cmake): enable s3/azure for the moment, hard-coded

* refactor(cmake): adjust BUILD_WITH_ANALYSIS option

* refactor(cmake):
1. enum AVROTYPE => enum enAVROTYPE
2. remove INSTALL(CODE ...) from tools/taos-tools/CMakeLists.txt

* refactor(cmake): portint macOS and typo

* refactor(cmake): ticking external cache key to force refresh external building

* refactor(cmake): remove obsoletes

* refactor(cmake): remove obsoletes

* refactor(cmake): remove obsoletes, contrib/deps-download related

* refactor(cmake): adding build.bat, and use build.bat in .github/workflows/tdengine-build.yml

* refactor(cmake):
1. call .\build.bat
2. remove POST_BUILD which is not valid in this signature of add_custom_command

* refactor(ci): use 'NMake Makefiles JOM'

* refactor(ci): typo

* refactor(ci): debug

* refactor(ci): debug

* refactor(ci): debug

* refactor(ci): use call on windows

* refactor(cmake): move utils/TSZ to contrib/TSZ

* chore(cmake): make github ci to run with empty change

* refactor(cmake): mv cmake.* -> *.cmake and related in CMakeLists.txt

* refactor(cmake): debugging integrating community

* refactor(cmake): debugging integrating community

* chore(cmake): make github ci to run with empty change

* refactor(cmake): revert geos.git back to 3.12.0

* refactor(cmake): revert doc/** to main branch

* refactor(cmake): revert tests/** to main branch, except tests/taosc_test/CMakeLists.txt

* refactor(cmake): revert test/cases/** to main branch

* refactor(cmake): revert test/docs/** to main branch

* refactor(cmake): revert tools/taos-tools/deps/{avro,jansson,libargp,snappy,zlib-1.2.13} to main branch, which need to be removed later

* refactor(ci): revert .github/workflows/{deploy-case-docs.yml,tdengine-doc-build.yml} to main branch

* refactor(cmake): revert tcommon.h and related to main branch

* refactor(cmake): revert test/* stuffs back to main branch

* refactor(cmake): revert libuv.git back to v1.49.2

* refactor(cmake): revert .h/.c and related to main branch

* refactor(cmake): typo format string

* refactor(cmake): typo correction, incorrect use of macro `assert`

* refactor(cmake): adding /wd4028 to bypass thttp.c under enterprise build for the moment

* refactor(cmake):
1. adding some rules to .gitignore for the last resort to save
2. adding source files manually to source/libs/parser/CMakeLists.txt to exclude potential sql.c generated by old build system

* refactor(cmake): add -Wno-unused-result for the moment

* refactor(cmake): adjust target dependencies

* refactor(cmake): tweaking building of mxml

* refactor(cmake): adjust externals CMAKE_ARGS to save compilation time

* refactor(cmake): temporarily use tomchon-patch-3

* refactor(cmake): temporarily adding dumpbin directives

* refactor(cmake): adding NODEFAULTLIB temporarily for debugging purpose

* refactor(cmake): adding NODEFAULTLIB:libcpmt temporarily for debugging purpose

* refactor(cmake): adding NODEFAULTLIB:cpmt temporarily for debugging purpose

* chore(cmake): make github ci to run with empty change

* chore(cmake): make github ci to run with empty change

* refactor(ci): use freemine.debug branch for the moment

* refactor(ci): revert back to use run-tests-on-windows.yml of tomchon-patch-3 branch

* chore(cmake): make github ci to run with empty change

* refactor(cmake): remove temporary debug directives

* refactor(cmake): move cmake_policy(..CMP0135..) to the root of this project

* refactor(cmake): Remove temporary debugging conditional and restore original logic

* chore(cmake): adding -B in army tests

* chore(cmake): make github ci to run with empty change

* chore(cmake): revert: adding -B in army tests

* refactor(cmake):
1. use ver-3.3.6.3 of taosadapter
2. pitfall fix: do not link with taosws for taos shell

* refactor(cmake): revert taosadapter to ver-3.3.6.0

* chore(cmake): adding logging message in more detail

* chore(cmake): adding logging message in more detail

* chore(ci): debugging by using ls -l

* chore(ci): typo

* chore(ci): typo

* refactor(cmake): revert to remove debug code, and taosadapter to ver-3.3.6.3

* refactor(cmake): remove DEP_ext_taosws

* chore(cmake): make github ci to run with empty change

* chore(cmake): make github ci to run with empty change

* chore(cmake): make github ci to run with empty change

* chore(cmake): make github ci to run with empty change

* chore(cmake): typo correction: utils/test/c/sml_test.c

* debug(cmake): test cases debug

* chore(cmake): make github ci to run with empty change

* chore(ci): using tomchon-patch-3 branch for run-tests-on-linux for the moment

* chore(cmake): make github ci to run with empty change

* debug(test): revert utils/test/c/sml_test.c back to tdengine/main branch

* debug(test): adding VERBOSE=1 and gcc --version

* debug(test): revert tests/parallel_test/cases.task back to tdengine/main branch

* debug: add log in run_case.sh

* fix: test log

* fix: docker run privileged

* tes: add debug test in  cases.task

* refactor(cmake): remove ref to \contrib\<dep-name> and ~/.cos-local-{1,2} from CMakeLists.txt(s)

* refactor(cmake): make taosudf link with libtaos.so

* chore(cmake): fix compiler warning as error problem

* refactor(cmake): revert 7d7f03832d

* chore(cmake): only udf.sim and udfpy.sim for the moment

* debug(ci): debug udf.sim for the moment

* debug(ci): debug udf.sim for the moment

* debug(ci): debug udf.sim for the moment

* debug(ci): debug only udf.sim for the moment

* debug(ci): debug only udf.sim for the moment, using tee

* debug(ci): debug only udf.sim for the moment, revert using tee

* debug(ci): debug only udf.sim for the moment, adding TAOS_LIB again

* debug(cmake): adding -rdynamic for the moment

* debug(cmake): re-adding udfpy.sim

* debug(cmake): do NOT run with Sanitizer for udfpy.sim for the moment

* chore(cmake): revert cases.task, and remove ubuntu-20.04 from tdengine-build.yml

* refactor(cmake): using ENABLE_EXPORTS to export symbols in taosudf

* chore(cmake): make github ci to run with empty change

* chore(cmake): remove stuffs for debug purpose and revert

---------

Co-authored-by: support-platform <it@taosdata.com>
Co-authored-by: Simon Guan <slguan@taosdata.com>
Co-authored-by: Simon Guan <guanshengliang@qq.com>
Co-authored-by: haoranchen <haoran920c@163.com>
2025-04-19 14:03:29 +08:00
WANG MINGMING
ca724865b9
fix(stream): parName is null (#30592) 2025-04-19 10:26:32 +08:00
Zhixiao Bao
e25da1cd35
fix: taosd fails to start after modifying the timezone configuration parameter. 2025-04-17 19:19:05 +08:00
dongming chen
c38f9eda0c
fix: TS-6336-get-epset (#30770) 2025-04-15 16:44:58 +08:00
dongming chen
d9b7511b0e
fix: [TS-6245]add-protection-when-drop-db (#30567) 2025-04-12 09:47:11 +08:00
dongming chen
f04b57dc9e
fix: [TS-6245]-add-trans-timeout (#30740) 2025-04-12 09:46:59 +08:00
dongming chen
b2fec10beb
fix[TS-6166]: when the transaction to create a database fails, repeatedly sending messages can cause high CPU usage.(#30724) 2025-04-10 20:41:38 +08:00
freemine
9897d85f86
enh(test): adding TD_ALWAYS_ASSERT macro for test cases to use (#30739) 2025-04-10 16:52:55 +08:00
Zhixiao Bao
34a41bf54c
fix: crash while upgrade taosd to 3362. (#30712) 2025-04-09 18:53:47 +08:00
dongming chen
8710e539a0
fix: [TD-34436] double-release (#30668) 2025-04-09 14:50:20 +08:00
dongming chen
712cb6652b
fix: TS-6276-grant-format (#30701) 2025-04-09 14:40:24 +08:00
dongming chen
c5c86af7b5
fix/TS-6226-send-check-when-not-sync (#30588) 2025-04-09 11:18:53 +08:00
Simon Guan
de5f5866a8
fix[TS-6276]: when the authorization statement exceeds a certain length, it cannot be displayed in full (#30657) 2025-04-07 09:21:13 +08:00
dongming chen
915a6e5609
fix: [TD-34250] wrong-createdb (#30555) 2025-04-01 10:02:05 +08:00
Zhixiao Bao
c6216a02ab
fix(s3): wrong condition where alter db ops s3_compact. (#30568) 2025-04-01 09:19:12 +08:00
WANG MINGMING
1c4507ab75
fix(stream): row size error (#30330) 2025-03-28 18:10:57 +08:00
Simon Guan
9a89db55ca
enh: allow ordinary users to execute the “show anodes” command. (#30464) 2025-03-25 20:17:16 +08:00
dongming chen
13475700c5
enh: [TD-34190] remove-wrong-log (#30408) 2025-03-25 12:37:20 +08:00
Haojun Liao
af06bea439
fix(stream): add start/stop message. (#30400) 2025-03-24 19:38:00 +08:00
Haojun Liao
cfcdfdd2e7
fix(stream): update the update node info. (#30364) 2025-03-24 10:18:06 +08:00
Haojun Liao
82ffeb82a9
fix(gpt): fix the bug for algo name case sensitive (#30362) 2025-03-23 13:20:24 +08:00
Simon Guan
e5a1539c6d
Merge pull request #30307 from taosdata/fix/TS-5467-3.0
fix(stream): fix stream processing for virtual tables
2025-03-20 22:24:45 +08:00
liuyao
f7beaeebfe
ci: adjust test && fix issue
ci: adjust test && fix issue
2025-03-20 17:54:20 +08:00
Zhixiao Bao
0b8bff92c6
enh: add keep option while show stable. (#30294) 2025-03-20 15:30:22 +08:00
Haojun Liao
cd10173454
fix(gpt): fix error in gpt (#30275)
* fix(stream): support packaging enterprise edition.

* feat(gpt): support lstm and do some internal refactor, add sample autoencoder model.

* feat(gpt): support lstm and do some internal refactor, add sample autoencoder model.

* test(gpt): disable model case.

* test(gpt): disable model case.

* doc: fix title error in doc.

* doc: add mlp doc.

* fix(gpt): add gpt

* fix(gpt): update the test cases.

* fix(gpt): update the test cases.

* test(gpt): disable gpt test in github action.

* fix(gpt): check the grant, check the rsp code.

* fix(gpt): limit the forecast rows.

* fix(gpt): use default gpt service

* fix(gpt): update error code.
2025-03-20 15:18:58 +08:00
Jinqing Kuang
c82398e175 fix(stream): fix stream processing for virtual tables
- Fixed status check in virtual table stream processing
- Fixed multi-way merge logic for super virtual tables
- Fixed generation of target subtable names during merge
2025-03-20 01:15:54 +08:00
Simon Guan
d02196a231
refactor: adjust some logs (#30245)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* docs: update keywords

* fix: compile errors in windows

* docs: minor changes

* docs: minor changes

* refactor: minor changes

* docs: fminro changes

* docs: format

* refactr: remove uncessory logs

* docs: format doc
2025-03-18 17:55:36 +08:00
Simon Guan
ddc05f168e
refactor: adjust the log to track the data writing process on the server via QID (#30195)
* refactor: adjust wal log infos

* refactor: adjust some log's level

* refactor: adjust some log's level

* fix: add traceId for sync module

* refactor: adjust log level

* refactor: adjust log level

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* refactor: adjust log levels

* fix: compile errors in windows

* refactor: minor changes
2025-03-18 16:21:44 +08:00
WANG MINGMING
3bc6635de3
feat[TS-6115]: support deletion of topics with active consumers. (#30232)
* feat[TS-6115]: drop topic & consumer group in force

* feat[TS-6115]: drop topic & consumer group in force

* feat[TS-6115]: drop topic & consumer group in force
2025-03-18 14:11:00 +08:00
Simon Guan
cf8d7cd0ac Merge branch 'main' into merge/mainto3.0 2025-03-17 21:42:42 +08:00
Yihao Deng
2151c5cae0
update telemInterval (#29703) 2025-03-17 17:42:24 +08:00
Haojun Liao
dfe32225de chore: merge main. 2025-03-17 14:51:11 +08:00
Haojun Liao
ab92886820
fix(stream): reduce the consensus checkpoint id trans. (#30105)
* fix(stream): reduce the consensus checkpoint id trans.

* refactor(stream): add some logs.

* refactor(stream): set the max checkpoint exec time 30min.

* refactor(stream): add checkpoint-consensus trans conflict check.

* refactor(stream): remove unused local variables.

* fix(stream): fix syntax error.

* fix(stream): 1. fix free memory error 2. continue if put result into dst hashmap failed.

* fix issue

* fix issue

* fix(mnd): follower mnode not processes the timer event.

* fix(stream): print correct error msg.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): truncate long subtable name

* fix(stream): add buffer len.

* refactor(stream): update some logs.

* fix issue

* refactor(stream): update some logs.

* refactor(stream): update some logs.

* fix(stream): check return value.

* fix(stream): fix syntax error.

* fix(stream): check return value.

* fix(stream): update the timer check in mnode.

* fix(stream): add restart stage tracking.

* fix(stream): track the start task stage for meta.

* fix(stream): fix error in log.

* refactor(stream): adjust log info.

* fix mem issue

* fix(stream): check the number of required tasks for consensus checkpointId.

* fix(stream): lock the whole start procedure.

* fix(stream): add lock during start all tasks.

* fix(stream): update logs.

* fix(stream): update logs.

* fix(stream): update logs.

* fix(stream): fix dead-lock.

* fix(stream): fix syntax error.

* fix(stream): not drop the scan-history task.

* fix(stream): fix syntax error.

* fix(stream): wait for executor stop before restarting.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): add some logs.

* fix(stream): disable some logs.

* fix(stream): reset the start info if no task left.

---------

Co-authored-by: 54liuyao <54liuyao@163.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
2025-03-17 10:20:17 +08:00
Jinqing Kuang
ac23a4fc55
feat(stream): support stream processing for virtual tables (#30144)
* enh: add client processing

* enh: add mnode vtables processing

* enh: add mnode vtable processing

* enh: add normal child vtable support

* fix: compile issues

* fix: compile issues

* fix: create stream issues

* fix: multi stream scan issue

* fix: remove debug info

* fix: agg task and task level issues

* fix: correct task output type

* fix: split vtablescan from agg

* fix: memory leak issues

* fix: add limitations

* Update 09-error-code.md

* Update 09-error-code.md

* fix: remove usless case

* feat(stream): extract original table data in source scan task

Implemented functionality in the source task to extract data
corresponding to the virtual table from the original table using WAL.
The extracted data is then sent to the downstream merge task for further
processing.

* feat(stream): multi-way merge using loser tree in virtual merge task

Implemented multi-way merge in the merge task using a loser tree to
combine data from multiple original table into a single virtual table.
The merged virtual table data is then pushed downstream for further
processing.  Introduced memory limit handling during the merge process
with configurable behavior when the memory limit is reached.

* fix(test): remove useless cases

---------

Co-authored-by: dapan1121 <wpan@taosdata.com>
Co-authored-by: Pan Wei <72057773+dapan1121@users.noreply.github.com>
2025-03-15 17:06:48 +08:00
Jing Sima
410324746b
feat:[TS-4897] virtual table (#30098)
* feat: [TS-4897] Support create/drop/alter/show/describe vtable

* feat: [TS-4897] Support vtable's query

* feat: [TS-4897] Support create virtual supertable

* feat: [TS-4897] Support explain analyze / where / count(*) and only select ts of vtable.

* feat: [TS-4897] Add create test and fix bugs

* feat: [TS-4897] Add alter/drop test and fix bugs

* feat: [TS-4897] Add describe/show test and fix bugs

* feat: [TS-4897] Add auth test and fix bugs

* feat: [TS-4897] Fix meta/catalog/cache bugs

* feat: [TS-4897] Support select tag from virtual child table

* feat: [TS-4897] Add select test and fix plenty of bugs

* feat: [TS-4897] Add optimize rule for vtable scan / support create vtable cross database / remove enterprise constraint / fix bugs.

* feat: [TS-4897] Fix 'schema is old'

* feat: [TS-4897] Support virtual stable query

* feat: [TS-4897] Add tests and Fix bugs

* feat: [TS-4897] resolve conflict.
2025-03-15 14:10:46 +08:00
liuyao
2de6736160
feat(stream): add new trigger continuous_window_close (#30125)
* opt stream build twa result

* opt force window close memory

* feat(stream):optimize new interval and scan operator

* adj log

* opt code

* opt code

* fill history

* fix issue for fill history

* add ci

* feat(stream): add new stream nonblock interval operator

* adjust code

* use new scan operator

* use new scan operator

* add log

* fix issue

* recover stream scan next function

* fix issue

* fix issue

* fix issue

* ignore disorder data

* fix issue

* fix issue for interval sliding

* fix issue

* fix ci issue

* fix ci issue

* fix ci issue

* add semi && final nonblock interval operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* refactor(stream): track the msgId for each upstream tasks.

* fix(stream): fix race condition.

* fix(stream): update the task last msgId when putting into input queue succ.

* fix issue

* fix issue

* put recalculate data to rocksdb

* fix issue

* fix issue

* enh(query)[TD-33071]: add support for saving and restoring tsdbReader scan progress

- Implement functionality to save scan progress during tsdbReader operations
- Enable resuming scans from the last saved position

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* add rec interval check

* enh(stream):add recalculate tasks.

* enh(stream): support the re-calculate the tasks.

* fix issue && do refactor

* do refactor

* fix issue

* fix issue

* update backend opt

* add new interface

* add new session operator

* support blob

* add new session operator

* fix issue

* add rec state for rec task

* fix invalid read

* add new session window

* enh(stream): update the stream tasks backend.

* new session operator

* add pull data

* fix(stream): fix error in expand stream backend.

* fix issue

* fix issue

* fix issue

* merge code

* fix issue

* fix(stream): check for null ptr.

* fix(stream): add more check.

* fix issue

* fix issue

* fix issue

* add debug code

* fix issue

* fix issue

* fix issue

* set rec end flag

* fix(stream): opt re-calculate stream tasks.

* fix issue

* fix issue

* add new operator

* enh(stream): dispatch recalculate block to agg tasks.

* fix issue

* fix issue

* fix(stream): adjust filter.

* fix issue

* refactor

* refactor(stream): adjust the recalculate end block.

* fix issue

* fix(stream): set correct create trigger block.

* fix issue

* fix(stream): fix error in build recalculate end block.

* fix(stream): check null ptr.

* add stream client && fix issue

* fix mem leak

* fix(stream): free msg.

* add stream client

* fix(stream): fix error.

* add stream client && fix issue

* add stream client

* refactor(stream): set the recalculate task complete.

* add wend and group_id for session window dest table

* feat(stream): refactor and set the recalcul agg task complete.

* add cfg for adapter

* fix issue

* add state && event operator

* feat(stream): support fill-history task.

* add new fill operator

* fix(stream): set correct backend when updating fill-history task to recalculate task.

* add new fill operator

* fix(stream): s2025-03-06 11:10:31.272
et ops always open in scan tsdb

* fix(stream):set the correct taskType for sink task.

* new fill operator

* adj stream fill operator

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* fix issue

* adj test

* fix issue

* fix(stream): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(steam): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix(stream): fix issue

* fix: ut com error

* fix(stream): fix mem leak and adjust operator type check rule

* fix(stream): fix mem leak and adjust test case

* refactor code

* fix(stream): free items.

* fix(stream): free fix memory leak.

* fix(stream): fix syntax error.

* fix: ignore unexpect block

* fix: adjust op type

---------

Co-authored-by: Haojun Liao <hjliao@taosdata.com>
Co-authored-by: Jinqing Kuang <kuangjinqingcn@gmail.com>
Co-authored-by: yihaoDeng <luomoxyz@126.com>
2025-03-14 20:14:01 +08:00