wangmm0220
4dd226b1a9
fix(stream): optimize val scan logic
2025-07-14 17:14:59 +08:00
Bomin Zhang
876979b5f5
feat[ts-6107]: shared storage ( #31552 )
...
* add API to use s3 as shared storage
* support using local file system as shared storage
* upload file to shared storage
* support read, compact and drop
* finish basic mnode & vnode msg processing
* follower sync migration state
* implement mnode transaction, and improve log
* send migration progress msg to dnode to avoid deadlock
* implement following migration
* remove mcount
* avoid redo migration on startup
* avoid follower deadlock when leader is down
* trigger migrate by timer, avoid compact after migration
* comment out the usage of 'tcs' functions in stream
* change config item prefix from s3 to ss
* change db option prefix from s3 to ss
* rename s3 data struct, function, file to ss
* rename s3 macro to ss
* update s3 sql to ss
* rename remaining s3 items to ss
* check ss configruation, improve s3 retry
* grant object storage -> shared storage, check ssEnabled
* fix memory leaks
* update build options
* omit sensitive information when dump config
* fix backward compatibility issue
* fix issues found in ci-checks
* fix some failed test cases
* avoid follower timeout and improve log
* fix: follower timeout because migration status not updated
* refuse migration if there's an in progress one
* fix ss test case
* remove garbage files and other minor improvement
* fix failed test cases
* update unit test
* fix failed test case
* fix failed test case
* update document
* update document and fix failed test cases
* fix minor issues in code, test and document
* check new commit after migration task is scheduled
* fix several issus
1. migrate information cannot be dropped sometimes because progress response was put into read queue.
2. memory leak in rare cases
3. data corruption in rare cases
4. failed test case
* add shared storage upgrade tool
* fix compile error
2025-07-14 16:33:53 +08:00
wangmm0220
0772b9f6c5
fix(stream): optimize val scan logic
2025-07-14 16:33:37 +08:00
wangmm0220
9d29d8ca60
fix(stream): tmq error
2025-07-14 16:12:41 +08:00
wangmm0220
3cdd41b27f
fix(stream): conflicts
2025-07-14 14:46:08 +08:00
wangmm0220
dcc4a6d1de
fix(stream): opti wal interface
2025-07-14 14:40:59 +08:00
WANG MINGMING
a051a73a62
Merge pull request #31857 from taosdata/feat/TS-6100-3.0-mm
...
Feat/ts 6100 3.0 mm
2025-07-14 14:40:10 +08:00
wangmm0220
a3de606eda
Merge remote-tracking branch 'origin/feat/TS-6100-3.0' into feat/TS-6100-3.0-mm
2025-07-14 11:27:00 +08:00
wangmm0220
b86d0959f5
fix(stream): scan wal with schema in that version
2025-07-14 11:26:31 +08:00
facetosea
0e0e4a2b1c
fix: error code check
2025-07-14 11:17:37 +08:00
Jinqing Kuang
69dd276f7f
Merge branch '3.0' into feat/TS-6100-3.0
2025-07-14 09:39:13 +08:00
facetosea
86cc038981
fix: gen crash
2025-07-12 16:21:07 +08:00
wangmm0220
35878f4b4a
fix(stream): memory leak
2025-07-12 16:19:43 +08:00
Kaili Xu
dc2b021778
feat: support mount data path ( #31677 )
2025-07-12 15:39:29 +08:00
WANG MINGMING
753624fa03
fix(tmq): [TS-6569]tdb error if write tmq meta data in multi thread ( #31808 )
2025-07-12 13:43:53 +08:00
wangmm0220
4145df030d
feat(stream): fix delete msg error
2025-07-11 17:39:51 +08:00
Zhixiao Bao
95cf8b751a
fix: set lcn before do s3 migrate. ( #31782 )
2025-07-11 16:24:46 +08:00
Kaili Xu
61ba6634ad
fix: overflow check in snprintf ( #31780 )
2025-07-11 09:08:47 +08:00
Tony Zhang
83b0fb43dc
fix: taosd crush in query when insufficient memory ( #31746 )
...
Co-authored-by: Tony Zhang <tonyzhang@taosdata.com>
2025-07-11 08:11:57 +08:00
wangmm0220
9b8b23b378
feat(stream): delete msg from wal
2025-07-10 19:40:00 +08:00
wangmm0220
25dd06bc50
feat(stream): fix tmq error
2025-07-10 14:20:35 +08:00
dongming chen
d7c25bba93
enh: add log for snapshot ( #31681 )
2025-07-10 09:48:18 +08:00
Pan Wei
baf6a52153
Merge branch '3.0' into feat/TS-6100-3.0
2025-07-10 08:25:34 +08:00
wangmm0220
7b341af2b3
fix(stream): cid error when scan wal
2025-07-09 10:28:36 +08:00
Mario Peng
808d720c1d
feat:insert into subquery ( #31401 ) ( #31710 )
2025-07-08 17:26:34 +08:00
wangmm0220
7ec5dd8990
fix(stream): set table scan operator param in vtable scan & fix filter error
2025-07-08 16:47:43 +08:00
Simon Guan
d497a960eb
Merge branch 'main' into merge/mainto3.0
2025-07-08 16:35:39 +08:00
wangmm0220
30c692309c
Merge remote-tracking branch 'origin/feat/TS-6100-3.0' into feat/TS-6100-3.0-mm
2025-07-08 14:30:25 +08:00
wangmm0220
9da567453a
fix(stream): set table scan operator param in vtable scan & fix filter error
2025-07-08 14:30:18 +08:00
facetosea
ce6e15a74d
fix: log level
2025-07-08 10:58:10 +08:00
Jinqing Kuang
d932cfcd88
fix(stream): fix hist calc of stream trigger task
2025-07-08 10:27:35 +08:00
Pan Wei
489990a787
fix: memory issues ( #31700 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
* fix: memory issues
* fix: memory leak issues
* fix: recalculate time range issue
* fix: add debug log
* fix: memory issues
* fix: enable case asan
* Update streamlist_for_ci.task
* fix: case asan issue
* fix: stream name issue
* fix: external window compile issues
* fix: deploy memory issue
* fix: ahandle issue
* fix: ahandle issue
* fix: ahandle issue
* fix: virtual table reader list issue
* fix: log info
* fix: msg error
* fix: virtual table addr list issue
* fix: memory issues
* fix: memory leak issue
* fix: memory issues
* fix: memory free issues
* fix: memory issues
* fix: snode deploy issue
* fix: mnode reader issue
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-08 09:32:12 +08:00
facetosea
083db4f837
fix: log level
2025-07-08 09:29:12 +08:00
Haojun Liao
4abc6c071d
fix(query): fix bug caused by shallow copy. ( #31638 )
2025-07-08 08:49:55 +08:00
Jinqing Kuang
968eeeeb0b
Merge branch '3.0' into feat/TS-6100-3.0-merge
2025-07-07 18:04:02 +08:00
wangmm0220
7dcc4d5292
fix(stream): release tsdb reader lock
2025-07-07 15:22:45 +08:00
wangmm0220
ed0aea6918
fix(stream): double free of notify content and timestamp not the firset col when getting tsdbmeta in reader
2025-07-05 14:17:52 +08:00
wangmm0220
4c4c3a8652
fix(stream): fix reset error in vtablescanoperator & modify log
2025-07-04 18:10:32 +08:00
Yihao Deng
f142278ec7
use no-default-index ( #31623 )
2025-07-04 14:16:26 +08:00
Jing Sima
a7adc8042f
feat: [TS-6100] Support TSMA: Part 1.
2025-07-04 13:31:48 +08:00
facetosea
31ff18a0be
fix: mem leak
2025-07-03 17:54:09 +08:00
Kaili Xu
a86e09e4e3
enh: grant support vnodes and storage size ( #31380 )
2025-07-03 16:44:29 +08:00
facetosea
b57ce4f4b4
fix: mem leak
2025-07-03 15:56:15 +08:00
WANG MINGMING
68d94bb85c
Merge branch 'feat/TS-6100-3.0' into feat/TS-6100-3.0-mm
2025-07-03 15:05:11 +08:00
wangmm0220
da400252e8
feat(stream): fix sarray init error in vTableInfo & optimize getting tsdb meta info for vtable
2025-07-03 14:59:30 +08:00
dongming chen
68f7094041
fix/TS-6785-fix-return-code ( #31593 )
2025-07-03 14:22:12 +08:00
facetosea
9bf99b0e51
log: format
2025-07-03 13:40:50 +08:00
Pan Wei
f6ab877e05
fix: case memory issues ( #31621 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
* fix: memory issues
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-03 13:26:22 +08:00
Simon Guan
fc2fec02c7
merge: from main to 3.0 branch ( #31562 )
2025-07-03 09:15:59 +08:00
wangmm0220
827db81912
feat(stream): fix per_filter disable & modify logic of getting tsdbmeta for vtable
2025-07-02 18:53:57 +08:00
Pan Wei
4270f35fb9
fix: crash and memory issues ( #31588 )
...
* enh: add operator reset func
* fix: merge join reset issue
* fix: memory issues
* fix: add debug assert
* fix: memory issues
* fix: memory leak
* fix: memory issues
* fix taos log miss
* fix: case issue
* fix: case issue
* fix: case issues
* fix: drop dnode issue
* fix: memory issues
---------
Co-authored-by: huohong <sallyhuo@taosdata.com>
2025-07-02 10:14:55 +08:00
facetosea
cf2a932034
fix: get vginfo
2025-07-01 23:06:56 +08:00
wangmm0220
dd8665a582
feat(stream): add skey/ekey/order for trigger
2025-07-01 17:34:00 +08:00
wangmm0220
138c2e41c3
feat(stream): fix coredump if do not set colId for triggerBlock with pseduCols
2025-07-01 11:20:09 +08:00
wangmm0220
6f9077f95f
feat(stream): add log
2025-07-01 10:17:07 +08:00
WANG MINGMING
3e0e339985
feat(stream): fix thread run out for runner ( #31559 )
2025-07-01 08:43:11 +08:00
facetosea
1064dbe38a
fix: fill
2025-06-30 17:34:39 +08:00
xiao-77
fee17fa423
fix: can not open vnodes while change sub dataDir.
2025-06-30 17:07:28 +08:00
wangmm0220
5610088604
fix(stream): add log
2025-06-30 14:13:54 +08:00
hongzhenliu
99b30bf096
fix(query) subscriptions memory leak ( #31533 )
...
Co-authored-by: hongzhenliu335 <ethanliu@taosdata.com>
2025-06-30 13:55:57 +08:00
Zhixiao Bao
62d8b35fd0
feat: add write metrics. ( #31198 )
2025-06-30 10:33:34 +08:00
wangmm0220
602de9eb87
feat(stream): fix datablockId not match in filter for calc
2025-06-30 10:21:50 +08:00
wangmm0220
209922b7c4
feat(stream): fix rtInfo is null when fetch data in reader
2025-06-28 15:46:33 +08:00
wangmm0220
c0de4b1a73
feat(stream): fix logic error
2025-06-28 15:24:08 +08:00
wangmm0220
272e7e476b
feat(stream): fix error if calc data
2025-06-28 14:25:29 +08:00
wangmm0220
564d585619
feat(stream): fix err that no rtInfo in stream fetch
2025-06-28 11:47:31 +08:00
wangmm0220
8858ebc250
feat(stream): optimize code
2025-06-28 10:02:32 +08:00
wangmm0220
6b08a4edd6
feat(stream): fix condition if %%trows
2025-06-27 19:37:02 +08:00
wangmm0220
c15deb10fe
feat(stream): fix condition if %%trows
2025-06-27 19:32:26 +08:00
dapan1121
b6e07d7cb5
fix: reader crash issue
2025-06-27 13:44:23 +08:00
wangmm0220
5bf462d11f
feat(stream): fix err if non table scan in calc plan
2025-06-26 15:58:08 +08:00
wangmm0220
01716fa4a0
feat(stream): last ver in wal
2025-06-26 15:42:29 +08:00
wangmm0220
5b60179ca1
feat(stream): modify blockIs in calcaCachaePlan & ver
2025-06-26 13:43:00 +08:00
Hongze Cheng
7508267906
enh: adjust log level
2025-06-26 09:38:45 +08:00
dapan1121
eb8150b6fc
Merge remote-tracking branch 'origin/feat/TS-6100-3.0.pw9' into feat/TS-6100-3.0
2025-06-26 07:58:03 +08:00
dapan1121
a42ed39535
enh: undeploy
2025-06-26 07:57:17 +08:00
wangmm0220
253f65b4f0
feat(stream): fix coredump when processing tag in reader
2025-06-25 19:07:12 +08:00
Yihao Deng
a4909ebe29
fix index perf ( #31474 )
2025-06-25 15:00:11 +08:00
dongming chen
73261908b5
fix: implement retry mechanism for vnodeProcessWriteMsg in vnodeApply… ( #31388 )
2025-06-24 16:53:11 +08:00
wangmm0220
81c31ad555
fix(stream): checkpoint
2025-06-24 15:08:40 +08:00
Jinqing Kuang
85fad8e178
Merge branch '3.0' into temp
2025-06-23 18:08:09 +08:00
wangmm0220
8669fd7863
fix(stream): interval reset error & checkpoint
2025-06-23 17:10:38 +08:00
wangmm0220
fb281e55eb
fix(stream): send no data if wal is end
2025-06-21 19:29:12 +08:00
Jinqing Kuang
652b6b52b4
fix(stream): fix rsp dangling pointer in wal meta rsp
2025-06-21 15:25:19 +08:00
Mario Peng
c65e1592fa
feat:insert into subquery ( #31401 )
2025-06-21 15:11:09 +08:00
wangmm0220
0c6a6c83a3
fix(stream): lastver in wal meta
2025-06-21 10:20:26 +08:00
wangmm0220
b09998b6a0
fix(stream): lastver in wal meta
2025-06-21 09:35:00 +08:00
wangmm0220
f30938ed1d
fix(stream): get tRowGet error
2025-06-20 19:04:56 +08:00
Simon Guan
9923aedcb6
Merge branch 'main' into merge/mainto3.0
2025-06-20 14:48:16 +08:00
wangmm0220
458cca1a98
fix(stream): groupid error
2025-06-20 13:39:25 +08:00
Simon Guan
441a2472d7
Merge branch '3.0' into merge/3.0tomain2
2025-06-19 18:26:02 +08:00
wangmm0220
0f4a4436fa
fix(stream): set calc reader running
2025-06-19 10:12:06 +08:00
wangmm0220
0cd27abc71
fix(stream): pWin is NULL in external window
2025-06-18 20:11:55 +08:00
facetosea
3a67777c14
fix: normal table exist
2025-06-18 19:22:39 +08:00
facetosea
061137c11a
fix: data insert suid error
2025-06-18 17:05:06 +08:00
Hongze Cheng
d9e56a61e4
enh: make info to debug
2025-06-18 16:45:29 +08:00
facetosea
b15d94fd90
fix: datainserter cache
2025-06-18 13:56:27 +08:00
Minglei Jin
082c1ebfa9
relean mutex when returning ( #31389 )
2025-06-18 09:40:14 +08:00
Minglei Jin
4ddfb76cb9
relean mutex when returning ( #31390 )
2025-06-18 09:40:03 +08:00
wangmm0220
b44770074a
fix(stream): only one ts condition in calc fetch like ts <= _twstart
2025-06-17 19:59:38 +08:00
wangmm0220
a288f3640b
fix(stream): ts condition error
2025-06-17 14:37:56 +08:00
Hongze Cheng
696d29d916
Merge branch 'main' of https://github.com/taosdata/TDengine into fix/TS-6672-taosd-main
2025-06-17 10:37:02 +08:00
facetosea
e1bb2a1d21
fix: table already exist, set response
2025-06-16 20:27:57 +08:00
Hongze Cheng
3e68b4cdd4
enh: add more check of submit message
2025-06-16 14:54:17 +08:00
Simon Guan
2838d2f063
Merge branch '3.0' into merge/mainto3.0
2025-06-14 10:41:12 +08:00
Hongze Cheng
8924124917
fix: check column number when alter table
2025-06-12 16:04:18 +08:00
wangmm0220
867625ff4b
fix(stream): add logic of %%tbname & checkpoint
2025-06-06 19:43:44 +08:00
Haojun Liao
e725e4ab24
fix(gpt): update the timeout parameter. ( #31267 )
2025-06-06 17:22:46 +08:00
Mario Peng
cc3fabdc08
fix:tsdb select error ( #31300 )
2025-06-06 15:09:54 +08:00
dapan1121
33ca255671
Merge remote-tracking branch 'origin/feat/TS-6100-3.0' into feat/TS-6100-3.0.pw9
2025-06-06 09:15:50 +08:00
dapan1121
bfce943ff0
enh: add more check
2025-06-06 09:09:40 +08:00
wangmm0220
4572c0243d
fix(stream): add logic for %%tbname
2025-06-05 17:26:09 +08:00
wangmm0220
f7eb3328c2
fix(stream): print block
2025-06-05 09:48:16 +08:00
dongming chen
87c969b168
fix: [TD-35276] add-log-apply-internal-error ( #31274 )
2025-06-04 17:24:49 +08:00
wangmm0220
3a82e07bc4
fix(stream): data error for external window & add pesudo function & multi instance for calc reader
2025-06-04 14:09:47 +08:00
wangmm0220
e7c84442dc
fix(stream): add get tsdb/wal interface for vtable
2025-06-03 14:09:25 +08:00
wangmm0220
db29960716
fix(stream): add get last/first ts interface for vtable
2025-05-30 18:54:18 +08:00
facetosea
734e09a22f
fix: table uid hash cache
2025-05-29 11:30:28 +08:00
wangmm0220
3a38149af6
fix(stream): add get OTableInfo interface
2025-05-28 19:41:10 +08:00
wangmm0220
e6dfbf6295
fix(stream): add interface of get vtable info
2025-05-28 18:34:19 +08:00
Hongze Cheng
98e23733ba
fix: add duplicate tolerance ( #31217 )
2025-05-28 16:54:48 +08:00
wangmm0220
1661db33b3
fix(stream): processTag coredump in walCaclRequest
2025-05-28 14:45:48 +08:00
Jing Sima
fa5b70d670
feat: [TS-6100] Parse create stream sql: Sql gen refacotr.
2025-05-28 13:52:56 +08:00
Jinqing Kuang
5f3d1df1fa
Merge branch '3.0' into feat/TS-6100-3.0
2025-05-27 19:31:36 +08:00
Hongze Cheng
430da1f872
enh: add memory usage monitor for vnode ( #31212 )
2025-05-27 16:40:03 +08:00
wangmm0220
9772e9846c
fix(stream): error if %%n with varchar tag
2025-05-26 19:56:41 +08:00
wangmm0220
3aa8c175f5
feat(stream): reset agg operator & reset tsdb reader in fetch request
2025-05-26 14:52:02 +08:00
Jinqing Kuang
cd5ef52a9d
fix(stream): remove unused files
2025-05-26 09:38:40 +08:00
Jinqing Kuang
18566df281
Merge branch '3.0' into kjq-merge
2025-05-24 14:56:21 +08:00
Haojun Liao
11665b6efe
fix(stream): control the global parameters.
2025-05-23 14:36:20 +08:00
Haojun Liao
8b2eb18323
fix(stream): update the stream info for update.
2025-05-23 11:01:28 +08:00
Haojun Liao
02d70186d2
refactor(stream): add parameters to config the stream queue size.
2025-05-22 19:56:49 +08:00
Haojun Liao
f5db6532dc
refactor(stream): add parameters to config the write queue size.
2025-05-22 19:54:54 +08:00
Haojun Liao
4743cc3683
refactor(stream): check for write queue size for sink tasks.
2025-05-22 17:55:47 +08:00
wangmm0220
2d717c3a4f
feat(stream): set tag value in trigger cols
2025-05-22 15:38:32 +08:00
Haojun Liao
231d8b416e
fix(stream): fix memory leak.
2025-05-22 09:55:18 +08:00
wangmm0220
b13967b8f3
feat(stream): move SStorageAPI from stream directory to vnode
2025-05-17 16:00:50 +08:00
wangmm0220
96ab95e33b
feat(stream): add group values
2025-05-17 11:41:46 +08:00
factosea
edcec9f02b
fix: create table has exist
2025-05-17 10:47:56 +08:00
wangmm0220
3f08f95dde
feat(stream): fix col id error
2025-05-16 17:03:55 +08:00
wangmm0220
2891e73399
feat(stream): fix compile error
2025-05-16 16:13:44 +08:00
wangmm0220
a5007823b8
feat(stream): add ctime for getting wal meta
2025-05-16 16:14:42 +08:00
wangmm0220
39015385f9
feat(stream): add some logs
2025-05-16 09:23:52 +08:00
wangmm0220
88414a1b09
feat(stream): use TimeRangeExpr to filter data in tsdb reader
2025-05-15 19:07:42 +08:00
WANG MINGMING
40216c6143
opti(tmq): save tmq offset in vnode commit time & change log level ( #30993 )
2025-05-15 18:15:01 +08:00
wangmm0220
0297c7be2c
feat(stream): add ts slotid/blockId in external window
2025-05-15 14:22:46 +08:00
wangmm0220
9ffe95755b
feat(stream): add interface of getting cols value of groupid
2025-05-15 10:03:57 +08:00
facetosea
0a7ac81aeb
fix: table schema version start from 1
2025-05-14 17:41:51 +08:00
wangmm0220
a1955fb3af
feat(stream): add ast for calc in stream reader
2025-05-14 17:39:02 +08:00
facetosea
11760abea2
fix: subtable schema check when table has exist
2025-05-14 11:12:14 +08:00
