Skip to content

[Do not merge] CI Trigger #15827

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 216 commits into from
Jul 15, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
216 commits
Select commit Hold shift + click to select a range
b7e7dad
partial
Caideyipi Jun 26, 2025
71f7c88
wal deletion
Caideyipi Jun 26, 2025
078a057
continue deletion
Caideyipi Jun 26, 2025
4dd437d
Update CheckpointManager.java
Caideyipi Jun 26, 2025
cb182d4
Release memory
Caideyipi Jun 26, 2025
f6b1cb3
Parital
Caideyipi Jun 26, 2025
12c21dd
update PipeInsertNodeTabletInsertionEvent
luoluoyuyu Jun 26, 2025
a078b1f
Remove useless
Caideyipi Jun 26, 2025
523b9c2
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jun 26, 2025
9241bcc
Update PipeRealtimeDataRegionHybridExtractor.java
Caideyipi Jun 26, 2025
09a0e61
Update Metric.java
Caideyipi Jun 26, 2025
ac7272a
setup
VGalaxies Jun 25, 2025
31a3a02
Fix
Caideyipi Jun 26, 2025
56f97e8
modify pipeStuckRestartMinIntervalMs
luoluoyuyu Jun 26, 2025
2cb4692
Pipe/IoTV2: Persist progress index locally before shutdown to accurat…
SteveYurongSu Jun 19, 2025
4bf2e71
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jun 26, 2025
9923977
add MaxWaitFinishTime
luoluoyuyu Jun 26, 2025
3ff163d
update PipeConfig
luoluoyuyu Jun 26, 2025
66d1631
Fix
Caideyipi Jun 26, 2025
c8014b1
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jun 26, 2025
6e7ff7f
Update WalDeleteOutdatedNewTest.java
Caideyipi Jun 26, 2025
213236d
Update PipeMemoryManager.java
Caideyipi Jun 26, 2025
a8cfd04
Update CommonConfig.java
Caideyipi Jun 26, 2025
4dc2072
Update PipeInsertNodeTabletInsertionEvent.java
Caideyipi Jun 26, 2025
2a3c861
Update PipeInsertNodeTabletInsertionEvent.java
Caideyipi Jun 26, 2025
0b35ba6
Bug fix
Caideyipi Jun 26, 2025
bbd8f7b
Update PipeConnectorConstant.java
Caideyipi Jun 26, 2025
5b3e275
add PipeDynamicMemoryBlock
luoluoyuyu Jun 26, 2025
f12da9b
fix
luoluoyuyu Jun 26, 2025
3ffb268
defaultval
Caideyipi Jun 26, 2025
db00211
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jun 26, 2025
07bba66
Update IoTDBDataRegionExtractor.java
Caideyipi Jun 27, 2025
d09466e
Fix
Caideyipi Jun 27, 2025
f9dae02
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jun 27, 2025
2208033
Total removal
Caideyipi Jun 27, 2025
1fe1bd6
Update CommonConfig.java
Caideyipi Jun 27, 2025
0326633
revert
Caideyipi Jun 27, 2025
25ca24f
Update ThreadName.java
Caideyipi Jun 27, 2025
cfc38e2
Update PipeResourceMetrics.java
Caideyipi Jun 27, 2025
26717ac
Update PipeDescriptor.java
Caideyipi Jun 27, 2025
1e241c2
add PIPE_FLOATING_MEMORY_USAGE
luoluoyuyu Jun 27, 2025
e8a3a72
fix PIPE_FLOATING_MEMORY_USAGE
luoluoyuyu Jun 27, 2025
5ef82cc
Change listen
Caideyipi Jun 30, 2025
874b57b
Changed isEmpty
Caideyipi Jun 30, 2025
3be74a0
Fix & use async flush
Caideyipi Jun 30, 2025
e3df0d6
Remove restart logic
Caideyipi Jun 30, 2025
d801fa9
Sync delete tsFile & Added pipeName
Caideyipi Jun 30, 2025
4196fc3
try using pipe dir
Caideyipi Jun 30, 2025
55d6355
Fixed the concurrent problem of floating memory
Caideyipi Jun 30, 2025
d467454
remove unnecessary flushes & restart parameters
Caideyipi Jun 30, 2025
05777b3
Only closed
Caideyipi Jun 30, 2025
e50a35c
Refactor historical flush & realtie listening
Caideyipi Jun 30, 2025
734eb1f
Nullable historical
Caideyipi Jun 30, 2025
eb023a2
Update PipeHistoricalDataRegionTsFileExtractor.java
Caideyipi Jun 30, 2025
47c6b26
Delete file only
Caideyipi Jun 30, 2025
3a911b6
Update IoTDBDataRegionExtractor.java
Caideyipi Jul 1, 2025
47ee0bb
partial
Caideyipi Jul 1, 2025
2d61e3a
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 1, 2025
6ca8bb6
simple refactor
Caideyipi Jul 1, 2025
fb5308b
Restore tsFile wait Close
Caideyipi Jul 1, 2025
8bbc14f
Refactor max progress index
Caideyipi Jul 1, 2025
6237535
Refactor resource Metrics && Fixed the metric multiple creation problem
Caideyipi Jul 1, 2025
e030052
Fixed tsFile manager test
Caideyipi Jul 1, 2025
da1752e
try refactor
Caideyipi Jul 1, 2025
ebf297d
Update PipeRealtimeDataRegionHybridExtractor.java
Caideyipi Jul 1, 2025
aef7866
May complete
Caideyipi Jul 1, 2025
7bc4820
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 1, 2025
472d1c0
Update TsFileInsertionDataContainerTest.java
Caideyipi Jul 1, 2025
567ec02
Update PipeTsFileResourceManager.java
Caideyipi Jul 1, 2025
ffeb881
introduce PipeCompactionManager for tsfile resource replacement in p…
SteveYurongSu Jul 1, 2025
3f56e3b
Help receiver GC
Caideyipi Jul 1, 2025
e3ed158
Add memory limit for creating Pipe
luoluoyuyu Jul 1, 2025
f271c18
May fix
Caideyipi Jul 1, 2025
377572b
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 1, 2025
cb74467
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into p…
SteveYurongSu Jul 1, 2025
d40a7aa
May fix
Caideyipi Jul 1, 2025
1064617
fix
luoluoyuyu Jul 1, 2025
55da540
partial fix
Caideyipi Jul 1, 2025
eb0ddaa
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 1, 2025
20e97a5
update IT
luoluoyuyu Jul 1, 2025
4ce201c
Update PipeTsFileResourceManagerTest.java
Caideyipi Jul 1, 2025
5d16d65
Fix 2/3 of the floating memory
Caideyipi Jul 1, 2025
6a69e5d
Fixed UT
Caideyipi Jul 1, 2025
751ac9d
impl replace in PipeRealtimePriorityBlockingQueue.java
SteveYurongSu Jul 1, 2025
ec98947
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into p…
SteveYurongSu Jul 1, 2025
1d8246e
fix
luoluoyuyu Jul 2, 2025
7b4f4cb
first delete assigner
Caideyipi Jul 2, 2025
db30ba7
Fix
Caideyipi Jul 2, 2025
0c6e61f
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 2, 2025
e9cf852
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 2, 2025
42bca9e
Update PipeTsFileResourceManagerTest.java
Caideyipi Jul 2, 2025
948bcf5
Update PipeDataNodeHardlinkOrCopiedFileDirStartupCleaner.java
Caideyipi Jul 2, 2025
3206928
Move
Caideyipi Jul 2, 2025
0d804a7
partial
Caideyipi Jul 2, 2025
f71e98c
partially
Caideyipi Jul 2, 2025
12f49fe
Update PipeRealtimeExtractTest.java
Caideyipi Jul 2, 2025
a37bf45
bug fix
Caideyipi Jul 2, 2025
90937d6
ready for PipeCompactedTsFileInsertionEvent constructor
SteveYurongSu Jul 2, 2025
08fa6b3
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into p…
SteveYurongSu Jul 2, 2025
009db93
made resource nullable
Caideyipi Jul 2, 2025
eab0c40
handle binding progress of PipeTsFileInsertionEvent
SteveYurongSu Jul 2, 2025
ef7ad11
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 2, 2025
c90fb2b
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 2, 2025
ee19849
partially
Caideyipi Jul 2, 2025
d0e5230
Fix
Caideyipi Jul 2, 2025
8cb1678
partial
Caideyipi Jul 2, 2025
ad5fda5
PipeCompactedTsFileInsertionEvent done
SteveYurongSu Jul 2, 2025
4c265de
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into p…
SteveYurongSu Jul 2, 2025
fcd0995
Update PipeCompactedTsFileInsertionEvent.java
SteveYurongSu Jul 2, 2025
f4d1bd1
may complete historical
Caideyipi Jul 2, 2025
2f03004
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 2, 2025
c6fc727
intro dummy events
SteveYurongSu Jul 2, 2025
1237e19
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into p…
SteveYurongSu Jul 2, 2025
3bedc36
Seq send by default
Caideyipi Jul 2, 2025
963212e
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 2, 2025
766abb0
Added the report frequency
Caideyipi Jul 2, 2025
aee6335
revert tsFile hardlink recover
Caideyipi Jul 2, 2025
93baf53
introduce segment index
Caideyipi Jul 2, 2025
16e2019
Update SegmentProgressIndex.java
Caideyipi Jul 2, 2025
88faaa2
polishing
Caideyipi Jul 2, 2025
6856dcf
Update SegmentProgressIndex.java
Caideyipi Jul 2, 2025
2aafa46
continue reverting
Caideyipi Jul 3, 2025
7b42852
test fix
Caideyipi Jul 3, 2025
68fd8e7
Optimize IoTProgressIndex
Caideyipi Jul 3, 2025
8d6abd1
Fix NPE
Caideyipi Jul 3, 2025
739daf3
add logs
SteveYurongSu Jul 3, 2025
39021f1
Add log for missing TsFileInsertionEvents replacement
SteveYurongSu Jul 3, 2025
334136f
NPE fix & heartbeat resp 2 common
Caideyipi Jul 3, 2025
bf1adc3
skip the commit of the later events when there are tsFiles
Caideyipi Jul 3, 2025
fd49232
report heartbeat
Caideyipi Jul 3, 2025
792a548
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 3, 2025
18aef84
try fix
Caideyipi Jul 3, 2025
c4e04a5
next fix
Caideyipi Jul 3, 2025
7c85c7f
Historical NPE fix
Caideyipi Jul 4, 2025
01f2e1c
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 4, 2025
7599506
realtime NPE fix & reduce log
Caideyipi Jul 4, 2025
07e0136
Update PipeEventCommitter.java
Caideyipi Jul 4, 2025
0ce9b20
Fix hardlink
Caideyipi Jul 4, 2025
d9acbaf
Update PipeTsFileEpochProgressIndexKeeper.java
Caideyipi Jul 4, 2025
e9a945d
rollback remaining
Caideyipi Jul 4, 2025
a5d4319
revert hard-link
Caideyipi Jul 4, 2025
031341b
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 4, 2025
fcde95e
fix assigner
luoluoyuyu Jul 4, 2025
58bdce8
Fix null pointer and ClientPool exceptions
luoluoyuyu Jul 5, 2025
dbe3ed6
Fix null pointer when DecreaseResource
luoluoyuyu Jul 5, 2025
2da9253
revert replace
Caideyipi Jul 6, 2025
64c07f8
Lock release
Caideyipi Jul 6, 2025
cabc774
fix-lock
Caideyipi Jul 6, 2025
cbdcbc6
Update PipeDataRegionAssigner.java
Caideyipi Jul 6, 2025
79fca84
Fix OOM caused by deleting gcSchemaInfo
luoluoyuyu Jul 7, 2025
a4b3345
Remove some useless
Caideyipi Jul 8, 2025
e60fc21
revert fileUtils
Caideyipi Jul 8, 2025
a64ba16
remove-useless
Caideyipi Jul 8, 2025
121b12a
heartbeat-fix
Caideyipi Jul 8, 2025
6428005
revert realtime supply
Caideyipi Jul 8, 2025
1cea612
Update PipeRealtimeDataRegionLogExtractor.java
Caideyipi Jul 8, 2025
7c21782
Remove synchornization
Caideyipi Jul 8, 2025
6c9fe86
Create CachedSchemaPatternMatcherTest.java
Caideyipi Jul 8, 2025
b4c96be
continue-kill-sync
Caideyipi Jul 8, 2025
31199d1
change-default
Caideyipi Jul 8, 2025
2042779
continue-progress-index-push-back-fix
Caideyipi Jul 8, 2025
24fce7e
Update PipeTsFileResourceManager.java
Caideyipi Jul 8, 2025
5335297
Fixed msg NPE
Caideyipi Jul 9, 2025
c2ee6bf
Update DataNodeSchemaLockManager.java
Caideyipi Jul 9, 2025
8b10eac
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 9, 2025
5118fce
Fix IT
Caideyipi Jul 9, 2025
6109284
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 9, 2025
a09eb7c
Update IoTDBPipeTypeConversionIT.java
Caideyipi Jul 9, 2025
a271983
Fixed tsfile concurrency problem
Caideyipi Jul 9, 2025
b482afa
reduce heartbeat logger
Caideyipi Jul 9, 2025
e4c2c76
better logger
Caideyipi Jul 9, 2025
f2da2c9
Replace exceptions with debug logs in schema checks
SteveYurongSu Jul 10, 2025
f019de4
Update PipeDataNodeSinglePipeMetrics.java
Caideyipi Jul 10, 2025
27b76ee
Added timeout in shutdown hook
Caideyipi Jul 10, 2025
71b80e1
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 10, 2025
9823ffe
Fixed the issue that the isTotallyGeneratedByPipe flag is invalid whe…
luoluoyuyu Jul 10, 2025
a23f942
Simplify the function to set the isTotallyGeneratedByPipe flag
luoluoyuyu Jul 10, 2025
9937a91
Update LoadTsFileAnalyzer.java
SteveYurongSu Jul 10, 2025
94e9585
Update PipeConfigNodeTaskAgent.java
Caideyipi Jul 10, 2025
611d7aa
Update PipeDataNodeTaskAgent.java
Caideyipi Jul 10, 2025
66de768
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 11, 2025
e2364c3
Update IoTDBPipeTypeConversionISessionIT.java
Caideyipi Jul 11, 2025
f7f25d7
Add Tablet Batch and TSFile Batch related metrics
luoluoyuyu Jul 11, 2025
3feb5f7
Update CompactionUtils.java
Caideyipi Jul 11, 2025
588a8f6
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 11, 2025
9ced07f
Partial
Caideyipi Jul 11, 2025
5afc004
Separated the pipe thread pool
Caideyipi Jul 12, 2025
7f74a90
Update PipeConnectorSubtaskManager.java
Caideyipi Jul 12, 2025
0a0ee9c
Fix the problem of creating too many PipeTransferTsFileHandlers, caus…
luoluoyuyu Jul 12, 2025
9db51d9
Optimized hybrid logger behaviour
Caideyipi Jul 12, 2025
e91b9a0
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 12, 2025
52f28f0
Optimized the downgrade logger to identify the tsFile
Caideyipi Jul 12, 2025
659a894
Modify PipeTransferTsFileHandler to apply for memory order
luoluoyuyu Jul 12, 2025
fac3bdb
Fixed memoryBlock
Caideyipi Jul 12, 2025
75cb7e7
spotless
Caideyipi Jul 12, 2025
d12d80b
Update IoTDBDataRegionAsyncConnector.java
Caideyipi Jul 12, 2025
3691d16
Merge branch 'dev/1.3' of https://github.com/apache/iotdb into multi-fix
Caideyipi Jul 12, 2025
3913efa
delete-uesless
Caideyipi Jul 12, 2025
a3a9ee4
Update PipeRealtimeExtractTest.java
Caideyipi Jul 12, 2025
938b202
rename tsFile pending thread & unify the realtime parallel limit
Caideyipi Jul 12, 2025
4ede5b0
Separate executor
Caideyipi Jul 12, 2025
78857eb
Fixed sleep logic
Caideyipi Jul 12, 2025
2b0246c
Reset default & Separate thread pool
Caideyipi Jul 14, 2025
df968f7
convert
Caideyipi Jul 14, 2025
db746af
revert-pom
Caideyipi Jul 14, 2025
a836f48
[To dev/1.3] Pipe: Add a tool for validating and repairing isGenerate…
SteveYurongSu Jul 14, 2025
bec300f
Fixed flag setting
Caideyipi Jul 14, 2025
7bc8a1b
Merge branch 'multi-fix' of https://github.com/Caideyipi/iotdb into m…
Caideyipi Jul 14, 2025
43558c7
Update PipeInsertionDataNodeListener.java
Caideyipi Jul 14, 2025
636f822
spotless
Caideyipi Jul 14, 2025
f6fac84
Fixed load tsFile
Caideyipi Jul 15, 2025
b3dba84
volatile
Caideyipi Jul 15, 2025
be62e31
Atomic-reference
Caideyipi Jul 15, 2025
aa1ffc8
fix
Caideyipi Jul 15, 2025
0217c0f
for-ci
Caideyipi Jul 15, 2025
71b7a4c
for-ci-2
Caideyipi Jul 15, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -429,6 +429,12 @@ public CommonConfig setSchemaRegionPerDataNode(double schemaRegionPerDataNode) {
return this;
}

@Override
public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck) {
setProperty("pipe_enable_memory_checked", String.valueOf(isPipeEnableMemoryCheck));
return this;
}

@Override
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
setProperty("pipe_air_gap_receiver_enabled", String.valueOf(isPipeAirGapReceiverEnabled));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -438,6 +438,13 @@ public CommonConfig setSchemaRegionPerDataNode(double schemaRegionPerDataNode) {
return this;
}

@Override
public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck) {
dnConfig.setIsPipeEnableMemoryCheck(isPipeEnableMemoryCheck);
cnConfig.setIsPipeEnableMemoryCheck(isPipeEnableMemoryCheck);
return this;
}

@Override
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
dnConfig.setPipeAirGapReceiverEnabled(isPipeAirGapReceiverEnabled);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -308,6 +308,11 @@ public CommonConfig setSchemaRegionPerDataNode(double schemaRegionPerDataNode) {
return this;
}

@Override
public CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck) {
return this;
}

@Override
public CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled) {
return this;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -138,6 +138,8 @@ CommonConfig setEnableAutoLeaderBalanceForIoTConsensus(

CommonConfig setSchemaRegionPerDataNode(double schemaRegionPerDataNode);

CommonConfig setIsPipeEnableMemoryCheck(boolean isPipeEnableMemoryCheck);

CommonConfig setPipeAirGapReceiverEnabled(boolean isPipeAirGapReceiverEnabled);

CommonConfig setDriverTaskExecutionTimeSliceInMs(long driverTaskExecutionTimeSliceInMs);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@
import org.junit.After;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
Expand Down Expand Up @@ -901,6 +902,7 @@ public void testLoadLocally() throws Exception {
}

@Test
@Ignore("Load with conversion is currently banned")
public void testLoadWithConvertOnTypeMismatch() throws Exception {

List<Pair<MeasurementSchema, MeasurementSchema>> measurementSchemas =
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,15 @@ protected void setupConfig() {
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,14 +59,16 @@ public void setUp() {
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS);
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS);
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ public void setUp() {
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS);
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

receiverEnv
.getConfig()
Expand All @@ -84,7 +85,8 @@ public void setUp() {
.setSchemaReplicationFactor(3)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS);
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -66,14 +66,17 @@ public void setUp() {
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setPipeAirGapReceiverEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -459,7 +459,6 @@ public void testAsyncLoadTsFileWithoutVerify() throws Exception {
testLoadTsFileWithoutVerify("async");
}

@Test
private void testLoadTsFileWithoutVerify(final String loadTsFileStrategy) throws Exception {
final DataNodeWrapper receiverDataNode = receiverEnv.getDataNodeWrapper(0);

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -71,13 +71,15 @@ public void setUp() {
// Disable sender compaction for tsfile determination in loose range test
.setEnableSeqSpaceCompaction(false)
.setEnableUnseqSpaceCompaction(false)
.setEnableCrossSpaceCompaction(false);
.setEnableCrossSpaceCompaction(false)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -65,13 +65,15 @@ public void setUp() {
// of the tested idempotent sql.
.setDefaultSchemaRegionGroupNumPerDatabase(1)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -59,13 +59,15 @@ public void setUp() {
.setAutoCreateSchemaEnabled(true)
.setTimestampPrecision("ms")
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,8 @@ private void innerSetUp(
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(dataRegionConsensus)
.setSchemaReplicationFactor(schemaRegionReplicationFactor)
.setDataReplicationFactor(dataRegionReplicationFactor);
.setDataReplicationFactor(dataRegionReplicationFactor)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
Expand All @@ -83,7 +84,8 @@ private void innerSetUp(
.setSchemaRegionConsensusProtocolClass(schemaRegionConsensus)
.setDataRegionConsensusProtocolClass(dataRegionConsensus)
.setSchemaReplicationFactor(schemaRegionReplicationFactor)
.setDataReplicationFactor(dataRegionReplicationFactor);
.setDataReplicationFactor(dataRegionReplicationFactor)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -61,13 +61,15 @@ public void setUp() {
// Disable sender compaction to test mods
.setEnableSeqSpaceCompaction(false)
.setEnableUnseqSpaceCompaction(false)
.setEnableCrossSpaceCompaction(false);
.setEnableCrossSpaceCompaction(false)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(true)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -48,13 +48,15 @@ protected void setupConfig() {
.getCommonConfig()
.setAutoCreateSchemaEnabled(false)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);
receiverEnv
.getConfig()
.getCommonConfig()
.setAutoCreateSchemaEnabled(false)
.setConfigNodeConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS);
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setIsPipeEnableMemoryCheck(false);

// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,7 @@
import org.apache.tsfile.write.record.Tablet;
import org.apache.tsfile.write.schema.IMeasurementSchema;
import org.apache.tsfile.write.schema.MeasurementSchema;
import org.junit.Ignore;
import org.junit.Test;
import org.junit.experimental.categories.Category;
import org.junit.runner.RunWith;
Expand Down Expand Up @@ -76,6 +77,7 @@ public void insertTablet() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertTabletReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand All @@ -94,6 +96,7 @@ public void insertAlignedTablet() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertAlignedTabletReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand All @@ -103,6 +106,7 @@ public void insertAlignedTabletReceiveByTsFile() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertRecordsReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -137,6 +141,7 @@ public void insertRecord() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertRecordReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -177,6 +182,7 @@ public void insertAlignedRecord() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertAlignedRecordReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -225,6 +231,7 @@ public void insertAlignedRecords() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertAlignedRecordsReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -253,6 +260,7 @@ public void insertStringRecordsOfOneDevice() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertStringRecordsOfOneDeviceReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -281,6 +289,7 @@ public void insertAlignedStringRecordsOfOneDevice() {
}

@Test
@Ignore("The receiver conversion is currently banned, will ignore conflict")
public void insertAlignedStringRecordsOfOneDeviceReceiveByTsFile() {
prepareTypeConversionTest(
(ISession senderSession, ISession receiverSession, Tablet tablet) -> {
Expand Down Expand Up @@ -381,13 +390,12 @@ private void createDataPipe(String diff, boolean isTSFile) {
String sql =
String.format(
"create pipe test%s"
+ " with source ('source'='iotdb-source','source.path'='root.test.**','realtime.mode'='%s','realtime.enable'='%s','history.enable'='%s')"
+ " with source ('source'='iotdb-source','source.path'='root.test.**','realtime.mode'='%s','realtime.enable'='%s','history.enable'='true')"
+ " with processor ('processor'='do-nothing-processor')"
+ " with sink ('node-urls'='%s:%s','batch.enable'='false','sink.format'='%s')",
diff,
isTSFile ? "file" : "forced-log",
!isTSFile,
isTSFile,
receiverEnv.getIP(),
receiverEnv.getPort(),
isTSFile ? "tsfile" : "tablet");
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -234,7 +234,7 @@ private void createDataPipe() {
String sql =
String.format(
"create pipe test"
+ " with source ('source'='iotdb-source','source.path'='root.test.**','realtime.mode'='forced-log','realtime.enable'='true','history.enable'='false')"
+ " with source ('source'='iotdb-source','source.path'='root.test.**','realtime.mode'='forced-log')"
+ " with processor ('processor'='do-nothing-processor')"
+ " with sink ('node-urls'='%s:%s','batch.enable'='false','sink.format'='tablet')",
receiverEnv.getIP(), receiverEnv.getPort());
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -36,6 +36,7 @@ public void setUp() {
env.getConfig().getCommonConfig().setAutoCreateSchemaEnabled(true);
// 10 min, assert that the operations will not time out
env.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
env.getConfig().getCommonConfig().setIsPipeEnableMemoryCheck(false);
env.initClusterEnvironment();
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -82,7 +82,8 @@ public void setUp() throws Exception {
.setSchemaRegionConsensusProtocolClass(ConsensusFactory.RATIS_CONSENSUS)
.setDataRegionConsensusProtocolClass(ConsensusFactory.IOT_CONSENSUS)
.setSchemaReplicationFactor(3)
.setDataReplicationFactor(2);
.setDataReplicationFactor(2)
.setIsPipeEnableMemoryCheck(false);

EnvFactory.getEnv().initClusterEnvironment(3, 3);
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,9 @@ protected void setUpConfig() {
// 10 min, assert that the operations will not time out
senderEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);
receiverEnv.getConfig().getCommonConfig().setDnConnectionTimeoutMs(600000);

senderEnv.getConfig().getCommonConfig().setIsPipeEnableMemoryCheck(false);
receiverEnv.getConfig().getCommonConfig().setIsPipeEnableMemoryCheck(false);
}

@Override
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -117,8 +117,11 @@ static final class SubscriptionInfo {
protected void setUpConfig() {
super.setUpConfig();

senderEnv.getConfig().getCommonConfig().setIsPipeEnableMemoryCheck(false);

// Enable air gap receiver
receiverEnv.getConfig().getCommonConfig().setPipeAirGapReceiverEnabled(true);
receiverEnv.getConfig().getCommonConfig().setIsPipeEnableMemoryCheck(false);
}

@Override
Expand Down
Loading
Loading