[Bug] Fix the memory expand 10~1000x of compression algorithm (#5504)

Fix the memory expand 10~1000x of compression algorithm in load and compaction
apache · Mar 12, 2021 · e9a73ee · e9a73ee
1 parent 543ed46
commit e9a73ee
Show file tree

Hide file tree

Showing 3 changed files with 6 additions and 2 deletions.
diff --git a/be/src/olap/rowset/segment_v2/page_io.cpp b/be/src/olap/rowset/segment_v2/page_io.cpp
@@ -50,6 +50,9 @@ Status PageIO::compress_page_body(const BlockCompressionCodec* codec, double min
         double space_saving = 1.0 - static_cast<double>(buf.size()) / uncompressed_size;
         // return compressed body only when it saves more than min_space_saving
         if (space_saving > 0 && space_saving >= min_space_saving) {
+            // shrink the buf to fit the len size to avoid taking
+            // up the memory of the size MAX_COMPRESSED_SIZE
+            buf.shrink_to_fit();
             *compressed_body = buf.build();
             return Status::OK();
         }

diff --git a/docs/en/administrator-guide/bucket-shuffle-join.md b/docs/en/administrator-guide/bucket-shuffle-join.md
@@ -1,4 +1,3 @@
-```
 ---
 {
     "title": "Bucket Shuffle Join",
@@ -24,7 +23,7 @@ KIND, either express or implied.  See the License for the
 specific language governing permissions and limitations
 under the License.
 -->
-```
+
 # Bucket Shuffle Join
 
 Bucket Shuffle Join is a new function officially added in Doris 0.14. The purpose is to provide local optimization for some join queries to reduce the time-consuming of data transmission between nodes and speed up the query.

diff --git a/docs/zh-CN/administrator-guide/bucket-shuffle-join.md b/docs/zh-CN/administrator-guide/bucket-shuffle-join.md
@@ -1,7 +1,9 @@
+---
 {
     "title": "Bucket Shuffle Join",
     "language": "zh-CN"
 }
+---
 
 <!-- 
 Licensed to the Apache Software Foundation (ASF) under one