Skip to content

Commit 426bb28

Browse files
committed
programming guide blurb
1 parent 8053fed commit 426bb28

File tree

1 file changed

+4
-0
lines changed

1 file changed

+4
-0
lines changed

docs/mllib-decision-tree.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -93,6 +93,10 @@ The recursive tree construction is stopped at a node when one of the two conditi
9393
1. The node depth is equal to the `maxDepth` training parammeter
9494
2. No split candidate leads to an information gain at the node.
9595

96+
### Max memory requirements
97+
98+
For faster processing, the decision tree algorithm performs simultaneous histogram computations for all nodes at each level of the tree. This could lead to high memory requirements at deeper levels of the tree leading to memory overflow errors. To alleviate this problem, a 'maxMemoryInMB' training parameter is provided which specifies the maximum amount of memory at the workers (twice as much at the master) to be allocated to the histogram computation. The default value is conservatively chosen to be 128 MB to allow the decision algorithm to work in most scenarios. Once the memory requirements for a level-wise computation crosses the `maxMemoryInMB` threshold, the node training tasks at each subsequent level is split into smaller tasks.
99+
96100
### Practical limitations
97101

98102
1. The implemented algorithm reads both sparse and dense data. However, it is not optimized for

0 commit comments

Comments
 (0)