runtime: 512GB memory limitation #10460
Comments
ianlancetaylor
changed the title
|
To be clear, I think you are talking about the value of MaxMem in runtime/malloc.go (tip) or runtime/malloc.h (Go 1.4), right? |
|
Yes, exactly. Couldn't that be a default, but have it changeable via a compiler flag? I mean when calling 6g, not when compiling 6g. |
|
How about use ulimit -v as a mechanism to increase the max heap?
If ulimit -v (RLIMIT_AS) is unlimited, we reserve for 128GB, however, if
it's limited to another amount, we reserve that amount. so if you raise
the ulimit -v for a Go program, it will be able to use more memory.
Another benefit is that you can actually reduce the VM footprint of a
Go process if needed.
|
|
I think an explicit command-line argument to the binary (handled by the runtime, perhaps; the GHC Haskell compiler, for example, does this with -rtsopts) is more intuitive and discoverable, and less likely to lead to surprising effects. That would be ideal; a reasonable fallback would be a compile-time flag. Certainly it's better than requiring users to patch the compiler source code! |
|
We can't add runtime command line flags to Go binaries, that's for sure.
(Had we reserved a -goruntime or similar flag before Go 1 is released,
we could probably do that, but now adding any universal runtime flag
might have conflict with user command line parsing.)
Using an environment variable, perhaps, but the runtime already takes
many environment variables, we shouldn't add one more lightly.
I think using ulimit to control VM usage is very natural and it also fixes
a few other issues (where on certain systems due to large amount of
VM space reserved, you can't fork a Go process and thus can't execute
another program.)
PS: changing MaxMem doesn't require patching the Go compiler. It's
a runtime change.
|
|
I certainly understand that something like a "-goruntime" flag can't be added until Go 2. I think there is definite interest in Go in data science, and the need for more than 128GB is going to be common enough. So, perhaps adding an environment variable is warranted. How does one change MaxMem at runtime (of the compiler, I presume)? Up until 1.4 it certainly required a source change to malloc.(go,h); I recall patching it for 1.0 when 16GB didn't cut it. |
bradfitz
changed the title
|
If we want to increase this to say a Terabyte I don't see why a Go flag or On Tue, Apr 14, 2015 at 11:07 PM, Noah Daniels notifications@github.com
|
|
It is not trivial to just change MaxMem at run time, and it's probably too late in the cycle to introduce code allowing MaxMem to be a variable, although I agree it is probably the right long-term solution. I will, however, bump MaxMem on 64-bit systems to 512 GB for Go 1.5. That should buy us some more headroom until we can fix things properly. |
|
CL https://golang.org/cl/10819 mentions this issue. |
|
"I don't have a sense of what, in terms The National Center for Biotechnology Information (NCBI) maintains a database of "non-redundant" (unique) protein sequences. That database is now over 60 million sequences, and for the past several years has been doubling every two years. PubChem (a public database of chemical compounds) is over 300GB in size on disk. It has also been growing, though not as quickly. If Go is going to be useful for big-data science, it needs to be able to use as much RAM as we can throw into a system. Terabyte-plus compute servers are common now. Now, I don't need a terabyte today, but I need more than 128GB. The right solution is for this not to be hardcoded. In my case, what consumes all the RAM is a large lookup table of kmers (very short sequences) indexed into every sequence in the database I'm trying to compress. Lots of pointers, lots of string data, and we don't want to write things out to disk until we're done. |
A workaround for #10460. Change-Id: I607a556561d509db6de047892f886fb565513895 Reviewed-on: https://go-review.googlesource.com/10819 Reviewed-by: Ian Lance Taylor <iant@golang.org> Reviewed-by: Austin Clements <austin@google.com>
rsc
changed the title
|
I wanted to raise this issue again, I know it's been a while but I've hit the 512GB limit on a system with 4TB. It took me a while to determine that Go's runtime was the issue, and not something else. Ideally, the out of memory error message could report that this is an internal Go limit reached and not what looks like a system OOM message. I feel 512GB is perfectly reasonable as a default, but it would be great to document and provide a way for FWIW, I'm also doing bioinformatics work like @ndaniels - Go is very well suited to it. |
|
/cc @aclements to think about for Go 1.10. |
|
+1 @pbnjay |
This would require threading some error information around that we currently don't, but should be pretty easy.
I'm not sure what problem you're getting at here. Memory mapped files have no effect on Go's heap arena limit since they don't live in the Go heap.
I would love to eliminate this limit, but doing so requires a complete redesign of the Go heap. I've actually been thinking about how to do that, but it's not a simple matter of just extending it at runtime. :) |
|
Change https://golang.org/cl/85887 mentions this issue: |
|
Change https://golang.org/cl/85889 mentions this issue: |
|
Change https://golang.org/cl/85885 mentions this issue: |
|
Change https://golang.org/cl/85888 mentions this issue: |
|
Change https://golang.org/cl/85884 mentions this issue: |
|
Change https://golang.org/cl/85882 mentions this issue: |
|
Change https://golang.org/cl/85883 mentions this issue: |
|
Change https://golang.org/cl/85886 mentions this issue: |
There are various places that assume the heap bitmap is contiguous and scan it sequentially. We're about to split up the heap bitmap. This commit modifies all of these except heapBitsSetType to use the heapBits abstractions so they can transparently switch to a discontiguous bitmap. Updates #10460. This is a step toward supporting sparse heaps. Change-Id: I2f3994a5785e4dccb66602fb3950bbd290d9392c Reviewed-on: https://go-review.googlesource.com/85882 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
This splits the heap bitmap into separate chunks for every 64MB of the heap and introduces an index mapping from virtual address to metadata. It modifies the heapBits abstraction to use this two-level structure. Finally, it modifies heapBitsSetType to unroll the bitmap into the object itself and then copy it out if the bitmap would span discontiguous bitmap chunks. This is a step toward supporting general sparse heaps, which will eliminate address space conflict failures as well as the limit on the heap size. It's also advantageous for 32-bit. 32-bit already supports discontiguous heaps by always starting the arena at address 0. However, as a result, with a contiguous bitmap, if the kernel chooses a high address (near 2GB) for a heap mapping, the runtime is forced to map up to 128MB of heap bitmap. Now the runtime can map sections of the bitmap for just the parts of the address space used by the heap. Updates #10460. This slightly slows down the x/garbage and compilebench benchmarks. However, I think the slowdown is acceptably small. name old time/op new time/op delta Template 178ms ± 1% 180ms ± 1% +0.78% (p=0.029 n=10+10) Unicode 85.7ms ± 2% 86.5ms ± 2% ~ (p=0.089 n=10+10) GoTypes 594ms ± 0% 599ms ± 1% +0.70% (p=0.000 n=9+9) Compiler 2.86s ± 0% 2.87s ± 0% +0.40% (p=0.001 n=9+9) SSA 7.23s ± 2% 7.29s ± 2% +0.94% (p=0.029 n=10+10) Flate 116ms ± 1% 117ms ± 1% +0.99% (p=0.000 n=9+9) GoParser 146ms ± 1% 146ms ± 0% ~ (p=0.193 n=10+7) Reflect 399ms ± 0% 403ms ± 1% +0.89% (p=0.001 n=10+10) Tar 173ms ± 1% 174ms ± 1% +0.91% (p=0.013 n=10+9) XML 208ms ± 1% 210ms ± 1% +0.93% (p=0.000 n=10+10) [Geo mean] 368ms 371ms +0.79% name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.17ms ± 1% 2.21ms ± 1% +2.15% (p=0.000 n=20+20) Change-Id: I037fd283221976f4f61249119d6b97b100bcbc66 Reviewed-on: https://go-review.googlesource.com/85883 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
This abstracts the remaining direct accesses to mheap.spans into new mheap.setSpan and mheap.setSpans methods. For #10460. Change-Id: Id1db8bc5e34a77a9221032aa2e62d05322707364 Reviewed-on: https://go-review.googlesource.com/85884 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
This splits the span map into separate chunks for every 64MB of the heap. The span map chunks now live in the same indirect structure as the bitmap. Updates #10460. This causes a slight improvement in compilebench and the x/benchmarks garbage benchmark. I'm not sure why it improves performance. name old time/op new time/op delta Template 185ms ± 1% 184ms ± 1% ~ (p=0.315 n=9+10) Unicode 86.9ms ± 1% 86.9ms ± 3% ~ (p=0.356 n=9+10) GoTypes 602ms ± 1% 599ms ± 0% -0.59% (p=0.002 n=9+10) Compiler 2.89s ± 0% 2.87s ± 1% -0.50% (p=0.003 n=9+9) SSA 7.25s ± 0% 7.29s ± 1% ~ (p=0.400 n=9+10) Flate 118ms ± 1% 118ms ± 2% ~ (p=0.065 n=10+9) GoParser 147ms ± 2% 147ms ± 1% ~ (p=0.549 n=10+9) Reflect 403ms ± 1% 401ms ± 1% -0.47% (p=0.035 n=9+10) Tar 176ms ± 1% 175ms ± 1% -0.59% (p=0.013 n=10+9) XML 211ms ± 1% 209ms ± 1% -0.83% (p=0.011 n=10+10) (https://perf.golang.org/search?q=upload:20171231.1) name old time/op new time/op delta Garbage/benchmem-MB=64-12 2.24ms ± 1% 2.23ms ± 1% -0.36% (p=0.001 n=20+19) (https://perf.golang.org/search?q=upload:20171231.2) Change-Id: I2563f8704ab9812434947faf293c5327f9b0d07a Reviewed-on: https://go-review.googlesource.com/85885 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
This replaces all uses of the mheap_.arena_* fields outside of mallocinit and sysAlloc. These fields fundamentally assume a contiguous heap between two bounds, so eliminating these is necessary for a sparse heap. Many of these are replaced with checks for non-nil spans at the test address (which in turn checks for a non-nil entry in the heap arena array). Some of them are just for debugging and somewhat meaningless with a sparse heap, so those we just delete. Updates #10460. Change-Id: I8345b95ffc610aed694f08f74633b3c63506a41f Reviewed-on: https://go-review.googlesource.com/85886 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Currently large sysReserve calls on some OSes don't actually reserve the memory, but just check that it can be reserved. This was important when we called sysReserve to "reserve" many gigabytes for the heap up front, but now that we map memory in small increments as we need it, this complication is no longer necessary. This has one curious side benefit: currently, on Linux, allocations that are large enough to be rejected by mmap wind up freezing the application for a long time before it panics. This happens because sysReserve doesn't reserve the memory, so sysMap calls mmap_fixed, which calls mmap, which fails because the mapping is too large. However, mmap_fixed doesn't inspect *why* mmap fails, so it falls back to probing every page in the desired region individually with mincore before performing an (otherwise dangerous) MAP_FIXED mapping, which will also fail. This takes a long time for a large region. Now this logic is gone, so the mmap failure leads to an immediate panic. Updates #10460. Change-Id: I8efe88c611871cdb14f99fadd09db83e0161ca2e Reviewed-on: https://go-review.googlesource.com/85888 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
Currently there's a detailed comment in lfstack_64bit.go about address space limitations on various architectures. Since that's now relevant to malloc, move it to a more prominent place in the documentation for memLimitBits. Updates #10460. Change-Id: If9708291cf3a288057b8b3ba0ba6a59e3602bbd6 Reviewed-on: https://go-review.googlesource.com/85889 Run-TryBot: Austin Clements <austin@google.com> TryBot-Result: Gobot Gobot <gobot@golang.org> Reviewed-by: Rick Hudson <rlh@golang.org>
In Go 1.4, the compiler still has a hardcoded 128GB memory limit. While this is much better than the 16GB limit from 1.0, it's presenting a real limitation to using Go for big-data scientific work, such as my research, such as https://github.com/ndaniels/cablastp2. While it's trivial for me to change the allocation limit in go's source and rebuild my own compiler, this presents a problem for other users using my work; if they find out that they need to patch the Go compiler just to build my source code, they're not going to like it. It's already enough of a barrier for some users to require they have the Go compiler. While I could provide statically compiled binaries for a variety of platforms, I can't target every OS and kernel version, and it becomes a nuisance to update those binaries whenever I update my source code.
I understand Go's memory allocation philosophy, and I realize that (for the time being) it needs to be hardcoded in a Go binary. But, it would be a huge improvement if a compiler flag could be used to alter the allocation limit for a particular package being compiled. Even better would be a runtime flag (much like Java's Xmx flag for the heap).
The text was updated successfully, but these errors were encountered: