GC overhead limit hit when reindexing large code base with 0.12-rc5

[tarzanek]

seems for large .jars we run out of memory, need to optimize my fix for #10

12:56:20 SEVERE: Problem updating lucene index database:
java.lang.OutOfMemoryError: GC overhead limit exceeded
        at org.apache.lucene.analysis.charfilter.HTMLStripCharFilter.<init>(HTMLStripCharFilter.java:30626)
        at org.opensolaris.opengrok.analysis.executables.JavaClassAnalyzer.analyze(JavaClassAnalyzer.java:111)
        at org.opensolaris.opengrok.analysis.executables.JarAnalyzer.analyze(JarAnalyzer.java:72)
        at org.opensolaris.opengrok.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:307)
        at org.opensolaris.opengrok.index.IndexDatabase.addFile(IndexDatabase.java:591)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:842)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
        at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
        at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:379)
        at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:168)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:744)

[vladak]

It seems we have hit this when reindexing with 0.12-rc5, the indexer got stuck after some time with the GC overhead limit exceeded message. The process tree of the indexer looks like this:

                21792 /bin/sh /var/run/opengrok-0.12/bin/OpenGrok index /ws-loc
                  21815 /usr/jdk/latest/bin/java -Xmx2048m -Dorg.opensolaris.op
                    8902  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8906  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8922  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13325 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13330 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13332 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13334 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13340 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13341 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13914 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    14109 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    15244 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    15645 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    2061  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    2499  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    3300  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    4795  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    6635  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    7550  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8045  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8094  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8263  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    8389  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    9734  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    9785  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    9832  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    9910  /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    12164 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    12497 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    12797 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo
                    13177 /usr/local/bin/ctags --c-kinds=+l --sql-kinds=+l --Fo

grok-hw:~$ ps -ly -p 21815
S    UID   PID  PPID   C PRI NI   RSS     SZ    WCHAN TTY         TIME CMD
S     80 21815 21792   0  40 20 274808 2093300        ? pts/3    2177:50 java

So the proces has RSS some 270 MB and SIZE almost 2 GB.

I just found out that the heap size is set by the OpenGrok script to 2 GB as we can also see above in the command arguments.

[vladak]

For completeness /usr/jdk/latest/bin/java is 32-bit application on our Solaris 11u1+SRU system.

[tarzanek]

another occurrence:
13:53:46 SEVERE: Problem updating lucene index database:
java.lang.OutOfMemoryError: GC overhead limit exceeded
at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl.(SAXParserImpl.java:137)
at com.sun.org.apache.xerces.internal.jaxp.SAXParserFactoryImpl.newSAXParser(SAXParserFactoryImpl.java:81)
at com.sun.beans.decoder.DocumentHandler$1.run(DocumentHandler.java:366)
at com.sun.beans.decoder.DocumentHandler$1.run(DocumentHandler.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.security.ProtectionDomain$1.doIntersectionPrivilege(ProtectionDomain.java:76)
at com.sun.beans.decoder.DocumentHandler.parse(DocumentHandler.java:363)
at java.beans.XMLDecoder$1.run(XMLDecoder.java:201)
at java.beans.XMLDecoder$1.run(XMLDecoder.java:199)
at java.security.AccessController.doPrivileged(Native Method)
at java.beans.XMLDecoder.parsingComplete(XMLDecoder.java:199)
at java.beans.XMLDecoder.readObject(XMLDecoder.java:250)
at org.opensolaris.opengrok.history.FileHistoryCache.readCache(FileHistoryCache.java:117)
at org.opensolaris.opengrok.history.FileHistoryCache.get(FileHistoryCache.java:296)
at org.opensolaris.opengrok.history.HistoryGuru.getHistory(HistoryGuru.java:212)
at org.opensolaris.opengrok.history.HistoryGuru.getHistoryReader(HistoryGuru.java:175)
at org.opensolaris.opengrok.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:284)
at org.opensolaris.opengrok.index.IndexDatabase.addFile(IndexDatabase.java:591)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:842)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:812)
at org.opensolaris.opengrok.index.IndexDatabase.update(IndexDatabase.java:379)
at org.opensolaris.opengrok.index.IndexDatabase$1.run(IndexDatabase.java:168)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

this seems like some JDK issue, it seems I will need to change some defaults in OpenGrok script

• one more
  13:53:46 SEVERE: Problem updating lucene index database:
  java.lang.OutOfMemoryError: GC overhead limit exceeded
  at org.opensolaris.opengrok.analysis.executables.JavaClassAnalyzer.linkDef(JavaClassAnalyzer.java:120)
  at org.opensolaris.opengrok.analysis.executables.JavaClassAnalyzer.getContent(JavaClassAnalyzer.java:140)
  at org.opensolaris.opengrok.analysis.executables.JavaClassAnalyzer.analyze(JavaClassAnalyzer.java:95)
  at org.opensolaris.opengrok.analysis.executables.JarAnalyzer.analyze(JarAnalyzer.java:72)
  at org.opensolaris.opengrok.analysis.AnalyzerGuru.populateDocument(AnalyzerGuru.java:307)
  at org.opensolaris.opengrok.index.IndexDatabase.addFile(IndexDatabase.java:591)
  at org.opensolaris.opengrok.index.IndexDatabase.indexDown(IndexDatabase.java:842)

[tarzanek]

% /usr/jdk/latest/bin/java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) Server VM (build 24.45-b08, mixed mode)

[tarzanek]

I am quite sure this wasn't there in previous JDK update, so jdk changed and lucene changed, one of those caused the prob and the prob is not really in jar analyzer (though it seems jar analyzer triggers this !)

[tarzanek]

(I will also try to backout fix for #10 and see if it makes any difference)

[tarzanek]

or disable this jdk feature -XX:-UseGCOverheadLimit ?
obviously we queue objects for lucene to process, so refs won't be freed that easily

[kahatlen]

We have seen problems with OOME in JarAnalyzer before. For example in #8.

Does anything stand out if you run "jmap -histo:live" on the process?

[vladak]

I had to use -F and strip live because of this error:

# /usr/jdk/instances/jdk1.7.0/bin/jmap -histo:live 21815
21815: Unable to open door: target process not responding or HotSpot VM not loaded

The (trimmed) output is then as follows:

# /usr/jdk/instances/jdk1.7.0/bin/jmap -F -histo 21815
Attaching to process ID 21815, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 24.45-b08
Iterating over heap. This may take a while...
Object Histogram:

num       #instances    #bytes  Class description
--------------------------------------------------------------------------
1:              737653  928206952       char[]
2:              20059   101838352       byte[]
3:              72082   58773424        int[]
4:              660301  15847224        java.lang.String
5:              35484   6181344 java.lang.Object[]
6:              173912  5565184 java.util.TreeMap$Entry
7:              32435   4022928 * ConstMethodKlass
8:              32435   2860104 * MethodKlass
9:              73942   2366144 org.apache.lucene.document.TextField
10:             25465   2164624 java.util.HashMap$Entry[]
11:             14442   1848576 org.apache.lucene.analysis.charfilter.HTMLStripCharFilter
12:             3116    1809896 long[]
13:             2787    1725688 * ConstantPoolKlass
14:             67493   1619832 org.opensolaris.opengrok.history.MercurialTagEntry
15:             36723   1468920 org.apache.lucene.util.fst.Builder$UnCompiledNode
16:             41134   1316288 org.apache.lucene.util.fst.Builder$Arc
17:             46972   1127328 java.util.ArrayList
18:             22645   1086960 java.util.HashMap
19:             45213   1085112 java.util.HashMap$Entry
20:             2787    1084432 * InstanceKlassKlass
21:             3625    1054120 * MethodDataKlass
22:             2429    981072  * ConstantPoolCacheKlass
23:             24413   781216  java.util.LinkedHashMap$Entry
24:             29157   699768  org.opensolaris.opengrok.analysis.IteratorReader
25:             28884   693216  org.apache.lucene.analysis.charfilter.HTMLStripCharFilter$TextSegment
26:             27095   650280  java.util.ArrayList$Itr
27:             37535   622240  org.apache.lucene.util.fst.Builder$Arc[]
28:             25168   565744  java.lang.String[]
29:             13269   543848  short[]
30:             14976   479232  java.io.StringReader
31:             9641    462768  java.util.TreeMap
32:             2777    444320  org.apache.derby.client.am.ColumnMetaData40
33:             188     398240  org.apache.lucene.index.IndexableField[]
34:             9429    377160  org.opensolaris.opengrok.history.HistoryEntry
35:             3051    374256  java.lang.Class
36:             1608    321600  org.apache.derby.client.am.PreparedStatement40
37:             1412    316288  org.apache.derby.client.net.NetResultSet40
38:             8727    279264  java.lang.ref.Finalizer
39:             11520   276480  java.util.Date
40:             4783    267848  java.util.LinkedHashMap
41:             13470   215520  java.util.HashMap$KeySet
42:             13406   214496  java.util.HashSet
43:             4066    208784  * System ObjArray
44:             7697    184728  org.opensolaris.opengrok.analysis.Definitions$Tag
45:             159     163904  org.apache.derby.client.net.FdocaSimpleDataArray[]
46:             9011    158360  boolean[]
47:             1032    156864  org.apache.derby.client.net.NetCursor
48:             9635    154160  java.util.TreeSet
49:             6372    152928  java.io.File
50:             9463    151408  java.lang.StringBuffer
51:             1544    148224  sun.util.calendar.Gregorian$Date
52:             1858    133776  org.apache.lucene.util.fst.FST$Arc
53:             5333    127992  org.apache.lucene.util.BytesRef
54:             432     118840  char[][]
55:             7007    112112  java.lang.Integer
56:             6734    107744  java.util.HashMap$EntrySet
57:             431     107352  org.apache.lucene.util.fst.FST$Arc[]
58:             6384    102144  org.apache.lucene.util.AttributeSource$State
59:             6022    96352   org.opensolaris.opengrok.analysis.Definitions$LineTagMap
60:             1863    89424   java.nio.HeapByteBuffer
61:             1536    86016   org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter$PendingTerm
62:             1691    81168   org.apache.derby.client.am.Section
63:             766     79664   org.opensolaris.opengrok.analysis.plain.PlainFullTokenizer
64:             247     75088   * ObjArrayKlassKlass
65:             3107    74568   java.util.concurrent.ConcurrentHashMap$HashEntry
66:             1765    70600   java.util.WeakHashMap$Entry
67:             117     61776   java.util.TimerTask[]
68:             662     58144   java.util.WeakHashMap$Entry[]
69:             2361    56664   org.apache.lucene.analysis.tokenattributes.CharTermAttributeImpl
70:             2313    55512   org.opensolaris.opengrok.history.GitTagEntry
71:             1608    51456   org.apache.derby.client.net.NetPreparedStatement
72:             2128    51072   org.apache.lucene.analysis.Analyzer$TokenStreamComponents
73:             2052    49248   java.util.Stack
74:             868     48608   org.opensolaris.opengrok.analysis.PathTokenizer
75:             6046    48368   java.lang.Object
76:             1157    46280   sun.nio.cs.UTF_8$Decoder
77:             1118    44720   sun.nio.cs.StreamDecoder
78:             1819    43656   java.util.Hashtable$Entry
79:             2708    43328   java.util.concurrent.atomic.AtomicInteger
80:             1744    41856   java.io.ExpiringCache$Entry
81:             2468    39488   java.util.HashMap$Values
82:             2361    37776   org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl
83:             2361    37776   org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl
84:             2361    37776   org.apache.lucene.util.AttributeSource$State[]
85:             662     37072   java.util.WeakHashMap
86:             1536    36864   org.apache.lucene.codecs.TermStats
87:             1535    36840   org.apache.lucene.codecs.BlockTreeTermsWriter$PendingTerm
88:             1081    34592   java.io.BufferedInputStream
89:             1079    34528   java.lang.ThreadLocal$ThreadLocalMap$Entry
90:             1996    31936   java.io.FileDescriptor
91:             995     31840   java.lang.ref.SoftReference
92:             1948    31168   org.apache.bcel.classfile.ConstantUtf8
93:             550     30800   java.util.zip.Deflater
94:             1265    30360   java.lang.ref.WeakReference
95:             338     29744   org.apache.lucene.index.TermsHashPerField
96:             908     29056   org.apache.lucene.analysis.ReusableStringReader
97:             310     28800   java.util.Hashtable$Entry[]
98:             1205    28560   int[][]
99:             267     27768   org.opensolaris.opengrok.analysis.plain.PlainSymbolTokenizer
100:            203     27608   java.text.DecimalFormat
101:            1115    26760   java.io.InputStreamReader
102:            1652    26432   org.apache.lucene.util.fst.Builder$CompiledNode
103:            70      25632   java.io.File[]
104:            194     25512   java.util.concurrent.ConcurrentHashMap$HashEntry[]
105:            774     24768   java.io.FileOutputStream
106:            509     24432   sun.nio.cs.UTF_8$Encoder
107:            428     23968   org.apache.lucene.analysis.core.StopFilter
108:            207     23184   java.util.GregorianCalendar
109:            477     22896   java.nio.HeapCharBuffer
110:            45      22320   byte[][]
111:            459     22032   sun.nio.cs.StreamEncoder
112:            172     21952   java.lang.ThreadLocal$ThreadLocalMap$Entry[]
113:            267     21360   java.lang.reflect.Method
114:            440     21120   java.lang.UNIXProcess$DeferredCloseInputStream
115:            429     20592   org.apache.lucene.analysis.util.CharArrayMap
116:            635     20320   java.io.FileInputStream
117:            79      20224   org.apache.derby.client.net.NetConnection40
118:            338     18928   org.apache.lucene.util.BytesRefHash
119:            1043    16688   java.io.ByteArrayOutputStream
120:            678     16272   java.lang.ref.ReferenceQueue
121:            308     14784   java.beans.MethodDescriptor
122:            17      14624   org.apache.bcel.classfile.Constant[]
123:            79      14536   org.apache.derby.client.net.NetAgent
124:            138     14352   org.apache.lucene.util.fst.FST
125:            355     14200   java.io.BufferedWriter
126:            220     14080   java.lang.reflect.Constructor
127:            287     13776   java.util.Hashtable
128:            117     13104   java.util.TimerThread
129:            409     13088   org.opensolaris.opengrok.history.RepositoryInfo
130:            544     13056   org.apache.bcel.classfile.ConstantNameAndType
131:            203     12992   java.text.DecimalFormatSymbols
132:            510     12240   org.opensolaris.opengrok.history.HistoryReader
133:            76      12120   org.apache.lucene.util.fst.Builder$UnCompiledNode[]
134:            148     11840   org.apache.lucene.util.packed.PackedInts$Reader[]
135:            184     11776   java.text.SimpleDateFormat
136:            353     11296   java.util.zip.GZIPOutputStream
137:            468     11232   org.apache.bcel.classfile.ConstantMethodref
138:            464     11136   java.util.concurrent.locks.ReentrantLock$NonfairSync
139:            459     11016   java.io.OutputStreamWriter
140:            442     10608   org.apache.lucene.util.CloseableThreadLocal
141:            204     9792    java.util.TreeMap$DescendingSubMap
142:            233     9320    org.apache.lucene.document.Field$StringTokenStream
143:            163     9128    java.text.DateFormatSymbols
144:            565     9040    java.util.zip.ZStreamRef
145:            158     8848    org.apache.derby.client.net.Typdef
146:            85      8840    java.lang.Thread
147:            181     8688    sun.util.calendar.ZoneInfo
148:            176     8448    org.apache.lucene.index.FreqProxTermsWriterPerField$FreqProxPostingsArray
149:            79      8216    java.net.SocksSocketImpl
150:            203     8120    java.text.DigitList
151:            169     8112    org.apache.lucene.index.FreqProxTermsWriterPerField
152:            169     8112    org.apache.lucene.index.FieldInfo
153:            169     8112    org.apache.lucene.index.TermVectorsConsumerPerField
154:            169     8112    org.apache.lucene.index.FieldInvertState
155:            162     7776    java.io.BufferedReader
156:            324     7776    org.opensolaris.opengrok.analysis.Definitions
157:            483     7728    java.lang.ThreadLocal
158:            192     7680    org.opensolaris.opengrok.history.MercurialRepository
159:            118     7552    java.beans.PropertyDescriptor
160:            456     7296    java.util.WeakHashMap$KeySet
161:            301     7224    org.opensolaris.opengrok.configuration.Project
162:            148     7104    org.apache.lucene.util.packed.AppendingDeltaPackedLongBuffer
163:            148     7104    org.apache.lucene.codecs.BlockTreeTermsWriter$PendingBlock
164:            394     7000    java.lang.Class[]
165:            79      6952    org.apache.derby.client.net.NetResultSetReply
166:            430     6880    org.apache.lucene.analysis.util.CharArraySet
167:            210     6720    org.apache.lucene.util.packed.Packed64
168:            93      6696    org.apache.lucene.index.SegmentCommitInfo
169:            165     6600    org.opensolaris.opengrok.history.SCCSRepository
170:            161     6440    org.apache.lucene.index.TermVectorsConsumerPerField$TermVectorsPostingsArray
171:            194     6208    java.util.concurrent.ConcurrentHashMap$Segment
172:            253     6072    java.sql.Timestamp
173:            375     6000    java.lang.Character
174:            148     5920    org.apache.lucene.index.NumericDocValuesWriter
175:            184     5888    org.apache.lucene.index.DocumentsWriterPerThreadPool$ThreadState
176:            55      5720    java.lang.UNIXProcess$2$1
177:            355     5680    java.util.zip.CRC32
178:            138     5520    org.apache.lucene.util.fst.BytesStore
179:            680     5440    java.lang.ref.ReferenceQueue$Lock
180:            339     5424    java.util.concurrent.ConcurrentLinkedQueue$Node
181:            169     5408    org.apache.lucene.index.DocFieldProcessorPerField
182:            169     5408    org.apache.lucene.index.NormsConsumerPerField
183:            338     5408    org.apache.lucene.index.TermsHashPerField$PostingsBytesStartArray
184:            169     5408    org.apache.lucene.index.DocInverterPerField
185:            218     5232    java.security.Provider$ServiceKey
186:            79      5056    org.apache.derby.client.net.NetResultSetRequest
187:            308     4928    sun.misc.FDBigInt
188:            75      4800    java.lang.reflect.Field
189:            196     4704    org.apache.lucene.index.IndexFileDeleter$RefCount
190:            195     4680    org.apache.lucene.util.IntsRef
191:            115     4600    java.lang.UNIXProcess
192:            113     4520    org.apache.lucene.util.packed.PagedGrowableWriter
193:            79      4424    org.apache.derby.client.am.SectionManager
194:            55      4400    java.util.concurrent.ConcurrentHashMap$Segment[]
195:            275     4400    java.lang.Long
196:            75      4200    java.security.Provider$Service
197:            75      4200    org.apache.lucene.util.fst.BytesRefFSTEnum
198:            171     4104    java.lang.ThreadLocal$ThreadLocalMap
199:            64      4096    sun.security.pkcs11.SunPKCS11$P11Service
200:            245     3920    org.apache.bcel.classfile.ConstantClass
201:            79      3792    org.apache.derby.client.net.NetDatabaseMetaData40
202:            79      3792    java.net.SocketInputStream
203:            79      3792    java.net.SocketOutputStream
204:            23      3680    org.apache.lucene.index.IndexWriter
205:            114     3648    sun.security.util.ObjectIdentifier
206:            114     3648    java.lang.ProcessBuilder
207:            32      3584    org.apache.lucene.index.DocFieldProcessorPerField[]
208:            148     3552    org.apache.lucene.util.OpenBitSet
209:            71      3408    java.lang.Package
210:            103     3296    org.opensolaris.opengrok.analysis.Ctags
211:            204     3264    java.util.TreeMap$KeySet
212:            102     3264    java.math.BigInteger
213:            79      3160    org.apache.derby.client.net.FdocaSimpleDataArray
214:            23      3128    org.apache.lucene.index.DocumentsWriterFlushControl
215:            64      3072    org.apache.lucene.index.BufferedUpdates
216:            190     3040    org.apache.bcel.classfile.ConstantString
217:            53      2968    java.net.URL
218:            72      2880    org.apache.derby.client.am.DateTimeValue
219:            118     2832    java.io.BufferedOutputStream
220:            176     2816    org.apache.lucene.util.packed.PackedInts$FormatAndBits
221:            116     2784    java.util.Timer
222:            58      2784    org.apache.lucene.index.SegmentInfo
223:            113     2712    org.apache.lucene.util.packed.GrowableWriter
224:            55      2640    java.util.concurrent.ConcurrentHashMap
225:            23      2576    org.apache.lucene.index.IndexWriterConfig
226:            23      2576    org.opensolaris.opengrok.analysis.sh.ShXref
227:            79      2528    org.apache.derby.client.am.ProductLevel
228:            52      2496    org.apache.lucene.store.FSDirectory$FSIndexOutput
229:            24      2496    java.util.concurrent.ThreadLocalRandom
230:            8       2432    * TypeArrayKlassKlass
231:            100     2400    java.util.LinkedList
232:            23      2392    org.opensolaris.opengrok.analysis.c.CSymbolTokenizer
233:            23      2392    org.opensolaris.opengrok.analysis.sh.ShSymbolTokenizer
234:            23      2392    org.apache.lucene.index.LiveIndexWriterConfig
235:            23      2392    org.opensolaris.opengrok.analysis.scala.ScalaSymbolTokenizer
236:            23      2392    org.opensolaris.opengrok.analysis.perl.PerlSymbolTokenizer
237:            99      2376    java.security.AccessControlContext
238:            21      2352    org.opensolaris.opengrok.analysis.document.TroffXref
239:            41      2296    org.apache.lucene.util.fst.Builder
240:            22      2288    org.opensolaris.opengrok.analysis.c.CxxSymbolTokenizer
241:            17      2240    java.beans.MethodDescriptor[]
242:            23      2208    org.opensolaris.opengrok.analysis.scala.ScalaXref
243:            23      2208    org.opensolaris.opengrok.analysis.plain.XMLXref
244:            69      2208    sun.security.pkcs11.SunPKCS11$Descriptor
245:            23      2208    org.opensolaris.opengrok.analysis.c.CXref
246:            23      2208    org.opensolaris.opengrok.analysis.plain.PlainXref
247:            21      2184    org.opensolaris.opengrok.analysis.document.TroffFullTokenizer
248:            45      2160    org.apache.lucene.index.IndexFileDeleter$CommitPoint
249:            66      2112    org.opensolaris.opengrok.analysis.FileAnalyzer
250:            66      2112    java.io.RandomAccessFile
251:            22      2112    org.opensolaris.opengrok.analysis.c.CxxXref
252:            22      2112    org.opensolaris.opengrok.analysis.perl.PerlXref
253:            20      2080    org.opensolaris.opengrok.analysis.java.JavaSymbolTokenizer
254:            43      2064    org.apache.lucene.index.SegmentInfos
255:            23      2024    java.text.SimpleDateFormat[]
256:            49      1960    java.util.IdentityHashMap
257:            61      1952    java.io.ObjectStreamField
258:            20      1920    org.opensolaris.opengrok.analysis.java.JavaXref
259:            79      1896    org.apache.derby.client.net.EbcdicCcsidManager
260:            79      1896    java.net.Socket
261:            79      1896    org.apache.derby.client.net.ResultSetReply
262:            59      1888    java.util.ArrayList$SubList
263:            18      1872    org.opensolaris.opengrok.analysis.python.PythonSymbolTokenizer
264:            117     1872    java.util.TaskQueue
265:            116     1856    java.util.Timer$1
266:            23      1840    java.nio.ByteBuffer[]
267:            23      1840    org.apache.lucene.index.TieredMergePolicy
268:            115     1840    java.util.concurrent.atomic.AtomicLong
269:            113     1808    org.apache.lucene.util.packed.PackedInts$Mutable[]
270:            32      1792    java.nio.DirectByteBuffer
271:            32      1792    org.apache.lucene.index.TermsHash
272:            73      1752    org.apache.bcel.classfile.ConstantFieldref
273:            109     1744    org.apache.lucene.util.fst.ReverseBytesReader
274:            31      1736    org.apache.lucene.codecs.BlockTreeTermsWriter$FieldMetaData
275:            18      1728    org.opensolaris.opengrok.analysis.python.PythonXref
276:            42      1680    org.apache.lucene.index.ByteSliceReader
277:            42      1680    sun.util.locale.LocaleObjectCache$CacheEntry
278:            23      1656    org.opensolaris.opengrok.index.IndexDatabase
279:            41      1640    org.apache.lucene.util.fst.NodeHash
280:            51      1632    org.apache.lucene.document.Field
281:            67      1608    java.util.Vector
282:            40      1600    org.apache.lucene.util.packed.BulkOperationPacked
283:            66      1584    org.apache.bcel.classfile.ConstantInterfaceMethodref
284:            98      1568    org.apache.lucene.util.packed.PackedInts$NullReader
285:            28      1568    sun.reflect.DelegatingClassLoader
286:            64      1536    sun.reflect.generics.tree.SimpleClassTypeSignature
287:            48      1536    java.util.regex.Pattern$Curly
288:            48      1536    org.apache.lucene.index.DocumentsWriterDeleteQueue
289:            62      1488    sun.reflect.NativeConstructorAccessorImpl
290:            23      1472    org.apache.lucene.index.DocumentsWriter
291:            45      1440    java.util.HashMap$KeyIterator
292:            16      1408    org.apache.lucene.index.DocumentsWriterPerThread
293:            59      1408    java.security.cert.Certificate[]
294:            44      1408    java.security.CodeSource
295:            44      1408    java.security.ProtectionDomain
296:            86      1376    java.util.concurrent.locks.ReentrantLock
297:            13      1352    org.opensolaris.opengrok.analysis.tcl.TclSymbolTokenizer
298:            11      1320    * KlassKlass
299:            1       1312    sun.misc.FDBigInt[]
300:            81      1296    java.lang.StringBuilder
301:            23      1288    org.apache.lucene.index.IndexFileDeleter
302:            80      1280    java.util.Collections$UnmodifiableRandomAccessList
303:            20      1280    java.util.regex.Pattern
304:            79      1264    sun.reflect.DelegatingConstructorAccessorImpl
305:            79      1264    org.opensolaris.opengrok.jdbc.ConnectionResource
306:            79      1264    org.apache.derby.client.am.CrossConverters
307:            12      1248    org.opensolaris.opengrok.analysis.sql.SQLXref
308:            12      1248    org.opensolaris.opengrok.analysis.fortran.FortranSymbolTokenizer
309:            13      1248    org.opensolaris.opengrok.analysis.tcl.TclXref
310:            38      1216    java.math.BigDecimal
311:            75      1200    org.apache.lucene.util.fst.BytesRefFSTEnum$InputOutput
312:            50      1200    org.apache.lucene.util.packed.Direct8
313:            73      1168    * CompilerICHolderKlass
314:            29      1160    sun.misc.Cleaner
315:            36      1152    java.util.zip.ZipCoder
316:            12      1152    org.opensolaris.opengrok.analysis.fortran.FortranXref
317:            72      1152    java.util.Collections$SynchronizedSet
318:            11      1144    org.opensolaris.opengrok.analysis.javascript.JavaScriptSymbolTokenizer
319:            70      1120    java.util.concurrent.atomic.AtomicBoolean
320:            14      1120    java.util.jar.JarFile$JarFileEntry
321:            23      1104    java.util.concurrent.ThreadPoolExecutor$Worker
322:            23      1104    sun.misc.URLClassPath$JarLoader
323:            23      1104    org.apache.lucene.index.DocumentsWriterPerThreadPool$ThreadState[]
324:            2       1096    float[]
325:            27      1080    org.apache.lucene.store.RAMOutputStream
326:            15      1080    org.apache.lucene.index.StandardDirectoryReader
327:            12      1072    java.lang.reflect.Method[]
328:            67      1072    java.util.Arrays$ArrayList
329:            2       1072    java.lang.Thread[]
330:            11      1056    org.opensolaris.opengrok.analysis.javascript.JavaScriptXref
331:            64      1056    sun.reflect.generics.tree.TypeArgument[]
332:            1       1040    java.lang.Integer[]
333:            1       1040    short[][]
334:            1       1040    java.lang.Long[]
335:            64      1024    org.apache.lucene.index.DocumentsWriterDeleteQueue$DeleteSlice
336:            32      1024    org.apache.lucene.util.IntBlockPool
337:            16      1024    org.apache.lucene.codecs.compressing.CompressingStoredFieldsWriter
338:            32      1024    org.apache.lucene.util.ByteBlockPool
339:            64      1024    sun.reflect.generics.tree.ClassTypeSignature
340:            42      1008    org.opensolaris.opengrok.index.CommandLineOptions$Option
341:            25      1000    java.util.logging.LogManager$LoggerWeakRef
342:            41      984     org.apache.lucene.util.fst.BytesStore$2
343:            39      936     java.lang.StringCoding$StringDecoder
344:            29      928     org.apache.lucene.store.RAMFile
345:            23      920     org.opensolaris.opengrok.analysis.plain.PlainAnalyzer
346:            23      920     org.opensolaris.opengrok.history.SubversionRepository
347:            23      920     org.apache.lucene.index.BufferedUpdatesStream
348:            23      920     org.apache.lucene.index.ConcurrentMergeScheduler
349:            23      920     org.opensolaris.opengrok.analysis.c.CAnalyzer
350:            23      920     org.opensolaris.opengrok.analysis.sh.ShAnalyzer
351:            23      920     org.opensolaris.opengrok.analysis.scala.ScalaAnalyzer
352:            16      896     java.util.jar.JarFile
353:            16      896     org.apache.lucene.index.TermVectorsConsumer
354:            16      896     org.apache.lucene.index.DocFieldProcessor
355:            55      880     java.lang.UNIXProcess$2
356:            22      880     org.opensolaris.opengrok.analysis.c.CxxAnalyzer
357:            22      880     org.opensolaris.opengrok.analysis.perl.PerlAnalyzer
358:            18      864     org.opensolaris.opengrok.history.CVSRepository
359:            35      840     java.util.ArrayDeque
360:            52      832     java.security.Provider$UString
361:            26      832     java.nio.DirectByteBuffer$Deallocator
362:            20      800     org.opensolaris.opengrok.analysis.java.JavaAnalyzer
363:            49      784     java.util.Collections$UnmodifiableCollection
364:            48      768     org.apache.lucene.index.DocumentsWriterDeleteQueue$Node
365:            16      768     org.apache.lucene.index.FieldInfo[]
366:            16      768     org.apache.lucene.codecs.compressing.CompressingStoredFieldsIndexWriter
367:            8       768     java.io.ObjectStreamClass
368:            41      744     java.lang.reflect.Type[]
369:            17      744     java.beans.PropertyDescriptor[]
370:            23      736     org.opensolaris.opengrok.analysis.plain.XMLAnalyzer
371:            23      736     java.util.concurrent.FutureTask
372:            46      736     org.apache.lucene.util.SetOnce
373:            23      736     org.apache.lucene.store.NIOFSDirectory
374:            7       728     org.opensolaris.opengrok.analysis.php.PhpSymbolTokenizer
375:            7       728     org.opensolaris.opengrok.analysis.php.PhpXref
376:            15      720     java.lang.management.PlatformComponent
377:            18      720     org.opensolaris.opengrok.analysis.python.PythonAnalyzer
378:            45      720     org.opensolaris.opengrok.analysis.Ctags$1
379:            30      720     org.apache.lucene.util.packed.Packed64SingleBlock$Packed64SingleBlock4
380:            30      720     java.security.Provider$EngineDescription
381:            15      720     java.util.zip.Inflater
382:            44      704     java.security.ProtectionDomain$Key
383:            44      704     java.lang.ProcessEnvironment$Value
384:            44      704     java.security.Principal[]
385:            17      680     java.beans.GenericBeanInfo
386:            17      680     java.beans.BeanDescriptor
387:            21      672     sun.util.locale.BaseLocale
388:            21      672     sun.util.locale.BaseLocale$Key
389:            21      672     org.opensolaris.opengrok.analysis.document.TroffAnalyzer
390:            6       672     org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter
391:            9       648     org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput
392:            20      640     org.apache.lucene.util.packed.AbstractAppendingLongBuffer$Iterator
393:            20      640     sun.reflect.generics.reflectiveObjects.TypeVariableImpl
394:            20      640     org.apache.lucene.index.NumericDocValuesWriter$NumericIterator
395:            40      640     org.apache.bcel.classfile.ConstantInteger
396:            16      640     org.apache.lucene.index.DocumentsWriterPerThread$DocState
397:            10      640     java.util.logging.Logger
398:            16      640     org.apache.lucene.index.StoredFieldsProcessor
399:            39      624     java.util.regex.Pattern$CharPropertyNames$1
400:            26      600     java.io.ObjectStreamField[]
401:            15      600     org.opensolaris.opengrok.history.GitRepository
402:            30      584     java.io.Closeable[]
403:            24      576     java.net.InetAddress$InetAddressHolder
404:            36      576     java.util.regex.Pattern$Ctype
405:            24      576     sun.security.x509.OIDMap$OIDInfo
406:            18      576     org.apache.bcel.classfile.Unknown
407:            10      560     sun.nio.ch.FileChannelImpl
408:            14      560     java.util.zip.ZipFile
409:            23      552     org.apache.lucene.index.FieldInfos$FieldNumbers
410:            23      552     org.apache.lucene.index.ThreadAffinityDocumentsWriterThreadPool
411:            23      552     java.util.regex.Pattern$GroupTail
412:            23      552     org.apache.lucene.index.DocumentsWriterFlushQueue
413:            23      552     java.lang.StringCoding$StringEncoder
414:            23      552     sun.nio.ch.Util$BufferCache
415:            23      552     org.apache.lucene.index.DocumentsWriterStallControl
416:            17      544     org.opensolaris.opengrok.analysis.executables.ELFAnalyzer
417:            17      544     sun.reflect.generics.repository.ClassRepository
418:            17      544     java.io.ObjectStreamClass$WeakClassKey
419:            22      528     sun.security.util.DerValue
420:            33      528     java.security.ProtectionDomain[]
421:            22      528     java.util.Locale
422:            1       528     java.lang.Character[]
423:            33      528     java.lang.ProcessEnvironment$Variable
424:            11      528     org.opensolaris.opengrok.util.Executor$2
425:            22      528     sun.security.util.DerInputBuffer
426:            22      528     java.util.logging.LogManager$LogNode
427:            13      520     org.opensolaris.opengrok.analysis.archive.GZIPAnalyzer
428:            13      520     org.opensolaris.opengrok.analysis.tcl.TclAnalyzer
429:            5       520     org.opensolaris.opengrok.analysis.vb.VBSymbolTokenizer
430:            13      520     java.io.ObjectStreamClass$FieldReflectorKey
431:            32      512     org.apache.lucene.util.Counter$SerialCounter
432:            21      504     sun.security.x509.AVAKeyword
433:            21      504     sun.security.x509.RDN
434:            12      480     org.opensolaris.opengrok.analysis.sql.SQLAnalyzer
435:            5       480     org.opensolaris.opengrok.analysis.vb.VBXref
436:            12      480     org.opensolaris.opengrok.analysis.fortran.FortranAnalyzer
437:            3       480     org.apache.lucene.codecs.lucene41.Lucene41PostingsWriter
438:            15      480     org.opensolaris.opengrok.analysis.executables.JarAnalyzer
439:            15      480     org.opensolaris.opengrok.analysis.executables.JavaClassAnalyzer
440:            19      456     org.opensolaris.opengrok.analysis.JFlexXref$Style
441:            19      456     java.util.Locale$LocaleKey
442:            7       448     java.util.regex.Matcher
443:            14      448     org.opensolaris.opengrok.analysis.archive.ZipAnalyzer
444:            14      448     java.util.concurrent.locks.AbstractQueuedSynchronizer$Node
445:            11      440     org.opensolaris.opengrok.analysis.javascript.JavaScriptAnalyzer
446:            3       432     org.apache.lucene.util.packed.PackedInts$Encoder[]
447:            3       432     org.apache.lucene.util.packed.PackedInts$Decoder[]
448:            9       432     org.apache.lucene.store.CompoundFileWriter$DirectCFSIndexOutput
449:            9       432     java.util.Properties
450:            4       416     org.opensolaris.opengrok.analysis.uue.UuencodeFullTokenizer
451:            2       416     org.apache.lucene.util.packed.BulkOperation[]
452:            13      416     org.apache.lucene.document.StringField
453:            13      416     org.opensolaris.opengrok.analysis.archive.TarAnalyzer
454:            4       408     java.math.BigInteger[]
455:            17      408     sun.reflect.generics.tree.ClassSignature
456:            25      400     sun.reflect.BootstrapConstructorAccessorImpl
457:            16      384     org.apache.lucene.util.GrowableByteArrayDataOutput
458:            12      384     java.util.HashMap$ValueIterator
459:            24      384     java.util.Collections$UnmodifiableCollection$1
460:            24      384     java.util.concurrent.ConcurrentLinkedQueue
461:            16      384     sun.security.pkcs11.wrapper.CK_ATTRIBUTE
462:            16      384     org.apache.lucene.index.DocInverter
463:            4       384     org.opensolaris.opengrok.analysis.uue.UuencodeXref
464:            16      384     java.util.LinkedList$Node
465:            16      384     sun.reflect.generics.reflectiveObjects.ParameterizedTypeImpl
466:            23      368     java.net.Inet4Address
467:            23      368     org.opensolaris.opengrok.index.IndexDatabase$1
468:            23      368     java.util.regex.Pattern$GroupHead
469:            23      368     org.apache.lucene.index.IndexWriter$ReaderPool
470:            23      368     org.apache.lucene.index.FlushByRamOrCountsPolicy
471:            23      368     java.util.concurrent.Executors$RunnableAdapter
472:            15      360     com.oracle.security.ucrypto.UcryptoMech
473:            5       360     sun.net.www.protocol.jar.URLJarFile
474:            11      352     sun.security.jca.ProviderConfig
475:            22      352     java.util.LinkedHashSet
476:            11      352     java.io.FilePermission
477:            22      352     java.util.jar.Attributes$Name
478:            4       352     java.util.jar.JarVerifier
479:            11      352     sun.security.pkcs11.TemplateManager$TemplateKey
480:            22      352     sun.security.util.DerInputStream
481:            22      352     org.apache.derby.client.am.PreparedStatement$PossibleTypes
482:            21      336     sun.security.x509.AVA
483:            14      336     org.opensolaris.opengrok.jdbc.PreparedQuery
484:            21      336     sun.security.x509.AVA[]
485:            21      336     sun.reflect.generics.tree.FieldTypeSignature[]
486:            14      336     org.apache.lucene.util.packed.BulkOperationPackedSingleBlock
487:            17      336     sun.reflect.generics.tree.ClassTypeSignature[]
488:            9       328     java.lang.reflect.Field[]
489:            20      320     sun.reflect.generics.tree.FormalTypeParameter
490:            20      320     sun.reflect.generics.tree.TypeVariableSignature
491:            3       312     double[]
492:            16      312     java.lang.management.PlatformComponent[]
493:            17      304     java.lang.reflect.TypeVariable[]
494:            13      304     sun.security.pkcs11.wrapper.CK_ATTRIBUTE[]
495:            17      304     sun.reflect.generics.tree.FormalTypeParameter[]
496:            12      288     org.opensolaris.opengrok.util.Executor
497:            18      288     java.text.DateFormat$Field
498:            12      288     java.security.Permissions
499:            6       288     java.nio.DirectLongBufferU
500:            9       288     org.opensolaris.opengrok.history.MercurialHistoryParser
501:            9       288     org.apache.lucene.store.CompoundFileWriter$FileEntry
502:            3       288     org.apache.lucene.codecs.lucene41.Lucene41SkipWriter
503:            12      288     java.lang.ProcessEnvironment$StringEnvironment
504:            18      288     java.util.regex.Pattern$CharPropertyNames$4
505:            9       288     java.util.HashMap$EntryIterator
506:            7       280     org.opensolaris.opengrok.analysis.php.PhpAnalyzer
507:            7       280     org.apache.lucene.document.FieldType
508:            17      272     java.util.Collections$SetFromMap
509:            17      272     sun.reflect.generics.factory.CoreReflectionFactory
510:            17      272     org.apache.bcel.classfile.ConstantPool
511:            17      272     sun.reflect.generics.scope.ClassScope
512:            16      256     org.apache.lucene.index.FreqProxTermsWriter
513:            16      256     org.apache.lucene.store.TrackingDirectoryWrapper
514:            8       256     sun.misc.ProxyGenerator$PrimitiveTypeInfo
515:            16      256     org.apache.lucene.codecs.compressing.CompressionMode$LZ4FastCompressor
516:            16      256     org.apache.lucene.index.FieldInfos$Builder
517:            8       256     sun.reflect.UnsafeQualifiedStaticLongFieldAccessorImpl
518:            16      256     org.apache.lucene.codecs.compressing.LZ4$HashTable
519:            16      256     org.apache.lucene.util.ByteBlockPool$DirectTrackingAllocator
520:            16      256     org.apache.lucene.index.TermVectorsConsumerPerField[]
521:            16      256     org.apache.lucene.index.DocValuesProcessor
522:            16      256     org.apache.lucene.index.TwoStoredFieldsConsumers
523:            16      256     org.apache.lucene.index.DocumentsWriterPerThread$IntBlockAllocator
524:            16      256     java.lang.Byte
525:            14      240     java.lang.Byte[]
526:            6       240     java.io.ObjectStreamClass$FieldReflector
527:            2       240     org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread
528:            10      240     org.opensolaris.opengrok.util.Executor$1
529:            10      240     sun.nio.ch.NativeThreadSet
530:            15      240     java.util.Collections$UnmodifiableSet
531:            15      240     org.opensolaris.opengrok.util.Getopt$Option
532:            15      240     org.apache.lucene.util.Version
533:            10      240     java.lang.RuntimePermission
534:            10      240     org.apache.lucene.util.packed.Packed64SingleBlock$Packed64SingleBlock1
535:            15      240     org.apache.lucene.index.SegmentReader[]
536:            7       224     java.lang.ClassLoader$NativeLibrary
537:            14      224     org.apache.bcel.util.ClassPath$Zip
538:            14      224     sun.misc.MetaIndex
539:            9       216     sun.util.logging.PlatformLogger$Level
540:            9       216     java.util.regex.Pattern$1
541:            9       216     org.opensolaris.opengrok.analysis.executables.ELFAnalyzer$E_Machine
542:            9       216     org.opensolaris.opengrok.analysis.executables.ELFAnalyzer$ELFIdentification
543:            9       216     java.util.logging.Level
544:            9       216     java.lang.StackTraceElement
545:            2       208     org.opensolaris.opengrok.analysis.lisp.LispSymbolTokenizer
546:            13      208     java.util.regex.Pattern$Single
547:            13      208     java.lang.reflect.Constructor[]
548:            2       208     org.opensolaris.opengrok.analysis.lisp.LispXref
549:            7       200     java.lang.Boolean[]
550:            5       200     org.opensolaris.opengrok.analysis.vb.VBAnalyzer
551:            4       192     sun.security.x509.X500Name
552:            8       192     org.apache.lucene.util.packed.Packed64SingleBlock$Packed64SingleBlock12
553:            12      192     org.apache.lucene.document.Document
554:            4       192     java.util.ResourceBundle$CacheKey
555:            12      192     java.security.CodeSigner[]
556:            8       192     sun.reflect.annotation.AnnotationInvocationHandler
557:            8       192     java.math.RoundingMode
558:            2       192     sun.security.jgss.SunProvider
559:            2       192     sun.security.provider.Sun
560:            6       192     org.apache.lucene.util.WeakIdentityMap$IdentityWeakReference
561:            6       192     java.util.Collections$SynchronizedMap
562:            6       192     com.sun.org.apache.xerces.internal.utils.XMLSecurityManager$Limit
563:            12      192     java.util.concurrent.CopyOnWriteArrayList
564:            23      184     org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy
565:            23      184     org.opensolaris.opengrok.analysis.AnalyzerGuru
566:            2       184     java.text.DateFormat$Field[]
567:            11      176     sun.security.pkcs11.TemplateManager$Template
568:            11      176     java.io.FilePermissionCollection
569:            11      176     java.text.NumberFormat$Field
570:            11      176     org.opensolaris.opengrok.util.Executor$SpoolHandler
571:            11      176     java.util.regex.Pattern$Dot
572:            11      176     sun.security.pkcs11.TemplateManager$KeyAndTemplate
573:            11      176     java.util.regex.Pattern$Begin
574:            11      176     java.util.regex.Pattern$CharProperty$1
575:            7       168     java.security.BasicPermissionCollection
576:            7       168     org.apache.lucene.util.packed.Direct16
577:            7       168     java.util.regex.Pattern$5
578:            7       168     org.apache.lucene.document.DateTools$Resolution
579:            7       168     sun.security.x509.AlgorithmId
580:            10      160     org.apache.lucene.index.NumericDocValuesWriter$1
581:            4       160     java.util.ResourceBundle$BundleReference
582:            4       160     java.util.Collections$SingletonMap
583:            4       160     org.opensolaris.opengrok.analysis.archive.BZip2Analyzer
584:            1       152     org.opensolaris.opengrok.configuration.Configuration
585:            4       144     sun.security.x509.RDN[]
586:            6       144     org.apache.lucene.store.IOContext
587:            9       144     java.util.logging.Level$KnownLevel
588:            5       144     java.lang.annotation.ElementType[]
589:            3       144     org.apache.lucene.codecs.BlockTreeTermsWriter
590:            6       144     com.sun.org.apache.xerces.internal.util.Status
591:            3       144     org.opensolaris.opengrok.history.SCCSHistoryParser
592:            6       144     org.opensolaris.opengrok.analysis.executables.ELFAnalyzer$E_Type
593:            2       144     org.apache.tools.tar.TarEntry
594:            9       144     java.nio.channels.spi.AbstractInterruptibleChannel$1
595:            1       136     sun.security.pkcs11.SunPKCS11
596:            2       136     java.math.BigDecimal[]
597:            16      128     org.apache.lucene.index.NormsConsumer
598:            1       128     sun.security.pkcs11.wrapper.CK_TOKEN_INFO
599:            2       128     org.apache.lucene.index.MergePolicy$OneMerge
600:            4       128     java.util.ResourceBundle$LoaderReference
601:            8       128     java.lang.annotation.ElementType
602:            4       128     java.lang.OutOfMemoryError
603:            2       128     java.security.SecureRandom
604:            8       128     java.util.regex.Pattern$CharPropertyNames$3
605:            2       128     sun.security.x509.X509CertImpl
606:            4       128     java.util.concurrent.atomic.AtomicReferenceFieldUpdater$AtomicReferenceFieldUpdaterImpl
607:            4       128     sun.security.util.MemoryCache
608:            4       128     sun.reflect.annotation.AnnotationType
609:            4       128     org.opensolaris.opengrok.analysis.uue.UuencodeAnalyzer
610:            5       120     org.apache.lucene.util.FixedBitSet
611:            5       120     org.opensolaris.opengrok.analysis.FileAnalyzer$Genre
612:            5       120     com.sun.org.apache.xerces.internal.utils.XMLSecurityManager$State
613:            5       120     sun.util.LocaleServiceProviderPool
614:            5       120     org.apache.lucene.util.packed.Packed64SingleBlock$Packed64SingleBlock2
615:            5       120     java.util.Collections$UnmodifiableMap
616:            7       112     java.util.regex.Pattern$Start
617:            2       112     java.util.regex.Pattern$GroupHead[]
618:            7       112     com.sun.management.VMOption$Origin
619:            2       112     java.io.ExpiringCache$1
620:            7       112     sun.security.x509.NetscapeCertTypeExtension$MapEntry
621:            2       112     java.nio.DirectByteBufferR
622:            2       112     sun.security.x509.X509CertInfo
623:            7       112     com.sun.org.apache.xerces.internal.util.FeatureState
624:            7       112     java.io.ObjectStreamClass$ClassDataSlot
625:            1       104     java.util.logging.LogManager$Cleaner
626:            1       104     java.lang.ref.Reference$ReferenceHandler
627:            1       104     org.opensolaris.opengrok.analysis.sql.PLSQLXref
628:            1       104     java.lang.ref.Finalizer$FinalizerThread
629:            6       104     java.io.ObjectStreamClass$ClassDataSlot[]
630:            1       104     org.apache.tools.bzip2.CBZip2InputStream
631:            1       96      com.sun.crypto.provider.SunJCE
632:            1       96      sun.security.pkcs11.Config
633:            1       96      sun.security.jca.ProviderList$1
634:            3       96      org.apache.lucene.index.FieldInfos
635:            1       96      sun.security.rsa.SunRsaSign
636:            1       96      com.sun.net.ssl.internal.ssl.Provider
637:            6       96      org.apache.lucene.codecs.BlockTreeTermsWriter$TermsWriter$FindBlocks
638:            6       96      java.util.regex.Pattern$CharPropertyNames$2
639:            2       96      sun.security.util.MemoryCache$SoftCacheEntry
640:            4       96      java.security.CodeSigner
641:            1       96      sun.security.smartcardio.SunPCSC
642:            1       96      org.jcp.xml.dsig.internal.dom.XMLDSigRI
643:            6       96      sun.misc.PerfCounter
644:            6       96      org.apache.derby.client.am.SqlCode
645:            3       96      sun.misc.URLClassPath
646:            4       96      org.apache.lucene.util.RamUsageEstimator$JvmFeature
647:            3       96      org.apache.lucene.util.BytesRefHash$1
648:            1       96      com.sun.security.sasl.Provider
649:            2       96      org.apache.lucene.index.SegmentWriteState
650:            2       96      org.apache.lucene.index.FrozenBufferedUpdates
651:            1       96      sun.security.pkcs11.Token
652:            1       96      sun.security.ec.SunEC
653:            2       96      java.lang.ThreadGroup
654:            1       96      com.oracle.security.ucrypto.UcryptoProvider
655:            1       88      org.opensolaris.opengrok.analysis.JFlexXref$Style[]
656:            11      88      sun.nio.ch.FileDispatcherImpl
657:            1       88      sun.security.util.ObjectIdentifier[]
658:            1       88      java.lang.Byte[][]
659:            1       88      java.util.logging.FileHandler
660:            2       80      boolean[][]
661:            5       80      java.util.regex.Pattern$BitClass
662:            2       80      org.opensolaris.opengrok.history.PerforceRepository
663:            5       80      org.opensolaris.opengrok.jdbc.InsertQuery
664:            5       80      java.util.regex.Pattern$Slice
665:            5       80      sun.security.jca.ServiceId
666:            2       80      java.util.StringTokenizer
667:            5       80      com.sun.org.apache.xerces.internal.utils.XMLSecurityPropertyManager$State
668:            2       80      org.opensolaris.opengrok.history.ClearCaseRepository
669:            1       80      sun.misc.Launcher$ExtClassLoader
670:            1       80      sun.misc.Launcher$AppClassLoader
671:            5       80      com.sun.org.apache.xerces.internal.util.PropertyState
672:            5       80      org.opensolaris.opengrok.history.CVSHistoryParser$ParseState
673:            2       80      org.opensolaris.opengrok.analysis.lisp.LispAnalyzer
674:            3       72      sun.security.pkcs11.Secmod$DbMode
675:            1       72      java.util.concurrent.ThreadPoolExecutor
676:            3       72      org.apache.lucene.codecs.lucene42.Lucene42TermVectorsFormat
677:            3       72      org.apache.bcel.classfile.ConstantLong
678:            3       72      org.opensolaris.opengrok.analysis.executables.ELFAnalyzer$EI_Data
679:            3       72      org.apache.lucene.store.ByteArrayDataOutput
680:            1       72      org.apache.lucene.util.Version[]
681:            1       72      org.apache.tools.bzip2.CBZip2InputStream$Data
682:            3       72      java.io.StringWriter
683:            3       72      com.sun.org.apache.xerces.internal.utils.XMLSecurityManager$NameMap
684:            1       72      java.util.logging.LogManager$RootLogger
685:            1       72      java.util.logging.LogRecord
686:            1       72      org.apache.bcel.util.ClassPath$PathEntry[]
687:            1       72      java.util.zip.ZipInputStream
688:            3       72      org.apache.lucene.codecs.lucene41.ForUtil
689:            3       72      java.io.FileReader
690:            3       72      org.apache.lucene.index.BufferedUpdatesStream$ApplyDeletesResult
691:            1       72      org.opensolaris.opengrok.history.Repository[]
692:            2       72      sun.security.jca.ProviderConfig[]
693:            1       72      com.oracle.security.ucrypto.UcryptoMech[]
694:            3       72      org.apache.lucene.codecs.lucene41.Lucene41StoredFieldsFormat
695:            3       72      org.opensolaris.opengrok.analysis.executables.ELFAnalyzer$EI_Class
696:            3       72      java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject
697:            3       72      org.apache.lucene.store.FlushInfo
698:            2       64      sun.security.x509.AuthorityKeyIdentifierExtension
699:            4       64      org.apache.lucene.index.FieldInfo$DocValuesType
700:            4       64      org.apache.lucene.store.IOContext$Context
701:            2       64      sun.nio.ch.FileChannelImpl$Unmapper
702:            4       64      java.lang.Boolean
703:            4       64      org.apache.lucene.codecs.lucene40.Lucene40SegmentInfoFormat
704:            2       64      org.apache.lucene.util.RamUsageEstimator$JvmFeature[]
705:            4       64      sun.security.x509.KeyIdentifier
706:            2       64      sun.util.locale.UnicodeLocaleExtension
707:            4       64      sun.security.provider.certpath.X509CertPath
708:            4       64      org.apache.lucene.index.FieldInfo$IndexOptions
709:            2       64      java.io.PrintStream
710:            2       64      java.util.Locale$Category
711:            2       64      java.io.ExpiringCache
712:            3       64      org.apache.lucene.store.RAMOutputStream[]
713:            1       64      sun.reflect.misc.MethodUtil
714:            4       64      org.apache.lucene.index.MergePolicy$MergeTrigger
715:            2       64      java.util.LinkedList$ListItr
716:            4       64      java.util.Collections$SingletonSet
717:            4       64      java.util.jar.JarVerifier$3
718:            4       64      java.text.Normalizer$Form
719:            2       64      java.util.TreeMap$ValueIterator
720:            2       64      sun.nio.fs.UnixPath
721:            4       64      sun.reflect.DelegatingMethodAccessorImpl
722:            2       64      org.apache.lucene.index.DocumentsWriterPerThread$FlushedSegment
723:            4       64      com.sun.beans.WeakCache
724:            2       64      java.lang.VirtualMachineError
725:            4       64      sun.security.pkcs11.wrapper.Functions$Flags
...

[vladak]

Is there a way to check the origin of these 900 MB of char[]s and 100 MB of byte[]s ?

[kahatlen]

I usually open the heap dump in VisualVM (the jvisualvm command in the JDK's bin directory) and inspect some sample instances. There's a function that shows the nearest gc root.

[vladak]

Dalibor did a dissection in Eclipse MAT and found out couple of interesting things.

[vladak]

We (me and Dalibor) discovered that the problem is JarAnalyzer indeed. There are cca 44000 /org.apache.lucene.document.TextField/ objects.

One of the interesting observation is that these objects are not referenced (some 890 MB if I remember correctly). This means GC is not able to keep up with the freeing, hence the GC error. In total there was 900 MB of unreferenced objects, waiting for GC to harvest them. This corresponds to the histogram data above - 890 MB of the chars was unreferenced, the remaining (so that it sums up to 928206952 bytes) were referenced.

[vladak]

I used JAVA_OPTS="-XX:-UseGCOverheadLimit -Xmx2048m" to perform the indexing and it finished fine. Once Jar analyzer is fixed this will be no longer needed.