Thanks, GC pauses looks like a very promising lead. We have 2 servers and the crashes appear to happen alternately between the 2.
The gc logs do reveal some pretty slow Full GC and CMS. Any hints on tuning this?
Current memory/gc options are:
-Xms30000m -Xmx30000m -XX:MaxPermSize=160m -XX:+UseCompressedOops -XX:NewSize=256m -XX:+UseConcMarkSweepGC -XX:+UseParNewGC -XX:+ExplicitGCInvokesConcurrent -XX:CMSInitiatingOccupancyFraction=65
...
1060.618: [Full GC 1060.618: [CMS: 5398766K->4439583K(30336640K), 11.9607200 secs] 5497761K->4439583K(30681664K), [CMS Perm : 74138K->73961K(74432K)], 11.9610270 secs] [Times: user=11.96 sys=0.00, real=11.96 secs]
...
202818.862: [GC [1 CMS-initial-mark: 15168671K(30336640K)] 15174214K(30681664K), 0.0338890 secs] [Times: user=0.01 sys=0.00, real=0.03 secs]
202818.896: [CMS-concurrent-mark-start]
202820.782: [CMS-concurrent-mark: 1.884/1.886 secs] [Times: user=9.98 sys=0.05, real=1.89 secs]
202820.782: [CMS-concurrent-preclean-start]
202820.878: [CMS-concurrent-preclean: 0.096/0.096 secs] [Times: user=0.11 sys=0.00, real=0.09 secs]
202820.878: [CMS-concurrent-abortable-preclean-start]
CMS: abort preclean due to time 202825.987: [CMS-concurrent-abortable-preclean: 4.504/5.109 secs] [Times: user=5.36 sys=0.00, real=5.11 secs]
202825.989: [GC[YG occupancy: 303127 K (345024 K)]202825.989: [Rescan (parallel) , 0.3003800 secs]202826.289: [weak refs processing, 0.1735160 secs] [1 CMS-remark: 15168671K(30336640K)] 15471798K(30681664K), 0.4853680 secs] [Times: user=5.20 sys=0.00, real=0.49 secs]
202826.475: [CMS-concurrent-sweep-start]
202826.761: [GC 202826.761: [ParNew: 311158K->4899K(345024K), 0.0259320 secs] 15179931K->14874599K(30681664K), 0.0261830 secs] [Times: user=0.42 sys=0.00, real=0.02 secs]
202843.170: [GC 202843.170: [ParNew: 311587K->3560K(345024K), 0.0183940 secs] 5621420K->5314131K(30681664K), 0.0186490 secs] [Times: user=0.29 sys=0.00, real=0.02 secs]
202844.061: [CMS-concurrent-sweep: 17.524/17.587 secs] [Times: user=23.48 sys=0.00, real=17.59 secs]
202844.061: [CMS-concurrent-reset-start]
202844.231: [CMS-concurrent-reset: 0.168/0.170 secs] [Times: user=0.11 sys=0.11, real=0.17 secs]