【求助】Impalad启动时JVM崩溃

最近往集群加了几个节点，在启动Impalad时，JVM崩溃，找了impala版本是cdh5-2.7.0_5.9.0,到出错的init.cc的213行是一个打印操作系统版本信息的一个调试操作，实在没辙了，有没有遇到过这个问题的？日志如下，
[mw_shl_code=applescript,true]        init.cc:213       
OS version: Linux version 3.10.0-514.21.2.el7.x86_64 (builder@kbuilder.dev.centos.org) (gcc version 4.8.5 20150623 (Red Hat 4.8.5-11) (GCC) ) #1 SMP Tue Jun 20 12:24:47 UTC 2017
Clock: clocksource: 'tsWrote minidump to /data/cloudera/var/log/impala-minidumps/impalad/5c4d3499-f55c-c204-74d4c233-4d58fef9.dmp
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGBUS (0x7) at pc=0x00007f73842cb47d, pid=97201, tid=140134281087232
#
# JRE version:  (7.0_67-b01) (build )
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.65-b04 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# j  java.lang.Object.<clinit>()V+0
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /run/cloudera-scm-agent/process/1856-impala-IMPALAD/hs_err_pid97201.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#
c', clockid_t: CLOCK_MONOTONIC[/mw_shl_code]

可能是文件句柄的问题。尝试下面操作，假如你的Linux为Ubuntu，用户为Impalad在文件 /etc/security/limits.conf 添加一行，如:

Impalad  -       nofile 32768

配nproc hard 和 softlimits. 在文件 /etc/security/limits.conf 添加一行下面内容:
Impalad soft/hard nproc 32000


在 /etc/pam.d/common-session 加上这一行:


session required  pam_limits.so


否则在 /etc/security/limits.conf上的配置不会生效.

还有注销再登录，这些配置才能生效!

langke93 发表于 2017-6-24 15:07
可能是文件句柄的问题。尝试下面操作，假如你的Linux为Ubuntu，用户为Impalad在文件 /etc/security/limits. ...

[mw_shl_code=applescript,true]cloudera-scm    soft  nofile  32768
cloudera-scm    soft  nproc   65536
cloudera-scm    hard  nofile  1048576
cloudera-scm    hard  nproc   unlimited
cloudera-scm    hard  memlock unlimited
[/mw_shl_code]
我用的是CDH这部分CDH有默认的配置

caoyun_87 发表于 2017-6-26 10:25
[mw_shl_code=applescript,true]cloudera-scm    soft  nofile  32768
cloudera-scm    soft  nproc   ...

看这个日志了吗
/run/cloudera-scm-agent/process/1856-impala-IMPALAD/hs_err_pid97201.log

qcbb001 发表于 2017-6-26 13:24
看这个日志了吗
/run/cloudera-scm-agent/process/1856-impala-IMPALAD/hs_err_pid97201.log

[mw_shl_code=applescript,true]Stack: [0x00007ffccd6ba000,0x00007ffccd7ba000],  sp=0x00007ffccd7b1080,  free space=988k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
j  java.lang.Object.<clinit>()V+0
v  ~StubRoutines::call_stub
V  [libjvm.so+0x5ffb65]  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x365
V  [libjvm.so+0x5fe5c8]  JavaCalls::call(JavaValue*, methodHandle, JavaCallArguments*, Thread*)+0x28
V  [libjvm.so+0x5c2a3a]  instanceKlass::call_class_initializer(Thread*)+0xca
V  [libjvm.so+0x5c2c94]  instanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x234
V  [libjvm.so+0x5c30ea]  instanceKlass::initialize(Thread*)+0x6a
V  [libjvm.so+0x5c2f2b]  instanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x4cb
V  [libjvm.so+0x5c30ea]  instanceKlass::initialize(Thread*)+0x6a
V  [libjvm.so+0x95ab02]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x402
V  [libjvm.so+0x638a34]  JNI_CreateJavaVM+0x74
[/mw_shl_code]

里面比较有用的是上面这些信息，堆栈是很空的。

[mw_shl_code=applescript,true]Heap
PSYoungGen      total 149504K, used 2570K [0x0000000759e00000, 0x0000000764480000, 0x0000000800000000)
  eden space 128512K, 2% used [0x0000000759e00000,0x000000075a082960,0x0000000761b80000)
  from space 20992K, 0% used [0x0000000763000000,0x0000000763000000,0x0000000764480000)
  to   space 20992K, 0% used [0x0000000761b80000,0x0000000761b80000,0x0000000763000000)
ParOldGen       total 339968K, used 0K [0x000000060da00000, 0x0000000622600000, 0x0000000759e00000)
  object space 339968K, 0% used [0x000000060da00000,0x000000060da00000,0x0000000622600000)
[/mw_shl_code]