日志

Window中调试HBase问题总结

已有 1258 次阅读2014-12-31 22:56

　1.好久没用log4j了，转到logback好多年了，hbase程序运行时，报缺少log4j配置，那么，就转去logback吧（以下的XXX表示版本号）。

　　原先lib包里面有log4j-XXX.jar、slf4j-api-XXX.jar、slf4j-log4j12-XXX.jar，干掉log4j-XXX.jar和slf4j-log4j12-XXX.jar，加入jcl-over-slf4j-XXX.jar，log4j-over-slf4j-XXX.jar、logback-classic-XXX.jar、logback-core-XXX.jar，至于原先的slf4j-api这个包，可以顺手替换成新的，然后，加入一个logback.xml到classpath下

<?xml version="1.0" encoding="UTF-8"?>








<configuration scan="true" scanPeriod="120 seconds" debug="false">
<appender name="FileApp" class="ch.qos.logback.core.rolling.RollingFileAppender">

<filter class="ch.qos.logback.classic.filter.ThresholdFilter">
<level>WARN</level>
</filter>
<File>log/HBaseClient.log</File>
<encoder>
<pattern>%d [%t] %-5p %c - %m%n</pattern>
</encoder>
<rollingPolicy class="ch.qos.logback.core.rolling.FixedWindowRollingPolicy">
<maxIndex>10</maxIndex>
<FileNamePattern>log/HBaseClient.log.%i</FileNamePattern>
</rollingPolicy>
<triggeringPolicy class="ch.qos.logback.core.rolling.SizeBasedTriggeringPolicy">
<MaxFileSize>1024KB</MaxFileSize>
</triggeringPolicy>
</appender>
<appender name="ConApp" class="ch.qos.logback.core.ConsoleAppender">
<encoder>
<pattern>%d [%t] %-5p %c - %m%n</pattern>
</encoder>
</appender>
<root level="ERROR">
<appender-ref ref="ConApp"/>
<appender-ref ref="FileApp"/>
</root>
</configuration>

这样，log4j和系统的日志系统就被logback接管了，至于为啥不用log4j，用logback，这个。。。自己去看吧，有空我可能会写点啥，logback有点小bug，就是有的linux系统下会找不到配置文件，以前修复过，不知道新版本有处理掉没。

　　2.Could not locate executable null\bin\winutils.exe in the Hadoop binaries

　　老掉牙的问题了，系统变量设置HADOOP_HOME，我还是不想去设置环境变量，还是一行代码来得快

System.setProperty("hadoop.home.dir", "G:/hadoop/hadoop-2.4.1");

3.Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

　　本地库是为了提高性能和包含一些java不支持的实现，比如集群中设置支持gzip lzo压缩后，在对压缩文件进行读取或者对输入文件压缩的时候要使用到hadoop的本地库。

　　找不到本地库，linux下可以自己编译一下，官方文档中也有介绍http://hadoop.apache.org/docs/r2.4.1/hadoop-project-dist/hadoop-common/NativeLibraries.html。

　　windows下。。。。我以前是在https://github.com/srccodes/hadoop-common-2.2.0-bin/tree/master/bin下载的，但是我win7 64位，就用不了，注册dll文件的时候，就报错了。官方文档中写着“The native hadoop library is supported on *nix platforms only.”。。。

　　找到一个windows下编译hadoop本地库的地址http://www.srccodes.com/p/article/38/build-install-configure-run-apache-hadoop-2.2.0-microsoft-windows-os

　　看了一遍，想想不划算，算了，到时候肯定是丢去linux系统下跑的，折腾这个干嘛啊，linux下又没有问题。

4.程序运行正常，但是当log级别改为Info时，突然发现了这么一条

2014-08-04 16:19:22,942 [main-SendThread(Master:2222)] INFO org.apache.zookeeper.ClientCnxn - Opening socket connection to server Master/192.168.117.128:2222. Will not attempt to authenticate using SASL (java.lang.SecurityException: 无法定位登录配置)

看到个Exception，什么情况？觉得哪里不对，但是程序又是一切正常的。

　　搜索了一通，发现好多一样的问题，但是貌似和我的没啥关系，别人是程序跑不通，我是程序正常得很。

　　有说是zookeeper版本不一致的，我看了下hadoop和hbase中的确实有区别，一个zookeeper-3.4.5.jar，一个zookeeper-3.4.6.jar，但是我总觉得不对，新旧怎么可能会不兼容啊。那就直接统一了试试，连客户端程序的都统一了，结果那个info信息还是存在，就和不换是一样的，果然不是这里的问题！

　　有说是配置和hosts的，我这里都没这问题的，都配置过了的。加参数？我在配置中都写好的，直接打印一看就知道有了。

　　这只是一个info信息，很奇怪为啥是个info，照说info级别的都是不影响啥的信息，程序确实是很正常的跑，但是看到这个信息就是不舒服。

　　自己查代码去算了，根据日志，是在ClientCnxn.java中出现的信息，好吧，查看源码

private void logStartConnect(InetSocketAddress addr) {
String msg = "Opening socket connection to server " + addr;
if (zooKeeperSaslClient != null) {
msg += ". " + zooKeeperSaslClient.getConfigStatus();
}
LOG.info(msg);
}

　汗，还得换去查ZooKeeperSaslClient.java，继续吧

public ZooKeeperSaslClient(final String serverPrincipal)
throws LoginException {
/**
* ZOOKEEPER-1373: allow system property to specify the JAAS
* configuration section that the zookeeper client should use.
* Default to "Client".
*/
String clientSection = System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client");
// Note that 'Configuration' here refers to javax.security.auth.login.Configuration.
AppConfigurationEntry entries[] = null;
RuntimeException runtimeException = null;
try {
entries = Configuration.getConfiguration().getAppConfigurationEntry(clientSection);
} catch (SecurityException e) {
// handle below: might be harmless if the user doesn't intend to use JAAS authentication.
runtimeException = e;
} catch (IllegalArgumentException e) {
// third party customized getAppConfigurationEntry could throw IllegalArgumentException when JAAS
// configuration isn't set. We can reevaluate whether to catch RuntimeException instead when more
// different types of RuntimeException found
runtimeException = e;
}
if (entries != null) {
this.configStatus = "Will attempt to SASL-authenticate using Login Context section '" + clientSection + "'";
this.saslClient = createSaslClient(serverPrincipal, clientSection);
} else {
// Handle situation of clientSection's being null: it might simply because the client does not intend to
// use SASL, so not necessarily an error.
saslState = SaslState.FAILED;
String explicitClientSection = System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY);
if (explicitClientSection != null) {
// If the user explicitly overrides the default Login Context, they probably expected SASL to
// succeed. But if we got here, SASL failed.
if (runtimeException != null) {
throw new LoginException("Zookeeper client cannot authenticate using the " + explicitClientSection +
" section of the supplied JAAS configuration: '" +
System.getProperty(Environment.JAAS_CONF_KEY) + "' because of a " +
"RuntimeException: " + runtimeException);
} else {
throw new LoginException("Client cannot SASL-authenticate because the specified JAAS configuration " +
"section '" + explicitClientSection + "' could not be found.");
}
} else {
// The user did not override the default context. It might be that they just don't intend to use SASL,
// so log at INFO, not WARN, since they don't expect any SASL-related information.
String msg = "Will not attempt to authenticate using SASL ";
if (runtimeException != null) {
msg += "(" + runtimeException + ")";
} else {
msg += "(unknown error)";
}
this.configStatus = msg;
this.isSASLConfigured = false;
}
if (System.getProperty(Environment.JAAS_CONF_KEY) != null) {
// Again, the user explicitly set something SASL-related, so they probably expected SASL to succeed.
if (runtimeException != null) {
throw new LoginException("Zookeeper client cannot authenticate using the '" +
System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client") +
"' section of the supplied JAAS configuration: '" +
System.getProperty(Environment.JAAS_CONF_KEY) + "' because of a " +
"RuntimeException: " + runtimeException);
} else {
throw new LoginException("No JAAS configuration section named '" +
System.getProperty(ZooKeeperSaslClient.LOGIN_CONTEXT_NAME_KEY, "Client") +
"' was found in specified JAAS configuration file: '" +
System.getProperty(Environment.JAAS_CONF_KEY) + "'.");
}
}
}
}