分享

使用sqoop 将hbase数据导入mysql 异常是key重复的导入

昨天在导入数据的时候
Caused by: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'EEE1007-J001-2016-01-01 00:00:00' for key 'PRIMARY'
        at org.apache.sqoop.mapred
16/06/15 19:23:09 INFO mapred.JobClient: Task Id : attempt_201606141801_0043_m_000003_2, Status : FAILED
java.io.IOException: Can't export data, please check failed map task logs
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:112)
        at org.apache.sqoop.mapreduce.TextExportMapper.map(TextExportMapper.java:39)
        at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:140)
        at org.apache.sqoop.mapreduce.AutoProgressMapper.run(AutoProgressMapper.java:64)
        at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330)
        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1438)
        at org.apache.hadoop.mapred.Child.main(Child.java:262)
Caused by: java.io.IOException: com.mysql.jdbc.exceptions.jdbc4.MySQLIntegrityConstraintViolationException: Duplicate entry 'EEE1007-J001-2016-01-01 00:00:00' for key 'PRIMARY'
        at org.apache.sqoop.mapred
16/06/15 19:23:18 INFO mapred.JobClient: Job complete: job_201606141801_0043
16/06/15 19:23:18 INFO mapred.JobClient: Counters: 26
16/06/15 19:23:18 INFO mapred.JobClient:   File System Counters
16/06/15 19:23:18 INFO mapred.JobClient:     FILE: Number of bytes read=0
16/06/15 19:23:18 INFO mapred.JobClient:     FILE: Number of bytes written=589536
16/06/15 19:23:18 INFO mapred.JobClient:     FILE: Number of read operations=0
16/06/15 19:23:18 INFO mapred.JobClient:     FILE: Number of large read operations=0
16/06/15 19:23:18 INFO mapred.JobClient:     FILE: Number of write operations=0
16/06/15 19:23:18 INFO mapred.JobClient:     HDFS: Number of bytes read=519752643
16/06/15 19:23:18 INFO mapred.JobClient:     HDFS: Number of bytes written=0
16/06/15 19:23:18 INFO mapred.JobClient:     HDFS: Number of read operations=9336
16/06/15 19:23:18 INFO mapred.JobClient:     HDFS: Number of large read operations=0
16/06/15 19:23:18 INFO mapred.JobClient:     HDFS: Number of write operations=0
16/06/15 19:23:18 INFO mapred.JobClient:   Job Counters
16/06/15 19:23:18 INFO mapred.JobClient:     Failed map tasks=1
16/06/15 19:23:18 INFO mapred.JobClient:     Launched map tasks=7
16/06/15 19:23:18 INFO mapred.JobClient:     Data-local map tasks=6
16/06/15 19:23:18 INFO mapred.JobClient:     Rack-local map tasks=1
16/06/15 19:23:18 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=1319967
16/06/15 19:23:18 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=0
16/06/15 19:23:18 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
16/06/15 19:23:18 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
16/06/15 19:23:18 INFO mapred.JobClient:   Map-Reduce Framework
16/06/15 19:23:18 INFO mapred.JobClient:     Map input records=6205617
16/06/15 19:23:18 INFO mapred.JobClient:     Map output records=6205617
16/06/15 19:23:18 INFO mapred.JobClient:     Input split bytes=236628
16/06/15 19:23:18 INFO mapred.JobClient:     Spilled Records=0
16/06/15 19:23:18 INFO mapred.JobClient:     CPU time spent (ms)=412460
16/06/15 19:23:18 INFO mapred.JobClient:     Physical memory (bytes) snapshot=516374528
16/06/15 19:23:18 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3176611840
16/06/15 19:23:18 INFO mapred.JobClient:     Total committed heap usage (bytes)=443416576
16/06/15 19:23:18 INFO mapreduce.ExportJobBase: Transferred 495.6747 MB in 395.6629 seconds (1.2528 MB/sec)
16/06/15 19:23:18 INFO mapreduce.ExportJobBase: Exported 6205617 records.
16/06/15 19:23:18 ERROR tool.ExportTool: Error during export: Export job failed!

然后我去mysql数据库 查询 数据库里面有这条数据 但是导入的时候 报key重复了  想的是重复导入了 怎么解决这个问题了

已有(2)人评论

跳转到指定楼层
请叫我野区养猪 发表于 2016-6-16 08:30:26
之前也报错了 把报错的那条数据删掉 然后这条数据又出来了 mysql那张表是空表 查询mysql表数据条数和hbase条数差了2000条
回复

使用道具 举报

langke93 发表于 2016-6-16 11:26:15
无论是哪种数据库,主键都是不允许重复的。楼主为何会出现重复key的情况。弄清这个,或许才能解决这个问题
回复

使用道具 举报

您需要登录后才可以回帖 登录 | 立即注册

本版积分规则

关闭

推荐上一条 /2 下一条