Hbase 总结(4)

2014-11-17 11:30:50,338 INFO [Thread-32] mapred.LocalJobRunner: Waiting for map tasks

2014-11-17 11:30:50,338 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: Starting task:

attempt_local1718376827_0001_m_000000_0

2014-11-17 11:30:50,365 INFO [LocalJobRunner Map Task Executor #0] mapred.Task: Using ResourceCalculatorProcessTree : [ ]

2014-11-17 11:30:50,368 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Processing split:

file:/BIDATA/hadoop/jyy/input/TM_CORP_SNMBR_TOP35_M.txt:0+84 2014-11-17 11:30:50,377 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Map output collector class =

org.apache.hadoop.mapred.MapTask$MapOutputBuffer

2014-11-17 11:30:50,405 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)

2014-11-17 11:30:50,405 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: mapreduce.task.io.sort.mb: 100

2014-11-17 11:30:50,405 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: soft limit at 83886080

2014-11-17 11:30:50,405 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: bufstart = 0; bufvoid = 104857600

2014-11-17 11:30:50,405 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: kvstart = 26214396; length = 6553600

2014-11-17 11:30:50,428 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner:

2014-11-17 11:30:50,428 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Starting flush of map output

2014-11-17 11:30:50,428 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Spilling map output

2014-11-17 11:30:50,428 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: bufstart = 0; bufend = 104; bufvoid = 104857600 2014-11-17 11:30:50,428 INFO [LocalJobRunner Map Task Executor #0]

mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214392(104857568); length = 5/6553600

2014-11-17 11:30:50,434 INFO [LocalJobRunner Map Task Executor #0] mapred.MapTask: Finished spill 0

2014-11-17 11:30:50,436 INFO [LocalJobRunner Map Task Executor #0] mapred.Task: Task:attempt_local1718376827_0001_m_000000_0 is done. And is in the process of committing

2014-11-17 11:30:50,442 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: map

2014-11-17 11:30:50,442 INFO [LocalJobRunner Map Task Executor #0] mapred.Task: Task 'attempt_local1718376827_0001_m_000000_0' done.

2014-11-17 11:30:50,442 INFO [LocalJobRunner Map Task Executor #0] mapred.LocalJobRunner: Finishing task:

attempt_local1718376827_0001_m_000000_0

2014-11-17 11:30:50,442 INFO [Thread-32] mapred.LocalJobRunner: Map task executor complete.

2014-11-17 11:30:50,451 INFO [Thread-32] mapred.Task: Using ResourceCalculatorProcessTree : [ ]

2014-11-17 11:30:50,455 INFO [Thread-32] mapred.Merger: Merging 1 sorted segments

2014-11-17 11:30:50,459 INFO [Thread-32] mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 98 bytes

2014-11-17 11:30:50,459 INFO [Thread-32] mapred.LocalJobRunner: 2014-11-17 11:30:50,463 INFO [Thread-32] Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords

2014-11-17 11:30:50,477 INFO [Thread-32] util.ChecksumType: Checksum using org.apache.hadoop.util.PureJavaCrc32

2014-11-17 11:30:50,478 INFO [Thread-32] util.ChecksumType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C

2014-11-17 11:30:50,551 INFO [Thread-32] mapred.Task:

Task:attempt_local1718376827_0001_r_000000_0 is done. And is in the process of committing

2014-11-17 11:30:50,552 INFO [Thread-32] mapred.LocalJobRunner: 2014-11-17 11:30:50,552 INFO [Thread-32] mapred.Task: Task

attempt_local1718376827_0001_r_000000_0 is allowed to commit now

2014-11-17 11:30:50,553 INFO [Thread-32] output.FileOutputCommitter: Saved output of task 'attempt_local1718376827_0001_r_000000_0' to

file:/BIDATA/hadoop/jyy/output/_temporary/0/task_local1718376827_0001_r_000000

2014-11-17 11:30:50,554 INFO [Thread-32] mapred.LocalJobRunner: Read 9 entries of class java.util.TreeSet(984) > reduce

2014-11-17 11:30:50,554 INFO [Thread-32] mapred.Task: Task 'attempt_local1718376827_0001_r_000000_0' done.

2014-11-17 11:30:51,306 INFO [main] mapreduce.Job: Job job_local1718376827_0001 running in uber mode : false

2014-11-17 11:30:51,308 INFO [main] mapreduce.Job: map 100% reduce 100% 2014-11-17 11:30:51,310 INFO [main] mapreduce.Job: Job job_local1718376827_0001 completed successfully

2014-11-17 11:30:51,328 INFO [main] mapreduce.Job: Counters: 28 File System Counters

FILE: Number of bytes read=40212818 FILE: Number of bytes written=41040346 FILE: Number of read operations=0

FILE: Number of large read operations=0 FILE: Number of write operations=0 Map-Reduce Framework Map input records=2 Map output records=2 Map output bytes=104

Map output materialized bytes=114 Input split bytes=120

Combine input records=0 Combine output records=0 Reduce input groups=2 Reduce shuffle bytes=0 Reduce input records=2 Reduce output records=18 Spilled Records=4 Shuffled Maps =0 Failed Shuffles=0

Merged Map outputs=0 GC time elapsed (ms)=0 CPU time spent (ms)=0

Physical memory (bytes) snapshot=0 Virtual memory (bytes) snapshot=0

Total committed heap usage (bytes)=505151488 ImportTsv

Bad Lines=0

File Input Format Counters Bytes Read=84

File Output Format Counters Bytes Written=1890

[9]使用completebulkload加载数据

hbase org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /BIDATA/hadoop/jyy/output TM_CORP_SNMBR_TOP35_M

hbase(main):002:0> [hadoop@bogon ~]$ hbase

org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles /BIDATA/hadoop/jyy/output TM_CORP_SNMBR_TOP35_M

2014-11-17 11:42:10,930 WARN [main] util.NativeCodeLoader: Unable to load

native-hadoop library for your platform... using builtin-java classes where applicable 2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client

environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT 2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client environment:host.name=bogon

2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client environment:java.version=1.7.0_60

2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation

2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client environment:java.home=/BIDATA/hadoop/jdk7/jre

2014-11-17 11:42:11,033 INFO [main] zookeeper.ZooKeeper: Client

environment:java.class.path=/BIDATA/hadoop/hbase/conf:/BIDATA/hadoop/jdk7/lib/tools.jar:/BIDATA/hadoop/hbase:/BIDATA/hadoop/hbase/lib/activation-1.1.jar:/BIDATA/hadoop/hbase/lib/aopalliance-1.0.jar:/BIDATA/hadoop/hbase/lib/asm-3.1.jar:/BIDATA/hadoop/hbase/lib/avro-

1.7.4.jar:/BIDATA/hadoop/hbase/lib/commons-beanutils-1.7.0.jar:/BIDATA/hadoop/hbase/lib/commons-beanutils-core-1.8.0.jar:/BIDATA/hadoop/hbase/lib/commons-cli-1.2.jar:/BIDATA/hadoop/hbase/lib/commons-codec-1.7.jar:/BIDATA/hadoop/hbase/lib/commons-collections-3.2.1.jar:/BIDATA/hadoop/hbase/lib/commons-compress-1.4.1.jar:/BIDATA/hadoop/hbase/lib/commons-configuration-1.6.jar:/BIDATA/hadoop/hbase/lib/commons-daemon-1.0.13.jar:/BIDATA/hadoop/hbase/lib/commons-digester-1.8.jar:/BIDATA/hadoop/hbase/lib/commons-el-1.0.jar:/BIDATA/hadoop/hbase/lib/commons-httpclient-3.1.jar:/BIDATA/hadoop/hbase/lib/commons-io-2.4.jar:/BIDATA/hadoop/hbase/lib/commons-lang-2.6.jar:/BIDATA/hadoop/hbase/lib/commons-logging-1.1.1.jar:/BIDATA/hadoop/hbase/lib/commons-math-2.1.jar:/BIDATA/hadoop/hbase/lib/commons-net-3.1.jar:/BIDATA/hadoop/hbase/lib/findbugs-annotations-1.3.9-1.jar:/BIDATA/hadoop/hbase/lib/gmbal-api-only-3.0.0-b023.jar:/BIDATA/hadoop/hbase/lib/grizzly-framework-2.1.2.jar:/BIDATA/hadoop/hbase/lib/grizzly-http-2.1.2.jar:/BIDATA/hadoop/hbase/lib/grizzly-http-server-2.1.2.jar:/BIDATA/hadoop/hbase/lib/grizzly-http-servlet-2.1.2.jar:/BIDATA/hadoop/hbase/lib/grizzly-rcm-2.1.2.jar:/BIDATA/hadoop/hbase/lib/guava-12.0.1.jar:/BIDATA/hadoop/hbase/lib/guice-3.0.jar:/BIDATA/hadoop/hbase/lib/guice-servlet-3.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-annotations-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-auth-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-client-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-common-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-hdfs-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-hdfs-2.2.0-tests.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-app-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-common-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-core-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-jobclient-2.2.0-tests.jar:/BIDATA/hadoop/hbase/lib/hadoop-mapreduce-client-shuffle-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-yarn-api-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-yarn-client-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-yarn-common-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-yarn-server-common-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hadoop-yarn-server-nodemanager-2.2.0.jar:/BIDATA/hadoop/hbase/lib/hamcrest-core-1.3.jar:/BIDATA/hadoop/hbase/lib/hbase-client-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-common-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-common-0.98.3-hadoop2-

tests.jar:/BIDATA/hadoop/hbase/lib/hbase-examples-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-hadoop2-compat-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-hadoop-compat-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-it-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-it-0.98.3-hadoop2-tests.jar:/BIDATA/hadoop/hbase/lib/hbase-prefix-tree-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-protocol-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-server-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-server-0.98.3-hadoop2-tests.jar:/BIDATA/hadoop/hbase/lib/hbase-shell-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-testing-util-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/hbase-thrift-0.98.3-hadoop2.jar:/BIDATA/hadoop/hbase/lib/high-scale-lib-1.1.1.jar:/BIDATA/hadoop/hbase/lib/htrace-core-2.04.jar:/BIDATA/hadoop/hbase/lib/httpclient-4.1.3.jar:/BIDATA/hadoop/hbase/lib/httpcore-4.1.3.jar:/BIDATA/hadoop/hbase/lib/jackson-core-asl-1.8.8.jar:/BIDATA/hadoop/hbase/lib/jackson-jaxrs-1.8.8.jar:/BIDATA/hadoop/hbase/lib/jackson-mapper-asl-1.8.8.jar:/BIDATA/hadoop/hbase/lib/jackson-xc-1.8.8.jar:/BIDATA/hadoop/hbase/lib/jamon-runtime-2.3.1.jar:/BIDATA/hadoop/hbase/lib/jasper-compiler-5.5.23.jar:/BIDATA/hadoop/hbase/lib/jasper-runtime-5.5.23.jar:/BIDATA/hadoop/hbase/lib/javax.inject-1.jar:/BIDATA/hadoop/hbase/lib/javax.servlet-3.1.jar:/BIDATA/hadoop/hbase/lib/javax.servlet-api-3.0.1.jar:/BIDATA/hadoop/hbase/lib/jaxb-api-2.2.2.jar:/BIDATA/hadoop/hbase/lib/jaxb-impl-2.2.3-1.jar:/BIDATA/hadoop/hbase/lib/jersey-client-1.9.jar:/BIDATA/hadoop/hbase/lib/jersey-core-1.8.jar:/BIDATA/hadoop/hbase/lib/jersey-grizzly2-1.9.jar:/BIDATA/hadoop/hbase/lib/jersey-guice-1.9.jar:/BIDATA/hadoop/hbase/lib/jersey-json-1.8.jar:/BIDATA/hadoop/hbase/lib/jersey-server-1.8.jar:/BIDATA/hadoop/hbase/lib/jersey-test-framework-core-1.9.jar:/BIDATA/hadoop/hbase/lib/jersey-test-framework-grizzly2-1.9.jar:/BIDATA/hadoop/hbase/lib/jets3t-0.6.1.jar:/BIDATA/hadoop/hbase/lib/jettison-1.3.1.jar:/BIDATA/hadoop/hbase/lib/jetty-6.1.26.jar:/BIDATA/hadoop/hbase/lib/jetty-sslengine-6.1.26.jar:/BIDATA/hadoop/hbase/lib/jetty-util-6.1.26.jar:/BIDATA/hadoop/hbase/lib/jruby-complete-1.6.8.jar:/BIDATA/hadoop/hbase/lib/jsch-0.1.42.jar:/BIDATA/hadoop/hbase/lib/jsp-2.1-6.1.14.jar:/BIDATA/hadoop/hbase/lib/jsp-api-2.1-6.1.14.jar:/BIDATA/hadoop/hbase/lib/jsr305-1.3.9.jar:/BIDATA/hadoop/hbase/lib/junit-4.11.jar:/BIDATA/hadoop/hbase/lib/libthrift-

百度搜索“77cn”或“免费范文网”即可找到本站免费阅读全部范文。收藏本站方便下次阅读，免费范文网，提供经典小说教育文库Hbase 总结(4)在线全文阅读。