hadoop 报错:Error: Could not find or load main class

时间:2021-01-22 15:57:37

Error: Could not find or load main class


报这个错是因为当前运行的 .class  文件不在 hadoop-env.sh文件中
export HADOOP_CLASSPATH=${HADOOP_HOME}//你运行的*.class 文件所在路径


[john@localhost ch02]$ ls
input  output  pom.xml  src  target
[john@localhost ch02]$ export $HADOOP_CLASSPATH=target/classes
bash: export: `build/classes=target/classes': not a valid identifier
[john@localhost ch02]$ export HADOOP_CLASSPATH=target/classes
[john@localhost ch02]$ echo $HADOOP_CLASSPATH
target/classes
[john@localhost ch02]$ hadoop MaxTemperature input/sample.txt output
14/12/13 12:27:14 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/12/13 12:27:14 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
Exception in thread "main" org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/output already exists
    at org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.checkOutputSpecs(FileOutputFormat.java:146)
    at org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:562)
    at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:432)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
    at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:415)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
    at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
    at org.apache.hadoop.mapreduce.Job.waitForCompletion(Job.java:1314)
    at MaxTemperature.main(MaxTemperature.java:31)

------

上面信息报错,output文件已经存在。hadoop输出文件路径需要,hadoop自己来建,我们创建就会报上面的错误。

修改方法:

[john@localhost ch02]$ rm -rf output/
[john@localhost ch02]$ ls
input  pom.xml  src  target
[john@localhost ch02]$ hadoop MaxTemperature input/sample.txt output
14/12/13 12:28:40 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
14/12/13 12:28:40 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
14/12/13 12:28:41 WARN mapreduce.JobSubmitter: Hadoop command-line option parsing not performed. Implement the Tool interface and execute your application with ToolRunner to remedy this.
14/12/13 12:28:41 WARN mapreduce.JobSubmitter: No job jar file set.  User classes may not be found. See Job or Job#setJar(String).
14/12/13 12:28:41 INFO input.FileInputFormat: Total input paths to process : 1
14/12/13 12:28:41 INFO mapreduce.JobSubmitter: number of splits:1
14/12/13 12:28:42 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1165163551_0001
14/12/13 12:28:42 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
14/12/13 12:28:42 INFO mapreduce.Job: Running job: job_local1165163551_0001
14/12/13 12:28:42 INFO mapred.LocalJobRunner: OutputCommitter set in config null
14/12/13 12:28:42 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
14/12/13 12:28:42 INFO mapred.LocalJobRunner: Waiting for map tasks
14/12/13 12:28:42 INFO mapred.LocalJobRunner: Starting task: attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/12/13 12:28:43 INFO mapred.MapTask: Processing split: file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/input/sample.txt:0+1485
14/12/13 12:28:43 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
14/12/13 12:28:43 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
14/12/13 12:28:43 INFO mapred.MapTask: soft limit at 83886080
14/12/13 12:28:43 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
14/12/13 12:28:43 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
14/12/13 12:28:43 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
14/12/13 12:28:43 INFO mapred.LocalJobRunner:
14/12/13 12:28:43 INFO mapred.MapTask: Starting flush of map output
14/12/13 12:28:43 INFO mapred.MapTask: Spilling map output
14/12/13 12:28:43 INFO mapred.MapTask: bufstart = 0; bufend = 99; bufvoid = 104857600
14/12/13 12:28:43 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
14/12/13 12:28:43 INFO mapred.MapTask: Finished spill 0
14/12/13 12:28:43 INFO mapred.Task: Task:attempt_local1165163551_0001_m_000000_0 is done. And is in the process of committing
14/12/13 12:28:43 INFO mapred.LocalJobRunner: map
14/12/13 12:28:43 INFO mapred.Task: Task 'attempt_local1165163551_0001_m_000000_0' done.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO mapred.LocalJobRunner: map task executor complete.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Waiting for reduce tasks
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Starting task: attempt_local1165163551_0001_r_000000_0
14/12/13 12:28:43 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
14/12/13 12:28:43 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@42cd2b59
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=333971456, maxSingleShuffleLimit=83492864, mergeThreshold=220421168, ioSortFactor=10, memToMemMergeOutputsThreshold=10
14/12/13 12:28:43 INFO reduce.EventFetcher: attempt_local1165163551_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
14/12/13 12:28:43 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local1165163551_0001_m_000000_0 decomp: 123 len: 127 to MEMORY
14/12/13 12:28:43 INFO reduce.InMemoryMapOutput: Read 123 bytes from map-output for attempt_local1165163551_0001_m_000000_0
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 123, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->123
14/12/13 12:28:43 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: finalMerge called with 1 in-memory map-outputs and 0 on-disk map-outputs
14/12/13 12:28:43 INFO mapred.Merger: Merging 1 sorted segments
14/12/13 12:28:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 116 bytes
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merged 1 segments, 123 bytes to disk to satisfy reduce memory limit
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merging 1 files, 127 bytes from disk
14/12/13 12:28:43 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
14/12/13 12:28:43 INFO mapred.Merger: Merging 1 sorted segments
14/12/13 12:28:43 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 116 bytes
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO mapreduce.Job: Job job_local1165163551_0001 running in uber mode : false
14/12/13 12:28:43 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
14/12/13 12:28:43 INFO mapreduce.Job:  map 100% reduce 0%
14/12/13 12:28:43 INFO mapred.Task: Task:attempt_local1165163551_0001_r_000000_0 is done. And is in the process of committing
14/12/13 12:28:43 INFO mapred.LocalJobRunner: 1 / 1 copied.
14/12/13 12:28:43 INFO mapred.Task: Task attempt_local1165163551_0001_r_000000_0 is allowed to commit now
14/12/13 12:28:43 INFO output.FileOutputCommitter: Saved output of task 'attempt_local1165163551_0001_r_000000_0' to file:/home/john/hadoop/book_res/code/hadoop-book-master/ch02/output/_temporary/0/task_local1165163551_0001_r_000000
14/12/13 12:28:43 INFO mapred.LocalJobRunner: reduce > reduce
14/12/13 12:28:43 INFO mapred.Task: Task 'attempt_local1165163551_0001_r_000000_0' done.
14/12/13 12:28:43 INFO mapred.LocalJobRunner: Finishing task: attempt_local1165163551_0001_r_000000_0
14/12/13 12:28:43 INFO mapred.LocalJobRunner: reduce task executor complete.
14/12/13 12:28:44 INFO mapreduce.Job:  map 100% reduce 100%
14/12/13 12:28:44 INFO mapreduce.Job: Job job_local1165163551_0001 completed successfully
14/12/13 12:28:44 INFO mapreduce.Job: Counters: 33
    File System Counters
        FILE: Number of bytes read=3652
        FILE: Number of bytes written=498027
        FILE: Number of read operations=0
        FILE: Number of large read operations=0
        FILE: Number of write operations=0
    Map-Reduce Framework
        Map input records=11
        Map output records=11
        Map output bytes=99
        Map output materialized bytes=127
        Input split bytes=142
        Combine input records=0
        Combine output records=0
        Reduce input groups=2
        Reduce shuffle bytes=127
        Reduce input records=11
        Reduce output records=2
        Spilled Records=22
        Shuffled Maps =1
        Failed Shuffles=0
        Merged Map outputs=1
        GC time elapsed (ms)=37
        CPU time spent (ms)=0
        Physical memory (bytes) snapshot=0
        Virtual memory (bytes) snapshot=0
        Total committed heap usage (bytes)=365953024
    Shuffle Errors
        BAD_ID=0
        CONNECTION=0
        IO_ERROR=0
        WRONG_LENGTH=0
        WRONG_MAP=0
        WRONG_REDUCE=0
    File Input Format Counters
        Bytes Read=1485
    File Output Format Counters
        Bytes Written=30
[john@localhost ch02]$


==========

思路来源:http://f.dataguru.cn/thread-167471-1-1.html