御笔剑客 发表于 2021-1-19 23:53:13

hadoop2.10执行自带的测试程序不输出output目录

执行的命令:./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs+'


使用的jdk8,hadoop2.0,执行测试数据时,并不会生成output目录,而是生成以下目录:

hadoop@code-shop:/usr/local/hadoop$ ./bin/hdfs dfs -ls
Found 2 items
drwxr-xr-x   - hadoop supergroup          0 2021-01-19 23:39 grep-temp-1442452675
drwxr-xr-x   - hadoop supergroup          0 2021-01-19 22:51 input
hadoop@code-shop:/usr/local/hadoop$

运行过程中的日志:貌似并未发现明显错误。

hadoop@code-shop:/usr/local/hadoop$ ./bin/hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-*.jar grep input output 'dfs+'
21/01/19 23:43:20 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
21/01/19 23:43:20 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
21/01/19 23:43:20 INFO input.FileInputFormat: Total input files to process : 8
21/01/19 23:43:21 INFO mapreduce.JobSubmitter: number of splits:8
21/01/19 23:43:22 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local1446384809_0001
21/01/19 23:43:22 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
21/01/19 23:43:22 INFO mapreduce.Job: Running job: job_local1446384809_0001
21/01/19 23:43:22 INFO mapred.LocalJobRunner: OutputCommitter set in config null
21/01/19 23:43:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/01/19 23:43:22 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/01/19 23:43:22 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
21/01/19 23:43:22 INFO mapred.LocalJobRunner: Waiting for map tasks
21/01/19 23:43:22 INFO mapred.LocalJobRunner: Starting task: attempt_local1446384809_0001_m_000000_0
21/01/19 23:43:22 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/01/19 23:43:22 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/01/19 23:43:22 INFO mapred.Task:Using ResourceCalculatorProcessTree : [ ]
21/01/19 23:43:22 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/hadoop-policy.xml:0+10206
21/01/19 23:43:49 INFO mapreduce.Job: Job job_local1446384809_0001 running in uber mode : false
21/01/19 23:43:49 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
21/01/19 23:43:49 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
21/01/19 23:43:49 INFO mapred.MapTask: soft limit at 83886080
21/01/19 23:43:49 INFO mapreduce.Job:map 0% reduce 0%
21/01/19 23:43:49 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
21/01/19 23:43:49 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
21/01/19 23:43:49 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
21/01/19 23:43:59 INFO mapred.LocalJobRunner:
21/01/19 23:43:59 INFO mapred.MapTask: Starting flush of map output
21/01/19 23:43:59 INFO mapred.MapTask: Spilling map output
21/01/19 23:43:59 INFO mapred.MapTask: bufstart = 0; bufend = 17; bufvoid = 104857600
21/01/19 23:43:59 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214396(104857584); length = 1/6553600
21/01/19 23:43:59 INFO mapred.MapTask: Finished spill 0
21/01/19 23:43:59 INFO mapred.Task: Task:attempt_local1446384809_0001_m_000000_0 is done. And is in the process of committing
21/01/19 23:43:59 INFO mapred.LocalJobRunner: map
21/01/19 23:43:59 INFO mapred.Task: Task 'attempt_local1446384809_0001_m_000000_0' done.
21/01/19 23:43:59 INFO mapred.Task: Final Counters for attempt_local1446384809_0001_m_000000_0: Counters: 23
        File System Counters
                FILE: Number of bytes read=304459
                FILE: Number of bytes written=803155
                FILE: Number of read operations=0
                FILE: Number of large read operations=0
                FILE: Number of write operations=0
                HDFS: Number of bytes read=10206
                HDFS: Number of bytes written=0
                HDFS: Number of read operations=5
                HDFS: Number of large read operations=0
                HDFS: Number of write operations=1
        Map-Reduce Framework
                Map input records=237
                Map output records=1
                Map output bytes=17
                Map output materialized bytes=25
                Input split bytes=122
                Combine input records=1
                Combine output records=1
                Spilled Records=1
                Failed Shuffles=0
                Merged Map outputs=0
                GC time elapsed (ms)=13
                Total committed heap usage (bytes)=234881024
        File Input Format Counters
                Bytes Read=10206
21/01/19 23:43:59 INFO mapred.LocalJobRunner: Finishing task: attempt_local1446384809_0001_m_000000_0
21/01/19 23:43:59 INFO mapred.LocalJobRunner: Starting task: attempt_local1446384809_0001_m_000001_0
21/01/19 23:43:59 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
21/01/19 23:43:59 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false
21/01/19 23:43:59 INFO mapred.Task:Using ResourceCalculatorProcessTree : [ ]
21/01/19 23:43:59 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/hadoop/input/capacity-scheduler.xml:0+8814
Killed

琉煌 发表于 2021-1-20 14:01:35

想知道小甲鱼最近在做啥?

lixiangyv 发表于 2021-2-6 08:37:03

看不懂

爱学习的小志 发表于 2021-3-27 15:30:29

兄弟我在源码确实找到了产生临时目录的代码

Path tempDir = new Path("grep-temp-" + Integer.toString(new Random().nextInt(2147483647)));

而你的日志中的16行

21/01/19 23:43:22 INFO output.FileOutputCommitter: FileOutputCommitter skip cleanup _temporary folders under output directory:false, ignore cleanup failures: false

显示跳过了清理临时目录,我觉得有可能是这个原因导致的output目录,但是我并没有找到解决方式,你可以检查一下MapReduce的配置文件是不是配置的有问题{:9_221:}

连帅帅 发表于 2021-4-1 16:57:48

我就是来混分的

kerln888 发表于 2021-4-11 16:58:27

看不懂啊

ridiculum 发表于 2021-6-16 13:02:51

看不懂

Dragon910623 发表于 2021-6-22 13:00:47

妙啊

robinyunhao 发表于 2021-6-26 18:45:17

hornwong 发表于 2021-6-28 09:17:31

感谢分享!

sunwenwu123 发表于 2021-7-10 11:07:52

学习一下

sunwenwu 发表于 2021-7-10 11:13:04

学习一下
页: [1]
查看完整版本: hadoop2.10执行自带的测试程序不输出output目录