Eclipse+Maven构建Hadoop项目的方法步骤

时间:2022-05-12 22:29:31

maven 翻译为”专家”、”内行”,是 apache 下的一个纯 java 开发的开源项目。基于项目对象模型(project object model 缩写:pom)概念,maven利用一个*信息片断能管理一个项目的构建、报告和文档等步骤。maven 是一个项目管理工具,可以对 java 项目进行构建、依赖管理。

在开发一些大型项目的时候,需要用到各种各样的开源包jar,为了方便管理及加载jar,使用maven开发项目可以节省大量时间且方便项目移动至新的开发环境。

开发环境

  • 系统:macos 10.14.1
  • hadoop:2.7.0
  • java:1.8.0
  • eclipse:4.6.2
  • maven: 3.3.9

maven安装

我使用的这个版本的eclipse已经自带了maven插件,不需要在自行安装,因此我也没有实际操作,本文就不介绍如何配置。

至于怎么知道自己使用的eclipse是否自带有maven,可以在eclipse->preference->maven->installations查看是否有maven及版本号。或者直接新建项目查看是否有maven选项。

Eclipse+Maven构建Hadoop项目的方法步骤

构建hadoop环境

创建maven项目

打开eclipse,file->new->project,选择maven,然后下一步next

Eclipse+Maven构建Hadoop项目的方法步骤

选择creat a simple project,然后下一步next

Eclipse+Maven构建Hadoop项目的方法步骤

输入group id和artifact id。然后finish。

groupid和artifactid被统称为“坐标”是为了保证项目唯一性而提出的,如果你要把你项目弄到maven本地仓库去,你想要找到你的项目就必须根据这两个id去查找。

groupid一般分为多个段,这里我只说两段,第一段为域,第二段为公司名称。域又分为org、com、cn等等许多,其中org为非营利组织,com为商业组织。举个apache公司的tomcat项目例子:这个项目的groupid是org.apache,它的域是org(因为tomcat是非营利项目),公司名称是apache,artigactid是tomcat。

比如我创建一个项目,我一般会将groupid设置为cn.snowin,cn表示域为中国,snowin是我个人姓名缩写,artifactid设置为testproj,表示你这个项目的名称是testproj,依照这个设置,你的包结构最后是cn.snowin.testproj打头。(引自 链接 )

Eclipse+Maven构建Hadoop项目的方法步骤

完成上述步骤后,就可以在project explorer中看到刚刚创建的maven项目。

Eclipse+Maven构建Hadoop项目的方法步骤

增加hadoop依赖

我使用的hadoop 2.7版本,以下是我的pom配置文件

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
<project xmlns="http://maven.apache.org/pom/4.0.0" xmlns:xsi="http://www.w3.org/2001/xmlschema-instance"
    xsi:schemalocation="http://maven.apache.org/pom/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelversion>4.0.0</modelversion>
 
    <groupid>practice.hadoop</groupid>
    <artifactid>simple-examples</artifactid>
    <version>0.0.1-snapshot</version>
    <packaging>jar</packaging>
 
    <name>simple-examples</name>
    <url>http://maven.apache.org</url>
 
    <properties>
        <project.build.sourceencoding>utf-8</project.build.sourceencoding>
    </properties>
 
    <dependencies>
        <dependency>
            <groupid>junit</groupid>
            <artifactid>junit</artifactid>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-common -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-common</artifactid>
            <version>2.7.0</version>
        </dependency>
 
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-hdfs</artifactid>
            <version>2.7.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-client -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-client</artifactid>
            <version>2.7.0</version>
        </dependency>
        
        <dependency>
            <groupid>org.apache.mrunit</groupid>
            <artifactid>mrunit</artifactid>
            <version>1.1.0</version>
            <classifier>hadoop2</classifier>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-core -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-mapreduce-client-core</artifactid>
            <version>2.7.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-yarn-api -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-yarn-api</artifactid>
            <version>2.7.0</version>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-auth -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-auth</artifactid>
            <version>2.7.0</version>
        </dependency>
 
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-minicluster -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-minicluster</artifactid>
            <version>2.7.0</version>
            <scope>test</scope>
        </dependency>
        <!-- https://mvnrepository.com/artifact/org.apache.hadoop/hadoop-mapreduce-client-jobclient -->
        <dependency>
            <groupid>org.apache.hadoop</groupid>
            <artifactid>hadoop-mapreduce-client-jobclient</artifactid>
            <version>2.7.0</version>
            <scope>provided</scope>
        </dependency>
 
    </dependencies>
</project>

在project explorer中右键该项目,选择build project,maven就会根据pom.xml配置文件下载所需要的jar包。

Eclipse+Maven构建Hadoop项目的方法步骤

稍等一段时间后,就可以看到maven dependencies中已经下载好的jar包。

Eclipse+Maven构建Hadoop项目的方法步骤

hadoop配置文件

运行 mapreduce 程序前,务必将 /usr/local/cellar/hadoop/2.7.0/libexec/etc/hadoop 中将有修改过的配置文件(如伪分布式需要core-site.xml 和 hdfs-site.xml),以及log4j.properties复制到 src/main/resources/

Eclipse+Maven构建Hadoop项目的方法步骤

mapreduce实例—wordcount

src/main/java/ 路径下,创建java文件,代码如下

?
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
import java.io.ioexception;
import java.util.stringtokenizer;
 
import org.apache.hadoop.conf.configuration;
import org.apache.hadoop.fs.path;
import org.apache.hadoop.io.intwritable;
import org.apache.hadoop.io.text;
import org.apache.hadoop.mapreduce.job;
import org.apache.hadoop.mapreduce.mapper;
import org.apache.hadoop.mapreduce.reducer;
import org.apache.hadoop.mapreduce.lib.input.fileinputformat;
import org.apache.hadoop.mapreduce.lib.output.fileoutputformat;
import org.apache.hadoop.util.genericoptionsparser;
 
public class wordcount {
 
    public static class tokenizermapper extends mapper<object, text, text, intwritable> {
 
        /**
         * longwritable, intwritable, text 均是 hadoop 中实现的用于封装 java
         * 数据类型的类,这些类实现了writablecomparable接口,
         * 都能够被串行化从而便于在分布式环境中进行数据交换,你可以将它们分别视为long,int,string 的替代品。
         */
        private final static intwritable one = new intwritable(1); // 值为1
        private text word = new text();
 
        public void map(object key, text value, context context) throws ioexception, interruptedexception {
            stringtokenizer itr = new stringtokenizer(value.tostring()); // 对字符串进行切分
            while (itr.hasmoretokens()) {
                word.set(itr.nexttoken());
                context.write(word, one);
            }
        }
    }
 
    public static class intsumreducer extends reducer<text, intwritable, text, intwritable> {
        private intwritable result = new intwritable();
 
        public void reduce(text key, iterable<intwritable> values, context context)
                throws ioexception, interruptedexception {
            int sum = 0;
            for (intwritable val : values) {
                sum += val.get();
            }
            result.set(sum);
            context.write(key, result);
        }
    }
 
    public static void main(string[] args) throws exception {
        configuration conf = new configuration();
    conf.addresource("classpath:/hadoop/core-site.xml");
    conf.addresource("classpath:/hadoop/hdfs-site.xml");
    conf.addresource("classpath:/hadoop/mapred-site.xml");
//      string[] otherargs = new genericoptionsparser(conf, args).getremainingargs();
        string[] otherargs = {"/input", "/output"};
        if (otherargs.length != 2) {
            system.err.println("usage: wordcount <in> <out>");
            system.exit(2);
        }
        job job = new job(conf, "word count");
        job.setjarbyclass(wordcount.class);
        job.setmapperclass(tokenizermapper.class);
        job.setcombinerclass(intsumreducer.class);
        job.setreducerclass(intsumreducer.class);
        job.setoutputkeyclass(text.class);
        job.setoutputvalueclass(intwritable.class);
        fileinputformat.setinputdirrecursive(job, true);
        fileinputformat.addinputpath(job, new path(otherargs[0]));
        fileoutputformat.setoutputpath(job, new path(otherargs[1]));
        system.exit(job.waitforcompletion(true) ? 0 : 1);
    }
 
}

以上就是本文的全部内容,希望对大家的学习有所帮助,也希望大家多多支持服务器之家。

原文链接:http://hareric.com/2019/02/01/Eclipse+Maven构建Hadoop项目/