MR并行算法编程过程中遇到问题的思考

时间:2023-03-08 17:08:34
MR并行算法编程过程中遇到问题的思考

1. Reducer 类中 reduce函数外定义的变量是在Reducer机器上属于全局变量的,因此,一台机器上reduce函数均可以对该变量的值做出贡献。如代码:(sum和count数据Reducer机器上的全局变量)‘

	public static class AvgCalReducer extends Reducer<EntityEntityWritable,FloatWritable,EntityEntityWritable,FloatWritable>
{
FloatWritable avg;
float sum=0;
int count=0;
public void reduce(EntityEntityWritable key,Iterable<FloatWritable>values,Context context) throws IOException, InterruptedException
{ System.out.println("reducer starting:");
for (FloatWritable value:values)
{
sum=sum+value.get();
count++;
System.out.println(" key = "+key+" value = "+value.get());
}
System.out.println("average:"+sum/count);
System.out.println("this reducer ending...");
avg=new FloatWritable(sum/count);
context.write(key, avg);
}
}

如果想使sum和count的值仅通过reduce函数进行改变,即只计算同一个key对应value的sum和count,则需要将sum和count放入reduce函数内,如下:

	public static class AvgCalReducer extends Reducer<EntityEntityWritable,FloatWritable,EntityEntityWritable,FloatWritable>
{
FloatWritable avg; public void reduce(EntityEntityWritable key,Iterable<FloatWritable>values,Context context) throws IOException, InterruptedException
{
float sum=0;
int count=0;
System.out.println("reducer starting:");
for (FloatWritable value:values)
{
sum=sum+value.get();
count++;
System.out.println(" key = "+key+" value = "+value.get());
}
System.out.println("average:"+sum/count);
System.out.println("this reducer ending...");
avg=new FloatWritable(sum/count);
context.write(key, avg);
}
}

2. 对于顺序组合式MapReduce作业:用两个job举例:

		Configuration conf1=new Configuration();
Job job1=new Job(conf1,"Job1");
job1.waitForCompletion(true); Configuration conf2=new Configuration();
Job job2=new Job(conf2,"Job2");
job2.waitForCompletion(true);

注意我们之前经常写的System.exit(job.waitForCompletion(true)?0:1)在这里不可以使用,比如第一个job处的(job1.waitForCompletion(true)改成System.exit(job.waitForCompletion(true)?0:1),则系统成功完成job1后正常退出系统,没有机会再去运行job2了。