Map Reduce Application(Partitioninig/Group data by a defined key)
Assuming we want to group data by the year(2008 to 2016) of their [last access date time]. For each year, we use a reducer to collect them and output the data in this group/partition(year of the last access datetime). So, we want the MR to partition our key by year. We will lean what's the default partitioner and see how to set custom partitioner.
The default partitioner:
1 public int getPartition(K key, V value, 2 int numReduceTasks) { 3 return (key.hashCode() & Integer.MAX_VALUE) % numReduceTasks; 4 }
Custom Partitioner:
job.setPartitionerClass(CustomPartitioner.class)
With blew partitioner, the data of different year of [last access date time] will be assigned to different / unique partition. The num of reduce tasks is 9.
1 public static class CustomPartitioner extends Partitioner<Text, Text>{ 2 @Override 3 public int getPartition(Text key, Text value, int numReduceTasks){ 4 if(numReduceTasks == 0){ 5 return 0; 6 } 7 return key-2008 8 }