Upgrade to Hadoop 2.8

[jbaiera]

Hadoop 2.8.0 was released on March 22 of this year, with a 2.8.1 release in process (at the time this ticket was created). Most notably in the list of changes for Hadoop 2.8.0 is the complete replacement of the apache commons httpclient 3.1 with the newer org.apache.httpcomponents httpclient 4.5.2. This poses a problem for upgrading since in 6.0 we will be supporting all the way back to Hadoop 2.2.0.

[whitfin]

Just dropping a specific error I'm seeing for reference in case people are searching for it. Can confirm that it's caused by 2.8.x.

java.lang.Exception: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpConnectionManager
    at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:489)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:556)
Caused by: java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpConnectionManager
    at org.elasticsearch.hadoop.rest.commonshttp.CommonsHttpTransportFactory.create(CommonsHttpTransportFactory.java:39)
    at org.elasticsearch.hadoop.rest.NetworkClient.selectNextNode(NetworkClient.java:99)
    at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:82)
    at org.elasticsearch.hadoop.rest.NetworkClient.<init>(NetworkClient.java:59)
    at org.elasticsearch.hadoop.rest.RestClient.<init>(RestClient.java:92)
    at org.elasticsearch.hadoop.rest.InitializationUtils.discoverEsVersion(InitializationUtils.java:273)
    at org.elasticsearch.hadoop.rest.RestService.createWriter(RestService.java:572)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.init(EsOutputFormat.java:173)
    at org.elasticsearch.hadoop.mr.EsOutputFormat$EsRecordWriter.write(EsOutputFormat.java:149)
    at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.write(ReduceTask.java:558)
    at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
    at org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.write(WrappedReducer.java:105)

[i-aggarwal]

We are using the elasticsearch-hadoop » 7.5.1 for writing data from cloud to Elasticsearch. We also have hadoop-common » 3.2.0 as a dependency. Even though we might be able to use the older version, is there any timeline in vision as to when the org.apache.httpcomponents » httpclient would be supported.

And yes I am seeing the exact same error.