Logging
editLogging
editelasticsearch-hadoop uses commons-logging library, same as Hadoop, for its logging infrastructure and thus it shares the same configuration means. Out of the box, no configuration is required - by default, elasticsearch-hadoop logs relevant information about the job progress at INFO level. Typically, whatever integration you are using (Map/Reduce, Hive, Pig), each job will print in the console at least one message indicating the elasticsearch-hadoop version used:
16:13:01,946 INFO main util.Version - Elasticsearch Hadoop v2.0.0.BUILD-SNAPSHOT [f2c5c3e280]
Configuring logging for Hadoop (or Hive and Pig) is outside the scope of this documentation, however in short, at runtime, Hadoop relies on log4j 1.2 as an actual logging implementation. In practice, this means adding the package name of interest and its level logging the log4j.properties file in the job classpath.
elasticsearch-hadoop provides the following important packages:
| Package | Purpose |
|---|---|
|
Apache Hive integration |
|
Map/Reduce functionality |
|
Apache Pig integration |
|
REST/transport infrastructure |
|
Serialization package |
|
Apache Spark package |
|
Apache Storm package |
The default logging level (INFO) is suitable for day-to-day use; if troubleshooting is needed, consider switching to DEBUG but be selective of the packages included. For low-level details, enable level TRACE however do remember that it will result in a significant amount of logging data which will impact your job performance and environment.
To put everything together, if you want to enable DEBUG logging on the Map/Reduce package make changes to the log4j.properties (used by your environment):
log4j.category.org.elasticsearch.hadoop.mr=DEBUG
See the log4j javadoc for more information.
Configure the executing JVM logging not the client
editOne thing to note is that in almost all cases, one needs to configure logging in the executing JVM, where the Map/Reduce tasks actually run and not on the client, where the job is assembled or monitored. Depending on your library, platform and version this can done through some dedicated settings.
In particular Map/Reduce-based libraries like Pig or Hive can be difficult to configure since at runtime, they create Map/Reduce tasks to actually perform the work. Thus, one needs to configure logging and pass the configuration to the Map/Reduce layer for logging to occur.
In both cases, this can be achieved through the SET command. In particular when using Hadoop 2.6, one can use mapreduce.job.log4j-properties-file along with an appropriate container-log4j.properties file.
It’s worth mentioning that Pig allows jobs to be executed locally and logging to be enabled through pig -x local -4 myLoggingFile someScript.pig