Starfish v0.3.0 Tutorial: 2. Job Profiling

Profiling a MapReduce Job Execution

Profiling a MapReduce job is simple using the provided bin/profile script. Please ensure you have installed the required BTrace classes and jars using the provided bin/install_btrace.sh, discussed in the Installation section.

  1. Set the following profiling parameters in bin/config.sh. All parameters below are optional:
    • PROFILE_TYPE: The type for profiling. Possible values: task, memory, all
    • RETAIN_TASK_PROFILES: Whether to retain the task-level profiles or not
    • COLLECT_DATA_TRANSFERS: Whether to collect the data transfers among the tasks or not
    • SAMPLING_MODE: The sampling mode. Possible values: off, profiles, tasks (explained below)
    • SAMPLING_FRACTION: The fraction of tasks to profile or run (explained below)
  2. Execute the bin/profile script:
    ./bin/profile hadoop jar jarFile args...

The expected parameters are identical to the parameters required by ${HADOOP_HOME}/bin/hadoop. Here is an example for profiling a WordCount MapReduce program.

./bin/profile hadoop jar contrib/examples/hadoop-starfish-examples.jar \
   wordcount -Dmapred.reduce.tasks=10 /input/path /output/path

When the job execution completes, all output files are placed in the directory specified by PROFILER_OUTPUT_DIR. This directory can be used later for analyzing the execution of MapReduce jobs and contains the following subdirectories:

  1. history: Contains a configuration file and a statistics file for each executed MapReduce job.
  2. job_profiles: Contains a job profile XML file for each profiled MapReduce job. A job profile is a concise statistical summary of the job's execution and consists of a vector of dataflow and cost fields. For the full listing and description of the fields, please see docs/explain_profiles.txt.
  3. task_profiles: Contains the raw monitoring data collected by the Profiler for each map or reduce task that was profiled. This information is available only when RETAIN_TASK_PROFILES is set to true.
  4. transfers: Contains detailed information regarding the transfer of map-output data from the map tasks to the the reduce tasks. This information is available only when COLLECT_DATA_TRANSFERS is set to true.

Known issues:

  1. If the job executes successfully but the directory PROFILER_OUTPUT_DIR is not created, it is possible that the parameter HADOOP_OPTS is overwritten instead of appended in ${HADOOP_HOME}/conf/hadoop-env.sh. For example, if HADOOP_OPTS is set to "-server", change it to be set to "-server $HADOOP_OPTS" (including the quotes).
  2. If the job fails to execute with a java.lang.ClassNotFoundException referring to the Profiler, it is possible that the parameter HADOOP_CLASSPATH is overwritten instead of appended in ${HADOOP_HOME}/conf/hadoop-env.sh. For example, if HADOOP_CLASSPATH is set to "/path/to/some/jar", change it to be set to "$HADOOP_CLASSPATH:/path/to/some/jar" (including the quotes).
  3. If the job fails to execute with a java.io.FileNotFoundException referring to a profile file, please ensure that SLAVES_BTRACE_DIR in bin/config.sh specifies an absolute path, not a relative path. You might need to reinstall the BTrace classes using bin/install_btrace.sh. See Section 1. Installation for details.
  4. If multiple tasks are failing with the error messages "java.lang.Throwable: Child Error" or "Error reading task output", then the problem is most likely caused from the path specified in SLAVES_BTRACE_DIR in bin/config.sh. Please ensure this path is absolute, it has the appropriate read permissions, and that each parent directory in the path has appropriate execute permissions. Note that in many case "/root/rest/of/path" will lead to this error.

Task-level Sampling when Profiling a MapReduce Job Execution

The Profiler supports two types of task-level sampling to generate approximate job profiles while keeping the run-time overhead low. The two techniques can be used by modifying the SAMPLING_MODE and SAMPLING_FRACTION in bin/config.sh or using the corresponing Hadoop parameters starfish.profiler.sampling.mode and starfish.profiler.sampling.fraction.

  1. The Profiler can collect task profiles for only a sample of the job's tasks, while all tasks are running. In the following snippet, all tasks will execute but only 20% of the map and reduce tasks will get profiled.
    ./bin/profile hadoop jar contrib/examples/hadoop-starfish-examples.jar \
       wordcount -Dmapred.reduce.tasks=10 \
       -Dstarfish.profiler.sampling.mode=profiles \
       -Dstarfish.profiler.sampling.fraction=0.2 \
       /input/path /output/path
  2. The Profiler can selectively execute (and profile) only a sample of the job's tasks. In the following snippet, only 20% of the map tasks will execute and profiled. (All reduce tasks will execute and profiled.)
    ./bin/profile hadoop jar contrib/examples/hadoop-starfish-examples.jar \
       wordcount -Dmapred.reduce.tasks=10 \
       -Dstarfish.profiler.sampling.mode=tasks \
       -Dstarfish.profiler.sampling.fraction=0.2 \
       /input/path /output/path

Memory Profiling during a MapReduce Job Execution

The Profiler supports three types of profiling specified using PROFILE_TYPE in bin/config.sh or using the corresponing Hadoop parameter starfish.profiler.profile.type. The three types are:

  1. task: This is the default task execution profile.
  2. memory: This option profiles only the heap memory usage of each task execution. You can find the recorded memory values in the task profiles under the PROFILER_OUTPUT_DIR/task_profiles directory. Memory profiling cannot be used for asking hypothetical questions or optimizing the execution of a MapReduce job.
  3. all: This option produces the task execution profile plus recordings for the minimum, maximum, and average heap memory usage during the task processing.

Here is an example for profiling the heap memory usage for a WordCount MapReduce program:

./bin/profile hadoop jar contrib/examples/hadoop-starfish-examples.jar \
   wordcount -Dstarfish.profiler.profile.type=memory /input/path /output/path