Starfish Hadoop Log Analyzer

Released: Dec 10, 2012

Requirements:

  • The Analyzer is launched by Java Web Start and requires permission to access your local file system. If you get security exceptions after confirming the warning, retry with a different choice.

Note:

  • Starfish Hadoop Log Analyzer is a standalone tool for Hadoop logs analysis and performance diagnosis. It is an improved version of the Starfish visualizer tailored for pure Hadoop logs.

How to use:

Starfish Profile, What-If and Cost-based Optimizer

Released: January 16, 2012

Requirements:

  • Starfish supports Apache Hadoop v0.20.2 and v0.20.203.0
  • The Starfish Visualizer requires JavaFX version 1.2.3
Starfish Visualizer
Starfish Visualizer

Starfish Components:

  • Profiler: The Profiler uses dynamic instrumentation to learn performance models, called job profiles, for unmodified MapReduce programs. The Profiler also exposes an interface for analyzing past MapReduce executions.
  • What-if Engine: The What-if Engine uses a mix of simulation and model-based estimation at the task phase level of MapReduce job execution, in order to predict the performance of a MapReduce job before the job's execution on a Hadoop cluster.
  • Cost-based Optimizer: The Cost-based Optimizer enumerates and searches through the high-dimensional space of job configuration settings, making appropriate calls to the What-if Engine, in order to find the optimal configuration settings to use for executing a MapReduce job.
  • Visualizer: The Visualizer provides a graphical user interface that allows the user to (a) analyze past MapReduce job executions, (b) ask hypothetical (what-if) questions on how the job behavior will change when parameter settings, cluster resources, or data proprties change, and, (c) ultimately optimize the job.

This release contains many improvements, new features, bug fixes, and optimizations. You will find the release notes and previous releases here.

The software release includes detailed documentation explaining how to install and use each component of the software. You will also find a tutorial here.

If you would like to stay up-to-date with the latest Starfish news or to pose any questions, please subscribe to our group below:

Subscribe to the Starfish group
Email:
Visit this group