mrjob

mrjob is a Python package that helps you write and run Hadoop Streaming jobs.

mrjob fully supports Amazon’s Elastic MapReduce (EMR) service, which allows you to buy time on a Hadoop cluster on an hourly basis. It also works with your own Hadoop cluster.

Table of Contents

• What’s New
  • 0.3.2
  • 0.3
    • Features
    • Changes and Deprecations
• Getting started
  • Writing and running a job
    • Running locally
    • Running on EMR
    • Running on your own Hadoop cluster
    • Running from another script
  • Common configuration tasks
    • Putting your source tree in the PYTHONPATH
    • Increasing the task timeout
    • Writing compressed output
  • Common EMR configuration tasks
    • Custom Python packages
    • Setting parameters in mapred-site.xml
    • Setting up Ganglia
    • Enabling Python core dumps
  • Example configuration file
• mrjob.job.MRJob - base class for all jobs
  • Basic
    • Writing one-step jobs
    • Writing multi-step jobs
    • Running the job
    • Parsing the output
    • Counters and status messages
  • Advanced
    • Protocols
    • Custom command-line options
    • Custom command-line types and actions
    • Job runner configuration
    • Running specific parts of jobs
    • Hadoop Configuration
  • Hooks for testing
• mrjob.protocol - input and output
  • Custom Protocols
• Runners - launching your job
  • mrjob.runner - base class for all runners
    • Running your job
    • Run Information
    • File management
  • mrjob.local - run locally for testing
  • mrjob.emr - run on EMR
    • Setting up EMR on Amazon
    • Job Runner
    • S3 Utilities
    • EMR Utilities
    • SSH Tunneling and Log Fetching
    • Choosing Type and Number of EC2 Instances
    • Advanced EMR Strategies
    • Troubleshooting
  • mrjob.hadoop - run on your Hadoop cluster
    • Utilities
  • mrjob.inline - alternate (debugger-friendly) local testing
• Runner configuration
  • mrjob.conf - parse and write config files
    • Reading and writing mrjob.conf
    • Combining options
  • Configuration options
    • All runners
    • Locally
    • On EMR
    • On your Hadoop cluster
    • Alternate (debugger-friendly) local testing
    • Getting configuration options out of runners
  • Quick Reference
    • Options that can’t be set from mrjob.conf (all runners)
    • Other options for all runners
    • Additional options for EMRJobRunner
    • Additional options for HadoopJobRunner
• Utilities
  • mrjob.compat - Hadoop version compatibility
  • mrjob.parse - log parsing
  • mrjob.retry - retry on transient errors
  • mrjob.util - general utility functions
• mrjob.tools.emr - managing EMR job flows
  • audit_usage
  • create_job_flow
  • fetch_logs
  • mrboss
  • report_long_jobs
  • s3_tmpwatch
  • terminate_job_flow
  • terminate_idle_job_flows
• Testing with mrjob
  • Anatomy of a Test Case
  • Counters

Indices and tables

• Search Page