Part Ⅰ．Hadoop Fundamentals

1．Meet Hadoop

Data!

Data Storage and Analysis

Querying All Your Data

Beyond Batch

Comparison with Other Systems

Relational Database Management Systems

Grid Computing

Volunteer Computing

A Brief History of Apache Hadoop

What's in This Book?

2．MapReduce

A Weather Dataset

Data Format

Analyzing the Data with Unix　Tools

Analyzing the Data with Hadoop

Map and Reduce

Java Map Reduce

Scaling Out

Data Flow

Combiner Functions

Running a Distributed Map Reduce Job

Hadoop Streaming

Ruby

Python

3．The Hadoop Distributed Filesystem

The Design of HDFS

HDFS Concepts

Blocks

Namenodes and Datanodes

Block Caching

HDFS Federation

HDFS High Availability

The Command-Line Interface

Basic Filesystem Operations

Hadoop Filesystems

Interfaces

The Java Interface

Reading Data from a Hadoop URL

Reading Data Using the FileSystem API

Writing Data

Directories

Querying the Filesystem

Deleting Data

Data Flow

Anatomy of a File Read

Anatomy of a File Write

Coherency Model

Parallel Copying with distcp

Keeping an HDFS Cluster Balanced

4．YARN

Anatomy of a YARN Application Run

Resource Requests

Application Lifespan

Building YARN Applications

YARN Compared to MapReduce 1

Scheduling in YARN

Scheduler Options

Capacity Scheduler Configuration

Fair Scheduler Configuration

Delay Scheduling

Dominant Resource Fairness

Further Reading

5．Hadoop I/O

Data Integrity

Data Integrity in HDFS

LocalFileSystem

ChecksumFileSystem

Compression

Codecs

Compression and Input Splits

Using Compression in MapReduce

Serialization

The Writable Interface

Writable Classes

Implementing a Custom Writable

Serialization Frameworks

File-Based Data Structures

SequenceFile

MapFile

Other File Formats and Column-Oriented Formats

Part Ⅱ．MapReduce

6．Developing a MapReduce Application

The Configuration API

Combining Resources

Variable Expansion

Setting Up the Development Environment

Managing Configuration

GeneficOptionsParser，Tool，and ToolRunner

Writing a Unit Test with MRUnit

Mapper

Reducer

Running Locally on Test Data

Running a Job in a Local Job Runner

Testing the Driver

Running on a Cluster

Packaging a Job

Launching a Job

The MapReduce Web UI

Retrieving the Results

Debugging a Job

Hadoop Logs

Remote Debugging

Tuning a Job

Profiling Tasks

MapReduce Workflows

Decomposing a Problem into MapReduce Jobs

JobControl

Apache Oozie

7．How Map Reduce Works

Anatomy of a MapReduce Job Run

Job Submission

Job Initialization

Task Assignment

Task Execution

Progress and Status Updates

Job Completion

Failures

Task Failure

Application Master Failure

Node Manager Failure

Resource Manager Failure

Shuffle and Sort

The Map Side

The Reduce Side

Configuration Tuning

Task Execution

The Task Execution Environment

Speculative Execution

Output Committers

8．MapReduce Types and Formats

MapReduce Types

The Default MapReduce Job

Input Formats

Input Splits and Records

Text Input

Binary Input

Multiple Inputs

Database Input(and Output)

Output Formats

Text Output

Binary Output

Multiple Outputs

Lazy Output

Database Output

9．MapReduce Features

Counters

Built-in Counters

User-Defined Java Counters

User-Defined Streaming Counters

Sorting

Preparation

Partial Sort

Total Sort

Secondary Sort

Joins

Map-Side Joins

Reduce-Side Joins

Side Data Distribution

Using the Job Configuration

Distributed Cache

MapReduce Library Classes

Part Ⅲ．Hadoop Operations

1O．Setting Up a Hadoop Cluster

Cluster Specification

Cluster Sizing

Network Topology

Cluster Setup and Installation

Installing Java

Creating Unix User Accounts

Installing Hadoop

Configuring SSH

Configuring Hadoop

Formatting the HDFS Filesystem

Starting and Stopping the Daemons

Creating User Directories

Hadoop Configuration

Configuration Management

Environment Settings

Important Hadoop Daemon Properties

Hadoop Daemon Addresses and Ports

Other Hadoop Properties

Security

Kerberos and Hadoop

Delegation Tokens

Other Security Enhancements

Benchmarking a Hadoop Cluster

Hadoop Benchmarks

User Jobs

11．Administering Hadoop

HDFS

Persistent Data Structures

Safe Mode

Audit Logging

Tools

Monitoring

Logging

Metrics and JMX

Maintenance

Routine Administration Procedures

Commissioning and Decommissioning Nodes

Upgrades

Part Ⅳ．Related Projects

12．Avro

Avro Data Types and Schemas

In-Memory Serialization and Deserialization

The Specific API

Avro Datafiles

Interoperability

Python API

Avro Tools

Schema Resolution

Sort Order

Avro MapReduce

Sorting Using Avro MapReduce

Avro in Other Languages

13．Parquet

Data Model

Nested Encoding

Parquet File Format

Parquet Configuration

Writing and Reading Parquet Files

Avro，Protocol Buffers，and Thrift

Parquet MapReduce

14．Flume

Installing Flume

An Example

Transactions and Reliability

Batching

The HDFS Sink

Partitioning and Interceptors

File Formats

Fan Out

Delivery Guarantees

Replicating and Multiplexing Selectors

Distribution：Agent Tiers

Delivery Guarantees

Sink Groups

Integrating Flume with Applications

Component Catalog

Further Reading

15．Sqoop

Getting Sqoop

Sqoop Connectors

A Sample Import

Text and Binary File Formats

Generated Code

Additional Serialization Systems

Imports：A Deeper Look

Controlling the Import

Imports and Consistency

Incremental Imports

Direct-Mode Imports

Working with Imported Data

Imported Data and Hive

Importing Large Objects

Performing an Export

Exports：A Deeper Look

Exports and Transactionality

Exports and SequenceFiles

Further Reading

16．Pig

Installing and Running Pig

Execution Types

Running Pig Programs

Grunt

Pig Latin Editors

An Example

Generating Examples

Comparison with Databases

Pig Latin

Structure

Statements

Expressions

Types

Schemas

Functions

Macros

User-Defined Functions

A Filter UDF

An Eval UDF

A Load UDF

Data Processing Operators

Loading and Storing Data

Filtering Data

Grouping and Joining Data

Sorting Data

Combining and Splitting Data

Pig in Pracfice

Parallelism

Anonymous Relations

Parameter Substitution

Further Reading

17．Hive

Installing Hive

The Hive Shell

An Example

Running Hive

Configuring Hive

Hive Services

The Metastore

Comparison with Traditional Databases

Schema on Read Versus Schema on Write

Updates，Transactions，and Indexes

SQL-on-Hadoop Alternatives

HiveQL

Data Types

Operators and Functions

Tables

Managed Tables and External Tables

Partitions and Buckets

Storage Formats

Importing Data

Altering Tables

Dropping Tables

Querying Data

Sorting and Aggregating

MapReduce Scripts

Joins

Subqueries

Views

User-Defined Functions

Writing a UDF

Writing a UDAF

Further Reading

18．Crunch

An Example

The Core Crunch API

Primitive Operations

Types

Sources and Targets

Functions

Materialization

Pipeline Execution

Running a Pipeline

Stopping a Pipeline

Inspecting a Crunch Plan

Iterative Algorithms

Checkpointing a Pipeline

Crunch Libraries

Further Reading

19．Spark

Installing Spark

An Example

Spark Applications，Jobs，Stages，and Tasks

A Scala Standalone Application

A Java Example

A Python Example

Resilient Distributed Datasets

Creation

Transformations and Actions

Persistence

Serialization

Shared Variables

Broadcast Variables

Accumulators

Anatomy of a Spark Job Run

Job Submission

DAG Construction

Task Scheduling

Task Execution

Executors and Cluster Managers

Spark on YARN

Further Reading

20．HBase

HBasics

Backdrop

Concepts

Whirlwind Tour of the Data Model

Implementation

Installation

Test Drive

Clients

Java

MapReduce

REST and Thrift

Building an Online Query Application

Schema Design

Loading Data

Online Queries

HBase Versus RDBMS

Successful Service

HBase

Praxis

HDFS

UI

Metrics

Counters

Further Reading

21．ZooKeeper

Installing and Running ZooKeeper

An Example

Group Membership in ZooKeeper

Creating the Group

Joining a Group

Listing Members in a Group

Deleting a Group

The ZooKeeper Service

Data Model

Operations

Implementation

Consistency

Sessions

States

Building Applications with ZooKeeper

A Configuration Service

The Resilient ZooKeeper Application

A Lock Service

More Distributed Data Structures and Protocols

ZooKeeper in Production

Resilience and Performance

Configuration

Further Reading

Part Ⅴ．Case Studies

22．Composable Data at Cerner

From CPUs to Semantic Integration

Enter Apache Crunch

Building a Complete Picture

Integrating Healthcare Data

Composability over Frameworks

Moving Forward

23．Biological Data Science：Saving Lives with Software

The Structure of DNA

The Genetic Code：Turning DNA Letters into Proteins

Thinking of DNA as Source Code

The Human Genome Project and Reference Genomes

Sequencing and Aligning DNA

ADAM，A Scalable Genome Analysis Platform

Literate programming with the Avro interface description language（IDL）

Column-oriented access with Parquet

A simple example：k-mer counting using Spark and ADAM

From Personalized Ads to Personalized Medicine

Join In

24．Cascading

Fields，Tuples，and Pipes

Operations

Taps，Schemes，and Flows

Cascading in Practice

Flexibility

Hadoop and Cascading at ShareThis

Summary

A．Installing Apache Hadoop

B．Cloudera's Distribution Including Apache Hadoop

C．Preparing the NCDC Weather Data

D．The Old and New Java MapReduce APIs

Index