Hadoop & Mapreduce Tutorial | Pig Installation and Modes

Installation and Modes

Installation

Pig core is written in Java and it works across operating systems. Pig’s shell, which executes the commands from the user, is a bash script and requires a UNIX system. Pig can also be run on Windows using Cygwin and Perl packages.

Java 1.6 is also mandatory for Pig to run. Optionally, the following can be installed on the same machine: Python 2.5, JavaScript 1.7, Ant 1.7, and JUnit 4.5. Python and JavaScript are for writing custom UDFs. Ant and JUnit are for builds and unit testing, respectively. Pig can be executed with different versions of Hadoop by setting HADOOP_HOME to point to the directory where we have installed Hadoop. If HADOOP_HOME is not set, Pig will run with the embedded version by default, which is currently Hadoop 1.0.0.

Requirements

Mandatory – Unix and Windows users need the following:

Hadoop 0.23.X, 1.X or 2.X – http://hadoop.apache.org/common/releases.html (You can run Pig with different versions of Hadoop by setting HADOOP_HOME to point to the directory where you have installed Hadoop. If you do not set HADOOP_HOME, by default Pig will run with the embedded version, currently Hadoop 1.0.4.)
Java 1.7 – http://java.sun.com/javase/downloads/index.jsp (set JAVA_HOME to the root of your Java installation)

Optional

Python 2.7 – https://www.python.org (when using Streaming Python UDFs)
Ant 1.8 – http://ant.apache.org/ (for builds)

Download Pig

To get a Pig distribution, do the following:

Download a recent stable release from one of the Apache Download Mirrors.
Unpack the downloaded Pig distribution, and then note the following:
The Pig script file, pig, is located in the bin directory (/pig-n.n.n/bin/pig). The Pig environment variables are described in the Pig script file.
The Pig properties file, pig.properties, is located in the conf directory (/pig-n.n.n/conf/pig.properties). You can specify an alternate location using the PIG_CONF_DIR environment variable.
Add /pig-n.n.n/bin to your path. Use export (bash,sh,ksh) or setenv (tcsh,csh). For example:

$ export PATH=/<my-path-to-pig>/pig-n.n.n/bin:$PATH

Test the Pig installation with this simple command: $ pig -help

Build Pig

To build pig, do the following:

Check out the Pig code from SVN: svn co http://svn.apache.org/repos/asf/pig/trunk
Build the code from the top directory: ant If the build is successful, you should see the pig.jar file created in that directory.
Validate the pig.jar by running a unit test: ant test
If you are using Hadoop 0.23.X or 2.X, please add -Dhadoopversion=23 in your ant command line in the previous steps

Pig Modes

You can run Pig (execute Pig Latin statements and Pig commands) using various modes.

	Local Mode	Tez Local Mode	Mapreduce Mode	Tez Mode
Interactive Mode	yes	experimental	yes	yes
Batch Mode	yes	experimental	yes	yes

Execution Modes

Pig has two execution modes or exectypes:

Local Mode – To run Pig in local mode, you need access to a single machine; all files are installed and run using your local host and file system. Specify local mode using the -x flag (pig -x local).
Tez Local Mode – To run Pig in tez local mode. It is similar to local mode, except internally Pig will invoke tez runtime engine. Specify Tez local mode using the -x flag (pig -x tez_local). Tez local mode is experimental. There are some queries which just error out on bigger data in local mode.
Mapreduce Mode – To run Pig in mapreduce mode, you need access to a Hadoop cluster and HDFS installation. Mapreduce mode is the default mode; you can, but don’t need to, specify it using the -x flag (pig OR pig -x mapreduce).
Tez Mode – To run Pig in Tez mode, you need access to a Hadoop cluster and HDFS installation. Specify Tez mode using the -x flag (-x tez).

You can run Pig in either mode using the “pig” command (the bin/pig Perl script) or the “java” command (java -cp pig.jar …).

Example

This example shows how to run Pig in local and mapreduce mode using the pig command.

/* local mode */

$ pig -x local …

/* Tez local mode */

$ pig -x tez_local …

/* mapreduce mode */

$ pig …

$ pig -x mapreduce …

/* Tez mode */

$ pig -x tez …

Interactive Mode

You can run Pig in interactive mode using the Grunt shell. Invoke the Grunt shell using the “pig” command (as shown below) and then enter your Pig Latin statements and Pig commands interactively at the command line.

Example

These Pig Latin statements extract all user IDs from the /etc/passwd file. First, copy the /etc/passwd file to your local working directory. Next, invoke the Grunt shell by typing the “pig” command (in local or hadoop mode). Then, enter the Pig Latin statements interactively at the grunt prompt (be sure to include the semicolon after each statement). The DUMP operator will display the results to your terminal screen.

grunt> A = load ‘passwd’ using PigStorage(‘:’);

grunt> B = foreach A generate $0 as id;

grunt> dump B;

Local Mode

$ pig -x local

… – Connecting to …

grunt>

Tez Local Mode

$ pig -x tez_local

… – Connecting to …

grunt>

Mapreduce Mode