Dowload Lastest Books On Hadoop

Amazon Best Deal !!

Powered by Blogger.

Sunday, December 4, 2016

INTRODUCTION TO APACHE PIG IN HADOOP



What is Apache Pig?

Apache Pig is an abstraction over MapReduce. It is a tool/platform which is used to analyze larger sets of data representing them as data flows. Pig is generally used with Hadoop; we can perform all the data manipulation operations in Hadoop using Apache Pig.

To write data analysis programs, Pig provides a high-level language known as Pig Latin. This language provides various operators using which programmers can develop their own functions for reading, writing, and processing data.

To analyze data using Apache Pig, programmers need to write scripts using Pig Latin language. All these scripts are internally converted to Map and Reduce tasks. Apache Pig has a component known as Pig Engine that accepts the Pig Latin scripts as input and converts those scripts into MapReduce jobs.

Why Do We Need Apache Pig?

Programmers who are not so good at Java normally used to struggle working with Hadoop, especially while performing any MapReduce tasks. Apache Pig is a boon for all such programmers.

Using Pig Latin, programmers can perform MapReduce tasks easily without having to type complex codes in Java.

Apache Pig uses multi-query approach, thereby reducing the length of codes. For example, an operation that would require you to type 200 lines of code (LoC) in Java can be easily done by typing as less as just 10 LoC in Apache Pig. Ultimately Apache Pig reduces the development time by almost 16 times.

Pig Latin is SQL-like language and it is easy to learn Apache Pig when you are familiar with SQL.

Apache Pig provides many built-in operators to support data operations like joins, filters, ordering, etc. In addition, it also provides nested data types like tuples, bags, and maps that are missing from MapReduce.

Features of Pig:-

Apache Pig comes with the following features −

Rich set of operators − It provides many operators to perform operations like join, sort, filer, etc.

Ease of programming − Pig Latin is similar to SQL and it is easy to write a Pig script if you are good at SQL.

Optimization opportunities − The tasks in Apache Pig optimize their execution automatically, so the programmers need to focus only on semantics of the language.

Extensibility − Using the existing operators, users can develop their own functions to read, process, and write data.

UDF’s − Pig provides the facility to create User-defined Functions in other programming languages such as Java and invoke or embed them in Pig Scripts.

Handles all kinds of data − Apache Pig analyzes all kinds of data, both structured as well as unstructured. It stores the results in HDFS.

Apache Pig Vs MapReduce:-


  • Listed below are the major differences between Apache Pig and MapReduce.
  • Apache Pig is a data flow language. MapReduce is a data processing paradigm.
  • It is a high level language.MapReduce is low level and rigid.
  • Performing a Join operation in Apache Pig is pretty simple. It is quite difficult in MapReduce to perform a Join operation between datasets.
  • Any novice programmer with a basic knowledge of SQL can work conveniently with Apache Pig. Exposure to Java is must to work with MapReduce.
  • Apache Pig uses multi-query approach, thereby reducing the length of the codes to a great extent. MapReduce will require almost 20 times more the number of lines to perform the same task.
  • There is no need for compilation. On execution, every Apache Pig operator is converted internally into a MapReduce job. MapReduce jobs have a long compilation process.

Apache Pig Vs SQL:-

  • Listed below are the major differences between Apache Pig and SQL.
  • Pig Latin is a procedural language. SQL is a declarative language.
  • In Apache Pig, schema is optional. We can store data without designing a schema (values are stored as $01, $02 etc.) Schema is mandatory in SQL.
  • The data model in Apache Pig is nested relational.The data model used in SQL is flat relational.
  • Apache Pig provides limited opportunity for Query optimization.There is more opportunity for query optimization in SQL.


In addition to above differences, Apache Pig Latin :-


  • Allows splits in the pipeline.
  • Allows developers to store data anywhere in the pipeline.
  • Declares execution plans.
  • Provides operators to perform ETL (Extract, Transform, and Load) functions.
  • Apache Pig Vs Hive
  • Both Apache Pig and Hive are used to create MapReduce jobs. And in some cases, Hive operates on HDFS in a similar way Apache Pig does. In the following table, we have listed a few significant points that set Apache Pig apart from Hive.


Apache Pig  Vs Hive:-


  • Apache Pig uses a language called Pig Latin. It was originally created at Yahoo. Hive uses a language called HiveQL. It was originally created at Facebook.
  • Pig Latin is a data flow language. HiveQL is a query processing language.
  • Pig Latin is a procedural language and it fits in pipeline paradigm. HiveQL is a declarative language.
  • Apache Pig can handle structured, unstructured, and semi-structured data. Hive is mostly for structured data.
  • Apache Pig is generally used by data scientists for performing tasks involving ad-hoc processing and quick prototyping. Apache Pig is used −


  • To process huge data sources such as web logs.
  • To perform data processing for search platforms.
  • To process time sensitive data loads.

7 comments:

  1. Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.

    Hadoop Online Training
    Data Science Online Training

    ReplyDelete
  2. The share you really give us excitement. Thanks for your sharing. If you feel tired at work or study try to participate in our games to bring the most exciting feeling. Thank you!Python Online Training | Learn Python Online

    ReplyDelete
  3. Greetings. I know this is somewhat off-topic, but I was wondering if you knew where I could get a captcha plugin for my comment form? I’m using the same blog platform like yours, and I’m having difficulty finding one? Thanks a lot.
    hardware and networking training in chennai

    hardware and networking training in tambaram

    xamarin training in chennai

    xamarin training in tambaram

    ios training in chennai

    ios training in tambaram

    iot training in chennai

    iot training in tambaram

    ReplyDelete
  4. Everyone wants to get unique place in the IT industry’s for that you need to upgrade your skills, your blog helps me improvise my skill set to get good career, keep sharing your thoughts with us.
    oracle training in chennai

    oracle training in omr

    oracle dba training in chennai

    oracle dba training in omr

    ccna training in chennai

    ccna training in omr

    seo training in chennai

    seo training in omr

    ReplyDelete
  5. Are you interested in doing Data Science Training in Chennai with a Certification Exam? Catch the best features of Data Science training courses with Infycle Technologies, the best Data Science Training & Placement institutes in and around Chennai. Infycle offers the best hands-on training to the students with the revised curriculum to enhance their knowledge. In addition to the Certification & Training, Infycle offers placement classes for personality tests, interview preparation, and mock interviews for clearing the interviews with the best records. To have all it in your hands, dial 7504633633 for a free demo from the experts.

    ReplyDelete