Partitioning and Bucketing in HIVE

Hello guys, I have came back with new topic in Big data environment. that is HIVE.

We all know HIVE is query engine tool to access the data on hdfs.

There are two optimization concepts in HIVE queries Partitioning and Bucketing .

We are going to see both of them and analyse the difference between the HIVE optimizations concepts

Partitioning :

Partitioning in hive is often used for distributing load horizontally in hive environment, this has performance benefit, and make the data in simple logical fashion. Example like if we are dealing with large student table and often run queries with WHERE clauses that restrict the results to a particular class or section. For making query to give response faster, Hive table can be PARTITIONED BY (class STRING, Section STRING), Partitioning tables changes how Hive structures the data storage and Hive will now create subdirectories under the main directory of student data reflecting the partitioning structure like . .../students/class=FirstYear/Section=Mechanical. If query limits for student from class FirstYear than it will only scan the contents of subdirectory ‘FirstYear’ under student directory. This can dramatically improve query performance, but only if the partitioning scheme reflects common filtering. Partitioning feature is very useful in Hive; however, a design that creates too many partitions may optimize some queries, but be detrimental for other important queries. Other drawback is having too many partitions is the large number of Hadoop files and directories that are created unnecessarily and overhead to NameNode since it must keep all metadata for the file system in memory.

Bucketing :

Bucketing is another technique for decomposing data sets into more manageable parts. For example, suppose a table using the marks as the top-level partition and the student_id as the second-level partition leads to too many small partitions. Instead, if we bucket the student table and use student_id as the bucketing column, the value of this column will be hashed by a user-defined number into buckets. Records with the same student_id will always be stored in the same bucket. Assuming the number of student_id is much greater than the number of buckets, each bucket will have many student_id. While creating table you can specify like CLUSTERED BY (student_id) INTO XY BUCKETS ; where XY is the number of buckets . Bucketing has several advantages. The number of buckets is fixed so it does not fluctuate with variety of data. If two tables have buckets on student_id, Hive can create a logically correct sampling. Bucketing also aids in doing efficient map-side joins etc.

Example:

1. marks=91

· 00000_0

· 00001_0

· 00002_0

· ........

· 00010_0

Here marks=91 is the partition and 000 files are the buckets in each partition. Buckets are calculated based on some hash functions, so rows with name=Sandy will always go in same bucket.

Comparison:

Features	Partition	Buckets
Size	The number of buckets is not fixed so it does fluctuate with data	The number of buckets is fixed so it does not fluctuate with data
Efficiency	Unnecessary may increase the load by creating many directories.	Enables more efficient queries
Distribution of data	Distributed according to condition we describe while creating partition	Hash(column) MOD(number of buckets) –evenly distributed
Query Optimization technique	Yes	Yes
Keyword	PARTITION	CLUSTERED
Execution	Queries for single itineraries by ID would be very fast but any other query would require to parse a huge amount of directories and files incurring serious overheads	We can optimize joins by bucketing ‘similar’ IDs so Hive can minimise the processing steps, and reduce the data needed to parse and compare for join operations

I suppose you like the post and please comments if you have any queries related to post or if you have any good ideas to share with me.

Enter your email address:

Delivered by FeedBurner

Send Image as binary data and string data via Socket programming in Play framework

Hello friends today we are going to see demo a simple example for sending binary data and string data via socket in play framework.

Application.java

It defines the Controller for the application. its Provide web socket and send simple binary data from socket.

package controllers;

import play.*;
import play.mvc.*;

import views.html.*;
import models.*;

public class Application extends Controller {

    // render index page
    public static Result index() {
        return ok(index.render());
    }

    // get the ws.js script
    public static Result wsJs() {
        return ok(views.js.ws.render());
    }

    // Websocket interface
    public static WebSocket wsInterface(){
        return new WebSocket(){

            // called when websocket handshake is done
            public void onReady(WebSocket.In in, WebSocket.Out out){
                SimpleChat.start(in, out);
            }
        };
    }
}

Simplechat.java

It defines the socket listener for sending messages and receiving messages.
For Sending string data you just have to replace byte[] to String in both the files

package models;
import play.mvc.*;
import play.libs.*;
import play.libs.F.*;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.util.*;
import java.awt.image.BufferedImage;
import java.io.BufferedOutputStream;
import java.io.ByteArrayOutputStream;
import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
import java.nio.ByteBuffer;
import java.util.Collections;
import java.util.HashSet;
import java.util.Set;

import javax.imageio.ImageIO;

public class SimpleChat{

    // collect all websockets here
    private static List> connections = new ArrayList>();

    public static void start(WebSocket.In in, WebSocket.Out out){


        File file = new File("C:\\Users\\saganlalp\\Pictures\\e.jpg");
       connections.add(out);
        in.onMessage(new Callback(){
            public void invoke(byte[] event){
                SimpleChat.notifyAll(event);
            }
        });

        in.onClose(new Callback0(){
            public void invoke(){
                //SimpleChat.notifyAll("A connection closed");
            }
        });
        try {
/*FileInputStream imageInFile = new FileInputStream(file);
                    byte imageData[] = new byte[(int) file.length()];*/

                  // server.getBroadcastOperations().sendEvent("fileevent", imageData);

                BufferedImage image = ImageIO.read(file);
                ByteArrayOutputStream baos = new ByteArrayOutputStream();
                ImageIO.write(image, "jpg", baos);
                byte[] byteArray = baos.toByteArray();
                /*OutputStream out = new BufferedOutputStream(new FileOutputStream("D:\\images\\new.jpg"));
                out.write(byteArray);
                if(out!=null){
                    out.close();
                }*/

                ByteBuffer buf = ByteBuffer.wrap(byteArray);
                       out.write(byteArray);
                    }
                     catch (IOException e) {

                        e.printStackTrace();
                 }
           }

    // Iterate connection list and write incoming message
    public static void notifyAll(byte[] message){
        for (WebSocket.Out out : connections) {
            out.write(message);
        }
    }


}

Index.scala.html

Its the index file of the application

@main("Small things jump around") {
    <section>
        <h1>Simple chat</h1>

        <input type="text" id="socket-input" />
        <div id="socket-messages"></div>
        <script type="text/javascript" charset="utf-8" src="@routes.Application.wsJs()"></script>
    </section>
}

main.scala.html

It is final file where data loads

@(title: String)(content: Html)

<!DOCTYPE html>

<html>
    <head>
        <title>@title</title>
        <link rel="stylesheet" media="screen" href="@routes.Assets.at("stylesheets/main.css")">
        <link rel="shortcut icon" type="image/png" href="@routes.Assets.at("images/favicon.png")">
        <script src="@routes.Assets.at("javascripts/jquery-1.9.0.min.js")" type="text/javascript"></script>

    </head>
    <body>
        @content
    </body>
</html>

ws.scala.js

Its js file that include Socket programming

$(function(){

    // get websocket class, firefox has a different way to get it
    var WS = window['MozWebSocket'] ? window['MozWebSocket'] : WebSocket;

    // open pewpew with websocket
    var socket = new WS('@routes.Application.wsInterface().webSocketURL(request)');
    socket.binaryType = "arraybuffer";
    var writeMessages = function(event){
        //$('#socket-messages').prepend(''+event.data+'
');
        //alert(event.data);
        if(event.data instanceof ArrayBuffer)
                        {
                        //alert(true);
                        showBinaryMessage(event);
                        }
        //$('#socket-messages').prepend(' Red dot

');
//$('#socket-messages').prepend('

');
    }
    function showBinaryMessage(evt)
            {
                //alert("Hi this is my message"+evt);

                var binary = '';
                var bytes = new Uint8Array(evt.data);
                var i;
                for(i=0;i< bytes.byteLength; i++)
                    {
                    binary +=String.fromCharCode(bytes[i]);
                    }
                    //alert(i);
                    //alert(bytes);
                    //alert(binary);

                $('#socket-messages').prepend(' Red dot

');
            }

    socket.onmessage = writeMessages;

    $('#socket-input').keyup(function(event){
        var charCode = (event.which) ? event.which : event.keyCode ;

        // if enter (charcode 13) is pushed, send message, then clear input field
        if(charCode === 13){
            socket.send($(this).val());
            $(this).val('');
        }
    });
});

Place e.jpg within the project directory or change the appropriate path of the image file

Just create new play project , copy this files in appropriate directory .
run the play project , your example for socket programming is ready

Hibernate and Spring Integration

Before knowing Hibernate you must understand JPA.

Introduction

Java Persistence API (JPA) provides POJO (Plain Old Java Object) standard and object relational mapping (OR mapping) for data persistence among applications. Persistence, which deals with storing and retrieving of application data, can now be programmed with Java Persistence API starting from EJB 3.0 as a result of JSR 220. This API has borrowed many of the concepts and standards from leading persistence frameworks like Toplink (from Oracle) and Hibernate (from JBoss). One of the great benefits of JPA is that it is an independent API and can nicely integrate with J2EE as well as J2SE applications. [POJO – Plain Old Java Object is a term used to refer Java objects that do not extend or implement some specialized classes. Therefore, all normal Java objects are POJO’s only. The following classes are not POJO classes

Hibernate

Hibernate is a high-performance Object/Relational persistence and query service which is licensed under the open source GNU Lesser General Public License (LGPL) and is free to download. Hibernate not only takes care of the mapping from Java classes to database tables (and from Java data types to SQL data types), but also provides data query and retrieval facilities.
This tutorial will teach you how to use Hibernate to develop your database based web applications in simple and easy steps.

JPA and Hibernate Difference
JPA is a specification for accessing, persisting and managing the data between Java objects and the relational database. As the definition says its API, it is only the specification. There is no implementation for the API. JPA specifies the set of rules and guidelines for developing the interfaces that follows standard. Straight to the point : JPA is just guidelines to implement the Object Relational Mapping (ORM) and there is no underlying code for the implementation.
Where as, Hibernate is the actual implementation of JPA guidelines. When hibernate implements the JPA specification, this will be certified by the JPA group upon following all the standards mentioned in the specification. For example, JPA guidelines would provide information of mandatory and optional features to be implemented as part of the JPA implementation.
Hibernate is a JPA provider.
- See more at: http://www.javabeat.net/jpa-vs-hibernate/#sthash.Qln9BBC6.dpuf

JPA and Hibernate Difference

JPA is a specification for accessing, persisting and managing the data between Java objects and the relational database. As the definition says its API, it is only the specification. There is no implementation for the API. JPA specifies the set of rules and guidelines for developing the interfaces that follows standard. Straight to the point : JPA is just guidelines to implement the Object Relational Mapping (ORM) and there is no underlying code for the implementation.

Where as, Hibernate is the actual implementation of JPA guidelines. When hibernate implements the JPA specification, this will be certified by the JPA group upon following all the standards mentioned in the specification. For example, JPA guidelines would provide information of mandatory and optional features to be implemented as part of the JPA implementation.

Hibernate is a JPA provider.

Now we start with our aim Hibernate and Spring Integration.

We can simply integrate hibernate application with spring application.
In hibernate framework, we provide all the database information hibernate.cfg.xml file.
But if we are going to integrate the hibernate application with spring, we don't need to create the hibernate.cfg.xml file. We can provide all the information in the applicationContext.xml file.

Advantage of Spring framework with hibernate

The Spring framework provides HibernateTemplate class, so you don't need to follow so many steps like create Configuration, BuildSessionFactory, Session, beginning and committing transaction etc.
So it saves a lot of code.
Understanding problem without using spring:
Let's understand it by the code of hibernate given below:

//creating configuration  
Configuration cfg=new Configuration();    
cfg.configure("hibernate.cfg.xml");    
    
//creating seession factory object    
SessionFactory factory=cfg.buildSessionFactory();    
    
//creating session object    
Session session=factory.openSession();    
    
//creating transaction object    
Transaction t=session.beginTransaction();    
        
Employee e1=new Employee(111,"arun",40000);    
session.persist(e1);//persisting the object    
    
t.commit();//transaction is commited    
session.close();    

As you can see in the code of sole hibernate, you have to follow so many steps.
Solution by using HibernateTemplate class of Spring Framework:
Now, you don't need to follow so many steps. You can simply write this:

Employee e1=new Employee(111,"arun",40000);
hibernateTemplate.save(e1);

Steps

Let's see what are the simple steps for hibernate and spring integration:

create table in the database It is optional.
create applicationContext.xml file It contains information of DataSource, SessionFactory etc.
create Employee.java file It is the persistent class
create employee.hbm.xml file It is the mapping file.
create EmployeeDao.java file It is the dao class that uses HibernateTemplate.
create InsertTest.java file It calls methods of EmployeeDao class.

Example of Hibernate and spring integration

In this example, we are going to integrate the hibernate application with spring. Let's see the directory structure of spring and hibernate example.

1) create the table in the database In this example, we are using the Oracle as the database, but you may use any database. Let's create the table in the oracle database

CREATE TABLE  "EMP558"   
   (    "ID" NUMBER(10,0) NOT NULL ENABLE,   
    "NAME" VARCHAR2(255 CHAR),   
    "SALARY" FLOAT(126),   
     PRIMARY KEY ("ID") ENABLE  
   )  
/  

2) Employee.java It is a simple POJO class. Here it works as the persistent class for hibernate.

package com.javatpoint;  
  
public class Employee {  
private int id;  
private String name;  
private float salary;  
  
//getters and setters  
  
}  

3) employee.hbm.xml This mapping file contains all the information of the persistent class.

'1.0' encoding='UTF-8'?>
"-//Hibernate/Hibernate Mapping DTD 3.0//EN"
"http://hibernate.sourceforge.net/hibernate-mapping-3.0.dtd">
<hibernate-mapping>
<class name="com.javatpoint.Employee" table="emp558">
<id name="id">
<generator class="assigned"></generator>
</id>
<property name="name"></property>
<property name="salary"></property>
</class>
</hibernate-mapping>

4) EmployeeDao.java It is a java class that uses the HibernateTemplate class method to persist the object of Employee class.

package com.javatpoint;  
import org.springframework.orm.hibernate3.HibernateTemplate;  
  
public class EmployeeDao {  
HibernateTemplate template;  
public void setTemplate(HibernateTemplate template) {  
    this.template = template;  
}  
  
public void saveEmployee(Employee e){  
    template.save(e);  
}  
  
public void updateEmployee(Employee e){  
    template.update(e);  
}  
  
public void deleteEmployee(Employee e){  
    template.delete(e);  
}  
}  

5) applicationContext.xml In this file, we are providing all the informations of the database in the BasicDataSource object. This object is used in the LocalSessionFactoryBean class object, containing some other informations such as mappingResources and hibernateProperties. The object of LocalSessionFactoryBean class is used in the HibernateTemplate class. Let's see the code of applicationContext.xml file.

<?xml version="1.0" encoding="UTF-8"?>
<beans
xmlns="http://www.springframework.org/schema/beans"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns:p="http://www.springframework.org/schema/p"
xsi:schemaLocation="http://www.springframework.org/schema/beans
http://www.springframework.org/schema/beans/spring-beans-3.0.xsd">
<bean id="dataSource" class="org.apache.commons.dbcp.BasicDataSource">
"driverClassName" value="oracle.jdbc.driver.OracleDriver">

</property>

</bean>

"dataSource" ref="dataSource">

</property>

<list>

<value>employee.hbm.xml</value>

</list>

</property>

<props>

<prop key="hibernate.dialect">org.hibernate.dialect.Oracle9Dialect</prop>

<prop key="hibernate.hbm2ddl.auto">update</prop>

</props>

</property>

</bean>

</bean>

</bean>

</beans>

6) InsertTest.java This class uses the EmployeeDao class object and calls its saveEmployee method by passing the object of Employee class.

package com.javatpoint;  
  
import org.springframework.beans.factory.BeanFactory;  
import org.springframework.beans.factory.xml.XmlBeanFactory;  
import org.springframework.core.io.ClassPathResource;  
import org.springframework.core.io.Resource;  
  
public class InsertTest {  
public static void main(String[] args) {  
      
    Resource r=new ClassPathResource("applicationContext.xml");  
    BeanFactory factory=new XmlBeanFactory(r);  
      
    EmployeeDao dao=(EmployeeDao)factory.getBean("d");  
      
    Employee e=new Employee();  
    e.setId(114);  
    e.setName("varun");  
    e.setSalary(50000);  
      
    dao.saveEmployee(e);  
      
}  
}  

Now, if you see the table in the oracle database, record is inserted successfully.

Enabling automatic table creation, showing sql queries etc.

You can enable many hibernate properties like automatic table creation by hbm2ddl.auto etc. in applicationContext.xml file. Let's see the code:

<property name = "hibernateProperties">
<props>
<prop key = "hibernate.dialect">org.hibernate.dialect.Oracle9Dialect</prop>
<prop key = "hibernate.hbm2ddl.auto">update</prop>
<prop key = "hibernate.show_sql">true</prop>
</props>

If you write this code, you don't need to create table because table will be created automatically.

Now this is all about Hibernate and Spring Integration. Do comment for any quiries.

Introduction to Java Persistence API(JPA)

ava Persistence API (JPA) provides POJO (Plain Old Java Object) standard and object relational mapping (OR mapping) for data persistence among applications. Persistence, which deals with storing and retrieving of application data, can now be programmed with Java Persistence API starting from EJB 3.0 as a result of JSR 220. This API has borrowed many of the concepts and standards from leading persistence frameworks like Toplink (from Oracle) a - See more at: http://www.javabeat.net/jpa/#sthash.iVuc8t0G.dpuf

Java Persistence API (JPA) provides POJO (Plain Old Java Object) standard and object relational mapping (OR mapping) for data persistence among applications. Persistence, which deals with storing and retrieving of application data, can now be programmed with Java Persistence API starting from EJB 3.0 as a result of JSR 220. This API has borrowed many of the concepts and standards from leading persistence frameworks like Toplink (from Oracle) and Hibernate (from JBoss). One of the great benefits of JPA is that it is an independent API and can nicely integrate with J2EE as well as J2SE applications. [POJO – Plain Old Java Object is a term used to refer Java objects that do not extend or implement some specialized classes. Therefore, all normal Java objects are POJO’s only. The following classes are not POJO classes - See more at: http://www.javabeat.net/jpa/#sthash.iVuc8t0G.dpuf

Introduction to Java Persistence API(JPA)

Installing Hadoop in Windows With Eclipse

Introduction

Hadoop is a powerful framework for automatic parallelization of computing tasks. Unfortunately programming for it poses certain challenges. It is really hard to understand and debug Hadoop programs. One way to make it a little easier is to have a simplified version of the Hadoop cluster that runs locally on the developer's machine. This tutorial describes how to set up such a cluster on a computer running Microsoft Windows. It also describes how to integrate this cluster with Eclipse, a prime Java development environment.

Prerequisites

The required Software that needed to be install are

Now this steps will use to create hadoop environment

Installing Cygwin

Installation method. Select “Install from Internet”.
Root Directory. The default is c:\cygwin. Accept this directory.
Local Package Directory (the directory where install files will be downloaded). The default is c:\cygwin-packages. Accept this directory.
Connection and download site.
A list of available packages will be displayed. The following packages are missing, so make sure to include them:
- openssh
- openssl
- tcp_wrappers
- diffutils
If several options are listed (eg: openssl) include them all.
Upon installation completion, it will create a Cygwin icon in the Desktop and/or Start menu. Click it to open a Cygwin window.

Installing Hadoop

This is a detailed step-by-step guide for installing Hadoop on Windows, Linux or MAC. It’s based in Hadoop 1.0.0, which is the current and first official stable version. It’s based in version 0.20.0 (note that there was a 0.21.0 version).
Installing Hadoop on Linux / MAC is pretty straight forward. However, having it run on Windows can be a bit tricky. You’d probably not run Hadoop on Windows on a productive environment, but it may result convenient as a development environment. If you are using Linux/MAC, just skip Windows information.

Windows installation

Hadoop can be installed on Windows using Cygwin (not inteded for production environments), but there are several Cygwin installation and configuration issues.

Windows: Download and install Cygwin

Cygwin is an implementation of a set of Linux commands and applications for Windows. Download the web installer from: http://cygwin.com/setup.exe and run it.
Installer will request some information before installing:

Installation method. Select “Install from Internet”.
Root Directory. The default is c:\cygwin. Accept this directory.
Local Package Directory (the directory where install files will be downloaded). The default is c:\cygwin-packages. Accept this directory.
Connection and download site.
A list of available packages will be displayed. The following packages are missing, so make sure to include them:
- openssh
- openssl
- tcp_wrappers
- diffutils
If several options are listed (eg: openssl) include them all.
Upon installation completion, it will create a Cygwin icon in the Desktop and/or Start menu. Click it to open a Cygwin window.

Now some configuration of cygwin to use with hadoop

Configuring SSH on Windows

Hadoop requires SSH (Secure SHell) to be running. To configure it, open a Cygwin window and type:

ssh-host-config

Use the following installation options:

Should privilege separation be used? (yes/no) no
Do you want to install sshd as a service? yes
Enter the value of CYGWIN for the daemon: [] ntsec
If requested for an account name, specify: cyg_server with a password you’ll remember.

Eg:

$ ssh-host-config

*** Info: Generating /etc/ssh_host_key
*** Info: Generating /etc/ssh_host_rsa_key
*** Info: Generating /etc/ssh_host_dsa_key
*** Info: Generating /etc/ssh_host_ecdsa_key
*** Info: Creating default /etc/ssh_config file
*** Info: Creating default /etc/sshd_config file
*** Info: Privilege separation is set to yes by default since OpenSSH 3.3.
*** Info: However, this requires a non-privileged account called 'sshd'.
*** Info: For more info on privilege separation read /usr/share/doc/openssh/README.privsep.
*** Query: Should privilege separation be used? (yes/no) no
*** Info: Updating /etc/sshd_config file

*** Query: Do you want to install sshd as a service?
*** Query: (Say "no" if it is already installed as a service) (yes/no) yes
*** Query: Enter the value of CYGWIN for the daemon: [] ntsec
*** Info: On Windows Server 2003, Windows Vista, and above, the
*** Info: SYSTEM account cannot setuid to other users -- a capability
*** Info: sshd requires.  You need to have or to create a privileged
*** Info: account.  This script will help you do so.

*** Info: You appear to be running Windows XP 64bit, Windows 2003 Server,
*** Info: or later.  On these systems, it's not possible to use the LocalSystem
*** Info: account for services that can change the user id without an
*** Info: explicit password (such as passwordless logins [e.g. public key
*** Info: authentication] via sshd).

*** Info: If you want to enable that functionality, it's required to create
*** Info: a new account with special privileges (unless a similar account
*** Info: already exists). This account is then used to run these special
*** Info: servers.

*** Info: Note that creating a new user requires that the current account
*** Info: have Administrator privileges itself.

*** Info: No privileged account could be found.

*** Info: This script plans to use 'cyg_server'.
*** Info: 'cyg_server' will only be used by registered services.
*** Query: Do you want to use a different name? (yes/no) no

*** Query: Create new privileged user account 'cyg_server'? (yes/no) yes
*** Info: Please enter a password for new user cyg_server.  Please be sure
*** Info: that this password matches the password rules given on your system.
*** Info: Entering no password will exit the configuration.
*** Query: Please enter the password:
*** Query: Reenter: Enter password

*** Info: User 'cyg_server' has been created with password '####'.
*** Info: If you change the password, please remember also to change the
*** Info: password for the installed services which use (or will soon use)
*** Info: the 'cyg_server' account.

*** Info: Also keep in mind that the user 'cyg_server' needs read permissions
*** Info: on all users' relevant files for the services running as 'cyg_server3'.
*** Info: In particular, for the sshd server all users' .ssh/authorized_keys
*** Info: files must have appropriate permissions to allow public key
*** Info: authentication. (Re-)running ssh-user-config for each user will set
*** Info: these permissions correctly. [Similar restrictions apply, for
*** Info: instance, for .rhosts files if the rshd server is running, etc].

*** Info: The sshd service has been installed under the 'cyg_server'
*** Info: account.  To start the service now, call `net start sshd' or
*** Info: `cygrunsrv -S sshd'.  Otherwise, it will start automatically
*** Info: after the next reboot.

*** Info: Host configuration finished. Have fun!

Installation script creates:

configuration files:
- /etc/ssh_config
- /etc/ssh_host_dsa_key
- /etc/ssh_host_ecdsa_key
- /etc/ssh_host_key
- /etc/ssh_host_rsa_key
- /etc/sshd_config
a cyg_server privilleged account.
a sshd Windows service, using the specified account and password, and listed under the name CYGWIN sshd.

IMPORTANT: Do not run ssh-host-config without removing existing files or account. The script changes access permissions on configuration files so that they can only be accessed by ssh services. If the sshd service, configuration files and account are not created together, the script fails to configure the file permissions and no error is reported.Cleaning up ssh
If you run into any issue, delete the above 6 files, remove the created service using:

cygrunsrv -R sshd

and start over.

You should be able to start sshd service and login using your password. However, in order to run Hadoop you need to create a server key, so that you can stablish a ssh session without specifying a password. To this type

ssh-keygen

and accept all default options (no passphrase).

$ ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/home/AccountName/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /home/AccountName/.ssh/id_rsa.
Your public key has been saved in /home/AccountName/.ssh/id_rsa.pub.
The key fingerprint is:
9b:51:11:ea:c4:a4:72:fe:70:e7:dd:f1:ea:34:ac:0f AccountName@ServerName
The key's randomart image is:
+--[ RSA 2048]----+
|        . o.     |
|       + . .     |
|    . o + .      |
|     + o .       |
|      o S .   .  |
|       + * . o o |
|        + . E = .|
|             + o |
|            .o+  |
+-----------------+

Copy the generated RSA public key into the authorized_keys file, to allow logging without password.

cd ~/.ssh
cat id_rsa.pub >> authorized_keys

Try connecting locally:

ssh localhost

You should be able to connect without specifying a password.

Install JAVA latest version and set JAVA_HOME path to system variable

Now Hadoop part :

Download Hadoop

Current Hadoop version is 1.0.0. Hadoop is organized as 3 projects:

Common: Common functionality to all projects (logging, utilities, etc).
HDFS: Hadoop Distributed File System.
MapReduce: Map-Reduce implementation. It allows performing distributed queries on the distributed file system. Explained later.

They are downloaded together from http://hadoop.apache.org/ as a single .tar.gz / .rpm / .deb file.
Unpack hadoop to any directory. Recommended install directory is /usr/local/hadoop-1.0.0, but you could use other directories.

Configure Hadoop

There are 3 basic configuration options for Hadoop:

Local (Standalone) Mode: All services run in a single node, with no replication.
Pseudo-Distributed Mode: Services run in a single node, but as separate Java processes.
Fully-Distributed Mode: Real distributed environment.

Hadoop configuration is stored in xml files located in /conf. They all share the same key-value format, stored as a sequence of:

  
    Property name
    Property value

Pseudo-Distributed Mode is the ideal development mode. Minimum configuration files for pseudo-distributed mode are shown below:

conf/core-site.xml:


  
    fs.default.name
    hdfs://localhost:9000

conf/hdfs-site.xml:
```
  
    dfs.replication
    1
  
```

conf/mapred-site.xml:


  
    mapred.job.tracker
    localhost:9001

If no path are specified, Hadoop temporary and data files are located in system tmp directory. So any implementation should begin by defining tmp and hdfs directories, as shown below:

conf/core-site.xml:


  
    hadoop.tmp.dir
    /tmp/hadoop-${user.name}
  

  
    fs.default.name
    hdfs://localhost:9000

conf/hdfs-site.xml:


  
    dfs.replication
    1
  

  
    dfs.name.dir
    /home/${user.name}/hdfs/name
  

  
    dfs.data.dir
    /home/${user.name}/hdfs/data

conf/mapred-site.xml:


  
    mapred.job.tracker
    localhost:9001

Under Windows, specify paths using full format. Eg:

  
    dfs.name.dir
    file:///c:/hdfs/name

Start hadoop

Format NameNode

Before starting Hadoop, you have to format the Name node. This is the node containing file structure. To format the Name node run:

cd /usr/local/hadoop-1.0.0
./bin/hadoop namenode -format

Several files will be created under the directory defined for the configuration key dfs.name.dir.

Start HDFS

bin/start-dfs.sh

Check HDFS is running by browsing to: http://localhost:50070/.
A webpage should be displayed with DFS information, where you can view and browse the directory structure.
If you run into any issue, check log files under hadoop-1.0.0/logs/ for errors.
You can also browse the file system using bin/hadoop fs -ls. Type bin/hadoop fs for the complete set of commands.
Under MAC/OSX you might get an “Unable to load realm info from SCDynamicStore” error. If you run into this issue, add the following line:

export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk

Start MapReduce (JobTracker):

bin/start-mapred.sh

Check JobTracker has started by browsing to:: http://localhost:50030/.
A page with scheduled jobs should be displayed.
Check hadoop-1.0.0/logs/ for errors.
Check HDFS and JobTracker by openning:

NameNode: http://localhost:50070/
JobTracker: http://localhost:50030/

Now steps to start all hadoop components:

Start the local hadoop cluster

Start the namenode in the first window by executing
cd hadoop-0.19.1
bin/hadoop namenode
Start the secondary namenode in the second window by executing
cd hadoop-0.19.1
bin/hadoop secondarynamenode
Start the job tracker the third window by executing
cd hadoop-0.19.1
bin/haoop jobtracker
Start the data node the fourth window by executing
cd hadoop-0.19.1
bin/haoop datanode
Start the task tracker the fifth window by executing
cd hadoop-0.19.1
bin/haoop tasktracker
Now you should have an operational hadoop cluster. If everthing went fine your screen should look like the image below:

Now the important step that post belong

Setup Hadoop Location in Eclipse

Launch the Eclipse environment.
Open Map/Reduce perspective by clicking on the open perspective icon (), select "Other" from the menu, and then select "Map/Reduce" from the list of perspectives.
After you switched to the Map/Reduce perspective. Select the Map/Reduce Locations tab located at the bottom portion of your eclipse environment. Then right click on the blank space in that tab and select "New Hadoop location...." from the context menu.
Fill in the following items, as shown on the figure above.
- Location Name -- localhost
- Map/Reduce Master
  - Host -- localhost
  - Port -- 9101
- DFS Master
  - Check "Use M/R Master Host"
  - Port -- 9100
- User name -- User
Then press the Finish button.
After you closed the Hadoop location settings dialog you should see a new location appearing in the "Map/Reduce Locations" tab.
In the Project Explorer tab on the lefthand side of the eclipse window, find the DFS Locations item. Open it up using the "+" icon on the left side of it, inside of it you should see the localhost location reference with the blue elephant icon. Keep opening up the items

Upload data to HDFS

Open a new CYGWIN command window.
Execute the following commands in the new CYGWIN window as shown on the image above.

cd hadoop-0.19.1
bin/hadoop fs -mkdir In
bin/hadoop fs -put *.txt In
When the last of the above commands will start execution you should see some activity happening in the rest of the hadoop windows as shown on the image below.

Creating and configuring Hadoop eclipse project.
Launch Eclipse
Right click on the blank space in the Project Explorer window and select New -> Project.. to create a new project.
Select Map/Reduce Project from the list of project types.

For Reference You can follow the below links :

iSignIndia - Website Design and Development

Looking for better Website development ?

Have a look here, Annually 50 websites turn over and 150 websites currently projects available. This company just get the Quality certificates from various organisations. Its good Company where people can forward websites projects. Just Make money by giving project to iSignIndia , have a personal talk to them any time and make your income start now.

This is the home page of iSignIndia, Just visit once , I assured you cannot neglect the design of website, and you definitely like the concept of design and development of iSignIndia.

iSign is a IT Company dealing with the web and software solutions, offering cost effective and high-quality professional web site design, web development & services for small to medium businesses. customers trust us to deliver technology solutions that help them do and achieve more, whether they’re at home, work, school or anywhere in their world. As we believe that our hardwork and dedication is “Of The People, By The People, For The People" and shall not perish from Humanity.

Enlightened Vision

To touch every individual through its presence in every home and office. Strive for the 100% customer satisfaction with our services and support.

Make iSign gloablly a respected name.
Improve the living and business standard of people.

Steadfast Mission

Our mission is to enable people and companies to increase their business value through the use of IT technologies. We aim at delivering services which will open new possibilities for our customers and allow them to work effectively and creatively.

Focus on customer rejoice. Build corporate image.
Better and effective communication.

Follow us socially

Contact Details

Website: www.iSignIndia.com

Email: info@isignindia.com

Partitioning and Bucketing in HIVE

Partitioning :

Bucketing :

Comparison:

Send Image as binary data and string data via Socket programming in Play framework

Application.java

Simplechat.java

Index.scala.html

main.scala.html

ws.scala.js

Hibernate and Spring Integration

Installing Hadoop in Windows With Eclipse

Introduction

Prerequisites

Installing Cygwin

Installing Hadoop

Windows installation

Windows: Download and install Cygwin

Now some configuration of cygwin to use with hadoop

Configuring SSH on Windows

Install JAVA latest version and set JAVA_HOME path to system variable

Now Hadoop part :

Download Hadoop

Configure Hadoop

Start hadoop

Format NameNode

Start HDFS

Start MapReduce (JobTracker):

Start the local hadoop cluster

Now the important step that post belong

Setup Hadoop Location in Eclipse

Upload data to HDFS

Creating and configuring Hadoop eclipse project.

iSignIndia - Website Design and Development

Enlightened Vision

Steadfast Mission

Follow us socially

Contact Details