Cloud Computing:Hadoop

1. Apache Hadoop

1.1. Overview

Apache Hadoop is a software solution for distributed computing of large datasets. Hadoop provides a distributed filesystem (HDFS) and a MapReduce implementation.
A special computer acts as the "name node". This computer saves the information about the available clients and the files. The Hadoop clients (computers) are called nodes.
The "name node" is currently a single point of failure. The Hadoop project is working on solutions for this.

1.2. Typical tasks

Apache Hadoop can be used to filter and aggregate data, e.g. a typical usecase would be the analysis of webserver logs file to find the most visisted pages. But MapReduce has been use to transverse graphs and other tasks.

1.3. Writing the map and reduce functions

Hadoop allow that map and reduce functions are written in Java. Hadoop provides also linker so that map and reduce funtions can be written in other languages, e.g. C++, Python, Per, etc.

2. Hadoop file system

The Hadoop file system (HDSF) is a distribute file system. It use an existing file system of the operating system but extends this with redundancy and distribution. HSDF hides the complexity of distributed storage and redundancy from the programmer.
In the standard configuration HDFS saves all files three times on different nodes. The "name node" (server) has the information where the files are stored.
Harddisks are very effective in reading large files sequentially but getting much slower during random access. HDFS is therefore optimized for large files.
To improve performance Hadoop also tries to move the computation to the nodes which store the data and not vice versa. Especially if you have very large data this help improving the performance as you can avoid that the network becomes the bottleneck.

3. MapReduce

Apache Hadoop jobs are working according to the MapRecude principle. See MapReduce for details.

4. Installation

Apache Hadoop can be downloaded from Hadoop Homepage. To get started with Hadoop you require the following sub-projects:

Comments

Post a Comment

Popular posts from this blog

Installing Qt Jambi For java

Installing from a Prebuilt Package After extracting the package, you are ready to run the Launcher, which is an application that runs the examples and demos that come with Qt Jambi. To run the examples and demo launcher on Windows run: qtjambi.exe On Linux and Mac OS X run: qtjambi.sh Running these files will verify the virtual machine version and set the correct environment for running Qt Jambi applications. When you are ready to distribute your own applications, see the  deployment  document. Manually Launching Qt Jambi Applications To manually launch applications, you need to include the following  .jar  files in your classpath: qtjambi-4.4.0_01.jar , which contains the Jambi classes. The platform archive, which contains the native libraries used by Qt Jambi. The archive is named after the operating system and compiler used to build the libraries; for instance,  qtjambi-win32-msvc2005-4.4.0_01.jar  for Windows and MSVC 2005. To run the examples or launcher, the class
Read more

Leach protocol installation in network simulator 2

Leach protocol installation in ns2 had to go through the following steps:  As far as I have studied, I got two patches for Leach 1. MIT patch 2. Mannasim patch- I didnt work much with it but if you dont want to meddle much with tcl, you can use it. Ok, the prerequisite is Ubuntu 10.04. We need to install ns-2.34  over that because LEACH MIT patch was made in 2000. Its a little painful to use the old OS, but it has to be done. Again you can have your OS on a virtual box on windows (only if you are not concerned with speed and not in love with your computer). 1. Download  ns-234-leach.tar.gz. Keep this in your /home/$userID/ns-allinone-2.34/ns-2.34. (Assuming you ns-allinone is in home dir) Leach ns2.34   2. Download the bash file "leach-setup.sh" into the directory " /home/$userID/ ns-allinone-2.34/ns-2.34".   leach-sh 3- Now find /home/arun/ns-allinone-2.35/ns-2.35 in the leach-setup.sh file and replace it with your own path. /home/$username/ns-a
Read more

What Is Tomcat Default Administrator Password ?

By default, Tomcat does not enable admin or manager access. To enable it, you have to edit the “ %TOMCAT_FOLDER%/conf/tomcat-users.xml ” manually. File : tomcat-users.xml (before update) , initially, Tomcat comments all users and roles like above. <?xml version = '1.0' encoding = 'utf-8' ?> <tomcat-users> <!-- <role rolename="tomcat"/> <role rolename="role1"/> <user username="tomcat" password="tomcat" roles="tomcat"/> <user username="both" password="tomcat" roles="tomcat,role1"/> <user username="role1" password="tomcat" roles="role1"/> --> </tomcat-users> File : tomcat-users.xml (after updated) <?xml version = '1.0' encoding = 'utf-8' ?> <tomcat-users> <!-- <role rolename="tomcat"/> <role rolename="role1"
Read more