Cloud Computing:Hadoop

1. Apache Hadoop

1.1. Overview

Apache Hadoop is a software solution for distributed computing of large datasets. Hadoop provides a distributed filesystem (HDFS) and a MapReduce implementation.
A special computer acts as the "name node". This computer saves the information about the available clients and the files. The Hadoop clients (computers) are called nodes.
The "name node" is currently a single point of failure. The Hadoop project is working on solutions for this.

1.2. Typical tasks

Apache Hadoop can be used to filter and aggregate data, e.g. a typical usecase would be the analysis of webserver logs file to find the most visisted pages. But MapReduce has been use to transverse graphs and other tasks.

1.3. Writing the map and reduce functions

Hadoop allow that map and reduce functions are written in Java. Hadoop provides also linker so that map and reduce funtions can be written in other languages, e.g. C++, Python, Per, etc.

2. Hadoop file system

The Hadoop file system (HDSF) is a distribute file system. It use an existing file system of the operating system but extends this with redundancy and distribution. HSDF hides the complexity of distributed storage and redundancy from the programmer.
In the standard configuration HDFS saves all files three times on different nodes. The "name node" (server) has the information where the files are stored.
Harddisks are very effective in reading large files sequentially but getting much slower during random access. HDFS is therefore optimized for large files.
To improve performance Hadoop also tries to move the computation to the nodes which store the data and not vice versa. Especially if you have very large data this help improving the performance as you can avoid that the network becomes the bottleneck.

3. MapReduce

Apache Hadoop jobs are working according to the MapRecude principle. See MapReduce for details.

4. Installation

Apache Hadoop can be downloaded from Hadoop Homepage. To get started with Hadoop you require the following sub-projects:

Comments

Post a Comment

Popular posts from this blog

Installing Qt Jambi For java

Installing from a Prebuilt Package After extracting the package, you are ready to run the Launcher, which is an application that runs the examples and demos that come with Qt Jambi. To run the examples and demo launcher on Windows run: qtjambi.exe On Linux and Mac OS X run: qtjambi.sh Running these files will verify the virtual machine version and set the correct environment for running Qt Jambi applications. When you are ready to distribute your own applications, see the  deployment  document. Manually Launching Qt Jambi Applications To manually launch applications, you need to include the following  .jar  files in your classpath: qtjambi-4.4.0_01.jar , which contains the Jambi classes. The platform archive, which contains the native libraries used by Qt Jambi. The archive is named after the operating system and compiler used to build the libraries; for instance,  qtjambi-win32-msvc2005-4.4.0_01.jar  for Windows and MSVC 2005. To run the ex...
Read more

What Is Tomcat Default Administrator Password ?

By default, Tomcat does not enable admin or manager access. To enable it, you have to edit the “ %TOMCAT_FOLDER%/conf/tomcat-users.xml ” manually. File : tomcat-users.xml (before update) , initially, Tomcat comments all users and roles like above. <?xml version = '1.0' encoding = 'utf-8' ?> <tomcat-users> <!-- <role rolename="tomcat"/> <role rolename="role1"/> <user username="tomcat" password="tomcat" roles="tomcat"/> <user username="both" password="tomcat" roles="tomcat,role1"/> <user username="role1" password="tomcat" roles="role1"/> --> </tomcat-users> File : tomcat-users.xml (after updated) <?xml version = '1.0' encoding = 'utf-8' ?> <tomcat-users> <!-- <role rolename="tomcat"/> <role rolename="role1"...
Read more

Install NS3

Install NS3 (ns-allinone-3.17) on Ubuntu 12.04/14.04 for beginners Installation steps for NS3 (ns-allinone-3.17): Follow the given steps for the installation of NS-3. First install some dependencies to enable the NS-3 execution environment on your Ubuntu 12.04/14.04 LTS machine. sudo apt-get install gcc g++ python sudo apt-get install gcc g++ python python-dev sudo apt-get install mercurial sudo apt-get install bzr sudo apt-get install gdb valgrind sudo apt-get install gsl-bin libgsl0-dev libgsl0ldbl sudo apt-get install flex bison libfl-dev sudo apt-get install g++-3.4 gcc-3.4 sudo apt-get install tcpdump sudo apt-get install sqlite sqlite3 libsqlite3-dev sudo apt-get install libxml2 libxml2-dev sudo apt-get install libgtk2.0-0 libgtk2.0-dev sudo apt-get install vtun lxc sudo apt-get install uncrustify sudo apt-get install doxygen graphviz imagemagick sudo apt-get install texlive texlive-extra-utils texlive-latex-extra sudo apt-get install python-s...
Read more