Skip to content

README.md

Latest commit

 

History

History
34 lines (28 loc) · 850 Bytes

README.md

File metadata and controls

34 lines (28 loc) · 850 Bytes
Raw

Learning Hadoop

  1. Platform:
    Ubunut 16.04
    Hadoop 2.9.0
    Spark 2.2.1
    Hbase 1.4.0
    Hive 2.1.1
    Pig 0.17.0

  2. Project of Recognize Spider:
    Recognize Spirder from IIS Logs with the percent of visit pages per IP, visit resource files or not, percent of visit time per IP.
    Analysis it on Spark by scala.
    files:
    /bin/run-recognize-spider.sh
    /bin/init-recognize-hbase.sh

  3. Projects of Hello World:

  • WordCount on MapReduce by JAVA.
    files:
    /bin/run-wordcount.sh
    /src/WordCount.java

  • WordCount on Spark by Scala.
    files: /bin/run-wordcount-scala.sh

  • One table join itself on MapReduce

    • Stored into Derby by Hive
      files: /bin/run-selfjoin-hive.sh    
    • Stored into Hbase by Hive
      files: /bin/run-selfjoin-hbase-hive.sh
    • Stored into Hbase by Pig
      files: /bin/run-selfjoin-pig.sh