      Hadoop
 


Apache Hadoop                 .         ,  ,   HDFS  MapReduce,  Hadoop.





 

      Hadoop















Hadoop         Apache                  .








Hadoop         ,        .

 Hadoop         2005 .

         Nutch Search Engine   .

,       Yahoo,       Cloudera,        .

      Hadoop,     ,     .

 ,    Hadoop  ,     .

Hadoop      .

,    Hadoop,   ,           .

    Hadoop  .

   Hadoop      ,         .

  ,      ,     ,    -    ,       .

  Apache Hadoop  MapReduce  HDFS      Google MapReduce    Google.

    ,   Hadoop,     .

    ,      ,    ,         ,  ,   .

 ,      ,     ,    ,       ,        ,     .








 Apache Hadoop    .

 Hadoop Common,    Hadoop  HDFS, Hadoop MapReduce  Hadoop YARN.

Hadoop Common    ,     Hadoop.

   Hadoop     ,          .

Hadoop YARN     ,                 .

 Hadoop MapReduce    ,      .

    Hadoop    ,      .








    HDFS, YARN, MapReduce     ,     ,          .

    ,   Apache PIG, Apache Hive, HBase  .

   ,  Java- MapReduce,         .

         .

 Apache PIG  Apache Hive    ,       .

  Hadoop       Java          C    .








,        Hadoop.

  HDFS   ?

 ,     ,   Java    Hadoop.

 Hadoop       Namenode    Datanode,     .

   HDFS   ,  ,      .

   HDFS     .








   HTFS      NameNote,       NameNote     , ,        .

  ,   Hadoop,  -   MapReduce.








  MapReduce    ,        MapReduce.

         ,    .

 ,  Hadoop MapReduce     JobTracker     TaskTracker,  -       slave.

MapReduce      ,       HDFS  ,     ,    .

 MapReduce    ,        map  reduce,      -.

Hadoop         map  reduce   .

 Hadoop      MapReduce     map  reduce.

 map   map     .

  reduce   reduce    ,   map,    .

 map  reduce     ,      .








Hadoop  1   HDFS  Map Reduce.

 Hadoop  1      MapReduce.

 Hadoop  2    HDFS  YARN/Map Reduce  2.

  Map Reduce,     ,     slave   .

          .

  ,     ,      .

 YARN  Yet Another Resource Negotiator      .








YARN          slave ,        ,         .

 Map Reduce     ,     .

 Hadoop  2, YARN      /    .

YARN         ,   Map Reduce    ,    YARN.

 , YARN    ,   ,       .

  Map Reduce   ,    ,    ,    mapper  reducer.

YARN       ,  YARN     MapReduce.

  resource manager YARN        ,  Map Reduce.

       ,       ,    ,       .








 HDFS Yarn    ,        .

     ,     Hadoop     .

   ,       .

      ,     ,      ?

 Hadoop    Google MapReduce    ,       .








   Google Big Data.

      Google GFS.

 Google ,          ,      .

  - ,       .

 ,  Google    MapReduce,        .

  Google ,    ,                ,   SQL.

    MySQL Gateway,      MapReduce      .

  ,           MapReduce     .

  Sawzall.

  Evenflow            .

  . Dremel      ,           .

 , ,   -,      .

  Chubby    ,       ,    .








   Facebook Big Data.

  ,   Facebook   .

  Zookeeper,  Chubby,       .

  HBase,    HBase       MapReduce.

  Hive  Databee,   SQL .

  Scribe,      ,         .








,      Yahoo,  ,      ,      ,     .

LinkedIn      .

  ,   ,     ,      .

 ,   ,       ,    .








   Hadoop  CDH  Cloudera's distribution for Hadoop  Cloudera.

Cloudera    ,   Apache Hadoop      Hadoop.

      Sqoop, ,        Hadoop    ,     .

  Flume         .

  HBase      ,   HDFS.

Oozie        .

 Pig  Hive      .

    Zookeeper        .

        Cloudera,         ,   ,    .

      ,       Hadoop.

         ,        .

  ,   ,  Yahoo, Google  Facebook   ,    - ,        ,   .

         Big Data.

       Apache Sqoop.








Sqoop  SQL  Hadoop.

    ,            HDFS.

     Java,      ,   .

          SQL   Hadoop   Map Reduce      .








    Hbase.

Hbase     Hadoop,      ,         .

 Hbase   Google Big Table      ,      .








Pig    ,        MapReduce   Hadoop.

   Pig Latin,          .

Pig ,         Hadoop,   pig.

 ,  pig,       ,   JRuby, JPython  Java.

 ,     PIG   .

 ,      PIG            .








  Apache Hive       ,      .

Hive            SQL-     ,      .

     Hive QL.








Oozie      ,      Hadoop.

   Oozie   ,    DAG  Directed Graphs.

  Oozie       Oozie,       .

Oozie      Hadoop        Hadoop.








    Zookeeper.

       ,        -  .

   ,   Zookeeper.

      Hadoop.

       ,              .








 Flume            .

      ,    .

 Flume     ,        .








     Impala,      Cloudera,    ,   Hadoop.

Impala   Hadoop     .

         ,   HTFS  Hbase,        .

Impala   Hadoop      .

         Hadoop.

    SQL-           .








   ,  Spark.

 Hadoop      ,      ,           Hadoop.

 Spark      .

Apache Spark     Hadoop         .

     Hadoop,    MapReduce   , Spark         ,             ,  ,                .

 Spark   Scala,       .

   Spark     Spark,     Spark  Hadoop Yarn.

   , Spark      ,  HDFS, Amazon S3   -   .




Cloudera QuickStart VM









        Cloudera,     Cloudera Hadoop.








    ,   .








   VirtualBox    ovf.








    Cloudera QuickStart       .

      ,  ,       Cloudera.

  Hue, Hadoop, HBase, Impala, Spark,  . .

    Cloudera Hadoop.

    ,     ,    ,     URL .

      ,     .








  Overview NameNode Hadoop.

      Hadoop.

  ,     .

         ,    . .








   Datanodes.

         Datanodes.

,   HDFS    NameNode,  ,            .

    Datanodes,      ,   ,   .








  RegionServer HBase/

HBase     ,        Hadoop.

   ,          HBase.

       ,      .








Impala     SQL-   ,   HDFS.

      25  ,     ,    ,       ,      .








,    Oozie.

      ,    ,  . .








,     -,  ,   Start Tutorial.

        Cloudera.








   ,             DataCo.








          ?

    ,      ,   ,   .

,         .

   Cloudera   ,           .

     Scoop.

 ,   Map Reduce       Hadoop    .

      ,      .

  ,            ,        .

   Cloudera    Sqoop.

Sqoop1    .

 Scoop2       ,        .

,      .

      Cloudera,         Hadoop (HDFS).

   ,           HDFS,    .

 Apache Sqoop   .

  Sqoop       MySQL  HDFS,    .








    ,     Sqoop.

    MapReduce       MySQL        Avro  HDFS.

     Avro,       Hive     Impala.

Impala     .

 Avro    ,   Hadoop.








 ,        .








  ,  ,     HDFS,      .

            .








 Sqoop         .








     avsc      .

 ,       .

    ,    .

  ,      .

        SQL.

      ,  ,      ,      - .    ,       .








,      Apache Hive,    .

        HDFS,  Hive      .

  ,       Hive.

 Hive  Impala      HDFS,       .

   ,  Hive  ,     MapReduce.

    Impala     ,        ,      .

 ,      Sqoop  HTFS,     Avro,    ,       .

 ,     .








    Hue,  Impala,      .

   ,         Hue.

Hue  -,     8888.








   Hue,  loudera      .








   Query Editors  Impala.








   ,   .








     ,    .








,     ,     ,    .

     SQL          10  ,  .








 ,  Hue,    .








   ,        Impala.

,     .

    ,     Cloudera     .








                     .




  .


   .

   ,     (https://www.litres.ru/pages/biblio_book/?art=65077172)  .

      Visa, MasterCard, Maestro,    ,   ,     ,  PayPal, WebMoney, ., QIWI ,       .


