Hadoop Conference 2011 Fall

日時：2011/09/26
場所：ベルサール汐留

イベントの詳細：
http://hadoop-conference-japan-2011-fall.eventbrite.com/
【重要】イベントのアーカイブ
http://mit.recruit.co.jp/hadoop/conference2011fall/info/archive.html
———————

『Hadoop 0.23 and MapReduce v2』
HortonWorks, Owen O’Malley

□Current Hadoop Branches
0.21はstableじゃないから使っちゃダメ

・0.20.203.0
added security
MapReduce job limits
Performance work

・0.20.204.0
fail in place
RPM & Debian package

・0.20.205.0
HBase support

・0.23
Expected to become the next stable release

a community effort from
cloudera, ebay, hortonworks, yahoo…

includes many new features:
– Hdfs federation
— a solution to HDFS Namenode scaling
— Entire HDFS namespace kept in NameNode’s RAM

– hdfs write-pipeline improvements with support for HBase supports
shuffle optimized by 30%
small mapreduece jobs optimization



Current Limitations
– Scalability
– SPOF
– Restart is very tricky due to complex state
– Hard Partition of resources into map and reduce slots
– Lacks support for alternate paradigms(Iterative applications ex. K-means, PageRank)
– Lack of wire-compatible(need to be same versions)


Design
- cluster resource management
- application life-cycle management


Improvements
- NO SPOF state saved in ZooKeeper

複数のバージョンのMapReduceが動く
決まった数のmapとreduceの設定の廃止

グラフ構造など次世代の処理に対応
（これまではイテレーションによる段数のコストが大きかったため）


status
ベータ版は2011年Q4

https://developer.yahoo.com/blogs/hadoop/posts/2011/02/mapreduce-nextgen

TAGS: None