読者です 読者をやめる 読者になる 読者になる

"Diary" インターネットさんへの恩返し

いつもソースコードコピペばかりなので,みなさまへ少しばかりの恩返しを

【Ubuntu】CDH5をインストール~サンプルを動かすまで



スポンサーリンク

Cloudera社が提供しているHadoopディストリビューションCDHをUbuntuにインストールする。こちらに従いインストールしています。


CDHとは

CDHは、Apache Hadoopや関連プロジェクトすべてを包含し、機能検証済み、かつ、世界でもっとも導入実績の多いディストリビューションです。本ディストリビューションは、100% Apacheラインセンスに基づくオープンソース製品であり、Hadoopソリューションとしては唯一、バッチ処理インタラクティブSQLインタラクティブ検索、ロールベースのアクセスコントロール機能などを備えています。他のディストリビューションよりも、さらに多くの企業ユーザーにダウンロードされ使用されているディストリビューションと言えます。

環境

1.パッケージのダウンロード

*1の"On Ubuntu and other Debian systems, do the following"に従い実施。

$ wget http://archive.cloudera.com/cdh5/one-click-install/trusty/amd64/cdh5-repository_1.0_all.deb

2.パッケージのインストール

$ sudo dpkg -i cdh5-repository_1.0_all.deb

Selecting previously unselected package cdh5-repository.
(Reading database ... 100266 files and directories currently installed.)
Preparing to unpack cdh5-repository_1.0_all.deb ...
Unpacking cdh5-repository (1.0) ...
Setting up cdh5-repository (1.0) ...
gpg: keyring `/etc/apt/secring.gpg' created
gpg: keyring `/etc/apt/trusted.gpg.d/cloudera-cdh5.gpg' created
gpg: /etc/apt/trustdb.gpg: trustdb created
gpg: key 02A818DD: public key "Cloudera Apt Repository" imported
gpg: Total number processed: 1
gpg:               imported: 1


リポジトリの登録。

$ curl -s http://archive.cloudera.com/cdh5/ubuntu/lucid/amd64/cdh/archive.key | sudo apt-key add -

OK


リポジトリのアップデートと、hadoopのインストール。

$ sudo apt-get update 
$ sudo apt-get install hadoop-0.20-conf-pseudo

・
・
・
・

/usr/lib/hadoop-0.20-mapreduce/hadoop-core-mr1.jar
 * Failed to start Hadoop jobtracker. Return value: 1
invoke-rc.d: initscript hadoop-0.20-mapreduce-jobtracker, action "start" failed.
Setting up hadoop-0.20-mapreduce-tasktracker (2.6.0+cdh5.4.2+567-1.cdh5.4.2.p0.4                                                  ~trusty-cdh5.4.2) ...
/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.6.0-mr1-cdh5.4.2.jar
/usr/lib/hadoop-0.20-mapreduce/hadoop-core-mr1.jar
starting tasktracker, logging to /var/log/hadoop-0.20-mapreduce/hadoop-hadoop-ta                                                  sktracker-vagrant-ubuntu-trusty.out
/usr/lib/hadoop-0.20-mapreduce/hadoop-core-2.6.0-mr1-cdh5.4.2.jar
/usr/lib/hadoop-0.20-mapreduce/hadoop-core-mr1.jar
 * Failed to start Hadoop tasktracker. Return value: 1
invoke-rc.d: initscript hadoop-0.20-mapreduce-tasktracker, action "start" failed                                                  .
Processing triggers for ureadahead (0.100.0-16) ...
Setting up hadoop-0.20-conf-pseudo (2.6.0+cdh5.4.2+567-1.cdh5.4.2.p0.4~trusty-cd                                                  h5.4.2) ...
update-alternatives: using /etc/hadoop/conf.pseudo.mr1 to provide /etc/hadoop/co                                                  nf (hadoop-conf) in auto mode
Processing triggers for libc-bin (2.19-0ubuntu6.6) ...

エラーがでた。。。きにせず、次に進む。

3.HDFSのフォーマット

$ sudo -u hdfs hdfs namenode -format

4.HDFSの開始

$ for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done

webアクセスして画面が開くか確認。
http://localhost:50070/
f:id:azumami:20150527220149p:plain

必要なディレクトリの作成。

$ sudo /usr/lib/hadoop/libexec/init-hdfs.sh

mapreduceの起動。

$ for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x start ; done

http://localhost:50030/ にアクセスして以下のページがみれるか確認。
f:id:azumami:20150527221152p:plain


ユーザ作成

$ groupadd map
$ useradd -d /home/map -g map -m map
$ passwd map
$ sudo -u hdfs hadoop fs -mkdir -p /user/map
$ sudo -u hdfs hadoop fs -chown map /user/map


inputディレクトリを作成し、設定ファイルをコピー。コピーされたかを確認する。

map$ hadoop fs -mkdir input
map$ hadoop fs -put /etc/hadoop/conf/*.xml input
map$ hadoop fs -ls input
Found 4 items
-rw-r--r--   1 map map       2133 2015-05-27 13:20 input/core-site.xml
-rw-r--r--   1 map map       3032 2015-05-27 13:20 input/fair-scheduler.xml
-rw-r--r--   1 map map       1875 2015-05-27 13:20 input/hdfs-site.xml
-rw-r--r--   1 map map        582 2015-05-27 13:20 input/mapred-site.xml


サンプルプログラムの実行。メモリエラー。ちーん。

map$ /usr/bin/hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'
15/05/28 08:47:32 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/05/28 08:47:33 INFO mapred.FileInputFormat: Total input paths to process : 4
15/05/28 08:47:33 INFO mapred.JobClient: Running job: job_201505280830_0003
15/05/28 08:47:35 INFO mapred.JobClient:  map 0% reduce 0%
15/05/28 08:48:01 INFO mapred.JobClient:  map 25% reduce 0%
15/05/28 08:48:02 INFO mapred.JobClient:  map 50% reduce 0%
15/05/28 08:48:34 INFO mapred.JobClient:  map 50% reduce 17%
15/05/28 08:48:48 INFO mapred.JobClient: Task Id : attempt_201505280830_0003_m_000002_0, Status : FAILED
java.lang.Throwable: Child Error
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:250)
Caused by: java.io.IOException: Task process exit with nonzero status of 1.
        at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:237)

attempt_201505280830_0003_m_000002_0: OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000f3b3a000, 32268288, 0) failed; error='Cannot allocate memory' (errno=12)
attempt_201505280830_0003_m_000002_0: #
attempt_201505280830_0003_m_000002_0: # There is insufficient memory for the Java Runtime Environment to continue.
attempt_201505280830_0003_m_000002_0: # Native memory allocation (malloc) failed to allocate 32268288 bytes for committing reserved memory.
attempt_201505280830_0003_m_000002_0: # An error report file with more information is saved as:
attempt_201505280830_0003_m_000002_0: # /var/lib/hadoop-hdfs/cache/mapred/mapred/local/taskTracker/map/jobcache/job_201505280830_0003/attempt_201505280830_0003_m_000002_0/work/hs_err_pid3532.log


そもそもメモリサイズって。

$ cat /proc/meminfo
MemTotal:         501492 kB
MemFree:          154976 kB
MemAvailable:     227880 kB
SwapTotal:        522236 kB
SwapFree:         225972 kB

$ free -m
             total       used       free     shared    buffers     cached
Mem:           489        345        144          0          5         66
-/+ buffers/cache:        273        216
Swap:          509        284        225


swapを1GB増やす。こちらを参照。

$ sudo dd if=/dev/zero of=/swapfile1 bs=1M count=1024
1024+0 records in
1024+0 records out
1073741824 bytes (1.1 GB) copied, 6.55761 s, 164 MB/s


swapファイルの確認。

$ ll /swapfile1
-rw-r--r-- 1 root root 1073741824 Jun  3 04:10 /swapfile1


パーミッション変更。

$ sudo chmod 600 /swapfile1


フォーマット

$ sudo mkswap /swapfile1
Setting up swapspace version 1, size = 1048572 KiB
no label, UUID=0314552f-ebe3-4998-92b7-0e9e492c2142


現状のスワップ設定を確認。

$ swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda5                               partition       522236  136616  -1


有効化。

$ sudo swapon /swapfile1


増えました。

$ swapon -s
Filename                                Type            Size    Used    Priority
/dev/sda5                               partition       522236  135552  -1
/swapfile1                              file            1048572 0       -2


確認。

$ cat /proc/meminfo | grep Swap
SwapCached:        18392 kB
SwapTotal:       1570808 kB
SwapFree:        1438384 kB


hdfs/mapreduce起動。

$su - map
$for x in `cd /etc/init.d ; ls hadoop-hdfs-*` ; do sudo service $x start ; done
$for x in `cd /etc/init.d ; ls hadoop-0.20-mapreduce-*` ; do sudo service $x start ; done


サンプル実行。次はoutputディレクトリが残っている旨エラー。

$/usr/bin/hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'

・
・
・
・

15/06/03 04:22:46 INFO mapred.JobClient: Cleaning up the staging area hdfs://localhost:8020/var/lib/                                                                    hadoop-hdfs/cache/mapred/mapred/staging/map/.staging/job_201506030345_0002
15/06/03 04:22:46 WARN security.UserGroupInformation: PriviledgedActionException as:map (auth:SIMPLE                                                                    ) cause:org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:8020/                                                                    user/map/output already exists
org.apache.hadoop.mapred.FileAlreadyExistsException: Output directory hdfs://localhost:8020/user/map                                                                    /output already exists


以下で削除。

$sudo -u hdfs hadoop fs -rmdir --ignore-fail-on-non-empty /user/map/output


以下で削除されたか確認。

$ sudo -u hdfs hadoop fs -ls -R /


消えていない。おそらくこの事象。以下でできた。

$ sudo -u hdfs hdfs dfs -rm -r -skipTrash /user/map/output
Deleted /user/map/output


今度は成功。

~$ /usr/bin/hadoop jar /usr/lib/hadoop-0.20-mapreduce/hadoop-examples.jar grep input output 'dfs[a-z.]+'
15/06/03 05:55:45 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/06/03 05:55:47 INFO mapred.FileInputFormat: Total input paths to process : 4
15/06/03 05:55:50 INFO mapred.JobClient: Running job: job_201506030555_0001
15/06/03 05:55:51 INFO mapred.JobClient:  map 0% reduce 0%
15/06/03 05:56:33 INFO mapred.JobClient:  map 50% reduce 0%
15/06/03 05:57:05 INFO mapred.JobClient:  map 100% reduce 0%
15/06/03 05:57:09 INFO mapred.JobClient:  map 100% reduce 67%
15/06/03 05:57:13 INFO mapred.JobClient:  map 100% reduce 100%
15/06/03 05:57:21 INFO mapred.JobClient: Job complete: job_201506030555_0001
15/06/03 05:57:22 INFO mapred.JobClient: Counters: 33
15/06/03 05:57:22 INFO mapred.JobClient:   File System Counters
15/06/03 05:57:22 INFO mapred.JobClient:     FILE: Number of bytes read=204
15/06/03 05:57:22 INFO mapred.JobClient:     FILE: Number of bytes written=1256351
15/06/03 05:57:22 INFO mapred.JobClient:     FILE: Number of read operations=0
15/06/03 05:57:22 INFO mapred.JobClient:     FILE: Number of large read operations=0
15/06/03 05:57:22 INFO mapred.JobClient:     FILE: Number of write operations=0
15/06/03 05:57:22 INFO mapred.JobClient:     HDFS: Number of bytes read=8041
15/06/03 05:57:22 INFO mapred.JobClient:     HDFS: Number of bytes written=320
15/06/03 05:57:22 INFO mapred.JobClient:     HDFS: Number of read operations=10
15/06/03 05:57:22 INFO mapred.JobClient:     HDFS: Number of large read operations=0
15/06/03 05:57:22 INFO mapred.JobClient:     HDFS: Number of write operations=2
15/06/03 05:57:22 INFO mapred.JobClient:   Job Counters
15/06/03 05:57:22 INFO mapred.JobClient:     Launched map tasks=4
15/06/03 05:57:22 INFO mapred.JobClient:     Launched reduce tasks=1
15/06/03 05:57:22 INFO mapred.JobClient:     Data-local map tasks=4
15/06/03 05:57:22 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=135539
15/06/03 05:57:22 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=38986
15/06/03 05:57:22 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
15/06/03 05:57:22 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
15/06/03 05:57:22 INFO mapred.JobClient:   Map-Reduce Framework
15/06/03 05:57:22 INFO mapred.JobClient:     Map input records=218
15/06/03 05:57:22 INFO mapred.JobClient:     Map output records=6
15/06/03 05:57:22 INFO mapred.JobClient:     Map output bytes=186
15/06/03 05:57:22 INFO mapred.JobClient:     Input split bytes=419
15/06/03 05:57:22 INFO mapred.JobClient:     Combine input records=6
15/06/03 05:57:22 INFO mapred.JobClient:     Combine output records=6
15/06/03 05:57:22 INFO mapred.JobClient:     Reduce input groups=6
15/06/03 05:57:22 INFO mapred.JobClient:     Reduce shuffle bytes=222
15/06/03 05:57:22 INFO mapred.JobClient:     Reduce input records=6
15/06/03 05:57:22 INFO mapred.JobClient:     Reduce output records=6
15/06/03 05:57:22 INFO mapred.JobClient:     Spilled Records=12
15/06/03 05:57:22 INFO mapred.JobClient:     CPU time spent (ms)=8640
15/06/03 05:57:22 INFO mapred.JobClient:     Physical memory (bytes) snapshot=735318016
15/06/03 05:57:22 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=3910836224
15/06/03 05:57:22 INFO mapred.JobClient:     Total committed heap usage (bytes)=822448128
15/06/03 05:57:22 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
15/06/03 05:57:22 INFO mapred.JobClient:     BYTES_READ=7622
15/06/03 05:57:22 WARN mapred.JobClient: Use GenericOptionsParser for parsing the arguments. Applications should implement Tool for the same.
15/06/03 05:57:22 INFO mapred.FileInputFormat: Total input paths to process : 1
15/06/03 05:57:23 INFO mapred.JobClient: Running job: job_201506030555_0002
15/06/03 05:57:24 INFO mapred.JobClient:  map 0% reduce 0%
15/06/03 05:57:41 INFO mapred.JobClient:  map 100% reduce 0%
15/06/03 05:57:51 INFO mapred.JobClient:  map 100% reduce 100%
15/06/03 05:57:57 INFO mapred.JobClient: Job complete: job_201506030555_0002
15/06/03 05:57:57 INFO mapred.JobClient: Counters: 33
15/06/03 05:57:57 INFO mapred.JobClient:   File System Counters
15/06/03 05:57:57 INFO mapred.JobClient:     FILE: Number of bytes read=204
15/06/03 05:57:57 INFO mapred.JobClient:     FILE: Number of bytes written=496351
15/06/03 05:57:57 INFO mapred.JobClient:     FILE: Number of read operations=0
15/06/03 05:57:57 INFO mapred.JobClient:     FILE: Number of large read operations=0
15/06/03 05:57:57 INFO mapred.JobClient:     FILE: Number of write operations=0
15/06/03 05:57:57 INFO mapred.JobClient:     HDFS: Number of bytes read=434
15/06/03 05:57:57 INFO mapred.JobClient:     HDFS: Number of bytes written=150
15/06/03 05:57:57 INFO mapred.JobClient:     HDFS: Number of read operations=4
15/06/03 05:57:57 INFO mapred.JobClient:     HDFS: Number of large read operations=0
15/06/03 05:57:57 INFO mapred.JobClient:     HDFS: Number of write operations=2
15/06/03 05:57:57 INFO mapred.JobClient:   Job Counters
15/06/03 05:57:57 INFO mapred.JobClient:     Launched map tasks=1
15/06/03 05:57:57 INFO mapred.JobClient:     Launched reduce tasks=1
15/06/03 05:57:57 INFO mapred.JobClient:     Data-local map tasks=1
15/06/03 05:57:57 INFO mapred.JobClient:     Total time spent by all maps in occupied slots (ms)=20861
15/06/03 05:57:57 INFO mapred.JobClient:     Total time spent by all reduces in occupied slots (ms)=9178
15/06/03 05:57:57 INFO mapred.JobClient:     Total time spent by all maps waiting after reserving slots (ms)=0
15/06/03 05:57:57 INFO mapred.JobClient:     Total time spent by all reduces waiting after reserving slots (ms)=0
15/06/03 05:57:57 INFO mapred.JobClient:   Map-Reduce Framework
15/06/03 05:57:57 INFO mapred.JobClient:     Map input records=6
15/06/03 05:57:57 INFO mapred.JobClient:     Map output records=6
15/06/03 05:57:57 INFO mapred.JobClient:     Map output bytes=186
15/06/03 05:57:57 INFO mapred.JobClient:     Input split bytes=114
15/06/03 05:57:57 INFO mapred.JobClient:     Combine input records=0
15/06/03 05:57:57 INFO mapred.JobClient:     Combine output records=0
15/06/03 05:57:57 INFO mapred.JobClient:     Reduce input groups=1
15/06/03 05:57:57 INFO mapred.JobClient:     Reduce shuffle bytes=204
15/06/03 05:57:57 INFO mapred.JobClient:     Reduce input records=6
15/06/03 05:57:57 INFO mapred.JobClient:     Reduce output records=6
15/06/03 05:57:57 INFO mapred.JobClient:     Spilled Records=12
15/06/03 05:57:57 INFO mapred.JobClient:     CPU time spent (ms)=1550
15/06/03 05:57:57 INFO mapred.JobClient:     Physical memory (bytes) snapshot=294739968
15/06/03 05:57:57 INFO mapred.JobClient:     Virtual memory (bytes) snapshot=1568473088
15/06/03 05:57:57 INFO mapred.JobClient:     Total committed heap usage (bytes)=211836928
15/06/03 05:57:57 INFO mapred.JobClient:   org.apache.hadoop.mapreduce.lib.input.FileInputFormatCounter
15/06/03 05:57:57 INFO mapred.JobClient:     BYTES_READ=234


/user/mapって書かないと見えない。どっかでおかしくなったな。

$ sudo hadoop fs -ls /user/map/output
Found 3 items
-rw-r--r--   1 map supergroup          0 2015-06-03 05:57 /user/map/output/_SUCCESS
drwxr-xr-x   - map supergroup          0 2015-06-03 05:57 /user/map/output/_logs
-rw-r--r--   1 map supergroup        150 2015-06-03 05:57 /user/map/output/part-00000||


inputフォルダ内のファイルをgrep dfs[a-z.]+ した際の出現回数が表示される。

$ sudo hadoop fs -cat /user/map/output/part-00000 | head
1       dfs.datanode.data.dir
1       dfs.namenode.checkpoint.dir
1       dfs.namenode.name.dir
1       dfs.replication
1       dfs.safemode.extension
1       dfs.safemode.min.datanodes

hdfsユーザ、一般ユーザ、suの使い分けがよくわからんな。そこちゃんと勉強しよう。

【参考】hdfs/mapreduceの停止コマンド。

$ for x in `cd /etc/init.d ; ls hadoop-*` ; do sudo service $x stop ; done