CentOS7 安装 Cloudera Manager

环境

配置 ssh 免密登陆

• 在 cm0 上配置 ssh 可免密登陆全部服务器

  1 2	ssh-keygen seq -f'192.168.1.%g' 100 106 | xargs -L1 ssh-copy-id

• 这里的私钥会在后面通过界面增加主机时用到

关闭防火墙和 selinux

• 在全部服务器上关闭防火墙

  1 2	systemctl stop firewalld systemctl disable firewalld

• 在全部服务器上关闭 selinux

  1 2	setenforce 0 sed -i '/^SELINUX=/cSELINUX=disabled' /etc/selinux/config

配置网络名称

• 在全部服务器上修改 /etc/hosts，增加如下解析记录

  1 2 3 4 5 6 7

  192.168.1.100 cm0.colben.cn cm0 192.168.1.101 gw0.colben.cn gw0 192.168.1.102 m0.colben.cn m0 192.168.1.103 m1.colben.cn m1 192.168.1.104 w0.colben.cn w0 192.168.1.105 w1.colben.cn w1 192.168.1.106 w2.colben.cn w2

• 在全部服务器上修改 /etc/sysconfig/network，增加各自主机名设置

  1	HOSTNAME=XXXX.colben.cn

• 在全部服务器上设置各自主机名

  1	hostnamectl set-hostname XXXX.colben.cn

配置时间同步

• 在 cm0 上配置修改 /etc/chrony.conf

  1 2	server ntp.aliyun.com iburst allow 192.168.1.0/24

• 在其他服务器上修改 /etc/chrony.conf

  1	server cm0.colben.cn iburst

• 在全部服务器上重启 chronyd 服务

  1	systemctl restart chronyd

配置 cloudera manager 内网安装源

• 在 cm0 上安装并启动 httpd 服务

  1 2 3

  yum install httpd systemctl enable httpd systemctl start httpd

• 在 cm0 上下载 cloudera manager yum 仓库

  1 2 3 4

  mkdir -p /var/www/html/cloudera-repos/cm6/6.3.1 wget https://archive.cloudera.com/cm6/6.3.1/repo-as-tarball/cm6.3.1-redhat7.tar.gz tar zxf cm6.3.1-redhat7.tar.gz -C /var/www/html/cloudera-repos/cm6/6.3.1 --strip-components=1 chmod -R ugo+rX /var/www/html/cloudera-repos/cm6

• 在 cm0 上下载 cdh yum 仓库(体积较小，需手动升级)

  1 2 3 4 5 6 7

  mkdir -p /var/www/html/cloudera-repos wget --recursive --no-parent --no-host-directories \ https://archive.cloudera.com/cdh6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos wget --recursive --no-parent --no-host-directories \ https://archive.cloudera.com/gplextras6/6.3.2/redhat7/ -P /var/www/html/cloudera-repos chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6 chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

• 在 cm0 上下载 cdh parcel 仓库(体积较大，可通过界面操作自动升级)

  1 2 3 4 5 6 7

  mkdir -p /var/www/html/cloudera-repos wget --recursive --no-parent --no-host-directories \ https://archive.cloudera.com/cdh6/6.3.2/parcels/ -P /var/www/html/cloudera-repos wget --recursive --no-parent --no-host-directories \ https://archive.cloudera.com/gplextras6/6.3.2/parcels/ -P /var/www/html/cloudera-repos chmod -R ugo+rX /var/www/html/cloudera-repos/cdh6 chmod -R ugo+rX /var/www/html/cloudera-repos/gplextras6

• 在 cm0 上创建 cloudera.repo，并重建 yum 缓存

  1 2 3 4 5 6 7 8 9

  cat > /etc/yum.repos.d/cloudera.repo <<-EOF [cloudera-manager] name=cloudera-manager baseurl=http://cm0.colben.cn/cloudera-repos/cm6/ enabled=1 gpgcheck=0 EOF yum clean all yum makecache fast

• 通过界面增加其他主机时，cloudera manager 会自动部署该 repo 文件

安装 jdk

• 在 cm0 上安装 jdk

  1	yum install oracle-j2sdk1.8

• 通过界面增加其他主机时，可自动安装 oracle-j2sdk1.8

安装 cloudera manager

• 在 cm0 上安装 cloudera-manager

  1	yum install cloudera-manager-daemons cloudera-manager-agent cloudera-manager-server

配置 cloudera manager 数据库

• 在 cm0 上安装 mysql，参考这里

• 官方推荐 my.cnf

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30

  [mysqld] transaction-isolation = READ-COMMITTED symbolic-links = 0 sql_mode=STRICT_ALL_TABLES key_buffer_size = 32M max_allowed_packet = 32M thread_stack = 256K thread_cache_size = 64 query_cache_limit = 8M query_cache_size = 64M query_cache_type = 1 max_connections = 600 log_bin=/var/lib/mysql/mysql_binary_log server_id=1 binlog_format = mixed read_buffer_size = 2M read_rnd_buffer_size = 16M sort_buffer_size = 8M join_buffer_size = 8M innodb_file_per_table = 1 innodb_flush_log_at_trx_commit = 2 innodb_log_buffer_size = 64M innodb_buffer_pool_size = 4G innodb_thread_concurrency = 8 innodb_flush_method = O_DIRECT innodb_log_file_size = 512M

• 在 cm0 上启动 mysql

  1 2	systemctl enable mysqld systemctl start mysqld

• 在 cm0 上的 mysql 中创建数据库和用户

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18

  CREATE DATABASE scm DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE amon DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE rman DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hue DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE metastore DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE sentry DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE nav DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE navms DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE oozie DEFAULT CHARSET utf8 DEFAULT COLLATE utf8_general_ci; GRANT ALL ON scm.* to scm@'%' identified by 'Pass-1234'; GRANT ALL ON amon.* to amon@'%' identified by 'Pass-1234'; GRANT ALL ON rman.* to rman@'%' identified by 'Pass-1234'; GRANT ALL ON hue.* to hue@'%' identified by 'Pass-1234'; GRANT ALL ON metastore.* to hive@'%' identified by 'Pass-1234'; GRANT ALL ON sentry.* to sentry@'%' identified by 'Pass-1234'; GRANT ALL ON nav.* to nav@'%' identified by 'Pass-1234'; GRANT ALL ON navms.* to navms@'%' identified by 'Pass-1234'; GRANT ALL ON oozie.* to oozie@'%' identified by 'Pass-1234';

• 在 cm0 上初始化数据库

  1 2 3 4 5 6 7 8 9

  /opt/cloudera/cm/schema/scm_prepare_database.sh mysql scm scm /opt/cloudera/cm/schema/scm_prepare_database.sh mysql amon amon /opt/cloudera/cm/schema/scm_prepare_database.sh mysql rman rman /opt/cloudera/cm/schema/scm_prepare_database.sh mysql hue hue /opt/cloudera/cm/schema/scm_prepare_database.sh mysql metastore hive /opt/cloudera/cm/schema/scm_prepare_database.sh mysql sentry sentry /opt/cloudera/cm/schema/scm_prepare_database.sh mysql nav nav /opt/cloudera/cm/schema/scm_prepare_database.sh mysql navms navms /opt/cloudera/cm/schema/scm_prepare_database.sh mysql oozie oozie

• 在全部服务器上安装 mysql connector

  1 2 3 4 5

  wget https://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.46.tar.gz tar zxvf mysql-connector-java-5.1.46.tar.gz mkdir -p /usr/share/java/ cp mysql-connector-java-5.1.46/mysql-connector-java-5.1.46-bin.jar \ /usr/share/java/mysql-connector-java.jar

启动 cloudera manager

• 在 cm0 上启动 cloudera manager

  1	systemctl start cloudera-scm-server

• 日志: /var/log/cloudera-scm-server/cloudera-scm-server.log

• 日志出现如下信息，表示 cloudera-scm-server 启动完成

  1	INFO WebServerImpl:com.cloudera.server.cmf.WebServerImpl: Started Jetty server.

• 浏览器访问

  • 地址: http://cm0.colben.cn:7180/
  • 用户: admin
  • 密码: admin