hdfs集成kerberos

@list l0:level1{ mso-level-number-format:decimal; mso-level-text:"%1."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level2{ mso-level-number-format:lower-alpha; mso-level-text:"%2."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level3{ mso-level-number-format:lower-roman; mso-level-text:"%3."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level4{ mso-level-number-format:decimal; mso-level-text:"%4."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level5{ mso-level-number-format:lower-alpha; mso-level-text:"%5."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level6{ mso-level-number-format:lower-roman; mso-level-text:"%6."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level7{ mso-level-number-format:decimal; mso-level-text:"%7."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level8{ mso-level-number-format:lower-alpha; mso-level-text:"%8."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level9{ mso-level-number-format:lower-roman; mso-level-text:"%9."; mso-level-number-position:left; font-family:Times New Roman; } @list l1:level1{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level2{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level3{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level4{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level5{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level6{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level7{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level8{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level9{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } 

• KERBEROS配置：

1.创建用户hdfs、所有节点配置免密

groupadd hadoop;

useradd hdfs -g hadoop -p hdfs;

2.服务端安装kerberos：

yum install -y krb5-libs krb5-server krb5-workstation

修改配置文件

vi /etc/krb5.conf

加密函数要加上

vi /var/kerberos/krb5kdc/kdc.conf

vi /var/kerberos/krb5kdc/kadm5.acl

3.初始化kerberos数据库

kdb5_util create -r HADOOP.COM -s 输入两次密码 HADOOP.COM是对应配置文件中的realm域名

初始化成功后会在/var/kerberos/krb5kdc目录下生成一些principal开头的文件

如果要重新初始化kerberos数据库，先删除/var/kerberos/krb5kdc目录下的principal文件 rm -f principal*

重启服务

systemctl restart krb5kdc

systemctl restart kadmin

***如果是麒麟V10，还要启动一个服务systemctl start sssd-kcm***

4.创建HADOOP.COM域内管理员

kadmin.local -q "addprinc root/admin" 输入两次密码

kadmin.local -q "ktadd root/admin" 导出管理员账户的keytab文件

klist -kt /etc/krb5.keytab

5.kerberos客户端配置

yum -y install krb5-libs krb5-workstation

将服务端/etc/krb5.conf文件拷贝到客户端/etc/krb5.conf

6.服务端创建hdfs的principal主体

kadmin.local -q "addprinc -randkey hdfs/node1@HADOOP.COM"

kadmin.local -q "addprinc -randkey hdfs/node2@HADOOP.COM"

kadmin.local -q "addprinc -randkey hdfs/node3@HADOOP.COM"

kadmin.local -q "addprinc -randkey HTTP/node1@HADOOP.COM"

kadmin.local -q "addprinc -randkey HTTP/node2@HADOOP.COM"

kadmin.local -q "addprinc -randkey HTTP/node3@HADOOP.COM"

kadmin.local -q "listprincs"

7.服务端创建keytab文件用于服务通过kerberos认证时免密

kadmin.local -q "xst -k hdfs-unmerged.keytab hdfs/node1@HADOOP.COM"

kadmin.local -q "xst -k hdfs-unmerged.keytab hdfs/node2@HADOOP.COM"

kadmin.local -q "xst -k hdfs-unmerged.keytab hdfs/node3@HADOOP.COM"

kadmin.local -q "xst -k http.keytab HTTP/node1@HADOOP.COM"

kadmin.local -q "xst -k http.keytab HTTP/node2@HADOOP.COM"

kadmin.local -q "xst -k http.keytab HTTP/node3@HADOOP.COM"

8.用kutil工具将生成的keytab文件合并成一个hdfs.keytab

ktutil

rkt hdfs-unmerged.keytab

rkt http.keytab

wkt hdfs.keytab

exit

klist -kt hdfs.keytab显示keytab文件列表

kinit -k -t hdfs.keytab hdfs/node1@HADOOP.COM

kinit -k -t hdfs.keytab HTTP/node1@HADOOP.COM

klist查看当前认证用户

9.服务端同步keytab文件到所有节点的keytab文件目录下

所有节点创建目录 mkdir /opt/hadoop-3.1.0/conf

scp hdfs.keytab node1:/opt/hadoop-3.1.0/conf/hdfs.keytab

scp hdfs.keytab node2:/opt/hadoop-3.1.0/conf/hdfs.keytab

scp hdfs.keytab node3:/opt/hadoop-3.1.0/conf/hdfs.keytab

node1节点：kinit -k -t /opt/hadoop-3.1.0/conf/hdfs.keytab hdfs/node1@HADOOP.COM

klist

node2节点：kinit -k -t /opt/hadoop-3.1.0/conf/hdfs.keytab hdfs/node2@HADOOP.COM

klist

node3节点：kinit -k -t /opt/hadoop-3.1.0/conf/hdfs.keytab hdfs/node3@HADOOP.COM

klist

10.安装JCE

CentOS和RedHat5.5或更高的版本中，对Kerberos 票证使用AES-256加密，因此必须在集群所有节点的JDK中安装Java Cryptography Extension（JCE）无限制强度加密策略文件

解压包unzip jce_policy-8.zip

所有节点备份文件目录

cp -r ${JAVA_HOME}/jre/lib/security ${JAVA_HOME}/jre/lib/security_bak

把UnlimitedJCEPolicyJDK8目录下的所有jar包(US_export_policy.jar和local_policy.jar)拷贝至所有节点

scp UnlimitedJCEPolicyJDK8/*.jar ${JAVA_HOME}/jre/lib/security

重启服务

systemctl restart krb5kdc

systemctl restart kadmin

@list l0:level1{ mso-level-number-format:decimal; mso-level-text:"%1."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level2{ mso-level-number-format:lower-alpha; mso-level-text:"%2."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level3{ mso-level-number-format:lower-roman; mso-level-text:"%3."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level4{ mso-level-number-format:decimal; mso-level-text:"%4."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level5{ mso-level-number-format:lower-alpha; mso-level-text:"%5."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level6{ mso-level-number-format:lower-roman; mso-level-text:"%6."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level7{ mso-level-number-format:decimal; mso-level-text:"%7."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level8{ mso-level-number-format:lower-alpha; mso-level-text:"%8."; mso-level-number-position:left; font-family:Times New Roman; } @list l0:level9{ mso-level-number-format:lower-roman; mso-level-text:"%9."; mso-level-number-position:left; font-family:Times New Roman; } @list l1:level1{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level2{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level3{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level4{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level5{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level6{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level7{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } @list l1:level8{ mso-level-number-format:bullet; mso-level-text:"¡"; mso-level-number-position:left; font-family:wingdings; } @list l1:level9{ mso-level-number-format:bullet; mso-level-text:""; mso-level-number-position:left; font-family:wingdings; } 

• HDFS配置：

1.设置环境变量

修改/etc/profile文件

export HADOOP_HOME=/opt/hadoop-3.1.0

export HADOOP_CONF_DIR=$HADOOP_HOME/etc/hadoop

export HADOOP_CLASSPATH=$HADOOP_HOME/share/hadoop

export PATH=$PATH:$HADOOP_CONF_DIR:$HADOOP_HOME/bin:$HADOOP_CLASSPATH

2.修改的配置文件有四个：core-site.xml、hdfs-site.xml、hadoop-env.sh、workers

按当前配置修改core-site.xml、hdfs-site.xml、workers文件

3.安装JSVC

由于datanode不走rpc协议而是采用http或https，gcdw目前只支持http协议的hdfs，datanode只用http需要安装JSVC，能够使一些Java程序使用root权限进行一些需要权限的操作，并进行用户之间的切换

tar zxvf commons-daemon-1.2.4-src.tar.gz

tar zxvf commons-daemon-1.2.4-bin.tar.gz

cd commons-daemon-1.2.4-src/src/native/unix/

./configure --with-java=/usr/local/jdk1.8.0_102

make

./jsvc --help

将编译后的jsvc程序 复制到所有节点的${HADOOP_HOME}/libexec 目录下

cp jsvc /opt/hadoop-3.1.0/libexec

删除所有节点${HADOOP_HOME}/share/hadoop/hdfs/lib下jar包commons-daemon-*.jar

复制解压后目录commons-daemon-1.2.4下的commons-daemon-1.2.4.jar到所有节点的${HADOOP_HOME}/share/hadoop/hdfs/lib 目录下

4.修改hadoop-env.sh

修改的地方有三处：

JSVC_HOME

JAVA_HOME

HDFS_DATANODE_SECURE_USER

5.拷贝配置好的core-site.xml、hdfs-site.xml、hadoop-env.sh、workers到所有节点

6.格式化namenode

namenode节点kinit -k -t /opt/hadoop-3.1.0/conf/hdfs.keytab hdfs/node1@HADOOP.COM

hadoop namenode -format

启动服务

hdfs --daemon start namenode

hdfs --daemon start secondarynamenode

cd $HADOOP_HOME && sbin/start-secure-dns.sh

问题：

除namenode外的两个节点datanode认证票据过期，貌似是kerberos在centos7.5上的bug

1560951 – RHEL7.5 - Hadoop Datanode service throws exception with Kerberos security enabled (redhat.com)

centos7.5yum下载的kerberos票据无限过期：

解决办法：换kerberos新版本

先上传较新版的rpm包（先卸载的话ssh依赖krb5-libs这个包中的库文件，卸载后ssh服务不可用，连不上服务器）

移除旧的包

rpm -e --nodeps krb5-devel-1.15.1-18.el7.x86_64 libkadm5-1.15.1-18.el7.x86_64 krb5-workstation-1.15.1-18.el7.x86_64 krb5-server-1.15.1-18.el7.x86_64 krb5-libs-1.15.1-18.el7.x86_64

安装包 rpm -ivh *.rpm 客户端不需要安装krb5-server-1.15.1-18.el7.x86_64

重新配置kerberos部分

gcdw安装kerberos认证的hdfs

主要参数：

GCDW_HDFS_NAMENODES=172.16.3.121:9870

GCDW_HDFS_URI=hdp://hdfs@172.16.3.121:9870/ -- 用principal中的hdfs用户，最后要有/

GCDW_HDFS_AUTH_MODE=kerberos

GCDW_HDFS_PRINCIPAL=hdfs/node1@HADOOP.COM

GCDW_HDFS_KEYTAB=/opt/hadoop-3.1.0/conf/hdfs.keytab -- 不需要新建principal导出keytab，可以与hdfs复用一个keytab

GCDW_HDFS_KERBEROS_CONFIG=/etc/krb5.conf

windows使用firefox浏览器访问带kerberos的namenodeweb界面：

Windows Kerberos客户端配置并访问CDH_mit kerberos客户端安装-CSDN博客

8a加载带kerberos的hdfs文件：

vi /etc/hosts补充hadoop节点域名解析

scp keytab文件到所有节点 用gbase用户kinit -kt keytab文件 principal主体

在gcluster层和gbase层均添加参数

gbase_hdfs_auth_mode=kerberos

gbase_hdfs_keytab=/opt/hdfs.keytab

gbase_hdfs_principal=hdfs/dws1@HADOOP.COM

gbase_hdfs_protocol=RPC -- 或使用http，下面的端口改成namenode的web端口9870

gbase_hdfs_port=9000

gbase_hdfs_namenode=dws1

_gbase_hdfs_rpcconfig='dfs.block.access.token.enable=true'

重启集群

加载hdfs文件

load data infile 'hdp://hdfs@172.16.3.121:9000/1.txt' into table t fields terminated by '|';