Hadoop And cloud : October 2015

Wednesday, 21 October 2015

Installation Kerberos

Kerberos Server configurations

kerberos server :- adp031.tuxhub.com
kerberos client :- adp032.tuxhub.com
RELEM :- TUXHUB.COM

1) Install Packages :

[root@adp031 ~]# yum install krb5-libs krb5-server krb5-workstation

2) Edit config file :

[root@adp031 ~]# cat /etc/krb5.conf

[logging]

default = FILE:/var/log/krb5libs.log

kdc = FILE:/var/log/krb5kdc.log

admin_server = FILE:/var/log/kadmind.log

[libdefaults]

default_realm = TUXHUB.COM

dns_lookup_realm = false

dns_lookup_kdc = false

ticket_lifetime = 24h

renew_lifetime = 7d

forwardable = true

[realms]

TUXHUB.COM = {

kdc = adp031.tuxhub.com

admin_server = adp031.tuxhub.com

}

[domain_realm]

.tuxhub.com = TUXHUB.COM

tuxhub.com = TUXHUB.COM

[root@adp031 ~]#

3) Edit kerberos database config file

[root@adp031 ~]# cat /var/kerberos/krb5kdc/kdc.conf

[kdcdefaults]

kdc_ports = 88

kdc_tcp_ports = 88

[realms]

TUXHUB.COM = {

master_key_type = aes256-cts

acl_file = /var/kerberos/krb5kdc/kadm5.acl

dict_file = /usr/share/dict/words

admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab

supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal

}

[root@adp031 ~]#

4) Setup database for kdc.

[root@adp031 ~]# kdb5_util create -s

Loading random data

Initializing database '/var/kerberos/krb5kdc/principal' for realm 'TUXHUB.COM',

master key name 'K/M@TUXHUB.COM'

You will be prompted for the database Master Password.

It is important that you NOT FORGET this password.

Enter KDC database master key:

Re-enter KDC database master key to verify:

5) Edit acl file :

[root@adp031 ~]# cat /var/kerberos/krb5kdc/kadm5.acl

*/admin@TUXHUB.COM *

[root@adp031 ~]#

6) Create kerberos principle

[root@adp031 ~]# kadmin.local

Authenticating as principal root/admin@TUXHUB.COM with password.

kadmin.local: addprinc root/admin

WARNING: no policy specified for root/admin@TUXHUB.COM; defaulting to no policy

Enter password for principal "root/admin@TUXHUB.COM":

Re-enter password for principal "root/admin@TUXHUB.COM":

Principal "root/admin@TUXHUB.COM" created.

7) Start services

[root@adp031 ~]# /etc/init.d/krb5kdc start

Starting Kerberos 5 KDC: [ OK ]

[root@adp031 ~]# /etc/init.d/kadmin start

Starting Kerberos 5 Admin Server: [ OK ]

[root@adp031 ~]#

[root@adp031 ~]# chkconfig krb5kdc on

[root@adp031 ~]# chkconfig kadmin on

8) Check configuration

[root@adp031 ~]# kinit root/admin@TUXHUB.COM

Password for root/admin@TUXHUB.COM:

9) Verify TGT

[root@adp031 ~]# klist

Ticket cache: FILE:/tmp/krb5cc_0

Default principal: root/admin@TUXHUB.COM

Valid starting Expires Service principal

10/21/15 15:50:57 10/22/15 15:50:57 krbtgt/TUXHUB.COM@TUXHUB.COM

renew until 10/21/15 15:50:57

[root@adp031 ~]#

Kerberos Client configurations:

1) Install packages :

[root@adp032 ~]$ sudo yum install krb5-libs krb5-workstation

2) Copy krb5.conf file from server to all client node.

[root@adp031 ~]# scp -pr /etc/krb5.conf adp032:/etc/

root@adp032's password:

krb5.conf 100% 437 0.4KB/s 00:00

Saturday, 17 October 2015

Yarn Memory and CPU allocation

1) Add below lines in yarn-site.xml

Note :- Calculate RAM per node. follow below links

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html OR

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<property>
<name>yarn.nodemanager.resource.memory-mb</name>
<value>3072</value>
</property>

<property>
<name>yarn.scheduler.minimum-allocation-mb</name>
<value>256</value>
</property>

<property>
<name>yarn.scheduler.maximum-allocation-mb</name>
<value>3072</value>
</property>

<property>
<name>yarn.app.mapreduce.am.resource.mb</name>
<value>512</value>
</property>

<property>
<name>yarn.app.mapreduce.am.command-opts</name>
<value>-Xmx409m</value>
</property>

2) Add below mapred-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml

<property>
<name>mapreduce.map.memory.mb</name>
<value>256</value>
</property>

<property>
<name>mapreduce.map.java.opts</name>
<value>-Xmx201m</value>
</property>

<property>
<name>mapreduce.reduce.memory.mb</name>
<value>512</value>
</property>

<property>
<name>mapreduce.reduce.java.opts</name>
<value>-Xmx410m</value>
</property>

3) Restart resourcemanager rand nodemanager on all nodes.

[hadoop@adp032 hadoop]$ yarn-daemon.sh stop resourcemanager;yarn-daemon.sh start resourcemanager

[hadoop@adp032 hadoop]$ yarn-daemon.sh stop nodemanager;yarn-daemon.sh start nodemanager

CPU Allocations :

4) Add below line in yarn-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<name>yarn.nodemanager.resource.cpu-vcores</name>

</property>

<name>mapreduce.map.cpu.vcores</name>

</property>

<name>mapreduce.reduce.cpu.vcores</name>

</property>

5) Perform step 3 . Restart demons.

6) Verify changes on resourcemanager webui.

7) You may face below error

Diagnostics: Container [pid=24068,containerID=container_1445068083523_0001_02_000001] is running beyond virtual memory limits. Current usage: 106.7 MB of 512 MB physical memory used; 1.1 GB of 1.0 GB virtual memory used. Killing container.

This is happening on Centos/RHEL 6 due to its aggressive allocation of virtual memory.

Add following property in yarn-site.xml

<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>

Ref : http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits

Friday, 16 October 2015

Hive Openldap Integration

1) Add below line in hive-site.xml

[hadoop@adp031 conf]$ vim /usr/local/hadoop/hive/conf/hive-site.xml

<property>
<name>hive.server2.authentication</name>
<value>LDAP</value>
</property>

<property>
<name>hive.server2.authentication.ldap.url</name>
<value>ldap://adp034</value>
</property>

<property>
<name>hive.server2.authentication.ldap.baseDN</name>
<value>ou=Users,dc=tuxhub,dc=com</value>
</property>

2) Restart hiveserver2.

[hadoop@adp031 ~]$ nohup hiveserver2 &

3) Connect via beeline

[hadoop@adp031 conf]$ beeline
Beeline version 1.2.1 by Apache Hive
beeline> !connect jdbc:hive2://adp031:10000
Connecting to jdbc:hive2://adp031:10000
Enter username for jdbc:hive2://adp031:10000: nitin
Enter password for jdbc:hive2://adp031:10000: *****
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://adp031:10000>

0: jdbc:hive2://adp031:10000> show tables;
+-----------+--+
| tab_name |
+-----------+--+
| hive1 |
+-----------+--+
1 row selected (0.326 seconds)
0: jdbc:hive2://adp031:10000>

Hadoop And cloud

Wednesday, 21 October 2015

Installation Kerberos

Saturday, 17 October 2015

Yarn Memory and CPU allocation

Friday, 16 October 2015

Hive Openldap Integration

Ansible Cheat sheet

Search This Blog