Tuesday, 17 November 2015

HUE Integration with Resource manager HA



1) Edit your hue.ini add below parameters in yarn_cluster section


[[[default]]]
     security_enabled=${security_enabled}
     mechanism=${mechanism}
     history_server_api_url=http://node6.tuxhub.com:19888
     ssl_cert_ca_verify=False
     logical_name=my.cluster.com
     submit_to=True


     [[[ha]]]
     resourcemanager_host=node6.tuxhub.com
     resourcemanager_api_url=http://node6.tuxhub.com:8088
     proxy_api_url=http://node6.tuxhub.com:8088
     logical_name=my.cluster.com

     [[[ha1]]]
     resourcemanager_host=node5.tuxhub.com
     resourcemanager_api_url=http://node5.tuxhub.com:8088
     proxy_api_url=http://node5.tuxhub.com:8088
     logical_name=my.cluster.com


2) Restart hue.


Wednesday, 4 November 2015

Hive Storage Authorization




<configuration>
 <property>
    <name>javax.jdo.option.ConnectionURL</name>
    <value>jdbc:mysql://setup1:3306/hive?createDatabaseIfNotExist=true</value>
    <description>JDBC connect string for a JDBC metastore</description>
</property>
 <property>
    <name>javax.jdo.option.ConnectionDriverName</name>
    <value>com.mysql.jdbc.Driver</value>
    <description>Driver class name for a JDBC metastore</description>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionUserName</name>
    <value>hive</value>
    <description>username to use against metastore database</description>
 </property>
 <property>
    <name>javax.jdo.option.ConnectionPassword</name>
    <value>hive</value>
    <description>password to use against metastore database</description>
 </property>
 <property>
    <name>hive.metastore.uris</name>
    <value>thrift://setup1:9083</value>
 </property>
 <property><name>hive.server2.authentication</name><value>NOSASL</value></property>

<!-- SECURITY -->

<property>
    <name>hive.metastore.pre.event.listeners</name>
    <value>org.apache.hadoop.hive.ql.security.authorization.AuthorizationPreEventListener</value>
</property>


<property>
    <name>hive.security.metastore.authorization.manager</name>
    <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
</property>


<property>
    <name>hive.security.metastore.authenticator.manager</name>
    <value>org.apache.hadoop.hive.ql.security.HadoopDefaultMetastoreAuthenticator</value>
</property>

<property>
    <name>hive.security.authorization.manager</name>
        <value>org.apache.hadoop.hive.ql.security.authorization.StorageBasedAuthorizationProvider</value>
        </property>


<property>
        <name>hive.security.authorization.enabled</name>
        <value>true</value>
</property>


<property>
        <name>hive.server2.enable.doAs</name>
        <value>true</value>
</property>
<!-- SECURITY CONFIG DONE -->

Wednesday, 21 October 2015

Installation Kerberos


Kerberos Server configurations

kerberos server :- adp031.tuxhub.com
kerberos client :-  adp032.tuxhub.com
RELEM           :-  TUXHUB.COM


1) Install Packages :

[root@adp031 ~]# yum install krb5-libs krb5-server krb5-workstation

2) Edit config file :

[root@adp031 ~]# cat /etc/krb5.conf
[logging]
 default = FILE:/var/log/krb5libs.log
 kdc = FILE:/var/log/krb5kdc.log
 admin_server = FILE:/var/log/kadmind.log

[libdefaults]
 default_realm = TUXHUB.COM
 dns_lookup_realm = false
 dns_lookup_kdc = false
 ticket_lifetime = 24h
 renew_lifetime = 7d
 forwardable = true

[realms]
 TUXHUB.COM = {
  kdc = adp031.tuxhub.com
  admin_server = adp031.tuxhub.com
 }

[domain_realm]
 .tuxhub.com = TUXHUB.COM
 tuxhub.com = TUXHUB.COM
[root@adp031 ~]#


3) Edit kerberos database config file

[root@adp031 ~]# cat /var/kerberos/krb5kdc/kdc.conf
[kdcdefaults]
 kdc_ports = 88
 kdc_tcp_ports = 88

[realms]
 TUXHUB.COM = {
  master_key_type = aes256-cts
  acl_file = /var/kerberos/krb5kdc/kadm5.acl
  dict_file = /usr/share/dict/words
  admin_keytab = /var/kerberos/krb5kdc/kadm5.keytab
  supported_enctypes = aes256-cts:normal aes128-cts:normal des3-hmac-sha1:normal arcfour-hmac:normal des-hmac-sha1:normal des-cbc-md5:normal des-cbc-crc:normal
 }
[root@adp031 ~]#


4) Setup database for kdc.

[root@adp031 ~]# kdb5_util create -s
Loading random data
Initializing database '/var/kerberos/krb5kdc/principal' for realm 'TUXHUB.COM',
master key name 'K/M@TUXHUB.COM'
You will be prompted for the database Master Password.
It is important that you NOT FORGET this password.
Enter KDC database master key:
Re-enter KDC database master key to verify:

5) Edit acl file :

[root@adp031 ~]# cat /var/kerberos/krb5kdc/kadm5.acl
*/admin@TUXHUB.COM      *
[root@adp031 ~]#

6) Create kerberos principle 

[root@adp031 ~]# kadmin.local
Authenticating as principal root/admin@TUXHUB.COM with password.
kadmin.local:  addprinc root/admin
WARNING: no policy specified for root/admin@TUXHUB.COM; defaulting to no policy
Enter password for principal "root/admin@TUXHUB.COM":
Re-enter password for principal "root/admin@TUXHUB.COM":
Principal "root/admin@TUXHUB.COM" created.


7) Start services 


[root@adp031 ~]# /etc/init.d/krb5kdc start
Starting Kerberos 5 KDC:                                   [  OK  ]

[root@adp031 ~]# /etc/init.d/kadmin start
Starting Kerberos 5 Admin Server:                          [  OK  ]
[root@adp031 ~]#


[root@adp031 ~]# chkconfig krb5kdc on
[root@adp031 ~]# chkconfig kadmin on


8) Check configuration 

[root@adp031 ~]# kinit root/admin@TUXHUB.COM
Password for root/admin@TUXHUB.COM:

9) Verify TGT 

[root@adp031 ~]# klist
Ticket cache: FILE:/tmp/krb5cc_0
Default principal: root/admin@TUXHUB.COM

Valid starting     Expires            Service principal
10/21/15 15:50:57  10/22/15 15:50:57  krbtgt/TUXHUB.COM@TUXHUB.COM
        renew until 10/21/15 15:50:57
[root@adp031 ~]#


Kerberos Client configurations:

1) Install packages :

[root@adp032 ~]$ sudo yum install krb5-libs krb5-workstation

2) Copy krb5.conf file from server to all client node.

[root@adp031 ~]# scp -pr  /etc/krb5.conf adp032:/etc/
root@adp032's password:
krb5.conf                                                                             100%  437     0.4KB/s   00:00


 


Saturday, 17 October 2015

Yarn Memory and CPU allocation




1) Add below lines in yarn-site.xml

Note :- Calculate RAM per node. follow below links

http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.0.9.1/bk_installing_manually_book/content/rpm-chap1-11.html    OR

http://www.cloudera.com/content/www/en-us/documentation/enterprise/latest/topics/cdh_ig_yarn_tuning.html


[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml


<property>
    <name>yarn.nodemanager.resource.memory-mb</name>
    <value>3072</value>
</property>


<property>
    <name>yarn.scheduler.minimum-allocation-mb</name>
    <value>256</value>
</property>

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>3072</value>
</property>


<property>
        <name>yarn.app.mapreduce.am.resource.mb</name>
        <value>512</value>
</property>

<property>
        <name>yarn.app.mapreduce.am.command-opts</name>
        <value>-Xmx409m</value>
</property>




2) Add below mapred-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml

<property>
        <name>mapreduce.map.memory.mb</name>
                <value>256</value>
                </property>

<property>
        <name>mapreduce.map.java.opts</name>
                <value>-Xmx201m</value>
                </property>

<property>
        <name>mapreduce.reduce.memory.mb</name>
                <value>512</value>
                </property>


<property>
        <name>mapreduce.reduce.java.opts</name>
                <value>-Xmx410m</value>
</property>


3) Restart resourcemanager rand nodemanager on all nodes.

[hadoop@adp032 hadoop]$  yarn-daemon.sh stop resourcemanager;yarn-daemon.sh start resourcemanager

[hadoop@adp032 hadoop]$ yarn-daemon.sh stop nodemanager;yarn-daemon.sh start nodemanager



CPU Allocations :

4) Add below line in yarn-site.xml

[hadoop@adp032 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>1</value>
</property>

<property>
        <name>mapreduce.map.cpu.vcores</name>
        <value>1</value>
</property>


<property>
         <name>mapreduce.reduce.cpu.vcores</name>
         <value>1</value>
</property>

5)  Perform step 3 . Restart demons.

6) Verify changes on resourcemanager webui. 




7)  You may face below error

Diagnostics: Container [pid=24068,containerID=container_1445068083523_0001_02_000001] is running beyond virtual memory limits. Current usage: 106.7 MB of 512 MB physical memory used; 1.1 GB of 1.0 GB virtual memory used. Killing container.

This is happening on Centos/RHEL 6 due to its aggressive allocation of virtual memory.


Add following property in yarn-site.xml


<property>
   <name>yarn.nodemanager.vmem-check-enabled</name>
    <value>false</value>
</property>
Ref : http://stackoverflow.com/questions/21005643/container-is-running-beyond-memory-limits


Friday, 16 October 2015

Hive Openldap Integration



1) Add below line in hive-site.xml

[hadoop@adp031 conf]$ vim /usr/local/hadoop/hive/conf/hive-site.xml

<property>
  <name>hive.server2.authentication</name>
  <value>LDAP</value>
</property>

<property>
      <name>hive.server2.authentication.ldap.url</name>
       <value>ldap://adp034</value>
</property>

<property>
  <name>hive.server2.authentication.ldap.baseDN</name>
  <value>ou=Users,dc=tuxhub,dc=com</value>
</property>



2) Restart hiveserver2.

[hadoop@adp031 ~]$ nohup hiveserver2  &

3) Connect via beeline

[hadoop@adp031 conf]$ beeline
Beeline version 1.2.1 by Apache Hive
beeline> !connect jdbc:hive2://adp031:10000
Connecting to jdbc:hive2://adp031:10000
Enter username for jdbc:hive2://adp031:10000: nitin
Enter password for jdbc:hive2://adp031:10000: *****
Connected to: Apache Hive (version 1.2.1)
Driver: Hive JDBC (version 1.2.1)
Transaction isolation: TRANSACTION_REPEATABLE_READ
0: jdbc:hive2://adp031:10000>

0: jdbc:hive2://adp031:10000> show tables;
+-----------+--+
| tab_name  |
+-----------+--+
| hive1     |
+-----------+--+
1 row selected (0.326 seconds)
0: jdbc:hive2://adp031:10000>





Wednesday, 30 September 2015

Hadoop 2.6 Historyserver Configuration



                                           Hadoop 2.6 Historyserver Configuration


1. Add below parameter in yarn-site.xml

[hadoop@adp031 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/yarn-site.xml

<property>
    <name>yarn.log-aggregation-enable</name>
        <value>true</value>
        </property>

<property>
        <name>yarn.nodemanager.remote-app-log-dir</name>
            <value>/tmp/logs</value>
            </property>

<property>
    <name>yarn.log-aggregation.retain-seconds</name>
        <value>259200</value>
        </property>

<property>
    <name>yarn.log-aggregation.retain-check-interval-seconds</name>
        <value>3600</value>
        </property>

<property>
<name>yarn.log.server.url</name>
<value>http://adp031:19888/jobhistory/logs/</value>
</property>


2. Add Below parameter in mapred-site.xml

[hadoop@adp031 hadoop]$ cat /usr/local/hadoop/hadoop/etc/hadoop/mapred-site.xml

<property>
<name>mapreduce.jobhistory.address </name>
<value>adp031:10020</value>
</property>

<property>
<name>mapreduce.jobhistory.webapp.address</name>
<value>adp031:19888</value>
</property>

<property>
<name>mapreduce.jobhistory.intermediate-done-dir</name>
<value>/mr-history/tmp</value>
</property>

<property>
<name>mapreduce.jobhistory.done-dir</name>
<value>/mr-history/done</value>
</property>


3.  Start historyserver

[hadoop@adp031 hadoop]$ mr-jobhistory-daemon.sh start historyserver

4. Run a job 

yarn  jar /usr/local/hadoop/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar  wordcount /user/hadoop/IN_wordcount /user/hadoop/out_wordcount_12/

5. Go to web UI type http://<history server name>:19888/jobhistory






Monday, 7 April 2014

PUPPET INSTALLATION AND CONFIGURATIONS



Step 1 :-

Install RHEL EPEL repository on Centos 6.x on each system 


wget http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
wget http://rpms.famillecollet.com/enterprise/remi-release-6.rpm
sudo rpm -Uvh remi-release-6*.rpm epel-release-6*.rpm

Step 2 :- 
Install Puppet on system 
On Master sever :- 
[root@PUPPET-MASTER ] # yum install puppet-server
[root@PUPPET-MASTER ] # /etc/init.d/puppetmaster start
On  all client agent system  :- 
[root@PUPPET-CLIENT1 ~] # yum install puppet
[root@PUPPET-CLIENT1 ~] # /etc/init.d/puppet restart

Step 3 :-

Configuration


On all client agent system :-

[root@PUPPET-CLIENT1 ~] # vim /etc/puppet/puppet.conf

Add below parameter into agent section

server = PUPPET-MASTER
runinterval = 120


[root@PUPPET-CLIENT1 ~] # /etc/init.d/puppet restart
Step 4 :-
On Master sever :-
List all cerificates from all clinet connected to server.
[root@PUPPET-MASTER ] # puppetca -l
  "PUPPET-CLIENT1" (E3:2B:85:FD:56:2E:34:A3:E0:FF:7A:33:3A:36:33:8C)
Sign all cerificate using below command 
[root@PUPPET-MASTER ] # puppetca -s PUPPET-CLIENT1

Step 5 :-
On all client agent system
[root@PUPPET-CLIENT1 ~] # puppet agent --test
info: Caching catalog for ug-th-0215-nn
notice: Finished catalog run in 0.10 seconds
[root@PUPPET-CLIENT1 ~] #

Above command say successful connection to sever with certificate .
Step 6 :-
Push customize conf files from master to server.
[root@PUPPET-MASTER ] # vim  /etc/puppet/manifests/site.pp
Add below entry into site.pp
import 'nodes.pp'
Step 7 :-
Create nodes.pp file 
[root@PUPPET-MASTER ] # vim  /etc/puppet/manifests/nodes.pp 
node 'PUPPET-CLIENT1' {
include nginx
}

node 'PUPPET-CLIENT2' {
include nginx
}
[r
Step 8 :-

[root@PUPPET-MASTER ] # mkdir -p /etc/puppet/modules/nginx/{manifests,files}

lets create the nginx class file which lives in /etc/puppet/modules/nginx/manifests/init.pp

[root@PUPPET-MASTER ] # vim /etc/puppet/modules/nginx/manifests/init.pp

# Manage nginx webserver
class nginx {
package { 'nginx':
ensure => installed,
}

service { 'nginx':
ensure => running,
}

file { 'nginxconfig':
name => '/etc/nginx/nginx.conf',
source => 'puppet:///modules/nginx/nginx.conf',
}
}


Add customize file into /etc/puppet/modules/nginx/files/

Step 9 :-

[root@PUPPET-MASTER ] # puppet apply /etc/puppet/manifests/site.pp

Step 10 :-

Login to client system and check for /etc/nginx/nginx.conf you will get your customize file , it will take 2 min to update client system. 


Step 11(Optional) :-

Hadoop Manifests

# Make sure /etc/puppet/module/ have permission of puppet puppet , unzip your hadoop file /etc/puppet/modules/hadoop/files/


class hadoop{


group { "hadoop":
        ensure => present,
        gid    => 1000,
}

user { "hadoop":
        ensure     => present,
        shell      => "/bin/bash",
       managehome        =>  true,
home => "/home/hadoop",
password => '$1$jE8T0sFs$PMKB4bfP21IRqzZ14mwTR/',
}

file { "Hadoophome":
name   => "/home/hadoop",
  ensure => "directory",
         owner  => "hadoop",
         group  => "hadoop",
         mode   => 700,
}

file { "hadoopconf":
name => "/usr/local/hadoop",
recurse => true,
ensure => "directory",
         owner  => "hadoop",
         group  => "hadoop",
mode   => 755,
source => 'puppet:///modules/hadoop/hadoop/',

}

Ansible Cheat sheet

Install Ansible  # yum install ansible Host file configuration  File  [ansible@kuber2 ~]$ cat /etc/ansible/hosts     [loca...