HDFS TO AWS S3

Step 1 :-

Login to Cloudera manager

Go to services and Click on hdfs

Go to configuration and click on view and edit roles

Click on service-wide configuration

Click on advance

Step 2 :-

Add below details in Cluster-wide Configuration Safety Value for core-site.xml

<property>

<name>fs.s3n.awsAccessKeyId</name>

<value>XXXXXXXXXXXXXXXXXXX</value>

</property>

<property>

<name>fs.s3n.awsSecretAccessKey</name>

<value>XXXXXXXXXXXXXXXXXXXXXXX</value>

</property>

Step 3 :-

Save configuration. 

Click on Action and Deploy Configuration 

Restart hdfs once 

Step 4:-

[nitin@nitin-ubuntu ~]# sudo -u hdfs hadoop distcp s3n://big-store/getting_price.sh hdfs://<CLSUTER1>:8020/user/nitin/

[nitin@nitin-ubuntu ~]# sudo -u hdfs hadoop distcp s3n://big-store/getting_price.sh hdfs://10.10.10.216:8020/user/nitin/

BACKUP HIVE META-STORE WITH POSTGRESQL IN CDH4.X

1) Go to scm server database

[nitin@nitin-ubuntu:~] # cd /var/lib/cloudera-scm-server-db/data

2) check file generated_password.txt . This file is cerated by cloudera manager

[nitin@nitin-ubuntu:~] # cat generated_password.txt

8UlBunj0MM

The password above was generated by /usr/share/cmf/bin/initialize_embedded_db.sh (part of the cloudera-manager-server-db package)

and is the password for the user 'cloudera-scm' for the database in the current directory.

Generated at 20140128-230553.

3) Login to PostgreSQL with password from above file that is 8UlBunj0MM

[nitin@nitin-ubuntu:~] # psql --user cloudera-scm --port=7432 –dbname=postgres

Password for user cloudera-scm:************

postgres=# \q

Note :- Make sure you give exact username and password

4) Take A dump

[nitin@nitin-ubuntu:~] # pg_dump hive -U cloudera-scm --port=7432 > hive.sql

Password:********

5) Its Done.

ACCESS HBASE TABLE WITH TABLEAU DESKTOP 8.0

Hope you have Tableau installed on system .

Concept :-

You can't directly connect to hbase  table via tableau you need to connect to hive table and hive internally mapped to hbase table.

Please check below link for more explanation :

http://nosql.mypopescu.com/post/17262685876/visualizing-hadoop-data-with-tableau-software-and

Step 1 :- 

Download Tableau driver for hive

http://www.cloudera.com/content/support/en/downloads/download-components/download-products/downloads-listing/connectors/tableau.html

Step 2 :- (Driver installation)

Install Above downloaded driver.

Step 3 :- (Configure ODBC driver)

Click on start go to Data Source (ODBC).

Click on System DSN.

Select Cloudera ODBC driver for Apache Hive.

Fill the details.

Save Setting.

Step 4 :- (Run Hive as Thrift service)

[ nitin@nitin-ubuntu:~ # ] $ sudo hive --service hiveserver --hiveconf /etc/hive/conf/hive-site.xml

Make sure you have auxpath set in above hive-site.xml and all jar present .

Below jar needed by hive client to talk to hbase and get data from hbase.

For example :- 

<property>

    <name>hive.aux.jars.path</name>

    <value>file:///usr/lib/hive/lib/hive-hbase-handler-0.10.0-cdh4.4.0.jar,file:///usr/lib/hbase/lib/hbase-0.94.6-cdh4.4.0.jar,file:///usr/lib/zookeeper/zookeeper-3.4.5-cdh4.4.0.jar,file:///usr/share/cmf/lib/guava-14.0.jar

   </value>

</property>

Step 4 :- (Connect tableau to hive tables)

Select tableau from start menu.

Go to data Click on connect data than click on cloudera database.

It will ask you to make connections.

Give your hive thrift  server IP and port as 10000.

Click on connect.

If its connected properly than you will get default in schema section.

Select table where you want to make computation.

Click OK.

HBASE TABLE SNAPSHOT

STEP 1 :-

Configuration

a)  Add a property.

[ nitin@nitin-ubuntu:~ # ] sudo vim /etc/hbase/conf/hbase-site.xml


  <property>
    <name>hbase.snapshot.enabled</name>
    <value>true</value>
  </property>

b) Restart hbase

 

[ nitin@nitin-ubuntu:~ # ] sudo /usr/lib/hbase/bin/stop_hbase.sh

[ nitin@nitin-ubuntu:~ # ] sudo /usr/lib/hbase/bin/start_hbase.sh

Step 2 :-

Take a Snapshot

[ nitin@nitin-ubuntu:~ # ] hbase shell

hbase> snapshot 'MY_TABLE', 'SNAP_MYTABLE'

Step 3 :-

Listing Snapshots

[ nitin@nitin-ubuntu:~ # ] hbase shell

hbase> list_snapshots

Step 4 :-

Deleting Snapshots

[ nitin@nitin-ubuntu:~ # ] hbase shell

hbase> delete_snapshot 'SNAP_MYTABLE'

Step 5 :-

Clone a table from snapshot

[ nitin@nitin-ubuntu:~ # ] hbase shell

hbase> clone_snapshot 'SNAP_MYTABLE', 'NEW_TABLE'

Step 6 :-

Export to another cluster:-

[ nitin@nitin-ubuntu:~ # ]  hbase  org.apache.hadoop.hbase.snapshot.ExportSnapshot -snapshot SNAP_Target_toys_Prods -copy-to hdfs://CLUSTER2:8020/hbase

HBASE TABLE ROW COUNT

STEP 1:-

Rowcounter  is a mapreduce job to count all the rows of a table. This is a good utility to use as a sanity check to ensure that HBase can read all the blocks of a table if there are any concerns of metadata inconsistency. It will run the mapreduce all in a single process but it will run faster if you have a MapReduce cluster in place for it to exploit.

hbase org.apache.hadoop.hbase.mapreduce.RowCounter <tablename>

[ nitin@nitin-ubuntu:~] # hbase org.apache.hadoop.hbase.mapreduce.RowCounter TARGET_TBL_NAME

QUERYING JSON RECORDS VIA HIVE

Step 1 :- 

Json file 

{

"Foo": "ABC",

"Bar": "20090101100000",

"Quux": {

"QuuxId": 1234,

"QuuxName": "Sam"

}
}








Step 2 :- 




Create Table 




CREATE TABLE json_table ( json string );







Step 3 :-




Upload data into hive table 

LOAD DATA LOCAL INPATH '/tmp/example.json'  INTO TABLE `json_table`;

Step 4 :- 

Retrieve data 

select get_json_object(json_table.json, '$') from json_table;

Step 5 :-

Retrieve Nested data

select get_json_object(json_table.json, '$.Foo') as foo,
       get_json_object(json_table.json, '$.Bar') as bar,       get_json_object(json_table.json, '$.Quux.QuuxId') as qid,       get_json_object(json_table.json, '$.Quux.QuuxName') as qname from json_table;

HBASE BACKUP AND RESTORE TABLE

STEP 1

EXPORT  :-

Export is a utility that will dump the contents of table to HDFS in a sequence file. Invoke via:

[ nitin@nitin-ubuntu:~ ]# hbase org.apache.hadoop.hbase.mapreduce.Export                   <tablename> <outputdir>

[ nitin@nitin-ubuntu:~ ]#  hbase org.apache.hadoop.hbase.mapreduce.Export HBASEEXPORTTABLE DUMP

STEP 2

IMPORT :- 

Import is a utility that will load data that has been exported back into HBase. Invoke via:

[ nitin@nitin-ubuntu:~ ]# hbase org.apache.hadoop.hbase.mapreduce.Import                   <tablename> <inputdir>

[ nitin@nitin-ubuntu:~ ]#  hbase org.apache.hadoop.hbase.mapreduce.Import HBASEIMPORTABLE DUMP