Hbase - Distributed , Scalable,Hadoop Database

HBase provides Bigtable-like capabilities on top of Hadoop and HDFS. Use Apache HBase™ when you need random, realtime read/write access to your Big Data. It is a NoSQL database ( it is a key/value store ) 

 Conceptually HBase is very much like Google's Bigtable ( multidimensional hashmap ) . See https://research.google.com/archive/bigtable.html  for more details.

 It stores the data by column families.  You can add a new column qualifier ( column family:qualifier)  to a existing column family anytime.  The column name is "column family name: qualifier" 

 Row Key,  Timestamp, Column Family1

 R1, t1, CF1:q1=""
R1,t2,CF1:q1=""

 Row Key,  Timestamp, Column Family2

 R1,t3,CF2:q2
R1,t4, CF2:q2

 Row Key,  Timestamp, Column Family3
R1,t5,CF3:q3

 Column families must be declared up front at schema definition time whereas columns do not need to be defined at schema time but can be conjured on the fly while the table is up and running.

 Physically, all column family members are stored together on the filesystem. Because tunings and storage specifications are done at the column family level, it is advised that all column family members have the same general access pattern and size characteristics.

 While rows and column keys are expressed as bytes, the version is specified using a long integer.
HBase is not suitable for transactional applications, large volume MapReduce jobs, relational analytics, etc. It is preferred when you have a variable schema with slightly different rows. It is also suitable when you are going for a key dependent access to your stored data.
HBase is also suitable when you need fault tolerant, fast and usable data management in a non-relational environment. 

 Real Use Cases
  • Use of HBase by Mozilla: They generally store all crash data in HBase
  • Use of HBase by Facebook: Facebook uses HBase storage to store real-time messages.
In Hbase  rowkey plays a role in partitioning the data.  So careful consideration should be given to the design of rowkey. The idea is also to keep the similar data rows together.  Hence rowkey should be meaningful  rather than arbitrary. 

Comments

Post a Comment

Popular posts from this blog

SQL

Joins - Not always very obvious ( Following example uses Hive ). create table x_temp ( name string, p_date int  ) insert into x_temp(name,p_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table y_temp ( name string, s_date int  ) insert into y_temp(name,s_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table z_temp ( name string, p_date int, s_date int ) insert into z_temp(name,p_date,s_date) values ('z1','20181025','20181025') select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date and  y.s_date != z.s_date) generates 4 records... x.name,x.p_date,y.name,y.s_date x2, 20181026, x2, 20181026 x3, 20181027, x2, 20181026 x2, 20181026, x3, 20181027 x3, 20181027, x3, 20181027 select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date or  y.s_date != z.s_date) generates 8 records x.name,x.p_date,y.
Read more

Analytics

Types of Analytics ( Notes from AWS Course )  Descriptive - What happened ?  hindsight  Diagnostic - Why did it  happen?  hindsight & insight  Predictive  - What will happen ? insight & foresight  Prescriptive - What should I do ? foresight  Cognitive & AI  - What are recommended actions ? foresight & hypothesis input  Popular BI Tools:  https://www.cio.com/article/3284415/10-bi-tools-for-data-visualization.html
Read more

HIVE

1. When doing multi table joins, except the last ( right most ) table, other tables are buffered in the reducers. So to have less memory footprint on reducers, largest tables should be the last one as it is not buffered but streamed to reducers.   For Impala , it is different. 2.  You can filter the unwanted records within the join itself. SELECT  a.*,   b. * FROM a LEFT OUTER JOIN b ON ( a.key=b.key  AND  b.name= 'Tim'   AND a .name= 'Kim' ) 3. Joins are left associative. So execution happens from left to right and the result of joins is  fed into next join operation. 4. To find the value of variable     set hive.execution.engine 5. To find values of all variables set  6. If you want to pass different hive configuration settings in hive shell, you can use --hiveconf. For example hive --hiveconf hivi.optimize.sort.dynamic.partition=true --hiveconf hive.exec.dynamic.partition.mode=nonstrict 7.  In case you accidentally drop the p
Read more