Data Lakes

With cloud computing and big data processing technological advances enabled data lakes. Data Lakes are becoming a natural choice for organizations to harness the power of data. It creates a central repository for all sorts of data; structure, semi-structured or unstructured. Data lakes store data from all sort of data sources and in all sort of formats. No preparation is  required before storing the data. Huge quantities of data can be stored in a cost-effective manner. Data pipelines are setup to cleanse & transform the data.  Data can be consumed in multiple ways; via interactive queries; exporting  into data warehousing or business intelligence solutions. 

Functional Areas
  • Data Ingestion or Collection : batch or streaming;
  • Catalog & Search : data cataloging, metadata creation, tagging 
  • Manage & Secure Data : cost effective storage ; security ( access restrictions & encryption at rest) 
  • Processing : cleansing & transformation, , ETL or ELT  pipelines,  raw data to processed data
  • Consumption : interactive query, data warehousing systems , BI tools
  • Access &User Interface :  Administrative tasks

AWS Services: 

Data Ingestion: DMS, Kinesis, Glue, Lambda
Catalog & Search: AWS Glue, DynamoDB, Amazon Elastic Search
Manage & Secure Data: S3, IAM, KMS, CloudTrail, CloudWatch
Processing: AWS EMR ( Hive, Spark ), AWS Glue(python, java, scala) , Apache Hive 
Consumption: Apache Hive , AWS Athena, AWS Redshift, AWS Quicksight 
User Interface: AWS AppSync, API Gateway, AWS Cognito

Non-AWS

Data Ingestion:
  
    Files : Stage the files & use Hadoop CLI 
    Database: Sqoop, no CDC for smaller tables but only for larger (10M+) , use -m option for large db        dump, NiFi is another option 
    Streaming: NiFi , Flume , Streamsets. NiFi is popular.

   
File Ingestion :   
    CSV  into TEXTFILE : move the file to HDFS, create an external  TEXTFILE table on top of the HDFS location.   
   CSV file into PARQUET:  you need to first create a temporary TEXTFILE table and create a PARQUET table and then insert from TEXTFILE into PARQUET table. 
    
    







Comments

Post a Comment

Popular posts from this blog

SQL

Joins - Not always very obvious ( Following example uses Hive ). create table x_temp ( name string, p_date int  ) insert into x_temp(name,p_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table y_temp ( name string, s_date int  ) insert into y_temp(name,s_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table z_temp ( name string, p_date int, s_date int ) insert into z_temp(name,p_date,s_date) values ('z1','20181025','20181025') select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date and  y.s_date != z.s_date) generates 4 records... x.name,x.p_date,y.name,y.s_date x2, 20181026, x2, 20181026 x3, 20181027, x2, 20181026 x2, 20181026, x3, 20181027 x3, 20181027, x3, 20181027 select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date or  y.s_date != z.s_date) generates 8 records x.name,x.p_date,y.
Read more

Analytics

Types of Analytics ( Notes from AWS Course )  Descriptive - What happened ?  hindsight  Diagnostic - Why did it  happen?  hindsight & insight  Predictive  - What will happen ? insight & foresight  Prescriptive - What should I do ? foresight  Cognitive & AI  - What are recommended actions ? foresight & hypothesis input  Popular BI Tools:  https://www.cio.com/article/3284415/10-bi-tools-for-data-visualization.html
Read more

HIVE

1. When doing multi table joins, except the last ( right most ) table, other tables are buffered in the reducers. So to have less memory footprint on reducers, largest tables should be the last one as it is not buffered but streamed to reducers.   For Impala , it is different. 2.  You can filter the unwanted records within the join itself. SELECT  a.*,   b. * FROM a LEFT OUTER JOIN b ON ( a.key=b.key  AND  b.name= 'Tim'   AND a .name= 'Kim' ) 3. Joins are left associative. So execution happens from left to right and the result of joins is  fed into next join operation. 4. To find the value of variable     set hive.execution.engine 5. To find values of all variables set  6. If you want to pass different hive configuration settings in hive shell, you can use --hiveconf. For example hive --hiveconf hivi.optimize.sort.dynamic.partition=true --hiveconf hive.exec.dynamic.partition.mode=nonstrict 7.  In case you accidentally drop the p
Read more