Amazon EMR

While trying to create EMR cluster I got following error.

 "Could not create cluster.The instance profile for the newly created default role is not yet visible. Please try after a few seconds." 

Tried again and again but no luck.  Ultimately I deleted all the default EMR roles which were there and I recreated the roles as following.

aws emr create-default-roles --profile admin --region us-east-1

After that I was able to create the cluster. 


https://aws.amazon.com/blogs/big-data/optimize-amazon-emr-costs-with-idle-checks-and-automatic-resource-termination-using-advanced-amazon-cloudwatch-metrics-and-aws-lambda/

Master node does not need high compute power.Core task nodes need both compute and storage and task  nodes don't need storage.  

You can not change instance type for master and core nodes while cluster is running but you can do so for task nodes.

By default  EMR schedules jobs in such a way that even if any task node gets terminated , jobs can still continue.  It schedles the master processes on core nodes ( nodes with label 'CORE' ) . 

you should only run core nodes on Spot Instances when partial HDFS data loss is tolerable.

When you launch EMR cluster , you have two options for picking instance : 1) Uniform Instance Group  or 2)  Instance Fleet.    Uniform Instance group will let you choose single type of instance for master , core and task nodes.  Though master , core and task instance types can be different from each other.   With instance fleet option you get more options . You can have mix instance types and mix purchase options ( spot vs. on-demand for core and task nodes ).  You can specify max spot price and target capacities ( spot vs on-demand )  in terms of units. For spot instances you can specify provisioning time-out and what do if spot instance capacity is not met in that time. See various options below when cluster composition is set to  instance fleet.


You can also specify the same via a json config file while using AWS CLI for creating cluster. 

Comments

Post a Comment

Popular posts from this blog

SQL

Joins - Not always very obvious ( Following example uses Hive ). create table x_temp ( name string, p_date int  ) insert into x_temp(name,p_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table y_temp ( name string, s_date int  ) insert into y_temp(name,s_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table z_temp ( name string, p_date int, s_date int ) insert into z_temp(name,p_date,s_date) values ('z1','20181025','20181025') select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date and  y.s_date != z.s_date) generates 4 records... x.name,x.p_date,y.name,y.s_date x2, 20181026, x2, 20181026 x3, 20181027, x2, 20181026 x2, 20181026, x3, 20181027 x3, 20181027, x3, 20181027 select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date or  y.s_date != z.s_date) generates 8 records x.name,x.p_date,y....
Read more

Analytics

Types of Analytics ( Notes from AWS Course )  Descriptive - What happened ?  hindsight  Diagnostic - Why did it  happen?  hindsight & insight  Predictive  - What will happen ? insight & foresight  Prescriptive - What should I do ? foresight  Cognitive & AI  - What are recommended actions ? foresight & hypothesis input  Popular BI Tools:  https://www.cio.com/article/3284415/10-bi-tools-for-data-visualization.html
Read more

DBeaver

Recently I used this tool for connecting to Hive via Knox gateway.  Good thing about this tool is that it automatically downloads the driver and configures it. That  is a pain in tools like Squirrel.  It also makes a JDBC connection to Hive.  Below is how the HOST and JDBC URL settings look like : HOST : <KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive JDBC URL :  jdbc:hive2://<KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive What I noticed on my setup was that if you keep the default port which 10000 and you set it to 8443 , it does not work. So you have to keep it blank.  That may be specific to my setup. Also you might notice that ssl=true but there are  no  sslTrustStore= < Path to the “.jks” file in the NiFi node > ;trustStorePassword= < TrustStore Password>  parameters. The reason is if you import the gateway cert  into the trus...
Read more