Azure HDInsight

You can quickly launch Azure HDInsight cluster using following link.

https://azure.microsoft.com/en-in/resources/templates/101-hdinsight-spark-linux/

A few things I learned while using above link. First thing is it lands you on a page where you need to enter cluster name, cluster admin name and  password , sshuser name and password.  When I entered required info and clicked "Purchase" , it failed. When I looked into the log, I found that issue was related to password. The message coming back is  not always correct. It says the password should be between  6-xx characters. In reality it needs the password to be at least 10 characters with at least one upper case , on small case and 1 number.  I had to spend 15-20 minutes to figure this out and get it working.

Once you launch the cluster , you can access the Ambari UI at   <clustername>.azurehdinsight.net. You can access spark cluster via commands like <clustername>.azurehdinsight.net/livy/sessions or <clustername>.azurehdinsight.net/livy/batches.  For more information see following link.

https://docs.microsoft.com/en-us/azure/hdinsight/spark/apache-spark-livy-rest-interface


When you launch cluster , it also provides you ssh access at   <sshuser>@<clustername>-ssh.azurehdinsight.net  . You can login there and upload files, spark code etc.  etc.  You can use hadoop commands here. For example you can upload "demo.jar" and then put that into hdfs location as following.

hadoop fs  -put -f demo.jar /example/data/codebase/jar

Now you can  access this file using following URL in livy comamnds.

"wasb://<clustername>@<storageaccountname>.blob.core.windows.net/example/data/codebase/jar/demo.jar"






Comments

Post a Comment

Popular posts from this blog

SQL

Joins - Not always very obvious ( Following example uses Hive ). create table x_temp ( name string, p_date int  ) insert into x_temp(name,p_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table y_temp ( name string, s_date int  ) insert into y_temp(name,s_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table z_temp ( name string, p_date int, s_date int ) insert into z_temp(name,p_date,s_date) values ('z1','20181025','20181025') select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date and  y.s_date != z.s_date) generates 4 records... x.name,x.p_date,y.name,y.s_date x2, 20181026, x2, 20181026 x3, 20181027, x2, 20181026 x2, 20181026, x3, 20181027 x3, 20181027, x3, 20181027 select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date or  y.s_date != z.s_date) generates 8 records x.name,x.p_date,y....
Read more

Analytics

Types of Analytics ( Notes from AWS Course )  Descriptive - What happened ?  hindsight  Diagnostic - Why did it  happen?  hindsight & insight  Predictive  - What will happen ? insight & foresight  Prescriptive - What should I do ? foresight  Cognitive & AI  - What are recommended actions ? foresight & hypothesis input  Popular BI Tools:  https://www.cio.com/article/3284415/10-bi-tools-for-data-visualization.html
Read more

DBeaver

Recently I used this tool for connecting to Hive via Knox gateway.  Good thing about this tool is that it automatically downloads the driver and configures it. That  is a pain in tools like Squirrel.  It also makes a JDBC connection to Hive.  Below is how the HOST and JDBC URL settings look like : HOST : <KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive JDBC URL :  jdbc:hive2://<KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive What I noticed on my setup was that if you keep the default port which 10000 and you set it to 8443 , it does not work. So you have to keep it blank.  That may be specific to my setup. Also you might notice that ssl=true but there are  no  sslTrustStore= < Path to the “.jks” file in the NiFi node > ;trustStorePassword= < TrustStore Password>  parameters. The reason is if you import the gateway cert  into the trus...
Read more