AWS, BIG DATA, JAVA

Recently I used this tool for connecting to Hive via Knox gateway.  Good thing about this tool is that it automatically downloads the driver and configures it. That  is a pain in tools like Squirrel.  It also makes a JDBC connection to Hive.  Below is how the HOST and JDBC URL settings look like : HOST : <KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive JDBC URL :  jdbc:hive2://<KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive What I noticed on my setup was that if you keep the default port which 10000 and you set it to 8443 , it does not work. So you have to keep it blank.  That may be specific to my setup. Also you might notice that ssl=true but there are  no  sslTrustStore= < Path to the “.jks” file in the NiFi node > ;trustStorePassword= < TrustStore Password>  parameters. The reason is if you import the gateway cert  into the trus...

Spring REST API App Deployment in Azure App Services  Within Azure App Services , Spring REST API application is deployed slightly different than a simple J2EE web application. A Sprint REST application can simply run inside JRE and don't need a container like tomcat. If you are planning to use azure-webapp-maven-plugin , it will need to be configured slightly differently. For a Spring REST API  application  , the plugin  configuration will look like following.     <plugin>         <groupId>com.microsoft.azure</groupId>         <artifactId>azure-webapp-maven-plugin</artifactId>         <version>1.7.0</version>         <configuration>             <!-- Specify v2 schema -->             <schemaVersion>v2</schemaVersion> ...

I learnt following lesson while trying to submit a job using livy  API which was part of azure HDInsight cluster. By default when you use following command. curl  -k --user "admin:<password> " -v  -d  '{"name":"gopal", "address":"1512 edwina ct" }'    https://<clustername>.azurehdinsight.net/livy/batches -H  "X-Requested-By: admin" It will treat the payload as  application/x-www-form-urlencoded. So you need to specify Content-Type header. curl  -k --user "admin:<password> " -v  -H "Content-Type: application/json"   -d  '{"name":"gopal", "address":"1512 edwina ct" }'    https://<clustername>.azurehdinsight.net/livy/batches  -H  "X-Requested-By: admin" This may work fine on linux but on window even this is going to be an issue. So you need put json payload into a file. Let us call the file "payl...

You can quickly launch Azure HDInsight cluster using following link. https://azure.microsoft.com/en-in/resources/templates/101-hdinsight-spark-linux/ A few things I learned while using above link. First thing is it lands you on a page where you need to enter cluster name, cluster admin name and  password , sshuser name and password.  When I entered required info and clicked "Purchase" , it failed. When I looked into the log, I found that issue was related to password. The message coming back is  not always correct. It says the password should be between  6-xx characters. In reality it needs the password to be at least 10 characters with at least one upper case , on small case and 1 number.  I had to spend 15-20 minutes to figure this out and get it working. Once you launch the cluster , you can access the Ambari UI at   <clustername>.azurehdinsight.net. You can access spark cluster via commands like <clustername>.azurehdinsight.net/l...

Few nice links: https://css-tricks.com/html5-progress-element/ https://stackoverflow.com/questions/7988962/continuous-css-rotation-animation-on-hover-animated-back-to-0deg-on-hover-out Remember even <br/> can impact the positioning of elements following it. 

1. Two very useful links if you are trying to setup Angular environment on windows 10 in a corporate network. https://www.zeolearn.com/magazine/setup-angular-windows https://yakucoding.com/npm-and-corporate-proxy-pac/ https://yakovfain.com/2016/11/06/angular-cli-speed-up-installing-dependencies-with-yarn/ https://coryrylan.com/blog/angular-ng-for-syntax 2. Very useful free courses/links for Angular: https://scrimba.com/g/gyourfirstangularapp https://nehalist.io/working-with-models-in-angular/ 3. You can change the style  dynamically using property binding. [style.background-color]="expression" 4. Be careful searching for help on Internet about Angular , as there is lots of documentation around old versions which no longer apply. 5. At times when you are making an  call in following manner and let us say getUser() is actually making an http call to a REST service. this . statusService . getUser (). subscribe (( value : Body ) => { this . resp = value...

Amazon Aurora allows creating up 15 read replicas in a region. Read replicas share the source storage. Storage scales automatically into 10GB increments up to 64 TB. There is option for Global Database which allows single database to span across multiple regions.  It uses storage based replication. By default Aurora database is replicated six ways across three AZs. It can handle loss of two copies without affecting writes and three copies without affecting reads. Aurora also has a backtrack feature, it can go back to point in time without restore from a backup.