AWS, BIG DATA, JAVA

1. Creating a partitioned table in  parquet format directly from an unpartitioned table in different format. create external table dbname.partitioned_table PARTITIONED BY ( col5,col6 ) STORED AS PARQUET LOCATION 'hdfs://nameservice/path/partitioned_table' AS select   col1 ,   col2 ,   col3,   col4   col5,   col6 from  dbname.unpartitioned_table 2.   Error while accessing HIVE TEXTFILE format table in Impala. Create a  TEXTFILE format table in HIVE.  CREATE EXTERNAL TABLE test_impala_do (   category STRING,   segment STRING,   level_1 STRING,   group_code INT,   d_xuid_count INT,   segment_code INT,   datetime STRING COMMENT 'DATETIME',   source_file STRING COMMENT 'SOURCE_FILE' ) PARTITIONED BY (   akey STRING COMMENT 'AKEY AS...

JVM argument -verbose:class  helps debug classloader.  With this option , you can see the  classes being loaded by the application. Use  -Xss<N><m|g>  argument to specify the stack size for the JVM. If the default JVM stack size is too small , you may get  java.lang.StackOverflowError from applications deep recursion. For example you can specify 128 mb of stack space by  -Xss128m  ClassLoader.findClass()  can throws  ClassNotFoundException if the jvm process does not have read permissions for the jar/class file , even if it is in classpath. 

1. Had to frequently pull files from a server. The file set  used to be in following format. abc123_file1.txt def23145_file2.txt r4rtyyy_file3.txt The task was to remove the prefix from names and put the file set back. It was too mechanical, so I wrote following script to do the same. ECHO OFF SETLOCAL EnableDelayedExpansion FOR %%I IN (*) DO (    REM ECHO %%I    SET  xname=%%I      for /f "tokens=1,2 delims=_" %%a in ("%%I") do (      REM ECHO %%a      REM ECHO %%b        REM ECHO !xname!      SET xname=!xname:%%a_=!      REM ECHO !xname!      SET ENDSWITH=!xname:~-4!      REM ECHO !ENDSWITH!      IF "!ENDSWITH!"==".csv"  REN %%I  !xname!      IF "!ENDSWITH!"=="done"  REN %%I  !xname!    ) ) 2.  I faced  a situation where I needed t...

* Good article on MapReduce http://www.bigsynapse.com/mapreduce-internals *  org.apache.hadoop.mapreduce.Mapper<KEYIN,VALUEIN,KEYOUT,VALUEOUT>  is generic class. There is an Mapper generic  interface too but it is in different package   org.apache.hadoop.mapre.Mapper<K1,V1,K2,V2> and has nothing to do with the Mapper class. The Reducer is also a generic class similar to Mapper. * Hadoop also has two special annotations to indicate  the audience and stability of any interface or class        -InterfaceStability        -InterfaceAudience * All the key value types in Hadoop Map Reduce programming must implement Writable interface, which results in type implementing efficient serialization using DataInput/DataOutput. * hadoop fs -put   input/sample.txt   hdfs://quickstart.cloudera:8020/sample.txt  returned "ConnectException"  because namenode service was not running.  Used f...

"Verifying  jdk......" I just got a new macbook pro OS X EL Capitan and I was not able to run applications needing java. I tried installing java but it always got stuck at "verifying jdk.......".  Finally I saw a post on google where it mentioned that issues was due to SIP ( System Integrity Protection) being enabled. I found an article how to disable it. http://osxdaily.com/2015/10/05/disable-rootless-system-integrity-protection-mac-os-x/ Once I disable , I was able to install jdk.

1. To get sum for all the values from column B1 to B6 where value in column C1 to C6 is "S1"  all  from Sheet1. =SUMIF(Sheet1!C1:C6,"S2",Sheet1!B1:B6) See "S2" in double quotes because without that, it will represent the value in cell S2. 2. In a formula, a single string can only 255 char long. So if you need to have longer string, you will have represent it by concatenation of multiple smaller strings like  for example   "abcdef" => "abc" & "def" 3.  If you are editing a long formula and getting errors, one thing you might want to look for is new lines in it esp. if you copied the text and pasted it from some place. 4. LEFT(txt , num_chars) => Returns  left num_chars characters from   txt. 5. How to escape double quote in string  as part of formula ? by double quote itself .   If you want to have "abc" as text where double quotes are included in it ...you have to put it  like ""...

1. How to enable secure cookies in an application.  Include below config in web.xml at the top level.   <session-config>     <cookie-config>       <http-only>true</http-only>       <secure>true</secure>     </cookie-config>   </session-config> Once you do that , you will have request server to set secure cookies or http only cookies and it will set those . Without this config , even if you request server to set secure cookies or http only cookies , it wont do it.  This is true at least for Tomcat. 2. Some versions of  Tomcat remove trailing =  in cookies.  To stop Tomcat from removing the trailing "="  signs, start Tomcat with  "-Dorg.apache.tomcat.util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE=true"  option.eds