AWS, BIG DATA, JAVA

1. Creating a partitioned table in  parquet format directly from an unpartitioned table in different format. create external table dbname.partitioned_table PARTITIONED BY ( col5,col6 ) STORED AS PARQUET LOCATION 'hdfs://nameservice/path/partitioned_table' AS select   col1 ,   col2 ,   col3,   col4   col5,   col6 from  dbname.unpartitioned_table 2.   Error while accessing HIVE TEXTFILE format table in Impala. Create a  TEXTFILE format table in HIVE.  CREATE EXTERNAL TABLE test_impala_do (   category STRING,   segment STRING,   level_1 STRING,   group_code INT,   d_xuid_count INT,   segment_code INT,   datetime STRING COMMENT 'DATETIME',   source_file STRING COMMENT 'SOURCE_FILE' ) PARTITIONED BY (   akey STRING COMMENT 'AKEY AS DEFINED' ) WITH SERDEPROPERTIES ('serialization.format'='1') STORED AS TEXTFILE LOCATION 'hdfs://nameservice1/..../test_impala_do' Populated using spark SQL job. INSERT OVERWRITE INTO

JVM argument -verbose:class  helps debug classloader.  With this option , you can see the  classes being loaded by the application. Use  -Xss<N><m|g>  argument to specify the stack size for the JVM. If the default JVM stack size is too small , you may get  java.lang.StackOverflowError from applications deep recursion. For example you can specify 128 mb of stack space by  -Xss128m  ClassLoader.findClass()  can throws  ClassNotFoundException if the jvm process does not have read permissions for the jar/class file , even if it is in classpath.