AWS S3

Use multipart upload to upload objects with size 5MB-5TB  to S3 bucket. Unless you abort or complete the multipart upload, you will be charged for the parts storage and any requests relating to those parts. You can also configure bucket  lifecycle policy to abort the multipart upload.

S3 is a global namespace and S3  bucket URL has to be globally unique.

if S3 uptime is below the SLA during any period(monthly) , AWS provides service credits.

Static Web Site Hosting 

 The bucket name has to be same as "Name" value in the "A" Record Set element ( inside the Route 53 Hosted Zone )  whose Alias Target is the s3 endpoint.

One thing to note is that Alias Target points to  region specific s3 endpoint and not bucket specific (
s3-website-us-east-1.amazonaws.com. )  but the specific bucket must exist and should have public access at least for viewing (reading) . It should say "Public"  under the Access column for that bucket. If that is not the case , you will not see it appearing in the "Alias Target" drop-down.

Public read permissions are also required  for the content files which you are planning to make available as part of static web site.

If you configure a bucket for static web site hosting and just opt to redirect the traffic to another bucket. It will be a browser redirect.

S3 URL 

virtual-hosted-style
  • http://bucket.s3.aws-region.amazonaws.com
  • http://bucket.s3.amazonaws.com : This won't work for regions created after March 2019.
path-style 
  • http://s3.amazonaws.com/bucket :  for us-east-1 region.
  • http://s3.aws-region.amazonaws.com/bucket : for rest of the regions







Comments

Post a Comment

Popular posts from this blog

SQL

Joins - Not always very obvious ( Following example uses Hive ). create table x_temp ( name string, p_date int  ) insert into x_temp(name,p_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table y_temp ( name string, s_date int  ) insert into y_temp(name,s_date) values ('x1',20181025),('x2', 20181026), ('x3',20181027) create table z_temp ( name string, p_date int, s_date int ) insert into z_temp(name,p_date,s_date) values ('z1','20181025','20181025') select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date and  y.s_date != z.s_date) generates 4 records... x.name,x.p_date,y.name,y.s_date x2, 20181026, x2, 20181026 x3, 20181027, x2, 20181026 x2, 20181026, x3, 20181027 x3, 20181027, x3, 20181027 select x.*, y.* from x_temp x, y_temp y, z_temp z  where x.p_date != z.p_date or  y.s_date != z.s_date) generates 8 records x.name,x.p_date,y....
Read more

Analytics

Types of Analytics ( Notes from AWS Course )  Descriptive - What happened ?  hindsight  Diagnostic - Why did it  happen?  hindsight & insight  Predictive  - What will happen ? insight & foresight  Prescriptive - What should I do ? foresight  Cognitive & AI  - What are recommended actions ? foresight & hypothesis input  Popular BI Tools:  https://www.cio.com/article/3284415/10-bi-tools-for-data-visualization.html
Read more

DBeaver

Recently I used this tool for connecting to Hive via Knox gateway.  Good thing about this tool is that it automatically downloads the driver and configures it. That  is a pain in tools like Squirrel.  It also makes a JDBC connection to Hive.  Below is how the HOST and JDBC URL settings look like : HOST : <KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive JDBC URL :  jdbc:hive2://<KNOX_HOST>:8443/;ssl=true;transportMode=http;httpPath=gateway/<topology>/hive What I noticed on my setup was that if you keep the default port which 10000 and you set it to 8443 , it does not work. So you have to keep it blank.  That may be specific to my setup. Also you might notice that ssl=true but there are  no  sslTrustStore= < Path to the “.jks” file in the NiFi node > ;trustStorePassword= < TrustStore Password>  parameters. The reason is if you import the gateway cert  into the trus...
Read more