Amazon Web Services (AWS)

Índex

General

Casos d'ús / Use cases
  • Use cases
  • Digital Media
    • Digital Media in the Cloud: Best Practices for Processing Media on AWS (YouTube)
      1. Media processing in AWS (AWS)
        • Cloud transcoding architecture
          • Phase 1:
            • Add transcoder instances to EC2
            • Use S3 to store file-based sources
            • Use S3 to store file-based outputs
            • Use CloudFront to distribute output streams
          • Phase 2:
            • Use acceleration and/or Direct Connect for ingest
            • Use Amazon Virtual Private Cloud to ringfence
            • Use EC2 Reserved Instances
            • Use EC2 Spot Instances
          • Phase 3:
            • Create a fleet of transcode workers
            • Use your on-premise workflow controller to orchestrate using SWF
            • Use SQS to create a cloud transcode queue
            • Use SNS for notifications
        • Securing content (14:00)
          • Local encryption: encrypt and mantain your own keys
          • Network encryption: use secured network transfer (SSL, VPC)
          • At REST encryption: S3 encrypts at REST using AES-256
          • DRM: integrate certificated-based DRM through third parties
          • Watermarking: Integrate digital watermarking through third parties
      2. Best practices for hybrid transcoding workflows (Elemental Technologies)
      3. Cloud-based content management (Ericsson)
      4. High performance media processing (Intel)

Cloudformation
  • AWS CloudFormation Product Details
  • User guide
  • 19 Best Practices for Creating Amazon CloudFormation Templates
  • Designer
  • Preserve resources after stack destruction
  • Info
  • CLI Cloudformation
  • EC2
    • Instance
    • Volume
    • VPC
    • single EC2 instance
      • single_ec2.json
        • {
              "Description": "Single EC2 instance",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Resources": {
                  "singleEC2": {
                      "Type": "AWS::EC2::Instance",
                      "Properties": {
                          "ImageId":"ami-xxxxxxxx",
                          "KeyName":"my_key_pair",
                          "InstanceType":"t2.micro"
                      }
                  }
              }
          }
    • single ec2 instance with an extra volume
      • single_ec2_volume.json
        • "Resources": {
          ...
              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":{"Ref" : "ImageId"},
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"my_server_key",
                  "InstanceType":{"Ref" : "InstanceType"},
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe\n",
                          "sudo mkfs -t xfs /dev/xvdh           \n",
                          "sudo mkdir /mnt/vol           \n",
                          "sudo chmod 777 /mnt/vol           \n",
                          "sudo mount /dev/xvdh /mnt/vol           \n",
                      ] ]
                      }
                  },
                  "Tags":[{"Key":"Name","Value":{"Ref":"BaseName"}}]
                  }
              },

          ...

              "NewVolume" : {
                  "Type" : "AWS::EC2::Volume",
                  "Properties" : {
                      "Size" : "100",
                      "AvailabilityZone" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ]}
                  }
              },
             
              "MountPoint" : {
                  "Type" : "AWS::EC2::VolumeAttachment",
                  "Properties" : {
                      "InstanceId" : { "Ref" : "EC2Instance" },
                      "VolumeId"  : { "Ref" : "NewVolume" },
                      "Device" : "/dev/xvdh"
                  }
              },
    • single EC2 with UserData and extra volume
      • Notes:
        • When using direct bash commands:
          • add "\n" at the end of each command
          • no need to call sudo
      • single_ec2_userdata_volume.json
        • {
              "Description": "Single EC2 instance with extra volume",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Parameters" : {
              "InstanceType" : {
                  "Description" : "EC2 instance type",
                  "Type" : "String",
                  "Default" : "t2.micro",
                  "AllowedValues" : [ "t1.micro", "t2.micro", "t2.small", "t2.medium", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"]
                  ,
                  "ConstraintDescription" : "must be a valid EC2 instance type."
              },
             
              "HostedZone" : {
                  "Type" : "String",
                  "Description" : "The DNS name of an existing Amazon Route 53 hosted zone",
                  "AllowedPattern" : "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
                  "ConstraintDescription" : "must be a valid DNS zone name.",
                  "Default" : "example.net"
              },
              "ImageId" : {
                  "Type" : "String",
                  "Description" : "The image_id for the ec2 instance"
              },
              "NewVolumeSize" : {
                  "Type" : "String",
                  "Description" : "The size of the new volume (GB)",
                  "Default": "5"
              }
              },   
              "Resources": {

              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":{"Ref" : "ImageId"},
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"wct_streaming_server",
                  "InstanceType":{"Ref" : "InstanceType"},
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe \n",
                          "while [ ! -e /dev/xvdh ]; do echo waiting for /dev/xvdh to attach; sleep 10; done \n",
                          "mkfs -t xfs /dev/xvdh \n",
                          "mkdir -p /mnt/vol1 \n",
                          "mount /dev/xvdh /mnt/vol1 \n",
                          "chmod 777 /mnt/vol1 \n"
                      ] ]
                      }
                  },
                  "Tags":[{"Key":"Name","Value":{"Ref":"BaseName"}}]
                  }
              },
            
              "NewVolume" : {
                  "Type" : "AWS::EC2::Volume",
                  "Properties" : {
                  "Size" : {"Ref" : "NewVolumeSize"},
                  "AvailabilityZone" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ]}
                  }
              },
             
              "MountPoint" : {
                  "Type" : "AWS::EC2::VolumeAttachment",
                  "Properties" : {
                  "InstanceId" : { "Ref" : "EC2Instance" },
                  "VolumeId"  : { "Ref" : "NewVolume" },
                  "Device" : "/dev/xvdh"
                  }
              },

          }
    • single EC2 entry with Route53
      • single_ec2_r53.json
        • {
              "Description": "Single EC2 instance",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Parameters" : {
              "InstanceType" : {
                  "Description" : "EC2 instance type",
                  "Type" : "String",
                  "Default" : "m1.small",
                  "AllowedValues" : [ "t1.micro", "t2.micro", "t2.small", "t2.medium", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"]
                  ,
                  "ConstraintDescription" : "must be a valid EC2 instance type."
              },
             
              "HostedZone" : {
                  "Type" : "String",
                  "Description" : "The DNS name of an existing Amazon Route 53 hosted zone",
                  "AllowedPattern" : "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
                  "ConstraintDescription" : "must be a valid DNS zone name."
              }
              },   
              "Resources": {
              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":"ami-437da730",
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"my_key_pair",
                  "InstanceType":"t2.micro"
                  }
              },
              "InstanceSecurityGroup" : {
                  "Type" : "AWS::EC2::SecurityGroup",
                  "Properties" : {
                  "GroupDescription" : "Enable SSH, HTTP, RTMP",
                  "SecurityGroupIngress" : [
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "22",
                      "ToPort" : "22",
                      "CidrIp" : "0.0.0.0/0"
                      },
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "80",
                      "ToPort" : "80",
                      "CidrIp" : "0.0.0.0/0"
                      },
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "1935",
                      "ToPort" : "1935",
                      "CidrIp" : "0.0.0.0/0"
                      }
                  ]
                  }
              },
              "MyDNSRecord" : {
                  "Type" : "AWS::Route53::RecordSet",
                  "Properties" : {
                  "HostedZoneName" : { "Fn::Join" : [ "", [{"Ref" : "HostedZone"}, "." ]]},
                  "Comment" : "DNS name for my instance.",
                  "Name" : { "Fn::Join" : [ "", [{"Ref" : "EC2Instance"}, ".", {"Ref" : "AWS::Region"}, ".", {"Ref" : "HostedZone"} ,"."]]},
                  "Type" : "A",
                  "TTL" : "900",
                  "ResourceRecords" : [ { "Fn::GetAtt" : [ "EC2Instance", "PublicIp" ] } ]
                  }
              }   
              },
              "Outputs" : {
              "InstanceId" : {
                  "Description" : "InstanceId of the newly created EC2 instance",
                  "Value" : { "Ref" : "EC2Instance" }
              },
              "AZ" : {
                  "Description" : "Availability Zone of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ] }
              },
              "PublicDNS" : {
                  "Description" : "Public DNSName of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "PublicDnsName" ] }
              },
              "PublicIP" : {
                  "Description" : "Public IP address of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "PublicIp" ] }
              },
              "DomainName" : {
                  "Description" : "Fully qualified domain name",
                  "Value" : { "Ref" : "MyDNSRecord" }
              }
              }
             
          }
  • Route53
  • Alarm
  • CloudFront
    • AWS::CloudFront::Distribution
      • "PriceClass" :
        • "PriceClass_100" -> Use Only U.S., Canada and Europe
        • "PriceClass_200" -> Use U.S., Canada, Europe, Asia and Africa
        • "PriceClass_All" -> Use All Edge Locations (Best Performance)
    • No cache for 404
      • "MyCloudFront" : {
            "Type" : "AWS::CloudFront::Distribution",
            "Properties" : {
                "DistributionConfig" : {
                    "CustomErrorResponses" : [ {
                        "ErrorCode" : "404",
                        "ErrorCachingMinTTL" : "2"
                    } ]
                    ...
                 }
                }
        }
    • Cache behaviour /  Forward headers: Whitelist
      • Configuring CloudFront to Cache Objects Based on Request Headers
      • To avoid problems with CORS and 403 response from CloudFront
      •                     "DefaultCacheBehavior" : {
                                "TargetOriginId" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                "ForwardedValues" : {
                                    "Headers" : ["Origin","Access-Control-Request-Headers","        Access-Control-Request-Method"],
                                    "QueryString" : "false",
                                    "Cookies" : { "Forward" : "none" }
                                },
                                "AllowedMethods" : ["GET", "HEAD", "OPTIONS"],
                                "CachedMethods" : ["GET", "HEAD", "OPTIONS"],
                                "ViewerProtocolPolicy" : "allow-all"
                            },
    • Full examples:
      • Origin is S3, with whitelist for forwarded headers ("Origin"):
        •     "Resources": {
                  "MyCloudFront" : {
                      "Type" : "AWS::CloudFront::Distribution",
                      "Properties" : {
                          "DistributionConfig" : {
                              "Origins" : [ {
                                  "DomainName": { "Fn::Join" : [ "", [{"Ref":"BucketName"}, ".s3.amazonaws.com"]]},
                                  "OriginPath": "my_dir",
                                  "Id" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                  "S3OriginConfig": {}
                              }],
                              "Enabled" : "true",
                              "Comment" : "My comments",
                              "DefaultCacheBehavior" : {
                                  "TargetOriginId" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                  "ForwardedValues" : {
                                      "Headers" : ["Origin"],
                                      "QueryString" : "false",
                                      "Cookies" : { "Forward" : "none" }
                                  },
                                  "ViewerProtocolPolicy" : "allow-all"
                              },
                              "PriceClass" : "PriceClass_100"
                          }      
                      }
                  }
              },
      • Origin is own http server, with no cache for 404 responses:
        •     "Resources": {
                  "MyCloudFront" : {
                      "Type" : "AWS::CloudFront::Distribution",
                      "Properties" : {
                          "DistributionConfig" : {
                              "Origins" : [ {
                                  "DomainName": "myserver.toto.org",
                                  "OriginPath": "/root_dir",
                                  "Id" : "oid-root_dir",
                                  "CustomOriginConfig": {
                                      "HTTPPort": "80",
                                      "HTTPSPort": "443",
                                      "OriginProtocolPolicy": "http-only"
                                  }
                              }],
                              "Enabled" : "true",
                              "Comment" : "My comments",
                              "DefaultCacheBehavior" : {
                                  "TargetOriginId" :           "oid-root_dir",
                                  "ForwardedValues" : {
                                      "QueryString" : "false",
                                      "Cookies" : { "Forward" : "none" }
                                  },
                                  "ViewerProtocolPolicy" : "allow-all"
                              },
                              "CustomErrorResponses" : [ {
                                  "ErrorCode" : "404",
                                  "ErrorCachingMinTTL" : "2"
                              } ],
                              "PriceClass" : "PriceClass_100"
                          }      
                      }
                  }
              },
    • Amazon CloudFront Template Snippets
    • Amazon CloudFront - Introduction
  • LoadBalancer

    		•  Application Load Balancer Classic Load Balancer
      protocols
      		 HTTP, HTTPS, HTTP/2, WebSockets
      		 HTTP, HTTPS, TCP, SSL

      		AWS::ElasticLoadBalancingV2::LoadBalancer
      		 AWS::ElasticLoadBalancing::LoadBalancer
      Fn::GetAtt
      		 CanonicalHostedZoneID
      		 CanonicalHostedZoneNameID
      DNSName
      		 CanonicalHostedZoneName
      LoadBalancerName
      		 -

      		AWS::ElasticLoadBalancingV2::Listener
      AWS::ElasticLoadBalancingV2::TargetGroup (TargetGroups)
      		 Listeners

    • Classic Load Balancer
      • listener with redirect ports http/80, http/8088, https/8089, tcp/1935 and LBCookieStickinessPolicy:
        • "MyLoadBalancer": {
              "Type": "AWS::ElasticLoadBalancing::LoadBalancer          ",
              "Properties": {
                  "LoadBalancerName": "MyLoadbalancerName",
                  "SecurityGroups" : [ ... ],
                  "AvailabilityZones": {
                      "Fn::GetAZs": ""
                  },
                  "CrossZone": "true",
                  "ConnectionSettings": {
                      "IdleTimeout" : 60
                  }
                  "Listeners": [
                      {
                          "LoadBalancerPort": "80",
                          "InstancePort": "80",
                          "Protocol": "HTTP",
                          "PolicyNames": ["MyFirstLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "8088",
                          "InstancePort": "8088",
                          "Protocol": "HTTP",
                          "PolicyNames": ["MySecondLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "8089",
                          "Protocol": "HTTPS",
                          "InstancePort": "8089",
                          "InstanceProtocol": "HTTPS",
                          "SSLCertificateId": "arn:aws:acm:eu-west-1:...",
                          "PolicyNames": ["MySecondLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "1935",
                          "InstancePort": "1935",
                          "Protocol": "TCP"
                      }
                  ],
                  "LBCookieStickinessPolicy" : [
                      {
                          "CookieExpirationPeriod" : "500",
                          "PolicyName" : "MyFirstLBCookieStickinessPolicy"
                      },
                      {
                          "CookieExpirationPeriod" : "1000",
                          "PolicyName" : "MySecondLBCookieStickinessPolicy"
                      }
                  ],
                  "HealthCheck": {
                      "Target": "HTTP:80/",
                      "HealthyThreshold": "3",
                      "UnhealthyThreshold": "5",
                      "Interval": "30",
                      "Timeout": "5"
                  }
              }
          }
      • listener with a certificate from ACM
        • ...
    • SecurityGroup
    • UDP
  • AutoScalingGroup
    • Prerequisites:
      • cfn-signal
        • Download from: Bootstrapping Applications using AWS CloudFormation
        • kixorz/ubuntu-cloudformation.json
        • Install
          • mkdir aws-cfn-bootstrap-latest
          • curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1
          • easy_install aws-cfn-bootstrap-latest
        • Problemes / Problems
          • Traceback (most recent call last):
              File "/bin/easy_install", line 9, in <module>
                load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
              [...]
              File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 701, in process_distribution
                distreq.project_name, distreq.specs, requirement.extras
            TypeError: __init__() takes exactly 2 arguments (4 given)
            • Diagnose
              • python
                • >>> from pkg_resources import load_entry_point
                  >>> load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
                  ...
                  pkg_resources.VersionConflict: (setuptools 25.1.4 (/usr/lib/python2.7/site-packages/setuptools-25.1.4-py2.7.egg), Requirement.parse('setuptools==0.9.8'))
            • Solució / Solution
              • sudo rm -rf /usr/lib/python2.7/site-packages/setuptools-25.1.4-py2.7.egg
      • You must install a UserData on your instance (AMI) that generates a cfn-signal received by CreationPolicy
        •     "LaunchConfig" : {
                  "Type" : "AWS::AutoScaling::LaunchConfiguration",
                  "Properties" : {
                  "ImageId":{"Ref" : "MyImageId"},
                  "SecurityGroups" : [ { "Ref" : "MySecurityGroup" } ],
                  "KeyName":"my_key",
                  "InstanceType":{"Ref" : "MyInstanceType"},
                  "IamInstanceProfile": "role_my_server",
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe\n",
                          "/usr/bin/cfn-signal -e 0 --stack ", { "Ref": "AWS::StackName" },
                          " --resource MyAutoscalingGroup ",
                          " --region ", { "Ref" : "AWS::Region" }, "\n"
                      ] ]
                      }
                  }
                  }
              },

    • Auto Scaling Template Snippets
    • AutoScalingGroup
      • AWS::AutoScaling::AutoScalingGroup
      • example.json
        •     "MyAutoScalingGroup" : {
                  "Type" : "AWS::AutoScaling::AutoScalingGroup",
                  "Properties" : {
                  "LaunchConfigurationName" : { "Ref" : "MyLaunchConfig" },
                  "MinSize" : "1",
                  "MaxSize" : "3",
                  "LoadBalancerNames" : [ { "Ref" : "MyLoadBalancer" } ],
                  "Tags":[
                      {
                      "Key":"Name",
                      "Value":{ "Fn::Join" : [ "", ["myinstance-", {"Ref" : "MyName"}]]},
                      "PropagateAtLaunch" : "true"
                      }
                  ]
                  },
                  "CreationPolicy" : {
                  "ResourceSignal" : {
                      "Timeout" : "PT15M",
                      "Count"   : "1"
                  }
                  },
                  "UpdatePolicy": {
                  "AutoScalingRollingUpdate": {
                      "MinInstancesInService": "1",
                      "MaxBatchSize": "1",
                      "PauseTime" : "PT15M",
                      "WaitOnResourceSignals": "true"
                  }
                  }
              },

    • ScalingPolicy
    • Alarm
      • AWS::...
      • example.json
        •     "MyUpScalingPolicy" : {
                  "Type" : "AWS::AutoScaling::ScalingPolicy",
                  "Properties" : {
                  "AdjustmentType" : "ChangeInCapacity",
                  "AutoScalingGroupName" : { "Ref" : "MyAutoScalingGroup" },
                  "Cooldown" : "60",
                  "ScalingAdjustment" : "1"
                  }
              },

              "MyCPUHighAlarm": {
                  "Type": "AWS::CloudWatch::Alarm",
                  "Properties": {
                  "EvaluationPeriods": "1",
                  "Statistic": "Average",
                  "Threshold": "80",
                  "AlarmDescription": "Alarm if CPU too high or metric disappears indicating instance is down",
                  "Period": "60",
                  "AlarmActions": [ { "Ref": "MyUpScalingPolicy" } ],
                  "Namespace": "AWS/EC2",
                  "Dimensions": [ {
                      "Name": "AutoScalingGroupName",
                      "Value": { "Ref": "MyAutoScalingGroup" }
                  } ],
                  "ComparisonOperator": "GreaterThanThreshold",
                  "MetricName": "CPUUtilization"
                  }
              },
    • LaunchConfiguration
    • Example:
      • my_scaling.json
        • ...
  • ...

IAM

ACM (AWS Certificate Manager)

EC2

S3
  • bucketname.s3.amazonaws.com
  • Bucket policy
    • AWS Policy Generator
    • Granting Read-Only Permission to an Anonymous User
    • Public Readable Amazon S3 Bucket Policy
      • {
          "Version": "2008-10-17",
          "Statement": [
            {
              "Sid": "AllowPublicRead",
              "Effect": "Allow",
              "Principal": {
                "AWS": "*"
              },
              "Action": [
                "s3:GetObject"
              ],
              "Resource": [
                "arn:aws:s3:::bucket_name/*"
              ]
            }
          ]
        }
      • Boto (Python):
        • # modify policy to make it publicly available
          policy_json = '{"Version":"2008-10-17","Statement":[{"Sid":"AllowPublicRead","Effect":"Allow","Principal":{"AWS":"*"},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::%s/*"]}]}' % (bucket_name)
          print policy_json
          bucket.set_policy(policy_json)
  • Cache
  • CORS
  • Static website
    • Virtual Hosting of Buckets
    • Hosting a Static Website on Amazon S3
    • Example: Setting Up a Static Website Using a Custom Domain
    • AngularJS
    • HTTPS
    • s3 bucket
      		 cloudfront
      		 route53
      		 http
      		 https
      name
      		 Static website hosting



      www.toto.org



      		 -
      		 -
      		 -
      		 http://s3-eu-west-1.amazonaws.com/www.toto.org/index.html
      		 https://s3-eu-west-1.amazonaws.com/www.toto.org/index.html
      -
      		 -
      		 -
      		 http://www.toto.org.s3.amazonaws.com/ (Access denied)
      http://www.toto.org.s3.amazonaws.com/index.html
      http://www.toto.org.s3.eu-west-1.amazonaws.com/ (Access denied)
      http://www.toto.org.s3.eu-west-1.amazonaws.com/index.html
      		 https://www.toto.org.s3.amazonaws.com/
      x
      		 -
      		 -
      		 http://www.toto.org.s3-website-eu-west-1.amazonaws.com/ https://www.toto.org.s3-website-eu-west-1.amazonaws.com/ (timeout: because of dots?)
      x
      		 -
      		 www.toto.org A ALIAS s3-website-eu-west-1.amazonaws.com.
      		 http://www.toto.org/ https://www.toto.org/
      x
      • Origin Domain Name: www.toto.org.s3.amazonaws.com
      • Default root object: index.html
      • Alternate Domain Names (CNAMEs): www.toto.org
      • Custom SSL Certificate: (choose one of the certificates previously uploaded to path /cloudfront/)
      		 www.toto.org A ALIAS www.toto.org (xxxxxx.cloudfront.net) http://www.toto.org/ https://www.toto.org/
      www-toto-org
      		 -
      		 -
      		 -
      		 http://s3-eu-west-1.amazonaws.com/www-toto-org/ (Access denied)
      http://s3-eu-west-1.amazonaws.com/www-toto-org/index.html
      		 https://s3-eu-west-1.amazonaws.com/www-toto-org/
      https://s3-eu-west-1.amazonaws.com/www-toto-org/index.html
      -
      		 -
      		 -
      		 http://www-toto-org.s3.amazonaws.com/ (Access denied)
      http://www-toto-org.s3.amazonaws.com/index.html
      http://www-toto-org.s3-eu-west-1.amazonaws.com/ (Access denied)
      http://www-toto-org.s3.eu-west-1.amazonaws.com/index.html
      		 https://www-toto-org.s3.amazonaws.com/ (Access denied)
      https://www-toto-org.s3.amazonaws.com/index.html
      https://www-toto-org.s3-eu-west-1.amazonaws.com/ (Access denied)
      https://www-toto-org.s3.eu-west-1amazonaws.com/index.html
      x
      		 -
      		 -
      		 http://www-toto-org.s3-website-eu-west-1.amazonaws.com/ https://www-toto-org.s3-website-eu-west-1.amazonaws.com/ (timeout)
      x
      • Origin Domain Name: www-toto-org.s3.amazonaws.com
      • Default root object: index.html
      • Alternate Domain Names (CNAMEs): www.toto.org
      • Custom SSL Certificate: (choose one of the certificates previously uploaded to path /cloudfront/)
      		 www.toto.org A ALIAS www.toto.org (xxxxxx.cloudfront.net) http://www.toto.org/ https://www.toto.org/

  • s3tools
    • s3cmd
      • Instal·lació / Installation
      • make bucket
        • s3cmd mb s3://...
      • list
        • s3cmd ls s3://bucket_name
      • upload
        • s3cmd put ...
  • yas3fs
    • Instal·lació / Installation
      • CentOS
        • sudo yum -y install fuse fuse-libs
          sudo easy_install pip
          sudo pip install yas3fs
          sudo sed -i'' 's/^# *user_allow_other/user_allow_other/' /etc/fuse.conf
          yas3fs s3://mybucket/path /mnt/local_path
        • fusermount -u mnt/local_path
  • s3fs-fuse
    • Wiki
    • maximum file size: 64GB
    • dependčncies / dependencies
      • Mageia
        • urpmi lib64fuse-devel
      • CentOS
        • yum install automake gcc-c++ fuse fuse-devel libcurl-devel libxml2-devel
    • compilació / compilation
      • git clone https://github.com/s3fs-fuse/s3fs-fuse.git
      • cd s3fs-fuse
      • ./autogen.sh
      • ./configure --exec-prefix=/usr
      • make
      • su; make install
    • ?
      • echo "user_allow_other" >> /etc/fuse.conf
    • utilització / usage
      • Wowza
      • when not using roles
        • ~/.passwd-s3fs (/etc/passwd-s3fs)
          • bucketName:accessKeyId:secretAccessKey
        • chmod 600 ~/.passwd-s3fs
      • mkdir /mnt/bucketName; chmod 755 /mnt/bucketName
      • s3fs bucketName /mnt/bucketName -ouse_cache=/tmp -o allow_other,ahbe_conf=/etc/ahbe.conf
      • with role:
        • s3fs bucketName /mnt/bucketName -o allow_other,ahbe_conf=/etc/ahbe.conf,iam_role=my_rolename
      • mount on boot (FAQ)
        • How to force s3fs mount on boot
          • option 1:
            • /etc/init.d/local
          • option 2:
            • /etc/fstab
              • s3fs#my_bucket /mnt/my_bucket fuse _netdev,nonempty,allow_other,ahbe_conf=/etc/ahbe.conf 0 0
              • s3fs#my_bucket /mnt/my_bucket fuse _netdev,nonempty,allow_other,ahbe_conf=/etc/ahbe.conf,iam_role=my_rolename 0 0
            • Problems:
              • d????????? ? ? ? ?            ? my_bucket
                • Solution
                  • check that _netdev option is present in /etc/fstab
    • Problemes / Problems
    • Cloudfront Cache-control
      • ahbe.conf
        • sample_ahbe.conf
        • # mpd and m3u8 files are cached for 2 seconds
          .mpd Cache-Control max-age=2
          .m3u8 Cache-Control max-age=1
      • s3fs bucketName /mnt/bucketName -o allow_other,ahbe_conf="/etc/ahbe.conf"

RDS
  • Using RDS with Django
  • Security group
    • Default rule make database available from outside
    • To be reachable from ec2 instances, add a rule for your VPC (e.g. 172.32.0.0/16)

CloudFront

Route53

VPC
  • VPCs and Subnets
  • Differences in CloudFormation when having a particular VPC

    		•  default VPC
      		 own VPC

      		default subnet
      		 own subnet own subnet





      AWS::EC2::Subnet
      AWS::EC2::RouteTable
      AWS::EC2::Route
      AWS::EC2::SubnetRouteTableAssociation      		 AWS::EC2::VPC
      AWS::EC2::InternetGateway
      AWS::EC2::VPCGatewayAttachment
      AWS::EC2::Subnet
      AWS::EC2::RouteTable
      AWS::EC2::Route
      AWS::EC2::SubnetRouteTableAssociation
      AWS::EC2::SecurityGroup

      		"VpcId" : {"Ref" : "MyVPC"}
      AWS::EC2::Instance
      		 "SecurityGroups" : [{ "Ref" : "MySecurityGroup" }] "SecurityGroupIds" : [{ "Ref" : "MySecurityGroup" }]
      "SubnetId" : {"Ref" : "MyFirstSubnet"}      		 "SecurityGroupIds" : [{ "Ref" : "MySecurityGroup" }]
      "SubnetId" : {"Ref" : "MyFirstSubnet"}
      AWS::ElasticLoadBalancing::LoadBalancer
      		 "AvailabilityZones" : {"Fn::GetAZs": ""}
      		 "Subnets" : [{"Ref" : "MyFirstSubnet"}] "Subnets" : [{"Ref" : "MyFirstSubnet"}]
      AWS::AutoScaling::AutoScalingGroup
      		 "AvailabilityZones" : {"Fn::GetAZs": ""} "VPCZoneIdentifier" : [{"Ref" : "MyFirstSubnet"}] "VPCZoneIdentifier" : [{"Ref" : "MyFirstSubnet"}]
  • Multicast
    • Overlay Multicast in Amazon Virtual Private Cloud
      • Info
        • Get all instances with a multicast tag, and filter those within a specific community (e.g.: foo)
          • aws --output json ec2 describe-instances --filters "Name=tag-key,Values=multicast" >instances_multicast.json
          • jq '.Reservations[].Instances[] | select( .Tags[] | . and .Key=="multicast" and (.Value | startswith("foo")) )' instances_multicast.json
          • from the selected instances, get only some information:
            • jq '.Reservations[].Instances[] | select( .Tags[] | . and .Key=="multicast" and (.Value | startswith("foo")) ) | [.InstanceId, .PrivateIpAddress, .PublicIpAddress, .Tags]' instances_multicast.json
      • Setup
        • Setup step 1: Create a subnet
          • Option 1: just create a subnet in a existing VPC
          • Option 2: Create new AWS VPC (vpc-xxxxxx) with a subnet and a route table to Internet:
            • Subnet
              • Name: Public subnet
              • IPv4 CIDR: 10.0.0.0/24
            • Cloudformation
              • {
                    "Resources": {
                        "MyVPC": {
                            "Type" : "AWS::EC2::VPC",
                            "Properties" : {
                                "CidrBlock": "10.0.0.0/16",
                                "EnableDnsSupport" : "true",
                                "EnableDnsHostnames" : "true",
                                "Tags" :[ { "Key" : "Name", "Value" : "my-vpc"} } ]
                            }
                        },

                        "MyInternetGateway" : {
                            "Type" : "AWS::EC2::InternetGateway"
                        },

                        "MyVPCGatewayAttachment" : {
                            "Type" : "AWS::EC2::VPCGatewayAttachment",
                            "Properties" : {
                                "InternetGatewayId" : {"Ref" : "MyInternetGateway"},
                                "VpcId" : {"Ref" : "MyVPC"}
                            }
                        },

                        "MyPublicSubnet" : {
                            "Type" : "AWS::EC2::Subnet",
                            "Properties" : {
                                "VpcId" : { "Ref" : "MyVPC" },
                                "CidrBlock" : "10.0.0.0/24",
                                "MapPublicIpOnLaunch" : "true",
                                "Tags" : [ { "Key" : "Name", "Value" : "my-subnet"} ]
                            }
                        },
                       
                        "PublicRouteTable" : {
                            "Type" : "AWS::EC2::RouteTable",
                            "Properties" : {
                                "VpcId" : {"Ref" : "MyVPC"}
                            }
                        },
                       
                        "PublicRoute" : {
                            "Type" : "AWS::EC2::Route",
                            "DependsOn" : "MyVPCGatewayAttachment",
                            "Properties"  : {
                                "RouteTableId" : {"Ref" : "PublicRouteTable"},
                                "DestinationCidrBlock" : "0.0.0.0/0",
                                "GatewayId" : {"Ref" : "MyInternetGateway"}
                            }
                        },
                       
                        "PublicSubnetRouteTableAssociation" : {
                            "Type" : "AWS::EC2::SubnetRouteTableAssociation",
                            "Properties" : {
                                "SubnetId" : {"Ref" : "MyPublicSubnet"},
                                "RouteTableId" : {"Ref" : "PublicRouteTable"}
                            }
                        },

                    }
                }
        • Setup step 2: Create AWS role with policy:
          • {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "Stmt1414071732000",
                        "Effect": "Allow",
                        "Action": [
                            "ec2:DescribeInstances",
                            "ec2:DescribeTags",
                            "ec2:DescribeRegions"
                        ],
                        "Resource": [
                            "*"
                        ]
                    }
                ]
            }
          • Cloudformation:
            • ...
        • Setup step 3: Create AWS security group (sg-yyyyyy) in your VPC (vpc-xxxxxx)
          • Inbound Rules:
            • Type: Custom Protocol Rule
            • Protocol: GRE (47)
            • Port Range: All
            • Source: sg-yyyyyy
          • Cloudformation (to avoid circular dependency, a AWS::EC2::SecurityGroupIngress must be created)
            •     "MyInboundRule" : {
                      "Type": "AWS::EC2::SecurityGroupIngress",
                      "Properties":{
                      "IpProtocol" : "47",
                      "SourceSecurityGroupId" : {"Ref" : "MySecurityGroup"},
                      "GroupId" : {"Ref" : "MySecurityGroup"}
                      }
                  },
                 
                  "MySecurityGroup" : {
                      "Type" : "AWS::EC2::SecurityGroup",
                      "Properties" : {
                          "GroupDescription" : "Enable ports 22 (ssh), GRE (47) (multicast)",
                          "VpcId" : {"Ref" : "MyVPC"},
                          "SecurityGroupIngress" : [
                              {
                                  "IpProtocol" : "tcp",
                                  "FromPort" : "22",
                                  "ToPort" : "22",
                                  "CidrIp" : "0.0.0.0/0"
                              }
                          ]
                      }
                  },
        • Setup step 4: Creation of several instances with this role and security group, with tag:
          • Name: multicast; Value: foo,172.16.0.7/24
          • Name: multicast; Value: foo,172.16.0.8/24
          • ...
        • Installation
          • Option 1:
            • Installation of Ruby script
              • CentOS
                • sudo yum install unzip bridge-utils ebtables curl ruby ruby-devel rubygem-nokogiri rubygem-daemons libxml2-devel
                • sudo gem install aws-sdk-v1 integration libxml-ruby
              • cd
              • wget https://s3.amazonaws.com/mcd-code/mcd-code-2014-07-11.zip
              • cd mcd-code-2014-07-11
              • sudo chmod 755 *
              • sudo mkdir -p /opt/mcast
              • sudo cp -pr * /opt/mcast'
              • sudo chown -R root:root /opt/mcast
            • Start Ruby script
              • temporarily
                • sudo ruby -d /opt/mcast/mcd
              • daemon
                • CentOS
                  • mcd.service
                    • [Unit]
                      Description=Multicast daemon for AWS EC2 instances
                      After=syslog.target network.target cloud-init.service

                      [Service]
                      Type=simple
                      #PIDFile=/run/mcd.pid
                      ExecStartPre=/usr/local/bin/mcd_setup.sh foo 172.16.0.0/24
                      ExecStart=/opt/mcast/mcd
                       ExecStartPost=/usr/local/bin/mcd_setup_route.sh foo
                      ExecReload=/bin/kill -s HUP $MAINPID
                      ExecStop=/bin/kill -s QUIT $MAINPID

                      [Install]
                      WantedBy=multi-user.target
                  • mcd_setup_route.sh
                    • #!/bin/bash

                      function print_help_and_exit {
                          cat <<EOF
                      Usage: `basename $0` multicast_name

                      Add route for multicast: mcbr-<multicast_name>

                      Examples:
                      - `basename $0` foo

                      EOF
                          exit 1
                      }

                      MIN_ARGS=1
                      MAX_ARGS=1
                      if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
                      then
                          print_help_and_exit
                      fi

                      # options
                      if ! params=$(getopt -o h --long help -n $0 -- "$@")
                      then
                          # invalid option
                          print_help_and_exit
                      fi
                      eval set -- ${params}

                      while true
                      do
                          case "$1" in
                              -h | --help ) print_help_and_exit;;
                              -- ) shift; break ;;
                              * ) break ;;
                          esac
                      done

                      # parameters
                      multicast_name=$1

                      # wait for bridge to exist
                      bridge_name="mcbr-${multicast_name}"

                      timeout=60
                      increment=5
                      t=0

                      while (( t < timeout )) && (brctl show ${bridge_name} 2>&1 1>/dev/null | grep -q "No such device")
                      do
                          echo "[`basename $0`] bridge ${bridge_name} is not available yet (${t}s/${timeout}s)"
                          (( t+=increment ))
                          sleep ${increment}
                      done

                      # add route for multicast
                      echo "[`basename $0`] adding route for multicast: mcbr-${multicast_name}"
                      route add -net 224.0.0.0/4 mcbr-${multicast_name}

                      exit 0
                  • mcd_setup.sh
                    • #!/bin/bash

                      function print_help_and_exit {
                          cat <<EOF
                      Usage: `basename $0` multicast_name multicast_cidr
                      Set the AWS EC2 tag: "multicast", "<multicast_name>,<multicast_cidr>"
                      Address multicast_cidr has the same number as local ip address, but converted to specified subnet
                      E.g.: if local address is 10.1.2.3/24 and you specify 172.16.0.0/24, the multicast_cidr=172.16.0.3/24
                      IMPORTANT: role for this ec2 instance must include a policy with: "ec2:CreateTags"

                      Examples:
                      - `basename $0` foo 172.16.0.0/24
                      EOF
                          exit 1
                      }

                      MIN_ARGS=2
                      MAX_ARGS=2
                      if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
                      then
                          print_help_and_exit
                      fi

                      # options
                      if ! params=$(getopt -o h --long help -n $0 -- "$@")
                      then
                          # invalid option
                          print_help_and_exit
                      fi
                      eval set -- ${params}

                      while true
                      do
                          case "$1" in
                              -h | --help ) print_help_and_exit;;
                              -- ) shift; break ;;
                              * ) break ;;
                          esac
                      done

                      # parameters
                      multicast_name=$1
                      multicast_cidr=$2

                      function translate_ip {
                          input_cidr=$1
                          output_cidr=$2

                          # remove subnet
                          input_address=${input_cidr%/*}

                          # get network and prefix
                          eval $(ipcalc -np $output_cidr)
                          output_prefix=$PREFIX
                          output_network=$NETWORK

                          # calculate number of bytes (n)
                          let output_positions=${output_prefix}/8

                          # remove first n bytes
                          input_array=(${input_address//./ })
                          input_significative=${input_array[@]:${output_positions}}

                          # get first n bytes
                          output_array=(${output_network//./ })
                          output_significative=${output_array[@]:0:${output_positions}}
                         
                          # join all bytes
                          total_address_array=(${output_significative[@]} ${input_significative[@]})
                          total_address=$(IFS='.';echo "${total_address_array[*]}";IFS=$'')
                          total_cidr="${total_address}/${output_prefix}"
                          echo $total_cidr
                      }

                      # check whether ip command is available
                      if ! (which ip >/dev/null 2>&1)
                      then
                          echo "ERROR: command ip not found. Consider running this script as root or sudo."
                          exit 1
                      fi

                      # get own information
                      local_cidr=$(ip -o address | awk '$2 !~ /lo/ && $3 ~ /^inet$/ {print $4; exit;}')
                      #local_ipv4=$(curl http://169.254.169.254/latest/meta-data/local-ipv4/)

                      multicast_cidr=$(translate_ip $local_cidr $multicast_cidr)

                      echo multicast_cidr: $multicast_cidr

                      # eu-west-1c
                      if ! aws_subregion=$(curl -s -m 4 http://169.254.169.254/latest/meta-data/placement/availability-zone)
                      then
                          echo "no aws_subregion found. Are you sure that you are running this script on an AWS instance?"
                          exit 1
                      fi
                      # eu-west-1
                      aws_region=${aws_subregion: : -1}

                      instance_id=$(curl -s -m 4 http://169.254.169.254/latest/meta-data/instance-id)

                      # create a tag
                      aws ec2 create-tags --region ${aws_region} --resources $instance_id --tags Key="multicast",Value="${multicast_name}\,${multicast_cidr}"

                      exit 0
          • Option 2: Install bash script
            • Installation of mcd.sh
            • Start bash script
              • temporarily
              • daemon
                • CentOS
        • Check
          • Process
            • sudo ps -edalf | grep mcd
          • Logs
            • tail -f /var/log/messages | grep mcd
          • Created bridges
            • brctl show
          • Created GRE tunnels
            • ip link show
          • Routes
            • route -n
          • Members
            • netstat -g
          • Omping
            • from each instance:
              • omping 172.16.0.7 172.16.0.8 ...
            • if it does not work:
              • check if some gretap points to a non existing address
                • to get the remote address, check the logs in /var/log/messages when the gretap was created
        • Ús / Usage
          • routes
            • in each instance:
              • route add -net 224.0.0.0/4 mcbr-foo
          • application:
            • from one instance:
              • ffmpeg ...
            • from the other one:
              • ...

    Big data and analytics

    CLI
    • AWS Command Line Interface
    • Instal·lació / Installation (python)
    • Ús / Usage
      • aws --version ...
    • Problemes / Problems
      • ImportError: No module named history
        • the problem is that a pair of awscli (1.14.28) and botocore (1.6.0), installed from yum (awscli.noarch), does not work
        • Solució / Solution
          • sudo pip install awscli
            • a pair of working packages is, e.g.: awscli==1.14.28, botocore==1.8.32
    • queries
      • --query <query-expression>
      • uses JMESPath
    • templates
      • --generate-cli-skeleton > skeleton.json
      • --cli-input-json file://skeleton.json
    • stdin/stdout
      • aws s3 cp s3://bucket/key - | bzip2 -best | aws s3 cp - s3://bucket/key.bz2
    • CLI reference
      • general options
        • --region eu-west-1
        • --profile myprofile
        • --output json
        • ...
      • aws autoscaling
        • launch configuration
        • autoscaling group
          • aws autoscaling create-auto-scaling-group --auto-scaling-group-name grup_stateless --launch-configuration-name lc_stateless_yyyymmdd_hhmm --min-size 1 --max-size 2 --load-balancer-names lb-stateless
          • aws autoscaling update-auto-scaling-group --auto-scaling-group-name grup_stateless --launch-configuration-name lc_stateless_yyyymmdd_hhmm
          • number of instances inside an autoscaling group, given its name
            • result=$(aws --output json autoscaling describe-auto-scaling-groups --auto-scaling-group-names $asg_name)
              number_instances=$(echo $result | jq '.AutoScalingGroups[0].Instances | length')
          • get autoscaling group, given its tag Name=myname:
            • asg_info=$(aws --output json autoscaling describe-auto-scaling-groups)
              asg_name_tag_value="myname"
              asg=$(echo "$asg_info" | jq ".AutoScalingGroups[] | select( .Tags[] | . and .Key==\"Name\" and .Value==\"${asg_name_tag_value}\")")
          • aws autoscaling set-desired-capacity --auto-scaling-group-name $asg_name --desired-capacity 2
          • Instance protection
            • aws autoscaling set-instance-protection --instance-ids i-93633f9b --auto-scaling-group-name my-auto-scaling-group --protected-from-scale-in
          • given an instance id, get the autoscaling group name it belongs to:
            • aws ec2 describe-tags --filters "Name=resource-id,Values=$instance_id" "Name=key,Values=aws:autoscaling:groupName" | jq '.Tags[] | .Value'
        • get all autoscaling groups (with pagination)
          • aws_options="--profile my_profile --output json --region eu-west-1"
            next_token=""
            max_items=50
            page_size=100
            total_number=0
            total_elements=$(jq -n '[]')

            while [[ $next_token != "null" ]]
            do
                if [[ "$next_token" ]]
                then
                    lc_info=$(aws ${aws_options} autoscaling describe-launch-configurations --max-items $max_items --page-size $page_size --starting-token $next_token)
                else
                    lc_info=$(aws ${aws_options} autoscaling describe-launch-configurations --max-items $max_items --page-size $page_size)
                fi
                #echo $lc_info | jq '.'
                returned_number=$(echo $lc_info | jq  '.LaunchConfigurations | length' )
                returned_elements=$(echo $lc_info | jq  '.LaunchConfigurations')
                echo $returned_elements | jq '.'

                total_elements=$( echo $total_elements | jq ". += $returned_elements")
                echo "returned_number: $returned_number"
                total_number=$(( total + returned_number ))
                echo "total_number: $total_number"
                echo $lc_info | jq  '.LaunchConfigurations[] | ([.LaunchConfigurationName] | join (" "))'

                next_token=$(echo $lc_info | jq '.NextToken')
                echo "next_token: $next_token"
               
            done


            echo "---------------------------"
            echo $total_elements | jq '.[].LaunchConfigurationName'

      • aws cloudformation
      • aws cloudfront
        • aws configure set preview.cloudfront true
        • list all distributions
          • aws cloudfront list-distributions --output json
        • get a specific distribution:
          • aws cloudfront get-distribution --id E1KBXTVP599T0A
        • get the config for a specific distribution
          • aws cloudfront get-distribution-config --id E1KBXTVP599T0A
        • update distribution
          • aws cloudfront get-distribution-config --id ${cloudfront_id} --output json > /tmp/${cloudfront_id}.json

            # get etag
            etag=$(jq -r '.ETag' /tmp/${cloudfront_id}.json)

            # modify             /tmp/${cloudfront_id}.json
            ...

            aws cloudfront update-distribution --id ${cloudfront_id} --if-match $etag --distribution-config $(jq -c '.DistributionConfig' /tmp/${cloudfront_id}.json) 
      • aws configure
        • generated files
          • ~/.aws/credentials (for all SDKs)
          • ~/.aws/config (only for CLI)
        • crearŕ / will create: ~/.aws/config
          • [default]
            output = json
            region = eu-west-1
            aws_access_key_id = xxx
            aws_secret_access_key = yyy
            [preview]
            cloudfront = true
        • i/and ~/.aws/credentials (? now included in ~/.aws/config)
          • [default]
            aws_access_key_id = xxx
            aws_secret_access_key = yyy
        • per a fer servir un altre perfil / to use another profile
          • aws --profile myprofile configure
          • will create ~/.aws/config
            • [profile myprofile]
              output = json
              region = eu-west-1
          • and ~/.aws/credentials
            • [myprofile]
               aws_access_key_id = xxx
              aws_secret_access_key = yyy
          • aws --profile myprofile ...
        • per a fer servir un altre fitxer de configuració / to use an alternate config file (e.g. /etc/aws/config):
      • aws ec2
        • instances
          • aws ec2 start-instances --instance-ids i-9b789ed8
          • aws ec2 describe-instances --filter Name=tag:Name,Values=ubuntu_1310
            • ...
              TAGS    Name    ubuntu_1310
            • get PublicIpAddress
              • aws --output json ec2 describe-instances --instance-ids i-9b789ed8 | jq -r '.Reservations[0].Instances[0].PublicIpAddress'
          • aws ec2 describe-instance-attribute --instance-id i-896b92c9 --attribute instanceType
          • aws ec2 stop-instances --instance-ids i-9b789ed8
          • aws ec2 terminate-instances --instance-ids i-9b789ed8
          • waiters
            • aws ec2 wait instance-running --instance-id i-896b92c9
            • aws ec2 wait instance-status-ok --instance-id i-896b92c9
            • aws ec2 wait instance-stopped --instance-id i-896b92c9
        • images
          • create an image:
            • instance_id=i-xxxxxxx
              image_prefix=image_u1404
              data=$(date '+%Y%m%d_%H%M')
              imatge_id=$(aws ec2 create-image --instance-id ${instance_id} --name "${image_prefix}_${data}" --description "My description")
              echo "${imatge_id} ${image_prefix}_${data} (from ${instance_id})"
          • create an image but root volume (/dev/sda1) will be destroyed on termination (useful when this image will be used in a launch configuration of an autoscaling group)
            • instance_id=i-xxxxxxx
              image_prefix=image_u1404
              data=$(date '+%Y%m%d_%H%M')
              imatge_id=$(aws ec2 create-image --instance-id ${instance_id} --name "${image_prefix}_${data}" --description "My description" --block-device-mappings "[{\"DeviceName\": \"/dev/sda1\",\"Ebs\":{\"VolumeType\":\"gp2\",\"DeleteOnTermination\":true}}]")
              echo "${imatge_id} ${image_prefix}_${data} (from ${instance_id})"
          • describe images:
            • get all own images
              • aws ec2 describe-images --owners self
              • aws_options="--profile my_profile --output json --region eu-west-1"
                ami_info=$(aws ${aws_options} ec2 describe-images --owners self)
                echo $ami_info | jq  '.Images | sort_by(.Name) | .[] | [.ImageId, .OwnerId, .Name] | join("  ")'
            • describe an image given its id
            • get an image by name:
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json
            • using wildcards:
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json
            • get image_id of an image with given its name:
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json | awk '/ImageId/ {print $2}' | tr -d '",'
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json | jq -r '.Images[].ImageId'
            • get a list of amis, sorted by creation date
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json | jq -r '(.Images | sort_by(.CreationDate) | .[] | [.CreationDate, .ImageId] | join(" ") )'
            • get image_id of the most recent image:
              • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json | jq -r '(.Images | sort_by(.CreationDate) | .[-1] | .ImageId )'
            • aws ec2 describe-image-attribute --image-id ami-6be62b1c --attribute description
          • waiters
        • snapshots
        • add tags to an:
          • instance:
            • aws ec2 create-tags --resources i-9b789ed8 --tags Key=Name,Value=ubuntu_1404
          • image:
            • aws ec2 create-tags --resources ami-6be62b1c --tags Key=Name,Value=image_ubuntu_1404
            • value="..."
              # escape commas and remove single quotes
              escaped_value=$(echo ${value//,/\\,} | tr -d "'")
              aws ec2 create-tags --resources ami-6be62b1c --tags Key=Name,Value="${escaped_value}"
        • get tags from an:
          • instance
            • aws ec2 describe-tags --filters "Name=resource-id,Values=i-1234567890abcdef8"
        • create an instance from an image:
          • aws ec2 run-instances --image-id ami-6be62b1c --security-groups launch-wizard-2 --count 1 --key-name my_keyname --placement AvailabilityZone='eu-west-1a',Tenancy='default' --instance-type t2.micro
          • get the instance_id:
            • reponse in json (parse with jq)
              • instance_description=$(aws --output json ec2 run-instances --image-id $image_id --security-groups $security_group_name --iam-instance-profile Name="role-nfs_server" --count 1 --key-name $key_name --placement AvailabilityZone=${availability_zone},Tenancy='default' --instance-type $instance_type --block-device-mappings 'DeviceName=/dev/sda1,Ebs={DeleteOnTermination=true,VolumeType=gp2}' )
              • instance_id=$(echo $instance_description | jq -r '.Instances[0].InstanceId')
            • response in text (parse with awk)
              • instance_id=$(aws --output text ec2 run-instances --image-id ami-6be62b1c --security-groups launch-wizard-2 --count 1 --key-name my_keyname --placement AvailabilityZone='eu-west-1a',Tenancy='default' --instance-type t2.micro | awk '/INSTANCES/ {print $8}')
              • aws ec2 describe-instances --instance-ids $instance_id
          • overwrite the delete on termination behaviour:
            • aws ec2 run-instances --image-id ami-6be62b1c ... --block-device-mappings 'DeviceName=/dev/sda1,Ebs={DeleteOnTermination=true,VolumeType=gp2}'
          • ...
            • dades d'usuari /user data:
              • when creating instance:
              • des de la instŕncia / from ec2 instance:
                • curl http://169.254.169.254/latest/user-data/
              • ec2-run-user-data (ec2ubuntu) (already installed in EC2 Ubuntu ami)
          • assign a role to the instance
          • Problems:
            • Client.InvalidParameterCombination: Could not create volume with size 10GiB and iops 30 from snapshot 'snap-xxxxx'
              • AutoScaling - Client.InvalidParameterCombination
              • Re: Stabilization Error (Again)
              • "Iops" should not be there (?)
              • Workaround: create image from web interface instead
              • Solution: add --block-device-mappings option (following example is for 10GiB volume)
                • aws ec2 create-image --instance-id i-xxxxxxxx --name AMIName --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeType":"gp2","DeleteOnTermination":"true","VolumeSize":10}}]'
        • volumes
          • create a volume
            • volume_description=$(aws ec2 create-volume --availability-zone $availability_zone --volume-type $volume_type --size $small_volume_size_gibytes)
            • volume_id=$(echo $volume_description | jq '.VolumeId')
          • wait for volume to be created
          • tag a volume with a name
            • aws ec2 create-tags --resources $volume_id --tags Key=Name,Value=$volume_name
          • describe a volume with specified name:
          • get availability zone of a volume
            • aws_cli_options="--profile my_profile --output json"
              volume_name=my-volume-name
              volume_description=$(aws $aws_cli_options ec2 describe-volumes --filters "Name=tag:Name,Values=$volume_name")
              availability_zone=$(echo $volume_description | jq -r '.Volumes[0].AvailabilityZone')
          • attach a volume to an instance
            • aws ec2 attach-volume --volume-id $volume_id --instance-id $instance_id --device /dev/sd${volume_letter}
          • detach volume
          • list all volumes
            • response=$(aws --output json ec2 describe-volumes)

              while IFS= read -r; do
                  volume=$REPLY
                  volume_id=$(echo $volume | jq -r '.VolumeId')
                  echo "---  VolumeId: $volume_id"

                  #tags=$(echo $volume | jq -c -r '(.Tags | values | .[] | select(.Key == "Name") )')
                  tags=$(echo $volume | jq -c -r '(.Tags | values | .[] )')
                  for tag in $tags
                  do
                      tag_key=$(echo $tag | jq '.Key')
                      tag_value=$(echo $tag | jq '.Value')
                      echo "    $tag_key: $tag_value"
                  done
              done < <(echo "$response" | jq -c -r '.[] | .[]')
      • aws elb
        • create a load balancer and associate to an instance:
          • aws elb create-load-balancer --load-balancer-name lb-stateful --listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=80 --availability-zones eu-west-1a eu-west-1b eu-west-1c --security-groups sg-ba33c2df
          • aws elb configure-health-check --load-balancer-name lb-stateful --health-check Target=TCP:80,Interval=30,Timeout=10,UnhealthyThreshold=2,HealthyThreshold=2
          • aws elb register-instances-with-load-balancer --load-balancer-name lb-stateful --instances i-9b789ed8
        • create a load balancer to be associated to an autoscaling group:
          • aws elb create-load-balancer --load-balancer-name lb-stateless --listeners Protocol=TCP,LoadBalancerPort=1935,InstanceProtocol=TCP,InstancePort=1935 Protocol=HTTP,LoadBalancerPort=8080,InstanceProtocol=HTTP,InstancePort=8080 --availability-zones eu-west-1a eu-west-1b eu-west-1c --security-groups sg-ba33c2df
          • aws elb configure-health-check --load-balancer-name lb-stateless --health-check Target=TCP:8080,Interval=30,Timeout=10,UnhealthyThreshold=2,HealthyThreshold=2
        • add an HTTPS listener with ARN of an uploaded IAM certificate
          • aws --region eu-west-1 elb create-load-balancer-listeners --load-balancer-name $load_balancer_name --listeners Protocol=HTTPS,LoadBalancerPort=443,InstanceProtocol=HTTPS,InstancePort=443,SSLCertificateId=$ARN
        • modify the certificate of an existing listener for a given port:
      • aws iam
        • certificats de servidor / server certificates
          • upload a certificate
            • obtained e.g. from Letsencrypt
              • letsencrypt_dirname=/etc/letsencrypt
                aws iam upload-server-certificate --server-certificate-name cert-${domain} \
                    --certificate-body file://${letsencrypt_dirname}/live/${domain}/cert.pem \
                    --private-key file://${letsencrypt_dirname}/live/${domain}/privkey.pem \
                    --certificate-chain file://${letsencrypt_dirname}/live/${domain}/chain.pem
            • self-signed, to be used in cloudfront:
              • openssl req -new -nodes -keyout www.toto.org.key -sha256 -x509 -days 365 -out www.toto.org.crt
                • Common Name: www.toto.org
              • aws iam upload-server-certificate \
                    --server-certificate-name cert-www.toto.org \
                    --certificate-body file://.../                www.toto.org.crt \
                    --private-key file://.../                www.toto.org.key \
                                    --certificate-chain file://.../www.toto.org.crt \
                    --path /cloudfront/
          • get a list of server certificates
          • get ARN of a certificate (will be specified when adding a listener to a ELB)
            • ARN=$(aws --output json iam get-server-certificate --server-certificate-name ${domain}.cert | jq '.ServerCertificate.ServerCertificateMetadata.Arn')
          • check if a certificate is available
            • ...
      • aws route53
        • Adding EC2 instances to Route53 (bash+boto)
        • aws route53 list-resource-record-sets --hosted-zone-id xxxxxx
        • aws route53 change-resource-record-sets --hosted-zone-id xxxxxx --change-batch file:///absolute_path_to/change_entry.json
          • change_entry.json (to modify record www.toto.org; e.g. TTL value)
            • {
                "Comment": "Modifying TTL to 55",
                "Changes": [
                  {
                    "Action": "UPSERT",
                    "ResourceRecordSet": {
                      "Name": "www.toto.org.",
                      "Type": "A",
                      "TTL": 55,
                      "ResourceRecords": [
                        {
                          "Value": "xx.xx.xx.xx"
                        }
                      ]
                    }
                  }
                ]
              }
        • Auto configuration of Route53 from EC2 at boot (Ubuntu Upstart)
          • previously, from any computer:
            • option 1 (preferred): create a role and assign it to the instance when launching it
              • ...
            • option 2: create a user
              • create a user and group that can only modify Route53 entries:
                • from web interface:
                  • Grup
                    • Group name
                      • grup_nomes_route53
                    • Policy
                      • {
                          "Version": "2012-10-17",
                          "Statement": [
                            {
                              "Effect": "Allow",
                              "Action": [
                                "route53:*"
                              ],
                              "Resource": [
                                "*"
                              ]
                            }
                          ]
                        }
                • alternatively, from aws cli:
                  • ...
          • once logged into EC2 instance:
            • only if used option 2 (not using a role):
            • /usr/local/bin/route53.sh
              • #!/bin/bash

                # zone_id for                 mydomain.org
                ZONE_ID=$(cat zone_id.txt)
                ip_address=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
                name=www.mydomain.org
                tmp_file=/tmp/modify_record_set.json
                rm -f $tmp_file

                #aws route53 list-resource-record-sets --hosted-zone-id $ZONE_ID

                cat > $tmp_file <<EOF
                {
                  "Comment": "Modifying ip address",
                  "Changes": [
                    {
                      "Action": "UPSERT",
                      "ResourceRecordSet": {
                        "Name": "${name}.",
                        "Type": "A",
                        "TTL": 60,
                        "ResourceRecords": [
                          {
                            "Value": "$ip_address"
                          }
                        ]
                      }
                    }
                  ]
                }
                EOF
                #source /opt/p27/bin/activate
                aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch file://$tmp_file
                #deactivate
                exit 0
            • init script
              • CentOS
                • /etc/systemd/system/route53.service
                  • [Unit]
                    Description=Description of my script
                    After=syslog.target network.target

                    [Service]
                    Type=oneshot
                    ExecStart=/usr/local/bin/route53.sh

                    [Install]
                    WantedBy=multi-user.target
                • sudo systemctl enable route53.service
                • sudo systemctl start route53.service
              • Ubuntu
                • /etc/init/route53.conf
                  • description "route53 daemon"
                     
                    start on (filesystem and net-device-up IFACE=lo)
                    stop on runlevel [!2345]
                     
                    env DAEMON=/usr/local/bin/route53.sh
                    env PID=/var/run/route53.pid

                    env AWS_CONFIG_FILE=/home/ubuntu/.aws/credentials

                    exec $DAEMON
      • aws s3
        • Note: no need to create dirs: they do not exist
        • list all buckets
          • aws s3 ls
        • list all files in a "directory":
          • aws s3 ls --recursive s3://my_bucket/my_dir1/
        • copy a single file to S3:
          • aws s3 cp toto.txt s3://my_bucket/
        • recursively copy to S3:
          • aws s3 ...
        • sync
          • aws s3 sync ... --exclude '*.png' --exclude 'log' ...
        • recursively copy from S3:
          • aws s3 cp s3://my_bucket/ . --recursive

    Boto (Python)
    • Instal·lació / Installation
      • v3
        • pip install boto3
      • v2
        • pip install boto
        • Alternative: get it from git and install it:
          • cd ~/src
          • git clone https://github.com/boto/boto.git
          • cd boto
          • [source /opt/PYTHON27/bin/activate]
          • python setup.py install
    • Config
      • aws configure
      • ~/.boto
        • [Credentials]
          aws_access_key_id = xxxx
          aws_secret_access_key = yyyy
    • Docs
    • Problemes / Problems
    • Usage
      • Amazon EC2 Basics For Python Programmers
      • autoscaling
      • s3
      • cloudformation
        • boto.cloudformation
        • Examples with boto3:
          • ...
        • Examples with boto v2:
          • single EC2 instance
            • single_ec2.json
            • single_ec2.py
              • import boto.cloudformation
                from django.conf import settings
                ...

                    try:
                        conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )
                        stack = conn.create_stack(self.name,
                            template_body=template_body,
                            template_url=None,
                            parameters=[],
                            notification_arns=[],
                            disable_rollback=False,
                            timeout_in_minutes=None,
                            capabilities=None)
          • single EC2 entry with Route53
            • single_ec2_r53.json
            • single_ec2_r53.py
              •         try:
                            # connect to the cloud and create the stack
                           
                            # connect to AWS
                            conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )

                            # check if the stack already exists
                            existing_stacks = [s.stack_name for s in conn.describe_stacks()]
                            logger.debug("   Existing stacks: %s" % existing_stacks)
                            if self.name in existing_stacks:
                                logger.error("   Stack %s is already created" % self.name)
                                raise Exception("Stack %s is already created" % self.name)
                           
                            # create the stack
                            conn.create_stack(self.name,
                                              template_body=template_body,
                                              template_url=None,
                                              parameters=[
                                                          ('HostedZone','toto.org'),
                                                          ],
                                              notification_arns=[],
                                              disable_rollback=False,
                                              timeout_in_minutes=None,
                                              capabilities=None)
                           
                            # wait for COMPLETE
                            ready = False
                            while not ready:
                                stacks = conn.describe_stacks(self.name)
                                if len(stacks) == 1:
                                    stack = stacks[0]
                                else:
                                    raise Exception("Stack %s has not been created" % self.name)
                                logger.debug("   stack status: %s" % stack.stack_status)
                                # CREATE_COMPLETE, ROLLBACK_COMPLETE
                                ready = (string.find(stack.stack_status, 'COMPLETE')) != -1
                                time.sleep(5)
                           
                            # get output information
                            outputs = dict()
                            for output in stack.outputs:
                                outputs[output.key] = output.value
                           
                            logger.debug("   DomainName: %s" % outputs['DomainName'])
          • ...
            • my_file.py
              •             # connect to AWS
                            conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )
                           
                            stacks = conn.describe_stacks(stack_name)
                            if len(stacks) == 1:
                                stack = stacks[0]
                            else:
                                raise Exception("Stack %s does not exist" % stack_name)

                            # CREATE_COMPLETE, ROLLBACK_COMPLETE
                            ready = (string.find(stack.stack_status, 'CREATE_COMPLETE')) != -1
                    
                            if ready:       
                                # get parameters (conversion from ResultSet to dictionary)
                                parameters = {item.key:item.value for item in stack.parameters}
                               
                                # get output information (conversion from ResultSet to dictionary)
                                outputs = {item.key:item.value for item in stack.outputs}
      • cloudfront
        • Create an invalidation:
          •     try:
                    import boto3
                except Exception as e:
                    print 'ERROR: %s' % e

                profile_name = 'my_profile'
                session = boto3.Session(profile_name=profile_name)
                cloudfront_client = session.client('cloudfront')

                distribution_id = 'xxxxxx'
                  
                # create invalidation
                import time
                response = cloudfront_client.create_invalidation(
                                DistributionId=distribution_id,
                                InvalidationBatch={
                                    'Paths': {
                                        'Quantity': 1,
                                        'Items': ['/*']
                                    },
                                'CallerReference': str(time.time())
                                }
                            )

                print response
      • autoscaling
        • Examples with boto3
          • Get ids of instances inside an autoscaling group with specified name:
            • # get the list of instances in the autoscaling group
              client_autoscaling = boto3.client('autoscaling', region_name='eu-west-1')
              paginator = client_autoscaling.get_paginator('describe_auto_scaling_instances')
              page_iterator = paginator.paginate(
                  PaginationConfig={'PageSize': 50}
              )
                         
              # ids for instances whose 'AutoScalingGroupName' == asg_name
              # http://boto3.readthedocs.io/en/latest/guide/paginators.html#filtering-results-with-jmespath
              filtered_instances = page_iterator.search(
                  'AutoScalingInstances[?AutoScalingGroupName == `{}`]'.format(asg_name)
              )
              instance_ids = [ i['InstanceId'] for i in filtered_instances ]
      • ec2
        • An Introduction to boto’s EC2 interface
        • EC2 (API reference)
        • Amazon EC2 Deployment with Boto
        • yourfile.py
          • conn = boto.ec2.connect_to_region("eu-west-1", aws_access_key_id=settings.AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
            reservation = conn.run_instances( 'ami-...', security_groups=['launch-wizard'], min_count=1, max_count=1, key_name='parell_key', placement='eu-west-1a', tenancy='default', instance_type='t1.micro')
            instance = reservation.instances[0]
                       
            while instance.state != 'running':
                time.sleep(5)
                instance.update() # Updates Instance metadata
                print "Instance state: %s" % (instance.state)

            # add Name tag
            instance.add_tag("Name","your_instance_name")

            print "Instance ID: %s" % instance.id
            print "Instance IP address: %s" % instance.ip_address
          • # get all the reservations with a given Name tag:
            reservations = conn.get_all_instances(filters={'tag:Name': '            your_instance_name'})
            # get the first reservation
            reservation = reservations[0]
          • # get all the reservations with a given instance_id
            reservations = conn.get_all_instances(instance_ids=["i-27823367"])
            # get the first reservation
            reservation = reservations[0]
            instance = reservation.instances[0]
            # get the value of tag "Name"
            name = instance.tags["Name"]
          • # connection
            conn = boto.ec2.autoscale.connect_to_region(settings.AWS_REGION,
                                                        aws_access_key_id=settings.AWS_ACCESS_KEY_ID_EC2,
                                                        aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY_EC2)
                   
            # get autoscaling group
            asg = conn.get_all_groups(names='my_name')[0]
                   
            # get instances
            instance_ids = [i.instance_id for i in asg.instances]
            print " Instances ID: %s" % (instance_ids)
                   
            # shutdown instances
            asg.shutdown_instances()
                   
            # wait for all instances to be shutdown
            instances = True
            while instances:
                time.sleep(5)
                asg = conn.get_all_groups('my_name')[0]
                if not asg.instances:
                    instances = False
                else:
                    logger.debug(" still some instances in group %s"%self.name)
                   
            # remove group
            asg.delete()
    • zip

    Mobile

    http://www.francescpinyol.cat/aws.html
    Primera versió: / First version: 2.X.2015
    Darrera modificació: 11 de desembre de 2018 / Last update: 11th December 2018

    Valid HTML 4.01!

    Cap a casa / Back home