Amazon Web Services (AWS)

Índex

General

Casos d'ús / Use cases
  • Use cases
  • Digital Media
    • Digital Media in the Cloud: Best Practices for Processing Media on AWS (YouTube)
      1. Media processing in AWS (AWS)
        • Cloud transcoding architecture
          • Phase 1:
            • Add transcoder instances to EC2
            • Use S3 to store file-based sources
            • Use S3 to store file-based outputs
            • Use CloudFront to distribute output streams
          • Phase 2:
            • Use acceleration and/or Direct Connect for ingest
            • Use Amazon Virtual Private Cloud to ringfence
            • Use EC2 Reserved Instances
            • Use EC2 Spot Instances
          • Phase 3:
            • Create a fleet of transcode workers
            • Use your on-premise workflow controller to orchestrate using SWF
            • Use SQS to create a cloud transcode queue
            • Use SNS for notifications
        • Securing content (14:00)
          • Local encryption: encrypt and mantain your own keys
          • Network encryption: use secured network transfer (SSL, VPC)
          • At REST encryption: S3 encrypts at REST using AES-256
          • DRM: integrate certificated-based DRM through third parties
          • Watermarking: Integrate digital watermarking through third parties
      2. Best practices for hybrid transcoding workflows (Elemental Technologies)
      3. Cloud-based content management (Ericsson)
      4. High performance media processing (Intel)

Cloudformation
  • AWS CloudFormation Product Details
  • User guide
  • 19 Best Practices for Creating Amazon CloudFormation Templates
  • Designer
  • Preserve resources after stack destruction
  • Info
  • CLI Cloudformation
  • EC2
    • Instance
    • Volume
    • VPC
    • single EC2 instance
      • single_ec2.json
        • {
              "Description": "Single EC2 instance",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Resources": {
                  "singleEC2": {
                      "Type": "AWS::EC2::Instance",
                      "Properties": {
                          "ImageId":"ami-xxxxxxxx",
                          "KeyName":"my_key_pair",
                          "InstanceType":"t2.micro"
                      }
                  }
              }
          }
    • single ec2 instance with an extra volume
      • single_ec2_volume.json
        • "Resources": {
          ...
              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":{"Ref" : "ImageId"},
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"my_server_key",
                  "InstanceType":{"Ref" : "InstanceType"},
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe\n",
                          "sudo mkfs -t xfs /dev/xvdh           \n",
                          "sudo mkdir /mnt/vol           \n",
                          "sudo chmod 777 /mnt/vol           \n",
                          "sudo mount /dev/xvdh /mnt/vol           \n",
                      ] ]
                      }
                  },
                  "Tags":[{"Key":"Name","Value":{"Ref":"BaseName"}}]
                  }
              },

          ...

              "NewVolume" : {
                  "Type" : "AWS::EC2::Volume",
                  "Properties" : {
                      "Size" : "100",
                      "AvailabilityZone" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ]}
                  }
              },
             
              "MountPoint" : {
                  "Type" : "AWS::EC2::VolumeAttachment",
                  "Properties" : {
                      "InstanceId" : { "Ref" : "EC2Instance" },
                      "VolumeId"  : { "Ref" : "NewVolume" },
                      "Device" : "/dev/xvdh"
                  }
              },
    • single EC2 with UserData and extra volume
      • Notes:
        • When using direct bash commands:
          • add "\n" at the end of each command
          • no need to call sudo
      • single_ec2_userdata_volume.json
        • {
              "Description": "Single EC2 instance with extra volume",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Parameters" : {
              "InstanceType" : {
                  "Description" : "EC2 instance type",
                  "Type" : "String",
                  "Default" : "t2.micro",
                  "AllowedValues" : [ "t1.micro", "t2.micro", "t2.small", "t2.medium", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"]
                  ,
                  "ConstraintDescription" : "must be a valid EC2 instance type."
              },
             
              "HostedZone" : {
                  "Type" : "String",
                  "Description" : "The DNS name of an existing Amazon Route 53 hosted zone",
                  "AllowedPattern" : "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
                  "ConstraintDescription" : "must be a valid DNS zone name.",
                  "Default" : "example.net"
              },
              "ImageId" : {
                  "Type" : "String",
                  "Description" : "The image_id for the ec2 instance"
              },
              "NewVolumeSize" : {
                  "Type" : "String",
                  "Description" : "The size of the new volume (GB)",
                  "Default": "5"
              }
              },   
              "Resources": {

              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":{"Ref" : "ImageId"},
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"wct_streaming_server",
                  "InstanceType":{"Ref" : "InstanceType"},
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe \n",
                          "while [ ! -e /dev/xvdh ]; do echo waiting for /dev/xvdh to attach; sleep 10; done \n",
                          "mkfs -t xfs /dev/xvdh \n",
                          "mkdir -p /mnt/vol1 \n",
                          "mount /dev/xvdh /mnt/vol1 \n",
                          "chmod 777 /mnt/vol1 \n"
                      ] ]
                      }
                  },
                  "Tags":[{"Key":"Name","Value":{"Ref":"BaseName"}}]
                  }
              },
            
              "NewVolume" : {
                  "Type" : "AWS::EC2::Volume",
                  "Properties" : {
                  "Size" : {"Ref" : "NewVolumeSize"},
                  "AvailabilityZone" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ]}
                  }
              },
             
              "MountPoint" : {
                  "Type" : "AWS::EC2::VolumeAttachment",
                  "Properties" : {
                  "InstanceId" : { "Ref" : "EC2Instance" },
                  "VolumeId"  : { "Ref" : "NewVolume" },
                  "Device" : "/dev/xvdh"
                  }
              },

          }
    • single EC2 entry with Route53
      • single_ec2_r53.json
        • {
              "Description": "Single EC2 instance",
              "AWSTemplateFormatVersion": "2010-09-09",
              "Metadata": {},
              "Parameters" : {
              "InstanceType" : {
                  "Description" : "EC2 instance type",
                  "Type" : "String",
                  "Default" : "m1.small",
                  "AllowedValues" : [ "t1.micro", "t2.micro", "t2.small", "t2.medium", "m1.small", "m1.medium", "m1.large", "m1.xlarge", "m2.xlarge", "m2.2xlarge", "m2.4xlarge", "m3.medium", "m3.large", "m3.xlarge", "m3.2xlarge", "c1.medium", "c1.xlarge", "c3.large", "c3.xlarge", "c3.2xlarge", "c3.4xlarge", "c3.8xlarge", "c4.large", "c4.xlarge", "c4.2xlarge", "c4.4xlarge", "c4.8xlarge", "g2.2xlarge", "r3.large", "r3.xlarge", "r3.2xlarge", "r3.4xlarge", "r3.8xlarge", "i2.xlarge", "i2.2xlarge", "i2.4xlarge", "i2.8xlarge", "d2.xlarge", "d2.2xlarge", "d2.4xlarge", "d2.8xlarge", "hi1.4xlarge", "hs1.8xlarge", "cr1.8xlarge", "cc2.8xlarge", "cg1.4xlarge"]
                  ,
                  "ConstraintDescription" : "must be a valid EC2 instance type."
              },
             
              "HostedZone" : {
                  "Type" : "String",
                  "Description" : "The DNS name of an existing Amazon Route 53 hosted zone",
                  "AllowedPattern" : "(?!-)[a-zA-Z0-9-.]{1,63}(?<!-)",
                  "ConstraintDescription" : "must be a valid DNS zone name."
              }
              },   
              "Resources": {
              "EC2Instance": {
                  "Type": "AWS::EC2::Instance",
                  "Properties": {
                  "ImageId":"ami-437da730",
                  "SecurityGroups" : [ { "Ref" : "InstanceSecurityGroup" } ],
                  "KeyName":"my_key_pair",
                  "InstanceType":"t2.micro"
                  }
              },
              "InstanceSecurityGroup" : {
                  "Type" : "AWS::EC2::SecurityGroup",
                  "Properties" : {
                  "GroupDescription" : "Enable SSH, HTTP, RTMP",
                  "SecurityGroupIngress" : [
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "22",
                      "ToPort" : "22",
                      "CidrIp" : "0.0.0.0/0"
                      },
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "80",
                      "ToPort" : "80",
                      "CidrIp" : "0.0.0.0/0"
                      },
                      {
                      "IpProtocol" : "tcp",
                      "FromPort" : "1935",
                      "ToPort" : "1935",
                      "CidrIp" : "0.0.0.0/0"
                      }
                  ]
                  }
              },
              "MyDNSRecord" : {
                  "Type" : "AWS::Route53::RecordSet",
                  "Properties" : {
                  "HostedZoneName" : { "Fn::Join" : [ "", [{"Ref" : "HostedZone"}, "." ]]},
                  "Comment" : "DNS name for my instance.",
                  "Name" : { "Fn::Join" : [ "", [{"Ref" : "EC2Instance"}, ".", {"Ref" : "AWS::Region"}, ".", {"Ref" : "HostedZone"} ,"."]]},
                  "Type" : "A",
                  "TTL" : "900",
                  "ResourceRecords" : [ { "Fn::GetAtt" : [ "EC2Instance", "PublicIp" ] } ]
                  }
              }   
              },
              "Outputs" : {
              "InstanceId" : {
                  "Description" : "InstanceId of the newly created EC2 instance",
                  "Value" : { "Ref" : "EC2Instance" }
              },
              "AZ" : {
                  "Description" : "Availability Zone of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "AvailabilityZone" ] }
              },
              "PublicDNS" : {
                  "Description" : "Public DNSName of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "PublicDnsName" ] }
              },
              "PublicIP" : {
                  "Description" : "Public IP address of the newly created EC2 instance",
                  "Value" : { "Fn::GetAtt" : [ "EC2Instance", "PublicIp" ] }
              },
              "DomainName" : {
                  "Description" : "Fully qualified domain name",
                  "Value" : { "Ref" : "MyDNSRecord" }
              }
              }
             
          }
  • EFS
    • AWS::EFS::FileSystem
    • AWS::EFS::MountTarget
    • exemple
      • ...
        "Resources"         : {
            "MyFileSystem" : {
                "Type": "AWS::EFS::FileSystem",
                "Properties": {
                "PerformanceMode": "generalPurpose",
                "FileSystemTags": [
                    {
                    "Key": "Name",
                    "Value": "my-fs"
                    }
                ]
                }
            },
           
            "MyMountTargetSecurityGroup" : {
                "Type" : "AWS::EC2::SecurityGroup",
                "Properties" : {
                "GroupDescription" : "Enable ports 2049 (nfs)",
                "VpcId" : {"Ref" : "VPCId"},
                "SecurityGroupIngress" : [
                    {
                    "IpProtocol" : "tcp",
                    "FromPort" : "2049",
                    "ToPort" : "2049",
                    "CidrIp" : {"Ref": "CidrSubnet"}
                    }
                ]
                }
            },

            "MyMountTarget" : {
                "Type": "AWS::EFS::MountTarget",
                "Properties": {
                "FileSystemId": { "Ref": "MyFileSystem" },
                "SubnetId": { "Ref": "MySubnet" },
                "SecurityGroups": [ { "Ref": "MyMountTargetSecurityGroup" } ]       
                }
            },
        ...
        "# mount efs from instance \n",
        "# mount -t nfs4 -o nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport ", {"Ref": "ProcessFileSystem"},".efs.",{"Ref" : "AWS::Region"},".amazonaws.com:/ /mnt/efs \n",
        "mkdir -p /mnt/efs \n",
        "echo ", {"Ref": "ProcessFileSystem"},".efs.",{"Ref" : "AWS::Region"},".amazonaws.com:/ /mnt/efs nfs4 nfsvers=4.1,rsize=1048576,wsize=1048576,hard,timeo=600,retrans=2,noresvport 0 0 >>/etc/fstab \n",
        "        mount /mnt/efs \n",
        ...

          }
  • Route53
  • Alarm
  • CloudFront
    • AWS::CloudFront::Distribution
      • "PriceClass" :
        • "PriceClass_100" -> Use Only U.S., Canada and Europe
        • "PriceClass_200" -> Use U.S., Canada, Europe, Asia and Africa
        • "PriceClass_All" -> Use All Edge Locations (Best Performance)
    • No cache for 404
      • "MyCloudFront" : {
            "Type" : "AWS::CloudFront::Distribution",
            "Properties" : {
                "DistributionConfig" : {
                    "CustomErrorResponses" : [ {
                        "ErrorCode" : "404",
                        "ErrorCachingMinTTL" : "2"
                    } ]
                    ...
                 }                 }
        }
    • Cache behaviour /  Forward headers: Whitelist
      • Configuring CloudFront to Cache Objects Based on Request Headers
      • To avoid problems with CORS and 403 response from CloudFront
      •                     "DefaultCacheBehavior" : {
                                "TargetOriginId" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                "ForwardedValues" : {
                                    "Headers" : ["Origin","Access-Control-Request-Headers","        Access-Control-Request-Method"],
                                    "QueryString" : "false",
                                    "Cookies" : { "Forward" : "none" }
                                },
                                "AllowedMethods" : ["GET", "HEAD", "OPTIONS"],
                                "CachedMethods" : ["GET", "HEAD", "OPTIONS"],
                                "ViewerProtocolPolicy" : "allow-all"
                            },
    • Full examples:
      • Origin is S3, with whitelist for forwarded headers ("Origin"):
        •     "Resources": {
                  "MyCloudFront" : {
                      "Type" : "AWS::CloudFront::Distribution",
                      "Properties" : {
                          "DistributionConfig" : {
                              "Origins" : [ {
                                  "DomainName": { "Fn::Join" : [ "", [{"Ref":"BucketName"}, ".s3.amazonaws.com"]]},
                                  "OriginPath": "my_dir",
                                  "Id" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                  "S3OriginConfig": {}
                              }],
                              "Enabled" : "true",
                              "Comment" : "My comments",
                              "DefaultCacheBehavior" : {
                                  "TargetOriginId" : { "Fn::Join" : [ "", ["S3-", {"Ref":"BucketName"}, "-my_dir" ] ]},
                                  "ForwardedValues" : {
                                      "Headers" : ["Origin"],
                                      "QueryString" : "false",
                                      "Cookies" : { "Forward" : "none" }
                                  },
                                  "ViewerProtocolPolicy" : "allow-all"
                              },
                              "PriceClass" : "PriceClass_100"
                          }      
                      }
                  }
              },
      • Origin is own http server, with no cache for 404 responses:
        •     "Resources": {
                  "MyCloudFront" : {
                      "Type" : "AWS::CloudFront::Distribution",
                      "Properties" : {
                          "DistributionConfig" : {
                              "Origins" : [ {
                                  "DomainName": "myserver.toto.org",
                                  "OriginPath": "/root_dir",
                                  "Id" : "oid-root_dir",
                                  "CustomOriginConfig": {
                                      "HTTPPort": "80",
                                      "HTTPSPort": "443",
                                      "OriginProtocolPolicy": "http-only"
                                  }
                              }],
                              "Enabled" : "true",
                              "Comment" : "My comments",
                              "DefaultCacheBehavior" : {
                                  "TargetOriginId" :           "oid-root_dir",
                                  "ForwardedValues" : {
                                      "QueryString" : "false",
                                      "Cookies" : { "Forward" : "none" }
                                  },
                                  "ViewerProtocolPolicy" : "allow-all"
                              },
                              "CustomErrorResponses" : [ {
                                  "ErrorCode" : "404",
                                  "ErrorCachingMinTTL" : "2"
                              } ],
                              "PriceClass" : "PriceClass_100"
                          }      
                      }
                  }
              },
    • Amazon CloudFront Template Snippets
    • Amazon CloudFront - Introduction
  • LoadBalancer

    		• Application Load Balancer Network Load Balancer
      		 Classic Load Balancer
      protocols
      		 HTTP, HTTPS, HTTP/2, WebSockets
      		HTTP, HTTPS, TCP, SSL

      		AWS::ElasticLoadBalancingV2::LoadBalancer
      		AWS::ElasticLoadBalancing::LoadBalancer
      Fn::GetAtt
      		 CanonicalHostedZoneID
      		CanonicalHostedZoneNameID
      DNSName
      		CanonicalHostedZoneName
      LoadBalancerName
      		-

      		AWS::ElasticLoadBalancingV2::Listener
      AWS::ElasticLoadBalancingV2::TargetGroup (TargetGroups)
      		Listeners
    • Classic Load Balancer
      • listener with redirect ports http/80, http/8088, https/8089, tcp/1935 and LBCookieStickinessPolicy:
        • "MyLoadBalancer": {
              "Type": "AWS::ElasticLoadBalancing::LoadBalancer          ",
              "Properties": {
                  "LoadBalancerName": "MyLoadbalancerName",
                  "SecurityGroups" : [ ... ],
                  "AvailabilityZones": {
                      "Fn::GetAZs": ""
                  },
                  "CrossZone": "true",
                  "ConnectionSettings": {
                      "IdleTimeout" : 60
                  }
                  "Listeners": [
                      {
                          "LoadBalancerPort": "80",
                          "InstancePort": "80",
                          "Protocol": "HTTP",
                          "PolicyNames": ["MyFirstLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "8088",
                          "InstancePort": "8088",
                          "Protocol": "HTTP",
                          "PolicyNames": ["MySecondLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "8089",
                          "Protocol": "HTTPS",
                          "InstancePort": "8089",
                          "InstanceProtocol": "HTTPS",
                          "SSLCertificateId": "arn:aws:acm:eu-west-1:...",
                          "PolicyNames": ["MySecondLBCookieStickinessPolicy"]
                      },
                      {
                          "LoadBalancerPort": "1935",
                          "InstancePort": "1935",
                          "Protocol": "TCP"
                      }
                  ],
                  "LBCookieStickinessPolicy" : [
                      {
                          "CookieExpirationPeriod" : "500",
                          "PolicyName" : "MyFirstLBCookieStickinessPolicy"
                      },
                      {
                          "CookieExpirationPeriod" : "1000",
                          "PolicyName" : "MySecondLBCookieStickinessPolicy"
                      }
                  ],
                  "HealthCheck": {
                      "Target": "HTTP:80/",
                      "HealthyThreshold": "3",
                      "UnhealthyThreshold": "5",
                      "Interval": "30",
                      "Timeout": "5"
                  }
              }
          }
      • listener with a certificate from ACM
        • ...
    • SecurityGroup
    • UDP
  • AutoScalingGroup
    • Prerequisites:
      • cfn-signal
        • Download from: Bootstrapping Applications using AWS CloudFormation
        • kixorz/ubuntu-cloudformation.json
        • Install
          • mkdir aws-cfn-bootstrap-latest
          • curl https://s3.amazonaws.com/cloudformation-examples/aws-cfn-bootstrap-latest.tar.gz | tar xz -C aws-cfn-bootstrap-latest --strip-components 1
          • easy_install aws-cfn-bootstrap-latest
        • Problemes / Problems
          • Traceback (most recent call last):
              File "/bin/easy_install", line 9, in <module>
                load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
              [...]
              File "/usr/lib/python2.7/site-packages/setuptools/command/easy_install.py", line 701, in process_distribution
                distreq.project_name, distreq.specs, requirement.extras
            TypeError: __init__() takes exactly 2 arguments (4 given)
            • Diagnose
              • python
                • >>> from pkg_resources import load_entry_point
                  >>> load_entry_point('setuptools==0.9.8', 'console_scripts', 'easy_install')()
                  ...
                  pkg_resources.VersionConflict: (setuptools 25.1.4 (/usr/lib/python2.7/site-packages/setuptools-25.1.4-py2.7.egg), Requirement.parse('setuptools==0.9.8'))
            • Solució / Solution
              • sudo rm -rf /usr/lib/python2.7/site-packages/setuptools-25.1.4-py2.7.egg
      • You must install a UserData on your instance (AMI) that generates a cfn-signal received by CreationPolicy
        •     "LaunchConfig" : {
                  "Type" : "AWS::AutoScaling::LaunchConfiguration",
                  "Properties" : {
                  "ImageId":{"Ref" : "MyImageId"},
                  "SecurityGroups" : [ { "Ref" : "MySecurityGroup" } ],
                  "KeyName":"my_key",
                  "InstanceType":{"Ref" : "MyInstanceType"},
                  "IamInstanceProfile": "role_my_server",
                  "UserData": {
                      "Fn::Base64": {
                      "Fn::Join" : [ "", [
                          "#!/bin/bash -xe\n",
                          "/usr/bin/cfn-signal -e 0 --stack ", { "Ref": "AWS::StackName" },
                          " --resource MyAutoscalingGroup ",
                          " --region ", { "Ref" : "AWS::Region" }, "\n"
                      ] ]
                      }
                  }
                  }
              },
    • Auto Scaling Template Snippets
    • AutoScalingGroup
      • AWS::AutoScaling::AutoScalingGroup
      • example.json
        •     "MyAutoScalingGroup" : {
                  "Type" : "AWS::AutoScaling::AutoScalingGroup",
                  "Properties" : {
                  "LaunchConfigurationName" : { "Ref" : "MyLaunchConfig" },
                  "MinSize" : "1",
                  "MaxSize" : "3",
                  "LoadBalancerNames" : [ { "Ref" : "MyLoadBalancer" } ],
                  "Tags":[
                      {
                      "Key":"Name",
                      "Value":{ "Fn::Join" : [ "", ["myinstance-", {"Ref" : "MyName"}]]},
                      "PropagateAtLaunch" : "true"
                      }
                  ]
                  },
                  "CreationPolicy" : {
                  "ResourceSignal" : {
                      "Timeout" : "PT15M",
                      "Count"   : "1"
                  }
                  },
                  "UpdatePolicy": {
                  "AutoScalingRollingUpdate": {
                      "MinInstancesInService": "1",
                      "MaxBatchSize": "1",
                      "PauseTime" : "PT15M",
                      "WaitOnResourceSignals": "true"
                  }
                  }
              },
    • ScalingPolicy
    • Alarm
      • AWS::...
      • example.json
        •     "MyUpScalingPolicy" : {
                  "Type" : "AWS::AutoScaling::ScalingPolicy",
                  "Properties" : {
                  "AdjustmentType" : "ChangeInCapacity",
                  "AutoScalingGroupName" : { "Ref" : "MyAutoScalingGroup" },
                  "Cooldown" : "60",
                  "ScalingAdjustment" : "1"
                  }
              },

              "MyCPUHighAlarm": {
                  "Type": "AWS::CloudWatch::Alarm",
                  "Properties": {
                  "EvaluationPeriods": "1",
                  "Statistic": "Average",
                  "Threshold": "80",
                  "AlarmDescription": "Alarm if CPU too high or metric disappears indicating instance is down",
                  "Period": "60",
                  "AlarmActions": [ { "Ref": "MyUpScalingPolicy" } ],
                  "Namespace": "AWS/EC2",
                  "Dimensions": [ {
                      "Name": "AutoScalingGroupName",
                      "Value": { "Ref": "MyAutoScalingGroup" }
                  } ],
                  "ComparisonOperator": "GreaterThanThreshold",
                  "MetricName": "CPUUtilization"
                  }
              },
    • LaunchConfiguration
    • Example:
      • my_scaling.json
        • ...
  • ElastiCache
  • ...

IAM

ACM (AWS Certificate Manager)

EC2

S3
  • Amazon S3 Path Deprecation Plan – The Rest of the Story
    • https://bucketname.s3.amazonaws.com/
    • old versions:
      • https://s3.amazonaws.com/bucketname/
      • https://s3-us-east-2.amazonaws.com/bucketname/
  • Bucket policy
    • AWS Policy Generator
    • Granting Read-Only Permission to an Anonymous User
    • Public Readable Amazon S3 Bucket Policy
      • {
          "Version": "2008-10-17",
          "Statement": [
            {
              "Sid": "AllowPublicRead",
              "Effect": "Allow",
              "Principal": {
                "AWS": "*"
              },
              "Action": [
                "s3:GetObject"
              ],
              "Resource": [
                "arn:aws:s3:::bucket_name/*"
              ]
            }
          ]
        }
      • Boto (Python):
        • # modify policy to make it publicly available
          policy_json = '{"Version":"2008-10-17","Statement":[{"Sid":"AllowPublicRead","Effect":"Allow","Principal":{"AWS":"*"},"Action":["s3:GetObject"],"Resource":["arn:aws:s3:::%s/*"]}]}' % (bucket_name)
          print policy_json
          bucket.set_policy(policy_json)
  • Cache
  • CORS
  • Static website
    • Virtual Hosting of Buckets
    • Hosting a Static Website on Amazon S3
    • Example: Setting Up a Static Website Using a Custom Domain
    • Redireccionament / Redirect
    • AngularJS
    • Directory list
    • HTTPS
    • s3 bucket
      		 cloudfront
      		 route53
      		 http
      		 https
      name
      		 Static website hosting



      www.toto.org



      		 -
      		 -
      		 -
      		 http://s3-eu-west-1.amazonaws.com/www.toto.org/index.html
      		 https://s3-eu-west-1.amazonaws.com/www.toto.org/index.html
      -
      		 -
      		 -
      		 http://www.toto.org.s3.amazonaws.com/ (Access denied)
      http://www.toto.org.s3.amazonaws.com/index.html
      http://www.toto.org.s3.eu-west-1.amazonaws.com/ (Access denied)
      http://www.toto.org.s3.eu-west-1.amazonaws.com/index.html
      		 https://www.toto.org.s3.amazonaws.com/
      x
      		 -
      		 -
      		 http://www.toto.org.s3-website-eu-west-1.amazonaws.com/ https://www.toto.org.s3-website-eu-west-1.amazonaws.com/ (timeout: because of dots?)
      x
      		 -
      		 www.toto.org A ALIAS s3-website-eu-west-1.amazonaws.com.
      		 http://www.toto.org/ https://www.toto.org/
      x
      • Origin Domain Name: www.toto.org.s3.amazonaws.com
      • Default root object: index.html
      • Alternate Domain Names (CNAMEs): www.toto.org
      • Custom SSL Certificate: (choose one of the certificates previously uploaded to path /cloudfront/)
      		www.toto.org A ALIAS www.toto.org (xxxxxx.cloudfront.net) http://www.toto.org/ https://www.toto.org/
      www-toto-org
      		 -
      		 -
      		 -
      		 http://s3-eu-west-1.amazonaws.com/www-toto-org/ (Access denied)
      http://s3-eu-west-1.amazonaws.com/www-toto-org/index.html
      		 https://s3-eu-west-1.amazonaws.com/www-toto-org/
      https://s3-eu-west-1.amazonaws.com/www-toto-org/index.html
      -
      		 -
      		 -
      		 http://www-toto-org.s3.amazonaws.com/ (Access denied)
      http://www-toto-org.s3.amazonaws.com/index.html
      http://www-toto-org.s3-eu-west-1.amazonaws.com/ (Access denied)
      http://www-toto-org.s3.eu-west-1.amazonaws.com/index.html
      		 https://www-toto-org.s3.amazonaws.com/ (Access denied)
      https://www-toto-org.s3.amazonaws.com/index.html
      https://www-toto-org.s3-eu-west-1.amazonaws.com/ (Access denied)
      https://www-toto-org.s3.eu-west-1amazonaws.com/index.html
      x
      		 -
      		 -
      		 http://www-toto-org.s3-website-eu-west-1.amazonaws.com/ https://www-toto-org.s3-website-eu-west-1.amazonaws.com/ (timeout)
      x
      • Origin Domain Name: www-toto-org.s3.amazonaws.com
      • Default root object: index.html
      • Alternate Domain Names (CNAMEs): www.toto.org
      • Custom SSL Certificate: (choose one of the certificates previously uploaded to path /cloudfront/)
      		 www.toto.org A ALIAS www.toto.org (xxxxxx.cloudfront.net) http://www.toto.org/ https://www.toto.org/
  • s3tools
    • s3cmd
      • Instal·lació / Installation
      • make bucket
        • s3cmd mb s3://...
      • list
        • s3cmd ls s3://bucket_name
      • upload
        • s3cmd put ...
  • yas3fs
    • Instal·lació / Installation
      • CentOS
        • sudo yum -y install fuse fuse-libs
          sudo easy_install pip
          sudo pip install yas3fs
          sudo sed -i'' 's/^# *user_allow_other/user_allow_other/' /etc/fuse.conf
          yas3fs s3://mybucket/path /mnt/local_path
        • fusermount -u mnt/local_path
  • s3fs-fuse
    • Wiki
    • maximum file size: 64GB
    • dependències / dependencies
      • Mageia
        • urpmi lib64fuse-devel
      • CentOS
        • yum install automake gcc-c++ fuse fuse-devel libcurl-devel libxml2-devel
    • compilació / compilation
      • git clone https://github.com/s3fs-fuse/s3fs-fuse.git
      • cd s3fs-fuse
      • ./autogen.sh
      • ./configure --exec-prefix=/usr
      • make
      • su; make install
      • Problemes / Problems
        • ./configure: line 4964: syntax error near unexpected token `common_lib_checking,'
          ./configure: line 4964: `PKG_CHECK_MODULES(common_lib_checking, fuse >= ${min_fuse_version} libcurl >= 7.0 libxml-2.0 >= 2.6 )'
          • ...
    • ?
      • echo "user_allow_other" >> /etc/fuse.conf
    • utilització / usage
      • Wowza
      • when not using roles
        • ~/.passwd-s3fs (/etc/passwd-s3fs)
          • bucketName:accessKeyId:secretAccessKey
        • chmod 600 ~/.passwd-s3fs
      • mkdir /mnt/bucketName; chmod 755 /mnt/bucketName
      • s3fs bucketName /mnt/bucketName -ouse_cache=/tmp -o allow_other,ahbe_conf=/etc/ahbe.conf
      • with role:
        • s3fs bucketName /mnt/bucketName -o allow_other,ahbe_conf=/etc/ahbe.conf,iam_role=my_rolename
      • mount on boot (FAQ)
        • How to force s3fs mount on boot
          • option 1:
            • /etc/init.d/local
          • option 2:
            • /etc/fstab
              • s3fs#my_bucket /mnt/my_bucket fuse _netdev,nonempty,allow_other,ahbe_conf=/etc/ahbe.conf 0 0
              • s3fs#my_bucket /mnt/my_bucket fuse _netdev,nonempty,allow_other,ahbe_conf=/etc/ahbe.conf,iam_role=my_rolename 0 0
            • Problems:
              • d????????? ? ? ? ?            ? my_bucket
                • Solution
                  • check that _netdev option is present in /etc/fstab
      • public permission for new files:
        • s3fs#my_bucket /mnt/my_bucket fuse _netdev,nonempty,allow_other,ahbe_conf=/etc/ahbe.conf,iam_role=my_rolename,default_acl=public-read 0 0
    • Problemes / Problems
    • Cloudfront Cache-control
      • ahbe.conf
        • sample_ahbe.conf
        • # mpd and m3u8 files are cached for 2 seconds
          .mpd Cache-Control max-age=2
          .m3u8 Cache-Control max-age=1
      • s3fs bucketName /mnt/bucketName -o allow_other,ahbe_conf="/etc/ahbe.conf"

RDS
  • Using RDS with Django
  • Security group
    • Default rule make database available from outside
    • To be reachable from ec2 instances, add a rule for your VPC (e.g. 172.32.0.0/16)

CloudFront

Route53

VPC
  • VPCs and Subnets
  • Differences in CloudFormation when having a particular VPC

    		• default VPC
      		 own VPC

      		default subnet
      		 own subnet own subnet





      AWS::EC2::Subnet
      AWS::EC2::RouteTable
      AWS::EC2::Route
      AWS::EC2::SubnetRouteTableAssociation      		 AWS::EC2::VPC
      AWS::EC2::InternetGateway
      AWS::EC2::VPCGatewayAttachment
      AWS::EC2::Subnet
      AWS::EC2::RouteTable
      AWS::EC2::Route
      AWS::EC2::SubnetRouteTableAssociation
      AWS::EC2::SecurityGroup

      		"VpcId" : {"Ref" : "MyVPC"}
      AWS::EC2::Instance
      		 "SecurityGroups" : [{ "Ref" : "MySecurityGroup" }] "SecurityGroupIds" : [{ "Ref" : "MySecurityGroup" }]
      "SubnetId" : {"Ref" : "MyFirstSubnet"}      		 "SecurityGroupIds" : [{ "Ref" : "MySecurityGroup" }]
      "SubnetId" : {"Ref" : "MyFirstSubnet"}
      AWS::ElasticLoadBalancing::LoadBalancer
      		 "AvailabilityZones" : {"Fn::GetAZs": ""}
      		 "Subnets" : [{"Ref" : "MyFirstSubnet"}] "Subnets" : [{"Ref" : "MyFirstSubnet"}]
      AWS::AutoScaling::AutoScalingGroup
      		 "AvailabilityZones" : {"Fn::GetAZs": ""} "VPCZoneIdentifier" : [{"Ref" : "MyFirstSubnet"}] "VPCZoneIdentifier" : [{"Ref" : "MyFirstSubnet"}]
  • Multicast
    • Overlay Multicast in Amazon Virtual Private Cloud
      • Info
        • Get all instances with a multicast tag, and filter those within a specific community (e.g.: foo)
          • aws --output json ec2 describe-instances --filters "Name=tag-key,Values=multicast" >instances_multicast.json
          • jq '.Reservations[].Instances[] | select( .Tags[] | . and .Key=="multicast" and (.Value | startswith("foo")) )' instances_multicast.json
          • from the selected instances, get only some information:
            • jq '.Reservations[].Instances[] | select( .Tags[] | . and .Key=="multicast" and (.Value | startswith("foo")) ) | [.InstanceId, .PrivateIpAddress, .PublicIpAddress, .Tags]' instances_multicast.json
      • Setup
        • Setup step 1: Create a subnet
          • Option 1: just create a subnet in a existing VPC
          • Option 2: Create new AWS VPC (vpc-xxxxxx) with a subnet and a route table to Internet:
            • Subnet
              • Name: Public subnet
              • IPv4 CIDR: 10.0.0.0/24
            • Cloudformation
              • {
                    "Resources": {
                        "MyVPC": {
                            "Type" : "AWS::EC2::VPC",
                            "Properties" : {
                                "CidrBlock": "10.0.0.0/16",
                                "EnableDnsSupport" : "true",
                                "EnableDnsHostnames" : "true",
                                "Tags" :[ { "Key" : "Name", "Value" : "my-vpc"} } ]
                            }
                        },

                        "MyInternetGateway" : {
                            "Type" : "AWS::EC2::InternetGateway"
                        },

                        "MyVPCGatewayAttachment" : {
                            "Type" : "AWS::EC2::VPCGatewayAttachment",
                            "Properties" : {
                                "InternetGatewayId" : {"Ref" : "MyInternetGateway"},
                                "VpcId" : {"Ref" : "MyVPC"}
                            }
                        },

                        "MyPublicSubnet" : {
                            "Type" : "AWS::EC2::Subnet",
                            "Properties" : {
                                "VpcId" : { "Ref" : "MyVPC" },
                                "CidrBlock" : "10.0.0.0/24",
                                "MapPublicIpOnLaunch" : "true",
                                "Tags" : [ { "Key" : "Name", "Value" : "my-subnet"} ]
                            }
                        },
                       
                        "PublicRouteTable" : {
                            "Type" : "AWS::EC2::RouteTable",
                            "Properties" : {
                                "VpcId" : {"Ref" : "MyVPC"}
                            }
                        },
                       
                        "PublicRoute" : {
                            "Type" : "AWS::EC2::Route",
                            "DependsOn" : "MyVPCGatewayAttachment",
                            "Properties"  : {
                                "RouteTableId" : {"Ref" : "PublicRouteTable"},
                                "DestinationCidrBlock" : "0.0.0.0/0",
                                "GatewayId" : {"Ref" : "MyInternetGateway"}
                            }
                        },
                       
                        "PublicSubnetRouteTableAssociation" : {
                            "Type" : "AWS::EC2::SubnetRouteTableAssociation",
                            "Properties" : {
                                "SubnetId" : {"Ref" : "MyPublicSubnet"},
                                "RouteTableId" : {"Ref" : "PublicRouteTable"}
                            }
                        },

                    }
                }
        • Setup step 2: Create AWS role with policy:
          • {
                "Version": "2012-10-17",
                "Statement": [
                    {
                        "Sid": "Stmt1414071732000",
                        "Effect": "Allow",
                        "Action": [
                            "ec2:DescribeInstances",
                            "ec2:DescribeTags",
                            "ec2:DescribeRegions"
                        ],
                        "Resource": [
                            "*"
                        ]
                    }
                ]
            }
          • Cloudformation:
            • ...
        • Setup step 3: Create AWS security group (sg-yyyyyy) in your VPC (vpc-xxxxxx)
          • Inbound Rules:
            • Type: Custom Protocol Rule
            • Protocol: GRE (47)
            • Port Range: All
            • Source: sg-yyyyyy
          • Cloudformation (to avoid circular dependency, a AWS::EC2::SecurityGroupIngress must be created)
            •     "MyInboundRule" : {
                      "Type": "AWS::EC2::SecurityGroupIngress",
                      "Properties":{
                      "IpProtocol" : "47",
                      "SourceSecurityGroupId" : {"Ref" : "MySecurityGroup"},
                      "GroupId" : {"Ref" : "MySecurityGroup"}
                      }
                  },
                 
                  "MySecurityGroup" : {
                      "Type" : "AWS::EC2::SecurityGroup",
                      "Properties" : {
                          "GroupDescription" : "Enable ports 22 (ssh), GRE (47) (multicast)",
                          "VpcId" : {"Ref" : "MyVPC"},
                          "SecurityGroupIngress" : [
                              {
                                  "IpProtocol" : "tcp",
                                  "FromPort" : "22",
                                  "ToPort" : "22",
                                  "CidrIp" : "0.0.0.0/0"
                              }
                          ]
                      }
                  },
        • Setup step 4: Creation of several instances with this role and security group, with tag:
          • Name: multicast; Value: foo,172.16.0.7/24
          • Name: multicast; Value: foo,172.16.0.8/24
          • ...
      • Installation
        • Option 1:
          • Installation of Ruby script
            • CentOS
              • sudo yum install unzip bridge-utils ebtables curl ruby ruby-devel rubygem-nokogiri rubygem-daemons libxml2-devel
              • sudo gem install aws-sdk-v1 integration libxml-ruby
            • cd
            • wget https://s3.amazonaws.com/mcd-code/mcd-code-2014-07-11.zip
            • cd mcd-code-2014-07-11
            • sudo chmod 755 *
            • sudo mkdir -p /opt/mcast
            • sudo cp -pr * /opt/mcast'
            • sudo chown -R root:root /opt/mcast
          • Start Ruby script
            • temporarily
              • sudo ruby -d /opt/mcast/mcd
            • daemon
              • CentOS
                • mcd.service
                  • [Unit]
                    Description=Multicast daemon for AWS EC2 instances
                    After=syslog.target network.target cloud-init.service

                    [Service]
                    Type=simple
                    #PIDFile=/run/mcd.pid
                    ExecStartPre=/usr/local/bin/mcd_setup.sh foo 172.16.0.0/24
                    ExecStart=/opt/mcast/mcd                     ExecStartPost=/usr/local/bin/mcd_setup_route.sh foo
                    ExecReload=/bin/kill -s HUP $MAINPID
                    ExecStop=/bin/kill -s QUIT $MAINPID

                    [Install]
                    WantedBy=multi-user.target
                • mcd_setup_route.sh
                  • #!/bin/bash

                    function print_help_and_exit {
                        cat <<EOF
                    Usage: `basename $0` multicast_name

                    Add route for multicast: mcbr-<multicast_name>

                    Examples:
                    - `basename $0` foo

                    EOF
                        exit 1
                    }

                    MIN_ARGS=1
                    MAX_ARGS=1
                    if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
                    then
                        print_help_and_exit
                    fi

                    # options
                    if ! params=$(getopt -o h --long help -n $0 -- "$@")
                    then
                        # invalid option
                        print_help_and_exit
                    fi
                    eval set -- ${params}

                    while true
                    do
                        case "$1" in
                            -h | --help ) print_help_and_exit;;
                            -- ) shift; break ;;
                            * ) break ;;
                        esac
                    done

                    # parameters
                    multicast_name=$1

                    # wait for bridge to exist
                    bridge_name="mcbr-${multicast_name}"

                    timeout=60
                    increment=5
                    t=0

                    while (( t < timeout )) && (brctl show ${bridge_name} 2>&1 1>/dev/null | grep -q "No such device")
                    do
                        echo "[`basename $0`] bridge ${bridge_name} is not available yet (${t}s/${timeout}s)"
                        (( t+=increment ))
                        sleep ${increment}
                    done

                    # add route for multicast
                    echo "[`basename $0`] adding route for multicast: mcbr-${multicast_name}"
                    route add -net 224.0.0.0/4 mcbr-${multicast_name}

                    exit 0
                • mcd_setup.sh
                  • #!/bin/bash

                    function print_help_and_exit {
                        cat <<EOF
                    Usage: `basename $0` multicast_name multicast_cidr
                    Set the AWS EC2 tag: "multicast", "<multicast_name>,<multicast_cidr>"
                    Address multicast_cidr has the same number as local ip address, but converted to specified subnet
                    E.g.: if local address is 10.1.2.3/24 and you specify 172.16.0.0/24, the multicast_cidr=172.16.0.3/24
                    IMPORTANT: role for this ec2 instance must include a policy with: "ec2:CreateTags"

                    Examples:
                    - `basename $0` foo 172.16.0.0/24
                    EOF
                        exit 1
                    }

                    MIN_ARGS=2
                    MAX_ARGS=2
                    if (( $# < $MIN_ARGS )) || (( $# > $MAX_ARGS ))
                    then
                        print_help_and_exit
                    fi

                    # options
                    if ! params=$(getopt -o h --long help -n $0 -- "$@")
                    then
                        # invalid option
                        print_help_and_exit
                    fi
                    eval set -- ${params}

                    while true
                    do
                        case "$1" in
                            -h | --help ) print_help_and_exit;;
                            -- ) shift; break ;;
                            * ) break ;;
                        esac
                    done

                    # parameters
                    multicast_name=$1
                    multicast_cidr=$2

                    function translate_ip {
                        input_cidr=$1
                        output_cidr=$2

                        # remove subnet
                        input_address=${input_cidr%/*}

                        # get network and prefix
                        eval $(ipcalc -np $output_cidr)
                        output_prefix=$PREFIX
                        output_network=$NETWORK

                        # calculate number of bytes (n)
                        let output_positions=${output_prefix}/8

                        # remove first n bytes
                        input_array=(${input_address//./ })
                        input_significative=${input_array[@]:${output_positions}}

                        # get first n bytes
                        output_array=(${output_network//./ })
                        output_significative=${output_array[@]:0:${output_positions}}
                       
                        # join all bytes
                        total_address_array=(${output_significative[@]} ${input_significative[@]})
                        total_address=$(IFS='.';echo "${total_address_array[*]}";IFS=$'')
                        total_cidr="${total_address}/${output_prefix}"
                        echo $total_cidr
                    }

                    # check whether ip command is available
                    if ! (which ip >/dev/null 2>&1)
                    then
                        echo "ERROR: command ip not found. Consider running this script as root or sudo."
                        exit 1
                    fi

                    # get own information
                    local_cidr=$(ip -o address | awk '$2 !~ /lo/ && $3 ~ /^inet$/ {print $4; exit;}')
                    #local_ipv4=$(curl http://169.254.169.254/latest/meta-data/local-ipv4/)

                    multicast_cidr=$(translate_ip $local_cidr $multicast_cidr)

                    echo multicast_cidr: $multicast_cidr

                    # eu-west-1c
                    if ! aws_subregion=$(curl -s -m 4 http://169.254.169.254/latest/meta-data/placement/availability-zone)
                    then
                        echo "no aws_subregion found. Are you sure that you are running this script on an AWS instance?"
                        exit 1
                    fi
                    # eu-west-1
                    aws_region=${aws_subregion: : -1}

                    instance_id=$(curl -s -m 4 http://169.254.169.254/latest/meta-data/instance-id)

                    # create a tag
                    aws ec2 create-tags --region ${aws_region} --resources $instance_id --tags Key="multicast",Value="${multicast_name}\,${multicast_cidr}"

                    exit 0
        • Option 2: Install bash script
          • Installation of mcd.sh
          • Start bash script
            • temporarily
            • daemon
              • CentOS
      • Check
        • Process
          • sudo ps -edalf | grep mcd
        • Logs
          • tail -f /var/log/messages | grep mcd
        • Created bridges
          • brctl show
        • Created GRE tunnels
          • ip link show
        • Routes
          • route -n
        • Members
          • netstat -g
        • Omping
          • from each instance:
            • omping 172.16.0.7 172.16.0.8 ...
          • if it does not work:
            • check if some gretap points to a non existing address
              • to get the remote address, check the logs in /var/log/messages when the gretap was created
      • Ús / Usage
        • routes
          • in each instance:
            • route add -net 224.0.0.0/4 mcbr-foo
        • application:
          • from one instance:
            • ffmpeg ...
          • from the other one:
            • ...

Big data and analytics

CLI
  • AWS Command Line Interface
  • Instal·lació / Installation (python)
  • Ús / Usage
    • aws --version ...
  • Problemes / Problems
    • ImportError: No module named history
      • the problem is that a pair of awscli (1.14.28) and botocore (1.6.0), installed from yum (awscli.noarch), does not work
      • Solucions / Solutions
        • Swap s3transfer packages (awscli-1.14.28-5):
          • yum swap python2-s3transfer python-s3transfer
        • Use pip
          • sudo pip install awscli
            • a pair of working packages is, e.g.: awscli==1.14.28, botocore==1.8.32
  • queries
  • templates
    • --generate-cli-skeleton > skeleton.json
    • --cli-input-json file://skeleton.json
  • stdin/stdout
    • aws s3 cp s3://bucket/key - | bzip2 -best | aws s3 cp - s3://bucket/key.bz2
  • CLI reference
    • general options
      • --region eu-west-1
      • --profile myprofile
      • --output json
      • ...
    • aws autoscaling
      • launch configuration
      • autoscaling group
        • aws autoscaling create-auto-scaling-group --auto-scaling-group-name grup_stateless --launch-configuration-name lc_stateless_yyyymmdd_hhmm --min-size 1 --max-size 2 --load-balancer-names lb-stateless
        • aws autoscaling update-auto-scaling-group --auto-scaling-group-name grup_stateless --launch-configuration-name lc_stateless_yyyymmdd_hhmm
        • number of instances inside an autoscaling group, given its name
          • result=$(aws --output json autoscaling describe-auto-scaling-groups --auto-scaling-group-names $asg_name)
            number_instances=$(echo $result | jq '.AutoScalingGroups[0].Instances | length')
        • get autoscaling group, given its tag Name=myname:
          • asg_info=$(aws --output json autoscaling describe-auto-scaling-groups)
            asg_name_tag_value="myname"
            asg=$(echo "$asg_info" | jq ".AutoScalingGroups[] | select( .Tags[] | . and .Key==\"Name\" and .Value==\"${asg_name_tag_value}\")")
        • aws autoscaling set-desired-capacity --auto-scaling-group-name $asg_name --desired-capacity 2
        • Instance protection
          • aws autoscaling set-instance-protection --instance-ids i-93633f9b --auto-scaling-group-name my-auto-scaling-group --protected-from-scale-in
        • given an instance id, get the autoscaling group name it belongs to:
          • aws ec2 describe-tags --filters "Name=resource-id,Values=$instance_id" "Name=key,Values=aws:autoscaling:groupName" | jq '.Tags[] | .Value'
      • get all autoscaling groups (with pagination)
        • aws_options="--profile my_profile --output json --region eu-west-1"
          next_token=""
          max_items=50
          page_size=100
          total_number=0
          total_elements=$(jq -n '[]')

          while [[ $next_token != "null" ]]
          do
              if [[ "$next_token" ]]
              then
                  lc_info=$(aws ${aws_options} autoscaling describe-launch-configurations --max-items $max_items --page-size $page_size --starting-token $next_token)
              else
                  lc_info=$(aws ${aws_options} autoscaling describe-launch-configurations --max-items $max_items --page-size $page_size)
              fi
              #echo $lc_info | jq '.'
              returned_number=$(echo $lc_info | jq  '.LaunchConfigurations | length' )
              returned_elements=$(echo $lc_info | jq  '.LaunchConfigurations')
              echo $returned_elements | jq '.'

              total_elements=$( echo $total_elements | jq ". += $returned_elements")
              echo "returned_number: $returned_number"
              total_number=$(( total + returned_number ))
              echo "total_number: $total_number"
              echo $lc_info | jq  '.LaunchConfigurations[] | ([.LaunchConfigurationName] | join (" "))'

              next_token=$(echo $lc_info | jq '.NextToken')
              echo "next_token: $next_token"
             
          done


          echo "---------------------------"
          echo $total_elements | jq '.[].LaunchConfigurationName'
    • aws cloudformation
    • aws cloudfront
      • aws configure set preview.cloudfront true
      • list all distributions
        • aws cloudfront list-distributions --output json
      • get a specific distribution:
        • aws cloudfront get-distribution --id E1KBXTVP599T0A
      • get the config for a specific distribution
        • aws cloudfront get-distribution-config --id E1KBXTVP599T0A
      • update distribution
        • aws cloudfront get-distribution-config --id ${cloudfront_id} --output json > /tmp/${cloudfront_id}.json

          # get etag
          etag=$(jq -r '.ETag' /tmp/${cloudfront_id}.json)

          # modify           /tmp/${cloudfront_id}.json
          ...

          aws cloudfront update-distribution --id ${cloudfront_id} --if-match $etag --distribution-config $(jq -c '.DistributionConfig' /tmp/${cloudfront_id}.json) 
    • aws configure
      • boto3 credentials
      • generated files
        • ~/.aws/credentials (for all SDKs)
        • ~/.aws/config (only for CLI)
      • crearà / will create: ~/.aws/config
        • [default]
          output = json
          region = eu-west-1           aws_access_key_id = xxx
          aws_secret_access_key = yyy
          [preview]
          cloudfront = true
      • i/and ~/.aws/credentials (? now included in ~/.aws/config)
        • [default]
          aws_access_key_id = xxx
          aws_secret_access_key = yyy
      • per a fer servir un altre perfil / to use another profile
        • aws --profile myprofile configure
        • will create ~/.aws/config
          • [profile myprofile]
            output = json
            region = eu-west-1
        • and ~/.aws/credentials
          • [myprofile] aws_access_key_id = xxx
            aws_secret_access_key = yyy
        • aws --profile myprofile ...
      • per a fer servir un altre fitxer de configuració / to use an alternate config file (e.g. /etc/aws/config):
    • aws efs
      • file system
        • name="fs-toto"
          creation_token=$(           openssl rand -hex 10)
          response=$(aws efs create-file-system --creation-token ${creation_token} --tags Key=Name,Value=${name})
          file_system_id=$(echo ${response} | jq -r '.FileSystemId')
          echo ${file_system_id}
      • mount targets
        • create a security group for NFS (port 2049)
          • group_name="sgroup-nfs-toto"
            vpc_id="vpc-..."
            description="Enable port 2049 (nfs)"
            response=$(aws ec2 create-security-group --group-name ${group_name} --vpc-id ${vpc_id} --description "${description}")
            group_id=$(echo ${response} | jq -r '.GroupId')
            echo ${group_id}
          • protocol=tcp
            port=2049
            cidr="" # cidr of the subnet inside the vpc
            aws ec2 authorize-security-group-ingress --group-id ${group_id} --protocol ${protocol} --port ${port} --cidr "${cidr}"
        • create a mount target
          • subnet_id="subnet-..."
            response=$(aws efs create-mount-target --file-system-id ${file_system_id} --subnet-id ${subnet_id} --security-groups ${group_id})
            mount_target_id=$(echo ${response} | jq -r '.MountTargetId')             echo ${mount_target_id}
        • get info about a mount target
          • response=$(aws efs describe-mount-targets --mount-target-id ${mount_target_id})
      • mount efs from an instance
    • aws ec2
      • instances
        • aws ec2 start-instances --instance-ids i-9b789ed8
        • aws ec2 describe-instances --filter Name=tag:Name,Values=ubuntu_1310
          • ...
            TAGS    Name    ubuntu_1310
          • get PublicIpAddress
            • aws --output json ec2 describe-instances --instance-ids i-9b789ed8 | jq -r '.Reservations[0].Instances[0].PublicIpAddress'
        • aws ec2 describe-instance-attribute --instance-id i-896b92c9 --attribute instanceType
        • aws ec2 stop-instances --instance-ids i-9b789ed8
        • aws ec2 terminate-instances --instance-ids i-9b789ed8
        • waiters
          • aws ec2 wait instance-running --instance-id i-896b92c9
          • aws ec2 wait instance-status-ok --instance-id i-896b92c9
          • aws ec2 wait instance-stopped --instance-id i-896b92c9
      • images
        • create an image:
          • instance_id=i-xxxxxxx
            image_prefix=image_u1404
            data=$(date '+%Y%m%d_%H%M')
            imatge_id=$(aws ec2 create-image --instance-id ${instance_id} --name "${image_prefix}_${data}" --description "My description")
            echo "${imatge_id} ${image_prefix}_${data} (from ${instance_id})"
        • create an image but root volume (/dev/sda1) will be destroyed on termination (useful when this image will be used in a launch configuration of an autoscaling group)
          • instance_id=i-xxxxxxx
            image_prefix=image_u1404
            data=$(date '+%Y%m%d_%H%M')
            imatge_id=$(aws ec2 create-image --instance-id ${instance_id} --name "${image_prefix}_${data}" --description "My description" --block-device-mappings "[{\"DeviceName\": \"/dev/sda1\",\"Ebs\":{\"VolumeType\":\"gp2\",\"DeleteOnTermination\":true}}]")
            echo "${imatge_id} ${image_prefix}_${data} (from ${instance_id})"
        • describe images:
          • get all own images
            • aws ec2 describe-images --owners self
            • aws_options="--profile my_profile --output json --region eu-west-1"
              ami_info=$(aws ${aws_options} ec2 describe-images --owners self)
              echo $ami_info | jq  '.Images | sort_by(.Name) | .[] | [.ImageId, .OwnerId, .Name] | join("  ")'
          • describe an image given its id
          • get an image by name:
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json
          • using wildcards:
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json
          • get image_id of an image with given its name:
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json | awk '/ImageId/ {print $2}' | tr -d '",'
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_name" --output json | jq -r '.Images[].ImageId'
          • get a list of amis, sorted by creation date
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json | jq -r '(.Images | sort_by(.CreationDate) | .[] | [.CreationDate, .ImageId] | join(" ") )'
          • get image_id of the most recent image:
            • aws ec2 describe-images --owners self --filters "Name=name,Values=my_image_basename*" --output json | jq -r '(.Images | sort_by(.CreationDate) | .[-1] | .ImageId )'
          • aws ec2 describe-image-attribute --image-id ami-6be62b1c --attribute description
        • waiters
      • snapshots
      • add tags to an:
        • instance:
          • aws ec2 create-tags --resources i-9b789ed8 --tags Key=Name,Value=ubuntu_1404
        • image:
          • aws ec2 create-tags --resources ami-6be62b1c --tags Key=Name,Value=image_ubuntu_1404
          • value="..."
            # escape commas and remove single quotes
            escaped_value=$(echo ${value//,/\\,} | tr -d "'")
            aws ec2 create-tags --resources ami-6be62b1c --tags Key=Name,Value="${escaped_value}"
      • get tags from an:
        • instance
          • aws ec2 describe-tags --filters "Name=resource-id,Values=i-1234567890abcdef8"
      • create an instance from an image:
        • aws ec2 run-instances --image-id ami-6be62b1c --security-groups launch-wizard-2 --count 1 --key-name my_keyname --placement AvailabilityZone='eu-west-1a',Tenancy='default' --instance-type t2.micro
        • get the instance_id:
          • reponse in json (parse with jq)
            • instance_description=$(aws --output json ec2 run-instances --image-id $image_id --security-groups $security_group_name --iam-instance-profile Name="role-nfs_server" --count 1 --key-name $key_name --placement AvailabilityZone=${availability_zone},Tenancy='default' --instance-type $instance_type --block-device-mappings 'DeviceName=/dev/sda1,Ebs={DeleteOnTermination=true,VolumeType=gp2}' )
            • instance_id=$(echo $instance_description | jq -r '.Instances[0].InstanceId')
          • response in text (parse with awk)
            • instance_id=$(aws --output text ec2 run-instances --image-id ami-6be62b1c --security-groups launch-wizard-2 --count 1 --key-name my_keyname --placement AvailabilityZone='eu-west-1a',Tenancy='default' --instance-type t2.micro | awk '/INSTANCES/ {print $8}')
            • aws ec2 describe-instances --instance-ids $instance_id
        • overwrite the delete on termination behaviour:
          • aws ec2 run-instances --image-id ami-6be62b1c ... --block-device-mappings 'DeviceName=/dev/sda1,Ebs={DeleteOnTermination=true,VolumeType=gp2}'
        • ...
          • dades d'usuari /user data:
            • when creating instance:
            • des de la instància / from ec2 instance:
              • curl http://169.254.169.254/latest/user-data/
            • ec2-run-user-data (ec2ubuntu) (already installed in EC2 Ubuntu ami)
        • assign a role to the instance
        • Problems:
          • Client.InvalidParameterCombination: Could not create volume with size 10GiB and iops 30 from snapshot 'snap-xxxxx'
            • AutoScaling - Client.InvalidParameterCombination
            • Re: Stabilization Error (Again)
            • "Iops" should not be there (?)
            • Workaround: create image from web interface instead
            • Solution: add --block-device-mappings option (following example is for 10GiB volume)
              • aws ec2 create-image --instance-id i-xxxxxxxx --name AMIName --block-device-mappings '[{"DeviceName":"/dev/sda1","Ebs":{"VolumeType":"gp2","DeleteOnTermination":"true","VolumeSize":10}}]'
      • volumes
        • create a volume
          • volume_description=$(aws ec2 create-volume --availability-zone $availability_zone --volume-type $volume_type --size $small_volume_size_gibytes)
          • volume_id=$(echo $volume_description | jq '.VolumeId')
        • wait for volume to be created
        • tag a volume with a name
          • aws ec2 create-tags --resources $volume_id --tags Key=Name,Value=$volume_name
        • describe a volume with specified name:
        • get availability zone of a volume
          • aws_cli_options="--profile my_profile --output json"
            volume_name=my-volume-name
            volume_description=$(aws $aws_cli_options ec2 describe-volumes --filters "Name=tag:Name,Values=$volume_name")
            availability_zone=$(echo $volume_description | jq -r '.Volumes[0].AvailabilityZone')
        • attach a volume to an instance
          • aws ec2 attach-volume --volume-id $volume_id --instance-id $instance_id --device /dev/sd${volume_letter}
        • detach volume
        • list all volumes
          • response=$(aws --output json ec2 describe-volumes)

            while IFS= read -r; do
                volume=$REPLY
                volume_id=$(echo $volume | jq -r '.VolumeId')
                echo "---  VolumeId: $volume_id"

                #tags=$(echo $volume | jq -c -r '(.Tags | values | .[] | select(.Key == "Name") )')
                tags=$(echo $volume | jq -c -r '(.Tags | values | .[] )')
                for tag in $tags
                do
                    tag_key=$(echo $tag | jq '.Key')
                    tag_value=$(echo $tag | jq '.Value')
                    echo "    $tag_key: $tag_value"
                done
            done < <(echo "$response" | jq -c -r '.[] | .[]')
    • aws elb
      • create a load balancer and associate to an instance:
        • aws elb create-load-balancer --load-balancer-name lb-stateful --listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=80 --availability-zones eu-west-1a eu-west-1b eu-west-1c --security-groups sg-ba33c2df
        • aws elb configure-health-check --load-balancer-name lb-stateful --health-check Target=TCP:80,Interval=30,Timeout=10,UnhealthyThreshold=2,HealthyThreshold=2
        • aws elb register-instances-with-load-balancer --load-balancer-name lb-stateful --instances i-9b789ed8
      • create a load balancer to be associated to an autoscaling group:
        • aws elb create-load-balancer --load-balancer-name lb-stateless --listeners Protocol=TCP,LoadBalancerPort=1935,InstanceProtocol=TCP,InstancePort=1935 Protocol=HTTP,LoadBalancerPort=8080,InstanceProtocol=HTTP,InstancePort=8080 --availability-zones eu-west-1a eu-west-1b eu-west-1c --security-groups sg-ba33c2df
        • aws elb configure-health-check --load-balancer-name lb-stateless --health-check Target=TCP:8080,Interval=30,Timeout=10,UnhealthyThreshold=2,HealthyThreshold=2
      • add an HTTPS listener with ARN of an uploaded IAM certificate
        • aws --region eu-west-1 elb create-load-balancer-listeners --load-balancer-name $load_balancer_name --listeners Protocol=HTTPS,LoadBalancerPort=443,InstanceProtocol=HTTPS,InstancePort=443,SSLCertificateId=$ARN
      • modify the certificate of an existing listener for a given port:
    • aws iam
      • certificats de servidor / server certificates
        • upload a certificate
          • obtained e.g. from Letsencrypt
            • letsencrypt_dirname=/etc/letsencrypt
              aws iam upload-server-certificate --server-certificate-name cert-${domain} \
                  --certificate-body file://${letsencrypt_dirname}/live/${domain}/cert.pem \
                  --private-key file://${letsencrypt_dirname}/live/${domain}/privkey.pem \
                  --certificate-chain file://${letsencrypt_dirname}/live/${domain}/chain.pem
          • self-signed, to be used in cloudfront:
            • openssl req -new -nodes -keyout www.toto.org.key -sha256 -x509 -days 365 -out www.toto.org.crt
              • Common Name: www.toto.org
            • aws iam upload-server-certificate \
                  --server-certificate-name cert-www.toto.org \
                  --certificate-body file://.../              www.toto.org.crt \
                  --private-key file://.../              www.toto.org.key \
                                --certificate-chain file://.../www.toto.org.crt \
                  --path /cloudfront/
        • get a list of server certificates
        • get ARN of a certificate (will be specified when adding a listener to a ELB)
          • ARN=$(aws --output json iam get-server-certificate --server-certificate-name ${domain}.cert | jq '.ServerCertificate.ServerCertificateMetadata.Arn')
        • check if a certificate is available
          • ...
    • aws route53
      • Adding EC2 instances to Route53 (bash+boto)
      • aws route53 list-resource-record-sets --hosted-zone-id xxxxxx
      • aws route53 change-resource-record-sets --hosted-zone-id xxxxxx --change-batch file:///absolute_path_to/change_entry.json
        • change_entry.json (to modify record www.toto.org; e.g. TTL value)
          • {
              "Comment": "Modifying TTL to 55",
              "Changes": [
                {
                  "Action": "UPSERT",
                  "ResourceRecordSet": {
                    "Name": "www.toto.org.",
                    "Type": "A",
                    "TTL": 55,
                    "ResourceRecords": [
                      {
                        "Value": "xx.xx.xx.xx"
                      }
                    ]
                  }
                }
              ]
            }
      • Auto configuration of Route53 from EC2 at boot (Ubuntu Upstart)
        • previously, from any computer:
          • option 1 (preferred): create a role and assign it to the instance when launching it
            • ...
          • option 2: create a user
            • create a user and group that can only modify Route53 entries:
              • from web interface:
                • Grup
                  • Group name
                    • grup_nomes_route53
                  • Policy
                    • {
                        "Version": "2012-10-17",
                        "Statement": [
                          {
                            "Effect": "Allow",
                            "Action": [
                              "route53:*"
                            ],
                            "Resource": [
                              "*"
                            ]
                          }
                        ]
                      }
              • alternatively, from aws cli:
                • ...
        • once logged into EC2 instance:
          • only if used option 2 (not using a role):
          • /usr/local/bin/route53.sh
            • #!/bin/bash

              # zone_id for               mydomain.org
              ZONE_ID=$(cat zone_id.txt)
              ip_address=$(curl http://169.254.169.254/latest/meta-data/public-ipv4)
              name=www.mydomain.org
              tmp_file=/tmp/modify_record_set.json
              rm -f $tmp_file

              #aws route53 list-resource-record-sets --hosted-zone-id $ZONE_ID

              cat > $tmp_file <<EOF
              {
                "Comment": "Modifying ip address",
                "Changes": [
                  {
                    "Action": "UPSERT",
                    "ResourceRecordSet": {
                      "Name": "${name}.",
                      "Type": "A",
                      "TTL": 60,
                      "ResourceRecords": [
                        {
                          "Value": "$ip_address"
                        }
                      ]
                    }
                  }
                ]
              }
              EOF
              #source /opt/p27/bin/activate
              aws route53 change-resource-record-sets --hosted-zone-id $ZONE_ID --change-batch file://$tmp_file
              #deactivate
              exit 0
          • init script
            • CentOS
              • /etc/systemd/system/route53.service
                • [Unit]
                  Description=Description of my script
                  After=syslog.target network.target

                  [Service]
                  Type=oneshot
                  ExecStart=/usr/local/bin/route53.sh

                  [Install]
                  WantedBy=multi-user.target
              • sudo systemctl enable route53.service
              • sudo systemctl start route53.service
            • Ubuntu
              • /etc/init/route53.conf
                • description "route53 daemon"
                   
                  start on (filesystem and net-device-up IFACE=lo)
                  stop on runlevel [!2345]
                   
                  env DAEMON=/usr/local/bin/route53.sh
                  env PID=/var/run/route53.pid

                  env AWS_CONFIG_FILE=/home/ubuntu/.aws/credentials

                  exec $DAEMON
    • aws s3
      • Note: no need to create dirs: they do not exist
      • list all buckets
        • aws s3 ls
      • list all files in a "directory":
        • aws s3 ls --recursive s3://my_bucket/my_dir1/
      • copy a single file to S3:
        • aws s3 cp toto.txt s3://my_bucket/
      • recursively copy to S3:
        • aws s3 cp --recursive . s3://my_bucket/
      • recursively copy from S3:
        • aws s3 cp --recursive s3://my_bucket/ .
      • sync
        • aws s3 sync ... --exclude '*.png' --exclude 'log' ...
        • Problemes / Problems

Boto (Python)
  • Instal·lació / Installation
    • v3
      • pip install boto3
    • v2
      • pip install boto
      • Alternative: get it from git and install it:
        • cd ~/src
        • git clone https://github.com/boto/boto.git
        • cd boto
        • [source /opt/PYTHON27/bin/activate]
        • python setup.py install
  • Credencials / Credentials
    • aws configure
    • ~/.boto
      • [Credentials]
        aws_access_key_id = xxxx
        aws_secret_access_key = yyyy
    • Django
      • if instance running Django has no IAM role, and Django uses boto3 (e.g. django-storages), the credentials must be set at settings.py
      • settings.py
        • AWS_ACCESS_KEY_ID = '...'
          AWS_SECRET_ACCESS_KEY = '...'
    • Celery
      • in an instance running Celery / Django without IAM role, if a process called by celery uses boto3 (e.g. django-storages), variables defined in Django settings.py are not set. They must explicitly be set at celery.conf (called as EnvironmentFile by celery.service)
      • celery.conf
        • AWS_ACCESS_KEY_ID = '...'
          AWS_SECRET_ACCESS_KEY = '...'
  • Docs
  • Problemes / Problems
  • Usage
    • Amazon EC2 Basics For Python Programmers
    • Difference in boto3 between resource, client, and session?

    		•

      		examples
      		 default session
      		 specific session



      		Amazon S3 Examples

      Session
      • stores configuration information (primarily credentials and selected region)
      • allows you to create service clients and resources
      • boto3 creates a default session for you when needed

      		-
      		 session = boto3.Session(profile_name='dev')
      Resource
      • higher-level, object-oriented API
      • generated from resource description
      • uses identifiers and attributes
      • has actions (operations on resources)
      • exposes subresources and collections of AWS resources
      • does not provide 100% API coverage of AWS services
      		 creation
      		s3_resource = boto3.resource('s3') ...
      methods
      		s3_resource.create_bucket(Bucket='mybucket')
      Client
      • low-level AWS service access
      • generated from AWS service description
      • exposes botocore client to the developer
      • typically maps 1:1 with the AWS service API
      • all AWS service operations are supported by clients
      • snake-cased method names (e.g. ListBuckets API => list_buckets method)
      		 creation
      		 from resource
      		 s3_client = s3_resource.meta.client
      from session
      		 s3_client = boto3.client('s3')
      		 s3_client = session.client('s3')

      		methods
      		 Creating and Using Amazon S3 Buckets
      • s3_client.list_buckets()
      • s3_client.create_bucket(Bucket='mybucket')
      • s3_client.upload_file()

      		...

      		Using an Amazon S3 Bucket as a Static Web Host
    • autoscaling
    • s3
    • cloudformation
      • boto.cloudformation
      • Examples with boto3:
        • ...
      • Examples with boto v2:
        • single EC2 instance
          • single_ec2.json
          • single_ec2.py
            • import boto.cloudformation
              from django.conf import settings
              ...

                  try:
                      conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )
                      stack = conn.create_stack(self.name,
                          template_body=template_body,
                          template_url=None,
                          parameters=[],
                          notification_arns=[],
                          disable_rollback=False,
                          timeout_in_minutes=None,
                          capabilities=None)
        • single EC2 entry with Route53
          • single_ec2_r53.json
          • single_ec2_r53.py
            •         try:
                          # connect to the cloud and create the stack
                         
                          # connect to AWS
                          conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )

                          # check if the stack already exists
                          existing_stacks = [s.stack_name for s in conn.describe_stacks()]
                          logger.debug("   Existing stacks: %s" % existing_stacks)
                          if self.name in existing_stacks:
                              logger.error("   Stack %s is already created" % self.name)
                              raise Exception("Stack %s is already created" % self.name)
                         
                          # create the stack
                          conn.create_stack(self.name,
                                            template_body=template_body,
                                            template_url=None,
                                            parameters=[
                                                        ('HostedZone','toto.org'),
                                                        ],
                                            notification_arns=[],
                                            disable_rollback=False,
                                            timeout_in_minutes=None,
                                            capabilities=None)
                         
                          # wait for COMPLETE
                          ready = False
                          while not ready:
                              stacks = conn.describe_stacks(self.name)
                              if len(stacks) == 1:
                                  stack = stacks[0]
                              else:
                                  raise Exception("Stack %s has not been created" % self.name)
                              logger.debug("   stack status: %s" % stack.stack_status)
                              # CREATE_COMPLETE, ROLLBACK_COMPLETE
                              ready = (string.find(stack.stack_status, 'COMPLETE')) != -1
                              time.sleep(5)
                         
                          # get output information
                          outputs = dict()
                          for output in stack.outputs:
                              outputs[output.key] = output.value
                         
                          logger.debug("   DomainName: %s" % outputs['DomainName'])
        • ...
          • my_file.py
            •             # connect to AWS
                          conn = boto.cloudformation.connect_to_region( settings.AWS_DEFAULT_REGION )
                         
                          stacks = conn.describe_stacks(stack_name)
                          if len(stacks) == 1:
                              stack = stacks[0]
                          else:
                              raise Exception("Stack %s does not exist" % stack_name)

                          # CREATE_COMPLETE, ROLLBACK_COMPLETE
                          ready = (string.find(stack.stack_status, 'CREATE_COMPLETE')) != -1
                  
                          if ready:       
                              # get parameters (conversion from ResultSet to dictionary)
                              parameters = {item.key:item.value for item in stack.parameters}
                             
                              # get output information (conversion from ResultSet to dictionary)
                              outputs = {item.key:item.value for item in stack.outputs}
    • cloudfront
      • Create an invalidation:
        •     try:
                  import boto3
              except Exception as e:
                  print 'ERROR: %s' % e

              profile_name = 'my_profile'
              session = boto3.Session(profile_name=profile_name)
              cloudfront_client = session.client('cloudfront')

              distribution_id = 'xxxxxx'
                
              # create invalidation
              import time
              response = cloudfront_client.create_invalidation(
                              DistributionId=distribution_id,
                              InvalidationBatch={
                                  'Paths': {
                                      'Quantity': 1,
                                      'Items': ['/*']
                                  },
                              'CallerReference': str(time.time())
                              }
                          )

              print response
    • autoscaling
      • Examples with boto3
        • Get ids of instances inside an autoscaling group with specified name (using JMESPath):
          • # get the list of instances in the autoscaling group
            client_autoscaling = boto3.client('autoscaling', region_name='eu-west-1')
            paginator = client_autoscaling.get_paginator('describe_auto_scaling_instances')
            page_iterator = paginator.paginate(
                PaginationConfig={'PageSize': 50}
            )
                       
            # ids for instances whose 'AutoScalingGroupName' == asg_name
            # http://boto3.readthedocs.io/en/latest/guide/paginators.html#filtering-results-with-jmespath
            filtered_instances = page_iterator.search(
                'AutoScalingInstances[?AutoScalingGroupName == `{}`]'.format(asg_name)
            )
            instance_ids = [ i['InstanceId'] for i in filtered_instances ]
        • Get all volumes in zones starting with 'eu-'
          • import boto3

            ec2_client = boto3.client('ec2', region_name='eu-west-1')
               
            paginator = ec2_client.get_paginator('describe_volumes')
            page_iterator = paginator.paginate(
                PaginationConfig={'PageSize': 50}
            )
                          
            zone_prefix = 'eu-'
            filtered_volumes = page_iterator.search(
                'Volumes[?starts_with(AvailabilityZone,`{}`)]'.format(zone_prefix)
            )
        • ... and size is greater than 80 (GiB):
          • zone_prefix = 'eu-' filtered_volumes = page_iterator.search(
                'Volumes[?(starts_with(AvailabilityZone,`{}`) && Size>`80`)]'.format(zone_prefix)
            )
        • Have a tag "Name" whose value starts with "my-":
          • prefix = 'my-'
            filtered_volumes = page_iterator.search(
                'Volumes[?(Tags[?Key==`Name`] | [?starts_with(Value,`{}`)])]'.format(prefix)
            )
    • ec2
      • An Introduction to boto’s EC2 interface
      • EC2 (API reference)
      • Amazon EC2 Deployment with Boto
      • yourfile.py
        • conn = boto.ec2.connect_to_region("eu-west-1", aws_access_key_id=settings.AWS_ACCESS_KEY_ID, aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY)
          reservation = conn.run_instances( 'ami-...', security_groups=['launch-wizard'], min_count=1, max_count=1, key_name='parell_key', placement='eu-west-1a', tenancy='default', instance_type='t1.micro')
          instance = reservation.instances[0]
                     
          while instance.state != 'running':
              time.sleep(5)
              instance.update() # Updates Instance metadata
              print "Instance state: %s" % (instance.state)

          # add Name tag
          instance.add_tag("Name","your_instance_name")

          print "Instance ID: %s" % instance.id
          print "Instance IP address: %s" % instance.ip_address
        • # get all the reservations with a given Name tag:
          reservations = conn.get_all_instances(filters={'tag:Name': '          your_instance_name'})
          # get the first reservation
          reservation = reservations[0]
        • # get all the reservations with a given instance_id
          reservations = conn.get_all_instances(instance_ids=["i-27823367"]) # get the first reservation
          reservation = reservations[0]
          instance = reservation.instances[0]
          # get the value of tag "Name"
          name = instance.tags["Name"]
        • # connection
          conn = boto.ec2.autoscale.connect_to_region(settings.AWS_REGION,
                                                      aws_access_key_id=settings.AWS_ACCESS_KEY_ID_EC2,
                                                      aws_secret_access_key=settings.AWS_SECRET_ACCESS_KEY_EC2)
                 
          # get autoscaling group
          asg = conn.get_all_groups(names='my_name')[0]
                 
          # get instances
          instance_ids = [i.instance_id for i in asg.instances]
          print " Instances ID: %s" % (instance_ids)
                 
          # shutdown instances
          asg.shutdown_instances()
                 
          # wait for all instances to be shutdown
          instances = True
          while instances:
              time.sleep(5)
              asg = conn.get_all_groups('my_name')[0]
              if not asg.instances:
                  instances = False
              else:
                  logger.debug(" still some instances in group %s"%self.name)
                 
          # remove group
          asg.delete()
  • zip

Mobile

CloudWatch
  • Log
    • Python
      • WatchTower
    • system logs
      • awslogs
      • Instal·lació / Installation
      • Configuració / Setup
        • Upload the CloudWatch Agent Configuration File to Systems Manager Parameter Store
        • /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
          • [credentials]
            [proxy]
            [region]
        • Manually Create or Edit the CloudWatch Agent Configuration File
          • /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json


            • agent
              • metrics_collection_interval
              • region
              • credentials
                • role_arn
              • debug
              • logfile
              metrics
              • namespace
              • append_dimensions
              • aggregation_dimensions
              • endpoint_override
              • metrics_collected
                • collectd
                • cpu
                  • resources
                  • totalcpu
                  • measurement[]
                    • rename
                    • unit
                  • metrics_collection_interval
                  • append_dimensions
                • disk
                  • resources
                  • measurement[]
                    • rename
                    • unit
                  • ignore_file_system_types
                  • metrics_collection_interval
                  • append_dimensions
                • diskio
                  • resources
                  • measurement[]
                    • rename
                    • unit
                  • metrics_collection_interval
                  • append_dimensions
                • swap
                  • measurement[]
                    • rename
                    • unit
                  • metrics_collection_interval
                  • append_dimensions
                • mem
                  • measurement[]
                    • rename
                    • unit
                  • metrics_collection_interval
                  • append_dimensions
                • net
                  • ...
                • netstat
                  • ...
                • processes
                  • ...
                • procstat
                • statsd
              • force_flush_interval
              • credentials
                • role_arn
              logs
              • logs_collected
                • files
                  • collect_list[]
                    • file_path
                    • log_group_name
                    • log_stream_name
                    • timezone
                    • multi_line_start_pattern
                    • encoding
                • windows_events
                  • collect_list[]
                    • event_name
                    • event_levels
                    • log_group_name
                    • log_stream_name
                    • event_format
                • log_stream_name
                • endpoint_override
                • force_flush_interval
                • credentials
                  • role_arn
        • Merge several json files:
          • awslogs_merge_conf.sh
            • #!/bin/bash

              EXPECTED_ARGS=2
              if (( $# != $EXPECTED_ARGS ))
              then
                  cat <<EOF
              Usage: `basename $0` config_dir config_file

              Examples:
              - `basename $0` /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.d /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
              EOF
                  exit 1
              fi

              # parameters
              config_dir=$1
              config_file=$2

              jq -s '.[0].logs.logs_collected.files.collect_list = [.[].logs.logs_collected.files.collect_list | add] | .[0]' ${config_dir}/*.json >${config_file}

              exit 0

http://www.francescpinyol.cat/aws.html
Primera versió: / First version: 2.X.2015
Darrera modificació: 21 de març de 2020 / Last update: 21st March 2020

Valid HTML 4.01!

Cap a casa / Back home