CloudTrail and Splunk

1) Navigate to CloudTrail > Add new trail
    Trail: my-cloudtrail
    Apply trail to all regions: Yes
    Create new S3 bucket: Yes
    S3 bucket: my-cloudtrail
    Advanced
      Send SNS notification for every log file delivery: Yes
      Create a new SNS topic: my-sns-topic-cloudtrail
    Create
   
2) Create SQS Queue
      Services > SQS > Create New Queue > Create
        Queue Name: my-sqs-cloudtrail

3) Subscribe SQS Queue to SNS Topic
      Services > SQS > my-sqs-cloudtrail > Queue Actions > Subscribe Queue to SNS Topic
      > Choose a Topic >  my-sns-topic-cloudtrail > Subscribe

4) Setup AWS permissions

5) Setup Data Inputs
       Settings > Data Inputs > CloudTrial >


 

Install node, npm and gulp on EC2

wget https://nodejs.org/dist/v5.10.1/node-v5.10.1-linux-x64.tar.xz
tar -xvf node-v5.10.1-linux-x64.tar.xz

Node and npm are installed.
Add the path to the .bash_profile and/or .bashrc

npm install -g gulp
npm install gulp --save-dev

Gulp is installed

Jenkins on AWS EC2 (Master/Slave setup using EC2 plugin)

1) Create user ciadmin
useradd ciadmin
passwd ciadmin

2) Create a file system /apps/jenkins
chown -R ciadmin:ciadmin jenkins

3) Install JDK 1.8, Apache Tomcat 9

4) Download and Copy Jenkins.war to webapps directory

5) Update .bash_profile
PATH=$PATH:$HOME/.local/bin:$HOME/bin
JAVA_HOME=/apps/jenkins/jdk1.8.0_77
JAVA_OPTS="$JAVA_OPTS -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses"
PATH=$JAVA_HOME/bin:$PATH
export PATH JAVA_HOME JAVA_OPTS

6) Update /etc/hosts for the tomcat to start properly (example below)
172.x.y.x ip-172-x-y-z ip-172-x-y-z.us-west-2.compute.internal


7) Hit Jenkins URL: http://ip:8080/jenkins

8) Install Amazon EC2 plugin
Manage Jenkins -> Manage Plugins -> Available > Cluster Management and Distributed Build > Amazon EC2 plugin > Install

9) Create 2 AWS IAM roles

9.1) jenkins-master-role
Attach the custom policy

{
    "Version": "xxxxx",
    "Statement": [
        {
            "Sid": "Stmtxxx",
            "Effect": "Allow",
            "Action": [
                "ec2:DescribeSpotInstanceRequests",
                "ec2:CancelSpotInstanceRequests",
                "ec2:GetConsoleOutput",
                "ec2:RequestSpotInstances",
                "ec2:RunInstances",
                "ec2:StartInstances",
                "ec2:StopInstances",
                "ec2:TerminateInstances",
                "ec2:CreateTags",
                "ec2:DeleteTags",
                "ec2:DescribeInstances",
                "ec2:DescribeKeyPairs",
                "ec2:DescribeRegions",
                "ec2:DescribeImages",
                "ec2:DescribeSecurityGroups",
                "ec2:DescribeSubnets"
            ],
            "Resource": [
                "*"
            ]
        }
    ]
}

Verify trust relationship

{
  "Version": "xxxxxx",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}


9.2) jenkins-slave-role

Verify trust relationship

{
  "Version": "xxxxxx",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ec2.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}

10) Setup EC2 Slave Config

Manage Jenkins > Configure System > Cloud section > Amazon Ec2

Name: aws-dev-vpc

Access Key: 

Secret Key: 

Use EC2 instance profile (Checked)

Region: us-west-2

EC2 Key Pair's Private Key: Place your private key

Test your AWS connection!!

AMIs

Description: celebrity-jenkins-slave

AMI ID: ami-xxxx

Instance Type: M4Large

EBS Optimized: Checked

Availability Zone: us-west-2a

Security group names: my-security-group

Remoe FS root: /mydir/subdir

Remote user: ciadmin

AMI Type: unix

Idle termination time: 60 

Note: Please note that slave will get terminated automatically after idle termination time with no activities kicked off by Jenkins

Stop/Disconnect on Idle Timeout: Checked

Note: If you do not want the slave to get terminated automatically but to stop, check the above.

Subnet ID for VPC: subnet-xxxx

Tags:
Name: project Value: myproject
Name: Name Value: myjenkins-slave

Instance Cap: 3
IAM Instance Profile: arn:aws:iam::MyAccountNo:instance-profile/cmyjenkins-slave-role

Connect by SSH Process: Checked

Save

11) Start a Slave
Manage Jenkins > Manage Nodes > provision via aws-de-vpc
Check the logs

Mounting EBS volume on EC2

[ec2-user@ip ~]$ lsblk
NAME    MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
xvda    202:0    0  10G  0 disk
ââxvda1 202:1    0   1M  0 part
ââxvda2 202:2    0  10G  0 part /
xvdb    202:16   0  50G  0 disk

[ec2-user@ip ~]$ sudo file -s /dev/xvdb
[ec2-user@ip ~]$ sudo mkfs -t ext4 /dev/xvdb
[ec2-user@ip ~]$ sudo mkdir -p /mydir/subdir
[ec2-user@ip ~]$ sudo mount /dev/xvdb /mydir/subdir

Add file system to fstab to remount on system reboot
[ec2-user@ip ~]$sudo cp /etc/fstab /etc/fstab.orig

Add the line to /etc/fstab
/dev/xvdb  /mydir/subdir  ext4    defaults,nofail        0       2

[ec2-user@ip ~]$sudo mount -a

CodeCommit vs BitBucket

CodeCommit does not have Pull Request faeture. Hence, using Bitbucket

AWS Codedeploy

Using Codedeploy to publish files from S3 to EC2 server


Preparing a POC application to use with Codedeploy

• Create index.html with some sample content
• Create appspec.yml with the following content

----------------------------------------------------------

version: 0.0

os: linux 

files:

 - source: /index.html

   destination: /home/ec2-user/myapp/

----------------------------------------------------------

• Please note all other unused config has been stripped down from original appsec.yml. It is important to remove all unused syntax. Otherwise, the application will not deploy properly using Codedeploy.
• Zip just the files index.html and appspec.yml. Name the zip file as myapp-poc.zip
• Please note when the zip is open, appspec.yml should be at the root level without any sub directories

Create S3 bucket and upload deploy artifacts

• Create S3 bucket myapp ( Left the default Grantee myaccount with all permissions)
• Upload myapp-poc.zip to S3 myapp S3 bucket ( Left the default Grantee myaccount with all permissions)

Create IAM policy and roles

• Create a policy myapp-codedeploy-ec2-policy S3 bucket myapp ( Left the default Grantee myaccount with all permissions)
• Upload myapp-poc.zip to S3 myapp S3 bucket ( Left the default Grantee myaccount with all permissions)
• Contents of myapp-codedeploy-policy

----------------------------------------------------------
{
   "Version": "xxxx",
   "Statement": [
{
   "Sid": "Stmtxxx",
   "Effect": "Allow",
   "Action": [
"s3:Get*",
"s3:List*"
   ],
   "Resource": [
"arn:aws:s3:::myapp/*",
"arn:aws:s3:::aws-codedeploy-us-east-1/*",
"arn:aws:s3:::aws-codedeploy-us-west-2/*",
"arn:aws:s3:::aws-codedeploy-us-west-1/*"
   ]
}
   ]
}

----------------------------------------------------------

• Create a role myapp-codedeploy-service-role 
• Choose AWSCodeDeploy from AWS Service Roles
• Attach the policy AWSCodeDeployRole (AWS managed) to myapp-codedeploy-service-role
• Edit trust relationship of myapp-codedeploy-service-role to read as 

----------------------------------------------------------

{

 "Version": "xxxx",

 "Statement": [

   {

     "Sid": "",

     "Effect": "Allow",

     "Principal": {

"Service": [

 "codedeploy.us-west-2.amazonaws.com",

 "codedeploy.us-west-1.amazonaws.com",

 "codedeploy.us-east-1.amazonaws.com"

]

     },

     "Action": "sts:AssumeRole"

   }

 ]

}

----------------------------------------------------------

• Create a role myapp-codedeploy-ec2-role
• Choose Amazon EC2 from AWS Service Roles
• Attach the policy myapp-codedeploy-ec2-policy (AWS managed) to myapp-codedeploy-ec2-role
• Edit trust relationship of myapp-codedeploy-ec2-role to read as 

----------------------------------------------------------

{

 "Version": "xxxx",

 "Statement": [

   {

     "Effect": "Allow",

     "Principal": {

"Service": "ec2.amazonaws.com"

     },

     "Action": "sts:AssumeRole"

   }

 ]

}

----------------------------------------------------------

Create an EC2 instance

• Spin up an EC2 instance with desired instance type.
• In "Configure Instance Details" section, choose proper vpc, subnet. choose the IAM role as myapp-codedeploy-ec2-role
• Please note that if you use Codedeploy wizard, it creates EC2 in the default vpc. If the default vpc does not exist, it fails. That is the reason behind spinning up a custom ec2 instance,
• Create tags on the EC2 instance: Ex: Tag name: project Tag value: myapp
• Please note tags are used by Codedeploy to discover instances.

Deploy Codedeploy agent on EC2 instance

• Login into EC2 instance
• Run the following commands
• sudo yum update
• sudo yum install ruby
• sudo yum install wget
• cd /home/ec2-user
• wget https://bucket-name.s3.amazonaws.com/latest/install
• chmod +x ./install
• sudo ./install auto
• sudo service codedeploy-agent status

Create Codedeploy config

• Create new application 
• Application name: myapp
• Deployment group name: myapp-deploy-stage
• Tags: Amazon EC2: Key: project Value: myapp (The no. of instances discovered will be displayed)
• Choose Service Role myapp-codedeploy-service-role
• Leave rest of them as default
• Create application
• In the deployment group, select the deployment myapp-deploy-stage
• Actions: Deploy new revision
• Create New Deployment
• Application: myapp
• Deployment Group: myapp-deploy-stage
• Revision Type: My application is stored in Amazon S3
• Revision Location: 

• Go to S3 console, select myapp-poc.zip. Copy the complete https url link and ETag url
• Form the revision URL as follows as an example
• https://s3-us-west-2.amazonaws.com/myapp/myapp-poc.zip?etag=962c02cb729b2f36745acbf4102129e1

• Paste the above URL with ETag in the Revision Location field
• Deploy

Enable forensic log in Apache within AWS Beanstalk

1) Change (or Uncomment) the line in file /etc/httpd/conf/httpd.conf
#LoadModule log_forensic_module modules/mod_log_forensic.so
to
LoadModule log_forensic_module modules/mod_log_forensic.so

2) Update /etc/httpd/conf.d/elasticbeanstalk.conf to include the line
ForensicLog /var/log/httpd/forensic_log 

3) Update /etc/httpd/conf.d/elasticbeanstalk.conf to add %{forensic-id}n at the end
LogFormat "%h (%{X-Forwarded-For}i) %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %{forensic-id}n"
This step is optional and help with forensic id correlation with access log.

4) Stop apache, check no http process is out there using ps-ef | grep http and start apache
/usr/sbin/apachectl stop
ps -ef | grep http (check for no http process)
/usr/sbin/apachectl start

CodeCommit

1) Create a userid and grant codecommitfullaccess policy for the poc purpose
2) Create a repository named jrepo in AWS CodeCommit
3) Copy the https url (example: https://git-codecommit.us-east-1.amazonaws.com/v1/repos/jrepo)
4) Setup AWS CLI
5) Install AWS Tools from https://aws.amazon.com/powershell/


6) Run windows command utility as administrator
7) cd %PROGRAMFILES(X86)%\AWS Tools\CodeCommit
8) git-credential-AWSS4.exe -p jcodecommit
    Note: jcodecommit is profile name stored in AWS config or credentials file  (under users home .aws dir)
    Choose Yes to generate sig
9) Run git config --global --edit and you should see a similar block
[credential]
helper = !'C:\\Users\\j\\AppData\\Roaming\\GitCredStore\\git-credential-AWSSV4.exe' --profile=jcodecommit
UseHttpPath = true

10) Create a local directory named codecommitrepos
11) cd c:\codecommitrepos
12) git clone https://git-codecommit.us-east-1.amazonaws.com/v1/repos/jrepo local-jrepo
13) git config --local user.name "developer1"
14) git config --local user.email developer1@email.com

15) cd c:\codecommitrepos\local-jrepo
15)  Create files index.html and index2.html
16) git add index.html
17) git commit -m "Added index.html'
16) git add index2.html
17) git commit -m "Added index2.html'

18) git push -u origin master

Voila! The files are now pushed to AWS CodeCommit.

ELB CLI Commands

1. Create internal load balancer aws elb create-load-balancer --load-balancer-name myelbname --listeners Protocol=HTTP,LoadBalancerPort=80,InstanceProtocol=HTTP,InstancePort=9080 --subnets mysubnet1-in-useast1a mysubnet2-in-useast1c--scheme internal --tags Key=environment,Value=stage Key=project,Value=myproject Key=app,Value=myapp Key=product,Value=elb 
   
   
2. Update application generated cookie stickiness aws elb create-app-cookie-stickiness-policy --load-balancer-name myelbname --policy-name myapp-cookie-policy --cookie-name JSESSIONIDX aws elb set-load-balancer-policies-of-listener --load-balancer-name myelbname --load-balancer-port 80 --policy-names myapp-cookie-policy
   
   
3. Update internal load balancer attributes (Combined - CrossZoneLoadBalancing, AccessLogs, ConnectionSettings, ConnectionDraining) 

aws elb modify-load-balancer-attributes --load-balancer-name myelbname --load-balancer-attributes {\"CrossZoneLoadBalancing\":{\"Enabled\":true},\"AccessLog\":{\"Enabled\":true,\"S3BucketName\":\"myelblogs\",\"S3BucketPrefix\":\"myapp\",\"EmitInterval\":5},\"ConnectionSettings\":{\"IdleTimeout\":60},\"ConnectionDraining\":{\"Enabled\":true,\"Timeout\":300}}  

# Individual Commands 

# Update load balancer attributes - CrossZoneLoadBalancing aws elb modify-load-balancer-attributes --load-balancer-name myelbname --load-balancer-attributes {\"CrossZoneLoadBalancing\":{\"Enabled\":true}} 

# Update internal load balancer attributes - AccessLogs aws elb modify-load-balancer-attributes --load-balancer-name myelbname --load-balancer-attributes {\"AccessLog\":{\"Enabled\":true,\"S3BucketName\":\"myelblogs\",\"S3BucketPrefix\":\"myapp\",\"EmitInterval\":5}} 

# Update internal load balancer attributes - ConnectionSettings aws elb modify-load-balancer-attributes --load-balancer-name myelbname --load-balancer-attributes "{\"ConnectionSettings\":{\"IdleTimeout\":60}}" 

# Update internal load balancer attributes - ConnectionDraining aws elb modify-load-balancer-attributes --load-balancer-name myelbname --load-balancer-attributes "{\"ConnectionDraining\":{\"Enabled\":true,\"Timeout\":300}}"

SSL upload

Upload

aws iam upload-server-certificate --server-certificate-name mycertname --certificate-body file://mycert.cer --private-key file://mycert.key

or

aws iam upload-server-certificate --server-certificate-name mycertname --certificate-body file://mycert.cer --private-key file://mycert.key  --certificate-chain file://certificate_chain_file

Verify

aws iam get-server-certificate --server-certificate-name mycertname

Commands to convert .crt and .key to .pem
openssl rsa -in server.key -text > my-private-key.pem
openssl x509 -inform PEM -in server.crt > my-private-cert.pem

Setup Amazon RDS CLI on Windows

Download RDSCLI.zip from http://aws.amazon.com/developertools/2928
Extract  the zip file C:\......\RDSCli\RDSCli-1.15.001

Set the path
set JAVA_HOME=C:\Program Files\Java\jdk1.7.0_55

setx AWS_CREDENTIAL_FILE C:\...\RDSCli\RDSCli-1.15.001\credential
(this file contains access keys)

setx AWS_RDS_HOME C:\...\RDSCli\RDSCli-1.15.001

setx PATH "%PATH%;%AWS_RDS_HOME%\bin"

> rds --help

Setup Amazon CLI on Windows

1. Download the AWS CLI MSI installer (AWSCLI64.msi) for Windows (64-bit) from https://s3.amazonaws.com/aws-cli/AWSCLI64.msi

2. Execute the default installation AWSCLI64.msi

3. Download python (python-3.4.1.amd64.msi) )from http://python.org/download

4. From windows command line, execute these commands
   > set PATH=%PATH%;C:\Python34
   > python --help
   > python -m pip install -U pip --proxy [user:password]@proxyserver:proxyport
   > pip --help
   > aws help
   > aws --version

5. Configure AWS
    > aws configure
       AWS Access Key ID [None]: enter-your-access-key-id-here
       AWS Secret Access Key [None]: enter-your-secret-access-key-here
       Default region name [None]: us-east-1
       Default output format [None]:

Creating and connecting to first EC2 UNIX instance

Login to AWS EC2
Once you sign up or EC2 account, login to AWS management console.
Select EC2 tab

Create Key Pair
From the navigation panel on the left, navigate to Network and Security section
Click on Key Pairs
Create new Key Pair. Provide name of your choice (eg: kpn1). A key pair name will be created. Fingerprint will be displayed on the key pair panel. A file will be downloaded to your desktop named kpn1.pem

Create Security Group
From the navigation panel on the left, navigate to Network and Security section
Click on Security Group
Create Security Group (eg: Name: sg1, Description: Security Group 1)

Create EC2 instance

1. From the navigation panel, go to EC2 dashboard
2. Click on Launch Instance button
3. If you want to start with a new instance to play, select Basic 32-bit Amazon Linux AMI 2011.02.1 Beta
4. In the next page, select Number of Instances - 1, Availability Zone - no preference, Instance Type: Micro. These are the defaults. Leave as it is. Continue.
5. Leave  the defaults on Advance Instance Options - Continue
6. On the key value panel, type in the key as 'Name' and value as 'Web Server' (you can label it appropriately based on your server type). Continue
7. Select 'Choose from your existing key pair'. Select the value  kpn1 from the dropdown. Continue
8. Select Security group sg1. Continue.
9. Click on Launch Instance. This step will create your EC2 instance.

Locating your EC2 IP
Navigate to Instances.
Select the server and review the bottom panel.
Public DNS: ec2-XX-XX-XX-XXX.compute-1.amazonaws.com
ec2-XX-XX-XX-XXX.compute-1.amazonaws.com is your hostname.
XX.XX.XX.XXX is  your IP address

Connecting to your EC2 Instance as ec2-user
ssh ec2-user@ec2-instance-ipaddress –i {filepath}/privkey.pem

or

1) Convert .pem to .ppk format using puttygen
Download puttygen.exe from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html
Execute puttygen
File -> Load Private Key -> Select kpn1.pem
You will get a prompt successfully imported foreign key.
File -> Save Private Key -> Save as kpn1.ppk

2)
Download putty.exe from http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html. This is a tool to connect to your ec2 unix instance.
Execute putty.exe
Session -> Host Name -> Type ec2 instance ip address
Navigate to Connection -> SSH - Auth -> Select the kpn1.ppk file
Open

3) Login as ec2-user

Connecting to your EC2 Instance as root
You still have to follow the steps to login as ec2-user
and then type command "sudo su"

Change the time in EC2 instances

1) Backup /etc/localtime
2) ln -sf /usr/share/zoneinfo/America/Los_Angeles /etc/localtime

Sign up

Sign up for Amazon AWS.

Setting up a website in AWS require planning and decision making.

Try to understand the pricing model.