New Website Content

Just pushed a new version of  www.civet-labs.com live, just more content than the last parked version.

Some interesting things for us all to remember:

One:

Always have someone test your work! (Thanks James) As you are bound to forget something. 

Two:

Always have a deploy script!  It makes life so much easier.

Here is the exceptionally simple one to release www.civet-labs.com to s3.  Using boto (again)

import boto.s3.connection
import boto.s3.key
import os, sys

bucket_names = {"peek":"peek.civet-labs.com", 
           "live":"www.civet-labs.com"}

default_name = "peek"
name = ""
if len(sys.argv)>1:
    name = sys.argv[1]

if name not in bucket_names:
    name = default_name

def s3Callback(bytes_transmitted, bytes_total):
    print "\t",bytes_transmitted, bytes_total

conn = boto.s3.connection.S3Connection()
bucket = conn.get_bucket(bucket_names[name])
print "Deploying to: ", bucket

filenames = map(lambda x:os.path.join("..",x), os.listdir(".."))
for filename in filenames:
    if os.path.isfile(filename):
        print filename
        key = boto.s3.key.Key(bucket, os.path.split(filename)[1])
        key.set_contents_from_filename(filename, 
                                       cb=s3Callback,
                                       policy='public-read'
                                       )

Three:

twistd.py web --path=.

runs a much better web server than

python -m SimpleHTTPServer

for really simple testing of a static site, if you have twisted installed.

Twisted Server on AWS EC2

This is a post on how I built a script to provision a AWS EC2 instance for a twisted server, for those that just want the finished script it is in my bit bucket repo here with instructions on how to use it - https://bitbucket.org/andrewcox/provisiontwistedserver

Wanting a public facing version of MailThrottler I started to look to see if there was an easy way to deploy it, like a web site on Heroku or AWS elastic beanstalk.

Looking at beanstalk it seemed easy, but I started hitting roadblocks due to the fact I wanted to run an non web TCP server.  So moving onto Cloud Formation I was quickly lost in a world of Chef and Ruby.  At this point I went to bed.

The next evening after a little more digging, I thought why not just do it all myself by creating a EC2 instance, using boto and fabric.

Boto is Amazon's python package that provides interfaces to Amazon Web Services, and Fabric is a python library for streamlining the use of SSH.

With Boto I could launch the instance and wait for the instance to be up and reachable by ssh, then with Fabric I could run the shell commands over ssh need to provision the mailthrottler server.

Starting in earnest I found that the actual launching of the server is easy.  Once you have the image id that you want, a keypair and a security group you can launch to your hearts content.  Though you will have lots of servers with nothing on them. 

The next challenge is to wait for the server to be reachable before you can ssh to it.  It is not enough to just wait for the instance to be running, you also have to wait for both status checks to be green lighted.  For a strange reason sometimes the response wouldn't give the correct object back, so that is why you will see a dummy one in the code.

After we have the instance launched and running fabric made it ridiculously easy to run the commands I needed, it is just a matter of setting the correct hosts and keyfile, then calling run() or sudo() as needed. 

The only real problem I had after that is fabric was exiting from calling the twistd command to quickly, so after some poking round I found it was just best to tack a && sleep 5 onto the end and everything was great. 

Like I said at the start of the post the script itself is located at https://bitbucket.org/andrewcox/provisiontwistedserver it has a readme with instructions on how to use it.

In the end the script took me about 2 evenings to write and cost a grand total of $0.20 in AWS fees

Let me know if it works for you!