Friday, 25 October 2013

New Website Content

Just pushed a new version of  www.civet-labs.com live, just more content than the last parked version.

Some interesting things for us all to remember:

One:

Always have someone test your work! (Thanks James) As you are bound to forget something. 

Two:

Always have a deploy script!  It makes life so much easier.

Here is the exceptionally simple one to release www.civet-labs.com to s3.  Using boto (again)

import boto.s3.connection
import boto.s3.key
import os, sys

bucket_names = {"peek":"peek.civet-labs.com", 
           "live":"www.civet-labs.com"}

default_name = "peek"
name = ""
if len(sys.argv)>1:
    name = sys.argv[1]

if name not in bucket_names:
    name = default_name

def s3Callback(bytes_transmitted, bytes_total):
    print "\t",bytes_transmitted, bytes_total

conn = boto.s3.connection.S3Connection()
bucket = conn.get_bucket(bucket_names[name])
print "Deploying to: ", bucket

filenames = map(lambda x:os.path.join("..",x), os.listdir(".."))
for filename in filenames:
    if os.path.isfile(filename):
        print filename
        key = boto.s3.key.Key(bucket, os.path.split(filename)[1])
        key.set_contents_from_filename(filename, 
                                       cb=s3Callback,
                                       policy='public-read'
                                       )
Three:

twistd.py web --path=.

runs a much better web server than

python -m SimpleHTTPServer

for really simple testing of a static site, if you have twisted installed.

Tuesday, 22 October 2013

Twisted Server on AWS EC2

This is a post on how I built a script to provision a AWS EC2 instance for a twisted server, for those that just want the finished script it is in my bit bucket repo here with instructions on how to use it - https://bitbucket.org/andrewcox/provisiontwistedserver

Wanting a public facing version of MailThrottler I started to look to see if there was an easy way to deploy it, like a web site on Heroku or AWS elastic beanstalk.

Looking at beanstalk it seemed easy, but I started hitting roadblocks due to the fact I wanted to run an non web TCP server.  So moving onto Cloud Formation I was quickly lost in a world of Chef and Ruby.  At this point I went to bed.

The next evening after a little more digging, I thought why not just do it all myself by creating a EC2 instance, using boto and fabric.

Boto is Amazon's python package that provides interfaces to Amazon Web Services, and Fabric is a python library for streamlining the use of SSH.

With Boto I could launch the instance and wait for the instance to be up and reachable by ssh, then with Fabric I could run the shell commands over ssh need to provision the mailthrottler server.

Starting in earnest I found that the actual launching of the server is easy.  Once you have the image id that you want, a keypair and a security group you can launch to your hearts content.  Though you will have lots of servers with nothing on them. 

The next challenge is to wait for the server to be reachable before you can ssh to it.  It is not enough to just wait for the instance to be running, you also have to wait for both status checks to be green lighted.  For a strange reason sometimes the response wouldn't give the correct object back, so that is why you will see a dummy one in the code.

After we have the instance launched and running fabric made it ridiculously easy to run the commands I needed, it is just a matter of setting the correct hosts and keyfile, then calling run() or sudo() as needed. 

The only real problem I had after that is fabric was exiting from calling the twistd command to quickly, so after some poking round I found it was just best to tack a && sleep 5 onto the end and everything was great. 

Like I said at the start of the post the script itself is located at https://bitbucket.org/andrewcox/provisiontwistedserver it has a readme with instructions on how to use it.

In the end the script took me about 2 evenings to write and cost a grand total of $0.20 in AWS fees

Let me know if it works for you!

Friday, 12 April 2013

Heroku Scheduler and the UnicodeEncodeError

Ruminate runs a scheduled task on Heroku every hour to grab the new entries from the configured news feeds.

I was noticing that some feeds were not updating, checking the logs I saw that I was getting a:
'ascii' codec can't encode character u'\u2022' in position 49: ordinal not in range(128)
This was odd because if I ran it from the command line on my machine it worked, and even stranger when I ran it with the heroku run it also worked.

Turning to Google I found this answer on StackOverflow - http://stackoverflow.com/a/11762169/27907

The long and short of it is that the print statement will happily default to ascii in python, if not told otherwise.  I am guessing that when run from my console it picks up that it is utf8, but when running detached it defaults back to ascii.  As the answer states this is an easy fix by just setting enviroment variable PYTHONIOENCODING to the codec that you want.  In this case utf8.

Running the following command sets the environment variable on the heroku app to the correct value:
heroku config:add PYTHONIOENCODING=utf8
This fixed the issue, and got my missing entries into the app. 

Hope this helps.

Tuesday, 2 April 2013

Announcing Ruminate


A week ago I challenged myself to write a simple Google reader clone. 

The reason being that I wanted to write a non-trivial piece of python code, and document the creation process.

It is now in version 0.1.  I have released it onto heroku and it works for my reading needs. 

You can see the Trello board at https://trello.com/b/ZqZWaSlr

and the code here https://bitbucket.org/andrewcox/ruminate

Currently Ruminate really only works for me, but in the coming weeks I hope to add more functionality (as seen on the trello board) if you want access find me at @vipox or follow the instructions on how to set it up on heroku yourself.

In the coming days I hope to blog the development of the app!

Wednesday, 13 March 2013

Installing .Net 1.1 on a AWS Windows 2008 machine

I am trying to get the .Net 1.1 applications that I work with installed on a AWS Windows 2008 instance so that it can act as a development server. 

After installing .Net 1.1 our windows services install and run OK. 

The problem comes when you try to run an ASP.Net application.  Searching I found this blog post, which has great instructions on how to get .Net 1.1 installed.

So big thanks to Ivor Bright, after following your instructions and restarting I was away laughing. 

Now I just have to work out if I cleared out to much of our database before I restored it into AWS.




Saturday, 23 February 2013

Installing Nikola on Windows 7

The new version of Nikola was released and wanted to know if I could use it for www.civet-labs.com.

I started by trying to pip install nikola into a virtualenv, but it didn't work out of the box.
The reason it was not installing out of the box was due to the fact windows doesn't have a build chain installed, and pip doesn't use eggs or exes.
Here are the steps I took to install nikola on my windows 7 laptop into a virtualenv

I will assume that you have python 2.7 installed (like me) and that you have also installed virtualenv

1) Create and activate a new virtual environment

2) Download pillow from pypi: https://pypi.python.org/pypi/Pillow/1.7.8#downloads

3) Run easy_install on the downloaded egg
easy_install %HOME%\Downloads\Pillow-1.7.8-py2.7-win32.egg
4) Download the unofficial build of lxml from http://www.lfd.uci.edu/~gohlke/pythonlibs/#lxml

5) Run easy_install on the downloaded exe
easy_install %HOME%\Downloads\lxml-3.1.0.win32-py2.7.exe 
6) pip install the nikola package
pip install nikola
Once I had installed it I tried the steps to set up a demo in the nikola handbook and found I had some issues as well.  They seem that they are going to be fixed but I will document them here for future reference.

1) I tried "nikola init mysite --demo" but got a "error not found" this is due to the fact that windows doesn't have executable scripts the same as *nix environments, and so the nikola script is not runnable by itself.  The easy way to fix this is just to copy the bat script from doit (installed as a requirement) to nikola.bat
copy venv\Scripts\doit.bat venv\Scripts\nikola.bat
2) Continuing on with the demo I found the nikola build command was failing because a temp file wasn't being deleted.  This should be being fixed in https://github.com/ralsina/nikola/issues/151.
Till then I just added
self.dodo.close()
to
venv/Lib/site-packages/nikola/plugins/command_build.py+67

Friday, 22 February 2013

Would your device lie to you?

The purpose of this post is the pose the question "Why would your device lie to you?" 

One of the assumptions of the "Internet of things"  is that the device is objective.  That the device will report the truth within the limits of its sensors. 

That is a useful assumption, but what if the devices weren't.  They were telling less than you the truth. 

I see that there are three reasons that your device may not be telling you the whole truth: Bugs, Marketing and Evil.

Bugs either in the software controlling the device or the software reporting the data cause the truth to be bent.  While this isn't lying in the traditional sense, more like a passer by miss reading their watch when you ask them for the time, the effect is the same.

It is important to consider this point as it could be used to cover up for the other two, and as a developer I know it is a very real possibility.

Marketing, or the white lie to push you in the device creators direction.  Imagine that you buy a sensor from a fertilizer company that should tell you when to feed your plant.  It is not hard to see how getting the device not to tell the exact truth would allow the fertilizer company to sell more fertilizer.  If the reason for the increased fertilizer was a lowered tolerance in the sensor, you can also see how one could blame "Bugs" for the marketing decisions.

Another example of this marketing lie may come from a fitness device that reports the calories used based on a gaming algorithm to increase your engagement with the device and system as a whole! 

These lies while falsehoods are like the lies that we tell each other every day to get them to do the things we want them to and pail in comparison to the next type of lie.

Evil.  Unlike the marketing lie which isn't really trying to harm you except to get you to spend more, these devices are actively trying to betray you.  The famous example of this in the wild is the Iranian nuclear centrifuges which were set to destroy themselves. 

You will unlikely meet these devices in the wild (unless you are also trying to do evil) but it is still not beyond the means of criminal networks to use your devices against you, or maybe just someone doing for the LOLZ.

So next time you look at data from your senors ask yourself "Is it lying to me?"