Inserting the BOM into a file

I have been working with Windows UTF-8 files a lot today, and I found that just saving the file as UTF-8 isn't enough for ASP.NET (well 1.1 at least). Even though emacs says that it will save the buffer with the BOM (Byte Order Mark) it doesn't seem to if the file doesn't start with one. So I wrote myself a little helper function to add the BOM into the start of the file. It works by going to the start of the buffer you are in, and adding the BOM FEFF. The exact bytes comprising the BOM for the Unicode character U+FEFF are converted into the UTF-8 format by emacs when it saves the file (which for reference are EF BB BF -- thanks to pnkfelix for pointing that out).

;;Insert the BOM at the start of a file for UTF
(defun insert-BOM()
  (interactive)
  (goto-char (point-min))
  (ucs-insert (string-to-number "FEFF" 16)) 
)

My first SSG blog post

I just posted my first blog post over at http://www.ssgl.com/blog. Its just a short comment on why my development team is better than the rest! You can see it here

Why didn't I follow the checklist!?!

Today I upgraded our Subversion server to 1.6.5. It was a nightmare and all because I tried to do it from memory. It seemed simple, just uninstall and install a few services and then I would be away laughing. However I was shown why pride comes before a fall. All was going well till Apache wouldn't start. After pushing and pulling at the configuration and reading the error logs, it turned out I should have installed mod_python. All shiny, but why isn't Trac working! Many hours later, oh yes I should reinstall the python bindings to Subversion. Look at that now it works. Later checking the wiki (to make sure it still works) I read the article I wrote on how to upgrade subversion. There in black and white was the steps as I should have taken them. Including the missing mod_python and python bindings. Why didn't I just follow my own checklist.

Serverside's new Blog

I finally finished putting the companies blog live. So if I blog there I will tell you all about here. You can find it at http://www.ssgl.com/blog. It is a simple dasBlog running off our UK cluster. I am hoping that the clustered set up that I went for is going to work.

My First Context Free Upload

So impressed by the Penny Arcade Automata Wall Paper that I have created my first context free upload. The idea is that I build up a set of units (I have called them Notes) that could be seen as patterns in the image. The patterns are meant to be based on the distribution English language letters so that the image looks more "natural" to the eye. Tell me what you think.

Emacs 23 - impressions

I am still waiting for Emacs 23.1 on my FreeBSD home laptop. However I am using Emacs 23.1 on my Windows machine at work and am loving it. The main things I am noticing are:

• The horizontal split when a new buffer is created for something like a *compile*
• The unicode support is much better, I no longer get the unicode "dot" at the start of a file saved by someone with Visual Studio.
• M-x butterfly - this is the sort of stuff that I love about Emacs, is it a hard core editor, yes, but it also wants to have fun!

I am yet to try the new daemon setting on the Windows machine, as I am still using gnuserv. But I don't really restart my machine that often, only on the day after update Tuesday really, so I don't really notice. So good reviews for Emacs 23.1, thanks to the whole team that have made it happen.

Emacs 23

I am eagerly awaiting Emacs 23, should be out any time soon. I hope!

Firefox 3.5

I have now upgraded to Firefox 3.5 on FreeBSD. It was a bit more tricky than I expected, it wouldn't just install using portinstall. Instead I had to make it from the port tree manually, in the end it the commands I used were:

cd /usr/ports/www/firefox35
make config
make
pkg_delete firefox-3.0.11,4
make install

I am guessing it was the fact that I needed to uninstall firefox3 before installing firefox35 was stopping portinstall from working. Also it turned out that I needed to load the sem module too, as per the pkg-message (I should really start reading them).

CouchDB

I am giving CouchDB a look over. I had heard about it from several blogs, and decided to take the plunge on the weekend. As always it was simple to install with "port-install couchdb" the longest part was waiting for the Erlang OTP platform to build. All in all it took a episode of some trash TV to install. I have a bit of a problem with the rc.d script, which doesn't seem work, but apart from that it seems pretty cool. Will try to port StarTracker to it over the next little while...

The 3 ways to stop gnome automounting

Problem I had some hard drive slices I didn't want gnome to mount when it started up. On first glance I didn't think this was so hard to fix. However after a bit of fruitless googling, I was none the wiser of where I would go to stop the auto mounting. I knew it was happening in gnome somewhere, but where. The journey On reading a blog about the new geom functionality in FreeBSD 7.2 I tried to change my fstab file. A few kernal panics later I have the file correct with the slice names in the file and the noauto option set but that did not to work. After more googling I hit upon the solution of turning off auto-mounting completely in Nautilus using a configuration setting. This works, but it seems to be a little heavy handed as it would stop my CD-ROM from auto-mounting when I put a disk in. So I followed the code back in Nautilus and I found that it was just asking HAL and the fstab file what drives it could mount and then mounting them, so my next solution was to make HAL ignore the slices, much reading later I had a fdi file that ignored the geom labels. This worked, but again I was troubled as to why the noauto route didn't work as it should. This is when I re-read the blog post again, and worked out that you should reference the slices with the geom label names rather than the slice names. It seems that the new GEOM functionality leaves the labels active and then HAL picks them up and offers them to Nautilus, so you have to set them in the fstab to noauto. Solutions

1. As of Gnome 2.22 Nautilus is responsible for doing the auto-mounting so the first solution is to turn off all auto-mounting completely in nautilus. We can do this by setting the configuration

   /apps/nautilus/preferences/media_automount

   to false. You can do this with the configuration editor or from the shell with

   gconftool-2 -s --type bool /apps/nautilus/preferences/media_automount false

   The problem with this solution is that it will stop Nautilus from auto-mounting anything.
2. The next option is to make HAL ignore the slices so they are not exposed to Nautilus. The way to do this is to create a fdi file that will tell HAL what to do. You can list the hal properties of everything on you system with lshal in my case the key is block.device and we can use the contains_ncase to find the slices we want to ignore.

   <?xml version="1.0" encoding="UTF-8"?>
   
   <deviceinfo version="0.2">
     <device>
       <match key="block.device" contains_ncase="/dev/ufsid/">
           <merge key="info.ignore" type="bool">true</merge>
       </match>
     </device>
   </deviceinfo>

   This works, but again it seems to be a bit of over kill.
3. The last solution is to set your fstab file to set the noauto option. This is the most simple. Just make sure to us the glabel name for the slice

Links to supporting docs

• HAL FAQ
• HAL spec (on block device)
• Where Nautilus uses the media_automount setting
• glabel(8)
• fstab(5)
• mount(8)
• GEOM Blog post

Python Decorators

I created my first python decorator today, I have been using them for a while in cherrypy and others, but never have had to create my own. In my case I was using it to ensure that a method attribute was converted to an int before it hit the actual function, as the attribute could be passed as a string. Having several functions that had this behaviour creating a decorator seemed the best option rather than adding the conversion code to each function, that and I wanted to know how they worked. Decorators work by creating a function that returns a function (though they can be classes, anything callable I believe). The function that is returned is the one that is to be run rather than the one that was originally written, which in my book is kind of funky (and that's good funky). Once you have created a decorator function you can then just add @functionname to the top of your actual function and then the actual function becomes the function that you output in your decorator.

Stack Overflow flair

Like the true sheep I am I have added the Stack Overflow and Server fault to my blog widgets. Now everyone can see that I follow the trends.

Upgrading to FreeBSD 7.2

Find the announcement. its as easy as:

• freebsd-update upgrade -r 7.2-RELEASE
• freebsd-update install
• shutdown -r now
• freebsd-update install
• shutdown -r now

Using pygame to develop with context free

A problem I have had while playing round with Context Free, is that every time I change the cfdg file, I have to rerun cfdg and then display the resulting image. This got me thinking that there must be a good way to monitor the file and every time it changes just re-run and display the image. My mind wondered back to a pycon UK when I saw a guy using pygame to display his slides. The reason being that as pygame is essentially a framework for displaying images and handling events without the tedious bits around the edges, he could very easily create a slide show that could run on anything without an extra program required. Putting this to work against my problem I could just create a python app that would monitor the file and every time the file changed, rerun cfdg and display the image in pygame. It turns out that this is ridiculously easy: 1) Initialise a pygame window 2) Start a thread to monitor the file. The thread just runs the cfdg with popen and displays the image created to the pygame window. I found it better to just run every second that way I could see the different images created by the random branching in context free. 3) Keep going till a keystoke kills the process. So here it is (python 2.5):

import pygame, sys, threading, time, os
pygame.init()

def setupScreen(width, height):
    screen = pygame.display.set_mode( (width,height) )
    pygame.display.flip()
    return screen
    
def waitForKeyStroke():
    runProgram = True
    while runProgram:
        event = pygame.event.wait()
        if event.type == pygame.QUIT or event.type == pygame.KEYDOWN:
            runProgram = False

def displayImage( screen, path ):
    px = pygame.image.load(path)
    screen.blit(px, px.get_rect())
    pygame.display.flip()
    return px

class updator(object):
    
    def __init__(self, screen, cfdgPath, imagePath):
        self.running = True
        self.screen = screen
        self.cfdgPath = cfdgPath
        self.imagePath = imagePath
        self.updatePeriod = 1

    def updateScreen(self):
        while self.running:
            f = os.popen('cfdg %s %s' % (self.cfdgPath, self.imagePath))
            txt = f.read()
            if f.close():
                print txt
            displayImage(self.screen,self.imagePath)
            time.sleep(self.updatePeriod)

def mainLoop(screen, cfdgPath, imagePath):
    u = updator(screen,cfdgPath,imagePath)
    t = threading.Thread(target=u.updateScreen)
    t.start()
    waitForKeyStroke()
    u.running = False
    t.join(10)
    
if __name__ == "__main__":
    "Usage cfdgviewer.py  
"
    screen = setupScreen(500,500)
    mainLoop(screen, sys.argv[1], sys.argv[2])

Emacs - Copying a whole line without killing it

For some reason I always find myself trying to copy a whole line and then pasting back somewhere else. I have never found a good way in Emacs to do this except for <home> C-k C-y until today, when I wrote this little function to do it for me and bound it to M-k because I never use kill-sentence anyway. (the excessive commenting is for people learning Lisp like me)

;Create a COPY line function
(defun copy-total-line ()
  "Copy the whole line that the cursor is on"
  (interactive)
   ;We want to return to where we started
  (save-excursion      
    ;Jump to the start of the line          
    (beginning-of-line)           
    ;Store the start of the line         
    (let* ((startpos (point)))    
      ;Move to the end of the line
      (end-of-line)
      ;Add it all to the kill ring as M-w would                       
      (kill-ring-save startpos (point) )  
      )
    )
  )
;Bind it to M-k (cause I never use kill sentence)
(define-key global-map "\M-k" 'copy-total-line)

On further searching the Emacs wiki has some other functions to do this, but this way I learned some lisp!

Context Free Art

A few months ago I stumbled upon Context Free Art. It is basically a way of drawing images by using a context free grammar. The images on the site were pretty nifty, but after unsuccessfully compiling the source I gave up forlorn that I would never be able to use a context free grammar to create images. Only to check back on the site every once and a while to see what new images others had created. Today I found myself with nothing better to do so I was going to give compiling the program another try. While seeing if I had the required libraries I had the brain wave to search every field in the FreeBSD ports website for "context". Bingo! A quick portinstall cfdg and I had it on my laptop. So now I am free to context free myself into odd fractal patterns. The moral of the story is that FreeBSD ports is a truly amazing tool.

Star tracker v0.2 release

Wow, StarTracker v0.2 is now live. With new Nomination usability fixes. Now you don't need to guess the username, and the "star" is easier to pick.

Using openssl to test https sites

Here is a little gem I found in Absolute FreeBSD - Chapter 9 that lets you test the certificate used by a web server to start a https connection. Basically what happens is that openssl opens a SSL connection to the web site, prints all the details of the connection and then you can enter a HTTP request as if you were connected via telnet to port 80. Reading the output gives you a good idea of what is wrong or right with your cert. Seeing I had to use it twice today to check our certificates I would say it is really useful.
Here is an example of use:

openssl s_client -connect www.google.com:443

Which yelds:

CONNECTED(00000003)
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDITCCAoqgAwIBAgIQPI06ZO4Y3RtzC6GS7viYGzANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wODA1MDIxNzAyNTVaFw0w
OTA1MDIxNzAyNTVaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHEw1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKEwpHb29nbGUgSW5jMRcw
FQYDVQQDEw53d3cuZ29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC
gYEAmxntXaVWr0lm23n9whx4Tk8RpYqs4pTu4+JLwAMlp5nMZeHslK6u8KeZvBDX
7YcwR81Q+a/T0/QLjUeKLuLOU5uRmX8eXPkb1umTZ+NK+M/EjAxo0ZdURw4KJDCn
gpSu3q4/v7oUxviykI42reHQvhaas15yOEnadKE//9KHge0CAwEAAaOB5zCB5DAo
BgNVHSUEITAfBggrBgEFBQcDAQYIKwYBBQUHAwIGCWCGSAGG+EIEATA2BgNVHR8E
LzAtMCugKaAnhiVodHRwOi8vY3JsLnRoYXd0ZS5jb20vVGhhd3RlU0dDQ0EuY3Js
MHIGCCsGAQUFBwEBBGYwZDAiBggrBgEFBQcwAYYWaHR0cDovL29jc3AudGhhd3Rl
LmNvbTA+BggrBgEFBQcwAoYyaHR0cDovL3d3dy50aGF3dGUuY29tL3JlcG9zaXRv
cnkvVGhhd3RlX1NHQ19DQS5jcnQwDAYDVR0TAQH/BAIwADANBgkqhkiG9w0BAQUF
AAOBgQAxCmyinulUGRZomZHWQ8trtMxszLD78e6BvwArb1ASxq8CKjbBKN7FTFYg
bfU9QrkYgSCy3Vdd674yhFBFUW7N5C4qOIifUu0o//yNV7WtZK5NDg7ZPay4/mZM
FY9EUvp8PATtfzdhBP7V6bmwnv6lEWnJY9ZGgW8A2HIvgjdEwQ==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1765 bytes and written 322 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: xxx
    Session-ID-ctx: 
    Master-Key: xxx
    Key-Arg   : None
    Start Time: 1237496532
    Timeout   : 300 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
---

Absolute FreeBSD 2nd edition

In a vain attempt to learn more about FreeBSD I bought Absolute FreeBSD 2nd edition. I am already up to the 5th Chapter and finding it a good read. It reads well as the author has a very conversational tone, while giving you more than enough information on the topic at hand. He drops in jokes and honest truths, as if you were talking to him in a bar. I imagine that it would be a bit lacking if you were a hard core Unix admin, but then I guess you wouldn't be reading the book. Chapters read so far: Chapter 1 - Getting More Help A very solid discussion into how to find information on the FreeBSD system in front of you. My favorite part is the section that describes how to write an email to the BSD user groups when you need support. I wish I could get anyone that sends me a support request to read this section before they do. If only so they can understand why they get so many polite emails back asking for more detail than "its broken". Chapter 2 - Installing BSD I skimmed this chapter as I already have FreeBSD installed, but will most likely go back and read it when I need to reinstall (due to human error) Chapter 3 - Start Me Up! The Boot Process This chapter was enlightening, I now have more insight into what the messages at boot time mean, and more importantly I found out that I could change the loader logo to a colour beastie. By adding the following to /boot/loader.conf:

loader_logo="beastie

Its this sort of useless fact that I love, and while maybe not as important as some other configuration options, I now know that there is a loader.conf and what it does. Chapter 4 - Read this before you break something else! (Backup and Recovery) I was ready to skip this chapter, when my eyes fell upon the section on Revision Control for config files. This is something I have been wondering how to do this properly for a while now, so I stopped skimming and read on. Once again this section was straight forward and explains both the whys and the hows. No more thinking "I should really have this under version control" for me :)

Star Tracker v 0.1 Live

Finally I have got my act together and got star tracker released to the Civet server. So you can see version 0.1 at http://caffeinesoftware.no-ip.org/startracker/index.

Changing the length of comments for C mode

For a long time I was irked by Emacs' habit off auto wrapping my comments especially when I was adding them to the end of lines in csharp-mode. Finally I have worked out what the problem was, after many hours of searching. As it turns out comments in c mode have been set to auto fill, and all I had to do was adjust my fill-column property to something large, (at the moment 200) and then it will only wrap the comment when the column length gets to 200. Genius. Given that I don't really use auto-fill for anything else it was a simple matter of:

• M-x customize-variable fill-column
• Set the value and "Save for Future Sessions"

More info at http://www.gnu.org/software/emacs/elisp/html_node/Filling.html

Book Reviews

I have finished both new python books, but I forgot how hard it is to write book reviews. Will try to post on them when I have re-read and taken notes!

A very cool idea

My python blog stream picked up http://www.snakebite.org/ today. It is a farm of computers with lots of different OSs on it so that Python can be constantly built and made to build on them. It is an amazing idea. I have always wondered how a sparse organisation, like a open source project, have the resources to build on so many platforms. It turns out they just rely on their members to have the OSs laying around to build on. Now Python doesn't have that problem. Big Win.

Name on Patent

My company just got one of its patents through and my name is on it. See it here For the record it is: European Patent Application No. 07712914.6 - Image Design System in the name of Serverside Group Limited - Our Ref: 601256PEP (066371-0075)

New Books

Amazon delivered "Expert Python Programming" and "CherryPy Essentials" today. Once they are read reviews will be written.

CherryPy

I am loving CherryPy. I just ported my star tracker to it in about 2 hours. Maybe I will put it up somewhere. Or just bring it back at work.

.emacs

Is it wrong to be becoming increasingly proud of my .emacs file? It is still very small, but I love it. Maybe it has something with slowly learning some lisp. Now I have to find a good way to sync it between work and home.

Running CherryPy behind Apache

There are a couple of options on to run CherryPy behind Apache. I have chosen to go for mod_python as the mod_rewrite runs the CherryPy server at the same time (from what I can tell) and it seems to be a little silly to run two servers at once.

It all comes down to the Apache config (which goes in /usr/local/etc/apache22/Includes/cherrypy.conf on my laptop) once you have mod_python installed and in your base Apache config

The actual configuration file for CherryPy takes the following form: <Location "/"> PythonPath "sys.path+['<path to any python modules required>']" SetHandler python-program PythonHandler cherrypy._cpmodpy::handler PythonOption cherrypy.setup <python module to run>::<function in module to run> PythonDebug On </Location>

so on my system with a basic app:

<Location "/">
    PythonPath "sys.path+['/usr/local/www/cherrypy/']"
    SetHandler python-program
    PythonHandler cherrypy._cpmodpy::handler
    PythonOption cherrypy.setup myapp::serverless
    PythonDebug On
</Location>

My references: The CherryPy wiki on the subject Mod_Python docs Now just to write some code that actually makes use of it all :)

Upgrading to FreeBSD 7.1

FreeBSD 7.1 is out (see here)