vicious.red.beam

In every game / movie which involves wizards, those guys are wearing robes… And I’m constantly amazed about it.

What is wrong with them? They’re up against warriors, demons, dragons and who knows what else wearing a robe that can protect them from the rain mostly. Any lame archer should be able to shoot them in the back most of the time, because they go to battle in next to nothing.

So you’re going to say that the armour limits their magical potential to cast spells probably… Fair enough, an interesting point. But that doesn’t make much sense to me either. I’m not talking about playing with telekinesis here, I’m talking about real wizards throwing fireballs at you, or creating a blizzard when they need one. I’m talking about guys who summon minions from hell and squash you with a meteor. That’s unreasonable to think some small sheet of metal will limit them. And if it’s too heavy, they can just do something about it, they’re mighty wizards after all!

So why bother with the fucking robe? The only worthy exception from the rule is Alphonse from the Fullmetal alchemist - he IS an armour and a kind of a wizard. So there - it’s possible, now stop with the battle bathrobes and let’s see a spell-casting brotherhood of steel finally, please, ok?

(no, no idea where that came from)

Some mobile versions of websites out there are silly and limiting, just an afterthought because you have to do out for some unspecified reason. Some others however are the greatest thing ever, because the interaction was very well designed. Not just how it looks, but the whole workflow that just matches the most common tasks. I wish my bank for example would make their main site behave like its mobile equivalent. The long content, faqs, links to other sections could be moved away to some place that doesn’t take 20% of the screen space.

So why not design like that in the first place? Map the most common interactions and build the site around them, only then add the long content in a way that doesn’t disrupt the flow.

Today I played a little with preseed, virt-install and similar things. There’s a couple of things I learnt that were hard to find, so here they go:

Polipo is cool for caching installation data if you plan to reinstall often. For example if you’re trying to install something with the config of network network=default, you can simply fire up polipo on your host machine and add to configuration (/etc/polipo/config):

proxyAddress = "0.0.0.0"
allowedClients = 127.0.0.1, 192.168.122.0/24

The localhost address is for usage from the host machine, the 192.168 address is for usage from the default libvirt network. Now you can use it to speedup installation via debian installer. In order to cache d-i itself, add http_proxy before your virt-install. For example:

sudo http_proxy=http://localhost:8123 virt-install ...

That will not affect where the packages are downloaded from. To do this you need to put the following in your preseed:

d-i mirror/http/proxy string http://192.168.122.1:8123

unless you want to actually mirror the whole debian repo on your disk. In that case - go ahead :)

Libvirt/kvm permissions are still there in squeeze, new ubuntus, etc. I have no idea why this doesn’t get fixed since lucid (hardy?) times, but for some reason virt-install and virt-manager get permission denied errors on libvirt’s socket file. After trying to debug it for a while and a long googling session, I’m back to the old solution again - set user and group in /etc/libvirt/qemu.conf to “root”. I surrender and don’t think it’s worth the effort. You do not need to disable apparmor though as some posts suggest, just make sure you restart the libvirt-bin daemon.

Libvirt can also complain about virtualisation platform not being available. What it really means is that qemu-kvm package is missing. Not a very good error description…

Location is what you need. —location that is, if you want to install via virt-install using preseed. It would be awesome if you could use the netboot cd (at least I think it would be), but that’s not possible. You need to point location at some url/path which contains the distributions debian-installer. For example

--location http://ftp.uk.debian.org/debian/dists/squeeze ↵
    /main/installer-amd64/

should do the trick.

Auto option is actually “auto=true”, not “auto” itself. At least in all recent distributions. This was not the case in the past, so watch out for old guides.

Stuck at hostname? This happens with dhcp setup. Instead of trying to specify the hostname in preseed, set it in —extra-args. Preseed script seems to be downloaded only after the host and domain are defined. For example:

--extra-args="auto=true url=http://192.168.122.1/preseed.cfg ↵
    hostname=temphost domain=tempdomain"

Some time ago I finally pushed the memcachedliked package to bitbucket (https://bitbucket.org/ViciousRedBeam/memcacheliked/src). It allows creating very simple servers which behave like memcached, but can use whatever data backend you want.

Why? Because you don’t need more interfaces than you already have. There are loads of libraries implementing memcache access in lots of languages. Why not reuse those instead of implementing another wire protocol? Many data exchange problems are limited to some put/get/remove interface anyway.

In addition, lots of other projects already allows talking to a memcached server, so if you have some other backing store, you can wrap it in memcache semantics, instead of trying to figure out how to add another import/export plugin to that product.

This project is not supposed to be fast, scalable, entreprisey, beautiful, etc. It’s supposed to save me (and maybe someone else) from reaching for SocketServer class and something similar on the client side. To get the feel for how it works, there’s a sample attached (https://bitbucket.org/ViciousRedBeam/memcacheliked/src/default/memcacheliked/sample.py) which implements the standard operations of memcached (with flags). It should be clear enough to get you started on your own backed.

This is not another one of those “git workflow” posts. Of course there’s http://nvie.com/posts/a-successful-git-branching-model/ and other “workflow” posts everyone will direct you to. They are all useful in some scenarios, but I found one issue missing in most of them: versioning your files. Not in a VCS sense, but rather in keeping the version numbers assigned to some revision and keeping the metadata clean.

I rarely work with software installed from source. If I do, it’s some temporary local build. Most of the time, I deal with packages, so this article might not correspond to what you’re used to.

It’s cool to keep the version in tags and just install the software directly from source. But if you iterate quickly and need to label your versions in a way where the source references the versions too, you’ll have a number of problems. The most obvious ones are: what version are commits between the versions? what version to put in the documentation before merging? how to reference the version you’re working on? These are very related ones of course.

Some of this may be obvious to many developers because of the workflow they use, but I run into those issues in both my projects and external ones enough times that I decided to write this down. So to start with - why would you reference current version in the file itself? There are many reasons, but my top 3 are:

• You’re writing something in documentation in branch feature/shiny-stuff: “… this function makes you feel good. Available in version >= x.y.z.”
• You work with metadata. For example every chef cookbook has a file metadata.rb which contains its version.
• Dependencies on versions which don’t exist yet. They do, but are in a different branch and your current branch does not really require them to build, so they’re developed separately.

The answer to most of the issues is - don’t change the version. Postpone it until you absolutely have to do the change. Then don’t change it in the branch itself, but rather on your release / master / other branch, depending on the workflow. That will allow you to avoid some bad surprises in the future:

Issue 1: Asynchronous branches. You’ll never know when someone else wants to merge their branch. If you worked on 1.5.0 and put the text “1.5.1” somewhere in your code, assuming that will be the next version, you will have to correct it later, possibly adding another commit and confusing someone browsing the history months from now, because actually you merged into 1.5.3.

Solution: Choose a marker. “X.Y.Z”, “{VER}”, “(current)”, whatever. Whenever you reference current version, use the marker. When you actually change the version (possibly on the merge commit itself), change the marker to the actual version it was implemented in. If you can afford it, make sure every commit on the master branch has its own version, so this happens on every merge and markers do not linger on in the main branch.

Issue 2: Reverting changes. When you try to revert some branch from long time ago, you’re reverting functionality, not versions. The version will actually go up, while you remove the invalid change. That means after the revert, you shouldn’t see your version string change and you shouldn’t see version conflict saying “you’re at 2.5, but the commit revert is supposed to change from 1.8 to 1.7”.

Solution: Don’t put the version change together with the code change. Change the version in the merge commit (merge —no-ff —no-commit). This way all code modifications are “version-free”.

Issue 3: What to call packages created from commits in between the releases?

Solution: Version all commits on the master / release branches if you can afford it. Branches logically become x.y.z-branch-commithash in that configuration. If you cannot do that, I recommend calling them (in debian world at least) with current.version-suffix, rather than next.version~suffix and that’s because of the previous points - you do not know what the next version will be. It might be 0.0.9, 0.9.1, or 9001. You might be planning for one thing and then revert everything because of a big change being completed faster than expected.

In summary: version every commit on master, change actual version strings only in merge commits, don’t mention any specific version in branches and you should stay away from most problematic situations.

Of course many workflows and developers will disagree. You’ll have to either decide which workflow suits you better, or challenge them to a knife fight. Either way, good luck.

For some time now I was looking for some sane music streaming service that works in UK. I didn’t mind paying for it too if it turned out to be good. Unfortunately according to http://en.wikipedia.org/wiki/List_of_online_music_databases#On-demand_streaming_music_services the situation sucks. Maybe not for everyone, but definitely for people in Europe. Ok - for people who try to make an informed choice at least.

The choices I have are: 8track, Deezer, Grooveshark, Last.fm, Mflow, Qriocity, Spotify, we7.

What I didn’t even try:

• Spotify because it requires a Facebook account. That’s a no-no for me.
• Qriocity because I refuse to use anything from Sony (due to security issues, spying on users, suing developers, etc. etc.)
• we7 because the player on their website simply didn’t work. At all. That’s not a good impression for a service offers… playing music.

Others were:

8track: looks nice, but this is not something I was looking for. I want to listen to complete albums.

Last.fm: I used it for a long time, but since it transformed into a social platform instead of a simple player it’s just annoying to use. I want to search for an album and play it. The less cruft there is around, the better. If I wanted to socialise I’d go to facebook instead, or you know… outside ;)

Grooveshark: it was cool for some time. Unfortunately they need fail hard at handling the metadata. Many albums are duplicated over and over again. Many albums have duplicate tracks or are missing some. Did I mention I just want to search for an album and play it? It would be great if they could add some moderation to their database - maybe I’d stay then. Also sometimes playback just stopped randomly.

Mflow: I tried a couple of queries but the tracks were not there, so I didn’t really look further.

Deezer:

Finally - a great service. I can actually listen to an album or create own playlists. There are still missing tracks in some albums but they handle it very nicely by showing the track, only greyed out. They provide the option of artist-based radio in case you get bored and you can upload your own tracks.

Although they really want you to join your account with Facebook you can skip this (unfortunately the “facebook-like” bar floating at the bottom of the screen will stay there anyway, sometimes preventing you from using some other elements of the page). They give around a month of free usage - first 2 weeks when you create an account, then 15 days after you decide to pay.

They do have some slight problems with the interface, but nothing preventing normal usage. Most annoying one is that they don’t like unicode. On import of mp3s they change the non-ascii characters to “?”, but you can still edit the names properly afterwards. All officially published albums seem to be stripped from accents - unfortunately search doesn’t change your query in the same way. Your uploaded tracks don’t show up on global search either.

All in all, it’s a good service and they’re going to get the monthly fee from me for a standard account. I’m not sure the pricing for another level is reasonable though - twice the price to be able to listen on your phone? I could pay that once to get their app, but not every month.

A day has been wasted… Apparently some versions of Ubuntu and Debian installers accept “auto”, but some require “auto=true”. Some require to manually kill the network and “netcfg” after installation (even after configuring static networking), some are sane.

I’d like it if there was an apt-get’able package containing dhcpd, tftpd, thttpd web interface for configuring the images + preseeds. If it was integrated with a puppet master, I’d be even happier.

Maybe one day…

Just a note that Memcacheliked module is published now… I’ll write up the how and why later on.

(Source: bitbucket.org)

Making a mistake in a Python metaclass? It’s like you’re aiming at your foot, but when you shoot, a bullet exits from all your body parts at once…

Here’s some useful, short code for processing website content. Think of it as an object mapper. No relations here, and only one way… so not an ORM, but it’s interesting to use anyways:

class PageParsingException(Exception): pass
class ModelParser(object):
   prefix_multi = False

   def __init__(self, page):
      if not hasattr(self, 'model'):
         raise PageParsingException("model not defined")

      if type(page) == str:

         builder = html5lib.treebuilders.getTreeBuilder("lxml")
         p = html5lib.HTMLParser(tree = builder)
         page = p.parse(page)

      if hasattr(self, 'prefix'):
         nodes = page.xpath(self.prefix)
         if len(nodes) > 1 and not self.prefix_multi:
            raise PageParsingException("prefix ambiguous")
         elif len(nodes) == 0:
            raise PageParsingException("prefix not found")

         page = nodes[0]

      for name, val in self.model.items():
         path, klass = val
         xpath_result = page.xpath(path)

         if type(klass) == type and issubclass(klass, ModelParser):
            setattr(self, name, [klass(x) for x in xpath_result])
         elif klass == str:
            setattr(self, name, ''.join(str(x) for x in xpath_result))
         elif hasattr(klass, "__call__"):
            setattr(self, name, klass(xpath_result))
         else:
            raise PageParsingException("unknown result class <%s>" % (klass,))

What does it do? Basically you can define a model of what needs to be found on the website you’re trying to scrape - using xpath in most cases. For example this:

class SomeTable(ModelParser):
   prefix = '''//table[@class="some_table"]'''
   model = {
         'item': ('''.//td[@class="one"]/text()''', str),
         'processed': ('''.//td[@class="two"]/text()''', custom_function),
         'other': ('''.//td[@class="three"]''', OtherModel),
         }

can be used by calling:

SomeTable(page_text)

in order to get an object with the ‘item’, ‘processed’ and ‘other’ fields. Of course there are similar frameworks out there. Unfortunately most of them are a part of some bigger project (eg. Scrapy) which might be simply not needed. This one is short and simple - just drop it in your project and use.