Category Archives: Programming

Printing actual SQLAlchemy queries

Programming, , , ,

Hi there. Long time no post.

I just spent a couple hours debugging a query with a DISTINCT ON clause. The DISTINCT ON clause is a PostgreSQL-specific extension to the standard DISTINCT clause, which lets you specify the columns on which you want PostgreSQL to operate for filtering out duplicate rows.

With the SQLAlchemy ORM (using the declarative extension,) here is one way to write a query to retrieve all unique names from a name table:

>>> q = DBSession.query(model.Name).distinct(model.Name.value) \
                 .order_by(model.Name.value)

Now, if you’d want to see what kind of SQL query the ORM will generate you’d print it like so:

>>> print str(q)
SELECT DISTINCT name.id AS name_id, name.value AS name_value
FROM name ORDER BY name.value

But as you can see, there’s no DISTINCT ON here. Dang. Continue reading Printing actual SQLAlchemy queries →

Share

Continuous integration on a zc.buildout-managed project with pyzmq

Programming, , , ,

Sometimes things are simple

So we’ve got that awesome Hudson server, checking out our projects at each commit, running all the tests and reporting on the outcome. That way we always know when something broke, what, and which committer just fell out of his Ballmer peak.

On that server we install as little stuff as possible and any non-Python dependency required for compiling python packages at install time is handled by buildout. That way we can keep our Hudson environment as clean as possible, preventing version conflicts and cruft accumulation.

For instance, the buildout for a project depending on GeoIP-Python contains the following snippet. It downloads and compiles GeoIP in a directory private to the project, then uses it when installing GeoIP-Python:

[geoip-python]
recipe = zc.recipe.egg:custom
eggs = GeoIP-Python
include-dirs = parts/geoip/include
library-dirs = parts/geoip/lib

[geoip]
recipe = zc.recipe.cmmi
url = http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz

Sometimes not

Enter pyzmq.

You would think that the following would work, but it doesn’t:

[pyzmq]
recipe = zc.recipe.egg:custom
egg = pyzmq
include-dirs = parts/zeromq/include
library-dirs = parts/zeromq/lib

[zeromq]
recipe = zc.recipe.cmmi
url = http://download.zeromq.org/zeromq-2.1.6.tar.gz

The solution

The pyzmq build scripts do things differently, and they override the provided directories for various reasons. This is kind of annoying, but I can live with it, as we can get the desired result by settings a few environment variables:

[pyzmq-env]
ZMQ_DIR = ${buildout:parts-directory}/zeromq
LIBRARY_PATH = ${buildout:parts-directory}/zeromq/lib
LD_LIBRARY_PATH = ${buildout:parts-directory}/zeromq/lib

[pyzmq]
recipe = zc.recipe.egg:custom
egg = pyzmq
environment = pyzmq-env

[zeromq]
recipe = zc.recipe.cmmi
url = http://download.zeromq.org/zeromq-2.1.6.tar.gz

The catch

Beware that in either case, the declaration of buildout:parts needs to have the above parts at the beginning. Order matters here, as it obviously does no good to install pyzmq (which could very well happen when buildout runs setup.py develop on the project) before zeromq is compiled.

This is mainly annoying because the project has a hudson-specific buildout file that contains these compiling parts, as developers use another buildout so they can use their local system libraries and not have to compile everything. This means that the bulk of buildout:parts has to be duplicated in both files as the += operator would otherwise add the additional parts at the end, thus breaking the build.

I’ve not yet found a way to insert parts at the beginning.

That’s it, happy testing!

Share

Django localeurl and reverse()

Programming, , , , , ,

If you use django-localeurl, you might run into the issue that on some machines, django.core.urlresolvers.reverse() works just fine and returns a URL with the locale prepended, while on other machines, it does not.

The reason is that, when localeurl is installed, it monkey-patches django.core.urlresolvers.reverse to change its behavior and have it prepend a locale to whatever URL is to be returned. And what happens is that the load order of the different files Django needs can change from one machine to another, thus makes the monkey-patching occur at a different timing, and you end up importing on some occasions the unpatched version of reverse().

The real solution, according to the localeurl author themselves, would be to devise a solution for localeurl which doesn’t involve monkey-patching at all.

Until this is done, here’s a work-around that works, even if a bit ugly. Instead of:

from django.core.urlresolvers import reverse
...
reverse('xyz')

Try this, you’ll make sure you always get the patched version:

from django.core import urlresolvers
...

urlresolvers.reverse('xyz')


Share

jQuery.getJSON() callback not firing with IE

Programming, , ,

Just a reminder to myself, so I won’t waste as much time if it happens again.

When a jQuery.getJSON() callback is called by Firefox but not by Internet Explorer, check if the JSON data returned really is squeaky-clean.

An extra comma at the end of an object declaration was doing okay with the more forgiving Firefox JSON parser, but tripped IE’s, with no error message, and no callback ever called.

I found the error by using jQuery.load() instead and eval()-ing the AJAX responseText in the callback, which threw a JavaScript error this time.

Share

JavaScript date parsing woes

Programming, ,

I just corrected an interesting bug in a web application’s client-side JavaScript code.

How come, most of the time, the string ‘2009-02-01’ gets parsed into a Date object with the correct February 1st, 2009 value, but sometimes results in March 1st, 2009?

This particular piece of code builds a financial periods menu by extracting dates out of some HTML returned by an AJAX request. Here’s what the function looks like – it may not be the best way to construct a date from a string in the form yyyy-mm-dd, but it works:
function makeDateFromIsoString(isoString)
{
if (isoString == null)
return null;
var date = new Date();
date.setFullYear(isoString.substr(0, 4));
date.setMonth(isoString.substr(5, 2) - 1);
date.setDate(isoString.substr(8, 2));
date.setHours(0);
date.setMinutes(0);
date.setSeconds(0);
return date;
}
That is, it works most of the time, but not at the end of some months, as I discovered.

Take yesterday, for instance – March 31st, 2009 – and let’s look at what makeDateFromIsoString(‘2009-02-01’) returned:

  • var date = new Date() created a new Date object, containing the current date and time: March 31st, 2009.
  • date.setFullYear(isoString.substr(0, 4)) changed the year of the data object to  ‘2009‘, so its value stayed the same.
  • date.setMonth(isoString.substr(5, 2) – 1) changed the month of the date object to  ‘02‘ (minus one, because months in JavaScript are zero-based,) so its value became February 31st, 2009. Of course, February 31st doesn’t exists, so because the real date three days after February 28th, 2009 is March 3rd, 2009, the date object now contained that value.
  • So by the the time the code was done with date.setDate(isoString.substr(8, 2)), the date value had become March 1st, 2009.

And this is how February became March, but only during the three last days of March. Interesting twist, isn’t it?

The fix was simple: call setDate() before setMonth().

Viva la Firebug for helping me nail this one down.

Update: In the end, I just replaced the whole function body with this: return new Date(isoString.replace(‘-‘, ‘/’, ‘g’)). Much simpler, and not subject to the above subtle, but interesting nevertheless, bug.

Share