Printing actual SQLAlchemy queries

Programming, , , ,

Hi there. Long time no post.

I just spent a couple hours debugging a query with a DISTINCT ON clause. The DISTINCT ON clause is a PostgreSQL-specific extension to the standard DISTINCT clause, which lets you specify the columns on which you want PostgreSQL to operate for filtering out duplicate rows.

With the SQLAlchemy ORM (using the declarative extension,) here is one way to write a query to retrieve all unique names from a name table:

>>> q = DBSession.query(model.Name).distinct(model.Name.value) \
                 .order_by(model.Name.value)

Now, if you’d want to see what kind of SQL query the ORM will generate you’d print it like so:

>>> print str(q)
SELECT DISTINCT name.id AS name_id, name.value AS name_value
FROM name ORDER BY name.value

But as you can see, there’s no DISTINCT ON here. Dang. Continue reading Printing actual SQLAlchemy queries →

Share

Continuous integration on a zc.buildout-managed project with pyzmq

Programming, , , ,

Sometimes things are simple

So we’ve got that awesome Hudson server, checking out our projects at each commit, running all the tests and reporting on the outcome. That way we always know when something broke, what, and which committer just fell out of his Ballmer peak.

On that server we install as little stuff as possible and any non-Python dependency required for compiling python packages at install time is handled by buildout. That way we can keep our Hudson environment as clean as possible, preventing version conflicts and cruft accumulation.

For instance, the buildout for a project depending on GeoIP-Python contains the following snippet. It downloads and compiles GeoIP in a directory private to the project, then uses it when installing GeoIP-Python:

[geoip-python]
recipe = zc.recipe.egg:custom
eggs = GeoIP-Python
include-dirs = parts/geoip/include
library-dirs = parts/geoip/lib

[geoip]
recipe = zc.recipe.cmmi
url = http://geolite.maxmind.com/download/geoip/api/c/GeoIP-1.4.6.tar.gz

Sometimes not

Enter pyzmq.

You would think that the following would work, but it doesn’t:

[pyzmq]
recipe = zc.recipe.egg:custom
egg = pyzmq
include-dirs = parts/zeromq/include
library-dirs = parts/zeromq/lib

[zeromq]
recipe = zc.recipe.cmmi
url = http://download.zeromq.org/zeromq-2.1.6.tar.gz

The solution

The pyzmq build scripts do things differently, and they override the provided directories for various reasons. This is kind of annoying, but I can live with it, as we can get the desired result by settings a few environment variables:

[pyzmq-env]
ZMQ_DIR = ${buildout:parts-directory}/zeromq
LIBRARY_PATH = ${buildout:parts-directory}/zeromq/lib
LD_LIBRARY_PATH = ${buildout:parts-directory}/zeromq/lib

[pyzmq]
recipe = zc.recipe.egg:custom
egg = pyzmq
environment = pyzmq-env

[zeromq]
recipe = zc.recipe.cmmi
url = http://download.zeromq.org/zeromq-2.1.6.tar.gz

The catch

Beware that in either case, the declaration of buildout:parts needs to have the above parts at the beginning. Order matters here, as it obviously does no good to install pyzmq (which could very well happen when buildout runs setup.py develop on the project) before zeromq is compiled.

This is mainly annoying because the project has a hudson-specific buildout file that contains these compiling parts, as developers use another buildout so they can use their local system libraries and not have to compile everything. This means that the bulk of buildout:parts has to be duplicated in both files as the += operator would otherwise add the additional parts at the end, thus breaking the build.

I’ve not yet found a way to insert parts at the beginning.

That’s it, happy testing!

Share

Django localeurl and reverse()

Programming, , , , , ,

If you use django-localeurl, you might run into the issue that on some machines, django.core.urlresolvers.reverse() works just fine and returns a URL with the locale prepended, while on other machines, it does not.

The reason is that, when localeurl is installed, it monkey-patches django.core.urlresolvers.reverse to change its behavior and have it prepend a locale to whatever URL is to be returned. And what happens is that the load order of the different files Django needs can change from one machine to another, thus makes the monkey-patching occur at a different timing, and you end up importing on some occasions the unpatched version of reverse().

The real solution, according to the localeurl author themselves, would be to devise a solution for localeurl which doesn’t involve monkey-patching at all.

Until this is done, here’s a work-around that works, even if a bit ugly. Instead of:

from django.core.urlresolvers import reverse
...
reverse('xyz')

Try this, you’ll make sure you always get the patched version:

from django.core import urlresolvers
...

urlresolvers.reverse('xyz')


Share

Opendata in Montreal

General, , ,

Just came out of the first Montreal Ouvert open meeting.

I was pleasantly surprised to see so many interested people showing up, and from so many different backgrounds. Developers, entrepreneurs, municipal employees, students, journalists, and more that I forgot about of course.

In the end they were all just citizens, enthusiastic about the possibilities of open data, ready to put in some grey matter and some work to help make it a reality. For some the work began some time ago, as much as a few years back.

And the possibilities of open data are many: availability of fresh, constantly updated data leads to increased government transparency, increasing its accountability and usefulness. It also means more tools for the population to get involved in the process and take informed action, making democracy work better.

Having data available in open, documented formats will tremendously benefit the city itself, as its own employees often struggle to obtain data from silo-ed, undocumented and not always fresh data.

And open data by the way is not only about having the government publish the data it has, it’s also about the free flow of data in general, as much as practically/legally possible, in both directions. Quality data exists out there which could complement what the government is producing, potentially making it more aware of what’s happening, sooner, making it again more responsive and relevant.

And last but not least, the economic consequences are real and tangible. Head over to Montreal  Ouvert’s blog and you’ll find multiple reports of other cities that are going through the process and reaping the benefits. The numbers are just staggering, both in the form of savings and local business generated.

So this first meeting brought all these different people together, and provided a space for a discussion to happen. All kinds of ideas where exchanged on how to make it happen, information about what has worked for other cities, about the realities and limitations of those and in the end, what would be the best things to do to make it work in the interest of all parties.

All in all, very exciting!

Montreal Ouvert as a collective is already pretty focused, and I’m sure there was enough material tonight to help it gather even more steam. At the very least, tonight’s meeting will have served to rally all sorts of isolated efforts, thus increasing greatly the potential power to make change happen, in the interest of everyone.

So, make sure you follow their blog, and contribute your ideas and comments wherever you can.

Share

On taking action

General, Life, , , , ,

So what happened in all that time since my last substantial blog post?

No, I wasn’t busy counting red cars passing on the street, or marveling at the cat’s anatomy as my kids do.

After mulling over the possibility for a while, I finally decided to quit my job and go solo, and that turned out to be one of the greatest, most satisfying choices I have ever made!

But it wasn’t an easy one. Continue reading On taking action →

Share

jQuery.getJSON() callback not firing with IE

Programming, , ,

Just a reminder to myself, so I won’t waste as much time if it happens again.

When a jQuery.getJSON() callback is called by Firefox but not by Internet Explorer, check if the JSON data returned really is squeaky-clean.

An extra comma at the end of an object declaration was doing okay with the more forgiving Firefox JSON parser, but tripped IE’s, with no error message, and no callback ever called.

I found the error by using jQuery.load() instead and eval()-ing the AJAX responseText in the callback, which threw a JavaScript error this time.

Share

On taking responsibility

Life, ,

And so I was asking myself today, after realizing that maybe, my current situation at work might not support in a sufficient way my personal and professional growth anymore, what would it take to reach out to the next level?

What is it that would be needed in order to break out of the protective shell that is a regular job, with its predictable income stream and limited responsibility?

Being employed as a programmer is, up to a point, pretty comfortable. You don’t have to worry all that much about getting new customers, and about accounting, office space, and a whole lot of other headaches.

In the end, you don’t have to take responsibility for those things. Somebody else does that for you. And I believe that this is the whole point: little responsibility.

Taking responsibility

A few years ago, I was looking at the rent our family paid each month to the landlord and realized that I would rather keep this money for myself, so we bought a duplex house and moved in. Not only the monthly payment went hereafter to capital repayment (minus interest of course,) but we got to collect a rent check every month for the second unit!

We went from tenants, with zero responsibility for the building and not much more for the apartment, little freedom about how we could rearrange or renovate the place, and total loss of the money we paid each month, to owners with full responsibility, money starting to pile up as well as coming partly from somebody else’s pocket.

Yes, sometimes a water tank needs to be replaced, a roof repaired, but in the end the trouble is way smaller than the benefits.

And now things are beginning to look an awful lot similar to me, professionally-wise. I get the paycheck in exchange of not having to worry inside that quite nice and shinny cage.

It looks that a lot of us are just happy to trade responsibility, freedom and expanded wealth-building abilities, for the easy ride, the one we can safely travel blindly without getting too much bothered.

Don’t get me wrong, I don’t want to come across as judgemental: I don’t think being an employee is intrinsically worse that being self-employed or an entrepreneur. It all depends on what you want to achieve, and what phase of your life your are in. Do what fits you best, and as a matter of fact I have no regrets whatsoever to having had a job for so long. It’s just that I’m now wondering, how farther could I go without the constraints a job affords.

So what’s next?

Well, that is the part which is not all clear yet, but I think the next step would be to define exactly where I’d like to go. Who am I, what are my strengths, what is the value that I could offer to the marketplace?

I guess my actual skills would make for a good start, in addition to the numerous improvements that I envisioned for our processes at work but was not allowed to implement.

Let’s see what comes off it.

Share

Web hosting adventures

Internet,

While A2Hosting has a pretty darn good bang for the buck, I recently had to look for a web host with even more advanced features. For instance, I needed to be able to compile a more recent version of the git DVCS, and use it on my account of course.

So I looked around some more and settled on WebFaction. I had wanted to try them out for a while now based on all the good reviews you can find on the Internet on them, and now that I have I must say that I’m pretty impressed.

I was able to compile git using my SSH account, along with the dependencies to be able to use git-svn, and it all went pretty flawlessly.

The control panel at WebFaction is awesome, easy to use, and it seems one can host just about everything on their servers: anything PHP, all sorts of Python and Ruby frameworks, it just looks fantastic.

In most cases you are running your own private Apache daemon, so you control your own configuration, add the modules you want, etc.

Pretty good stuff, check them out!

Share