8th Light Blog: Storing Binary Data in Postgres? Beware!

Storing Binary Data in Postgres? Beware!

Micah — Sun, 18 Feb 2007 06:05:00 +0000

If your situation matches the following conditions, beware!

You’re working in rails.
You’re using Postgresql.
You’re storing binary data in the database.

This was the situation on a project of mine. We were storing PDFs and PNG images in our Postgrs database. Everything was fine during development and testing where we used files that ranged up to a few dozen kilobytes in size. The situation went rapidly downhill when file sizes got up in the hundreds of kilobytes to megabytes. The worst part about it was that the errors we got were misleading and seemingly random. They included:

undefined method `<<' for nil:NilClass
invalid end of buffer
undefined class/module Packet (Packet being a model class which was definitely defined)

These errors sent us on a wile goose chase. The clue that finally pointed us toward the problem was the fact that it took 6 seconds to load a 4M PDF document from the database. That was far too long especially considering the same document could be loaded from a file instantaneously.

Apparently, binary data stored in Postgresql’s bytea data type has to be parsed on save and load to escape and unescape certain characters. Unfortunately the native C postgres gem doesn’t do the parsing. It’s done in the PostgreSQLAdapter.unescape_bytea and PostgreSQLAdapter.escape_bytea methods (Ruby code) of ActiveRecord and the parsing is a bit too intensive for Ruby. This is where the meltdown begins. It consumes too much memory, or too much processing power, or … well I don’t know exactly. But I know it breaks.

We refactored our model such that all the binary data gets stored in flat files on disk rather than in the database. After this, our Rails app came back to life. It was MUCH faster too!

Here’s hoping that if and when you encounter this problem, Google points you to this blog entry and you find it helpful.

"Storing Binary Data in Postgres? Beware!" by fotos@funtasia.gr

Mon, 16 Jul 2007 17:26:11 +0000

Well you are mostly correct. The ruby escaping/unescaping happens only if you use the pure ruby postgresql adapter and not the native one.

If you install this gem: postgres-pr then you get the pure ruby (pr) adapter. If you install this gem: postgres you get the native (as in C) adapter (homepage: http ://ruby.scripting.ca/postgres/). I think it also goes by the name ruby-postgres (http://rubyforge.org/projects/ruby-postgres/).

You can see that from activerecord/lib/activerecord/connectionadapters/postgresqladapter.rb that if the PGconn object responds to the ‘escapebytea’ method it wi ll try to do it that way. And the native adapter does it in C, using the libpq interface (http://www.postgresql.org/docs/8.2/static/libpq-exec.html#LIBPQ-EXEC-ESCAP E-BYTEA), which should be fast enough.

If you run windows you can get the native adapter (precompiled) from http://www.vandomburg.net/pages/postgres-ruby-windows.

Also if you use PostgreSQL 8.2 you need to patch postgresql_adapter and add an ‘E’ for the quoting of binary strings.

Have fun and do a better research next time! :)

-fot

"Storing Binary Data in Postgres? Beware!" by Fotos Georgiadis

Wed, 04 Jul 2007 19:55:13 +0000

Well you are mostly correct. The ruby escaping/unescaping happens only if you use the pure ruby postgresql adapter and not the native one.

If you install this gem: postgres-pr then you get the pure ruby (pr) adapter. If you install this gem: postgres you get the native (as in C) adapter (homepage: http://ruby.scripting.ca/postgres/). I think it also goes by the name ruby-postgres (http://rubyforge.org/projects/ruby-postgres/).

You can see that from activerecord/lib/activerecord/connectionadapters/postgresqladapter.rb that if the PGconn object responds to the ‘escapebytea’ method it will try to do it that way. And the native adapter does it in C, using the libpq interface (http://www.postgresql.org/docs/8.2/static/libpq-exec.html#LIBPQ-EXEC-ESCAPE-BYTEA), which should be fast enough.

If you run windows you can get the native adapter (precompiled) from http://www.vandomburg.net/pages/postgres-ruby-windows.

Have fun and do a better research next time! :)

-fot