Installation

The software is available on pgfoundry and easy enough to install on a debian system, as it supports it from the source code, using the yada debian package making facility. So it summaries as:

$ apt-get install postgresql-server-dev-8.2 python-psycopg
$ apt-get install libevent-dev yada devscripts
$ make deb82
$ dpkg -i skytools_2.1.3_i386.deb skytools-modules-8.2_2.1.3_i386.deb
$ londiste.py -h

On a RedHat system (CentOS 4), I had to manually install python-egenix-mxdatetime to be able to have python-psycopg importable and run londiste.

Important points not found in documentation

The skytools projects really lacks documentation, but comes with a test suite, and a mailing-list which archives revealed soon to be a must-read!

Now, here's the quick howto for the impatient, from things read in the archives, in the documentation, in the source code, and learnt from errors:

1. for londiste to run, you need a running pgq ticker;
2. ticker has to be installed and run on the provider site;
3. run londiste.py from the subscriber, even the provider installation step;
4. install both pgq and londiste as a PostgreSQL superuser;
5. then alter owner to the user writing into the replicated tables to the londiste schema and its tables, or GRANT your users the privileges to write into londiste.* schema and tables;
6. create yourself the table to be replicated on the subscriber site;
7. subscriber data are not enforced to be read-only, take care of it yourself;
8. londiste replicates schema.table to schema.table.

Then some constraints to respect for being able to play with the tool, some of them requires reading the source (thanksfully written in python) to understand:

9. tables to replicate have to have a primary key defined;

10. on the subscriber site, don't define any foreign key constraint, this will only give you the following message:

    ERROR found trigger on table xxx (143480)
    

    where xxx is the table name and the magic number its system oid;

11. database encoding on provider and subscriber must be the same (for copy to work sanely whatever the database content);

Running londiste and replicating data

The configuration of both londiste and pgq being straightforward, won't be cover in this document. We'll assume you've done it by now, and we're ready to replicate asynchronoulsy some tables!

On the provider site, first run the ticker:

$ pgqadm.py -d ticker.ini ticker

Then, on the subscriber site, install londiste:

$ londiste.py conf.ini provider install
$ londiste.py conf.ini subscriber install

Those two commands are to be run being connected as a PostgreSQL superuser. As the subscriber data will be read-only—remember that you have to enforce this yourself—, having data owned by a superuser seems acceptable. After installing londiste on provider, it's possible to alter ownership of londiste schema and its tables—or to simply GRANT permission on them—to a non-superuser user, the one who does the data updates for example.

When you're positive about the data users to be granted the privileges to use londiste.* tables (from triggers which insert, update and delete internal data), we can run londiste:

$ londiste.py conf.ini provider add foo bar baz
$ londiste.py conf.ini subscriber add foo bar baz
$ londiste.py -d conf.ini replay
$ tail -f londiste.conf.log

And there it is, the replication is ongoing, beginning with some splitted COPY-ing of the data from the provider to the subscriber.

Of course, you can add tables to provider and subscriber after having launched the replication manager—the replay command—, you just need to have added a table to the provider before to add it to the subscriber.

TroubleShooting

Some easy to trap errors:

• after adding a table to subscriber, provider site can't insert/update/delete into it

  The user issuing the write-queries hasn't been GRANT-ed permissions to write into londiste.* tables.

• WARNING No pidfile, process not running

  You issued some add or setup command and canceled it, using kill or a keyboard interrupt, and now londiste seems to be lost.

  First, it certainly is still running, check logs and main .pid file.

  Now, check that the pid given into a %(name)s.copy.pid or %(name)s.setup.pid is no more running on your system, then remove the file.

Testing

After having made different installations, including 3 weeks of often disconnected provider/subscriber setup, and production provider to testing system subscriber where PostgreSQL is sometimes restarted, I have confidence into the software.

The production server where londiste has been installed showed no impact on performance and on resources usage—basic top and dstat like measuring, and slow queries monitoring—, expect for when adding or removing tables to replication: it create or drop a trigger, which requires some locking.

Some more testing will have to come, though, performance stress tests using Tsung load testing tool and data validity checking. It would also be neat to have some information about replication latency—maybe pgqadm.py ticker status is simply this.