Ross Patterson's Blog - Posted in 2010

Migrating a blobstorage

2010-12-01T00:00:00+00:00

Addressing the “ZODB.blob WARNING (PID) The lawn blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the bushy layout.” warnings.

Update: Filed a ZODB bug.

I’ve been getting he following log message ever since I migrated to ZODB BLOBs:

ZODB.blob WARNING (PID) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.

It’s not obvious what to do about this, even after initial googling, so I thought I’d document it. Firstly, we need to use the ZODB.scripts.migrateblobs script so create a console script for it that has all the necessary eggs available. Unfortunately, the ZODB3 egg doesn’t define a console_scripts entry point for this script so I used zc.recipe.egg to work around this:

[buildout]
...
parts +=
    zeo
...
    migrateblobs
...
[migrateblobs]
recipe = zc.recipe.egg
eggs = ZODB3
entry-points = migrateblobs=ZODB.scripts.migrateblobs:main

After re-running buildout and shutting down ZEO, I used the generated script to migrate the blobstorage:

$ mv var/blobstorage var/blobstorage-lawn
$ bin/migrateblobs var/blobstorage-lawn/ var/blobstorage
(PID) Blob directory `var/blobstorage-lawn/` has layout marker set. Selected `lawn` layout.
(PID) The `lawn` blob directory layout is deprecated due to scalability issues on some file systems, please consider migrating to the `bushy` layout.
(PID) Blob directory '.../var/blobstorage/' does not exist. Created new directory.
(PID) Blob temporary directory 'var/blobstorage/tmp' does not exist. Created new directory.
Migrating blob data from `var/blobstorage-lawn/` (lawn) to `var/blobstorage` (bushy)
        OID: 0x###### - 1 files
...

Now I’m able to fire up ZEO and I can access AT file and image objects fine without the log warning.

Updated on 01 December 2010

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

Trying out Chameleon

2010-10-07T00:00:00+00:00

My experiences with Chameleon, testing compatibility and z3c.form

I’m about to upgrade a sizable client deployment to Plone 4 and wanted to see if I could switch to the Chameleon template engine at the same time. I’m recording things I found here:

Testing both with and without Chameleon

For some add-ons of mine, I found it useful to be able to run the tests both with and without Chameleon. It turns out to be remarkably simple with buildout.

Integrating cmf.pt and z3c.form

There are some gotchas when using both cmf.pt and z3c.form together. You have to be sure to include z3c.ptcompat, which you can do with z3c.form’s poorly named “extra” extras_require. I did this in my project’s setup.py by adding the following:

      install_requires=[
...
          'z3c.form[extra]<2.3dev',
...

Unfortunately, z3c.ptcompat uses an environment variable to switch on the use of Chameleon so you have to add the environment variable to the buildout instances and test runners:

[instance]
...
environment-vars =
  PREFER_Z3C_PT True
...
[test]
...
environment = testenv
...
[testenv]
PREFER_Z3C_PT = True

Updated on 07 October 2010

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

Does the Tool Really Deserve the Credit?

2010-08-16T00:00:00+00:00

When we rave about a tool being superior because we used it to solve a given problem, is it really the tool or the familiarity with the problem we gained in learning to use the tool?

To get it out of the way, this is not about virtualenv or z3c.form. I have nothing but praise for virtualenv, I just use it very rarely. I have much to criticize about z3c.form, but that’s another topic and may be a problem endemic to all forms libraries. I’d like to keep this and any ensuing discussion to the more abstract issue of incidental familiarity and how it affects our perspectives on the value of a given tool.

I use Ubuntu Linux for my development platform (and everything else). As such, I’ve had the luxury of a pretty good OS installed python development environment. I really don’t know much about virtualenv. I do know its one of those tools that many people swear by with a bit of zeal. Some won’t assist with a problem unless you first setup a virtualenv to eliminate variables. The thing is, it’s never actually solved a problem of mine that I can think of by eliminating a variable. I have on a very few occasions found it useful to smash my way out of a horrible environment when I’m not under one of my trusty Ubuntu environments, but even then just to get up and running. So for me, it’s a tool I don’t know well that I will use only when I need to to meet someone else’s requirements or to solve a problem quickly when finding the root cause isn’t worth my time.

Plone 3, unfortunately (go Plone 4, GO!), still requires Python 2.4 and Ubuntu has, quite appropriately, recently dropped 2.4. While struggling to keep my Plone 3 buildouts running and with some RHEL deployments, I found myself reaching for virtualenv. The problems were a mess of path issues (PATH, PYTHONPATH, LD_LIBRARY_PATH, etc.). I don’t know much about these issues and I don’t want to. I prefer to delegate to my trusty OS and to buildout. In the end, though virtualenv helped somewhat, I still had to come to understand far more about that path issues than I feel I should.

It occurred to me, however, that if I’d been working on something other than Ubuntu all these years, I would have run into many more path and similar problems and I probably would have fallen in love with virtualenv. It also occurred to me, that in the process of learning to use virtualenv to help resolve these issues, I would probably have learned a lot more about all the path issues and would probably be fairly competent at comprehending such issues. I would probably also not be aware that I had gained such further familiarity and might attribute my new-found greater ease with my Python environments disproportionately to virtualenv.

This reminded me of my experience with zope.formlib and z3c.form. A while back I’d built a number of forms with zope.formlib/zope.app.form. I found it initially much more productive than Archetypes, mostly due to much more appropriate decoupling, but as time went on it seemed likely I would have to put at least as much time into understanding things I don’t want to as I have had to do with Archetypes. Given I already have the requisite familiarity with Archetypes and can be quite productive in it, I limited my investment in zope.formlib. When z3c.form came around, I was very excited at the prospect that I might get much of the benefit of zope.fomrlib forms without having to know as many things I don’t want to know. I’m now considerably more invested in z3c.form than I ever was in zope.formlib and I can safely say I need to know just about as much I don’t feel I should have to as I did with zope.formlib.

I have noted that the advocates of z3c.form often seem to find the appropriate approach to a given problem to be very clear to them and take a lot of pride in being able to point to the right place in the docs. For my experience, however, I find that even after reading the right place in the docs and reviewing several other docs, I still have to do way too much UTSL to solve my problem. I also have the strong sense that after putting this much time into learning z3c.form, the obviousness of “The Right Way” will only come to me once I’ve had to learn nearly as much I shouldn’t have to as I’ve had to do with Archetypes. I take from this that it may be another case of somewhat erroneously attributing the value of comprehension, understanding, and experience to the tool whose use lead to this familiarity largely incidentally.

(To be very clear, I don’t want to discuss z3c.form here. I think investing in some successor to AT is very necessary at this point and we may not be able to afford waiting for something else. The documentation is also absolutely worth being proud of. I think most of the value of z3c.form will be in putting our collective weight behind something. I use z3c.form and will continue to do so.)

Now for the disturbing part, I know I’ve done this very thing many times but I can’t yet list them. It’s much easier to come to see a pattern like this when looking critically at or feeling frustrated with tools I’m just using but not heavily invested in than it is to do so with tools I’m building. I hope to temper my strategic decisions for adopting and suggesting tools with this critical awareness. I also hope to temper my own perspective of the value of tools I’m developing or becoming deeply invested in. I also hope our development community can learn to better ask whether the perceived value of a given tool is being unduly influenced by the incidental familiarity we’ve developed in building or using that tool.

Updated on 16 August 2010

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

ExtensionClass ZODB Migration Script

2010-04-02T00:00:00+00:00

A "bin/instance run" script for updating ExtensionClass instances pickles in the ZODB. Useful to apply Hanno’s pickle patch.

Updated on 02 April 2010

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

Dogfood Dinner Bell

2010-04-01T00:00:00+00:00

It’s that time again to upgrade my Blog to the latest Plone.

With the first Plone 4 Beta out it’s time to eat my own dogfood and upgrade my blog. This site is now running on Plone 4.0b1-1. So come poke around if you like and tell me what’s broken. :)

The upgrade went pretty smoothly for my main site. I had some other, older, Plone portals where I’ve installed various add-ons in the past and for those I had to do some manual cleanup of some broken objects.

The unified folder implementation makes removing broken objects very difficult. The manage_delObjects(), _delObject, and _delOb methods are all broken before the folders have been migrated but the migration won’t complete if there are broken objects in the folder. I had to do something like this where “idx” is the index in _objects of the broken object and ‘foo’ is the id:

>>> folder._objects = folder._objects[:idx]+folder._objects[idx+1:]
>>> delattr(folder, 'foo')
>>> import transaction
>>> transaction.commit()

On two of my portals I also had some regular TextIndex’es in the catalog, which I guess are no longer around. Those I was able to just delete in the “Indexes” tab of the ZMI for portal_catalog. Since these were core indexes like “Title” I used portal_setup to install just the catalog import step from the Plone base profile. Being a base profile this means running the import step cleared the indexes so I also had to update those catalogs.

After all that was done I just uninstalled the classic theme and Kupu and then installed and configured plone.app.caching. Everything seems to be working well.

Just to polish things off, I wanted to make sure I was getting the full benefit of Hanno’s ExtensionClass/ZODB fix. The one thing I can’t find is a clear migration procedure. As far as I understand the ZODB, the objects created before the fix would need to be written to the ZODB again to see the benefit from the fix. So I used the following at a “bin/instance debug” prompt:

>>> app.ZopeFindAndApply(app, search_sub=1, apply_func=lambda obj, path: setattr(obj, '_p_changed', True))
>>> import transaction
>>> transaction.commit()

That should do it, but if anyone knows how that’s insufficient or knows a better way to accomplish the migration, please do leave a comment.

Updated on 01 April 2010

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.