.. Ross Patterson's Blog imported post, created by `$ ./bin/rfc822-to-post` on Mar 15, 2021. .. meta:: :description: Getting a 6-7 fold decrease in zc.buildout run times. :keywords: Plone, Zope .. post:: Jan 27, 2012 :tags: Plone, Zope :author: Ross Patterson :redirect: @@redirect-to-uuid/3ce57c8a138e48d0b42914d017b34bdf ################################# Buildout Performance Improvements ################################# Getting a 6-7 fold decrease in zc.buildout run times. I've been `baffled `_ by the amount of time running buildout takes. In particular, it seemed like a lot of duplicate work was being done when multiple parts had identical or very similar sets of required eggs/distributions. After a bunch of `profiling `_ `and `_ `timing `_, I've found there are a number of ways to increase your buildout performance. Use `buildout.dumppickedversions `_ version 0.5 or later: If you're using `buildout.dumppickedversions `_, use version 0.5. This release includes `a fix `_ I contributed which yields a `3-4 fold decrease `_ in runtime if buildout is run with just the ``-N`` option and no ``-v`` options. If your're using `buildout-versions `_ instead of buildout.dumppickedversions, unfortunately `it suffers from the same problem `_. The author knows about this and may address it but until then you may want to switch to buildout.dumppickedversions. Use `zc.buildout version 1.5 `_ or later: This seems to yield a `2-3 fold decrease `_ in buildout runtime when run with ``-N`` and addresses a `similar hot loop `_ as found in buildout.dumppickedversions. In fact, it's resolution of required distributions seems to be much more efficient in general. If you're stuck with 1.4, I've `contributed `_ the same fix as for buildout.dumppickedversions to the 1.4 branch so it should make it into a next release if someone makes a new 1.4 release. Use the `-N `_ option: With the above newest releases of zc.buildout and buildout.dumppickedversions, you can now exercise a lot of control of your run times `depending on what command-line options you use `_. In particular, make sure `newest `_ is false and that `verbosity `_ is not increased. Both of these can be influenced by options in the buildout configuration, so keep an eye out for that, but if no such options are interfering then this means just invoking buildout like ``$ bin/buildout -N -c buildout.cfg``. IOW, use one ``-N`` option and no ``-v`` options. Before the improvements in zc.buildout 1.5 and the fix to buildout.dumppickedversions, a lot of time was wasted doing work that would only actually be used if ``-v`` had been given. Clean out your egg cache and use ``virtualenv --no-site-packages``: Some of the remaining inefficiencies in zc.buildout are `proportional to the number of distributions `_ available on ``sys.path``. IOW if you reduce the number of distributions to be scanned, you can increase performance. If you're using a `shared `_ `eggs-directory `_ or have a buildout that has been around for a long time, there may be a lot of different versions of the same packages in ``eggs-directory``. By cleaning those out you reduce the number of distributions that buildout needs to scan. I recommend just emptying your ``eggs-directory`` and then letting buildout re-download all the eggs it actually needs now. I takes a while once, but then it's done. Similarly, using ``virtualenv --no-site-packages`` can reduce the number of dists buildout needs to scan. Help me get `zc.buildout/branches/env-cache `_ merged and released: Having said the above about cleaning out the egg cache, ``buildout``, ``distribute``, and ``setuptools`` shouldn't be scanning these paths multiple time anyways. Along with package indexes, these paths are global in nature and so the dists found there should only be scanned for once and cached globally. I've started this work on the `env-cache `_ branch of zc.buildout. In my timing tests, this branch seems to yield another `~60% decrease `_ in run time. The zc.buildout tests, however, `are a PITA `_ and Jim Fulton is `a very busy man `_ with little time to evaluate this branch. If anyone else can help me shepherd this branch through to merge and then release, that would be great. I've `written up `_ how to use this branch with your existing buildouts. With all this in effect I took a production buildout and another development buildout from 2 minute run times down to 20 second run times. This really makes re-running buildout something I no longer feel an urge to avoid. So use zc.buildout 1.5.2, buildout.dumppickedversions 0.5, ``-N`` and do what you can to help get the ``env-cache`` branch merged. .. update:: Jan 27, 2012 Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.