Ross Patterson's Blog - Posts tagged Python

Using Directories for Sphinx Pages

2021-04-05T00:00:00+00:00

Creating Sphinx pages as ./foo/index.rst has a number of benefits over ./foo.rst including path consistency and organizing content.

When I first started playing with ABlog, and thus learning more about Python’s wonderful documentation tool Sphinx, I observed that Sphinx actually creates a directory in the built HTML output for each reStructuredText ./**.rst file in the source. For example, if the source for this post had been ./blog/sphinx-page-dirs.rst, Sphinx would render that to ./_website/blog/sphinx-page-dirs/index.html.

One consequence of this is that a relative link to another area of the site will be broken in the rendered HTML build output if it’s correct in the ./**.rst source file. For example, if ./about.rst contains a link such as:

See `the blog area to see all posts <./blog/>`_.

The Sphinx HTML build output will render that ./about.rst file to ./about/index.html and thus the link will point to /about/blog/ instead of the intended /blog/. I dimly recall encountering such path issues with other types of reST path references as well. Path consistency is also particularly handy for editors or other tools that can infer enough to open files from path references in other files, such as FFAP in Emacs. Writing the source reST for a page as ./about/index.rst in the first place nicely avoids those problems and empowers such tools.

I’m not a big fan of embedding code of one sort into content (or code) of another sort. For example, as big an Org-mode fan as I am, I’ve never been able to get into using Babel. In this case, when writing documentation or other text content, I resist embedding code into source text files and prefer to have such code in separate files next to the text. If the code I want to include in documentation is actual used code, as was the case with the export/import scripts that migrated this blog, then it saves time copying and pasting and prevents out-of-date content when forgetting to do so. Even if it is example or demonstration code of more than a few lines, having it in separate sibling files supports using all our favorite tools (static analysis, IDEs, editors, etc.) which further prevents incorrect or misleading examples. So instead of reStructuredText’s :: literal blocks or Markdown’s indented code blocks I’d much rather include code from separate files. Using a directory for such pages, e.g. ./blog/migrating-from-plone-to-ablog/, also allows us to keep such related files organized neatly together.

Finally, using a directory for a Sphinx page supports putting all related content into a separate VCS repository checkout, such as via $ git submodule .... This can be very useful for sharing code related to one page without having to share the source and VCS history for the whole site. For example, I have an upcoming post with a non-trivial amount of working and tested demo code I want to share. I can use this approach to make that code and VCS history public while still preserving the option of having private content or VCS history for the rest of this site.

Migrating From Plone To ABlog

2021-03-16T00:00:00+00:00

My aspiration to return to blogging thanks to reStructuredText, ABlog and GitLab CI/CD.

I stopped blogging years ago. I was never very good at blogging regularly, I always seem to prefer adding further polish to whatever I’m working on over sharing what I’ve learned. But it didn’t help that my old blogging platform, self-hosted Plone, was too heavy for the job. The maintenance work kept piling up and every time I thought of blogging, I’d remember that TODO list and then do something else instead. :-/

I want to be very clear. Plone, the open-source enterprise CMS, is a simply awesome piece of software and a remarkable open-source project. The list of features provided is both extensive and powerful and the capacity to extend those features and frameworks to build custom CMS’ish applications is even more powerful. Finally and foremost, being involved in the open-source community that develops and maintains that software ecosystem has been the most rewarding part of my career in software. Lovely people.

Plone is, however, laughably overkill for a software developer’s professional blog. My intention was to eat my own dog-food by using the CMS that I spent most of my time working with at the time. That intention is perfectly reasonable but my work stream has mostly moved away from Plone in the intervening years and the maintenance burden is never small for such a large feature set, even if you’re using almost none of those features. I’ll try to learn the lesson here to moderate the impulse to dog-food.

Blogging is writing. Text (as opposed to a WYSIWYG word processor, for example) is the format for writing that least gets in the way when one is proficient with a programmer’s editor. Python’s magnificent reStructuredText is the most expressive text markup language out there and provides a number of features that one inevitably needs when doing much more than writing a heading, 3 paragraphs, 2 lists and 5 links. Sphinx is the lingua franca for writing documentation in the Python world and beyond, and for very good reason.

I’ve known for some time, given the above, that I wanted to blog in reST and I also want to become more familiar with Sphinx, so I went looking for static site generators based on Sphinx. I evaluated ABlog and Tinkerer, both blogging-oriented static site generators based on Sphinx. I like that ABlog seems to be a little less opinionated and calls itself a Sphinx extension since I may well want to use this platform for static site content beyond just blogging. It also seems like ABlog is actively maintained while the last release of Tinkerer was in 2017. Given what I read in a few other reviews and comments I made the call to go with Ablog.

Next up I needed to decide whether to preserve my Plone blogging content or start afresh. Given that I’m bad at getting around to blogging, I thought exporting my old posts would help fill things out a but, though one glance at the dates gives that game away. ;-) It also seemed like a fun problem to solve.

Given that the target is text files and a static site generator, I decided to write the export script oriented towards files on the filesystem and as agnostic about the internals of Plone as possible. I’m pretty happy with it for such a pragmatic bit of “software” and I think it does a surprisingly decent job of capturing as much as possible the data and metadata of Plone content in a format most appropriate for files in a filesystem and oriented to file and text-oriented tools. In particular, it exports text fields the format of the “native” MIME type of that field instance, mostly so that posts I wrote in reST would remain in that format on export. It will also export multiple, separate file-like fields to separate files preserving any per-field filename if available. For example, the lead images for Plone News Item instances, were written next to the corresponding reST file with the original uploaded image basename. Finally, it also uses the FTP/WebDAV support built into Zope to export most, if not all, other fields to a corresponding *.properties file in RFC822 format.

Now that I had files reflecting the Plone representation of the content, I created a new repository, copied the files there and began working on massaging them into ABlog posts. I learned about the reST ‘.. raw:: html’ directive for the Plone content I wrote using the WYSIWYG editor and was thus in HTML format in the exported files. It works nicely and some quick shell commands created an *.rst file corresponding to each *.html file:

.. raw:: html
   :file: ./{id}.html

Thus having *.rst files for each exported post, it was time to convert them into ABlog posts. I wanted to preserve all metadata from the *.properties files that corresponded to ABlog post metadata or could reasonably be added to the reST surrounding the main content of the post. Again, I’m pretty happy with how much the import script was able to preserve. I don’t think I’ll miss any of the excluded metadata:

#######
{title}
#######

    {description}

.. raw:: html
   :file: ./{id}.html


.. meta::
   :description: {description}
   :keywords: {subject}

.. post:: {creation_date}
   :tags: {subject}
   :author: Ross Patterson
   :redirect:  @@redirect-to-uuid/{UID}

.. update:: {modification_date}

   Imported from Plone on {now}.  The date for this update is the last
   modified date in Plone.

If any of the above export/import code would be useful to anyone else, LMK and I’ll throw it up to a separate GitLab repository under an open-source license along with my VCS history.

With all the content showing up nicely in the output from the ABlog watchdog, auto-rebuild development server, I fixed a few formatting issues in the imported content, updated the boilerplate content, and went looking for a theme. I do prefer dark themes but mostly, much to my chagrin, I’ve noticed that many if not most people seem to think less of you if you show them something with the default theme, even when the person showing you is decidedly not a designer. ;-) I found a dark Solarized theme that extends the default theme and contributed some fixes. It’s certainly not perfect, but I like it better than the default and it certainly expresses that this is the blog of a developer.

Finally, I created a private GitLab repository and whipped up a simple GitLab CI/CD configuration that builds and deploys the built static site to GitLab Pages when I push to master. I added some GitLab redirects for the few paths that changed. BTW, this is one of the nice things about ABlog so far, most paths stayed the same. I pushed to master and viola!

Do LMK if you find any bugs. This ended up being more complete than I would have expected so I’d love to know of any issues I missed.

Emerald Sprint Report Out

2013-02-13T00:00:00+00:00

Merging versioning and and constrained types into Dexterity near Beautiful Seattle.

Thanks to the fabulous Seattle Plone folk, we had another west coast sprint, the Emerald Sprint, on lovely Camano Island. The focus was polishing Dexterity, it was a gorgeous location, a very productive sprint, and I very much hope we can make it an annual thing. We might have a better chance at that if we choose a topic next year such that David can just be another sprinter for once. :-)

Firstly, I have to give a huge shout out to our Maven of Fine Dining, Sally, and to Alice for yet more above-and-beyond contributions to the culinary experience of this sprint. The food was simply incredible and I have to say I love a sprint that has that element. I was spoiled before by the cooking at the Cioppino Sprint and the Sea Sprint and the food at this sprint may have out done them!

The sprint location was an excellent choice and everything was very well organized by Cris, Luke, Fulvio and David. I was very sad to learn that Cris wasn’t going to be able to stay for the sprint and wish him the best. Thanks also to Ian Anderson and Andy Leeb for treating us to a delicious dinner out on our first night.

We did a brainstorming session where we all voted for what we found to be the most important. Then David and Matthew marked the items that they thought were clear low-hanging fruit that didn’t require a lot of design or other discussion. After seeing what the other sprinters chose for I opted for tackling some of the remaining low-hanging fruit. That ended up being two tasks both of which involved merging work done by others previously in add-ons or other packages not in Dexterity core where they belong: versioning and constrained types.

Working on versioning involved merging two add-on packages into Dexterity core. The collective.cmfeditionsdexteritycompat add-on makes it possible to use the history view on Dexterity content and I merged it into plone.app.versioningbehavior. The collective.dexteritydiff add-on brings parity with Archetypes when viewing diffs between versions and I merged it into Products.CMFDiffTool. Both packages were done by Rafael Oliveira and worked perfectly, my only job was doing the merge and reconciling the tests. Fortunately, Rafael had written tests and had done so using plone.app.testing. Unfortunately, Products.CMFEditions does not use plone.app.testing and since such tests can not run in the same test run as ZopeTestCase tests are run, that meant I had to back-port the tests Rafael wrote off of plone.app.testing back to ZopeTestCase. I hate going backwards but enough things build on the Products.CMFEditions tests that I guessed it would take too long and be to disruptive to convert all those tests. After that, I spent the rest of the sprint trying to complete my plone.app.testing.api branch which is intended to make such conversions easier.

I also had the opportunity to bring Gin Apocrypha with me and bar tend for the event. It was my first time trying to check enough items on the plane to be able to mix my whole menu and it actually worked out very well. So I think I may be willing to check a bag in the future to be able to bring Gin Apocrypha to events I don’t drive to in the future!

It was a great sprint and I love the Seattle area so I really hope this happens again. This sprint being focused on Dexterity and David being Dexterity’s maintainer and fearless leader, I know I pestered him a lot as I needed to ask questions and get perspective, but he did a great job facilitating everything. For that reason if no other, I hope the next Emerald Sprint can be on a topic such that David can get back to basics and enjoy being just another sprinter again. :-)

Updated on 13 February 2013

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

POSKeyError During Commit

2012-11-21T00:00:00+00:00

A terrifying tale of ZODB errors that only showed during the hardest time to debug.

Believe it or not, I’m a POSKeyError virgin, I’ve never run into one… Until now!

I was working on a very small Plone 3.1.1 to 4.2.2 upgrade. Not too many add-ons, simple skin/theme customizations, 1,810 objects in ~800MB. No problem, right?

Wrong. I got a POSKeyError after the upgrade step that converts files and images to BLOBs. The wierd part is that it only happened during the transaction.commit(), not during ZODB object access:

  File "/opt/src/eggs/Zope2-2.13.18-py2.7.egg/Zope2/App/startup.py", line 301, in commit
    transaction.commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_manager.py", line 89, in commit
    return self.get().commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 329, in commit
    self._commitResources()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 443, in _commitResources
    rm.commit(self)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 572, in commit
    oid, serial, transaction)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/BaseStorage.py", line 416, in checkCurrentSerialInTransaction
    committed_tid = self.getTid(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 770, in getTid
    pos = self._lookup_pos(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 403, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x055350

I tried all the tools in ZODB.scripts and zc.zodbdgc but they couldn’t tell me anything about the oid in question. That’s probably because it was a different oid every time I attempted the upgrade step on the same ZODB.

Without being able to use the typical tools to find out what ZODB objects were referring to those oids I was really stuck. Then David Glick came to my rescue. Actually, he was staying at my apartment so I drafted him. :-) He suggested that I might have to create a mapping of oids to stack frames at the time when objects were added to the transaction so that I could figure out what had been going on once I knew which oid would be the problem. This worked very well and much thanks to David.

I added the following to the place where the objects were being added to the transaction, ZODB.Connection.Connection.readCurrent():

if not hasattr(self, 'XXX'):
    self.XXX = {}
self.XXX[ob._p_oid] = sys._getframe()

Then once I had my POSKeyError in pdb.post_mortem() I could invoke pdb at the place where the offending oid had been added:

import pdb; pdbr = pdb.Pdb() ; pdbr.reset() ; pdbr.interaction(self.XXX[oid], None)

Using that I was able to find out what objects were involved. They were all portal_catalog/path index Trees.IIBTree.IITreeSet objects. What’s odd is that the error happens even if I do a catalog clear and rebuild first.

After much more debugging and discussion, we determined that the ZODB.Connection.Connection.readCurrent() support was involved somehow. Also, as I deleted the images that were involved I noticed another migration error earlier on went away for each image I removed. That’s the key, when the BLOB migration fails for one file or image, it uses a savepoint to rollback just that image’s migration and proceed with the rest. It sure seems like there’s a bug in the ZODB where when a savepoint is rolled back the objects registered with the transaction are cleaned up but the objects registered for readCurrent are not. Gee, someone should write a test case. :-)

In the meantime, it turns out the reason the image migrations were failing was something else all together but I thought I should share this debugging technique for capturing stack information when you can’t know which stack you need to inspect until later. I also hope this ZODB bug gets squashed.

Updated on 21 November 2012

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

WSGI apps on IIS

2012-10-30T00:00:00+00:00

iiswsgi provides a WSGI server providing a FCGI gateway to IIS and setup.py commands for distributing apps as WebPI packages

The iiswsgi module implements a FastCGI to WSGI gateway that is compatible with IIS’s variation of the FastCGI protocol. It also provides distutils commands for building, distributing and installing Microsoft Web Deploy (MSDeploy) packages through the Web Platform Installer (WebPI).

The goals of the code in iiswsgi are to do the following for deploying WSGI apps on IIS:

make it open source as far as possible, right up to IIS
be Pythonic as far as possible, right up to the MSDeploy packaging
re-use our existing tool-chain for distributing packages
share the maintenance burden for a WSGI on Windows story across the community

For the Plone project, it’s always been simultaneously a necessity that we support a Windows deployment story and one of our biggest pain points. The Windows installers have always been very different from the other installers. They have had different layouts from our user and developer documentation and even from each other. They have never been maintained or supported by more than one entity, either a company or an individual, and as such have often and ultimately languished. And as for the poor individuals who have tackled the Windows installers, they have almost always burned out and can no longer provide any significant Windows support at all. This is not a healthy open source community dynamic. And yet there is wide consensus that it’s not an option not to have a Windows deployment story.

My hope is that by generalizing the IIS deployment architecture as a WSGI server and distutils commands, it can be of use to the general Python WSGI world. I also hope that by doing things ‘The Right Way’, it will be something that will be clearer and easy to support and maintain. With those two together maybe we can solve the burnout issue by distributing the maintenance load. I’d very much appreciate any help to that end, particularly including feedback on how to get there. I don’t care where the code lives and would be happy to see some of it merged back into the packages it derives from or moved into larger packages. So please let me know if you’d like to coordinate moving things around with me.

Help Needed

Any contributions are very welcome. Here are a few things I’m looking for in particular:

addressing Known Issues
IIS app name and Python dist name conventions
fostering community ownership
writing tests

I’m particularly apologetic for the last one, I’m ashamed by the lack of tests. In my defense, this whole problem was such a fog for me when I started that I just needed to start writing things and poking around. Believe me this is not my usual MO, I almost always do TDD and beg your forgiveness. :-)

I look forward to getting this going!

Updated on 30 October 2012

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.