Ross Patterson's Blog - Posts tagged Plone

Reflections on TTW Programming from the Future

2021-04-07T00:00:00+00:00

Playing with IFTTT and Zapier has me remembering the TTW programming fallout and debate.

I’ve been working on feeding my ABlog posts to social media. I’ve been experimenting with some external services for this, primarily because implementing something self-hosted seems like overkill for this and it makes an excuse to explore something I’ve been wanting to. I’m intrigued by these services that claim to let you connect little pieces of functionality, through an intuitive UI, to create “custom” mini-“applications”.

I’m old enough to have cut my teeth on some of the first Through-The-Web (TTW) programming offerings and thus long enough to watch the problems that became of that. The consensus among the communities I’ve been involved in is that it’s a bad idea to invite users and tinkerers to build their own custom applications with the promise that it’s easier and achievable because they can do it in their browser. The issue is not that non-programmers can’t learn to program. They certainly can and I object to the kind of rarefied elitism that leads others to argue otherwise. The issue is that no one can build the kind of custom application one can build with a full programming language without also learning most, if not all, of the suite of programming best practices. Without those best practices, one is all but guaranteed to end up with a poorly engineered, unmaintainable mess. The best case scenario is that your application fails to become useful and thus never has a chance to become a burden. That leaves those who manage to create an actually useful, initially successful application that users come to depend on. These successful tinkerers are eventually rewarded by hitting multiple walls at once. Thinking about, planning for, and executing on maintainability and other best practices are simply costs that must be paid. The apparent savings of not having to think about such things is an illusion and those costs have been compounding all along behind your back.

While the 10th feature was as easy to implement as the 1st, the 100th feature is 10 times as difficult as the 10th and breaks features 20-23; you can infer the curve. You took a stab at a major version upgrade of the framework months ago, hit another wall, and set it aside. Now the security/LTS of your major version is about to expire meaning your custom application will soon be vulnerable to multiple publicly available exploits. Change management is becoming a nightmare with duplicated code all over the place in an effort to compensate. A while back, you learned about VCS and the current best-of-breed in that domain but you also realized that you couldn’t use those tools to manage your TTW code. There are better ways to build custom applications even within the framework you’re using, but not TTW, the effort is comparable to re-implementing the application, and those that can do it justly charge much more. BTW, having learned so much from this, you’re now worth a lot more yourself and are recruited into your first junior programming job. Those last two are good things, but the end result is that a team of users is now dependent on a custom application that’s going to cost them a lot to either replace or maintain. They became a company in the business of developing software without ever intending to, planning on or budgeting for.

That whole process breeds resentment in a way that doesn’t come from even an equally blundering and costly but conscious effort to develop one’s own custom software. Like much of human happiness, it’s expectations, the surprise, that’s the issue. That’s how I understand the issue with TTW programming. If a framework provides anything approaching a full programming language, and offers or promises to save users-come-developers some portion of software development “hassles”, then everyone is almost certainly in for some bad-will inducing surprises.

So that means that any system that allows users to assemble “applications” through a UI meant to spare them from some portion of software development discipline needs to have clear, rigid boundaries on expectations. It needs to narrowly define what can be done, place firm limits on the size of what can be built (roughly, the quantity of components). It doesn’t hurt to provide users an eject button so that they can export what they’ve done to the filesystem and continue developing it incrementally using a full software development tool chain and process. These are the lessons the Plone community tried to learn from our Zope TTW roots. The Plone community built web UI based on these conclusions for both custom content types with custom schemata and for custom themes.

Given my involvement in that long (for software) history, I’ve been wanting to check out IFTTT. Such custom applet “development” and hosting platforms are solving different problems, very simple workflow applications connecting various internet services, but I imagine theses same principles apply. IFTTT seems to employ these principles quite firmly. I’m not aware of any eject button that could meaningfully allow one to incrementally build software on top of IFTTT applets, but that wouldn’t be terribly useful anyways given how firmly they restrict the scope of what one can do. I don’t have any current need for anything beyond IFTTT’s simple, two-step composition model, the this component and the that component in “If this then that”, but I could well imagine wanting more than that. There’s also the limit on 3 applets on the free tier of their freemium pricing plan, I am already using 2. Finally, I couldn’t find a then component for posting to LinkedIn as a service, that’s the one that really lead me to seek other options.

I ran across Zapier a while back and was similarly intrigued, so when that showed up in my search for something that could post to LinkedIn, I eagerly tried it out. They offer 5 applets, which they call “Zaps” (bit of a groaner of a term), on the free tier of their freemium pricing plan, so point for Zapier. I haven’t read much of their docs, but it’s apparent in the web UI that there’s support for at least a somewhat arbitrary chaining of more than 2 components, another win for Zapier. I also found their options for dealing with different “fields” in data, atom feed items in this case, to be significantly more rich than those offered by IFTTT, Zapier streaks ahead. They do indeed have components that interact with LinkedIn, so Zapier FTW in this very cursory and superficial scoring.

In the current moment, I’m very curious to see how the more restrictive IFTTT compares in the long term to the more feature-rich and composible Zapier in terms of user experience, user maintainability, and internal maintainability. Even if I were to pay enough attention to keep up on this, not very likely, I’m even less likely to get access to the kind of proprietary internal discussion that could yield such insight. C’est la capitalisme. At any rate, It’s been fun to reflect on the issue while automating some things.

Migrating From Plone To ABlog

2021-03-16T00:00:00+00:00

My aspiration to return to blogging thanks to reStructuredText, ABlog and GitLab CI/CD.

I stopped blogging years ago. I was never very good at blogging regularly, I always seem to prefer adding further polish to whatever I’m working on over sharing what I’ve learned. But it didn’t help that my old blogging platform, self-hosted Plone, was too heavy for the job. The maintenance work kept piling up and every time I thought of blogging, I’d remember that TODO list and then do something else instead. :-/

I want to be very clear. Plone, the open-source enterprise CMS, is a simply awesome piece of software and a remarkable open-source project. The list of features provided is both extensive and powerful and the capacity to extend those features and frameworks to build custom CMS’ish applications is even more powerful. Finally and foremost, being involved in the open-source community that develops and maintains that software ecosystem has been the most rewarding part of my career in software. Lovely people.

Plone is, however, laughably overkill for a software developer’s professional blog. My intention was to eat my own dog-food by using the CMS that I spent most of my time working with at the time. That intention is perfectly reasonable but my work stream has mostly moved away from Plone in the intervening years and the maintenance burden is never small for such a large feature set, even if you’re using almost none of those features. I’ll try to learn the lesson here to moderate the impulse to dog-food.

Blogging is writing. Text (as opposed to a WYSIWYG word processor, for example) is the format for writing that least gets in the way when one is proficient with a programmer’s editor. Python’s magnificent reStructuredText is the most expressive text markup language out there and provides a number of features that one inevitably needs when doing much more than writing a heading, 3 paragraphs, 2 lists and 5 links. Sphinx is the lingua franca for writing documentation in the Python world and beyond, and for very good reason.

I’ve known for some time, given the above, that I wanted to blog in reST and I also want to become more familiar with Sphinx, so I went looking for static site generators based on Sphinx. I evaluated ABlog and Tinkerer, both blogging-oriented static site generators based on Sphinx. I like that ABlog seems to be a little less opinionated and calls itself a Sphinx extension since I may well want to use this platform for static site content beyond just blogging. It also seems like ABlog is actively maintained while the last release of Tinkerer was in 2017. Given what I read in a few other reviews and comments I made the call to go with Ablog.

Next up I needed to decide whether to preserve my Plone blogging content or start afresh. Given that I’m bad at getting around to blogging, I thought exporting my old posts would help fill things out a but, though one glance at the dates gives that game away. ;-) It also seemed like a fun problem to solve.

Given that the target is text files and a static site generator, I decided to write the export script oriented towards files on the filesystem and as agnostic about the internals of Plone as possible. I’m pretty happy with it for such a pragmatic bit of “software” and I think it does a surprisingly decent job of capturing as much as possible the data and metadata of Plone content in a format most appropriate for files in a filesystem and oriented to file and text-oriented tools. In particular, it exports text fields the format of the “native” MIME type of that field instance, mostly so that posts I wrote in reST would remain in that format on export. It will also export multiple, separate file-like fields to separate files preserving any per-field filename if available. For example, the lead images for Plone News Item instances, were written next to the corresponding reST file with the original uploaded image basename. Finally, it also uses the FTP/WebDAV support built into Zope to export most, if not all, other fields to a corresponding *.properties file in RFC822 format.

Now that I had files reflecting the Plone representation of the content, I created a new repository, copied the files there and began working on massaging them into ABlog posts. I learned about the reST ‘.. raw:: html’ directive for the Plone content I wrote using the WYSIWYG editor and was thus in HTML format in the exported files. It works nicely and some quick shell commands created an *.rst file corresponding to each *.html file:

.. raw:: html
   :file: ./{id}.html

Thus having *.rst files for each exported post, it was time to convert them into ABlog posts. I wanted to preserve all metadata from the *.properties files that corresponded to ABlog post metadata or could reasonably be added to the reST surrounding the main content of the post. Again, I’m pretty happy with how much the import script was able to preserve. I don’t think I’ll miss any of the excluded metadata:

#######
{title}
#######

    {description}

.. raw:: html
   :file: ./{id}.html


.. meta::
   :description: {description}
   :keywords: {subject}

.. post:: {creation_date}
   :tags: {subject}
   :author: Ross Patterson
   :redirect:  @@redirect-to-uuid/{UID}

.. update:: {modification_date}

   Imported from Plone on {now}.  The date for this update is the last
   modified date in Plone.

If any of the above export/import code would be useful to anyone else, LMK and I’ll throw it up to a separate GitLab repository under an open-source license along with my VCS history.

With all the content showing up nicely in the output from the ABlog watchdog, auto-rebuild development server, I fixed a few formatting issues in the imported content, updated the boilerplate content, and went looking for a theme. I do prefer dark themes but mostly, much to my chagrin, I’ve noticed that many if not most people seem to think less of you if you show them something with the default theme, even when the person showing you is decidedly not a designer. ;-) I found a dark Solarized theme that extends the default theme and contributed some fixes. It’s certainly not perfect, but I like it better than the default and it certainly expresses that this is the blog of a developer.

Finally, I created a private GitLab repository and whipped up a simple GitLab CI/CD configuration that builds and deploys the built static site to GitLab Pages when I push to master. I added some GitLab redirects for the few paths that changed. BTW, this is one of the nice things about ABlog so far, most paths stayed the same. I pushed to master and viola!

Do LMK if you find any bugs. This ended up being more complete than I would have expected so I’d love to know of any issues I missed.

Emerald Sprint Report Out

2013-02-13T00:00:00+00:00

Merging versioning and and constrained types into Dexterity near Beautiful Seattle.

Thanks to the fabulous Seattle Plone folk, we had another west coast sprint, the Emerald Sprint, on lovely Camano Island. The focus was polishing Dexterity, it was a gorgeous location, a very productive sprint, and I very much hope we can make it an annual thing. We might have a better chance at that if we choose a topic next year such that David can just be another sprinter for once. :-)

Firstly, I have to give a huge shout out to our Maven of Fine Dining, Sally, and to Alice for yet more above-and-beyond contributions to the culinary experience of this sprint. The food was simply incredible and I have to say I love a sprint that has that element. I was spoiled before by the cooking at the Cioppino Sprint and the Sea Sprint and the food at this sprint may have out done them!

The sprint location was an excellent choice and everything was very well organized by Cris, Luke, Fulvio and David. I was very sad to learn that Cris wasn’t going to be able to stay for the sprint and wish him the best. Thanks also to Ian Anderson and Andy Leeb for treating us to a delicious dinner out on our first night.

We did a brainstorming session where we all voted for what we found to be the most important. Then David and Matthew marked the items that they thought were clear low-hanging fruit that didn’t require a lot of design or other discussion. After seeing what the other sprinters chose for I opted for tackling some of the remaining low-hanging fruit. That ended up being two tasks both of which involved merging work done by others previously in add-ons or other packages not in Dexterity core where they belong: versioning and constrained types.

Working on versioning involved merging two add-on packages into Dexterity core. The collective.cmfeditionsdexteritycompat add-on makes it possible to use the history view on Dexterity content and I merged it into plone.app.versioningbehavior. The collective.dexteritydiff add-on brings parity with Archetypes when viewing diffs between versions and I merged it into Products.CMFDiffTool. Both packages were done by Rafael Oliveira and worked perfectly, my only job was doing the merge and reconciling the tests. Fortunately, Rafael had written tests and had done so using plone.app.testing. Unfortunately, Products.CMFEditions does not use plone.app.testing and since such tests can not run in the same test run as ZopeTestCase tests are run, that meant I had to back-port the tests Rafael wrote off of plone.app.testing back to ZopeTestCase. I hate going backwards but enough things build on the Products.CMFEditions tests that I guessed it would take too long and be to disruptive to convert all those tests. After that, I spent the rest of the sprint trying to complete my plone.app.testing.api branch which is intended to make such conversions easier.

I also had the opportunity to bring Gin Apocrypha with me and bar tend for the event. It was my first time trying to check enough items on the plane to be able to mix my whole menu and it actually worked out very well. So I think I may be willing to check a bag in the future to be able to bring Gin Apocrypha to events I don’t drive to in the future!

It was a great sprint and I love the Seattle area so I really hope this happens again. This sprint being focused on Dexterity and David being Dexterity’s maintainer and fearless leader, I know I pestered him a lot as I needed to ask questions and get perspective, but he did a great job facilitating everything. For that reason if no other, I hope the next Emerald Sprint can be on a topic such that David can get back to basics and enjoy being just another sprinter again. :-)

Updated on 13 February 2013

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

prgmr.com VPS: “We don’t assume you are stupid”

2013-02-04T00:00:00+00:00

Excellent, affordable VPS service done in a way that feels very much like participating in an open-source community.

I just wanted to drop a quick note praising prgmr.com for their excellent and very affordable VPS service. Unfortunately, I had to change hosting providers for other reasons but would have been happy to stay with them indefinitely.

Their slogan is “We don’t assume you are stupid” and that’s very apt. There’s a combination of no hand-holding combined with support. The support is a particularly good part of their service. It’s generous but in a particular way that works very well for me. They’ll help you up with no real hierarchy or bureaucracy up to a limit that I think almost anyone who has participated significantly in an open-source project will find familiar. Beyond that, you’re on your own. I really like this balance and always found the lack of bureaucracy more than worth it for any relative lack of hand-holding.

At any rate, if you want an excellent and affordable VPS in the Bay Area, I can recommend no one better.

Updated on 04 February 2013

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

POSKeyError During Commit

2012-11-21T00:00:00+00:00

A terrifying tale of ZODB errors that only showed during the hardest time to debug.

Believe it or not, I’m a POSKeyError virgin, I’ve never run into one… Until now!

I was working on a very small Plone 3.1.1 to 4.2.2 upgrade. Not too many add-ons, simple skin/theme customizations, 1,810 objects in ~800MB. No problem, right?

Wrong. I got a POSKeyError after the upgrade step that converts files and images to BLOBs. The wierd part is that it only happened during the transaction.commit(), not during ZODB object access:

  File "/opt/src/eggs/Zope2-2.13.18-py2.7.egg/Zope2/App/startup.py", line 301, in commit
    transaction.commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_manager.py", line 89, in commit
    return self.get().commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 329, in commit
    self._commitResources()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 443, in _commitResources
    rm.commit(self)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 572, in commit
    oid, serial, transaction)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/BaseStorage.py", line 416, in checkCurrentSerialInTransaction
    committed_tid = self.getTid(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 770, in getTid
    pos = self._lookup_pos(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 403, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x055350

I tried all the tools in ZODB.scripts and zc.zodbdgc but they couldn’t tell me anything about the oid in question. That’s probably because it was a different oid every time I attempted the upgrade step on the same ZODB.

Without being able to use the typical tools to find out what ZODB objects were referring to those oids I was really stuck. Then David Glick came to my rescue. Actually, he was staying at my apartment so I drafted him. :-) He suggested that I might have to create a mapping of oids to stack frames at the time when objects were added to the transaction so that I could figure out what had been going on once I knew which oid would be the problem. This worked very well and much thanks to David.

I added the following to the place where the objects were being added to the transaction, ZODB.Connection.Connection.readCurrent():

if not hasattr(self, 'XXX'):
    self.XXX = {}
self.XXX[ob._p_oid] = sys._getframe()

Then once I had my POSKeyError in pdb.post_mortem() I could invoke pdb at the place where the offending oid had been added:

import pdb; pdbr = pdb.Pdb() ; pdbr.reset() ; pdbr.interaction(self.XXX[oid], None)

Using that I was able to find out what objects were involved. They were all portal_catalog/path index Trees.IIBTree.IITreeSet objects. What’s odd is that the error happens even if I do a catalog clear and rebuild first.

After much more debugging and discussion, we determined that the ZODB.Connection.Connection.readCurrent() support was involved somehow. Also, as I deleted the images that were involved I noticed another migration error earlier on went away for each image I removed. That’s the key, when the BLOB migration fails for one file or image, it uses a savepoint to rollback just that image’s migration and proceed with the rest. It sure seems like there’s a bug in the ZODB where when a savepoint is rolled back the objects registered with the transaction are cleaned up but the objects registered for readCurrent are not. Gee, someone should write a test case. :-)

In the meantime, it turns out the reason the image migrations were failing was something else all together but I thought I should share this debugging technique for capturing stack information when you can’t know which stack you need to inspect until later. I also hope this ZODB bug gets squashed.

Updated on 21 November 2012

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.