Ross Patterson's Blog - Posts tagged Zope

Reflections on TTW Programming from the Future

2021-04-07T00:00:00+00:00

Playing with IFTTT and Zapier has me remembering the TTW programming fallout and debate.

I’ve been working on feeding my ABlog posts to social media. I’ve been experimenting with some external services for this, primarily because implementing something self-hosted seems like overkill for this and it makes an excuse to explore something I’ve been wanting to. I’m intrigued by these services that claim to let you connect little pieces of functionality, through an intuitive UI, to create “custom” mini-“applications”.

I’m old enough to have cut my teeth on some of the first Through-The-Web (TTW) programming offerings and thus long enough to watch the problems that became of that. The consensus among the communities I’ve been involved in is that it’s a bad idea to invite users and tinkerers to build their own custom applications with the promise that it’s easier and achievable because they can do it in their browser. The issue is not that non-programmers can’t learn to program. They certainly can and I object to the kind of rarefied elitism that leads others to argue otherwise. The issue is that no one can build the kind of custom application one can build with a full programming language without also learning most, if not all, of the suite of programming best practices. Without those best practices, one is all but guaranteed to end up with a poorly engineered, unmaintainable mess. The best case scenario is that your application fails to become useful and thus never has a chance to become a burden. That leaves those who manage to create an actually useful, initially successful application that users come to depend on. These successful tinkerers are eventually rewarded by hitting multiple walls at once. Thinking about, planning for, and executing on maintainability and other best practices are simply costs that must be paid. The apparent savings of not having to think about such things is an illusion and those costs have been compounding all along behind your back.

While the 10th feature was as easy to implement as the 1st, the 100th feature is 10 times as difficult as the 10th and breaks features 20-23; you can infer the curve. You took a stab at a major version upgrade of the framework months ago, hit another wall, and set it aside. Now the security/LTS of your major version is about to expire meaning your custom application will soon be vulnerable to multiple publicly available exploits. Change management is becoming a nightmare with duplicated code all over the place in an effort to compensate. A while back, you learned about VCS and the current best-of-breed in that domain but you also realized that you couldn’t use those tools to manage your TTW code. There are better ways to build custom applications even within the framework you’re using, but not TTW, the effort is comparable to re-implementing the application, and those that can do it justly charge much more. BTW, having learned so much from this, you’re now worth a lot more yourself and are recruited into your first junior programming job. Those last two are good things, but the end result is that a team of users is now dependent on a custom application that’s going to cost them a lot to either replace or maintain. They became a company in the business of developing software without ever intending to, planning on or budgeting for.

That whole process breeds resentment in a way that doesn’t come from even an equally blundering and costly but conscious effort to develop one’s own custom software. Like much of human happiness, it’s expectations, the surprise, that’s the issue. That’s how I understand the issue with TTW programming. If a framework provides anything approaching a full programming language, and offers or promises to save users-come-developers some portion of software development “hassles”, then everyone is almost certainly in for some bad-will inducing surprises.

So that means that any system that allows users to assemble “applications” through a UI meant to spare them from some portion of software development discipline needs to have clear, rigid boundaries on expectations. It needs to narrowly define what can be done, place firm limits on the size of what can be built (roughly, the quantity of components). It doesn’t hurt to provide users an eject button so that they can export what they’ve done to the filesystem and continue developing it incrementally using a full software development tool chain and process. These are the lessons the Plone community tried to learn from our Zope TTW roots. The Plone community built web UI based on these conclusions for both custom content types with custom schemata and for custom themes.

Given my involvement in that long (for software) history, I’ve been wanting to check out IFTTT. Such custom applet “development” and hosting platforms are solving different problems, very simple workflow applications connecting various internet services, but I imagine theses same principles apply. IFTTT seems to employ these principles quite firmly. I’m not aware of any eject button that could meaningfully allow one to incrementally build software on top of IFTTT applets, but that wouldn’t be terribly useful anyways given how firmly they restrict the scope of what one can do. I don’t have any current need for anything beyond IFTTT’s simple, two-step composition model, the this component and the that component in “If this then that”, but I could well imagine wanting more than that. There’s also the limit on 3 applets on the free tier of their freemium pricing plan, I am already using 2. Finally, I couldn’t find a then component for posting to LinkedIn as a service, that’s the one that really lead me to seek other options.

I ran across Zapier a while back and was similarly intrigued, so when that showed up in my search for something that could post to LinkedIn, I eagerly tried it out. They offer 5 applets, which they call “Zaps” (bit of a groaner of a term), on the free tier of their freemium pricing plan, so point for Zapier. I haven’t read much of their docs, but it’s apparent in the web UI that there’s support for at least a somewhat arbitrary chaining of more than 2 components, another win for Zapier. I also found their options for dealing with different “fields” in data, atom feed items in this case, to be significantly more rich than those offered by IFTTT, Zapier streaks ahead. They do indeed have components that interact with LinkedIn, so Zapier FTW in this very cursory and superficial scoring.

In the current moment, I’m very curious to see how the more restrictive IFTTT compares in the long term to the more feature-rich and composible Zapier in terms of user experience, user maintainability, and internal maintainability. Even if I were to pay enough attention to keep up on this, not very likely, I’m even less likely to get access to the kind of proprietary internal discussion that could yield such insight. C’est la capitalisme. At any rate, It’s been fun to reflect on the issue while automating some things.

Emerald Sprint Report Out

2013-02-13T00:00:00+00:00

Merging versioning and and constrained types into Dexterity near Beautiful Seattle.

Thanks to the fabulous Seattle Plone folk, we had another west coast sprint, the Emerald Sprint, on lovely Camano Island. The focus was polishing Dexterity, it was a gorgeous location, a very productive sprint, and I very much hope we can make it an annual thing. We might have a better chance at that if we choose a topic next year such that David can just be another sprinter for once. :-)

Firstly, I have to give a huge shout out to our Maven of Fine Dining, Sally, and to Alice for yet more above-and-beyond contributions to the culinary experience of this sprint. The food was simply incredible and I have to say I love a sprint that has that element. I was spoiled before by the cooking at the Cioppino Sprint and the Sea Sprint and the food at this sprint may have out done them!

The sprint location was an excellent choice and everything was very well organized by Cris, Luke, Fulvio and David. I was very sad to learn that Cris wasn’t going to be able to stay for the sprint and wish him the best. Thanks also to Ian Anderson and Andy Leeb for treating us to a delicious dinner out on our first night.

We did a brainstorming session where we all voted for what we found to be the most important. Then David and Matthew marked the items that they thought were clear low-hanging fruit that didn’t require a lot of design or other discussion. After seeing what the other sprinters chose for I opted for tackling some of the remaining low-hanging fruit. That ended up being two tasks both of which involved merging work done by others previously in add-ons or other packages not in Dexterity core where they belong: versioning and constrained types.

Working on versioning involved merging two add-on packages into Dexterity core. The collective.cmfeditionsdexteritycompat add-on makes it possible to use the history view on Dexterity content and I merged it into plone.app.versioningbehavior. The collective.dexteritydiff add-on brings parity with Archetypes when viewing diffs between versions and I merged it into Products.CMFDiffTool. Both packages were done by Rafael Oliveira and worked perfectly, my only job was doing the merge and reconciling the tests. Fortunately, Rafael had written tests and had done so using plone.app.testing. Unfortunately, Products.CMFEditions does not use plone.app.testing and since such tests can not run in the same test run as ZopeTestCase tests are run, that meant I had to back-port the tests Rafael wrote off of plone.app.testing back to ZopeTestCase. I hate going backwards but enough things build on the Products.CMFEditions tests that I guessed it would take too long and be to disruptive to convert all those tests. After that, I spent the rest of the sprint trying to complete my plone.app.testing.api branch which is intended to make such conversions easier.

I also had the opportunity to bring Gin Apocrypha with me and bar tend for the event. It was my first time trying to check enough items on the plane to be able to mix my whole menu and it actually worked out very well. So I think I may be willing to check a bag in the future to be able to bring Gin Apocrypha to events I don’t drive to in the future!

It was a great sprint and I love the Seattle area so I really hope this happens again. This sprint being focused on Dexterity and David being Dexterity’s maintainer and fearless leader, I know I pestered him a lot as I needed to ask questions and get perspective, but he did a great job facilitating everything. For that reason if no other, I hope the next Emerald Sprint can be on a topic such that David can get back to basics and enjoy being just another sprinter again. :-)

Updated on 13 February 2013

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

prgmr.com VPS: “We don’t assume you are stupid”

2013-02-04T00:00:00+00:00

Excellent, affordable VPS service done in a way that feels very much like participating in an open-source community.

I just wanted to drop a quick note praising prgmr.com for their excellent and very affordable VPS service. Unfortunately, I had to change hosting providers for other reasons but would have been happy to stay with them indefinitely.

Their slogan is “We don’t assume you are stupid” and that’s very apt. There’s a combination of no hand-holding combined with support. The support is a particularly good part of their service. It’s generous but in a particular way that works very well for me. They’ll help you up with no real hierarchy or bureaucracy up to a limit that I think almost anyone who has participated significantly in an open-source project will find familiar. Beyond that, you’re on your own. I really like this balance and always found the lack of bureaucracy more than worth it for any relative lack of hand-holding.

At any rate, if you want an excellent and affordable VPS in the Bay Area, I can recommend no one better.

Updated on 04 February 2013

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

POSKeyError During Commit

2012-11-21T00:00:00+00:00

A terrifying tale of ZODB errors that only showed during the hardest time to debug.

Believe it or not, I’m a POSKeyError virgin, I’ve never run into one… Until now!

I was working on a very small Plone 3.1.1 to 4.2.2 upgrade. Not too many add-ons, simple skin/theme customizations, 1,810 objects in ~800MB. No problem, right?

Wrong. I got a POSKeyError after the upgrade step that converts files and images to BLOBs. The wierd part is that it only happened during the transaction.commit(), not during ZODB object access:

  File "/opt/src/eggs/Zope2-2.13.18-py2.7.egg/Zope2/App/startup.py", line 301, in commit
    transaction.commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_manager.py", line 89, in commit
    return self.get().commit()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 329, in commit
    self._commitResources()
  File "/opt/src/eggs/transaction-1.1.1-py2.7.egg/transaction/_transaction.py", line 443, in _commitResources
    rm.commit(self)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/Connection.py", line 572, in commit
    oid, serial, transaction)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/BaseStorage.py", line 416, in checkCurrentSerialInTransaction
    committed_tid = self.getTid(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 770, in getTid
    pos = self._lookup_pos(oid)
  File "/opt/src/eggs/ZODB3-3.10.5-py2.7-linux-x86_64.egg/ZODB/FileStorage/FileStorage.py", line 403, in _lookup_pos
    raise POSKeyError(oid)
POSKeyError: 0x055350

I tried all the tools in ZODB.scripts and zc.zodbdgc but they couldn’t tell me anything about the oid in question. That’s probably because it was a different oid every time I attempted the upgrade step on the same ZODB.

Without being able to use the typical tools to find out what ZODB objects were referring to those oids I was really stuck. Then David Glick came to my rescue. Actually, he was staying at my apartment so I drafted him. :-) He suggested that I might have to create a mapping of oids to stack frames at the time when objects were added to the transaction so that I could figure out what had been going on once I knew which oid would be the problem. This worked very well and much thanks to David.

I added the following to the place where the objects were being added to the transaction, ZODB.Connection.Connection.readCurrent():

if not hasattr(self, 'XXX'):
    self.XXX = {}
self.XXX[ob._p_oid] = sys._getframe()

Then once I had my POSKeyError in pdb.post_mortem() I could invoke pdb at the place where the offending oid had been added:

import pdb; pdbr = pdb.Pdb() ; pdbr.reset() ; pdbr.interaction(self.XXX[oid], None)

Using that I was able to find out what objects were involved. They were all portal_catalog/path index Trees.IIBTree.IITreeSet objects. What’s odd is that the error happens even if I do a catalog clear and rebuild first.

After much more debugging and discussion, we determined that the ZODB.Connection.Connection.readCurrent() support was involved somehow. Also, as I deleted the images that were involved I noticed another migration error earlier on went away for each image I removed. That’s the key, when the BLOB migration fails for one file or image, it uses a savepoint to rollback just that image’s migration and proceed with the rest. It sure seems like there’s a bug in the ZODB where when a savepoint is rolled back the objects registered with the transaction are cleaned up but the objects registered for readCurrent are not. Gee, someone should write a test case. :-)

In the meantime, it turns out the reason the image migrations were failing was something else all together but I thought I should share this debugging technique for capturing stack information when you can’t know which stack you need to inspect until later. I also hope this ZODB bug gets squashed.

Updated on 21 November 2012

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.

WSGI apps on IIS

2012-10-30T00:00:00+00:00

iiswsgi provides a WSGI server providing a FCGI gateway to IIS and setup.py commands for distributing apps as WebPI packages

The iiswsgi module implements a FastCGI to WSGI gateway that is compatible with IIS’s variation of the FastCGI protocol. It also provides distutils commands for building, distributing and installing Microsoft Web Deploy (MSDeploy) packages through the Web Platform Installer (WebPI).

The goals of the code in iiswsgi are to do the following for deploying WSGI apps on IIS:

make it open source as far as possible, right up to IIS
be Pythonic as far as possible, right up to the MSDeploy packaging
re-use our existing tool-chain for distributing packages
share the maintenance burden for a WSGI on Windows story across the community

For the Plone project, it’s always been simultaneously a necessity that we support a Windows deployment story and one of our biggest pain points. The Windows installers have always been very different from the other installers. They have had different layouts from our user and developer documentation and even from each other. They have never been maintained or supported by more than one entity, either a company or an individual, and as such have often and ultimately languished. And as for the poor individuals who have tackled the Windows installers, they have almost always burned out and can no longer provide any significant Windows support at all. This is not a healthy open source community dynamic. And yet there is wide consensus that it’s not an option not to have a Windows deployment story.

My hope is that by generalizing the IIS deployment architecture as a WSGI server and distutils commands, it can be of use to the general Python WSGI world. I also hope that by doing things ‘The Right Way’, it will be something that will be clearer and easy to support and maintain. With those two together maybe we can solve the burnout issue by distributing the maintenance load. I’d very much appreciate any help to that end, particularly including feedback on how to get there. I don’t care where the code lives and would be happy to see some of it merged back into the packages it derives from or moved into larger packages. So please let me know if you’d like to coordinate moving things around with me.

Help Needed

Any contributions are very welcome. Here are a few things I’m looking for in particular:

addressing Known Issues
IIS app name and Python dist name conventions
fostering community ownership
writing tests

I’m particularly apologetic for the last one, I’m ashamed by the lack of tests. In my defense, this whole problem was such a fog for me when I started that I just needed to start writing things and poking around. Believe me this is not my usual MO, I almost always do TDD and beg your forgiveness. :-)

I look forward to getting this going!

Updated on 30 October 2012

Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.