.. Ross Patterson's Blog imported post, created by `$ ./bin/rfc822-to-post` on Mar 15, 2021. .. meta:: :description: Can Plone work better on Windows? I have a working proof-of-concept with Web Platform Installer, Web Matrix and IIS Express that may improve Plone's installation story. Help me get it out the door. :keywords: pse12, Plone, Zope .. post:: Mar 31, 2012 :tags: pse12, Plone, Zope :author: Ross Patterson :redirect: @@redirect-to-uuid/77cf907ee10b493b91457c497d838eab ###################### Python Web Apps on IIS ###################### Can Plone work better on Windows? I have a working proof-of-concept with Web Platform Installer, Web Matrix and IIS Express that may improve Plone's installation story. Help me get it out the door. Please don't get me wrong, I hate Microsoft and Windows as much as the next OSS or Python web developer. I just feel it's important to be honest with ourselves about *why* they're bad. I grow a little tired of all the griping sessions whenever the sorry state of `Plone's `_ Windows `installer `_ support is discussed. I think there's plenty to gripe about and there's plenty that Microsoft does poorly but lets also be honest with ourselves. If Microsoft didn't do some things really well they wouldn't be a problem for us and we wouldn't even be discussing whether to support them as a platform. At any rate, the discussions about the Windows installer at the 2012 `Cioppino Sprint `_ were as frustrating and disappointing as ever. When everyone was through griping, however, I couldn't find anyone who at the end of that discussion thought we could afford strategically to drop the Windows installer or leave it to a 3rd party. I've started putting Microsoft's Web Platform Installer, Web Matrix, and IIS Express together for `Plone 4.2 and Python 2.7 `_ and have a proof-of-concept working on my Windows VM that actually gives us Open Source all the way up to IIS, see the screenshot above. Plone is actually really snappy running this way even on my very slow Windows VM. Below I document what I've learned as I proceeded and my final findings. If you read nothing else please read the `Help Needed!`_ section and let me know if you or anyone else can help me get this to a place where others can try it out. I'll be sprinting on this at the `post-PSE sprints `_ so anyone who has IIS or Windows installer experience, I'd love to sprint with you on this. .. contents:: ============ Help Needed! ============ Primarily, I need help with things that I can't do with just Web Deploy and WebPI. For example, the way IIS does FastCGI means that a configuration change has to be made to the global IIS config for *each* FastCGI app. IOW, it can't be done in the app/deployment-local ``web.config`` file. I know the shell command to run to make the change, but I don't know how to package that properly so that it runs with escalated privileges when necessary. Similarly, we need to figure out how to handle ZEO in the way that is closest to "correct" for IIS. Should we just install an autostart service? IIS have a database provisioning and control framework. Can that be adapted to also manage ZODB databases and control running ZEO processes? Or should we just wrap it up in such a way that ZEO is started the first, and only the first time, IIS launches the IIS process and shuts it down when stopping the app? If so, how? Since we're using WSGI via FastCGI using Flup, we're dependent on Zope's WSGI server. Unfortunately it lacks the publication hooks used by things like `plone.app.theming `_ and `plone.app.caching `_. This is really a bug in the Zope2 WSGI publisher and as such affects all WSGI deployments, not just Windows. `@Hanno `_ and `@davisagli `_ think they'll be able to get to this on in the next few days. Finally, if I'm wrong about any of the *technical* stuff in here, I'd love to hear about it. The documentation is crap for all this stuff and it has been way too hard to figure it all out, so I'd love any leg up I can get. ================== Supporting Windows ================== One thing that came out of the discussion, that I thought was interesting, was the sentiment that if we were going to support Windows we shouldn't do it while telling Windows users to take a leap at the same time. IOW, we say we support windows out of one side of our mouths, and blame users for choosing windows out the other side of our mouths when they encounter fundamental problems or find them selves encountering an unfamiliar learning curve. I see two things behind such problems: lack of integration with typical Windows tool chains, and differences in the documentation. The differences in the documentation largely come from the fact that the current windows installer isn't based off the `unified installer `_ so the buildout/package layout is different than what's in most of our docs. To that end, I started looking into what it would take to build a Windows installer from the unified installer. The lack of integration with typical Windows tool chains is a much bigger issue and much of that issue is a problem for most Python web applications, not just Plone. My research did, however, turn up some very promising new tools from Microsoft that we may be able to use to provide a better experience for Windows Plone developers and Windows Plone deployments. There are also two major audiences that tend to make use of Plone's Windows support: developers and deployments. `Steve McMahon `_ believes that most of the Windows downloads are by developers looking to get started doing Plone integrations or custom Plone development. Another target the installer has been used for, and this is more of what I've seen in my experience as a developer, is as a basis for production Plone deployments running on Windows servers. I definitely trust Steve's sense more than my own, but the good news is that this new Windows tool chain seeks to provide a nicely integrated experience from web app development through to web app deployment. So for the latter half of the sprint and ever since then, I've been obsessed with the Web Platform Installer (WebPI or WPI), IISExpress, and Web Matrix tool chain. The WebPI is actually a `fairly open framework `_ for describing web frameworks and web apps including dependencies and arbitrary installation commands in an `extended atom feed `_. Helicon uses this to provide a `Django install story `_ nicely integrated with this Windows tool chain. Using IISExpress and Web Matrix also allows developers to work in a local environment isolated to their user directory without needing to have the full IIS ($$$) installed. ================== Notes and Findings ================== Below is a loosely organized grab-bag of notes and findings I recorded while working on this. I publish it here only for reference. Hosting ======= * `ISAPI-WSGI `_ doesn't support IISExpress The ISAPI-WSGI OSS project is used by a number of `Python WSGI projects `_ to support IIS. `ISAPI-WSGI depends on py2win32 which in turn depends on IIS 6 `_ or the IIS 7 plugin providing IIS 6 compatibility. The IIS 6 compatibility plugin `isn't supported on IISExpress `_. * IIS only supports FastCGI This is how the WebPI and Web Matrix tool chain support PHP apps or other apps, such as plone, that need to have long running separate processes to run efficiently. This `FCGI support `_ is also restricted to using Windows named pipes. IOW, no TCP sockets to a separately running FCGI server process. * No current FCGI to WSGI gateway works with Windows The `FastCGI spec `_ calls for the ``STDIN_FILENO`` to refer to a socket which is then used for two-way communication between the server and the process handling requests. Naturally, Microsoft has embraced and extended this standard in IIS such that instead of a single socket it uses two Windows Named Pipes one each for receiving and sending. IIS may also support TCP sockets behind the scenes. This means that anything that expects to use normal sockets for FCGI, like `flup `_ won't work with IIS. I can find no other OSS FCGI to WSGI gateway that supports Windows named pipes. * `IIRF `_ doesn't support IISExpress We might be able to use `Ionic's Isapi Rewrite Filter `_ to proxy IIS requests through to a separately running Python process. This is less then ideal since it may make the Web Matrix experience less integrated and requiring more un-Windows-like knowledge. IIRF doesn't support IISExpress, at any rate, though it may be possible to manually install it into IISExpress. * `Helicon Zoo Module `_ I suppose the lack of other working options is exactly why Helicon Tech built it's own solution for this. I prefer to have OSS all the way up to IIS itself, but that's difficult when you play in the Microsoft sandbox. At least it looks like Helicon has paid some attention to `performance `_. Furthermore, having company support may yield better long term maintenance for IIS support than an OSS project in the Microsoft universe. That still doesn't mean I like it. Part of the Helicon Zoo Module is a `zoofcgi.py` script which is their own FCGI-WSGI gateway that seems to use ``STDIN_FILENO`` instead of sockets to do the FCGI communication. IOW, Helicon has written a totally new FCGI-WSGI gateway that works with IIS's broken FCGI implementation. In their implementation I see a lot that looks familiar from `flup `_. * Modifying zoofcgi.py to run Plone Unfortunately, Helicon's `zoofcgi.py` only supports a Django WSGI app or a example wsgi app, with no way that I saw to use it to run an arbitrary WSGI app. Replace the `run_example_app()` function with the following to `enable loading of an arbitrary WSGI app `_ from a `Paste `_ `*.ini` file: .. code-block:: python from paste.script.util.logging_config import fileConfig from paste.deploy import loadapp def run_example_app(): config = os.environ.get('WSGI_CONFIG_FILE') if config: config = os.path.abspath( config ) fileConfig(config) application = loadapp('config:%s'%(config,)) else: application = example_application if __debug__: logging.info('run_fcgi: STARTED') FCGIServer(application).run() if __debug__: logging.info('run_fcgi: EXITED') In `~\My Documents\IISExpress\config\applicationhost.config` change ` `web.config` in the site root: .. code-block:: xml It should be possible to re-use `zoofcgi.py` without modifying it by making it importable, IOW putting it's dir on `sys.path`. Ideally, I'd like to see instead is a library that can wrap the named pipes provided by IIS such that they emulate a normal socket. * `iiswsgi `_ and `filesocket `_ I implemented a crude `socket-like implementation called filesocket `_ that wraps two open files and acts like a socket. Then I copied and pasted all the bits from `flup `_ that depend too rigidly on actual socket or on things not available in windows and `adapted them as needed `_. Finally, I wrapped all that up with the necessary `paste.deploy `_ bits so that IIS can launch Plone as a WSGI app, after adding bits from the `WSGI buildout `_ to the unified installer, from a Paste ini file, using the `paster config without a server `_ since we're interfacing directly with IIS. Finally, add the following to ``configuration/system.webServer/fastCgi`` in ``~\My Documents\IISExpress\config\applicationhost.config``: .. code-block:: xml This is one of the things I need help with. This can be done using the IIS `appcmd.exe `_ program but I need to know how to do this during the Web PI install process. It works and now we have Open Source all the way up to IIS! * Zope namespace URLs Zope has a lot of special URL structures like `++resource++foo.css` but `IIS chokes on these `_. Add the `following `_ to web.config: .. code-block:: xml ... * WebMatrix not launching instance until it's been run in the foreground I'm noticing that the Plone instance keeps restarting. Before I fixed the pluses in URL issue, it at least eventually got running stably after several restarts, but now it's restarting much more frequently. I thought it was caused by putting the resource registries in development mode because it means more requests and the number of requests that IIS lets build up before restarting the FCGI process is reached. When Plone is warming up, almost all of these request pile up. But I've also seen it be perfectly stable with the resource registries development mode on so I'll need to catch this in the act again to debug. Since then, I think the problem is that for some reason it needs to be started in foreground mode before WebMatrix/IISExpress can successfully start it. * How to restart in debug-mode? Logging? What is the best way to make debugging information accessible to the MS toolchain? What's the best way give WebMatrix developers and IIS admins to restart the instance in foreground-mode and/or debug-mode? What about giving them more integrated access to the logs? Should the default buildout configuration of the unified installer for Windows use event log handlers? Currently, the error reporting out of WebMatrix/IISExpress when launching the instance or anything else fails is miserable. Is there something we can do to better integrate with it's trace logs such that more Zope/Plone/Console/UNIXy specific output is reflected somewhere? * Fixed two Windows bugs `Docutils 0.7 conflicts with PIL `_ causing:: AccessInit: hash collision: 3 for both 1 and 1 Upgrading to `Docutils 0.9 fixes this `_. Also `contributed a fix to Zope2 `_ that addresses many of the stale lock file problems on windows. Packaging ========= * Creating a Web Deploy Package Microsoft provides some `docs `_ for creating Web Deploy packages. It may also be possible to use the `msdeploy `_ tool to make a package in a more automated way. More helpful pages about making web deploy packages: * `Lessons Learned from Packaging our ASP.NET Application for the Microsoft Web Platform Installer `_ * `Package an Application for the Windows Web Application Gallery `_ * Bootstrapping the Unified Installer The first time building the Web Deploy package based on the Unified Installer, some things need to be installed and configured that can't really be automated. Some of these steps shouldn't be necessary in the long run, since it should be possible to use the existing Web PI feed to install the dependencies now that the feed is working. Much of this should also be added to the Unified Installer in some sort of platform specific way. But if the Web PI feed ever needs to be created anew, or maybe when switching Python versions, it may be necessary to do the same things I had to do on my Windows VM to begin creating the Web PI feed and Plone Web Deploy package. I'm documenting all that in a `README in the Unified Installer `_. * Non-buildout root Web Deploy Package? For my proof-of-concept, I manually created a ``web.config`` file in the ``zeocluster`` folder of the buildout created by the Unified Installer and then manually added that as a "site" in WebMatrix. Eventually, we need to figure out how to make the Unified Installer buildout root also be the root of the site for WebMatrix/IISExpress/IIS or how to have a web deploy package where the root of the installed site is a subdirectory of the web deploy package. * Deploying to a path with spaces I ran into the following error when the buildout was at a location with spaces in the path:: WindowsError: [Error 87] The parameter is incorrect To try to narrow down the issue, I used buildout to create a debug-mode only script with the following part: .. code-block:: cfg [debug] recipe = zc.recipe.egg eggs = ${instance:eggs} entry-points = debug=Zope2.Startup.run:run initialization = import sys sys.argv.extend(["-C", r"${instance:location}\etc\zope.conf", "-X", "debug-mode=on"]) The script this produces works just fine, so the problem is in the `plone.recipe.zope2instance `_ recipe. * Installing FastCGI application, running bootstrap and buildout Using ```` in ``manifest.xml`` is possible in theory: - http://forums.iis.net/t/1161375.aspx - http://stackoverflow.com/questions/1511792/runcommand-provider-in-msdeploys-manifest-xml-file I tried adding doing this and then just adding a site from a folder in WebMatrix, nothing. But it may be that WebMatrix doesn't do anything with ``manifest.xml`` when adding a site that way so I'll have to try it by adding it to the WebPI feed. May also want to experiment with manually using ``msdeploy.exe``. Installing ========== * Writing the Web Platform Installer Atom Feed Microsoft provides a `reference `_ to what elements and attributes it adds to the ATOM namespace. * `bdist_wininst `_ Python Installers aren't silent Windows MSI installers seem to support a `/verysilent` flag. Unfortunatley, the Python `distutils` support for building Windows installers has `no silent option `_. A `workaround `_ may be to use Web PI's support for arbitrary installer commands to extract the installer without running it. It may also work to convert the wininst packages into `MSI packages `_. In short, we need MSI's for `PIL `_, and `pywin32 `_, and binary windows eggs for `lxml `_. Researching free software to build MSI's. We need custom actions support, so `Advanced Installer won't do `_. It turns out that `easy_install `_ can make an egg out of a ``bdist_wininst`` exe installer. So we can now move all those dependencies into buildout by using `find-links` that point to the ``bdist_wininst`` exe downloads. This is how I'm getting pywin32 now. * Get the SHA1 Hash of the Web Deploy Package Microsoft provides a `tool `_ for generating this hash. I'm not sure if using this tool is strictly necessary or if there may be a way to get the `msdeploy` tool to do this as a part of a more automated packaging process. * Web App Gallery Microsoft actually has a `web application gallery `_ that they say applications can `submit `_ their application to. If approved, these applications would be available in Web PI without requiring the user to enter a custom feed. Miscellaneous Links and References ================================== * `Paster/INI flup FCGI-WSGI config `_ * `launching CGI/FCGI processes `_ * `IIS Configuration Reference `_ * `So called IISExpress docs `_ .. update:: Nov 12, 2017 Imported from Plone on Mar 15, 2021. The date for this update is the last modified date in Plone.