union.cms released on Python 3

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology aka using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4 on Python 3. It is used as CMS for large organisations.

In this post we want to present our process how we tackled the migration and briefly discuss occurred issues. Our migration plan looked like the following:

Port union.cms to Zope 4 while still running on Python 2.7.
Rollout this version to production – this already has happened in December 2018, see union.cms launched on Zope 4.
Port the code and the tests to Python 3 while keeping it compatible with Python 2.7 – this was done without branching a dedicated Python 3 branch, instead with a continuous integration system running against Python 2.7 and 3.7 to ensure the possibility of a prompt deployment.
Have some releases to production during the migration phase – this included also releases with new features developed in parallel.
Migrate the ZODB based database to be readable by an application server running on Python 3 – Thank you to everyone who contributed to zodbupdate which allowed to have a smooth migration, details see below.
Run manual tests on a staging system – by ourselves and by the customer to find edge cases not detected by the automatic tests. (There where only a few which where easy to reproduce and to fix.)
Rollout to production on Python 3 – this was done at the beginning of November 2019.
Drop the Python 2 support code – this is still open but of low risk.

During the preparation of the project the migration of the Data.fs seemed to be the hardest part. There where no tools in the beginning and the migration had to be done because of the str vs. binary issues between Python 2 and 3. (For details on this topic see Migrate a Zope ZODB Data.fs to Python 3.)

Actually the hardest part was the migration from Zope 2 to Zope 4. There are some internal changes where it is not easy to see what to do to make our own code compatible. Additionally we updated all dependencies to the newest versions which support both Python 2 and 3 to prepare the final switch to Python 3.

The migration of the code to Python 3 was a lot of work. It included to change some dependencies to other packages which have already been ported to Python 3 instead of depending on unmaintained ones. But most of the dependencies were in a usable state. pylint was used on Python 2 to detect code which will cause problems on Python 3. (This requires to use a pylint version older than version 2. We called it using pylint --py3k --disable=no-absolute-import src/** setup.py.) Most parts of the migration could be done automatically using modernize leaving the more trickier ones for the developers.

The migration of the database ran smoothly. The only issue was hidden inside ZCatalog where some index contents were stored as binary but str was expected. This could be solved by creating and running a migration script. (Details see Products.ZCatalog#83.)

The rollout to production went without problems even though databases of more than 10 GByte size had to be migrated. Thankfully it was possible to do the migration offline instead of being forced to do a live migration.

The whole migration project went about two years. We decided for a slower migration with at least some deployments to production to prove the already done steps in a live environment and to allow new features and bug fixes during the migration project. This approach went well, so we can suggest it for other migration projects.

By now union.cms runs live to Zope 4.1 using Python 3.7. It’s time to celebrate that Zope 4 on Python 3 can be used for actual projects in a live environment. 🎉

union.cms launched on Zope 4

union.cms is a content management system which was once developed on Zope 2. It was one of the early adopters of the Five technology: Using Zope 3 components in Zope 2. Now it is one of the proud early adopters of Zope 4. It is used as CMS for large organisations.

The porting of union.cms to Zope 4 was not that hard because union.cms does not use things from the dirty corners of Zope 2 which have been cleaned up in Zope 4.

Zope 4 is still in its late beta phase, this required more intense testing. There where no critical problems found before the launch. After launch we had an increased number of ConflictErrors which required a deeper investigation. But even this problem was solvable in a fair amount of time so it does not hurt the editors working daily with the CMS. (Details see Zope#413.) The users even claim it is running faster.

union.cms currently runs live to Zope 4 although it still uses Python 2.7. We are working hard to port it to Python 3. The launch is planned for the end of 2019.

Zope 4 comes with ZODB 5 which seems to be a big step from ZODB 3 which is used in Zope 2. There were no migration steps necessary besides using the current versions of ZODB and ZEO as union.cms still runs on Python 2.7. Upgrading to Python 3 will require a migration of the database contents. This migration is the last blocker before Zope 4 will get a final release.

Now it’s time to celebrate that the beta version of Zope 4 can be used for actual projects in a live environment. 🎉

Saltlabs Sprint: Zope and Plone sprint in a new location

After Earl Zope II is now nearly relocated to the Python 3 wonderland, gocept will move to a new head quarter in the next months. This is the right time to celebrate with a new sprint, as we have now even more space for sprinters. The new location is called the “Saltlabs”, a place for IT companies in Halle (Saale), Germany.

Sprint information

Date: Monday, 1st until Friday, 5th of October 2018
Location: Leipziger Str. 70, Halle (Saale), Germany

Sprint topics

This sprint has three main topics:

Create a final Zope 4 release

Before releasing a final version of Zope 4 we want to resolve about at least 40 issues: Some bugs have to be fixed, some functions have to be polished and documentation has to be written resp. reviewed. On the other hand there is the re-brush of the ZMI using Bootstrap which should be completed beforehand, as it modernizes the ZMI and allows for easier customisation, but might also be backwards incompatible with certain test suites. There is an Etherpad to write down ideas, tasks, wishes and work proposals, which are not currently covered by the issue tracker.

Port Plone to Python 3

The following tasks are currently open and can be fixed at the sprint:

successfully run all Plone tests and even the robotframework tests on Python 3
Zope 4 lost the WebDAV support: find resp. create a replacement
document the WSGI setup and test it in a production ready environment
port as many as possible add-ons to Python 3 (e.g. Mosaic and Easyform)
work on the Migration of ZODB contents (Data.fs) to Python 3
improve the test setup with tox.
start to support Python 3.7

Polish Plone 5.2

The upcoming Plone 5.2 release will appreciate some love and care at the following items:

new navigation with dropdown and better performance
Barceloneta theme: ease the customisation and improve responsiveness
parallelise the tests so they run faster
remove Archetypes and other obsolete packages

See also the list of topics on plone.org for this sprint.

Organisational Remarks

In order to coordinate the participation for this sprint, we ask you to join us on Meetup. We can then coordinate the catering and requirements for space.

As this sprint will be running longer than usual (five days), it is also possible to join only for a part of the week. As October 3rd is the national holiday, we are trying to organise some social event for those who are interested in having a small break.

For a better overview, please indicate your participation also on this doodle poll.

“allow-hosts” in buildout considered harmful

Today we had the following error message when re-installing a project from scratch:

 While:
   Installing.
   Getting section application.
   Initializing section application.
   Installing recipe zc.zope3recipes.
   Getting distribution for 'zc.zope3recipes==0.13.0'.
 Error: Couldn't find a distribution for 'zc.zope3recipes==0.13.0'.

Yes this is a really old recipe but it still exists on PyPI. We are using zc.buildout in Version 2.10, and do not use a custom index. So being forced to use HTTPS to access PyPI does not seem be the problem.

After searching way too long we found that .buildout/default.cfg contains the following statement:

allow-hosts =
   *.python.org
   *.gocept.com
   *.gocept.net
   effbot.org
   dist.plone.org

It restricts the allowed hosts for download but it seems to restrict the index, too. https://pypi.python.org/simple nowadays redirects to https://pypi.org/simple which is not on the list.

Suggestion: Remove allow-hosts if possible. It is more harmful than good, especially because packages are nowadays downloaded from https://files.pythonhosted.org.

UPDATE:

I filed an issue for zc.buildout at GitHub: buildout/buildout#447

September, 18th–20th: DevOps Sprint

Since we have a strong history in web development, but also were involved in operating web applications we developed, the DevOps movement hit our nerves.

Under the brand name “Flying Circus” we are establishing a platform respecting the DevOps principles.

A large portion of our day-to-day work is dedicated to DevOps related topics. We like to collaborate by sharing ideas and work on tools we all need to make operations and development of web applications a smooth experience. A guiding question: how can we improve the operability of web applications?

A large field of sprintable topics comes to our mind:

Logging

Enable web application developers to integrate logging mechanisms into their apps easily. By using modern tools like Logstash for collecting and analyzing of the data, operators are able to find causes performance or other problems efficiently.

Live-Debugging and Monitoring

Monitoring is a must when operation software. At least for some people (including ourselves), Nagios is not the best fit for DevOps teams.

Deploying

We always wanted to have reproducable automated deployments. Coming from the Zope world, started with zc.buildout, we developed our own deployment tool batou. More recently upcoming projects, such as ansible, and tools (more or less) bound to cloud services like heroku.

Backup

After using bacula for a while, we started to work on backy, which aims to work directly on volume files of virtual machines.

and more…

Join us to work on these things and help to make DevOps better! The sprint will take place at our office, Forsterstraße 29, Halle (Saale), Germany. On September, 20th we will have a great party in the evening.

If you want to attend, please sign up on http://www.meetup.com/DevOps-Sprint/events/191582682/.

Accomodation

For your stay in Halle, we can recommend the following Hotels: “City Hotel am Wasserturm”, “Dorint Hotel Charlottenhof”, “Dormero Hotel Rotes Ross”. For those on budget, there is the youth hostel Halle (http://halle.djh-sachsen-anhalt.de/). Everything is in walking distance from our office.

Reproducable automated deployments on RaspberryPi with batou

For continuous integration during development, we use Jenkins to automatically run tests for all projects we maintain. Some time ago we wanted to increase visibility of the results, so we set up a Raspberry Pi driving a few meters of LPD8806-based LED strip on which we can address single LEDs to represent the status of individual or aggregated builds.

Automating deployments is a good idea…

After an SD Card failure we were painfully remembered how hard it can be to set up a service where all parts were deployed manually. Fortunately we wrote at least some minimal documentation on how to set everything up, so after a few days we were presented with many broken builds. Of course nobody cared about the build status with all LEDs being dark. 🙁

Let’s automate!

Today we wondered if we can use our deployment-tool batou to make reproduceable deployments to a raspberry pi, and did some tests on a vanilla raspbian image (2013-07-26 “Wheezy”).

Preparing your Raspberry Pi

Of course, you can not deploy to it without some simple preparations. First thing is, batou needs to be able to log on the target host with a public ssh key, so we copied our public key to the raspi which has the address 192.168.0.5 in this example:

local> ssh-copy-id pi@192.168.0.5
pi@192.168.0.5's password:
Type password of user pi, default: "raspberry"

(If you don’t have the ssh-copy-id, you have to manually append your ssh public key to /home/pi/.ssh/authorized_keys, which you will need to create on a plain installation)

Manually install minimal requirements

Batou does also have a few requirements which are needed to bootstrap the environment:

mercurial – to pull the buildout which sets up batou
python-virtualenv – to create a clean python environment for the buildout
python-dev – to compile libcrypto against

Note: We are currently working on batou 1.0 which most likely will no longer need any of these.

You can install all the requirements at once with the following command on your raspi:

pi> sudo aptitude install mercurial python-virtualenv python-dev

Prepare batou

Now you are ready to do your first batou deployment to a raspberry pi. For our experiments we created a small hello-world batou deployment, containing a test component which deploys a file /tmp/test which contains “foo” to a raspberry pi specified by an IP address.

To begin, clone the repository on your local machine:

local> hg clone https://bitbucket.org/gocept/batou-on-raspberrypi
local> cd batou-on-raspberrypi

Now, edit environments/pi.cfg and set the IP-address of your Raspi.

To create the nessecary scripts to do the deployment, run buildout to create a sandbox containing all dependencies of batou and the scripts you can use to deploy:

local> python bootstrap.py
…
local> bin/buildout

Deploy!

After some minutes, your batou deployment sandbox will be ready for use. You most likely modified environments/pi.cfg so you need to check in that change first, because batou refuses to deploy a dirty working copy.

local> hg ci -m 'change ip of my raspi'

To run the deployment, call batou-remote with the name of the environment (“pi”, which corresponds to environments/pi.cfg). Because the ssh user you use to connect with the target host differs from your local user, you have to specify it with --ssh-user.

local> bin/batou-remote pi --ssh-user=pi

Batou will now set up itself on the remote side and deploys all components specified in pi.cfg. To show it worked, check if the deployed file contains the correct content:

pi> cat /tmp/test
foo

TL;DR

Create reproducable automated deployments for your software is great fun.
Preparing a raspi to be a target host for batou 0.2.12 based deployments is easy:
- Install python-virtualenv, mercurial and python-dev.
- Put your ssh public key on the raspi.
Example deployment can be found on bitbucket.

Sprint report: Deploying Python web applications – platforms and applications

Last week I met Stephan Diehl, Michael Hierweck, Veit Schiele, and Jens Vagelpohl in Berlin for a sprint. Our chosen topic was “Python web application deployment”. In this post I’d like to recap our discussions, gocept’s perspective on those, and the deployment tool “batou” that we have been incubating in the last months.

Continue reading “Sprint report: Deploying Python web applications – platforms and applications”

Tag: deployment