Richard Jones' Log Richard Jones' Log: Python

This is getting frustrating. I subscribe to the PyPI RSS feed and far too often entries posted contain no description or they have something equally useless like "A wrapper for Shrubbery". What the hell is a "Shrubbery" and why should I care?

The second frustration is when a package has a somewhat useful short description "A package for doing spammish things" but there's no further information: there's no longer description, and the home page link points at the PyPI page itself. Nothing for the curious Python developer to get an idea of what the package does. Compare this with the Roundup Issue Tracker PyPI entry.

I have to ask in all seriousness why even bother uploading to the index if you can't be arsed to put in a description?

This is part four of my continuing adventure in packaging my Python application.

In part three I was close to a solution. In the end I have decided to:

1. Use the "build_apps" distutils command I wrote to generate zip application bundles for the three target platforms. The zip file contains all libraries (pure python, thankfully) and a launcher script to run the application.
2. Split out the examples from the rest so there's just one examples zip file and the other zip files are about half as heavy.
3. Generate a standard distutils source distribution which is intended for Linux system package maintainers.
4. Upload all those files to Google Code using a modified googlecode_upload.py script.
5. Still register with PyPI but set the download_url to point to Google Code.

The upload script was modified in two ways:

1. First, it's broken with the svn module available under Ubuntu (and others). I modified the code to directly parse the svn passwords. It's a gross hack and probably not widely useful so I won't submit a patch. For the curious the code is below.
2. Secondly I hard-coded it to upload the five files I want to send, just to make my life easier.

I'll await feedback from actual users but I believe that this solution will be good enough.

By the way, I just released a new version of the application in question. It's Bruce, the Presentation Tool (who puts reStructuredText in your projector). The 3.2 release is pretty nice with support for external styles, recording, automated playback of various types, gradual exposure of lists and some other stuff. It's pretty cool.

On to the change to googlecode_upload.py, which replaces get_svn_auth() with:

def get_svn_auth(project_name, config_dir):
  """Return (username, password) for project_name in config_dir."""
  realm = (' Google Code Subversion Repository'
           % project_name)
  authdir = os.path.join(config_dir, 'auth', 'svn.simple')
  for fname in os.listdir(authdir):
      info = {}
      key = None
      for line in open(os.path.join(authdir, fname)):
          line = line.strip()
          if line[0] in 'KV':
              continue
          if line == 'END':
              break
          if line in 'password username svn:realmstring'.split():
              key = line
          else:
              info[key] = line
      if info['svn:realmstring'] == realm:
          return (info['username'], info['password'])
  return (None, None)

This is part three of my continuing adventure in packaging my Python application.

Thanks again for the advice given in response to my last post on Linux-specific application packaging issues. I've decided to leave packaging up to the experts and help them as much as possible. To this end I've:

1. split the installation instructions out of README.txt into INSTALL.txt and put in a notice that the INSTALL.txt is not for regular users,
2. looked into how to best support more organised system package maintainers (the INSTALL.txt is my start at that), and
3. generated stand-alone application downloads for Linux, Windows and OS X.

The approach I've taken for the stand-alone packages is as follows:

1. bundle all of bruce, pyglet, cocos, docutils (including roman.py) and pygments into a library zip file,
2. create a simple script for each O/S to run bruce using that zip file, and
3. create the distribution zip files with those contents alongside the README.txt, HOWTO.txt and PKG-INFO.

Since the examples directory is pretty big I've also created a separate "examples" zip file.

This is implemented as a new setup.py command "build_apps" (as BuildApps.) I didn't use zipfile's PyZipFile but rather implemented my own. PyZipFile only includes compiled modules (.pyc or .pyo). The problem I have with this is that the compiled modules are not compatible across Python minor releases (modules compiled for Python 2.5 are not compatible with 2.6 and vice versa.) I would have to include separate library zip files for each system interpreter I wished to support. After some very simple checks I determined that the performance difference is negligible if I just bundle the original .py files instead.

I wrote "build_apps" so it'd fake being an "sdist" command enabling me to upload the resultant file to the Python Package Index (PyPI). This part is really quite inelegant but I'd like to keep the downloads all in one place. Unfortunately the file listing in PyPI lists the files as "Source" which isn't good. PyPI has no concept of an "application" though.

The Windows and OS X application bundles are completely untested.

I intend to investigate py2app and py2exe in the longer term to produce more system-friendly programs (no need to install Python, have icons, etc) but for now I believe this solution is good enough.

Just focusing on the Linux side for a moment.

Thanks to everyone who responded to yesterday's post about Python application packaging. My efforts will now be focused on creating files suitable for Linux package managers. This would initially be Debian software package (.deb) and Red Hat Package Manager (.rpm) - I considered Gentoo Linux's ebuild but really the audience for that is too narrow and they have tools to convert rpm and deb files to their own format.

Tasks for my application:

1. I'll have to organise my install and generate meta-data files according to the two formats: Debian package basics and maintainer's guide (which may differ from Ubuntu's) and RPM package basics,
2. Generate the package files: .deb via stdeb (there's also easy-deb) and .rpm using the built-in distutils bdist_rpm), and
3. Host the files somewhere.

If you don't want to use stdeb or easy-deb there's a nice ShowMeDo screencast of creating a .deb for a Python program by Horst Jens.

One of the commenters mentioned above, Daniel, has already produced .deb files using stdeb. I say files because he produced one each for pyglet, cocos2d, and bruce. The docutils and pygments libraries are already available in Ubuntu's package repository.

Already I have a problem: Ubuntu also packages a version of pyglet. The pyglet version packaged is 1.0 which is quite out of date compared to the version I'd like to package, 1.1 (or even 1.2 pre-release). If I install the Ubuntu version and then try to upgrade to Daniel's version I get the rather unfriendly error:

richard@shiny:/tmp$ sudo dpkg -i python-pyglet_1.1.2-1_all.deb
(Reading database ... 165323 files and directories currently installed.)
Preparing to replace python-pyglet 1.0.dfsg-1 (using .../python-pyglet_1.1.2-1_all.deb) ...
Unpacking replacement python-pyglet ...
Setting up python-pyglet (1.1.2-1) ...
pycentral: pycentral pkginstall: not overwriting local files
pycentral pkginstall: not overwriting local files
dpkg: error processing python-pyglet (--install):
subprocess post-installation script returned error exit status 1
Processing triggers for python-support ...
Errors were encountered while processing:
python-pyglet

Removing the existing python-pyglet package resolves this issue of course. I don't know what the state is for other Linux distributions.

So, there's still some issues:

1. There's a potential issue when a Linux distribution starts packaging bruce, pyglet and cocos2d itself. As mentioned above there's already one conflict on Ubuntu.
2. I've still got an issue with compatibility. While I control the package files that's not a big deal but once the distributions start doing so I no longer have any control, and
3. Is it better to require users to download .deb files or would it be better to be able to just add "deb http://pypi.python.org/debian/ pypi" to Debian's apt configuration, and whatever the similar configuration is for rpm-based systems (yum or urpmi)?

I also noticed getdeb and CNR (click n run) while poking around. They could be good application promotion tools. There's probably others.

(Updated to include link to Ubuntu packaging guide)

I've got an application that I'd like to make more user-friendly to install.

1. On OS X and Windows applications have everything (program, libraries, data) bundled. This is good as it removes issues around library compatibility and installation. In a previous job I've handled installing on these platforms but I also had ready access to them to develop & test on and now as a FOSS developer I don't.
2. On OS X I know about py2app. Since there's no "installation" this is effectively done. That'll help.
3. On Windows I know about py2exe but I still need to create an installer for the application (to add it to "Program Files", the start menu and possibly associate file extensions). There's a link to Inno Setup from the py2exe tutorial and I think it'll help.
4. On Linux applications may be a stand-alone bundle but are encouraged to use a standard install pattern (programs in /usr/bin, libraries in /usr/lib/, data files in /usr/share). If I don't follow that pattern I won't get picked up by Linux distributions.
5. In the Python world we have two technologies: distutils and setuptools. Parts of distutils are used in py2app and py2exe. At first glance distutils and setuptools appear to be suitable for the Linux case, except...
6. Installing with distutils can cause problems if the internal library contents change, resulting in asking users to manually remove any old install.
7. Python eggs, and the associated Python library search path / site.py shenanigans, cause developer and end-user pain when an application has its own local copy of an installed egg library. Many scary user warnings out on the console, and also possibly incorrect behavior. virtualenv can help solve this but would require bundling a build of Python with the application.
8. Shared libraries can cause compatibility issues that I'd like to avoid.

So, does anyone have any advice?

1. Is there a py2exe/py2app for Linux that I've not found? Do I have to use auto(conf|tools)?
2. I don't believe I should package Python 2.5 but if I'm not using a "local" version then I've a good chance of running into site-packages egg issues and I'd rather not have to:

   import sys
   sys.path = [e for e in sys.path
       if not e.endswith('site-packages') and not e.endswith('.egg')]

   at the start of my application.
3. Also, are there handy systems I could access to run my py2app/py2exe?

Aside: having just finished writing this post, proof-reading it and looking for one more link I stumbled upon this and this and the huge discussion that lead up to those. The discussion appears to be focusing on packages (libraries) rather than applications.

Update: I've had to withdraw the presentation due to a clash of scheduling that means I won't be able to attend PyCon 2009.

Yes, just the one :)

The summary as submitted was:

I will talk about the way Python is used in the operations of ekit.com Inc, a telecommunications provider for travellers. I will briefly touch on the overall architecture before delving into some specific examples of how Python is being used to rapidly develop robust and flexible systems. This talk is meant to be entertaining, informative and possibly instructive.

My thanks go to the thoughtful comments and reviews from the program committee which helped focus the proposal and should help guide final delivery.

Python 3.0 that is.

As someone who has to deal with unicode and charset issues I'm really looking forward to the day I can ditch the current two string types and all the confusion and bugs that arise from them. Unicode / encoding will be so much easier to explain now!

I also learnt something new in Anthony's "what's new in Python" talk at OSDC: comprehensions are everywhere now!

>>> [chr(c) for c in (83, 80, 65, 77)]
['S', 'P', 'A', 'M']
>>> {chr(c) for c in (83, 80, 65, 77)}
{'A', 'P', 'S', 'M'}
>>> {c:chr(c) for c in (83, 80, 65, 77)}
{80: 'P', 65: 'A', 83: 'S', 77: 'M'}
>>> ''.join(chr(c) for c in (83, 80, 65, 77))
'SPAM'

That's awesome (for those new to things, that's a list, set, dictionary and generator comprehension).

Also, kudos to Ubuntu. "apt-get install python3" worked, and more importantly didn't replace the "python" command (rather requiring "python3" to invoke it). Unfortunately the same isn't true under Windows where the installer replaces the existing ".py" file association to point to Python 3 instead of 2.6, thus breaking all installed Python scripts. Boo.

Yeah, this blog still exists... and here's a meme to prove it...

(all the cool kids are doing it, apparently...)

Taken with the crappy webcam in my Dell laptop using the KDE screenshot tool to capture the window contents of the configuration panel of Kopete and using the Gimp to crop out the picture. I did have to play with the color balance of the webcam so I didn't come out too purple. I hope that's not against "the rules."

I just realised that I released the latest version of Roundup (1.4.5, a maintenance release) 7 years almost to the day after my first public release (0.1.1, released on the 18th of August, 2001.)

I've been tasked to do some performance analysis on our new Travel Journal product (now "officially" with Fire Eagle support). The last time I worked for ekit I did a bunch of website performance testing and improvements.

I even wrote a tool to help me at the time! Last touched 5 years ago, and still at "rc1" :)

I'm now updating it (to Python 2.3 heh) and will release that 1.0 final that should've been released back then...

Excellent timing too, with the next challenge in a month's time!

If you're not already subscribed, head over to the Python Magazine site to read the article.

Bruce, the Presentation Tool has grown syntax-highlighting support thanks to Pygments. It's optional, and used if the presentation has either a doctest block or a ".. code:: <language>" block.

Bruce the Presentation Tool

Bruce the Presentation Tool is for people who are tired of fighting with presentation tools. Presentations are composed (edited) in plain text files. In its basic form it allows text, code, image, interative Python sessions and video. It uses pyglet to render in OpenGL.

Get it from PyPI

.. load-style:: big-centered

Text displayed centered on the default white background.

----

A new page, separated from the previous using the four
dashes.

Ut enim ad minim veniam.

A Page Title
------------

Pages may optionally have titles which are displayed
centered at the top by default.

Bruce, the Presentation Tool is getting incrementally better. It's a nice little hobby project because there's always just a little more that could be done, and never a huge task at any given step. Thus I can do a little each time when I'm on the train (30 minutes of hacking on my EEE PC).

ReStructuredText is now the only input format - the old "Socrates"-based implementation is gone. Everything that it could do in terms of formatting may now be done in ReST - and more.

Some of the stuff I mentioned in my last post has been resolved - though it's not final I'm definitely leaning towards staying with directives as they give me more, and Bruce will be able to be more intelligent at producing HTML than the generic html4css2 writer (when dealing with Bruce presentations). Title-less pages are there, so are video pages. Titles are incorporated into the page decoration, allowing some nice tricks. Enumerated list generation handles more styles.

Next up I'll be looking at reintroducing the python interpreter and python code page types - but in the New Bruce they're actually just page elements that may form part of any page.

ps. for the people who asked to see it in action - grab the SVN and run something like "python -m bruce examples/example.rst" or any of the .rst in the examples directory. Bruce is being developed against the pyglet SVN HEAD though (it might work against 1.1, I'm not sure)

As previously hinted Bruce, the Presentation Tool may now display presentations authored in ReStructuredText. I've had a chance to do some more work on it lately, and need to take a step back and think about what I'm trying to do :)

The ReST capabilities currently available are:

1. Sections denote pages (just like all the other ReST presentation tools),
2. Lists are handled (some features missing). A lot of inline markup is handled. Images are handled, both inline and stand-alone.
3. The stylesheet and other configuration may be changed on the fly with ".. config::" directives.
4. The background decoration* may be specified with a ".. decoration::" directive.

*: the decoration stuff is new as of very recently too. Currently it controls the background colour, but also allows rendering of quads (with colour gradient if you like) and images in the background. There's still much to do like scaling the decoration layer to the screen size, and adding more toys to decorate with like lines and possibly splines. Not sure how far to take it.

Missing from the ReST side though is:

1. Pages without titles (this would require some sort of "page" directive to indicate a new page has begun).
2. Other page types like the Python interpreter, Python code and Video.
3. Handling notes and running-sheet HTML generation sensibly.
4. Allowing custom page types, perhaps through ".. custom:: <module name>"

Those things aren't insurmountable. I'm becoming increasibly convinced that ReST is a better way to go than the custom markup format, but I'm having trouble with the final decision to give up on the old format.

Of course maintaining two parsers is ... silly.

I'm pretty sure I've made my decision, but thought I'd throw this post out anyway in case anyone had any thoughts or encouragement...

PyWeek 6 is all done. The entries are in and rated. It's been another great PyWeek with continued growth in participation and games produced at the end. So many games that I'm considering extending the two weeks for judging!

Greg Ewing has proposed that there be an additional Pyggy Awards challenge which encourages developers to take games produced during the latest PyWeek and improve on them over three months. At the end we run a new round of awards.

python -m bruce presentation.rst

I'm unwinding from PyCon in my cousin's apartment in Pittsburgh. Mostly this involves catching up with Nat, trying to get some sleep (unsuccessfully - I seem to have some sort of delayed-jetlag insomnia), catch up on email (partially successfully) and dump a bunch of photos from the camera. Photos have been updated and I now have a new title image on my blog page (thanks Toby for the old one :). I've compiled all the panoramas that were worth it and I think there's some nice ones in there. I got some nice photos of Evelyn and the PyAr guys from our downtown touristing (somehow Gui managed to dodge the camera... will have to wait to see Lucio's photos).

It's snowing outside.

Can't wait to be home.

The conference itself met and exceeded expectations. I saw some enlightening talks had some great conversations with some bright people. As already mentioned pyglet was a hit. Bruce was too, which was fun. I think I may have a contributor in Brian Dorsey, who was really nice and had some ideas and made the "python code" (in a presentation page write/edit a python program and hit F4 to execute it) Bruce page work.

Well, the pyglet talk's done. The pic above is a little panorama I took about 10 minutes before the presentation (a nice man turned off the incredibly bright spotlight down the other end of the room). The room filled up a lot more as we got closer to the start time with people sitting in the aisles.

The talk went really well, with only one Bruce problem -- I couldn't see the projection screen from the podium. I intend to code up a facility to show the contents of the projection screen on the second screen (ie. laptop) alongside the notes but I just didn't have time. This combined with a very cheap mouse that didn't always click when it was supposed to meant sometimes I was talking to the wrong slide content. Oh well. It was well-received.

Oh, and I couldn't show off the t-shirt I had specially printed and organised delivery to Phil Hassey so I'd have it for the presentation. I was wearing a pullover because it was so damned cold in the presentation room :(

And then there was the BoF afterwards during which I just answered more questions for about an hour.

pyglet's got a considerable buzz here :)

(I have another panorama on flickr that I took during Guido's plenary)

Update: I've uploaded the HTML presentation notes.

I'm at PyCon. It's been a pretty busy time so far - have seen a bunch of great talks, have talked to a bunch of great people and have been almost totally let down by technology. My EEE now refuses to charge up, so I'm really glad I brought the work laptop too. Though it's having problems - I can't get onto the wireless network using it, so I've been running in Vista. Fortunately pyglet (and hence Bruce) runs just fine in Vista, so my presentation is going ahead as planned. My work phone isn't working for some reason.

It's good to be here though :)

ps. if you're at PyCon and care, there's going to be a pyglet BOF at 2:45 today, right after the presentation. See the board downstairs.