Richard Jones' Log

Mon, 20 Jan 2014
Python 3.3 and virtualenv

We're kicking off some new projects using Python 3 (yay!) but had some issues getting virtualenvs working. Which is kinda ironic given that Python 3.3 included virtualenv in it, as pyvenv. Unfortunately, pyvenv isn't quite the same thing as virtualenv, and in particular it doesn't install/include pip and setuptools. There's also some additional issues introduced under Ubuntu.

First, you'll need to obtain Python 3.3. Some of the methods you could use will work and some are known to produce a non-viable environment. In particular:

  • OS X: get it from homebrew ("brew install python3"). I've not tried other avenues, but this works and is the easiest approach in my opinion.
  • Ubuntu: get it from source, building like so:
    sudo apt-get install build-essential libsqlite-dev sqlite3 bzip2 libbz2-dev
    wget http://python.org/ftp/python/3.3.3/Python-3.3.3.tar.bz2
    tar jxf ./Python-3.3.3.tar.bz2
    cd ./Python-3.3.3
    ./configure --prefix=/opt/python3.3
    make && sudo make install
    Do not attempt to use any currently-available pre-built packages (eg. from a PPA) as they will create broken virtualenvs. See this discussion for some enlightenment, but note the lack of a reasonable solution.
  • Windows: no idea, sorry.

Now that you've got a Python 3.3 installation, you can create your virtual environment. You do this with this command combination:

pyvenv-3.3 
. /bin/activate
wget https://bitbucket.org/pypa/setuptools/raw/bootstrap/ez_setup.py
python3.3 ez_setup.py
wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
python3.3 get-pip.py

Now you should have a viable, working Python 3.3 virtual environment.

Fortunately Python 3.4 is going to improve on this by installing pip alongside python.

Also, pip 1.5.1's "get-pip.py" will let you skip that extra setuptools install above when it's out (real soon).

category: Python | permanent link
Thu, 25 Jul 2013
Python Game Programming Challenge (PyWeek) #17 is coming!

The 17th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 1st to the 8th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Fri, 12 Jul 2013
PyCon AU 2013 and Things You Should Not Do In Python

So PyCon Australia is done for another year and it was a corker. I'm somewhat overwhelmed and Graeme Cross has an great summary of the event, but I really must repeat what an amazing job the organisers did in running a quality, smooth event packed with Python. See also Katie Miller's writeup about the 10 things that PyCon AU does particularly well.

Personally, I pleased as punch that my talk Don't Do This went as well as it did - I had a blast giving it and it was well-received.

Also, Ben Finney talked a bunch about PyCon AU in Wednesday's Byte Into It on 3RRR.

category: Python | permanent link
Thu, 21 Mar 2013
How awesome was PyCon?

PyCon was awesome.

It started with the Young Coders tutorial where I had the privilege of helping out as an assistant. The kids were amazing. Barbara Shaurette and Katie Cunningham were amazing. It was awesome.

The next day I popped in to help set up that room again but then spent the rest of the day focusing on my afternoon tutorial teaching people to write games in Python. It was packed - 35 or so attendees had signed up but then a bunch of the kids from the first day of Young Coders also came along to watch and learn. Including an 8 year old girl who told her dad she wanted to write video games. Awesome.

Then the conference proper started with an inspiring opening speech by the conference chair Jesse Noller about his vision of Change the future - education, outreach, politeness, respect, tenacity and vision. A broad, ambitious and awesome vision, and one I can fully get behind. Then Eben Upton took the stage to tell us about his journey with the Raspberry Pi project and a little of his vision. Lots of similarities. And then everyone got a Raspberry Pi. Awesome.

People wanting help figuring out how to teach kids to program their Raspberry Pi could try the free Raspberry Pi Education Manual PDF.

I spent a lot of time during the conference split between talking through various things in the hallway (I'm the Cheeseshop BDFL and there's many things afoot there and PyCon is a great time to move things forward very quickly), and spending time in the Raspberry Pi lab, chairing talk sessions for amazing speakers and generally having my mind awesomely expanded.

I taught many people the basics of pygame. I ran into the Kivy developers, and talked to them about the Pi. During the sprints they ported Kivy to the Pi and even developed some simple games using Kivy, the Pi and some simple hardware sensors (I'm playing the game with a tilt sensor).

I had an inspiring conversation with James Bennett during which a light went off in my head and the result is Python Enhancement Proposal (PEP) 439 for "Inclusion of pip bootstrap in Python installation". (apologies if you get a badly-formatted version; it should be fixed shortly.)

I'm exhausted. The people here - the community - is amazing and vibrant and brilliant and huge. 2500 people at the conference, 20% women. Awesome.

Seriously, look how many people were there...

category: Python | permanent link
Mon, 18 Feb 2013
Python Game Programming Challenge (PyWeek) #16 is coming!

The 16th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 14th to the 21st of April.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Fri, 15 Feb 2013
PyPI password-related security changes
TL;DR: please log into PyPI and change your password.

Recently we have been auditing and improving security of the Python Package Index (PyPI) and other python.org hosts.

You may be aware that the wiki.python.org host was compromised. Since we must assume that all passwords stored in that system are also compromised, and we also assume that some users share passwords between python.org systems, we are performing a password reset of all PyPI accounts in one week's time, at 2013-02-22 00:00 UTC.

If you log in before that deadline and change your password then you'll be fine, otherwise you'll need to use the password recovery form after the reset has occurred.

Additionally, we would ask you to begin to access PyPI using HTTPS through the web. We're in the process of installing a new SSL certificate so the current Big Red Certificate Warning should go away very soon.

We are in the process of updating the Python packaging toolset to use HTTPS.

These steps are but a couple of those we're intending to take to better secure PyPI. If you are interested in these matters I encourage you to participate in the discussion on the catalog SIG.

Finally, we apologise for any inconvenience these changes have caused.

category: Python | permanent link
Sat, 09 Feb 2013
Introducing pythonhosted.org - the new packages.python.org

We've just added a new domain to the python.org stable, pythonhosted.org, to replace the existing packages.python.org domain. All accesses to the old domain are redirected to the new - all existing references will work. Google et al should catch up eventually.

This was done as part of an overall review of security of the PyPI website. Warning, that thread is longish :-)

category: Python | permanent link
Fri, 04 Jan 2013
Compiling cx_Oracle on OS X

Occasionally I need to compile cx_Oracle on OS X and the 32- and 64-bit worlds collide head-on in a mess of "implicit conversion shortens 64-bit value into a 32-bit value" and "file was built for unsupported file format which is not the architecture being linked (x86_64)" errors.

I keep forgetting the various steps needed to make this work correctly so here they are:

  1. Create a virtualenv with a python interpreter stripped to 32-bit only. In the virtualenv bin directory ($WORKON_HOME/name_of_virtualenv/bin):
    % mv python python.fat
    % lipo python.fat -remove x86_64 -output python
    Doing this saves all the hassles of that "arch", "VERSIONER_PYTHON_PREFER_32_BIT" and "defaults write blah blah" guff and doesn't affect any other virtualenvs.
  2. Now that we have a "thin" Python compatible with the Oracle library, we can build cx_Oracle. Unfortunately even though Python is thin its configuration is not so we need to force things using an obscure environment variable that distutils will pick up:
    ARCHFLAGS="-arch i386" pip install cx_Oracle

There, that was easy, wasn't it?

category: Python | permanent link
Thu, 09 Aug 2012
Python Game Programming Challenge (PyWeek) #15 is coming!

The 15th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 9th to the 16th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

Check out the help page for how to compete and the growing resources message board post.

category: Python | permanent link
Thu, 23 Feb 2012
PyWeek #14 date change

The next PyWeek will be in May from the 6th to 13th. Not April.

category: Python | permanent link
Tue, 21 Feb 2012
Python Game Programming Challenge (PyWeek) #14 is coming!

The 14th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 6th to the 13th of May.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

If you're in the US and can make it I'm co-presenting a 3 hour pygame tutorial at PyCon in March.

category: Python | permanent link
Mon, 30 Jan 2012
PyPI is an OpenID provider

PyPI is now an OpenID provider.

To use this OpenID provider, enter pypi.python.org into any form that expects an OpenID*. Should the service not support OpenID 2, you will have to enter pypi.python.org/id/username instead (using your PyPI username.) Log into PyPI and visit your details page if you'd like to cut-n-paste the URL.

We follow the emerging approach that you have to sign into PyPI before signing into the actual services. This is intended to prevent phishing, as otherwise the relying party may fake PyPI's login page and collect your PyPI password (which they can still do if you fall for it.) It also avoids "nested" logins (i.e. where you need to log into PyPI with an OpenID while trying to login elsewhere with the PyPI id.)

If you find any problems with this service, please report them to the PyPI bug tracker.

*: of course for sites that extend PyPI this can be simplified to a simple button saying "link to my PyPI account".

category: Python | permanent link
Fri, 06 Jan 2012
Tips for Testing Twisted

This post contains some tips for testing Twisted applications. Note that I'm not using trial, the official Twisted test runner, for a variety of reasons.

Twisted Deferreds in Testing

You could use trial, but if you need to use another testing tool (nose, or behave) then you'll need to do a little more work.

To let your deferreds run (and make sure they do finish) wrap your test functions in the "@deferred" decorator from nose.twistedtools:

@deferred(1)

This will run the reactor for 1 second before declaring it hung.

If you want to see deferreds that are not behaving you will need to enable logging of Twisted events if they're not already enabled:

defer.setDebugging(1)
observer = log.PythonLoggingObserver()
observer.start()

and then patch nose.twistedtools as follows:

--- original/nose/twistedtools.py
+++ installed/nose/twistedtools.py
@@ -39,12 +39,19 @@

 _twisted_thread = None

+_deferreds = []
+
 def threaded_reactor():
     """
     Start the Twisted reactor in a separate thread, if not already done.
     Returns the reactor.
     The thread will automatically be destroyed when all the tests are done.
     """
+    from twisted.internet import defer
+    def __init__(self):
+        _deferreds.append(self)
+    defer.DebugInfo.__init__ = __init__
+
     global _twisted_thread
     try:
         from twisted.internet import reactor
@@ -135,6 +142,8 @@
             def errback(failure):
                 # Retrieve and save full exception info
                 try:
+                    if issubclass(failure.type, StandardError):
+                        print failure
                     failure.raiseException()
                 except:
                     q.put(sys.exc_info())
@@ -156,6 +165,9 @@
             try:
                 error = q.get(timeout=timeout)
             except Empty:
+                for d in _deferreds:
+                    if hasattr(d, 'creator') and not hasattr(d, 'invoker'):
+                        print d._getDebugTracebacks()
                 raise TimeExpired("timeout expired before end of test (%f s.)"
                                   % timeout)
             # Re-raise all exceptions

(I tried a couple of times to figure out how trial does the above but failed miserably)

This will result in a display of all the deferreds still pending when the deferred decorator timeout fires. Note that the deferred decorator introduces a deferred that will not have terminated.

One outstanding problem I've not solved is that inlineCallbacks aren't displayed usefully in the failure debug output - you get told that it is some inline callback that is still hanging, but not where it was created.

Twisted Web and Selenium

Testing Twisted web using Selenium is made more complicated by the need to allow the Twisted reactor to run at the same time as the selenium webdriver is trying to poke at it.

In other applications I've tested like this I just run up the server being tested in a separate thread but I can't do that in the case of Twisted.

To make this work any selenium operation that will cause a Twisted reactor event must be deferred to a thread, hence the liberal sprinkling of deferToThread throughout this code. Some sections do not touch the reactor (for example, filling out a form once the form page has been loaded doesn't), so it can be straight selenium calls.

Note also that the pages are generated in a server dynamically using deferreds so I use a "done" tag marker at the end to indicate to the tests that the page has been fully generated. Selenium's timeout on element access allows me to cause the test to fail if the done marker does not appear.

For example this step function directly from a behave feature implementation:

@then('the message should be sent')
@deferred(1)
@inlineCallbacks
def step(context):
    # wait until we're all done
    context.browser.implicitly_wait(1)
    yield threads.deferToThread(lambda: get_element(context.browser, id="done"))

    # there should be a result container
    results = yield threads.deferToThread(lambda: find_elements(context.browser, id="result"))
    result = results[-1]
    status = yield threads.deferToThread(lambda: get_element(result, id="status"))

    # the status could be a number of values depending on synchronisiation of
    # the tests / runner
    assert status.text in 'created new accepted done'.split()

    message = context.db.messages.values()[0]
    if message.status != message.STATUS_DONE:
        yield esme._wait_until_done(context)

More tips as I happen to think of them or discover them :-)

category: Python | permanent link
Thu, 22 Dec 2011
New Year Python Meme

Thanks, Tarek, for this fun idea.

1. What's the coolest Python application, framework or library you have discovered in 2011?
I was pretty happy I discovered the awesomeness of bottle when researching my web micro-framework battle (video.)

I've started using Python 2.7 and 3.2 which is pretty cool (having been stuck in 2.3, gasp!)

I've been working with Twisted again after a couple of years' break and have discovered txpostgres and Twisted's own inlineCallbacks. Both are pretty cool. inlineCallbacks make Twisted programming quite bearable to think about :-)

2. What new programming technique did you learn in 2011?
I've been honing my testing skills and learned about Behaviour Driven Development. I've also learned to use both Mercurial and Git (and the latter still drives me insane sometimes) and their related websites bitbucket and github.

I also started using virtualenv and Fabric way more this year.

3. What's the name of the open source project you contributed the most in 2011? What did you do?
This year I created parse, overload and ooch and contributed to behave. I also helped run PyCon Australia.

4. What was the Python blog or website you read the most in 2011?
Planet Python and the New Packages feed from PyPI.

5. What are the three top things you want to learn in 2012?
I can never predict what I'll end up learning. Or what software I'll be writing.

I know I'll be learning a lot more about Mercuruial (and Git, I suppose) and the vagaries of modern software deployment.

I plan on working more on my test- and behaviour-driven development practices.

I also want to learn how to bake more things thanks to the Great British Bake Off :-)

6. What are the top software, app or lib you wish someone would write in 2012?
I'm hoping that someone will do something cool with the new PyPI OpenID provider ;-)

Want to do your own list? here's how:

  1. copy-paste the questions and answer to them in your blog
  2. tweet it with the #2012pythonmeme hashtag
category: Python | permanent link
Tue, 23 Aug 2011
Cheese Shop (PyPI) sprinting at PyCon AU

We've had a fun couple of days sprinting on the Cheese Shop at PyCon AU where a number of contributors have fixed bugs and improved or added features (though always with the goal of keeping the service simple of course.)

In particular:

  1. Andy Todd helped clean up some aspects of the underlying database and fix up some of the sql.
  2. Capel Brunker added some more XML-RPC functionality, performed some tracker triage and also addressed some bugs and security issues.
  3. Kaleb Ufton, in his first contribution to Open Source development, added a bug tracker URL field to packages (which persists across releases and you must enter by editing through the website.) He also helped me sort out some twisted Apache configuration issues.
  4. I finally got around to writing the "newest packages" RSS feed.

There's another secret project we kicked off that will hopefully appear in the next couple of days, and some additional work that will hopefully come to fruition within a week or so. Stay tuned :-)

Thanks to everyone who contributed!

category: Python | permanent link
Wed, 27 Jul 2011
Using Solaris "priv" with Fabric

I needed support for "priv" instead of Fabric's built-in "sudo" support. I went through a number of (sometimes quite horrific) iterations before I settled on this relatively simple solution:

import contextlib

@contextlib.contextmanager
def priv(user):
    '''Context manager to cause all run()'ed operations to be
    priv('user')'ed.
    
    Replaces env.shell with the priv command for the duration of the
    context.
    '''
    save_shell = env.shell
    env.shell = 'priv su - %s -c' % user
    yield
    env.shell = save_shell

This is then used in a fabfile like so:

    with priv('remote_user'):
        run('do some remote command as remote_user')
        run('another remote command as remote_user')
category: Python | permanent link
Fri, 22 Jul 2011
PyWeek 13 (September 2011) is coming!

The 13th Python Game Programming Challenge (PyWeek) is coming. It'll run from the 11th to the 18th of September.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will hopefully increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

If you've never written a game before and would like to try things out then perhaps you could try either:

  1. The tutorial I presented at LCA 2010, Introduction to Game Programming, or
  2. The book Invent Your Own Computer Games With Python
category: News | permanent link
Fri, 29 Apr 2011
PyCon Australia 2011 - Registration Open and CFP Almost Over

A reminder that the Call for Proposals for PyCon Australia 2011 will be closing soon. We've had some great proposals so far, but there is still time left and program to fill.

PyCon Australia is Australia's only conference dedicated exclusively to the Python programming language, and will be held at the Sydney Masonic Center over the weekend of August 20 and 21. See below for more information and updates on:

  1. Call For Proposals
  2. Registration is Open
  3. More Sponsors Announced

Please share this message on to those you feel may be interested.

Call For Proposals

The deadline for proposal submission is the 2nd of May. That's only a few days away!

We are looking for proposals for talks on all aspects of Python programming from novice to advanced levels; applications and frameworks, or how you have been involved in introducing Python into your organisation. We're especially interested in short presentations that will teach conference-goers something new and useful. Can you show attendees how to use a module? Explore a Python language feature? Package an application?

We welcome first-time speakers; we are a community conference and we are eager to hear about your experience. If you have friends or colleagues who have something valuable to contribute, twist their arms to tell us about it! Please also forward this Call for Proposals to anyone that you feel may be interested.

The earlier you submit your proposal, the more time we will have to review and give you feedback before the program is finalised.

Speakers receive free registration for the conference, including a seat at the conference dinner. Don't miss out, submit your proposal today!

Registration is Open

We offer three levels of registration for PyCon Australia 2011:

Corporate - $440
If your company is paying for you to attend PyCon, please register at the corporate rate. You'll be helping to keep the conference affordable for all, especially for students and those needing financial aid. Government employees should also register at the corporate rate.
Full (Early Bird) - $165
This is the registration rate for regular attendees. We are offering a limited Early Bird rate for the first 50 registrations until the end of May. Once the Early Bird period ends, or when all Early Bird slots are filled, registration will increase to $198. Full registration includes one seat at the conference dinner on Saturday night.
Student - $44
For students able to present a valid student card we're offering this reduced rate. Student registrations do not include a seat at the conference dinner.

Additional seats at the conference dinner may be purchased for $77 each.

All prices include GST.

Information about the registration process is on the PyCon Australia website.

More Sponsors Announced

We are delighted to announce that ComOps has joined as a Gold Sponsor. Thank you to the following companies for their continuing support of Python and for helping to make PyCon Australia 2011 a reality:

Gold: Google
Gold: Microsoft
Gold: ComOps

Silver: Anchor
Silver: Enthought
Silver: Python Software Foundation

Thanks also to Linux Australia, who provide the overarching legal and organisational structure for PyCon Australia.

category: Python | permanent link
Tue, 26 Apr 2011
Review: Lennart Regebro's "Porting to Python 3"

Lennart Regebro has provided me with a free copy of his self-published book "Porting to Python 3" so that I may review it. He has considerable experience in porting code from Python 2 to Python 3, and it shows in much of his advice and examples. Regebro has assembled an excellent cast of helpers: the technical reviewer Martin von L÷wis also has much expertise with Python 3 (he implemented the first port of the popular Django web framework). Brett Canon's introduction also provides a good starting point for someone new to the story of Python 3.

The book's structure is well thought-out. The first chapter immediately invites the reader to make sure they really need to port their code, and what considerations might be taken into account when deciding to do so. The book even has advice, backed up with useful information, for those who are currently unable to port their code but may start preparing for doing so in the future.

The second chapter discusses strategies for moving to Python 3 (view it online). These are presented clearly with one section per strategy. Within each section there are clear references to the other, more detailed chapters of the book which may be used to implement each strategy.

Once you've decided which strategy to apply you can focus on the pertinent remaining chapters. These follow a pretty logical progression:

  • firstly preparing the way by making your code more modern,
  • introducing the 2to3 tool for automatically converting code to Python 3 and the various options for how to use it,
  • talking through some common migration problems and some simple solutions to them,
  • presenting some modern Python (2.6+) idioms that your code may use in Python 2 and 3, and
  • supporting Python 2 and 3 without using the 2to3 tool.

The final two chapters are for more limited audiences and cover migrating C extensions and writing custom 2to3 tool fixers. Even though these are topics which most readers won't need to worry about they are still covered in good detail with minimum fuss. It is in this final chapter that Regebro's experience with porting gives weight to the advice and examples he presents.

Finally there's a couple of pretty comprehensive appendices covering over the main language incompatibilities and library changes which makes it a good reference.

As previously mentioned, the book is very well-organised giving the reader an easy path through the material depending on their situation. Throughout the book there are many pertinent and clear references to software written by others. The book suffers from some typesetting and grammatical errors, though these are minor and none could cause any confusion regarding the content.

CONCLUSION
There's a lot of information out there for porting from Python 2 to 3, but Regebro has produced a concise, well-organised and complete reference for doing so. It's not skimped on any detail though; the book covers all areas related to moving to Python 3. It's not just about porting code; it's also a handy book for any programmer who's grown up with Python 2 (or 1!) and is looking to move to Python 3.

category: Python | permanent link
Mon, 07 Mar 2011
PyWeek 12 (April 2011) is registration is open!

The 12th Python Game Programming Challenge (PyWeek) is almost upon us. It'll run from the 3rd to the 10th of April. Registration for teams and individuals is now open on the website.

The PyWeek challenge:

  1. Invites entrants to write a game in one week from scratch either as an individual or in a team,
  2. Is intended to be challenging and fun,
  3. Will hopefully increase the public body of game tools, code and expertise,
  4. Will let a lot of people actually finish a game, and
  5. May inspire new projects (with ready made teams!)

If you've never written a game before and would like to try things out then perhaps you could try either:

  1. The tutorial I presented at LCA 2010, Introduction to Game Programming, or
  2. The book Invent Your Own Computer Games With Python
category: Python | permanent link