That is to say: do one thing, do it well.

Decorators (not to be confused with the decorator pattern) can add behaviors or side-effects to a method, and this can be dangerous. It seems harmless, because by adding a decorator, you’re likely taking boilerplate code and shuffling it elsewhere. However, they often encourage badness because of how easy it is to add these behaviors.

On testing decorated methods:

Adding an @decorator to a python object unarguably makes the undecorated code difficult to test in isolation. Decorators are applied at compile-time, and cannot be mocked or made to not-execute without some pain.

There are certainly a few common tricks that can help test a decorated method with minimal side-effects, but they require changes to the decorator itself. There’s just plain and simple no way to un-decorate a method for isolated testing.

Let’s look at a few common examples:

@expose: register a URL for a view function in a web framework

class BlogPostController(object):
    @expose("/blog/{post_id}")
    def index(self, request, post_id=None):
        """ show a blog post """
        post = adapter.get_post(post_id)
        return render("show_entry.html", dict(post=post))

Without the `@expose`, your `show_entry` knows how to get a given post and render it in the proper template. With the decorator, it also now knows which URL corresponds to that. You now have multiple reasons for this block of code to change, including pointing to a different URL or using a different template. It’s preferred to have a separate module for managing which urls point to which views.

Harm factor: low. Ick factor: medium – high.

@cache: try to get the result from memcache, otherwise, execute the function and stick it in cache for next time

class PostAdapter(DataAdapter):
    @cache
    def get_post(self, post_id)
        """ grab a blog post from the database """
        return self.query(Post).filter(id=post_id).one() # SQLAlchemy folks need to talk to Mr Demeter...

OK this seems cool right? You only hit the database when you have to, otherwise we get it even quicker by looking it up in the cache.

What happens if you want to disable caching? A separate cache abstraction layer would reduce volatility in your data adapter.

And what does the unit test look like?

class TestGettingAPost(object):
    def setup(self):
        self.query = Mock() #don't hit the production database!
        self.post_adapter = PostAdapter(query=self.query)
    def test_getting_a_post(self):
        assert self.post_adapter.get_post(123)

Damn, there’s no way to mock out the @cache decorator so it doesn’t run. Try to, I dare you. You’re likely going to actually get post 123 from your production memcache. Crappy. The only thing you can do is make the @cache grab the cache implementation from the PostAdapter instance (and mock that out in you test), or find some other sneaky way of disabling caching for test runs. But the @cache decorator isn’t all self-contained and fun anymore.

Harm factor: medium – high.

@validate: Make sure the request matches the specified schema, otherwise hand-off to error handler

class EditPostController(object):
    def _save_error(self, request, errors):
        """ @validate decorator kicked flow here, redisplay edit page with errors """
        ...
    @validate(schema, error_handler=_save_error)
    def save(self, request)
        post = self.schema.to_python(request.params)
        self.post_adapter.save_post(post)
        return redirect("/blog/{0}".format(post_id))

Without this @validate, there would be a lot of boilerplate code inside the `save` method. With it, any unit test for the save method will be likely linked to your schema. You’d have to make an actual valid request in order to test this method. That’s outside of any tests for your schema directly. That means double-coverage but 2 tests to update when requirements shift.

Harm factor: low-medium

Conclusions (or tl;dr):

As easy as it is to become infatuated with Python decorators, they definitely encourage you to violate the SRP. This can create a myriad of problems:

• Difficultly in isolating system under test
• Added complexity to enable testing
• Redundant redundant unit tests
• Making a code module more volatile than it ought to

They still have some valid use cases and can lead to cleaner code, however, as Master Yoda once said, “when you look at the dark side, careful you must be…”

I’ve cleanup up that sample code and tucked it into a script. You can pass a list of files to the script, and it will apply a sepia effect to each, making sure to backup the originals.

#!/usr/bin/python
"""
Apply sepia filter in batch to images
 
Usage:
    python batch_sepia.py [--no-backup] file1 [file2] ...
"""
 
import Image as PIL_Image
import shutil, os
from optparse import OptionParser
 
 
def open_image(filename):
    """ grab a PIL image from the given location
    """
    image = PIL_Image.open(filename)
    image.load()
    return image
 
def save_image(image, filename, quality=95):
    """ save the PIL image to disk
    """
    image.save(filename, "JPEG", quality=quality)
 
def make_linear_ramp(white):
    """ generate a palette in a format acceptable for `putpalette`, which
        expects [r,g,b,r,g,b,...]
    """
    ramp = []
    r, g, b = white
    for i in range(255):
        ramp.extend((r*i/255, g*i/255, b*i/255))
    return ramp
 
def apply_sepia_filter(image):
    """ Apply a sepia-tone filter to the given PIL Image
        Based on code at: http://effbot.org/zone/pil-sepia.htm
    """
    # make sepia ramp (tweak color as necessary)
    sepia = make_linear_ramp((255, 240, 192))
 
    # convert to grayscale
    orig_mode = image.mode
    if orig_mode != "L":
        image = image.convert("L")
 
    # optional: apply contrast enhancement here, e.g.
    #image = ImageOps.autocontrast(image)
 
    # apply sepia palette
    image.putpalette(sepia)
 
    # convert back to its original mode
    if orig_mode != "L":
        image = image.convert(orig_mode)
 
    return image
 
def convert_image(filename, make_backup):
    """ convert an image at the given path to sepia tone.
        @param filename
        @param make_backup - if True, will copy original file to file.bak
    """
    if not os.path.exists(filename):
        print 'Skipping %s' % filename
        return
    print 'Processing %s...' % filename
    if make_backup:
        shutil.copyfile(filename, '%s.bak' % filename)
    save_image(apply_sepia_filter(open_image(filename)), filename)
    print 'Done.'
 
def convert_images(files, make_backup=True):
    """ convert the list of filenames to sepia tone.
        @param filename
        @param make_backup - if True, will copy original file to file.bak
    """
    map(lambda f: convert_image(f, make_backup), files)
 
 
if __name__ == '__main__':
    parser = OptionParser()
    parser.add_option("-x", "--no-backup", dest="no_backup", default=False,
            action="store_true")
    (options, files) = parser.parse_args()
    convert_images(files, make_backup=not options.no_backup)

You can execute this from the command line as:

$ python batch_sepia.py image1.jpg image2.jpg image3.jpg

If you want to recursively apply the filter to a bunch of images, you might consider mixing this with some find/xargs-fu:

$ find $HOME/pictures -name "*.jpg" | xargs python batch_sepia.py

Once you’ve got your photos in order, head over to photoworks.com to get them printed!</shameless-plug>

:inoremap # X<C-H>#

See http://vim.wikia.com/wiki/Restoring_indent_after_typing_hash

Here are some notes I jotted down while at the conference. It’s a bit overwhelming, but I wanted to throw this out there for now, then I’ll come back and reorganize it.

Tutorial: PyCon 401 – Some Advanced Topics, by Steve Holden
* String interpolation
* Iterators
* Generators
* Descriptors and name lookups
* Metaclasses

———————————-
String Interpolation
———————————-

Can use locals() trick

def fmt(date, x):
    amount = 3
    return "%(date)12s $(x)-25s %(amount)8.2f" % locals()

———————————-
Iteration
———————————-

Iterable object must have next() and __iter__()
* next returns next item, or raises StopIteration
* __iter__ returns an “iterable” object (e.g. self)

———————————-
Generators
———————————-

Any function with “yield” in it is automagically assumed to be a generator by
the compiler
* Never actually return anything
* Obey the iterator protocol, and thus have a next() method

def g(s):
  for c in s:
    yield c+'!'
g1 = g('hello')
assert [c for c in g1] == ['h!', 'e!', 'l!', 'l!', 'o!']

———————————-
Decorators and other cool shit
———————————-

functools.wrap
* when making a decorator, decorate your decorator with this to preserve
“dunder_name” __name__ and dunder-doc __doc__

functools.partial
* copy a function to a new place and provide different kwarg defaults

———————————-
Descriptors and Properties
———————————-

HACKATTACK: Property definition without namespace polluion
* see ~/src/sandbox/src/pycon/tests/test_property.py

Descriptor protocol just talks about how to look up stuff in a class
* not all that cool :-/

———————————-
Metaclasses
———————————-

factories
* produce and return functions
* produce and return classes

metaclass
* can do cool shit to a class without cluttering up the class definition
* keep your classes closer to application domain model and responsibilities

Example metaclass: Automagically Decorate Every Method in Class
* page 35 of notes defines a Tracer class factory with __new__, that wraps
every callable, non-dundered (__x__) function with a tracing decorator
* you add a __metaclass__ attribute to your class, then when it gets spun up,
the tracer factory decorates its shit
* see ~/src/sandbox/src/pycon/tests/test_metaclass.py

Notes from Friday

———————
Windmill
———————
in-browser, cross-browser functional testing framework
* started as fork of selenium
* great visual recorder
* dom inspector cross-browser
python api
continuous integration w/ hudson
reporting hooks
issues:
* no ssl support

http://www.getwindmill.com

LOOKS AWESOME!

———————
Profiling
———————
Optimization makes code harder to maintain
But necessary sometimes because Python is slow in real world
Premmature optimization is the root of all evil. Forget about small efficiencies. Focus on hot spots.
Improve your algorithms for the 3% of worst things
* unroll loop
* add caching
Use profiling to figure out which pieces are the 3%

track entry and exit times to functions using Timer

How to:

import cProfile
cProfile.run("python.swallow(gopher)")

? pstats.Stats(‘1.profile’, ‘2.profile’)
** not good for threaded systems… need seperate profiler per thread

KCacheGrind can help explore profile output (runs on wxpython)

Look for
* way too many calls
* large local time in single function (can be spot-optimized)
* large cumulative per call (possibly cache results)
* high waiting times (IO blocked ?)

———————
Testing AJAX Web Apps
———————
It’s hard, and Kumar has failed. This is just an overview of what has/hasn’t worked.

Example: pylons+dojo aintjustsoul.net
* ajax to fetch album thumbnails

Strategy:
– test data handlers (in python)
– test javascript
– lint it first for syntax errors (e.g. javascriptlint.com)
– rhino runner (perhaps with john resig’s env.js to simulate DOM)
– spidermonkey ?
– Selenium, jquery.simulate, YUI.test to simulate user events in browser
(now it gets trickier)
– isolate UI for testing
– DI your connection stuff to mock out data and use a fake server, at the
cost of a little more indirection
– automate UI tests
– selenium remote control automates browser interactions via python code
– gridify your tests
– “selenium grid” to speed up tests
– future: test swarm
– crowd sourced continuous integration
– seems like an awful security flaw

http://bit.ly/testing-ajax (mailing list on google groups)

JSBridge can help to launch firefox from terminal
* it fakes X-server so mozilla runs but doesn’t put anything to display
* you can still run your tests in an actual browser rather than rhino’s implementation
* mozilla ubiquity plugin tested this way

———————
Auomated QA Infrastructure
———————

Ingredients:
* VCS
– mainline branch.. merges trigger full QA battery
– devs use QA system on dev branches
* reporting
– if you don’t do this, nothing else matters
– “your QA system is ONLY as good as its reporting of results”
* tests (discovery, coverage metrics, static source analysis, performance profiling)
– discovery = nose
– coverage = figleaf
– static analysis = pylint
– profiling = ????

CI frameworks:
* svnchecker (svn post-commit hooks)
* bitten
* pybots
* buildbot ***** <-- best general purpose python-based open-source framework now

Buildbot Architecture
* master: coordinate, dispatch, report
* build slaves: runs tests as directed
- BuildFactory defines the build process
>>> factory.addStep(SVN(url))
>>> factory.addStep(Compile(command=['python', 'setup.py', 'build'])
>>> factory.addStep(nose.run()) #????
# they used Trial(testpath=’.') for Twisted
>>> factory.addStep(Figleaf(…
>>> factory.addStep(PyLint(…
* changes: encapsulates events that require testing
* schedulers: makes decisions about when/where to order tests
* pollers/triggers: signals the base change events
– built-in SVNPoller

———————
Twisted/MQ/RPC
———————
… (notes in moleskin) …

———————
Testing large, untested codebases
~ Titus
———————

Example: biological something or other
on the science background of code under test
* example: pygr – python graph database
– provides data abstration with pygr.Data
* problem in biology
– tons and tons of data, parsed lists, etc

on the code and authors
* da code
– ~8k of Py, ~2k of Pyrex
– almost all library and framework, no UI fluff
* the people
– written by 1 dude ==> lots of technical debt
– no tests, interface contracts, etc.
– code ownership is issue by new people that want to contribute.
don’t want to step on original author’s code
– idiosyncratic code since author was a smart dude

statement: code coverage is of limited utility bc it doesn’t measure branch coverage
* TITUS SAYS WRONG WRONG WRONG
* it’s not just bc it doesn’t measure branch coverage. there are other more
valid reasons why it might be flawed

grokking code through coverage
* howto:
– given a big chunk o python
– import it, write test to exercise some of it, check coverage to see what it done
– add additional statement to test
– repeat
* lets you pull apart highly complex/coupled code
*

problem: no good way keep track of tests in a large codebase
* some things may or may not be getting run by your runner
* new project “tattle” that keeps track of master list of tests
– when running, checks the master list and if something is missing or new, it warns you
– not decided how to keep track of what’s changed
– yaml ?
– tag the source ?
– check the diff ? (requires “saving” coverage state, then comparing on next run)

“for those that use nose, each additional dot is like another hit of smack”

code review:
* pep8
* pep257 – docstring format!!!

“each time you’ve pissed on another line of code, it’s like it’s yours now”

reformatting vs “search/replace” vs real functional changes
* what you gonna break?

coverage is an invaluable lever for prying into other people’s code bases

what’s next?
* trace code coverage between commits (and which lines are no longer covered)
* branch coverage (titus on GSoC project: “i have no idea how to do this, that’s what students are for”)
* simplify CI. buildbot too complex

Notes from Saturday

——————————
!!!! LIGHTNING TALKS !!!!
——————————

== thestatusisnotquo.com ==
@mpirnat’s socially redeeming python mini-community of hackers

== sqlpython ==
write sql in unix command shell
have access to all other cool unix stuff (grep, sed, cat, etc)

== webob ==
presentation was cool… just played through the doctests in a
fake interactive interpreter. flawless demo FTW!

== DVCS ==
mercurial (hg) is super easy to use
statement: hg on bitbucket is better than svn on google code
successful DVCS don’t really use the D
but django 1.1 delayed by a month because of DVCS crap
what’s the benefit:
* better merging (but shouldn’t require DVCS)
* easier repository creation (as opposed to svnadmin)
svn isn’t quite dead yet (but it’s not getting better)

== pymite ==
python on microcontrollers.. runs on 8bit devices with 64kb ram
easily embed native code in docstrings

== web2py ==
MVC web framework. used for pycon registration.
online DB admin tool ala django
online python code editor
integrates with bug ticketing system and links back to original stack trace
templating system looks like advanced pyfusion

——————————
Guido’s keynote
——————————
py3.1/2.7 coming
no decision on dvcs
tool to backport from py3 to py2 coming. then we can write code in 3 and give it to the 2x world also.

——————————
State of Django
——————————
… (see moleskin) …

** statement: everyone should be using sphinx for technical docs **

——————————
Lack of Design Patterns
——————————
“python isn’t java without the compile”

define lack of patterns:
* comp.lang.python 100k messages
* visitor/flyweight/strategy/state pattern was most often referenced
(but still less than “sausage”)

why no talk of patterns?
* patterns are built-in
– first class functions
– metaprogramming (can manipulate functions/definitions)
– iterators
– closures (even if not a “true” closure in python, but close enough)

examples:
* factory
>>> {‘thing1′:Thing1, ‘thing2′:Thing2}[which_one]()

conclusions:
* look to features of langauge before patterns
* reduce patterns –> shortercode
* if need a pattern, add a language feature (e.g. concurrency stuff)

presentor wants “channel pattern” (as a model for processes to replace threads)
* look at PyCSP (or stackless?) for concurrency

——————————
Class Decorators
——————————

just like function decorators
* definition: a callable that accepts at least one argument and returns something

available as of py2.6

def identity(ob):
    return ob 
 
@identiy
class C():
    pass

in 2.5, you can do same type of thing, but you can’t use the @decorator syntax

class decorators are not inheritable!!!
but you can stack them on top of a single class
(as opposed to metaclasses, which are inherited but not stackable)

default method decorator vs. mixins (eg partial class)
* ???

popular patterns (besides just replacing what was done with metaclasses)
* register
* augment (mixin replacement)
– example: take the __lt__ method and automagically make an __eq__ and
__gt__ for class
* fixup (monkeypatching decorator)
– example: replace all dunder-attribute with single underscores, so can get
access to the stuff that was privatized. can monkeypatch a decorator into
a thirdparty library too! (named @stop_doing_java)
* verify (assert stuff about the class)

registration pattern
* presentors best use case
* add stuff to a list that’s kept elsewhere

@cron.schedule(cron.NIGHTLY)
class SalesReport(Report):
    def run(self):
        #blah

——————————
ORM Panel
——————————
…

——————————
Scripting in Python (super beginner)
——————————

Compromise Layout pattern
Project
/bin
script.py
/scriptlib
__init__.py
scriptimpl.py
setup.py
Conditional main pattern
>>> if __name__ == ‘main’:
… sys.exit(main(sys.argv))
3 layers of IO pattern:
* should you deal with filenames, file objects, or generators ? YES!
* main should take a filename. invokes something that takes a file instanece.
below that use generator layer for efficiency.
Use optparse pattern
wtf. not exactly what i was expecting from this talk. time to go drink.

——————————
Drop ACID and think about data
——————————
ACID: a promise ring your DBMS wears
* atomicity – all or nothing
* consistency – no explosions during crashes/etc
* isolation – no fighting from writers, can pretend transactions are serial
* durability – no lying to you.. assume things worked

scalability and reliability: requires a bunch of stuff from rdbms to get there

CAP theorem says pick 2:
* consistency, availability, partition tolerance

BASE:
* basically available, soft state (doesn’t have to be consistent immediately), eventually consistent
* Google BigTable
– append-only data makes it faster to insert
– fantastic data compression due to hybrid row/column storage
* Amazon Dynamo key/value store
– consistent hashing to distribute data to appropriate node
* Facebook Cassandra
– OPEN SOURCE!! incrementally scalable
– no compression, not quite polished

distributed db’s are the new hotness, but none are awesome yet.
* we don’t need another half-baked dynamo clone

memcached: key/value store used as cache
* RAM only. throws data away. lightning fast. everyone’s doing it.
* invalidation is hard and failure-prone
– be careful with mutable data. consistency will be lost over DB solution.

Tokyo Cabinet/Tyrant: super enhanced berkeleydb key/value store
* disk persistent, but still very performant
* actively developed
* replication similar to mysql
!! possible solution for photoworks session

document databases
* schema free, easy to use,
* easier to migrate. some documents will just have different properties.
– but application still needs to be able to deal with this
!! we kind of got this with photoworks project meta-data
* couchdb – document db poster child
* mongodb – way faster than couchdb. json/bson (binary json-ish)

column databases:
* sequential reads are awesome
* compresses much better than rows
* monetdb – DON’T USE THIS

——————————
Concurrency and Distributed systems
——————————
concurrent > parallel > distributed

goals:
* decrease latency, increase throughput

issue?
* global interpreter lock (GIL). one thread at a time.
– ugh. slow. boo.
– perhaps fixed in unladen-swallow ?
* but you can build concurrent regardless of GIL

multiprocessing in 2.6+
* processes and IPC via pipes to allow parallelism
* similar API to thread/queue

more on GIL workarounds…
* Jython
– allows free-threading via java.util.concurrent, at the cost of losing C extensions
* IronPy
– also unrestricted threading, and some C extensions via ironclad
* PyPy
– complete rethink of interpreter
– not production ready

Twisted
* asynchronous, event driven multitasking
* supports thread usage, but may not always be threadsafe

Kamaelia
* easy to understand. very easy to get up and running. IPCs, Threads, etc abstracted away
* compontents talk vis mailbox. cooperative multitasking via generators
* lots of fantastic examples
!!! try this out

problem w/ frameworks
* twisted/kamaelia radically different
* you end up adapting the methodology in app, which can be dangerous

messaging/rpc flooded with libraries
* Pyro for RPC should be checked out
* memcached for shared memory
* Stomp, ApacheActiveMQ for message queues
– space flooded b/c heavy use by web apps for asynch processesing right now

Notes from Sunday

—————————————–
Paver
—————————————–
build process helpers
* cmd line
* config
* work w/ files
* svn, sphinx, virtualenv, etc.
* distutils, setuptools

without paver, you end up writing a seperate script for everything you want to do

$ easy_install Paver
$ paver paverdocs #displays Paver documentation

build files…
$ vi pavement.py

options(
    setup = setup_meta,  #via setup.py
    virtualenv=Bunch(
        packages_to_install=["", "", ""],
...

tasks are the heart of paver

@task
def clean():
   """ cleans up paver dict,  removes virtualenv tree and build dir """

other fun decorators too
@cmdopts – integration w/ optparser to set up cmd line options for your tasks

paver.easy
* logging, unix command running, dry-run stuff against command line

paver.path
* jason orendorff’s path.py module built in

paver.doctools
* sphinx (generate html docs)
* ned batch’s Cog support. “section off” pieces of code and pull into your docs dynamically.

paver.ssh, paver.svn, paver.virtual (virtualenv)

deployment of code delegated to pytoss (?)

getting started: http://bit.ly/starting_paver

—————————————–
Functional Testing Panel
—————————————–
windmill
* python proxy – avoids same domain security in browser
* python browser launching
* python CI reporting
* javascript simulate user interactions – tries to get it right (bubbling, mouseclick events)
* javascript waits for dynamic content
* javascript assertions

selenium:
* “record/playback is the root of all evil”. none of creators wanted to do it
* next version will have more native C hooks. javascript event faking could only get so far.
* really only good at testing in firefox, rest of browsers (including chrome) suck

twill:
* technically just a small DSL on top of mechanize/clientform/beautifulsoup
* forms, cookies, redirects, link checking
* but no javascript support. ever. not even a little.

webtest (ian bicking)
* runs fake request/responses on WSGI frameworks
* written to test paste
* doesn’t actually run http

from myapp import wsgi_app
from webtest import TestApp
app=TestApp(wsgi_app)
app.go('/')
print app.resp

nose
* nose.cfg for default cmd line opts
* –with-django plugin to help set up env

py.test 1.0
* distributed testing. central reporting.
– run your tests on all different envs (clouds, o/s’s, py versions, etc)
* plugins similar to nosetests. but not documented all that well
* runs unittests, doctest, restdoc, pocoo (sends failures to pastebin)
* tetamap.wordpress.com, pytest.org

vision: make automated deployment merge w/ actual deployment, and have success
be mandated for deploy.

—————————————–
Lightning talks
—————————————–

== argparse ==
all features of optparse, but better docstrings and never touch sys.argv
sub command parsers (e.g. svn add) easier than optparse
mutually exclusive arguments

http://code.google.com/p/argparse

== python 3.1 ==
some performance improvements. IO now almost as fast as 2x
unittest improvements. assertRaises trick
OrderedDict finally arrives

from __future__ import bad_syntax_ideas
* braces, no colons, no indentation, implied ruby blocks, etc.
* pep 3117

== Poor Man’s Continuous Integration ==
try to do it without buildbot using your existing tools
* capastrano, maestro, git/svn
using “fabric” instead of “git push”
$ fab push -d staging_server deploy
* wraps up the test run and git push steps

== cluemapper – software project manager ==
example: lots of software projects
* each w/ issue tracker, svn
* want to share users across projects
built on top of trac. “trac for teams”
* basic reporting and time-tracking
* trac plugins to extend core
* wsgi + dojo ajax
components
* irc bot
* theming
* pypi egg server
* bzr integration
* pastebin
* timetracking, reporting
nice source control user manager for each repository
roadmap
* agile/scrum/xp stories and whiteboard
– drag/drop between stories and iterations
!!!!!!!!!!!!!!!!!!!!!!! yayz !!!!!!!!!!!!!!!!!!
cluemapper.org

== ReleaseBot ==
release mgr process watches buildbot
push button to tag a version for deploy
* workflow for notifying QA that this is what needs tested
buildbot notices tag and preps for release
not yet developed, just idea phase

== pyjamas ==
python to javascript compiler, with built-in widget set
JSONProxy seems fun!
rest is pure evil.

== \ is evil ==
don’t split lines with a backslash
use parentheses instead

my_str = ("asdlfkjasdf   asdlfkjdsf"
        "  asdflkj   asldkjf "
        " alskjdf asdlkjj ")

insetad of

evil_nasty_str = "asdlfkjasdf   asdlfkjdsf" \
        "  asdflkj   asldkjf " \
        " alskjdf asdlkjj "

Mini-conclusion
Things for follow-up:
* windmill. first round of experimenting at openspace didn’t prove to be as awesome as i had hoped. but i’m willing to give it another try. scripting these in python feels like an afterthought, and it wouldn’t be easy to get integrated with BuildBot.
* ClueMapper. looking forward to the scrum/xp tools built into Trac.
* webtest instead of twill for scripting end-to-end tests
* py.test for distributed testing
* tokyo cabinet for key/value store with persistence
* web2py web framework. the data schema admin manager doesn’t feel quite as polished as django, and i don’t like the online editor. but their MVC ideas and templating implementation might be useful.
* mercurial or bzr. i need more practice with DVCS. i committed all my code throughout the weekend to a local mercurial instance, but never pushed up to a central repository.

That’s all she wrote for now. Looking forward to future PyCon’s and to trying out some of these awesome tools.

I was impressed, but needed to check the accuracy of such a statement. So I dialed up a python session.

import re
awesomedrome = """Dammit I'm mad.
Evil is a deed as I live.
God, am I reviled? I rise, my bed on a sun, I melt.
To be not one man emanating is sad. I piss.
Alas, it is so late. Who stops to help?
Man, it is hot. I'm in it. I tell.
I am not a devil. I level "Mad Dog".
Ah, say burning is, as a deified gulp,
In my halo of a mired rum tin.
I erase many men. Oh, to be man, a sin.
Is evil in a clam? In a trap?
No. It is open. On it I was stuck.
Rats peed on hope. Elsewhere dips a web.
Be still if I fill its ebb.
Ew, a spider... eh?
We sleep. Oh no!
Deep, stark cuts saw it in one position.
Part animal, can I live? Sin is a name.
Both, one... my names are in it.
Murder? I'm a fool.
A hymn I plug, deified as a sign in ruby ash,
A Goddam level I lived at.
On mail let it in. I'm it.
Oh, sit in ample hot spots. Oh wet!
A loss it is alas (sip). I'd assign it a name.
Name not one bottle minus an ode by me:
"Sir, I deliver. I'm a dog"
Evil is a deed as I live.
Dammit I'm mad."""
 
# strip out all punctuation
just_the_chars = re.compile('\W').sub('', awesomedrome).lower()
 
def test_palindrome():
    # chars[::-1] gives the reverse of the resulting string
    assert just_the_chars == just_the_chars[::-1]

I stripped out any non-word characters (punctuation, space, etc) with the regular expression, lower-cased it all, and compared it front to back.

$ python is_demtri_a_god.py
not exactly

As you can see, it’s not palindromic.

So close.

$ nosetests test_demetri.py
.
----------------------------------------------------------------------
Ran 1 test in 0.001s
 
OK

Update 4/22:
Not sure why the original script used to doubted the claim to palindromity, but I’ve fixed it now. Demetri really does rock our socks off!

Although I’ve been enamored with Python for the last 13 or so months, I’ve often dreamt longingly for days of Java glory. Things from the past seem so desirable at times — IDEs with good code completion and integrated debugging, staticly-typed compiler hand-holding, mature application servers and deployment tools. There was never much black magic tying things together. You could easily trace where and why something was breaking, and throw a fix up quickly.

It only takes a few minutes of reading through a large Java codebase to bring me back to reality. The horribly cluttered JSPs with code spewing into and around markup. MEGA-JARs. XML files and properties files galore. 65 lines of code to parse a querystring (not because there wasn’t a better way, but because many Java coders don’t value beauty by simplicity). Seriously.

I know there are pie-in-the-sky Java apps that give developers the warm-and-fuzzies inside, but most of what I’ve seen in practice gives me the chills. I guess part of what is so alluring is the notion that I could take an ugly Java app, apply what I’ve learned from Python, and behold the beauty of an amazing creation.

Alas, Java is like the bad ex that you think you want back. A few months after the breakup, you think back on all the good times, the marathon sex sessions, and forget entirely why it was you broke up. You want a second chance to make it work. In reality, the sex was never really that good, it was just comfortable. There were countless little things that annoyed you and caused you so many headaches, the same few things that you always fought about and never really resolved.

Just so that I don’t forget, here’s a list of 5 things I love about Python or hate about Java.

1. from future import antigravity; Seriously, just easy_install it, import it, and you have it. So much more is available at your fingertips in Python than was in Java.
2. The fact that Python was willing to break backwards compatibility with the advent Python 3K, which is a serious testament to the community. Java can’t do that because its user base is notoriously full of late adopters who are resistant to change and are generally less agile.
3. Collections. Yes there are all the collection types in the Java standard library, but they’re still so klunky to work with. Everything relies on hashtables and arrays, so why not make them as easy to use as Python?
4. XML fever. Why is it so buzzy still in Java? There is a better way.
5. The lack of an awesome, blow-me-away, clear-winner of a web framework in Java. I know Python doesn’t quite have that either, but Django is making great strides. And in a weird way, not having one is good for Python, in that it’s fostering some great new developments. Granted, I don’t know anything about Struts 2, but I have a feeling it’s the same-old same-old.

The moral of the story? Don’t forget how good you got it.

Sometimes, you’d like to test actual file handling in a unit test. If you do, you should provide a sample data set packaged with your code in a data directory within your test directory.

Your package would then resemble the following structure:

/package
    /tests
        /data
            data.txt
        __init__.py
        test_module.py
    __init__.py
    module.py

In your test, you’ll need to provide a relative path to the data file. To do this, use the special __file__ attribute of the package to provide a starting point. This will let you know where the module lives on disk regardless of where the package is installed.

import os
from package.tests import __file__ as test_directory
 
def _data_dir():
    return os.path.join(os.path.dirname(test_directory), 'data')
 
def test_sample_data():
    file_path = os.path.join(_data_dir(), 'data.txt')
    with open(file_path) as f:
        package.module.do_something(f)

There you have it!

  <div>I'm some
               fancy text that needs
               to be found</div>

I tried my very simple pattern match using the re.MULTILINE flag to no avail. The solution ended up being:

p = re.compile('some\s*fancy', re.DOTALL)
p.search(input_text)

re.MULTILINE is not the same as re.DOTALL. Use DOTALL if you want your dot to match everything, including new lines. See more at this handy dandy reference site.

Doh!

Option 1: encode=string-ify, decode=eval
The idea here is take a dict and turn it into a string. Then running eval would give you back the original dictionary. If the world were full of with people who danced in flower-filled meadows, this would be ok. The problem is that if you need to pass “s” through an HTTP request, it can be altered into something, like perhaps “os.rmdir(’/’)”

d={'var1':1, 'var2':2}
s="{'var1':1, 'var2':2}"
new_d = eval(s)

Result: VULNERABLE to arbitrary code execution.

Option 2: encode=string-ify, decode=slightly-safter eval
The eval method in python does allow you to tighten it up a bit. There are several articles detailing how to make it slightly safer, by only allowing a certain subset of methods to be available.

new_d = eval(s, {"__builtins__":None}, {})

Result: VULNERABLE to arbitrary code execution.

Option 3: encode=pickle, decode=unpickle
So what if instead of just eval’ing, we used the pickle module to serialize and then deserialize the dictionary? That feels safer at first glance. But what if the object to be deserialized was no longer a dictionary? Perhaps an attacker pickled their own evil object, whose constructor or maybe get() method was modified to do wreak havoc.
Result: VULNERABLE to arbitrary code execution.

Option 4: encode=jsonify, decode=un-jsonify
One technique that didn’t appear in the results was using simplejson to encode and decode. (Thanks Mike Pirnat for suggesting the now-obvious). The simplejson decoder is a much safer one, because it scans the input to make sure it really does look like a dictionary (with all keys and values being safe types) before turning it into a python dict.
User’s can still tamper with the keys and values, but it is no more susceptible than other types of parameter manipulation rampant on the web, so validation still need be there.
Result: much better

Option 5: encode=jsonify(jsonify+hash), decode=un-jsonify plus hash check
To an extra level of comfort, let’s try to make sure that the dictionary wasn’t tampered with. We can take a hash of what we just json-ified, pack it with the data, and json-ify that result. Then when we unpack it, we can double-check that the hash that came with the data still matches a new calculation of the hash of the data. Assuming we have a strong, protected key for our hash algorithm, if the user tried to tamper with the data, they wouldn’t be able to replicate the corresponding change in the hash value, allowing us to raise an exception.
Result: best yet

There ye have it, a safer way of turning a dictionary into a string suitable for passing around the website. Just make sure your hash function is strong and your key is protected.

You can download encode_dict.py to see the final result.