Mike's Blabberings

Just wrapping up a fantastic few days at PyCon Chicago 2009. I went with a group of about 10 guys from AGI, and we managed to have a little bit of fun in between some great mind-enriching talks.

Here are some notes I jotted down while at the conference. It’s a bit overwhelming, but I wanted to throw this out there for now, then I’ll come back and reorganize it.

Tutorial: PyCon 401 – Some Advanced Topics, by Steve Holden
* String interpolation
* Iterators
* Generators
* Descriptors and name lookups
* Metaclasses

———————————-
String Interpolation
———————————-

Can use locals() trick

def fmt(date, x):
    amount = 3
    return "%(date)12s $(x)-25s %(amount)8.2f" % locals()

———————————-
Iteration
———————————-

Iterable object must have next() and __iter__()
* next returns next item, or raises StopIteration
* __iter__ returns an “iterable” object (e.g. self)

———————————-
Generators
———————————-

Any function with “yield” in it is automagically assumed to be a generator by
the compiler
* Never actually return anything
* Obey the iterator protocol, and thus have a next() method

def g(s):
  for c in s:
    yield c+'!'
g1 = g('hello')
assert [c for c in g1] == ['h!', 'e!', 'l!', 'l!', 'o!']

———————————-
Decorators and other cool shit
———————————-

functools.wrap
* when making a decorator, decorate your decorator with this to preserve
“dunder_name” __name__ and dunder-doc __doc__

functools.partial
* copy a function to a new place and provide different kwarg defaults

———————————-
Descriptors and Properties
———————————-

HACKATTACK: Property definition without namespace polluion
* see ~/src/sandbox/src/pycon/tests/test_property.py

Descriptor protocol just talks about how to look up stuff in a class
* not all that cool :-/

———————————-
Metaclasses
———————————-

factories
* produce and return functions
* produce and return classes

metaclass
* can do cool shit to a class without cluttering up the class definition
* keep your classes closer to application domain model and responsibilities

Example metaclass: Automagically Decorate Every Method in Class
* page 35 of notes defines a Tracer class factory with __new__, that wraps
every callable, non-dundered (__x__) function with a tracing decorator
* you add a __metaclass__ attribute to your class, then when it gets spun up,
the tracer factory decorates its shit
* see ~/src/sandbox/src/pycon/tests/test_metaclass.py

Notes from Friday

———————
Windmill
———————
in-browser, cross-browser functional testing framework
* started as fork of selenium
* great visual recorder
* dom inspector cross-browser
python api
continuous integration w/ hudson
reporting hooks
issues:
* no ssl support

http://www.getwindmill.com

LOOKS AWESOME!

———————
Profiling
———————
Optimization makes code harder to maintain
But necessary sometimes because Python is slow in real world
Premmature optimization is the root of all evil. Forget about small efficiencies. Focus on hot spots.
Improve your algorithms for the 3% of worst things
* unroll loop
* add caching
Use profiling to figure out which pieces are the 3%

track entry and exit times to functions using Timer

How to:

import cProfile
cProfile.run("python.swallow(gopher)")

? pstats.Stats(‘1.profile’, ‘2.profile’)
** not good for threaded systems… need seperate profiler per thread

KCacheGrind can help explore profile output (runs on wxpython)

Look for
* way too many calls
* large local time in single function (can be spot-optimized)
* large cumulative per call (possibly cache results)
* high waiting times (IO blocked ?)

———————
Testing AJAX Web Apps
———————
It’s hard, and Kumar has failed. This is just an overview of what has/hasn’t worked.

Example: pylons+dojo aintjustsoul.net
* ajax to fetch album thumbnails

Strategy:
– test data handlers (in python)
– test javascript
– lint it first for syntax errors (e.g. javascriptlint.com)
– rhino runner (perhaps with john resig’s env.js to simulate DOM)
– spidermonkey ?
– Selenium, jquery.simulate, YUI.test to simulate user events in browser
(now it gets trickier)
– isolate UI for testing
– DI your connection stuff to mock out data and use a fake server, at the
cost of a little more indirection
– automate UI tests
– selenium remote control automates browser interactions via python code
– gridify your tests
– “selenium grid” to speed up tests
– future: test swarm
– crowd sourced continuous integration
– seems like an awful security flaw

http://bit.ly/testing-ajax (mailing list on google groups)

JSBridge can help to launch firefox from terminal
* it fakes X-server so mozilla runs but doesn’t put anything to display
* you can still run your tests in an actual browser rather than rhino’s implementation
* mozilla ubiquity plugin tested this way

———————
Auomated QA Infrastructure
———————

Ingredients:
* VCS
– mainline branch.. merges trigger full QA battery
– devs use QA system on dev branches
* reporting
– if you don’t do this, nothing else matters
– “your QA system is ONLY as good as its reporting of results”
* tests (discovery, coverage metrics, static source analysis, performance profiling)
– discovery = nose
– coverage = figleaf
– static analysis = pylint
– profiling = ????

CI frameworks:
* svnchecker (svn post-commit hooks)
* bitten
* pybots
* buildbot ***** <-- best general purpose python-based open-source framework now

Buildbot Architecture
* master: coordinate, dispatch, report
* build slaves: runs tests as directed
- BuildFactory defines the build process
>>> factory.addStep(SVN(url))
>>> factory.addStep(Compile(command=['python', 'setup.py', 'build'])
>>> factory.addStep(nose.run()) #????
# they used Trial(testpath=’.') for Twisted
>>> factory.addStep(Figleaf(…
>>> factory.addStep(PyLint(…
* changes: encapsulates events that require testing
* schedulers: makes decisions about when/where to order tests
* pollers/triggers: signals the base change events
– built-in SVNPoller

———————
Twisted/MQ/RPC
———————
… (notes in moleskin) …

———————
Testing large, untested codebases
~ Titus
———————

Example: biological something or other
on the science background of code under test
* example: pygr – python graph database
– provides data abstration with pygr.Data
* problem in biology
– tons and tons of data, parsed lists, etc

on the code and authors
* da code
– ~8k of Py, ~2k of Pyrex
– almost all library and framework, no UI fluff
* the people
– written by 1 dude ==> lots of technical debt
– no tests, interface contracts, etc.
– code ownership is issue by new people that want to contribute.
don’t want to step on original author’s code
– idiosyncratic code since author was a smart dude

statement: code coverage is of limited utility bc it doesn’t measure branch coverage
* TITUS SAYS WRONG WRONG WRONG
* it’s not just bc it doesn’t measure branch coverage. there are other more
valid reasons why it might be flawed

grokking code through coverage
* howto:
– given a big chunk o python
– import it, write test to exercise some of it, check coverage to see what it done
– add additional statement to test
– repeat
* lets you pull apart highly complex/coupled code
*

problem: no good way keep track of tests in a large codebase
* some things may or may not be getting run by your runner
* new project “tattle” that keeps track of master list of tests
– when running, checks the master list and if something is missing or new, it warns you
– not decided how to keep track of what’s changed
– yaml ?
– tag the source ?
– check the diff ? (requires “saving” coverage state, then comparing on next run)

“for those that use nose, each additional dot is like another hit of smack”

code review:
* pep8
* pep257 – docstring format!!!

“each time you’ve pissed on another line of code, it’s like it’s yours now”

reformatting vs “search/replace” vs real functional changes
* what you gonna break?

coverage is an invaluable lever for prying into other people’s code bases

what’s next?
* trace code coverage between commits (and which lines are no longer covered)
* branch coverage (titus on GSoC project: “i have no idea how to do this, that’s what students are for”)
* simplify CI. buildbot too complex

Notes from Saturday

——————————
!!!! LIGHTNING TALKS !!!!
——————————

== thestatusisnotquo.com ==
@mpirnat’s socially redeeming python mini-community of hackers

== sqlpython ==
write sql in unix command shell
have access to all other cool unix stuff (grep, sed, cat, etc)

== webob ==
presentation was cool… just played through the doctests in a
fake interactive interpreter. flawless demo FTW!

== DVCS ==
mercurial (hg) is super easy to use
statement: hg on bitbucket is better than svn on google code
successful DVCS don’t really use the D
but django 1.1 delayed by a month because of DVCS crap
what’s the benefit:
* better merging (but shouldn’t require DVCS)
* easier repository creation (as opposed to svnadmin)
svn isn’t quite dead yet (but it’s not getting better)

== pymite ==
python on microcontrollers.. runs on 8bit devices with 64kb ram
easily embed native code in docstrings

== web2py ==
MVC web framework. used for pycon registration.
online DB admin tool ala django
online python code editor
integrates with bug ticketing system and links back to original stack trace
templating system looks like advanced pyfusion

——————————
Guido’s keynote
——————————
py3.1/2.7 coming
no decision on dvcs
tool to backport from py3 to py2 coming. then we can write code in 3 and give it to the 2x world also.

——————————
State of Django
——————————
… (see moleskin) …

** statement: everyone should be using sphinx for technical docs **

——————————
Lack of Design Patterns
——————————
“python isn’t java without the compile”

define lack of patterns:
* comp.lang.python 100k messages
* visitor/flyweight/strategy/state pattern was most often referenced
(but still less than “sausage”)

why no talk of patterns?
* patterns are built-in
– first class functions
– metaprogramming (can manipulate functions/definitions)
– iterators
– closures (even if not a “true” closure in python, but close enough)

examples:
* factory
>>> {‘thing1′:Thing1, ‘thing2′:Thing2}[which_one]()

conclusions:
* look to features of langauge before patterns
* reduce patterns –> shortercode
* if need a pattern, add a language feature (e.g. concurrency stuff)

presentor wants “channel pattern” (as a model for processes to replace threads)
* look at PyCSP (or stackless?) for concurrency

——————————
Class Decorators
——————————

just like function decorators
* definition: a callable that accepts at least one argument and returns something

available as of py2.6

def identity(ob):
    return ob 
 
@identiy
class C():
    pass

in 2.5, you can do same type of thing, but you can’t use the @decorator syntax

class decorators are not inheritable!!!
but you can stack them on top of a single class
(as opposed to metaclasses, which are inherited but not stackable)

default method decorator vs. mixins (eg partial class)
* ???

popular patterns (besides just replacing what was done with metaclasses)
* register
* augment (mixin replacement)
– example: take the __lt__ method and automagically make an __eq__ and
__gt__ for class
* fixup (monkeypatching decorator)
– example: replace all dunder-attribute with single underscores, so can get
access to the stuff that was privatized. can monkeypatch a decorator into
a thirdparty library too! (named @stop_doing_java)
* verify (assert stuff about the class)

registration pattern
* presentors best use case
* add stuff to a list that’s kept elsewhere

@cron.schedule(cron.NIGHTLY)
class SalesReport(Report):
    def run(self):
        #blah

——————————
ORM Panel
——————————
…

——————————
Scripting in Python (super beginner)
——————————

Compromise Layout pattern
Project
/bin
script.py
/scriptlib
__init__.py
scriptimpl.py
setup.py
Conditional main pattern
>>> if __name__ == ‘main’:
… sys.exit(main(sys.argv))
3 layers of IO pattern:
* should you deal with filenames, file objects, or generators ? YES!
* main should take a filename. invokes something that takes a file instanece.
below that use generator layer for efficiency.
Use optparse pattern
wtf. not exactly what i was expecting from this talk. time to go drink.

——————————
Drop ACID and think about data
——————————
ACID: a promise ring your DBMS wears
* atomicity – all or nothing
* consistency – no explosions during crashes/etc
* isolation – no fighting from writers, can pretend transactions are serial
* durability – no lying to you.. assume things worked

scalability and reliability: requires a bunch of stuff from rdbms to get there

CAP theorem says pick 2:
* consistency, availability, partition tolerance

BASE:
* basically available, soft state (doesn’t have to be consistent immediately), eventually consistent
* Google BigTable
– append-only data makes it faster to insert
– fantastic data compression due to hybrid row/column storage
* Amazon Dynamo key/value store
– consistent hashing to distribute data to appropriate node
* Facebook Cassandra
– OPEN SOURCE!! incrementally scalable
– no compression, not quite polished

distributed db’s are the new hotness, but none are awesome yet.
* we don’t need another half-baked dynamo clone

memcached: key/value store used as cache
* RAM only. throws data away. lightning fast. everyone’s doing it.
* invalidation is hard and failure-prone
– be careful with mutable data. consistency will be lost over DB solution.

Tokyo Cabinet/Tyrant: super enhanced berkeleydb key/value store
* disk persistent, but still very performant
* actively developed
* replication similar to mysql
!! possible solution for photoworks session

document databases
* schema free, easy to use,
* easier to migrate. some documents will just have different properties.
– but application still needs to be able to deal with this
!! we kind of got this with photoworks project meta-data
* couchdb – document db poster child
* mongodb – way faster than couchdb. json/bson (binary json-ish)

column databases:
* sequential reads are awesome
* compresses much better than rows
* monetdb – DON’T USE THIS

——————————
Concurrency and Distributed systems
——————————
concurrent > parallel > distributed

goals:
* decrease latency, increase throughput

issue?
* global interpreter lock (GIL). one thread at a time.
– ugh. slow. boo.
– perhaps fixed in unladen-swallow ?
* but you can build concurrent regardless of GIL

multiprocessing in 2.6+
* processes and IPC via pipes to allow parallelism
* similar API to thread/queue

more on GIL workarounds…
* Jython
– allows free-threading via java.util.concurrent, at the cost of losing C extensions
* IronPy
– also unrestricted threading, and some C extensions via ironclad
* PyPy
– complete rethink of interpreter
– not production ready

Twisted
* asynchronous, event driven multitasking
* supports thread usage, but may not always be threadsafe

Kamaelia
* easy to understand. very easy to get up and running. IPCs, Threads, etc abstracted away
* compontents talk vis mailbox. cooperative multitasking via generators
* lots of fantastic examples
!!! try this out

problem w/ frameworks
* twisted/kamaelia radically different
* you end up adapting the methodology in app, which can be dangerous

messaging/rpc flooded with libraries
* Pyro for RPC should be checked out
* memcached for shared memory
* Stomp, ApacheActiveMQ for message queues
– space flooded b/c heavy use by web apps for asynch processesing right now

Notes from Sunday

—————————————–
Paver
—————————————–
build process helpers
* cmd line
* config
* work w/ files
* svn, sphinx, virtualenv, etc.
* distutils, setuptools

without paver, you end up writing a seperate script for everything you want to do

$ easy_install Paver
$ paver paverdocs #displays Paver documentation

build files…
$ vi pavement.py

options(
    setup = setup_meta,  #via setup.py
    virtualenv=Bunch(
        packages_to_install=["", "", ""],
...

tasks are the heart of paver

@task
def clean():
   """ cleans up paver dict,  removes virtualenv tree and build dir """

other fun decorators too
@cmdopts – integration w/ optparser to set up cmd line options for your tasks

paver.easy
* logging, unix command running, dry-run stuff against command line

paver.path
* jason orendorff’s path.py module built in

paver.doctools
* sphinx (generate html docs)
* ned batch’s Cog support. “section off” pieces of code and pull into your docs dynamically.

paver.ssh, paver.svn, paver.virtual (virtualenv)

deployment of code delegated to pytoss (?)

getting started: http://bit.ly/starting_paver

—————————————–
Functional Testing Panel
—————————————–
windmill
* python proxy – avoids same domain security in browser
* python browser launching
* python CI reporting
* javascript simulate user interactions – tries to get it right (bubbling, mouseclick events)
* javascript waits for dynamic content
* javascript assertions

selenium:
* “record/playback is the root of all evil”. none of creators wanted to do it
* next version will have more native C hooks. javascript event faking could only get so far.
* really only good at testing in firefox, rest of browsers (including chrome) suck

twill:
* technically just a small DSL on top of mechanize/clientform/beautifulsoup
* forms, cookies, redirects, link checking
* but no javascript support. ever. not even a little.

webtest (ian bicking)
* runs fake request/responses on WSGI frameworks
* written to test paste
* doesn’t actually run http

from myapp import wsgi_app
from webtest import TestApp
app=TestApp(wsgi_app)
app.go('/')
print app.resp

nose
* nose.cfg for default cmd line opts
* –with-django plugin to help set up env

py.test 1.0
* distributed testing. central reporting.
– run your tests on all different envs (clouds, o/s’s, py versions, etc)
* plugins similar to nosetests. but not documented all that well
* runs unittests, doctest, restdoc, pocoo (sends failures to pastebin)
* tetamap.wordpress.com, pytest.org

vision: make automated deployment merge w/ actual deployment, and have success
be mandated for deploy.

—————————————–
Lightning talks
—————————————–

== argparse ==
all features of optparse, but better docstrings and never touch sys.argv
sub command parsers (e.g. svn add) easier than optparse
mutually exclusive arguments

http://code.google.com/p/argparse

== python 3.1 ==
some performance improvements. IO now almost as fast as 2x
unittest improvements. assertRaises trick
OrderedDict finally arrives

from __future__ import bad_syntax_ideas
* braces, no colons, no indentation, implied ruby blocks, etc.
* pep 3117

== Poor Man’s Continuous Integration ==
try to do it without buildbot using your existing tools
* capastrano, maestro, git/svn
using “fabric” instead of “git push”
$ fab push -d staging_server deploy
* wraps up the test run and git push steps

== cluemapper – software project manager ==
example: lots of software projects
* each w/ issue tracker, svn
* want to share users across projects
built on top of trac. “trac for teams”
* basic reporting and time-tracking
* trac plugins to extend core
* wsgi + dojo ajax
components
* irc bot
* theming
* pypi egg server
* bzr integration
* pastebin
* timetracking, reporting
nice source control user manager for each repository
roadmap
* agile/scrum/xp stories and whiteboard
– drag/drop between stories and iterations
!!!!!!!!!!!!!!!!!!!!!!! yayz !!!!!!!!!!!!!!!!!!
cluemapper.org

== ReleaseBot ==
release mgr process watches buildbot
push button to tag a version for deploy
* workflow for notifying QA that this is what needs tested
buildbot notices tag and preps for release
not yet developed, just idea phase

== pyjamas ==
python to javascript compiler, with built-in widget set
JSONProxy seems fun!
rest is pure evil.

== \ is evil ==
don’t split lines with a backslash
use parentheses instead

my_str = ("asdlfkjasdf   asdlfkjdsf"
        "  asdflkj   asldkjf "
        " alskjdf asdlkjj ")

insetad of

evil_nasty_str = "asdlfkjasdf   asdlfkjdsf" \
        "  asdflkj   asldkjf " \
        " alskjdf asdlkjj "

Mini-conclusion
Things for follow-up:
* windmill. first round of experimenting at openspace didn’t prove to be as awesome as i had hoped. but i’m willing to give it another try. scripting these in python feels like an afterthought, and it wouldn’t be easy to get integrated with BuildBot.
* ClueMapper. looking forward to the scrum/xp tools built into Trac.
* webtest instead of twill for scripting end-to-end tests
* py.test for distributed testing
* tokyo cabinet for key/value store with persistence
* web2py web framework. the data schema admin manager doesn’t feel quite as polished as django, and i don’t like the online editor. but their MVC ideas and templating implementation might be useful.
* mercurial or bzr. i need more practice with DVCS. i committed all my code throughout the weekend to a local mercurial instance, but never pushed up to a central repository.

That’s all she wrote for now. Looking forward to future PyCon’s and to trying out some of these awesome tools.