For those of you following us on LiveJournal, I'm pleased to announce that we have a new blog up here:

blogs.divmod.com

Please update your bookmarks and/or news readers, and enjoy the new look :-)

I know everyone is all aglow with the new web development offering from Google, but let me do the unpopular thing and put some things into perspective: there are limitations.

In fact, the limitations that exist will prevent me from using App Engine with all of my projects, save one (that one being a very simple web site). First, the limitations that prevent me from using App Engine (from one of their FAQs):

• Sockets are disabled with Google App Engine
• The system does not allow you to invoke subprocesses, as a result some os module methods are disabled
• Threading is not available

This means that I can't write a deferred wrapper for their data layer, I can't use Twisted for such things as XML-RPC or AMP-based communications, and I can't use an async templating system (like Nevow). I'm stuck with CGI and blocking code. And for all but the simplest projects, that's a big "No Thank You" from me.

This doesn't mean that I won't use it -- I will. I have one project that this will be perfect for... but it's for someone else, not me.

However, these limitations are actually good news :-) Here's the silver lining:

As Glyph as alluded to in his recent blog post (and in our tweets), we've recently completed a massive week-long BizDev Divmod sprint in Boston. One of the results of this is based on community feedback we've had over the last year, and which culminated at PyCon 2008 in Chicago with multiple requests for particular services from The Twisted Company. That result is a set of tools, features, and management options folks will be able to use with our software (app server, smart object db, network services, etc.). People really want to start using our stuff in cloud/grid computing environments. They need support for multiple and diverse network services, inter-store communications, massive deployments, etc. Two months before PyCon, we started working on tickets to support this, and we're making excellent progress toward providing the requested features.

We're still unclear as to which parts of this will be open source, as that will be driven by a combination of business and community demand. Regardless, Google's lack of support for this stuff has (for now) left the field wide open for us. And that, folks, is a big "Thank You Google!" :-)

Technorati Tags: business, cloud, community, divmod, google, grid, mantissa, networking, nevow, pycon, python, services, software

As Glyph just said, our community/development site is "reloaded" with a fresh look and the beginnings of some new structure. This comes as a result of many interdependent activities in the offing here at Divmod, most of which are still in the oven (and we wouldn't want to ruin the surprise now, would we?).

At PyCon, many of you approached us about more than the Twisted stuff we've been working on, and we had some good conversations. We've listened to all of you and have been making Herculean strides to provide a clearer view into what we do and how it can help you. We've got a long way to go regarding site content improvements and enhancements to documentation, both of which are genuinely at the top of our list right now. Yes, we've got a vision; but that is truly nothing without the continued interest and support of curious and creative folks like you, who want to architect extraordinary software.

There's been an extraordinary amount of energetic development, conversation, brainstorming and contribution that's been happening on IRC, at our offices, and in the code base, by employees and community members, at work and at play -- it's terrific to be a part of something so genuine and organic (the rumors of inorganic overlords have been greatly exaggerated).

Come take a look! ... And stay for a while :-)

P.S. If there are any CSS-on-IE Super Freaks in the House, will you please stand up? We're trying to work on some styling oddities that exist on the site and we could use some help!

Update: Paul Hummer has been amazingly helpful and got us up and running on IE when he had spare moments today. Thanks Paul!

Technorati Tags: business, community, design, divmod, mantissa, nevow, python, web, work

After a last minute rush, there will be a significant Divmod representation at PyCon this year, complete with sprinting! We're focusing on a community release for Mantissa with improvements in many areas of the Twisted application server as well as Nevow. Those without experience are just as welcome as seasoned hackers, since part of our release goal is to make Mantissa as easy to understand as possible :-)

Sign up to join us here:

• http://wiki.python.org/moin/PyCon2008/SprintSignups/NevowMantissa

We've added a bunch of useful links that you can use to prepare for the sprint (which starts after the PyCon talks, on Monday). If you have any questions, please come by #divmod on irc.freenode.com to get clarifications. You can also participate in the sprint remotely via #divmod, if you're not making it to PyCon this year.

Check out the complete list of sprints here:

• http://us.pycon.org/2008/sprints/projects/

Special thanks to Facundo Batista for setting things up for us!

In several previous blog posts, private email conversations with some of you, and Facebook conversations with others, I have referenced Divmod's new product. We quietly released our first beta on November 9th and are currently about 85% through the second beta (to be released some time in the next couple of weeks). I'm blogging about it now because I just can't wait any longer :-) And we've been getting some really great feedback from users.

"But what is this mystery product?" you ask. Say no more! I am talking about Blendix. As the front page says, this first incarnation of Blendix is primarily a tool for consolidating web properties into a single URL, for your convenience and that of your friends/family. A striking example of this is my page on Blendix: https://blendix.com/users/oubiwann/. As you can see from the "Now Blending" panel, you'd have to visit 14 URLs at 10 different sites to see this same content. By checking me out on blendix.com instead, you are not only saving yourself the time and trouble, but you're getting a more complete picture of what I've been up to.

"Um, yeah... looks like an RSS reader to me..." True, there are a lot of similarities. But there's another twist: you can add your friends, too. Even the ones that don't want to create an account on Blendix :-) I'd say that it's a safe bet most of us have friends who publish some sort of content online but who also can't be bothered to join yet another social networking or similarly service-oriented site. With Blendix, that's not a problem -- you don't have to wait on anyone :-) You simply add them to your contacts where you can provide feeds, email addresses, and user names. Then, their public content (i.e., "feeds") will be displayed at a unique Blendix URL.

Right now, anything that provides an RSS feed can be added to your profile (and your contacts' profiles). Additionally, we're getting custom content from such sources as Amazon wishlists, last.fm playlists, weather, Google maps, and Flickr images... with lots more planned :-)

I wanted to write about Blendix here (on my blog) before we made any official press releases because you guys -- the members of the python community -- are our "home town" :-) We want you to get "first dibs" (as it were) and we wanted you to hear the news from one of your own, not from the cold lips of some news story. We're keeping things small for now, but if you have any interest (or if you use similar features with a related service), please come check us out (oh, and this too), sign up, take an account for a spin, and email us with any problems, questions or suggestions.

The team has been in amazing spirits during this multi-month push, working really hard, and adding new features left and right. We've been at it for a while, so competition is growing in this increasingly important user space. We may not be the shiniest or have the most users, but it is consistently apparent that our product is addressing needs and features that others seem to be skipping. Our development process may have taken longer than others due to the infrastructure we decided was essential to build, but this has given us an edge over other shops with similar plans.

And boy, do we have some nice surprises in store for you in the months to come :-)

Technorati Tags: business, community, divmod, internet, software, web

So I've been working at Divmod for several months now, and I thought I'd report on things a little bit. I've actually wanted to do this for a while, but things have just been soooo busy here, that I honestly haven't gotten the chance.

Simply put, this is the best company I have ever worked for. The team is brilliant and hard-working; the (virtual/distributed) working environment is highly creative one; the CEO is amazingly supportive and gracious; the software and supporting architecture is fascinating; and our product is actually useful and people like it :-)

I've done my time in Corporate America and swore I'd never go back. I've been a consultant since 2002, and the fact that I am actually an employee again really says something about the company I working for and team I'm working with.

Technorati Tags: divmod, work

As promised earlier, here is the new Athena tutorial!

Currently, the tutorial consists of a topical introduction and two example applications, built with the reader step-by-step. Note that this is still in branch and hasn't been reviewed for inclusion in trunk. Please feel free to leave comments to this blog post about the tutorial (good, bad and ugly). This will help us improve the content and prepare it for inclusion in trunk.

The tutorial includes detailed instructions on setting up an environment, obtaining the source code, and running the applications. There are also pointers on how to conceptually approach development with Nevow/Athena.

Due to concerns over code compactness, we pulled out the CSS that nicely styled the echo and chat tutorial apps. The curious can view the styled versions here:




And if you want to pretify your example apps, you can update the fragment templates per the links given below.

• Echo App
• Chat App

Technorati Tags: ajax, comet, divmod, documentation, python, twisted

Some of you may remember when, given 5 seconds to say something about Nevow's AJAX capabilities at the PyCon Web Framework panel, I said "we were there first." I was serious about that. If you take the time to dig through the dusty attics of Twisted's and Divmod's svn repositories, you'll see it too.

Not that I blame anyone for not knowing, though. If there's any two crimes that Twisted devs are guilty of (among potentially uncountably many) it's:

1. being frighteningly clever, and
2. not sharing the useful bits of the cleverness with the world at large.

We're trying hard to really start sharing the goodness, though, honest!

Along these lines, we've got a series of tutorials about Athena that will be published "real soon, now." Initially, we're going to put out some intro text and two basic tutorials. Follow-on tutorials will cover advanced topics and/or basic ones in more detail.

It will be very exciting, though, to bring the power and elegance of Athena to the masses :-)

Personally, I've found writing Athena apps (with its tightly bound Python and JS) provides ease of use and peace of deterministic structure in an arena (AJAX/COMET application development) that is usually littered with the bodies and casualties of poor planning, poorly established boundaries between application components (in requirements, architecture, and implementation), and poor education.

I look forward to community feedback on the tutorials: it's a complicated topic that will be condensed into only a little code in each fully functional example. The need for clarity and the defeat of confusion is paramount.

Technorati Tags: ajax, comet, divmod, documentation, python, twisted

When I was on the development team for Zenoss, I tried to get them to use branch-based development, including Divmod Combinator. They eventually did switch to using branches for any big changes, but they didn't embrace Combinator. When I'd taken over development of the Zenoss community code, there were only two developers, so I only implemented a very basic branch management process. But having used Combinator a lot recently, I wish that I'd taken that opportunity to get them using Combinator... man, their lives would be so much easier right now if I'd taken that step.

I am now managing all of my svn-based projects with Combinator. Even the ones were I am sole developer. The time saved and convenience gained is enormous.

Below is a perfectly good example of how Combinator makes svn management less painful. I don't know about you, but every time I have to manually to push new trunk changes into my current development branch, my brain does a little freak-out dance. I stress, thinking "Crap; I have to get this right. What are the exact commands I have to use again?" Thanks to Combinator, this is no longer a problem, and my brain does a happy dance instead.

These days, I am using Combinator on an almost hourly basis. It's very simple (to get setup with Combinator, see the tutorial and other example usage).


Background


Assuming you've got a Combinator-managed project called "Project", let's do some background first. Say you want to branch trunk for a new feature, one that you're tracking in ticket #836. Here's how:

$ mkbranch Project viking-feature-836

Now, just to make sure that you are no longer in trunk, check which branch you are in:

$ whbranch
Project: viking-feature-836
Divmod: trunk
Twisted: trunk

(Note that every project that you are managing with Combinator will be listed.)

Say you've got another ticket you're working on, #1066. You created the branch with this command, just like the other branch:

$ mkbranch Project norman-feature-1066

After starting this branch, you complete 836 and merge to trunk:

$ chbranch Project viking-feature-836
$ unbranch Project
$ svn commit -m "Completed 836."

unbranch updates trunk with the latest changes and then merges the branch it just came from (viking-feature-836) into trunk. Running svn diff will show you just how true this is.


Merging


So much for the background! Now for the merging I'd promised:

If the norman feature depends on the viking feature, we need to "merge" trunk to branch. Using Divmod Combinator, this is how that is accomplished:

$ chbranch Project norman-feature-1066
$ unbranch Project
$ mkbranch Project norman-feature-1066-2
$ cd ~/Project/branches/norman-feature-1066-2
$ svn commit -m "Merged changes from norman-feature-1066."

In other words, we:

1. merge our work to date from norman-feature-1066 to trunk (without committing!)
2. make a new branch, based on that trunk, which includes the norman feature to-date and all the latest changes made to trunk
3. commit the branch changes to our new branch

We're using trunk to get what we need by only making changes to trunk in the working directory, without making any changes in the repository. Keep in mind that the changes from the first branch were added (and not committed) to trunk before creating the second branch. Since the second branch was created from this modified trunk, we will need to commit those additions. Also, you will most definitely want to revert your tweaked trunk once you have finished creating your new, updated branch!


Paths


As a side note, Combinator is especially excellent for python projects, because of its python path management features. Any python tools, scripts, pluggins, or applications that use the code in your Combinator-managed projects will import from the current, active code (whatever shows up in whbranch). So when you're working in branch and you run your unit tests, you know that it's the branch code that's getting tested. When you switch back to trunk, all your code is running against trunk. Very nice :-)

Technorati Tags: branch-based development, divmod, python, twisted, version control

Tags: branch-based development, divmod, python, twisted, version control

Have you ever written a module that looked like this?

subscribers = []

def addSubscriber(subscriber):
    subscribers.append(subscriber)

def publish(message):
    for subscriber in subscribers:
        subscriber.notify(message)


And then used it like this?

from publisher import publish

class worker:
    def work(self):
        publish(self)


I've done this many times myself.

I used to think that this was the "right" way to implement Singletons in Python.  Other languages had static members and synchronized static accessors and factory methods; all kinds of rigamarole to achieve this effect, but Python simply had modules.

Now, however, I realize that there is no "right" way to implement Singleton in Python, because singletons are simply a bad thing to have.  As Wikipedia points out, "It is also considered an anti-pattern since it is often used as a euphemism for global variable."

The module above is brittle, and as a result, unpleasant to test and extend.

It's difficult to test because the call to "publish" cannot be indirected without monkeying around with the module's globals - generally recognized to be poor style, and prone to errors which will corrupt later, unrelated tests.

It makes code that interacts with it difficult to test, because while you can temporary mangle global variables in the most egregious of whitebox tests, tests for code that is further away shouldn't need to know about the implementation detail of "publish".  Furthermore, code which adds subscribers to the global list will destructively change the behavior of later tests (or later code, if you try to invoke your tests in a running environment, since we all know running environments are where the interesting bugs occur).

It's difficult to extend because there is no explicit integration point with 'publish', and all instances share the same look-up.  If you want to override the behavior of "work" and send it to a different publisher, you can't call to the superclass's implementation.

Unfortunately, this probably doesn't seem particularly bad, because bad examples abound.  It's just the status quo.  Twisted's twisted.python.log module is used everywhere like this.  The standard library's sys.path, sys.stdin/out/err, warnings.warn_explicit, and probably a dozen examples I can't think of off the top of my head, all work like this.

And there's a good reason that this keeps happening.  Sometimes, you feel as though your program really does need a "global" registry for some reason; you find yourself wanting access to the same central object in a variety of different places.  It seems convenient to have it available, and it basically works.

Here's a technique for implementing that convenience, while still allowing for a clean point of integration with other code.

First, make your "global" thing be a class.

class Publisher:
    def __init__(self):
        self.subscribers = []

    def addSubscriber(self, subscriber):
        self.subscribers.append(subscriber)

    def publish(self, message):
        for subscriber in subscribers:
            subscriber.notify(message)

thePublisher = Publisher()

Second, decide and document how "global" you mean.  Is it global to your process?  Global to a particular group of objects?  Global to a certain kind of class?  Document that, and make sure it is clear who should use the singleton you've created.  At some point in the future, someone will almost certainly come along with a surprising requirement which makes them want a different, or wrapped version of your global thing,  Documentation is always important, but it is particularly important when dealing with globals, because there's really no such thing as completely global, and it is difficult to determine from context just how global you intend for something to be.

Third, and finally, encourage using your singleton by using it as a default, rather than accessing it directly.  For example:

from publisher import thePublisher

class Worker:
    publisher = thePublisher

    def work(self):
        self.publisher.publish(self)


In this example, you now have a clean point of integration for testing and extending this code.  You can make a single Worker instance, and change its "publisher" attribute before calling "work".  Of course, if you're willing to burn a whole extra two lines of code, you can make it an optional argument to the constructor of Worker.  If you decide that in fact, your publisher isn't global at all, but system-specific, this vastly decreases the amount of code you have to change.

Does this mean you should make everything into objects, and never use free functions?  No.  Free functions are fine, but functions in Python are for functional programming.  The hint is right there in the name.  If you are performing computations which return values, and calling other functions which do the same thing, it makes perfect sense to use free functions and not bog yourself down with useless object allocations and 'self' arguments.

Once you've started adding mutable state into the mix, you're into object territory.  If you're appending to a global list, if you're setting a global "state" variable, even if you're writing to a global file, it's time to make a class and give it some methods.