A Toolkit for Design Docs

This post is reproduced from its original home on blog.e2.gg, where Evan and I share our thoughts on engineering, management, and remote work.

As I have (somewhat unexpectedly) been drawn into the world of management, one of my favourite parts of the job has been mentoring and uplifting then engineers in my organization. One of the most critical parts of that process is the fitting of incrementally more ambitious and open-ended projects to devs at each step of their learning curve, and the most important part of that, in terms of ensuring their success, is act of writing effective design documents for their projects.

This isn’t a post about why design docs are fundamentally critical to the design of good software (Joel Spolsky articulated that more effectively than I ever will, as have, honestly, manymany others) but it is worth noting that single most common root cause of failure that I’ve seen hamstring projects, particularly those run by more junior engineers, has been inadequately thorough design documents. This is the singular moment during which more senior engineers can scrutinize each element of the design, particularly the nasty and easy to miss failure cases, in the context of the whole (other than, of course, after the project launches and things start breaking). However, the exercise of communicating technical intent, complete and unambiguous, is a new and foreign challenge of its own – and even just getting started can be a daunting task to someone writing their first design doc.

So instead this is a post about what you need to get started writing your first (or your hundredth) design doc, in as brief and functional terms as is possible. With the following, I hope to provide all the structure and tooling you need to create something comprehensive for review from your peers, as well as offering a few tips and tricks to keep in mind as you fill out the format along the way.


There’s nothing more intimidating, as a writer, than a blank sheet of paper – where do I even start? To that end, for any informative documents I’m required to produce, I start with all the sections and headers I can identify, and turn the problem from “filling out the page” into (as much as is possible) “filling in the blanks.” Although the following may not quite be universal – you shouldn’t shy away from altering it to fit your domain – I find that generally starting with this format gives me enough structure to start breaking down the document into smaller problems to digest.

  1. Overview
    1. Stakeholders/key personnel – Start off with who are you (author), who owns the thing you’re building (teams), and who cares about it (key stakeholders). Although this may be unnecessary in some smaller docs, for a broader audience it really helps the folks asking questions if they know who they’re talking to.
    2. Background – This is where you start the narrative. It’s always important to think about how the content of your document flows from section to section, each conveying unique information that supports the conclusions that follow, and this is where you can lay your groundwork for your readers’ comprehension of that narrative. Keep it short, this isn’t the time to justify all the specifics of your solution, but set the stage with the impetus for the feature, and any requisite technological context.
    3. Essential terminology – Although there’s a good chance you can skip this section, take a moment and consider – are there any terms that are specific to this domain that your readers are unlikely to know, or are you introducing novel terminology for the purposes of your solution? I’ve seen countless technical discussions spiral into frustration because speaking unambiguously in english is actually really hard, and anything you can do to preempt potential confusion will save you effort down the line.
    4. Objectives
      1. Goals – What are you going to accomplish?
      2. Non-goals – What are you not trying to accomplish? Don’t put world peace. Many projects have intuitable extensions, or adjacent problem domains that you may be pushed towards rolling into scope. Push back – be clear regarding your boundaries (in documentation as in life).
  2. Technical Summary
    1. General Overview – Explain the logic and flow of your solution in two to three paragraphs, or if you’re feeling ambitious, five or six bullet points.
    2. System Diagrams – At least a broad system overview, with diagrams for key subsystems as necessary, with complimentary notes providing context as necessary.
    3. Key Components – Using a new AWS offering? Have a really complicated class heirarchy that’s central to the solution? Offering a new API to internal clients? Enumerate them here.
  3. Technical Flows – This is a chance to explain how what you’re building will be used. Whether it’s a library, an API, or elsewise, understanding it through its flows rather than it’s atomic points of interaction is a much more informative lens for critiquing the design. Sequence diagrams are immensely useful here, and I’d strongly recommend getting familiar with them for anyone who writes client-facing code (more on them later).
  4. Contracts/API – Okay, so now that we understand the flows, what exactly are our points of contact? The prior segment should have had most of these listed by name, and now that the reader understands how they’ll be used you can step through and give more detail on specific arguments, return values, and encapsulated business logic.
  5. Persistence – Are you storing data? Great, now let’s get into the details. How is that data modelled? How is it indexed? Is it scalable, and is it flexible enough to support future use cases? Bad modelling at the persistence level often causes problems down the road – particularly for the second version of a feature as you expand beyond the tightly scoped initial problem domain, so making sure the constraints of your design are well understood at the outset can be enormously valuable.
  6. Observability
    1. Metrics – What are you counting?
    2. Monitors – What are you analyzing?
    3. Alerts – What do you want to respond to?
  7. Risks and Risk Mitigation – Where could this project go off the rails? What business outcomes or technology constraints might result in failure cases? What might extend the timeline beyond your current estimates for delivery?
  8. Deployment/Rollout Strategy – “Turning it on” is another common point of failure for new features. Particularly if your project is a new/alternative flow or resource, ensuring that that any ramp up is smooth and well monitored is essential. How to manage deployments is the subject of another post, but at a minimum cover timelines, rollback plans, backwards compatibility, monitors to track during deployment, and metrics to be used as success criteria.
  9. Appendix – At this point you’ve covered your bases! Long form references, or content that isn’t strictly necessary to the pitch but may be useful to a curious reader can be plugged into an appendix, making it clear that this isn’t part of the body of the proposal. You can drop basically anything in here, but a few likely entries follow:
    1. Alternatives – Are there other strategies that are still worth considering, or that you want to justify in their exclusion?
    2. Required Sign-offs – In some cases explicit sign off from a set of technical partners (for inter-service contributions, for example) may be useful to have on paper. Drop it in here!
    3. Follow-on projects – Where are you taking this project post-MVP?
    4. Specific scope exclusions – Any potential expansions on the non-Goals as listed above.
    5. Dependencies – Projects being executed in parallel, requirements of non technical partners, really anything that may block the project from proceeding.


Of course, not everything you’ll be sticking in that doc will raw text – it’s helpful to have diagrams and tables to present specific structured information, and so I’ve highlighted a few of the most lightweight and useful tools I’ve come to appreciate in my time producing design docs.


Where do you actually write the doc? Honestly, “wherever you’re most comfortable” is a good place to start, and the first few hours should mostly be spent dumping as many ideas as possible out of your head and into their respective sections in preparation for organization and editing. That said, once you’ve got a semi-mature draft I strongly recommend finding a buddy, copying the content into a collaborative editor like google docs, and soliciting feedback as early as possible. You don’t have to have every potential question answered and every spelling eror fixed before you start getting criticism, and it’ll save you having to make larger changes when your document is more complete. Is this obvious? Sure. Have most people put off reaching out to peers in favour of adding a little more polish first, to their net detriment? Absolutely.

System diagrams

A high level system overview is near essential for any system describing more than one client and one server. Although you can get by with just Google Draw (or, for that matter, MS Paint), there are better solutions out there. Diagrams.net is free, easy to use (many of the tools in this space are far more complicated than necessary for the general case), and produces diagrams that aren’t an eyesore. If you don’t want to think about what to use here (and you shouldn’t, generally, that’s not the problem you’re trying to solve), just use this. Technically this site addresses almost every use case listed here, but due to my preference for text-based diagramming tools (over wysiwyg), I’ll include more specialized tools in the following sections.

Sequence diagrams

These are the bees knees, and criminally underused. Many technical specifications are most usefully understood through the the flow of data and the sequence of their use, but it can be hard to express these things in raw text without being so verbose as to obfuscate the key information. Sequence diagrams offer a flexible and fun way to render these flows, and in many cases can replace hundreds of words of text with a few easily understood images. I’ve been using websequencediagrams.com for years, and would strongly recommend them for their simplicity and clarity.

Entity Relationship Diagrams

Got a bunch of models, and those models are related to each other? Just like they hammered home in CS101, entity relationship diagrams are your friend. They’re simple and easy to read, so although they can be skipped over in more simply modelled domains, they’re an extremely efficient utility to have in your toolkit when necessary. Diagram.codes has a bevy of diagramming options, is well designed, and it’s ER syntax is particularly clear and simple.

API definitions

For long term documentation I think that APIs should almost always have their docs written inline with their method signatures and docs should be generated from there. However, before the API exists in code, tooling for expressing the contract you’ll be putting in place is a lot more sparse. Most popular options, like Swagger, are a little heavyweight for the brief and high-level definitions that might exist in a design doc, and I haven’t found any solutions that I’ve really loved. As such, I basically just copy and paste a rich-text version of this snippet, and now you can make your own too!

| Endpoint Name/Path                                                                   |
| Description        | A little about the endpoint, how it operates, and why it exists |
| Arguments:         | |Parameter| Type |            Description            |          |
|                    | | ------- | ---- | --------------------------------- |          |
|                    | |   id    |  int | the uuid of the referenced object |          |
| Response:          | Serialized response object                                      |


So now you have a structure, a general sense of content, and a few handy tools for mocking up the diagrams that you may want to include – you’re pretty much good to go! Before you rush off though, I’d like to offer a few more points to consider as you draft your first (or fiftieth) design doc.

  • Good headers help set expectations for readers and break up your document into easily parsable components. Good formatting seems like a nicety that many developers see as unnecessary frivolity, but self explanatory headers with good visual distinction between h1/h2/h3/etc. make a huge difference for reader comprehension.
  • Don’t just cover the happy path. Failure modes are an essential component of a design doc, and warrant equal emphasis along with the success cases.
  • The design doc should not be a product doc. Linking to the product doc near the top is a great idea, but retreading product/business outcomes, unless the project is strictly technical (and even then you could pull those out into another doc), is often counterproductive because it invites debate on a tangential subject, rather than focusing attention on the technical design.
  • Emphasize complex elements, and unique/novel (within your context) elements, leave explanations of common infrastructure out or note them only briefly.
  • This is a long doc structure. It can feel unduly burdensome for smaller projects, and even for larger projects it’s easy for this to feel like busywork once the author feels they’ve fully internalized the solution. Again, I don’t want to litigate that these docs are valuable (they are), but if you do feel like this is too much – do less. Something is better than nothing. Skipping some sections in the name of velocity or flexibility or just in the name of saving your sanity after a long week, can be totally fine so long as the doc still covers the core design elements of your solution. Edit the format to fit your needs and your patience, but remember the more thorough the doc, the more detailed feedback you’ll get from your peers, and the more value it will return to the project.

Finally, one last note before you go – this guide is intended to inform a formal proposal of a solution, which may be several docs deep in the design process already. You might write out a complete list of approaches to be debated, early on after identifying a problem space; you might throw together a one-pager outlining your preferred solution when you’re first seeking approval from your peers or boss to explore the direction; those are good an important, but should not be as exhaustive as the document outlined above. A design doc should assume an opinionated position regarding the correct solution, and justify that opinion through its thorough treatment of the problem space. If you’re not ready to tackle the design doc yet, that’s okay! Seek advice from peers or mentors, do a lightweight comparison of alternative solutions, and make clear to your stakeholders that you need a little more time – this is a big commitment! But when you are ready, I hope this guide helps make your experience of writing your design doc just a little bit easier.

Why I Can’t Buy Lightbulbs

I like shopping. Not that I particularly enjoy going to stores, or the acquisition of goods, or spending money, but for me shopping is really about research. And I love research.

When I discover (or decide) I need some new thing, I’ll immediately and excitedly block off a few hours for my upcoming study session. Which are the best 1“Best” meaning here a healthy mix of objective superiority and subjective preference. versions of that thing? What brands are reliable? Are knockoffs good value, or compromised imitations? Where can I get the most value for my Canadian dollar? And, most importantly, what are the core attributes I should check for when browsing options?

To take a recent example, I recently decided that the green coating of pollen covering our windows had to go and bought myself a pressure washer. These days, unless you have to do some serious industrial work, the best pressure washer is going to be electric (lower maintenance, more reliable, lighter, no need for gas, albeit less powerful), and the best of those is going to have a brushless motor. There are a few big-name brands, but knockoffs are surprisingly acceptable given that even the most popular models have reliability issues, so the delta between mainstream quality and off-brand isn’t necessarily that large (though neither is the cost difference). That said, you should still make sure that you get one with metal fixtures (most common point of failure), and that hits a minimum of about 1800psi and 1.2 gallons per minute. Take those criteria, match it against your personal preference, budget, and preference for certain bells and whistles, and it’s easy to find a model that’ll be a quality fit for your needs.

There are a few notable kinks to this process – particularly in categories where there simply are no good options, like printers. In those cases I’ll waste a few quiet hours of my life, getting gradually more desperate as I slowly disqualify every major product line and eventually settle for something that looks pretty (because if it’s going to end up a paperweight, might as well be a pretty one, right?). But, for the most part, it serves me well and brings a great deal of enjoyment to my life.

Until a light burns out. Every time I see a bulb flicker my stomach drops, my anxiety rises, and I consign myself to a few more days circling the drain of lightbulb shopping. What’s unique about lightbulbs? Well, the problem is that, unlike printers, there are good options. There’s a surprising range of quantifiable aspects of lightbulbs that make a given bulb “good,” but actually finding one to buy that meets all those success metrics is an endless cycle of frustration. So what makes it so challenging? Hey, if you’ve made it this far, I’m assuming you’re interested, so let’s go down the rabbit hole of quantifying lightbulbs.

What makes a good lightbulb?


If you’ve shopped for lightbulbs before, this is probably the first thing that came to mind. Lights can be “warm,” which is to say that they have a yellow/orange cast or “cool,” in which case they have a much bluer tint. Warmer lights (lower on the scale) are cozier and softer, but too warm and things start to look dirty. Colder lights (higher, starting around 4000k) have more of a crisp, clean, bright look, but can be harsh and antiseptic as you get too far down the scale. Picking the right warmth for the right room (warm for bedroom, cool for bathroom, etc.,) is important, but I find my all-purpose warmth is about 3500 kelvin. 3000 kelvin is the most popular option but verges on a bit too warm and muddy for my taste (and don’t give me any of that 2800k nonsense), whereas by 4000k things are starting to look a little harsh. Thankfully, 3500 kelvin isn’t overly uncommon, so this isn’t the most significant constraint.


Not too much needs to be said about brightness, but it’s still necessary to consider to weed out the real oddball bulbs. Brightness is measured in a unit called “lumens,” and the sweet spot for general purpose lighting is about 800 lumens. This is about equivalent to a 60 watt incandescent bulb, for those of you old enough to remember when those were the only option. Any high end use cases will usually call for closer to 1600 lumens, and a moody side light might be closer to 400.

CRI Rating

CRI rating is where things start getting tricky. CRI stands for Colour Reproduction Index and, put simply, is a measure of how closely the light output by a bulb will reproduce colours as they would be seen under natural sunlight. Because the light emitted by a light source is not produced in uniform intensity across the visible spectrum, a light source with an excessively skewed output will mute or oversaturate certain colours, dependent on the distribution.

Sunlight, with a relatively uniform distribution, is treated as the norm, and so we measure lights on their ability to reproduce a key set of 15 colours, scoring each on a scale of 1-100, and taking the average to get the aggregate CRI scores.

A light with an average CRI score of 90+ is generally seen as having good performance, while top performers are rated 95+. It’s notable that there are a number of complaints with how manufacturers grade their performance against this scale (and with the system itself), but broadly a 95+ CRI rating will mean that the colours in your home (both paintings and people) will look bright and lively. Except…

R9 Rating

Given that we have this convenient measure of colour reproduction across 15 different swatches, you’d think we’d be taking the average of the full set, right? Hah, nice idea, but no. Turns out the CRI that is advertised on most bulbs is the “General CRI” (sometimes referred to as Ra or CRIa), which only takes the average of the the first eight colours. So even if a light performs abysmally on the next seven, it could still get a 95+ CRI score by specifically optimizing for the first set. Why is this the standard? I honestly have no idea, but I’m chalking it up to the deep pockets of Big Lightbulb.

This is particularly relevant because of the recent technological shifts in bulb making. I’ll review the options a little later on, but suffice to say that the most popular bulb type these days is LED, and it just so happens that it’s extremely common for LED bulbs to perform terribly on the ninth colour of the CRI swatch, generally referred to as R9. So those high scoring LEDs that you just rushed out to buy? They’re going to make anything with red tones look awful (which, notably, includes some skin tones, a major drawback to LED lighting). Unfortunately, most manufacturers don’t advertise their R9 scores anywhere, outside of a few specialty manufacturers. Thankfully there are, as always, intrepid internet researchers who have spent a great deal of money on specialty gear to measure the R9 scores of popular bulbs as well as the CRIe score, which is short for CRI Extended and represents the average of the first 14 swatches 2TCS 15 is was added much later to the core set, hence CRAe being generally measured as just the first 14. The 15th was added by the Japanese Industrial Standards organization (see JIS C 1609-1:2006 Illuminance Meters).

Shape and Socket

Of course, whether or not a light is good or not is irrelevant if it doesn’t fit the socket or the space. Most of the time hopefully you can just buy A19s and spend no more time thinking about bulbs than is strictly necessary (‘A’ lights are the familiar bulb shape with a standard light-socket fitting, 19 is a measurement of size and is the most common measurement). But, if you’re lucky like me, you may have no less than six different bulbs to shop for, each of varying levels of obscurity. For the most part my house calls for BR30s or, in common parlance, wide-ish flood lights. But I also have to accommodate T3s, GU10s, B10s (with their cute little base), E11s, and of course the venerable A19s.


In some cases, when there’s a mire of mediocre options in a product category, you can solve most of your problems and get something much better by paying just marginally more than average. Pens are a good example of this – pens you might pick up at the grocery store frequently have cloggy ink, write scratchily, and have only a passing understanding of ergonomics. But, for just a few dollars more you can get a four pack of Uni-ball Vision Elite Roller pens and have a much better time writing for years to come. Unfortunately, when it comes to lightbulbs, that’s very much not the case. Not only is there a pretty significant variance in price among even the more mediocre entries, but the price for premium products goes up extremely quickly. As an example, the truly excellent 95+ CRI BR30 bulb from San Francisco’s Waveform Lighting goes for $28 US dollars, and that’s certainly not the top of the line.

There’s not much I can offer in terms of in advice in this particular dimension, of course everyone’s ideal price point is going to be different, but unless you have an absurdly high budget for lighting (or very few sockets), this isn’t going to be a problem that a little more money will take care of.

Other things to consider

Of course like any other product, there are many more qualities that differentiate specific entries in the category, but that are not of sufficient importance to affect my decision making outside of a head-to-head between otherwise ideal options. That said, these are still worth enumerating, as they may be more important in certain contexts or could more strongly affect your preferences than mine. Directionality and uniform diffusement may matter more or less, depending on your specific need, as may dimmability (and make sure you have a compatible dimmer, or they will buzz and flicker like there’s no tomorrow!). Some might argue that the lighting technology is a matter of choice (incandescent, fluorescent, LED), but in all earnestness LED is the only reasonable choice – longer lasting, more energy efficient, and better for the environment than any of the alternatives.

The one quality which I haven’t included as a primary deciding factor, but may be a point of contention for the more discerning lightbulb enthusiasts, is lightbulb flicker. Depending on how an LED cycles, it’ll have different presentations of frequency, modulation, and ‘duty cycle’, but in my experience no bulbs that I’ve bought of reasonable quality have had visible flicker issues. As such I don’t give it particular consideration, but if you are particularly sensitive to that sort of thing, it’s definitely worth taking into account.

Was that really so bad?

So it’s simple then, right? Just find a listing for a 3500k, 800 lumen BR30 lightbulb, with a CRIa score of 90+, a good R9/CRIe score as calculated by some strangers on a forum, and that costs less than $10/bulb. Well, turns out that even if you do manage that and buy a dozen, if you go back six months later that particular bulb is likely to be gone, replaced by a marginally but critically different model that is somehow lacking or at least undocumented. There exists a mind-boggling array of bulbs from any given vendor, and the turnover is bizarrely rapid, making any precise model extremely challenging to find from any larger company. Moreover, and this is likely relevant to less of my audience (pretending that any of this was relevant, and that I have an audience), but it insanely hard to find lightbulbs in Canada. Many major brands, like Cree and and Waveform are near impossible to find (especially for specific high-performing models), and others are many multiples more expensive than their US counterparts. Many, many times I’ve finally found an high-quality, appropriately priced bulb that seems common enough that surely someone in Canada must carry it, only to be stymied time and again.

So what’s the takeaway here? Well, at the least I hope you learned something about lightbulbs, but more-so this has been a selfish act of catharsis for me, putting to words something that’s been bothering me for years. In the past decade I’ve lost nearly a week of my life to googling lightbulbs and that’s not something I’m proud of. Perhaps I could just get less-great bulbs, but now I know enough to be bothered by their small underperformances. The healthiest option is likely to never buy lightbulbs again, to hand it off to someone else who can not stress the details quite so much, but same problems goes. So I guess the real takeaway is that it’s too late for me, but maybe you can use this to save yourself some trouble while still getting a good product, next time you have to buy a lightbulb yourself.

Thinking About Remote Work

In amongst all the tragedy and trauma of our current circumstance, I find some solace in the newfound solidarity formed and felt in our communities—individuals of many disparate circumstances coming together and lending their expertise to support and uplift each other. I have felt the growth of the online community of workers particularly acutely, as thousands of Canadians bring their daily routines onto the internet and transition to being remote workers. This is a field in which I have some expertise; I have spent the last six years working from a home office for various companies, and many of the newfound frustrations of my friends and family trying out remote work for the first time are very familiar to me. As such I’d like to offer a few thoughts, to both give better definition to the problems we face, and also to offer a few solutions wherever I have some experience.

Working from home, working distributed

Although an exhaustive analysis is outside the scope of this document, I think it’s constructive to consider our new work circumstances in two dimensions rather than lumping it all together under “remote work.” First, we are now working from home: this means dealing with distractions, new challenges to our work life balance, and other largely environmental complications to our processes. Secondly, we are working distributed, newly remote from our peers and collaborators: this comes with issues of communication and organization, and of staying on top of information that used to be at our literal physical fingertips. I think by considering our new circumstances through this lens we can move beyond listicles of tips and tricks, and instead provide a framework for more deeply understanding our challenges at hand and finding solutions tailored to our individual needs.

Working From Home

Although I’m a fan of and advocate for the many advantages of working from home (notably, no corporate policies against your cat sleeping on your desk), I’m more than happy to acknowledge that it also comes with its fair share of challenges and drawbacks. The loss of enforced boundaries is a common frustration for people newly transitioning to working from home, and somehow it can be simultaneously challenging to avoid the distractions and temptations of the home while also frustrating to find ways to disengage from work and not let it bleed into the rest of your life. Similarly, those new to home offices will find themselves suddenly deprived of the many enriching aspects of office life (as strange as that may sound) that exist at the margins of work and that provide essential social and physical variety to our lives. 

Boundaries of Work

Boundaries are going to come in a few different forms but the first, and most obvious, category to address is physical boundaries. Not that you have to build up blockades to keep out your family, but a clear definition of where you work and where you don’t will give you great mileage for both defining to others when you’re in “work mode” and not to be distracted, and also for training your brain to reflexively understand the same and help you find that “flow” that’s so much easier in a dedicated collocated office. Again, maybe you don’t have a home office, or maybe not even a desk, but that doesn’t mean you can’t carve out a dedicated space for thinking and working. Maybe between 9am and 5pm the kitchen table is strictly off limits to non-workers, or maybe for the duration of staying at home you only sit in your comfiest chair when you’re working.

And that nicely brings us to barriers around time—if the table stops being an office at 5pm, then you have to leave that office and go back to real life. It’s so easy to just keep going when you don’t have to physically leave work behind, but by all appearances this is going to be much more of a marathon than a sprint, and making sure that you’re not accidentally burning yourself out is going to be critical to your long term health. The inverse also applies —you should be very cautious about allowing yourself to take on other tasks during your work hours, like tidying up around the house or taking time for video games (not that you shouldn’t at all, just that you should be conservative and deliberate in your choices). If you’re already struggling to remain focused, giving yourself more chances to feel more ‘at home’ rather than ‘at work’ will further cement your mentality of not being singularly focused the way you would be in an office.

Finally, social boundaries play a critical role as well. Those who aren’t working from home—but are home with you—are likely not going to immediately appreciate how important and how hard it is to keep a clear(ish) work-life divide. That doesn’t mean you can’t plan and enjoy moments with the people around you, but if the people around you feel they have unfettered access to your person, it’s on you to have the willpower to say no to them each and every time, and pretty often it’ll be all too easy to say yes. Being clear and up front about the requirements of your job and the times when you’ll be occupied, prevents regular interrupts and distractions and makes the times that you do surprise them with an afternoon break all the more special.

Lost Margins

The other, subtler change you have to account for when you start working from home is the loss of the margins around your work that existed when you were in an office. There’s a significant portion of your day between the moment you leave for work to the time when you get home that isn’t occupied with what would traditionally be considered ‘work,’ and while much of that time is spent on things that are frustrating and unnecessary (I’m looking at you, commute), there are significant segments that are remarkably critical to our health and happiness. The first thing that you may notice is that you are significantly less active than you were before. Although office activities aren’t exactly a rigorous workout, the bustle of getting to your place of work, shuffling between meetings and events, and running to the bathroom that is for some reason on the opposite side of the building is a lot more active than rolling out of bed and grabbing your laptop (in my experience, the difference between about 3500 and 500 steps a day). If you’re not that active, like me, then you’ve just lost a meaningful percentage of your daily exercise (about 30% if we’re aiming for 10k steps and going by my numbers), and finding a way to replace that is important for staving off sluggishness and fatigue. If you struggle sticking to exercise regimes, I find replacing the lost activity with other ‘movement with purpose’ is easier to maintain; when I’m having a hard time with motivation I’ll allocate an hour in the afternoons for tidying or minor projects around the house. It keeps me moving when I’m not up for going for a run, and has the added bonus of making my home slightly less of a disaster.

Another significant loss in transitioning to working from home is the loss of social contact. You may not chat a lot with your colleagues, you may not even like most of them, but your day was likely still punctuated with casual social interactions that have just disappeared entirely. I’m sure a psychologist could get into a lot more detail about how social deprivation affects us and why humans just aren’t wired for it, but those little moments when you said hello as you rolled into the office, or chatted after a meeting, or paused to catch up in a hallway, those were important. It can be hard to understand at first why something feels off, but in time you’ll likely find just how much you appreciate what little casual social contact you have left. Fostering these connections, and making deliberate time for interactions that previously would have been incidental (I personally favour leaning into casual catch-ups during the start of calls, as we wait for everyone to get on and ready), is a valuable investment in your mental health and, as I’ll cover in the next section, an equally valuable investment in your productivity.

Working Distributed

Working in a distributed manner is likely something that most folks have at least some passing familiarity with, even if they haven’t had to be particularly thoughtful about it before. Any time you’re working with someone who isn’t in the seat right next to you, you’re dealing with some degree of distribution, although for most people this is generally with others with whom they are more professionally distanced, rather than close collaborators. The uniqueness of the current situation comes from the rapid displacement of previously collocated collaborators—people with whom you are expected to be tightly integrated are suddenly inaccessible in ways you likely haven’t had to cope with before. Bridging that gap—making your team as accessible as it is integrated—is fully possible (and the crux of successful distributed organizations), but the transition is extremely jarring when we have an established set of norms that rely on seeing each other in person on a daily basis.

Reestablishing Presence

If there’s one thing to focus on when trying to reconnect a team that’s been recently distributed, it’s re-establishing a sense of presence. An enormous component of the accessibility that exists between members of a collocated team is their immediate knowledge of their coworkers location and status. A quick glance can instantly establish whether a team mate is at work, and furthermore whether they’re readily interruptible, based on whether they’re wearing headphones, posture, and many other small cues that are obvious in person. That immediate understanding of the presence of one’s colleagues makes choosing when and how to engage with them about as low friction as possible, as opposed to when they’re on the other side of the internet and it may be hard to establish whether a coworker is even in “the office” that day or not. There are a myriad of ways to communicate presence remotely—status messages in your favourite chat app, announcing your arrival/departure in a public channel, and automated activity trackers that exist in many modern tools (like Slack or Teams)—be considerate of what works for you and your team and make sure that it is used widely and consistently.

Tools and Technology

There’s a whole separate blog post to be written on the best tooling for distributed work, so once again for the sake of brevity I’ll stick to the core concepts and personal highlights as best I can. When it comes to sharing information you’re generally going to have tooling that fits into three categories: synchronous communication (video/phone calls), asynchronous communication (chat/email), and information repositories (Jira/Google Docs); and broadly these map to three different modes of exchange: freeform exploration/debate, structured discussion, and the establishment of a referenceable library of information. The boundaries and definitions of each category are not firm, the study of communication and its mediums is deep, broad, and outside of the scope of this post, but understanding your toolset through that lens may help you better select which tool is most appropriate (just because that meeting could have been an email doesn’t mean it should have been). Similarly, this perspective can be used to maximize the ease of access to any given information source. To clarify with some examples, consider the barriers in play when an individual is looking to either video conference with a colleague, or extract documented knowledge from a google doc. To reduce friction hindering them from starting a conference call you may need to focus on ensuring that they know who to contact, that they are comfortable reaching out to the domain expert, and they know when the individual with the required knowledge is available. On the opposite end of the spectrum, to reduce friction when sourcing information from an information repository like a google doc, you should focus on good naming, searchability, and on ensuring that in your information library there exists only a single source of truth for each piece of information (or at least that duplicates don’t contradict each other). 

Although I’ve excluded matters of hardware entirely from this discussion so far (in short, a good mic, HD camera, and fast and hardwired internet valuable additions if available), I would like to make one specific recommendation. Given that you can’t always rely upon your coworkers to have high quality hardware of their own, I find a set of good headphones to be invaluable for making out the more muddled speakers on a call. The flip side of this of course is that headphones have the disadvantage of muffling your own voice, which many people find disconcerting and distracting while speaking. It turns out that you can have the best of both worlds—get a nice mic with aux out, and wire your headphones through that. This way the audio of your voice that the mic is picking up will be looped back to your ears sufficiently quickly that it’s almost indistinguishable from hearing yourself speak with no headphones on at all, and lets you hear people clearly without impediments to your own speech.

Keeping teams social

Perhaps the least obvious aspect of interconnectedness and accessibility of a team is the social component—the ways we reduce the friction in our teams’ communication by being friendly with each other. Reaching out to a friend, even if they’re less accessible in other ways, is almost always preferable to connecting with someone with whom your relationship has soured (or didn’t exist in the first place). Not that you should necessarily be trying to force a team to become best buddies, but it is crucial to recognize the value derived from the socialization amongst your members—specifically the ways in which it eases the processes of outreach and communication. We tend to implicitly acknowledge this with collocated groups in the form of team events, where we ostensibly build camaraderie. Certainly those are valuable, and their digital analog—the Zoom cocktail hour—can be useful as well, but there’s an enormous amount of invisible socialization that goes on in the office that we should also attempt to replicate for our distributed workers. Moments like walking to meetings together, chatting over coffee, and exchanging thoughts at a shared desk, tragically, have no obvious analogue. Instead we must engineer space for alternative activities. As I mentioned before I’m particularly partial to the start-of-meeting social which, despite initially feeling like a waste of time (at least to some) pays significant social dividends in time. That supplemented with a generous allocation of semi-social 1-1 meetings keeps me feeling connected with my organization socially, as well as professionally. And if you are the individual in charge of a team, I strongly recommend ramping up loosely structured meetings like standups and planning sessions where you can leave ample room for social margins to keep your team talking.

In a team that is tightly integrated, one of the primary goals of the teams’ organizers should be making the act of drawing upon the knowledge of a collaborator as low-friction as possible. By considering our tools, practices, and processes through that lens of accessibility, it is possible to make considerable improvements to our collective experience as distributed workers.

A Final Thought

As I mentioned before, there are many invaluable posts written by incredibly thoughtful people espousing theories of remote work, or providing lists of helpful advice for managing your new situation. The challenge then is not how to find advice, but to find advice that works for you—successful remote work is a deeply personal matter and there are few one-size-fits-all solutions. Instead, I hope that this post can provide some guidance to help you sift through your thoughts on being a remote worker and what works best for you when it comes to staying focused at home, and connected at work. 

Stay safe.


Tweetable Python

Sometimes when I have nothing better to do, or when I do have something better to do but I really don’t want to do it, I play code golf with snippets of my work – trying to compress my bits of logic down to something tiny enough that I can tweet it. They aren’t efficient, they’re certainly not pretty, but there’s some satisfaction in squeezing out every last character I can while still maintaining the original functionality. I tend to write my golfs as standalone functions, so even though there’s no sane reason to use them as they’re currently formatted, they’re fully functional so I can play with them later. This isn’t a particularly educational post, I just enjoy these as puzzles and I love to share games like this with people who enjoy the same types of problems.

Dictionary Inversion – 75

Dictionary inversion, at least in python, is a nice simple operation (particularly with the dictionary comprehensions of 2.7+). Still, a basic implementation (disregarding handling of duplicate values) is a good warmup for a problem I had to tackle a couple of years ago.

def a(d):return{v:k for k,v in d.items()}
# for 2.6ers, def a(d):return dict(v,k for k,v in d.items()

The actual task I was given was to take a dictionary of values to lists of values and reverse it such that each entry in the lists pointed to a list of keys that pointed to lists which contained that element. I’ve included an example below to clarify the requirements.

# Given a dictionary of the form
d = {1: [1,2], 2: [2,3]}
# The function should return a dictionary of the form
{1: [1], 2: [1,2], 3: [2]}

# Solution:
def a(d):return{i:[k for k in d if i in d[k]]for v in d.values()for i in v}

# Or, in a slightly more readable form:
def a(d):
[k for k in d if i in d[k]]
for v in d.values()for i in v

Pretty printer – 132

The goal here was to take a two-dimensional array of strings and print them neatly in a tab-delineated table. This implementation assumes an eight-space tab size, but is easily adaptable to other sizes.

My ‘pretty printer’ is one of the few old code golf examples where I still have some records of my previous iterations of compression. In the interest of sharing my process for code golfing, here are a few of the steps I still have copies of.

# So, given the following input
table = [
["Name", "Age", "City"],
["Bob", "23", "Detroit"],
["Angelina", "103", "San Francisco"]

# The function will print
# Name          Age     City
# Bob           23      Detroit
# Angelina      103     San Francisco

def pretty_printer(rows):
columns = zip(*rows)
max_widths = [max(len(value) for value in column) for column in columns]
for row in rows:
print '\t'.join(value + "\t" * (max_widths[col]/8 - len(value)/8) for col, value in enumerate(row))

The first iteration, as you can see, is entirely uncompressed. The variables are well named for clarity, and I even use a for-loop instead of a comprehension to keep things nice and simple. The only particularly interesting line is columns = zip(*rows), which splats the sequence of rows into zip(), which then combines all the first, second, third, etc., elements into new lists, giving us the columns instead.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8)for i, v in enumerate(r)) for r in t)

# Or, with line breaks
def p(t):
v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8)
for i, v in enumerate(r))
for r in t)

My first pass striped out all the nice naming, most of the whitespace, and compressed it down to one line by putting the variable value generation in where ‘columns’ and ‘max_widths’ used to be. Instead of printing line by line I also switched to using ‘\n’.join(), which combines a list of strings with line breaks between each.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(map(len,c))for c in zip(*t)][i]/8-len(v)/8)for i,v in enumerate(r))for r in t)

# Again, with some line breaks for readability
def p(t):
v+'\t'*([max(len(e)for e in c)for c in zip(*t)][i]/8-len(v)/8)
for i, v in enumerate(r))
for r in t)

I shaved off the final 10 characters by trimming out the whitespace I missed on the first pass, and replacing a comprehension with a map.

Not shown here are more than a few false starts that I remember trying but unfortunately didn’t write down. The difference between pretty good and excellent code golfing is generally just a few characters, so there’s a lot of experimentation that goes on where you do things like trying to see if a particular case works best as a list comprehension or a map operation, or whether it’s worth the overhead of creating an interim variable if you can avoid the character-cost of recomputation later.

Threaded Numbers – 83

I wrote this problem specifically for a code golf competition, so while it doesn’t have any practical use it makes for a good little puzzle in more than a few languages. I remember distinctly seeing a pretty impressive selection of approaches, even just within python (though there were plenty of interesting other language entries as well). If you’re looking to play around with code golfing, this would be one of the first I recommend you try out.

The object of the exercise is to open a file in the same directory as your program, which will contain a sequence of randomly ordered integers that are delineated by spaces. Your program should read the integers, and then output them in a very specific sorted format. The outputted list should also be space-delineated, but should be ordered such that the smallest number is the leftmost in the sequence, the second smallest is the rightmost, the third smallest the second leftmost, and so on.

# If the file 'in.txt' contains "1 3 5 8 2 9 4 6 8 10"
# the program should output 1 3 5 8 9 10 8 6 4 2

print' '.join(x[::2]+x[1::2][::-1])

And just for fun, here’s one of my other (less successful) attempts at the same problem. It relies on the trick of passing the same iterator twice to zip, so that items are pulled off the sorted list in pairs and dropped into two separate new sequences.

print' '.join(a[0]+a[1][::-1])

Python Tuple Tutorial – What they are and when to use them

My goal for today’s post is to give a deeper understanding of what tuples are and how those distinctions make them useful, saving their many novel applications involving packing/unpacking for a maybe-someday follow-up post so that I can address it a a psudo-distinct concept. Then, for those interested in what I’ve been writing, I hope to follow that up with tutorials on sequence types generally, lists specifically, usages of map, reduce and filter, and then the itertools module. Apparently my enthusiasm for sequences has a ways to go before it plays itself out. I make no promise of sticking to that schedule, but if I do I hope to release each of the above a slightly accelerated pace compared to my usual posting speed.

That all being said, for those of you who came here to learn (or better yet, teach me something!), I’ll get started on the tutorial now.

What They Are

Python tuples belong to the group of sequential data types, a subset of containers which is comprised of strings, Unicode strings, lists, tuples, bytearrays, buffers, and xrange objects. Tuples, like lists, can contain heterogenous ordered data; and all of the standard sequence operations can be applied to them. The primary difference between lists and tuples is that tuples are immutable, which means that once created, they cannot be altered. New tuples can be created from old tuples, and sub-tuples can be extracted, but once a tuple has been instantiated it is left utterly unchanged until the garbage collector cleans it up.

There are two very similar syntaxes for creating tuples, either just a comma separated list of values, or a comma separated list of values enclosed by technically optional parenthesis. I say ‘technically optional’ because to the very best of my understanding to exclude the parenthesis is uncommon and unpopular. I personally favour parenthesis as well, as I feel they keep visual consistency with lists while also clearly indicating that the created object is a tuple.

# Although you can do this:
no_parens_tuple = 1, 'b', 2

# Personally I lean towards 
parens_tuple = ('a', 2, 'c')

What I feel helps my case in favour of the usage of parens is that the syntax to create a single item tuple is the element with a comma after it, with or without parens. Using just the comma without the parens is easy to miss and makes the code easier to misunderstand. To further emphasize my point, a zero-element tuple can only be created by just a pair of parenthesis with nothing between them. Given that these two uses both make a strong case for using the parens (even as uncommon as they may be), for the sake of consistency they should be used for longer tuples as well.

readable_tuple = ('spectacular',)

ambiguous_tuple = 'spectacular',

empty_tuple = ()

Please note that the trailing comma is required for a single item tuple, even if parenthesis are included. Otherwise the parens are disregarded and the result is just the object intended for the tuple.

remembered = (10,)
forgot = (10)

The gathering of values into a tuple is called ‘tuple packing.’ The converse also exists, called ‘tuple unpacking,’ and I’ll delve into it in part two of this series. It’s a concept that deserves individual addressing.

Tuples have a couple more interesting (or challenging) characteristics that are born of their immutable nature. Firstly, they are hashable, which means (amongst other things) that they can be used as keys in dictionaries. Were you to try using a list as a key in a dict it would fail, and with good reason; as soon as the list was altered its hash value would change and it would point to a different or nonexistent bucket of items in the dictionary. To be honest, there aren’t many cases where hashing a tuple is going to be essential, but there’s a pretty healthy number where it’s neat, intuitive, or more pythonic than the alternative.

Functionally, it sometimes helps to think of tuples as a subset of lists; they forgo all the non-standard-sequence functions (such as append() and index assignment) but in exchange are a faster and more lightweight data type. If performance is a concern and it’s an option, tuples are the way to go. Before further addressing when and why to use tuples, I’d like to illustrate a little further the distinctions between mutable lists and immutable tuples below:

person_tuple = ('James Woods', 45, 'Probably Hollywood', 'Actor')
person_list = ['Luke Skywalker', 25, 'Far Far Away', 'Jedi Knight']

# Of course appending to a list works like you'd expect
person_list.append('Yoda's School of the Jedi Arts')

# But we can't do this with tuples
person_tuple.append('School of Hard Knocks')
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
AttributeError: 'tuple' object has no attribute 'append'

# Assigning new values doesn't work either
person_tuple[3] = 'Celebrity'
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
TypeError: 'tuple' object does not support item assignment

# Tuples can be used to derive new tuples:
slice_tuple = person_tuple[1:3] # Using slices
combined_tuple = person_tuple + person_tuple # Appending one to another
copied_tuple = person_tuple[:] # Duplicating tuple
tuple_val = person_tuple[0] # Index retrieval works as you'd expect


When To Use Them

Because lists and tuples are so similar, and lists grant additional flexibility, it’s easily to fall back on the argument that lists will do just fine, particularly because they’re so comfy and familiar to most people by the time they discover tuples. And to be honest, a great majority of the time using lists will be perfectly adequate, and if by the end of this tutorial you’re still confused by tuples and unsure of where they belong, go ahead and use that list, it’s okay. Chances are it’ll work just fine and no one will ever even think to question you. That being said, there are times that the advantages of tuples (such as their immutability, hashability, and performance gains) are significant enough to justify their usage. My goal here is to give you just a bit of a sense of when these cases might be.

You may note that many of my examples so far have contained heterogenous data constructed to represent a complex object (like a person). This is because unlike with lists we have a guaranteed contract of consistency with tuples, meaning that if I create a tuple with a string, a number, and then a bytearray, I can know with absolute certainty when I operate on that tuple later that the properties and their types will match the schema and values with which I created them earlier. Because there is no guarantee of order or data type with a list after it’s been created you lose that contract of consistency; and therefor I find that as a rule of thumb although lists techinically support heterogenous data, I try to keep them to sets of a single data type and use them in cases where I’m iterating over the elements and performing a uniform operation on all of them.

Tuples on the other hand are write-protected, almost like they have an implicit assert statement verifying that their data will be correctly formed. For this reason I use them typically for cases where I need a quick stand-in for an object, one that’s light-weight, easy reconfigurable at a later date, and not needed for use elsewhere. For example in a quick script to pull rows of data from a static source (database, csv, or other similar store) where you’re doing all the heavy lifting for one reason or another, it makes sense to handle rows as tuples because it wouldn’t make sense to modify, reorder, or resize those rows; it would create inconsistency of data that would be difficult to process. If our rows are tuples then when we use code like the following:

for row_num, name, email, first_pet, mothers_maiden_name in rows:
  print 'Processing %s (%d/%d)' % (name, row_num, len(rows))
  recover_password(email, first_pet, mothers_maiden_name)

We know our code won’t fail because of an accidentally extended row or a swapped name and email (unless of course the original data was corrupt, which is a whole different story). Python itself is written around this concept, that tuples have an implied semantic and an index has meaning. A great example of this is the return value of python’s database API specification, where fetchmany() and fetchall() return a list (ordered sequence of homogenous data) of tuples representing rows (hetergenous set where order has specific meaning). Cool, eh?

Another of tuples’ advantages needs less longwinded addressing, their improved performance, as it is relatively self explanatory. More lightweight than lists, dictionaries, or instantiating custom objects, creating and iterating over tuples is simply faster. Of course, this isn’t carte blanche, wrapping a list in a one-element tuple won’t speed up processing of that list, but for tuples that contain only hashable data types (simple and immutable types) there’s a pretty decent speed boost. Rarely will you find yourself writing python where the difference between list and tuples performance is a make or break choice, but if you’re dealing with large amounts of performance sensitive code and data, sticking to tuples can be one of many choices you make to make sure it all runs at top speed.

Finally, the hashability of tuples is another simple but neat little feature. This, like the performance gains, is particular to tuples that contain hashable data, but can be applied in some pretty fun places. Generally, it makes any situation where you have a couple of shared properties between objects that you would like to group and access them by a lot easier. For example, lat/long coordinate pairs for people or residencies, or month_of_year, day_of_month pairs make it easy to group date-tracked data while neatly handling that pesky leap-year problem.

lat_long_tup = (49, -117)
lat_long_list = [37, -122]

# If I were storing a bunch of house locations, it'd be easy to group them by lat/long coordinates.
# I do this by coordinates instead of by city because I used to live in a forest and no,
# Vancouver isn't a close enough approximation. It's eight hours away!
peoples_houses = {}
peoples_houses[lat_long_tup] = [my_old_house, my_old_neighbour]

#This will fail due to lists being unhashable
peoples_houses[lat_long_list] = [my_new_house, my_old_neighbour]
Traceback (most recent call last):
File '', line 1, in
TypeError: unhashable type: 'list'

Extra Reading

A shorter, better, and cooler description of tuples and how they compare to lists
An excellent tuple tutorial, and it’s illustrated!
Sequence types. The table in this section is an invaluable reference
Python tuples aren’t just constant lists

List Comprehensions for Fun and Profit

List comprehensions are awesome. They’re an incredibly intuitive and readable way to create, filter, join and otherwise modify lists. In the interest of helping young pyrates understand how they can use this simple and elegant tool to their advantage this tutorial will start with the very basics of what list comprehension are, and then gradually scale up to some of their cooler and more esoteric applications. If you’re already well acquainted with the concept, skip down a few sections for the more exciting stuff, or jump right to the end for the really juicy bits.

A list comprehension is a concise way of generating a list from an existing set or another iterable object. The syntax for the one of the most basic list comprehensions is as follows: animal_names = [animal.name for animal in list_of_animals] 

The idea, as I’m sure is evident or familiar to many of you, is that the new list will be the set of items as specified at the beginning of the list comprehension (before the ‘for’ clause), extracted from the list at the end of the comprehension (after the ‘in’ clause). The item in the middle, our ‘animal’ variable in the example above, is the variable that will represent each item in the list being processed, is scoped to the comprehension, and may be acted upon the same as any other object. The example above extracts a list of animal names from an original list of animals and could also be written as:

animal_names = []
for animal in list_of_animals:

Another popular example to introduce list comprehensions is the list of perfect squares of the numbers in a range. Easy enough to compute with a few lines of code but with list comprehensions we can do it in one and, I’d argue, make the statement even more readable:

squares = [x*x for x in range(15)]
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196]

Predicates and Generator Expressions

The first way we’re going to make list comprehensions more useful is by adding an if-statement to dictate the inclusion of terms extracted from the source list in the resulting list. This if-statement, known as the predicate, is added at the end of the comprehension, and maintains the almost english-like readable syntax. To continue our mathematical examples (list comprehensions in fact have their roots in traditional mathematic expressions), what follows is the set of numbers between zero and one hundred that are divisible by seven:

div_by_seven = [x for x in range(100) if x%7 == 0]
[0, 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98]

This allows us to easily filter out list items that are undesirable. (Note that the above example can also be accomplished with range(0, 100, 7). This example is illustrative only.)

I’d also like to touch on the fact that the value at the start of the list comprehension, the value that will be included in your final list, can be the result of an expression as well as a static value. Much like lambda functions there can only be a single expression at the start of your comprehension, which restricts you from writing large blocks of logic within the comprehension. I’m personally in favor of this limitation, as it enforces the terse and readable structure of the comprehension and prevents wild misuse. This functionality allows us to take advantage of other bits of python’s syntactic sugar such as ternary operators:

moods = ['sad' if person.is_alone() else 'happy' 
  for person in people_at_party]

For those of you unsure of what exactly expressions and statements are in python a brief (and only mostly accurate) definition would be that all expressions evaluate to a value, whereas statements are a superset which describe pretty much any piece of python code and therefor include the set of expressions. For quick reference I use the ‘does it look like it belongs on the righthand-side of an assignment’ check to evaluate whether a statement is an expression and therefor can be used in a comprehension. A few of the most common examples of these are variables, ternary operators, and function calls. For a better understanding of this concept I recommend checking out these links.

Nested and Chained Comprehensions

List comprehensions are very flexible, and can be extended in a number of ways. The first way, and perhaps the simplest, is by using nested comprehensions. Nested comprehensions are a single-line equivalent of nested for-loops (as you may have been able to infer), and as such extend the functionality we’ve established in exactly the way you’d expect. A neat little nested comprehension can be used to generate the identity matrix that will be familiar to anyone who’s dabbled in matrix algebra. For those who haven’t, this produces a 4×4 matrix of ones and zeroes such that the ones form a diagonal line from the top left to the bottom right:

ident_matrix = [[1 if col_index==row_index else 0 for row_index 
  in range(4)] for col_index in range(4)]
[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

You’ll note that the value generators in these two comprehensions are slightly more complex than what we’ve seen previously. For the inner expression we take advantage of python’s ternary syntax, while for the outer comprehension its generator expression is the nested comprehension, which is what gives us the list-of-lists structure that emulates the matrix.

While by nesting list comprehensions we can easily build lists of lists, we can deconstruct the same by using chained comprehensions. Chained list comprehensions similarly iterate over multiple lists just like nested for loops, but produce a single array instead of a multidimensional one. I recommend using chained comprehensions sparingly, as their syntax is somewhat less intuitive and they can quickly become un-pythonic in their complexity. Given that matrix we generated above, here’s a quick example of how we could flatten it back out:

flattened = [num for row in ident_matrix for num in row]
[1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]

In more traditional syntax, this could be written:

flattened = []
for row in ident_matrix:
  for num in row:

To make this a little more complex, let’s flatten the identity matrix and get only the zeros, ignoring the ones entirely.

flat_zeroes = [num for row in ident_matrix for num in row if num==0]
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Or how about only the zeroes from the even numbered rows?

even_zeroes = [num for row in ident_matrix if 
  ident_matrix.index(row)%2==0 for num in row if num==0]
[0, 0, 0, 0, 0, 0]

You can see how the awkward alternations between our assignments and our predicates quickly make our list comprehension much harder to read and follow. This is where list comprehensions begin to approach the limit of their usefulness. As much as it’s nice to have everything on one line, list comprehensions are meant as a convenience tool; if they’re making your code more complex and unreadable, they should be replaced by a more traditional structure. If your code looks indecipherable and fits on one line but could easily be replaced by a few more lines that could be written by a coder with only minimal experience with python, you’re trying too hard to impress someone.

# Perhaps overly simplified, but straightforward and unambiguous.
even_zeroes = []
for row in ident_matrix[::2]:
  for val in row:
    even_zeroes.append(val) if val == 0 else None

Dict and Set Comprehensions

Python 2.7 added dictionary comprehensions, a long sought after feature. Sadly, as much of my work is still done on projects that rely on python 2.4-6, these are not available to me. That being said, by cleverly combining list comprehensions and python’s built in dict() function we can create our own simple dictionary comprehensions in a pinch. The dict() function can construct a dictionary from a number of different inputs, but the one that’ll be the most useful to us is the list of tuples. Calling the following:

dict([('cat', 'meow'), ('dog', 'woof')])
{'dog': 'woof', 'cat': 'meow'}

Gives us a dictionary that maps the keys ‘cat’ and ‘dog’ to their respective animal noises. As I’m sure many of you can see this allows us to dynamically create such a list by using a similar expression within a comprehension like this:

inverted_dict = dict([(val, key) for key, value in 

Python 2.7+ dict comprehensions are semantically equivalent, but more intuitive and less expensive due to not having to create an intermediate list to pass to the dict constructor. To do the same as we did above in later versions of python, we can use:

inverted_dict = {val : key for key, val in old_dict.items()}

Much simpler, and much more pythonic.

Set comprehensions are also a new addition as of python 2.7, which rather than generating a list or dict, they create a set (only contain unique items). The syntax is similar to dict comprehensions (in that it uses the ‘{‘ symbol), but instead of giving it a pair of items, we provide just one. This can be emulated in python 2.6- by calling set() on the generated list.

unique_names = {name for name in all_names} 
# Is equivalent to
unique_names = set(all_names)

Generator Objects

A slight variation to list comprehensions is generator objects which are syntactically and functionally nearly identical, but rather than generating a list when the generator object is instantiated they pull from the source list when the generator object is iterated upon. This removes the need for intermediate storage and provides a not insignificant performance gain in many cases. Additionally, if the list being generated from is altered after the instantiation of the generator object but before iterating over the generator’s product then the change will be reflected in the resulting list. Finally, because we instantiate a generator instead of a list we can’t access the list members like we would be able to usually; instead the object is effectively opaque until it is iterated over and then we have access to only a single member at a time. Because this is an easy concept to get tripped up on I’m going to do my best to illustrate it below. Please note that the generator is instantiated by using ‘(…)’ instead of ‘[…]’ notation.

source = [1, 2] #list from which we'll generate 
comp_list = [num + 1 for num in source] #Stores complete result
gen_obj = (num + 1 for num in source) #Stores procedure for generating list

comp_list[0] #Regular list access
gen_obj[0] #Doesn't work
Traceback (most recent call last):
  File 'stdin', line 1, in module
TypeError: 'generator' object is unsubscriptable

source.append(3) #alter list

# Comprehension yields the product of unmodified list
for num in comp_list:
  print num

# Generator yields product of augmented list
for num in gen_obj:
  print num

Obviously, as with all things in life and coding, using list comprehensions vs generator objects is a tradeoff, in this case between flexibility and performance. It’s up to you to apply them judiciously.

Esoterica (Or, The Good, The Bad, and The Ugly)

Let’s start with the good:

almost_csv = [line.split('::') for line in open('my_report.csv')]

I’ve seen mixed opinions on this comprehension but I’ve done enough not-comma-delimited file handling to find this quick and dirty solution useful. Because files are closed when the corresponding file object is deallocated and the file object here is scoped to just the list comprehension, this neatly opens a file, splits it by lines, and then splits lines on ‘,’ before closing the file again. All in the space of just a few characters. Admittedly it means you’re loading the entire file into memory, but hey, sometimes that’s okay. Alternately, if you used a generator instead, you’d get the same concise syntax with non of the memory overhead! Please note that for any case where your delimiter is a single character, the csv module should be used.

Here are a couple more neat little perks, annotated with comments as needed:

# Tuple unpacking works in list comprehensions!
unpacked = ['%s + %s' % (a, b) for a, b in enumerate(range(100, 0, -1))]

# Comprehensions play well with python's built in functions
sorted_owners = sorted(getOwner(dog) for dog in dogs) 
# Note that the above creates a generator object and shares parenthesis with the sorted function

As for the bad, I’m going to unashamedly steal the form of this one from Chris Leary‘s blog post which you should definitely read if you want a deeper and more entertaining analysis of why you shouldn’t try this:

ord_lists = [2,3,4,5,3,4,23,24,25,26,45,46,9,13,14]
sub_seqs = []
[sub_seqs[-1].append(x) if sub_seqs and x == sub_seqs[-1][-1] 
  + 1 else sub_seqs.append([x]) for x in ord_lists]
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
[[2, 3, 4, 5], [3, 4], [23, 24, 25, 26], [45, 46], [9], [13, 14]]

While this works, and perhaps gets some points for being concise, it’s going strictly against the designated purpose of list comprehensions. List comprehensions are for list construction, not shorthand for loops. All this is doing is saving the developer the inconvenience of writing the proper loop syntax. Because append() does not return a value we’re generating a garbage list of Nones the length of the list that’s being modified; and more importantly any developer who reads this will be forced to re-comprehend the meaning of the code after realizing that the safe and familiar comprehensions that they’re used to are being bent to a completely foreign purpose. List comprehensions are nice, be nice to them too.

The ugly award goes to a list comprehension I saw that used a neat trick that I suppose could be super useful in certain situations, but if you find yourself using it you better have a pretty excellent argument for your case. Python lets you access a dictionary of all local variables, even the unnamed ones, and list comprehensions are assigned the designator ‘_[1]’. This gives us the pretty powerful ability to access and pass to functions the generated list as it’s being generated. For example, here are two comprehensions, the first of which uses a isPrime function that we define to generate the list of prime numbers from zero to fifty, and the latter of which is a concise way of getting the first N numbers in the Fibonacci sequence:

def isPrime(num, primes):
  for prime in primes:
    if prime == 0 or prime == 1:
    elif num % prime == 0:
      return False
  return True

[num for num in range(50) if isPrime(num, locals()['_[1]'])]
[0, 1, 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]

# Fibonacci Sequence
[locals()['_[1]'][-1] + locals()['_[1]'][-2] if 
  len(locals()['_[1]']) > 1 else 1 for i in range(15)]
[1, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]

However once again, this is pretty unreadable, and relies on an undocumented feature that has no guarantee of consistency or maintenance. Watch out.

Further Reading

PEP 202, List Comprehensions
A nice warning about abusing list comprehensions, from the desk of Jesse Noller
A guide to Python 3 Comprehensions


There were a couple of mistakes and a totally untested (and wrong) example. Thank you to all who pointed it out, and I’ll try to continually improve this tutorial as I learn more.

Bootstrap Popover/Tooltip Click Trigger


  1. Narration
  2. Breakdown
  3. Demo


Web development has it’s quirks. And by I mean that those of us like myself who are used to the simple and safe territory of a well defined, well documented, and most of all  consistent server-side programming language usually end up somewhere between daunted and devastated each time we subject ourselves to the tempestuous whims of front-end development. Like a peaceful Vault-dweller, each time I poke my head into the Capital Wasteland of browser-compatibility I either retreat in fear, or venture forth to discover that these badlands are just as much of a deathtrap as I believed them to be.

Unfortunately, my job requires on a daily basis for me to venture out into this dangerous wilderness and test my mettle (and much more so my patience) by working with that special trifecta of HTML, CSS, and Javascript. I also have a terrible memory, so I run into the same problems and reinvestigate them far more often than I should. So I’m making a start to writing down those problems, both for my reference and yours.

Let me begin by saying that I have a confession to make. I use Bootstrap. And I love it. I rely on it. It makes a lot of this terrible business a lot more bearable. When it goes wrong it makes me sad. Not just because I then actually have to do my own work, but because it’s like my close friend has suddenly grown a second, angrier head and has started hurling insults and rocks my way. In short, I feel betrayed.

Most recently this happened with the Twitter Bootstrap popover extension. Normally a lovely tool, I found all of a sudden it would refuse to appear when I attached it to an anchor tag and set it’s trigger event to ‘click’.

For some reason, the ‘click’ trigger maps to an event that requires the element to receive focus. Although this is just fine in Firefox (and IE!), in WebKit based browsers like Chrome and Safari certain tags cannot receive focus from the mouse if they don’t have a tabindex. Oddly enough, the anchor tag is given a place in the tab-order by default in all the browsers I’ve investigated so you can trigger onfocus events by tabbing through to the element, but not by clicking it. It took me a lot longer than it should to figure that out. If you just came here to get your popover working on click you can go home now. Just add “tabindex=’0′” to the elements that you want to have popovers or tooltips and it should work just fine. If you’re more curious about why this is, and how it works, I’ve got a couple more paragraphs for you.

The trouble is, we don’t really have a standard for this. The closest we have is the DOM Level 2 HTML specification that actually defines focus methods for only four tags; HTMLInputElement, HTMLSelectElement, HTMLTextAreaElement and HTMLAnchorElement. What is focusable and how (by keyboard navigation or by mouse clicking) is left somewhat open ended. Because of this, different browsers have chosen to allow focus events by default on a wide variety of elements.

This puts the tabindex attribute in the position of not only dictating the tab order, but also the focusability of elements. Once you get past the now apparent misnaming of the attribute, it’s easy to use in this manner, and should resolve further problems of focus quickly and easily (at least it has for me). For quick analysis of whether this issue is the source of your difficulties I highly recommend referencing this well organized table which details browser behaviour by element in regards to focus.



Boostrap popovers/tooltips using ‘click’ trigger aren’t working on certain elements in Chrome and Safari.

Root Cause

Many HTML elements can’t obtain focus (in some cases this only applies to gaining focus from a mouse click! -_-) in certain browsers by default.


If the you want to be able to give the elements focus by mouse, but exclude them from the tab-order (and thereby keyboard focussing), give them a tab index of -1.

If you would like the elements to maintain their tab-order give them a tabindex of 0, and they will be able to gain focus from both mouse and keyboard.

Further Reading

  1. DOM Level 2 HTML Specification
  2. Reference table of browser behaviour
  3. A better written, more technical, and more informative look at much the same topic