This post is reproduced from its original home on blog.e2.gg, where Evan and I share our thoughts on engineering, management, and remote work.

As I have (somewhat unexpectedly) been drawn into the world of management, one of my favourite parts of the job has been mentoring and uplifting then engineers in my organization. One of the most critical parts of that process is the fitting of incrementally more ambitious and open-ended projects to devs at each step of their learning curve, and the most important part of that, in terms of ensuring their success, is act of writing effective design documents for their projects.

This isn’t a post about why design docs are fundamentally critical to the design of good software (Joel Spolsky articulated that more effectively than I ever will, as have, honestly, many, many others) but it is worth noting that single most common root cause of failure that I’ve seen hamstring projects, particularly those run by more junior engineers, has been inadequately thorough design documents. This is the singular moment during which more senior engineers can scrutinize each element of the design, particularly the nasty and easy to miss failure cases, in the context of the whole (other than, of course, after the project launches and things start breaking). However, the exercise of communicating technical intent, complete and unambiguous, is a new and foreign challenge of its own – and even just getting started can be a daunting task to someone writing their first design doc.

So instead this is a post about what you need to get started writing your first (or your hundredth) design doc, in as brief and functional terms as is possible. With the following, I hope to provide all the structure and tooling you need to create something comprehensive for review from your peers, as well as offering a few tips and tricks to keep in mind as you fill out the format along the way.

Structure

There’s nothing more intimidating, as a writer, than a blank sheet of paper – where do I even start? To that end, for any informative documents I’m required to produce, I start with all the sections and headers I can identify, and turn the problem from “filling out the page” into (as much as is possible) “filling in the blanks.” Although the following may not quite be universal – you shouldn’t shy away from altering it to fit your domain – I find that generally starting with this format gives me enough structure to start breaking down the document into smaller problems to digest.

1. Overview
   1. Stakeholders/key personnel – Start off with who are you (author), who owns the thing you’re building (teams), and who cares about it (key stakeholders). Although this may be unnecessary in some smaller docs, for a broader audience it really helps the folks asking questions if they know who they’re talking to.
   2. Background – This is where you start the narrative. It’s always important to think about how the content of your document flows from section to section, each conveying unique information that supports the conclusions that follow, and this is where you can lay your groundwork for your readers’ comprehension of that narrative. Keep it short, this isn’t the time to justify all the specifics of your solution, but set the stage with the impetus for the feature, and any requisite technological context.
   3. Essential terminology – Although there’s a good chance you can skip this section, take a moment and consider – are there any terms that are specific to this domain that your readers are unlikely to know, or are you introducing novel terminology for the purposes of your solution? I’ve seen countless technical discussions spiral into frustration because speaking unambiguously in english is actually really hard, and anything you can do to preempt potential confusion will save you effort down the line.
   4. Objectives
      1. Goals – What are you going to accomplish?
      2. Non-goals – What are you not trying to accomplish? Don’t put world peace. Many projects have intuitable extensions, or adjacent problem domains that you may be pushed towards rolling into scope. Push back – be clear regarding your boundaries (in documentation as in life).
2. Technical Summary
   1. General Overview – Explain the logic and flow of your solution in two to three paragraphs, or if you’re feeling ambitious, five or six bullet points.
   2. System Diagrams – At least a broad system overview, with diagrams for key subsystems as necessary, with complimentary notes providing context as necessary.
   3. Key Components – Using a new AWS offering? Have a really complicated class heirarchy that’s central to the solution? Offering a new API to internal clients? Enumerate them here.
3. Technical Flows – This is a chance to explain how what you’re building will be used. Whether it’s a library, an API, or elsewise, understanding it through its flows rather than it’s atomic points of interaction is a much more informative lens for critiquing the design. Sequence diagrams are immensely useful here, and I’d strongly recommend getting familiar with them for anyone who writes client-facing code (more on them later).
4. Contracts/API – Okay, so now that we understand the flows, what exactly are our points of contact? The prior segment should have had most of these listed by name, and now that the reader understands how they’ll be used you can step through and give more detail on specific arguments, return values, and encapsulated business logic.
5. Persistence – Are you storing data? Great, now let’s get into the details. How is that data modelled? How is it indexed? Is it scalable, and is it flexible enough to support future use cases? Bad modelling at the persistence level often causes problems down the road – particularly for the second version of a feature as you expand beyond the tightly scoped initial problem domain, so making sure the constraints of your design are well understood at the outset can be enormously valuable.
6. Observability
   1. Metrics – What are you counting?
   2. Monitors – What are you analyzing?
   3. Alerts – What do you want to respond to?
7. Risks and Risk Mitigation – Where could this project go off the rails? What business outcomes or technology constraints might result in failure cases? What might extend the timeline beyond your current estimates for delivery?
8. Deployment/Rollout Strategy – “Turning it on” is another common point of failure for new features. Particularly if your project is a new/alternative flow or resource, ensuring that that any ramp up is smooth and well monitored is essential. How to manage deployments is the subject of another post, but at a minimum cover timelines, rollback plans, backwards compatibility, monitors to track during deployment, and metrics to be used as success criteria.
9. Appendix – At this point you’ve covered your bases! Long form references, or content that isn’t strictly necessary to the pitch but may be useful to a curious reader can be plugged into an appendix, making it clear that this isn’t part of the body of the proposal. You can drop basically anything in here, but a few likely entries follow:
   1. Alternatives – Are there other strategies that are still worth considering, or that you want to justify in their exclusion?
   2. Required Sign-offs – In some cases explicit sign off from a set of technical partners (for inter-service contributions, for example) may be useful to have on paper. Drop it in here!
   3. Follow-on projects – Where are you taking this project post-MVP?
   4. Specific scope exclusions – Any potential expansions on the non-Goals as listed above.
   5. Dependencies – Projects being executed in parallel, requirements of non technical partners, really anything that may block the project from proceeding.

Tools

Of course, not everything you’ll be sticking in that doc will raw text – it’s helpful to have diagrams and tables to present specific structured information, and so I’ve highlighted a few of the most lightweight and useful tools I’ve come to appreciate in my time producing design docs.

Authorship

Where do you actually write the doc? Honestly, “wherever you’re most comfortable” is a good place to start, and the first few hours should mostly be spent dumping as many ideas as possible out of your head and into their respective sections in preparation for organization and editing. That said, once you’ve got a semi-mature draft I strongly recommend finding a buddy, copying the content into a collaborative editor like google docs, and soliciting feedback as early as possible. You don’t have to have every potential question answered and every spelling eror fixed before you start getting criticism, and it’ll save you having to make larger changes when your document is more complete. Is this obvious? Sure. Have most people put off reaching out to peers in favour of adding a little more polish first, to their net detriment? Absolutely.

System diagrams

A high level system overview is near essential for any system describing more than one client and one server. Although you can get by with just Google Draw (or, for that matter, MS Paint), there are better solutions out there. Diagrams.net is free, easy to use (many of the tools in this space are far more complicated than necessary for the general case), and produces diagrams that aren’t an eyesore. If you don’t want to think about what to use here (and you shouldn’t, generally, that’s not the problem you’re trying to solve), just use this. Technically this site addresses almost every use case listed here, but due to my preference for text-based diagramming tools (over wysiwyg), I’ll include more specialized tools in the following sections.

Sequence diagrams

These are the bees knees, and criminally underused. Many technical specifications are most usefully understood through the the flow of data and the sequence of their use, but it can be hard to express these things in raw text without being so verbose as to obfuscate the key information. Sequence diagrams offer a flexible and fun way to render these flows, and in many cases can replace hundreds of words of text with a few easily understood images. I’ve been using websequencediagrams.com for years, and would strongly recommend them for their simplicity and clarity.

Entity Relationship Diagrams

Got a bunch of models, and those models are related to each other? Just like they hammered home in CS101, entity relationship diagrams are your friend. They’re simple and easy to read, so although they can be skipped over in more simply modelled domains, they’re an extremely efficient utility to have in your toolkit when necessary. Diagram.codes has a bevy of diagramming options, is well designed, and it’s ER syntax is particularly clear and simple.

API definitions

For long term documentation I think that APIs should almost always have their docs written inline with their method signatures and docs should be generated from there. However, before the API exists in code, tooling for expressing the contract you’ll be putting in place is a lot more sparse. Most popular options, like Swagger, are a little heavyweight for the brief and high-level definitions that might exist in a design doc, and I haven’t found any solutions that I’ve really loved. As such, I basically just copy and paste a rich-text version of this snippet, and now you can make your own too!

+--------------------+-----------------------------------------------------------------+
| Endpoint Name/Path                                                                   |
+--------------------+-----------------------------------------------------------------+
| Description        | A little about the endpoint, how it operates, and why it exists |
| Arguments:         | |Parameter| Type |            Description            |          |
|                    | | ------- | ---- | --------------------------------- |          |
|                    | |   id    |  int | the uuid of the referenced object |          |
| Response:          | Serialized response object                                      |
+--------------------+-----------------------------------------------------------------+

Techniques

So now you have a structure, a general sense of content, and a few handy tools for mocking up the diagrams that you may want to include – you’re pretty much good to go! Before you rush off though, I’d like to offer a few more points to consider as you draft your first (or fiftieth) design doc.

• Good headers help set expectations for readers and break up your document into easily parsable components. Good formatting seems like a nicety that many developers see as unnecessary frivolity, but self explanatory headers with good visual distinction between h1/h2/h3/etc. make a huge difference for reader comprehension.
• Don’t just cover the happy path. Failure modes are an essential component of a design doc, and warrant equal emphasis along with the success cases.
• The design doc should not be a product doc. Linking to the product doc near the top is a great idea, but retreading product/business outcomes, unless the project is strictly technical (and even then you could pull those out into another doc), is often counterproductive because it invites debate on a tangential subject, rather than focusing attention on the technical design.
• Emphasize complex elements, and unique/novel (within your context) elements, leave explanations of common infrastructure out or note them only briefly.
• This is a long doc structure. It can feel unduly burdensome for smaller projects, and even for larger projects it’s easy for this to feel like busywork once the author feels they’ve fully internalized the solution. Again, I don’t want to litigate that these docs are valuable (they are), but if you do feel like this is too much – do less. Something is better than nothing. Skipping some sections in the name of velocity or flexibility or just in the name of saving your sanity after a long week, can be totally fine so long as the doc still covers the core design elements of your solution. Edit the format to fit your needs and your patience, but remember the more thorough the doc, the more detailed feedback you’ll get from your peers, and the more value it will return to the project.

Finally, one last note before you go – this guide is intended to inform a formal proposal of a solution, which may be several docs deep in the design process already. You might write out a complete list of approaches to be debated, early on after identifying a problem space; you might throw together a one-pager outlining your preferred solution when you’re first seeking approval from your peers or boss to explore the direction; those are good an important, but should not be as exhaustive as the document outlined above. A design doc should assume an opinionated position regarding the correct solution, and justify that opinion through its thorough treatment of the problem space. If you’re not ready to tackle the design doc yet, that’s okay! Seek advice from peers or mentors, do a lightweight comparison of alternative solutions, and make clear to your stakeholders that you need a little more time – this is a big commitment! But when you are ready, I hope this guide helps make your experience of writing your design doc just a little bit easier.

def p(t):
print’\n’.join(‘\t’.join(v+’\t’*([max(len(e)for e in c)for c in zip(*t)][i]/8-len(v)/8)for i, v in enumerate(r))for r in t) #golf

— Evan Brynne (@evanbrynne) December 17, 2013

Sometimes when I have nothing better to do, or when I do have something better to do but I really don’t want to do it, I play code golf with snippets of my work – trying to compress my bits of logic down to something tiny enough that I can tweet it. They aren’t efficient, they’re certainly not pretty, but there’s some satisfaction in squeezing out every last character I can while still maintaining the original functionality. I tend to write my golfs as standalone functions, so even though there’s no sane reason to use them as they’re currently formatted, they’re fully functional so I can play with them later. This isn’t a particularly educational post, I just enjoy these as puzzles and I love to share games like this with people who enjoy the same types of problems.

Dictionary Inversion – 75

Dictionary inversion, at least in python, is a nice simple operation (particularly with the dictionary comprehensions of 2.7+). Still, a basic implementation (disregarding handling of duplicate values) is a good warmup for a problem I had to tackle a couple of years ago.

def a(d):return{v:k for k,v in d.items()}
# for 2.6ers, def a(d):return dict(v,k for k,v in d.items()

The actual task I was given was to take a dictionary of values to lists of values and reverse it such that each entry in the lists pointed to a list of keys that pointed to lists which contained that element. I’ve included an example below to clarify the requirements.

# Given a dictionary of the form
d = {1: [1,2], 2: [2,3]}
# The function should return a dictionary of the form
{1: [1], 2: [1,2], 3: [2]}

# Solution:
def a(d):return{i:[k for k in d if i in d[k]]for v in d.values()for i in v}

# Or, in a slightly more readable form:
def a(d):
return{i:
[k for k in d if i in d[k]]
for v in d.values()for i in v
}

Pretty printer – 132

The goal here was to take a two-dimensional array of strings and print them neatly in a tab-delineated table. This implementation assumes an eight-space tab size, but is easily adaptable to other sizes.

My ‘pretty printer’ is one of the few old code golf examples where I still have some records of my previous iterations of compression. In the interest of sharing my process for code golfing, here are a few of the steps I still have copies of.

# So, given the following input
table = [
["Name", "Age", "City"],
["Bob", "23", "Detroit"],
["Angelina", "103", "San Francisco"]
]

# The function will print
# Name          Age     City
# Bob           23      Detroit
# Angelina      103     San Francisco

def pretty_printer(rows):
columns = zip(*rows)
max_widths = [max(len(value) for value in column) for column in columns]
for row in rows:
print '\t'.join(value + "\t" * (max_widths[col]/8 - len(value)/8) for col, value in enumerate(row))

The first iteration, as you can see, is entirely uncompressed. The variables are well named for clarity, and I even use a for-loop instead of a comprehension to keep things nice and simple. The only particularly interesting line is columns = zip(*rows), which splats the sequence of rows into zip(), which then combines all the first, second, third, etc., elements into new lists, giving us the columns instead.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8)for i, v in enumerate(r)) for r in t)

# Or, with line breaks
def p(t):
print'\n'.join(
'\t'.join(
v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8)
for i, v in enumerate(r))
for r in t)

My first pass striped out all the nice naming, most of the whitespace, and compressed it down to one line by putting the variable value generation in where ‘columns’ and ‘max_widths’ used to be. Instead of printing line by line I also switched to using ‘\n’.join(), which combines a list of strings with line breaks between each.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(map(len,c))for c in zip(*t)][i]/8-len(v)/8)for i,v in enumerate(r))for r in t)

# Again, with some line breaks for readability
def p(t):
print'\n'.join(
'\t'.join(
v+'\t'*([max(len(e)for e in c)for c in zip(*t)][i]/8-len(v)/8)
for i, v in enumerate(r))
for r in t)

I shaved off the final 10 characters by trimming out the whitespace I missed on the first pass, and replacing a comprehension with a map.

Not shown here are more than a few false starts that I remember trying but unfortunately didn’t write down. The difference between pretty good and excellent code golfing is generally just a few characters, so there’s a lot of experimentation that goes on where you do things like trying to see if a particular case works best as a list comprehension or a map operation, or whether it’s worth the overhead of creating an interim variable if you can avoid the character-cost of recomputation later.

Threaded Numbers – 83

I wrote this problem specifically for a code golf competition, so while it doesn’t have any practical use it makes for a good little puzzle in more than a few languages. I remember distinctly seeing a pretty impressive selection of approaches, even just within python (though there were plenty of interesting other language entries as well). If you’re looking to play around with code golfing, this would be one of the first I recommend you try out.

The object of the exercise is to open a file in the same directory as your program, which will contain a sequence of randomly ordered integers that are delineated by spaces. Your program should read the integers, and then output them in a very specific sorted format. The outputted list should also be space-delineated, but should be ordered such that the smallest number is the leftmost in the sequence, the second smallest is the rightmost, the third smallest the second leftmost, and so on.

# If the file 'in.txt' contains "1 3 5 8 2 9 4 6 8 10"
# the program should output 1 3 5 8 9 10 8 6 4 2

x=sorted(open('in.txt').read().split(),key=int)
print' '.join(x[::2]+x[1::2][::-1])

And just for fun, here’s one of my other (less successful) attempts at the same problem. It relies on the trick of passing the same iterator twice to zip, so that items are pulled off the sorted list in pairs and dropped into two separate new sequences.

x=iter(sorted(open('in.txt').read().split(),key=int))
a=zip(*zip(x,x))
print' '.join(a[0]+a[1][::-1])

My goal for today’s post is to give a deeper understanding of what tuples are and how those distinctions make them useful, saving their many novel applications involving packing/unpacking for a maybe-someday follow-up post so that I can address it a a psudo-distinct concept. Then, for those interested in what I’ve been writing, I hope to follow that up with tutorials on sequence types generally, lists specifically, usages of map, reduce and filter, and then the itertools module. Apparently my enthusiasm for sequences has a ways to go before it plays itself out. I make no promise of sticking to that schedule, but if I do I hope to release each of the above a slightly accelerated pace compared to my usual posting speed.

That all being said, for those of you who came here to learn (or better yet, teach me something!), I’ll get started on the tutorial now.

What They Are

Python tuples belong to the group of sequential data types, a subset of containers which is comprised of strings, Unicode strings, lists, tuples, bytearrays, buffers, and xrange objects. Tuples, like lists, can contain heterogenous ordered data; and all of the standard sequence operations can be applied to them. The primary difference between lists and tuples is that tuples are immutable, which means that once created, they cannot be altered. New tuples can be created from old tuples, and sub-tuples can be extracted, but once a tuple has been instantiated it is left utterly unchanged until the garbage collector cleans it up.

There are two very similar syntaxes for creating tuples, either just a comma separated list of values, or a comma separated list of values enclosed by technically optional parenthesis. I say ‘technically optional’ because to the very best of my understanding to exclude the parenthesis is uncommon and unpopular. I personally favour parenthesis as well, as I feel they keep visual consistency with lists while also clearly indicating that the created object is a tuple.

# Although you can do this:
no_parens_tuple = 1, 'b', 2

# Personally I lean towards 
parens_tuple = ('a', 2, 'c')

What I feel helps my case in favour of the usage of parens is that the syntax to create a single item tuple is the element with a comma after it, with or without parens. Using just the comma without the parens is easy to miss and makes the code easier to misunderstand. To further emphasize my point, a zero-element tuple can only be created by just a pair of parenthesis with nothing between them. Given that these two uses both make a strong case for using the parens (even as uncommon as they may be), for the sake of consistency they should be used for longer tuples as well.

readable_tuple = ('spectacular',)

ambiguous_tuple = 'spectacular',

empty_tuple = ()

Please note that the trailing comma is required for a single item tuple, even if parenthesis are included. Otherwise the parens are disregarded and the result is just the object intended for the tuple.

remembered = (10,)
remembered
(10,)
 
forgot = (10)
forgot
10

The gathering of values into a tuple is called ‘tuple packing.’ The converse also exists, called ‘tuple unpacking,’ and I’ll delve into it in part two of this series. It’s a concept that deserves individual addressing.

Tuples have a couple more interesting (or challenging) characteristics that are born of their immutable nature. Firstly, they are hashable, which means (amongst other things) that they can be used as keys in dictionaries. Were you to try using a list as a key in a dict it would fail, and with good reason; as soon as the list was altered its hash value would change and it would point to a different or nonexistent bucket of items in the dictionary. To be honest, there aren’t many cases where hashing a tuple is going to be essential, but there’s a pretty healthy number where it’s neat, intuitive, or more pythonic than the alternative.

Functionally, it sometimes helps to think of tuples as a subset of lists; they forgo all the non-standard-sequence functions (such as append() and index assignment) but in exchange are a faster and more lightweight data type. If performance is a concern and it’s an option, tuples are the way to go. Before further addressing when and why to use tuples, I’d like to illustrate a little further the distinctions between mutable lists and immutable tuples below:

person_tuple = ('James Woods', 45, 'Probably Hollywood', 'Actor')
person_list = ['Luke Skywalker', 25, 'Far Far Away', 'Jedi Knight']

# Of course appending to a list works like you'd expect
person_list.append('Yoda's School of the Jedi Arts')

# But we can't do this with tuples
person_tuple.append('School of Hard Knocks')
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
AttributeError: 'tuple' object has no attribute 'append'

# Assigning new values doesn't work either
person_tuple[3] = 'Celebrity'
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
TypeError: 'tuple' object does not support item assignment

# Tuples can be used to derive new tuples:
slice_tuple = person_tuple[1:3] # Using slices
combined_tuple = person_tuple + person_tuple # Appending one to another
copied_tuple = person_tuple[:] # Duplicating tuple
tuple_val = person_tuple[0] # Index retrieval works as you'd expect

When To Use Them

Because lists and tuples are so similar, and lists grant additional flexibility, it’s easily to fall back on the argument that lists will do just fine, particularly because they’re so comfy and familiar to most people by the time they discover tuples. And to be honest, a great majority of the time using lists will be perfectly adequate, and if by the end of this tutorial you’re still confused by tuples and unsure of where they belong, go ahead and use that list, it’s okay. Chances are it’ll work just fine and no one will ever even think to question you. That being said, there are times that the advantages of tuples (such as their immutability, hashability, and performance gains) are significant enough to justify their usage. My goal here is to give you just a bit of a sense of when these cases might be.

You may note that many of my examples so far have contained heterogenous data constructed to represent a complex object (like a person). This is because unlike with lists we have a guaranteed contract of consistency with tuples, meaning that if I create a tuple with a string, a number, and then a bytearray, I can know with absolute certainty when I operate on that tuple later that the properties and their types will match the schema and values with which I created them earlier. Because there is no guarantee of order or data type with a list after it’s been created you lose that contract of consistency; and therefor I find that as a rule of thumb although lists techinically support heterogenous data, I try to keep them to sets of a single data type and use them in cases where I’m iterating over the elements and performing a uniform operation on all of them.

Tuples on the other hand are write-protected, almost like they have an implicit assert statement verifying that their data will be correctly formed. For this reason I use them typically for cases where I need a quick stand-in for an object, one that’s light-weight, easy reconfigurable at a later date, and not needed for use elsewhere. For example in a quick script to pull rows of data from a static source (database, csv, or other similar store) where you’re doing all the heavy lifting for one reason or another, it makes sense to handle rows as tuples because it wouldn’t make sense to modify, reorder, or resize those rows; it would create inconsistency of data that would be difficult to process. If our rows are tuples then when we use code like the following:

for row_num, name, email, first_pet, mothers_maiden_name in rows:
  print 'Processing %s (%d/%d)' % (name, row_num, len(rows))
  recover_password(email, first_pet, mothers_maiden_name)

We know our code won’t fail because of an accidentally extended row or a swapped name and email (unless of course the original data was corrupt, which is a whole different story). Python itself is written around this concept, that tuples have an implied semantic and an index has meaning. A great example of this is the return value of python’s database API specification, where fetchmany() and fetchall() return a list (ordered sequence of homogenous data) of tuples representing rows (hetergenous set where order has specific meaning). Cool, eh?

Another of tuples’ advantages needs less longwinded addressing, their improved performance, as it is relatively self explanatory. More lightweight than lists, dictionaries, or instantiating custom objects, creating and iterating over tuples is simply faster. Of course, this isn’t carte blanche, wrapping a list in a one-element tuple won’t speed up processing of that list, but for tuples that contain only hashable data types (simple and immutable types) there’s a pretty decent speed boost. Rarely will you find yourself writing python where the difference between list and tuples performance is a make or break choice, but if you’re dealing with large amounts of performance sensitive code and data, sticking to tuples can be one of many choices you make to make sure it all runs at top speed.

Finally, the hashability of tuples is another simple but neat little feature. This, like the performance gains, is particular to tuples that contain hashable data, but can be applied in some pretty fun places. Generally, it makes any situation where you have a couple of shared properties between objects that you would like to group and access them by a lot easier. For example, lat/long coordinate pairs for people or residencies, or month_of_year, day_of_month pairs make it easy to group date-tracked data while neatly handling that pesky leap-year problem.

lat_long_tup = (49, -117)
lat_long_list = [37, -122]

# If I were storing a bunch of house locations, it'd be easy to group them by lat/long coordinates.
# I do this by coordinates instead of by city because I used to live in a forest and no,
# Vancouver isn't a close enough approximation. It's eight hours away!
peoples_houses = {}
peoples_houses[lat_long_tup] = [my_old_house, my_old_neighbour]

#This will fail due to lists being unhashable
peoples_houses[lat_long_list] = [my_new_house, my_old_neighbour]
Traceback (most recent call last):
File '', line 1, in
TypeError: unhashable type: 'list'

Extra Reading

A shorter, better, and cooler description of tuples and how they compare to lists
An excellent tuple tutorial, and it’s illustrated!
Sequence types. The table in this section is an invaluable reference
Python tuples aren’t just constant lists

List comprehensions are awesome. They’re an incredibly intuitive and readable way to create, filter, join and otherwise modify lists. In the interest of helping young pyrates understand how they can use this simple and elegant tool to their advantage this tutorial will start with the very basics of what list comprehension are, and then gradually scale up to some of their cooler and more esoteric applications. If you’re already well acquainted with the concept, skip down a few sections for the more exciting stuff, or jump right to the end for the really juicy bits.

A list comprehension is a concise way of generating a list from an existing set or another iterable object. The syntax for the one of the most basic list comprehensions is as follows: animal_names = [animal.name for animal in list_of_animals] 

The idea, as I’m sure is evident or familiar to many of you, is that the new list will be the set of items as specified at the beginning of the list comprehension (before the ‘for’ clause), extracted from the list at the end of the comprehension (after the ‘in’ clause). The item in the middle, our ‘animal’ variable in the example above, is the variable that will represent each item in the list being processed, is scoped to the comprehension, and may be acted upon the same as any other object. The example above extracts a list of animal names from an original list of animals and could also be written as:

animal_names = []
for animal in list_of_animals:
  animal_names.append(animal.name)

Another popular example to introduce list comprehensions is the list of perfect squares of the numbers in a range. Easy enough to compute with a few lines of code but with list comprehensions we can do it in one and, I’d argue, make the statement even more readable:

squares = [x*x for x in range(15)]
squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196]

Predicates and Generator Expressions

The first way we’re going to make list comprehensions more useful is by adding an if-statement to dictate the inclusion of terms extracted from the source list in the resulting list. This if-statement, known as the predicate, is added at the end of the comprehension, and maintains the almost english-like readable syntax. To continue our mathematical examples (list comprehensions in fact have their roots in traditional mathematic expressions), what follows is the set of numbers between zero and one hundred that are divisible by seven:

div_by_seven = [x for x in range(100) if x%7 == 0]
div_by_seven
[0, 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98]

This allows us to easily filter out list items that are undesirable. (Note that the above example can also be accomplished with range(0, 100, 7). This example is illustrative only.)

I’d also like to touch on the fact that the value at the start of the list comprehension, the value that will be included in your final list, can be the result of an expression as well as a static value. Much like lambda functions there can only be a single expression at the start of your comprehension, which restricts you from writing large blocks of logic within the comprehension. I’m personally in favor of this limitation, as it enforces the terse and readable structure of the comprehension and prevents wild misuse. This functionality allows us to take advantage of other bits of python’s syntactic sugar such as ternary operators:

moods = ['sad' if person.is_alone() else 'happy' 
  for person in people_at_party]

For those of you unsure of what exactly expressions and statements are in python a brief (and only mostly accurate) definition would be that all expressions evaluate to a value, whereas statements are a superset which describe pretty much any piece of python code and therefor include the set of expressions. For quick reference I use the ‘does it look like it belongs on the righthand-side of an assignment’ check to evaluate whether a statement is an expression and therefor can be used in a comprehension. A few of the most common examples of these are variables, ternary operators, and function calls. For a better understanding of this concept I recommend checking out these links.

Nested and Chained Comprehensions

List comprehensions are very flexible, and can be extended in a number of ways. The first way, and perhaps the simplest, is by using nested comprehensions. Nested comprehensions are a single-line equivalent of nested for-loops (as you may have been able to infer), and as such extend the functionality we’ve established in exactly the way you’d expect. A neat little nested comprehension can be used to generate the identity matrix that will be familiar to anyone who’s dabbled in matrix algebra. For those who haven’t, this produces a 4×4 matrix of ones and zeroes such that the ones form a diagonal line from the top left to the bottom right:

ident_matrix = [[1 if col_index==row_index else 0 for row_index 
  in range(4)] for col_index in range(4)]
ident_matrix
[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

You’ll note that the value generators in these two comprehensions are slightly more complex than what we’ve seen previously. For the inner expression we take advantage of python’s ternary syntax, while for the outer comprehension its generator expression is the nested comprehension, which is what gives us the list-of-lists structure that emulates the matrix.

While by nesting list comprehensions we can easily build lists of lists, we can deconstruct the same by using chained comprehensions. Chained list comprehensions similarly iterate over multiple lists just like nested for loops, but produce a single array instead of a multidimensional one. I recommend using chained comprehensions sparingly, as their syntax is somewhat less intuitive and they can quickly become un-pythonic in their complexity. Given that matrix we generated above, here’s a quick example of how we could flatten it back out:

flattened = [num for row in ident_matrix for num in row]
flattened
[1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]

In more traditional syntax, this could be written:

flattened = []
for row in ident_matrix:
  for num in row:
    flattened.append(num)

To make this a little more complex, let’s flatten the identity matrix and get only the zeros, ignoring the ones entirely.

flat_zeroes = [num for row in ident_matrix for num in row if num==0]
flat_zeroes
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Or how about only the zeroes from the even numbered rows?

even_zeroes = [num for row in ident_matrix if 
  ident_matrix.index(row)%2==0 for num in row if num==0]
even_zeroes
[0, 0, 0, 0, 0, 0]

You can see how the awkward alternations between our assignments and our predicates quickly make our list comprehension much harder to read and follow. This is where list comprehensions begin to approach the limit of their usefulness. As much as it’s nice to have everything on one line, list comprehensions are meant as a convenience tool; if they’re making your code more complex and unreadable, they should be replaced by a more traditional structure. If your code looks indecipherable and fits on one line but could easily be replaced by a few more lines that could be written by a coder with only minimal experience with python, you’re trying too hard to impress someone.

# Perhaps overly simplified, but straightforward and unambiguous.
even_zeroes = []
for row in ident_matrix[::2]:
  for val in row:
    even_zeroes.append(val) if val == 0 else None

Dict and Set Comprehensions

Python 2.7 added dictionary comprehensions, a long sought after feature. Sadly, as much of my work is still done on projects that rely on python 2.4-6, these are not available to me. That being said, by cleverly combining list comprehensions and python’s built in dict() function we can create our own simple dictionary comprehensions in a pinch. The dict() function can construct a dictionary from a number of different inputs, but the one that’ll be the most useful to us is the list of tuples. Calling the following:

dict([('cat', 'meow'), ('dog', 'woof')])
{'dog': 'woof', 'cat': 'meow'}

Gives us a dictionary that maps the keys ‘cat’ and ‘dog’ to their respective animal noises. As I’m sure many of you can see this allows us to dynamically create such a list by using a similar expression within a comprehension like this:

inverted_dict = dict([(val, key) for key, value in 
   old_dict.items()])

Python 2.7+ dict comprehensions are semantically equivalent, but more intuitive and less expensive due to not having to create an intermediate list to pass to the dict constructor. To do the same as we did above in later versions of python, we can use:

inverted_dict = {val : key for key, val in old_dict.items()}

Much simpler, and much more pythonic.

Set comprehensions are also a new addition as of python 2.7, which rather than generating a list or dict, they create a set (only contain unique items). The syntax is similar to dict comprehensions (in that it uses the ‘{‘ symbol), but instead of giving it a pair of items, we provide just one. This can be emulated in python 2.6- by calling set() on the generated list.

unique_names = {name for name in all_names} 
# Is equivalent to
unique_names = set(all_names)

Generator Objects

A slight variation to list comprehensions is generator objects which are syntactically and functionally nearly identical, but rather than generating a list when the generator object is instantiated they pull from the source list when the generator object is iterated upon. This removes the need for intermediate storage and provides a not insignificant performance gain in many cases. Additionally, if the list being generated from is altered after the instantiation of the generator object but before iterating over the generator’s product then the change will be reflected in the resulting list. Finally, because we instantiate a generator instead of a list we can’t access the list members like we would be able to usually; instead the object is effectively opaque until it is iterated over and then we have access to only a single member at a time. Because this is an easy concept to get tripped up on I’m going to do my best to illustrate it below. Please note that the generator is instantiated by using ‘(…)’ instead of ‘[…]’ notation.

source = [1, 2] #list from which we'll generate 
comp_list = [num + 1 for num in source] #Stores complete result
gen_obj = (num + 1 for num in source) #Stores procedure for generating list

comp_list[0] #Regular list access
2
gen_obj[0] #Doesn't work
Traceback (most recent call last):
  File 'stdin', line 1, in module
TypeError: 'generator' object is unsubscriptable

source.append(3) #alter list

# Comprehension yields the product of unmodified list
for num in comp_list:
  print num
2
3

# Generator yields product of augmented list
for num in gen_obj:
  print num
2
3
4

Obviously, as with all things in life and coding, using list comprehensions vs generator objects is a tradeoff, in this case between flexibility and performance. It’s up to you to apply them judiciously.

Esoterica (Or, The Good, The Bad, and The Ugly)

Let’s start with the good:

almost_csv = [line.split('::') for line in open('my_report.csv')]

I’ve seen mixed opinions on this comprehension but I’ve done enough not-comma-delimited file handling to find this quick and dirty solution useful. Because files are closed when the corresponding file object is deallocated and the file object here is scoped to just the list comprehension, this neatly opens a file, splits it by lines, and then splits lines on ‘,’ before closing the file again. All in the space of just a few characters. Admittedly it means you’re loading the entire file into memory, but hey, sometimes that’s okay. Alternately, if you used a generator instead, you’d get the same concise syntax with non of the memory overhead! Please note that for any case where your delimiter is a single character, the csv module should be used.

Here are a couple more neat little perks, annotated with comments as needed:

# Tuple unpacking works in list comprehensions!
unpacked = ['%s + %s' % (a, b) for a, b in enumerate(range(100, 0, -1))]

# Comprehensions play well with python's built in functions
sorted_owners = sorted(getOwner(dog) for dog in dogs) 
# Note that the above creates a generator object and shares parenthesis with the sorted function

As for the bad, I’m going to unashamedly steal the form of this one from Chris Leary‘s blog post which you should definitely read if you want a deeper and more entertaining analysis of why you shouldn’t try this:

ord_lists = [2,3,4,5,3,4,23,24,25,26,45,46,9,13,14]
sub_seqs = []
[sub_seqs[-1].append(x) if sub_seqs and x == sub_seqs[-1][-1] 
  + 1 else sub_seqs.append([x]) for x in ord_lists]
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
sub_seqs
[[2, 3, 4, 5], [3, 4], [23, 24, 25, 26], [45, 46], [9], [13, 14]]

While this works, and perhaps gets some points for being concise, it’s going strictly against the designated purpose of list comprehensions. List comprehensions are for list construction, not shorthand for loops. All this is doing is saving the developer the inconvenience of writing the proper loop syntax. Because append() does not return a value we’re generating a garbage list of Nones the length of the list that’s being modified; and more importantly any developer who reads this will be forced to re-comprehend the meaning of the code after realizing that the safe and familiar comprehensions that they’re used to are being bent to a completely foreign purpose. List comprehensions are nice, be nice to them too.

The ugly award goes to a list comprehension I saw that used a neat trick that I suppose could be super useful in certain situations, but if you find yourself using it you better have a pretty excellent argument for your case. Python lets you access a dictionary of all local variables, even the unnamed ones, and list comprehensions are assigned the designator ‘_[1]’. This gives us the pretty powerful ability to access and pass to functions the generated list as it’s being generated. For example, here are two comprehensions, the first of which uses a isPrime function that we define to generate the list of prime numbers from zero to fifty, and the latter of which is a concise way of getting the first N numbers in the Fibonacci sequence:

def isPrime(num, primes):
  for prime in primes:
    if prime == 0 or prime == 1:
      pass
    elif num % prime == 0:
      return False
  return True

[num for num in range(50) if isPrime(num, locals()['_[1]'])]
[0, 1, 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]

# Fibonacci Sequence
[locals()['_[1]'][-1] + locals()['_[1]'][-2] if 
  len(locals()['_[1]']) > 1 else 1 for i in range(15)]
[1, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]

However once again, this is pretty unreadable, and relies on an undocumented feature that has no guarantee of consistency or maintenance. Watch out.

Further Reading

PEP 202, List Comprehensions
A nice warning about abusing list comprehensions, from the desk of Jesse Noller
A guide to Python 3 Comprehensions

Edits

There were a couple of mistakes and a totally untested (and wrong) example. Thank you to all who pointed it out, and I’ll try to continually improve this tutorial as I learn more.

Navigation

1. Narration
2. Breakdown
3. Demo

Narration

Web development has it’s quirks. And by I mean that those of us like myself who are used to the simple and safe territory of a well defined, well documented, and most of all  consistent server-side programming language usually end up somewhere between daunted and devastated each time we subject ourselves to the tempestuous whims of front-end development. Like a peaceful Vault-dweller, each time I poke my head into the Capital Wasteland of browser-compatibility I either retreat in fear, or venture forth to discover that these badlands are just as much of a deathtrap as I believed them to be.

Unfortunately, my job requires on a daily basis for me to venture out into this dangerous wilderness and test my mettle (and much more so my patience) by working with that special trifecta of HTML, CSS, and Javascript. I also have a terrible memory, so I run into the same problems and reinvestigate them far more often than I should. So I’m making a start to writing down those problems, both for my reference and yours.

Let me begin by saying that I have a confession to make. I use Bootstrap. And I love it. I rely on it. It makes a lot of this terrible business a lot more bearable. When it goes wrong it makes me sad. Not just because I then actually have to do my own work, but because it’s like my close friend has suddenly grown a second, angrier head and has started hurling insults and rocks my way. In short, I feel betrayed.

Most recently this happened with the Twitter Bootstrap popover extension. Normally a lovely tool, I found all of a sudden it would refuse to appear when I attached it to an anchor tag and set it’s trigger event to ‘click’.

For some reason, the ‘click’ trigger maps to an event that requires the element to receive focus. Although this is just fine in Firefox (and IE!), in WebKit based browsers like Chrome and Safari certain tags cannot receive focus from the mouse if they don’t have a tabindex. Oddly enough, the anchor tag is given a place in the tab-order by default in all the browsers I’ve investigated so you can trigger onfocus events by tabbing through to the element, but not by clicking it. It took me a lot longer than it should to figure that out. If you just came here to get your popover working on click you can go home now. Just add “tabindex=’0′” to the elements that you want to have popovers or tooltips and it should work just fine. If you’re more curious about why this is, and how it works, I’ve got a couple more paragraphs for you.

The trouble is, we don’t really have a standard for this. The closest we have is the DOM Level 2 HTML specification that actually defines focus methods for only four tags; HTMLInputElement, HTMLSelectElement, HTMLTextAreaElement and HTMLAnchorElement. What is focusable and how (by keyboard navigation or by mouse clicking) is left somewhat open ended. Because of this, different browsers have chosen to allow focus events by default on a wide variety of elements.

This puts the tabindex attribute in the position of not only dictating the tab order, but also the focusability of elements. Once you get past the now apparent misnaming of the attribute, it’s easy to use in this manner, and should resolve further problems of focus quickly and easily (at least it has for me). For quick analysis of whether this issue is the source of your difficulties I highly recommend referencing this well organized table which details browser behaviour by element in regards to focus.

Breakdown

Problem

Boostrap popovers/tooltips using ‘click’ trigger aren’t working on certain elements in Chrome and Safari.

Root Cause

Many HTML elements can’t obtain focus (in some cases this only applies to gaining focus from a mouse click! -_-) in certain browsers by default.

Solution

If the you want to be able to give the elements focus by mouse, but exclude them from the tab-order (and thereby keyboard focussing), give them a tab index of -1.

If you would like the elements to maintain their tab-order give them a tabindex of 0, and they will be able to gain focus from both mouse and keyboard.

Further Reading

1. DOM Level 2 HTML Specification
2. Reference table of browser behaviour
3. A better written, more technical, and more informative look at much the same topic