Tweetable Python


Sometimes when I have nothing better to do, or when I do have something better to do but I really don’t want to do it, I play code golf with snippets of my work – trying to compress my bits of logic down to something tiny enough that I can tweet it. They aren’t efficient, they’re certainly not pretty, but there’s some satisfaction in squeezing out every last character I can while still maintaining the original functionality. I tend to write my golfs as standalone functions, so even though there’s no sane reason to use them as they’re currently formatted, they’re fully functional so I can play with them later. This isn’t a particularly educational post, I just enjoy these as puzzles and I love to share games like this with people who enjoy the same types of problems.

Dictionary Inversion – 75

Dictionary inversion, at least in python, is a nice simple operation (particularly with the dictionary comprehensions of 2.7+). Still, a basic implementation (disregarding handling of duplicate values) is a good warmup for a problem I had to tackle a couple of years ago.

def a(d):return{v:k for k,v in d.items()}
# for 2.6ers, def a(d):return dict(v,k for k,v in d.items()

The actual task I was given was to take a dictionary of values to lists of values and reverse it such that each entry in the lists pointed to a list of keys that pointed to lists which contained that element. I’ve included an example below to clarify the requirements.

# Given a dictionary of the form
d = {1: [1,2], 2: [2,3]}
# The function should return a dictionary of the form
{1: [1], 2: [1,2], 3: [2]}

# Solution:
def a(d):return{i:[k for k in d if i in d[k]]for v in d.values()for i in v}

# Or, in a slightly more readable form:
def a(d):
 return{i:
  [k for k in d if i in d[k]]
  for v in d.values()for i in v
 }

Pretty printer – 132

The goal here was to take a two-dimensional array of strings and print them neatly in a tab-delineated table. This implementation assumes an eight-space tab size, but is easily adaptable to other sizes.

My ‘pretty printer’ is one of the few old code golf examples where I still have some records of my previous iterations of compression. In the interest of sharing my process for code golfing, here are a few of the steps I still have copies of.

# So, given the following input
table = [
  ["Name", "Age", "City"],
  ["Bob", "23", "Detroit"],
  ["Angelina", "103", "San Francisco"]
]

# The function will print
# Name          Age     City
# Bob           23      Detroit
# Angelina      103     San Francisco

def pretty_printer(rows):
  columns = zip(*rows)
  max_widths = [max(len(value) for value in column) for column in columns]
  for row in rows:
    print '\t'.join(value + "\t" * (max_widths[col]/8 - len(value)/8) for col, value in enumerate(row))

The first iteration, as you can see, is entirely uncompressed. The variables are well named for clarity, and I even use a for-loop instead of a comprehension to keep things nice and simple. The only particularly interesting line is columns = zip(*rows), which splats the sequence of rows into zip(), which then combines all the first, second, third, etc., elements into new lists, giving us the columns instead.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8)for i, v in enumerate(r)) for r in t)

# Or, with line breaks
def p(t):
  print'\n'.join(
    '\t'.join(
      v+'\t'*([max(len(e) for e in c) for c in zip(*t)][i]/8-len(v)/8) 
    for i, v in enumerate(r)) 
  for r in t)

My first pass striped out all the nice naming, most of the whitespace, and compressed it down to one line by putting the variable value generation in where ‘columns’ and ‘max_widths’ used to be. Instead of printing line by line I also switched to using ‘\n’.join(), which combines a list of strings with line breaks between each.

def p(t):print'\n'.join('\t'.join(v+'\t'*([max(map(len,c))for c in zip(*t)][i]/8-len(v)/8)for i,v in enumerate(r))for r in t)


# Again, with some line breaks for readability
def p(t):
  print'\n'.join(
    '\t'.join(
      v+'\t'*([max(len(e)for e in c)for c in zip(*t)][i]/8-len(v)/8)
    for i, v in enumerate(r))
  for r in t)

I shaved off the final 10 characters by trimming out the whitespace I missed on the first pass, and replacing a comprehension with a map.

Not shown here are more than a few false starts that I remember trying but unfortunately didn’t write down. The difference between pretty good and excellent code golfing is generally just a few characters, so there’s a lot of experimentation that goes on where you do things like trying to see if a particular case works best as a list comprehension or a map operation, or whether it’s worth the overhead of creating an interim variable if you can avoid the character-cost of recomputation later.

Threaded Numbers – 83

I wrote this problem specifically for a code golf competition, so while it doesn’t have any practical use it makes for a good little puzzle in more than a few languages. I remember distinctly seeing a pretty impressive selection of approaches, even just within python (though there were plenty of interesting other language entries as well). If you’re looking to play around with code golfing, this would be one of the first I recommend you try out.

The object of the exercise is to open a file in the same directory as your program, which will contain a sequence of randomly ordered integers that are delineated by spaces. Your program should read the integers, and then output them in a very specific sorted format. The outputted list should also be space-delineated, but should be ordered such that the smallest number is the leftmost in the sequence, the second smallest is the rightmost, the third smallest the second leftmost, and so on.

# If the file 'in.txt' contains "1 3 5 8 2 9 4 6 8 10" 
# the program should output 1 3 5 8 9 10 8 6 4 2

x=sorted(open('in.txt').read().split(),key=int)
print' '.join(x[::2]+x[1::2][::-1])

And just for fun, here’s one of my other (less successful) attempts at the same problem. It relies on the trick of passing the same iterator twice to zip, so that items are pulled off the sorted list in pairs and dropped into two separate new sequences.

x=iter(sorted(open('in.txt').read().split(),key=int))
a=zip(*zip(x,x))
print' '.join(a[0]+a[1][::-1])

CodeToasters!

Anyone wondering where I went? Sorry about that. Took on a lot at work, not quite more than I could chew, but enough that my extra curricular activities got cut down pretty slim. So Most of what I’ve been working on I’m pretty certain I’m still not allowed to talk about, but there’s one thing that’s come out of the last few weeks that I’m both permitted and excited to share, and that’s CodeToasters!

CodeToasters is a group that meets weekly to present and solve algorithm problems after work (Wednesdays at five!) that I started a while back and that has been formalized into a company wide (it used to just be me and a few team mates) meetup! Although the format is still in flux what we have at the moment seems to be working pretty well. We start with a bref discussion of what people came up with for the bonus take-home problem that was on last week’s agenda. After that someone (a different person each week) does a walkthrough of an algorithm or a simple interview question. This prompts each presenter to learn a problem or concept extremely thoroughly so that they can communicate it effectively and answer any questions that the other attendees might have.

The last segment (usually the last ~30 minutes of the hour block) is spent with the group collectively solving a presented problem. After an explanation of the problem and a few sample cases, there is usually a few minute quiet period as people first attempt to approach the problem. Then either a group discussion is started or people split off into subgroups to mull over possible solutions, as the presenter wanders between groups providing gentle guidance. People who have grokked the problem exceptionally quickly sometimes take a whiteboard and start sketching out approaches to the problem, which sparks further group discussion.

I’ve been writing a lot of the problems we’ve been solving, which has been a blast, but unfortunately I can’t put many of them up here as we’ve started using some of them as interview questions at LinkedIN. Today’s batch though, not only did I write all of them (actually, I didn’t write quicksort, and the bonus problem is an adaptation of a riddle my friend asked me once) but none of them have been claimed yet, so I’m pretty certain I’m totally free to put them up here. With that in mind I’ve embeded today’s agenda below for people to take a look at. Feedback welcome!

Anyway I’d love to hear what people think of this one, and if you have any interest in getting your own CodeToasters going, in solutions to the posted problems, or in just chatting about the idea and format, just ping me at ebrynne@gmail.com!

CodeToastersW4

Python Tuple Tutorial – What they are and when to use them

My goal for today’s post is to give a deeper understanding of what tuples are and how those distinctions make them useful, saving their many novel applications involving packing/unpacking for a maybe-someday follow-up post so that I can address it a a psudo-distinct concept. Then, for those interested in what I’ve been writing, I hope to follow that up with tutorials on sequence types generally, lists specifically, usages of map, reduce and filter, and then the itertools module. Apparently my enthusiasm for sequences has a ways to go before it plays itself out. I make no promise of sticking to that schedule, but if I do I hope to release each of the above a slightly accelerated pace compared to my usual posting speed.

That all being said, for those of you who came here to learn (or better yet, teach me something!), I’ll get started on the tutorial now.

What They Are

Python tuples belong to the group of sequential data types, a subset of containers which is comprised of strings, Unicode strings, lists, tuples, bytearrays, buffers, and xrange objects. Tuples, like lists, can contain heterogenous ordered data; and all of the standard sequence operations can be applied to them. The primary difference between lists and tuples is that tuples are immutable, which means that once created, they cannot be altered. New tuples can be created from old tuples, and sub-tuples can be extracted, but once a tuple has been instantiated it is left utterly unchanged until the garbage collector cleans it up.

There are two very similar syntaxes for creating tuples, either just a comma separated list of values, or a comma separated list of values enclosed by technically optional parenthesis. I say ‘technically optional’ because to the very best of my understanding to exclude the parenthesis is uncommon and unpopular. I personally favour parenthesis as well, as I feel they keep visual consistency with lists while also clearly indicating that the created object is a tuple.

# Although you can do this:
no_parens_tuple = 1, 'b', 2

# Personally I lean towards 
parens_tuple = ('a', 2, 'c')

What I feel helps my case in favour of the usage of parens is that the syntax to create a single item tuple is the element with a comma after it, with or without parens. Using just the comma without the parens is easy to miss and makes the code easier to misunderstand. To further emphasize my point, a zero-element tuple can only be created by just a pair of parenthesis with nothing between them. Given that these two uses both make a strong case for using the parens (even as uncommon as they may be), for the sake of consistency they should be used for longer tuples as well.

readable_tuple = ('spectacular',)

ambiguous_tuple = 'spectacular',

empty_tuple = ()

Please note that the trailing comma is required for a single item tuple, even if parenthesis are included. Otherwise the parens are disregarded and the result is just the object intended for the tuple.

remembered = (10,)
remembered
(10,)
 
forgot = (10)
forgot
10

The gathering of values into a tuple is called ‘tuple packing.’ The converse also exists, called ‘tuple unpacking,’ and I’ll delve into it in part two of this series. It’s a concept that deserves individual addressing.

Tuples have a couple more interesting (or challenging) characteristics that are born of their immutable nature. Firstly, they are hashable, which means (amongst other things) that they can be used as keys in dictionaries. Were you to try using a list as a key in a dict it would fail, and with good reason; as soon as the list was altered its hash value would change and it would point to a different or nonexistent bucket of items in the dictionary. To be honest, there aren’t many cases where hashing a tuple is going to be essential, but there’s a pretty healthy number where it’s neat, intuitive, or more pythonic than the alternative.

Functionally, it sometimes helps to think of tuples as a subset of lists; they forgo all the non-standard-sequence functions (such as append() and index assignment) but in exchange are a faster and more lightweight data type. If performance is a concern and it’s an option, tuples are the way to go. Before further addressing when and why to use tuples, I’d like to illustrate a little further the distinctions between mutable lists and immutable tuples below:

person_tuple = ('James Woods', 45, 'Probably Hollywood', 'Actor')
person_list = ['Luke Skywalker', 25, 'Far Far Away', 'Jedi Knight']

# Of course appending to a list works like you'd expect
person_list.append('Yoda's School of the Jedi Arts')

# But we can't do this with tuples
person_tuple.append('School of Hard Knocks')
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
AttributeError: 'tuple' object has no attribute 'append'

# Assigning new values doesn't work either
person_tuple[3] = 'Celebrity'
Traceback (most recent call last):
  File '<stdin>', line 1, in <module>
TypeError: 'tuple' object does not support item assignment

# Tuples can be used to derive new tuples:
slice_tuple = person_tuple[1:3] # Using slices
combined_tuple = person_tuple + person_tuple # Appending one to another
copied_tuple = person_tuple[:] # Duplicating tuple
tuple_val = person_tuple[0] # Index retrieval works as you'd expect

 

When To Use Them

Because lists and tuples are so similar, and lists grant additional flexibility, it’s easily to fall back on the argument that lists will do just fine, particularly because they’re so comfy and familiar to most people by the time they discover tuples. And to be honest, a great majority of the time using lists will be perfectly adequate, and if by the end of this tutorial you’re still confused by tuples and unsure of where they belong, go ahead and use that list, it’s okay. Chances are it’ll work just fine and no one will ever even think to question you. That being said, there are times that the advantages of tuples (such as their immutability, hashability, and performance gains) are significant enough to justify their usage. My goal here is to give you just a bit of a sense of when these cases might be.

You may note that many of my examples so far have contained heterogenous data constructed to represent a complex object (like a person). This is because unlike with lists we have a guaranteed contract of consistency with tuples, meaning that if I create a tuple with a string, a number, and then a bytearray, I can know with absolute certainty when I operate on that tuple later that the properties and their types will match the schema and values with which I created them earlier. Because there is no guarantee of order or data type with a list after it’s been created you lose that contract of consistency; and therefor I find that as a rule of thumb although lists techinically support heterogenous data, I try to keep them to sets of a single data type and use them in cases where I’m iterating over the elements and performing a uniform operation on all of them.

Tuples on the other hand are write-protected, almost like they have an implicit assert statement verifying that their data will be correctly formed. For this reason I use them typically for cases where I need a quick stand-in for an object, one that’s light-weight, easy reconfigurable at a later date, and not needed for use elsewhere. For example in a quick script to pull rows of data from a static source (database, csv, or other similar store) where you’re doing all the heavy lifting for one reason or another, it makes sense to handle rows as tuples because it wouldn’t make sense to modify, reorder, or resize those rows; it would create inconsistency of data that would be difficult to process. If our rows are tuples then when we use code like the following:

for row_num, name, email, first_pet, mothers_maiden_name in rows:
  print 'Processing %s (%d/%d)' % (name, row_num, len(rows))
  recover_password(email, first_pet, mothers_maiden_name)

We know our code won’t fail because of an accidentally extended row or a swapped name and email (unless of course the original data was corrupt, which is a whole different story). Python itself is written around this concept, that tuples have an implied semantic and an index has meaning. A great example of this is the return value of python’s database API specification, where fetchmany() and fetchall() return a list (ordered sequence of homogenous data) of tuples representing rows (hetergenous set where order has specific meaning). Cool, eh?

Another of tuples’ advantages needs less longwinded addressing, their improved performance, as it is relatively self explanatory. More lightweight than lists, dictionaries, or instantiating custom objects, creating and iterating over tuples is simply faster. Of course, this isn’t carte blanche, wrapping a list in a one-element tuple won’t speed up processing of that list, but for tuples that contain only hashable data types (simple and immutable types) there’s a pretty decent speed boost. Rarely will you find yourself writing python where the difference between list and tuples performance is a make or break choice, but if you’re dealing with large amounts of performance sensitive code and data, sticking to tuples can be one of many choices you make to make sure it all runs at top speed.

Finally, the hashability of tuples is another simple but neat little feature. This, like the performance gains, is particular to tuples that contain hashable data, but can be applied in some pretty fun places. Generally, it makes any situation where you have a couple of shared properties between objects that you would like to group and access them by a lot easier. For example, lat/long coordinate pairs for people or residencies, or month_of_year, day_of_month pairs make it easy to group date-tracked data while neatly handling that pesky leap-year problem.

lat_long_tup = (49, -117)
lat_long_list = [37, -122]

# If I were storing a bunch of house locations, it'd be easy to group them by lat/long coordinates.
# I do this by coordinates instead of by city because I used to live in a forest and no,
# Vancouver isn't a close enough approximation. It's eight hours away!
peoples_houses = {}
peoples_houses[lat_long_tup] = [my_old_house, my_old_neighbour]

#This will fail due to lists being unhashable
peoples_houses[lat_long_list] = [my_new_house, my_old_neighbour]
Traceback (most recent call last):
File '', line 1, in
TypeError: unhashable type: 'list'

Extra Reading

A shorter, better, and cooler description of tuples and how they compare to lists
An excellent tuple tutorial, and it’s illustrated!
Sequence types. The table in this section is an invaluable reference
Python tuples aren’t just constant lists

List Comprehensions for Fun and Profit

List comprehensions are awesome. They’re an incredibly intuitive and readable way to create, filter, join and otherwise modify lists. In the interest of helping young pyrates understand how they can use this simple and elegant tool to their advantage this tutorial will start with the very basics of what list comprehension are, and then gradually scale up to some of their cooler and more esoteric applications. If you’re already well acquainted with the concept, skip down a few sections for the more exciting stuff, or jump right to the end for the really juicy bits.

A list comprehension is a concise way of generating a list from an existing set or another iterable object. The syntax for the one of the most basic list comprehensions is as follows: animal_names = [animal.name for animal in list_of_animals] 

The idea, as I’m sure is evident or familiar to many of you, is that the new list will be the set of items as specified at the beginning of the list comprehension (before the ‘for’ clause), extracted from the list at the end of the comprehension (after the ‘in’ clause). The item in the middle, our ‘animal’ variable in the example above, is the variable that will represent each item in the list being processed, is scoped to the comprehension, and may be acted upon the same as any other object. The example above extracts a list of animal names from an original list of animals and could also be written as:

animal_names = []
for animal in list_of_animals:
  animal_names.append(animal.name)

Another popular example to introduce list comprehensions is the list of perfect squares of the numbers in a range. Easy enough to compute with a few lines of code but with list comprehensions we can do it in one and, I’d argue, make the statement even more readable:

squares = [x*x for x in range(15)]
squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81, 100, 121, 144, 169, 196]

Predicates and Generator Expressions

The first way we’re going to make list comprehensions more useful is by adding an if-statement to dictate the inclusion of terms extracted from the source list in the resulting list. This if-statement, known as the predicate, is added at the end of the comprehension, and maintains the almost english-like readable syntax. To continue our mathematical examples (list comprehensions in fact have their roots in traditional mathematic expressions), what follows is the set of numbers between zero and one hundred that are divisible by seven:

div_by_seven = [x for x in range(100) if x%7 == 0]
div_by_seven
[0, 7, 14, 21, 28, 35, 42, 49, 56, 63, 70, 77, 84, 91, 98]

This allows us to easily filter out list items that are undesirable. (Note that the above example can also be accomplished with range(0, 100, 7). This example is illustrative only.)

I’d also like to touch on the fact that the value at the start of the list comprehension, the value that will be included in your final list, can be the result of an expression as well as a static value. Much like lambda functions there can only be a single expression at the start of your comprehension, which restricts you from writing large blocks of logic within the comprehension. I’m personally in favor of this limitation, as it enforces the terse and readable structure of the comprehension and prevents wild misuse. This functionality allows us to take advantage of other bits of python’s syntactic sugar such as ternary operators:

moods = ['sad' if person.is_alone() else 'happy' 
  for person in people_at_party]

For those of you unsure of what exactly expressions and statements are in python a brief (and only mostly accurate) definition would be that all expressions evaluate to a value, whereas statements are a superset which describe pretty much any piece of python code and therefor include the set of expressions. For quick reference I use the ‘does it look like it belongs on the righthand-side of an assignment’ check to evaluate whether a statement is an expression and therefor can be used in a comprehension. A few of the most common examples of these are variables, ternary operators, and function calls. For a better understanding of this concept I recommend checking out these links.

Nested and Chained Comprehensions

List comprehensions are very flexible, and can be extended in a number of ways. The first way, and perhaps the simplest, is by using nested comprehensions. Nested comprehensions are a single-line equivalent of nested for-loops (as you may have been able to infer), and as such extend the functionality we’ve established in exactly the way you’d expect. A neat little nested comprehension can be used to generate the identity matrix that will be familiar to anyone who’s dabbled in matrix algebra. For those who haven’t, this produces a 4×4 matrix of ones and zeroes such that the ones form a diagonal line from the top left to the bottom right:

ident_matrix = [[1 if col_index==row_index else 0 for row_index 
  in range(4)] for col_index in range(4)]
ident_matrix
[[1, 0, 0, 0], [0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1]]

You’ll note that the value generators in these two comprehensions are slightly more complex than what we’ve seen previously. For the inner expression we take advantage of python’s ternary syntax, while for the outer comprehension its generator expression is the nested comprehension, which is what gives us the list-of-lists structure that emulates the matrix.

While by nesting list comprehensions we can easily build lists of lists, we can deconstruct the same by using chained comprehensions. Chained list comprehensions similarly iterate over multiple lists just like nested for loops, but produce a single array instead of a multidimensional one. I recommend using chained comprehensions sparingly, as their syntax is somewhat less intuitive and they can quickly become un-pythonic in their complexity. Given that matrix we generated above, here’s a quick example of how we could flatten it back out:

flattened = [num for row in ident_matrix for num in row]
flattened
[1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1]

In more traditional syntax, this could be written:

flattened = []
for row in ident_matrix:
  for num in row:
    flattened.append(num)

To make this a little more complex, let’s flatten the identity matrix and get only the zeros, ignoring the ones entirely.

flat_zeroes = [num for row in ident_matrix for num in row if num==0]
flat_zeroes
[0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0]

Or how about only the zeroes from the even numbered rows?

even_zeroes = [num for row in ident_matrix if 
  ident_matrix.index(row)%2==0 for num in row if num==0]
even_zeroes
[0, 0, 0, 0, 0, 0]

You can see how the awkward alternations between our assignments and our predicates quickly make our list comprehension much harder to read and follow. This is where list comprehensions begin to approach the limit of their usefulness. As much as it’s nice to have everything on one line, list comprehensions are meant as a convenience tool; if they’re making your code more complex and unreadable, they should be replaced by a more traditional structure. If your code looks indecipherable and fits on one line but could easily be replaced by a few more lines that could be written by a coder with only minimal experience with python, you’re trying too hard to impress someone.

# Perhaps overly simplified, but straightforward and unambiguous.
even_zeroes = []
for row in ident_matrix[::2]:
  for val in row:
    even_zeroes.append(val) if val == 0 else None

Dict and Set Comprehensions

Python 2.7 added dictionary comprehensions, a long sought after feature. Sadly, as much of my work is still done on projects that rely on python 2.4-6, these are not available to me. That being said, by cleverly combining list comprehensions and python’s built in dict() function we can create our own simple dictionary comprehensions in a pinch. The dict() function can construct a dictionary from a number of different inputs, but the one that’ll be the most useful to us is the list of tuples. Calling the following:

dict([('cat', 'meow'), ('dog', 'woof')])
{'dog': 'woof', 'cat': 'meow'}

Gives us a dictionary that maps the keys ‘cat’ and ‘dog’ to their respective animal noises. As I’m sure many of you can see this allows us to dynamically create such a list by using a similar expression within a comprehension like this:

inverted_dict = dict([(val, key) for key, value in 
   old_dict.items()])

Python 2.7+ dict comprehensions are semantically equivalent, but more intuitive and less expensive due to not having to create an intermediate list to pass to the dict constructor. To do the same as we did above in later versions of python, we can use:

inverted_dict = {val : key for key, val in old_dict.items()}

Much simpler, and much more pythonic.

Set comprehensions are also a new addition as of python 2.7, which rather than generating a list or dict, they create a set (only contain unique items). The syntax is similar to dict comprehensions (in that it uses the ‘{‘ symbol), but instead of giving it a pair of items, we provide just one. This can be emulated in python 2.6- by calling set() on the generated list.

unique_names = {name for name in all_names} 
# Is equivalent to
unique_names = set(all_names)

Generator Objects

A slight variation to list comprehensions is generator objects which are syntactically and functionally nearly identical, but rather than generating a list when the generator object is instantiated they pull from the source list when the generator object is iterated upon. This removes the need for intermediate storage and provides a not insignificant performance gain in many cases. Additionally, if the list being generated from is altered after the instantiation of the generator object but before iterating over the generator’s product then the change will be reflected in the resulting list. Finally, because we instantiate a generator instead of a list we can’t access the list members like we would be able to usually; instead the object is effectively opaque until it is iterated over and then we have access to only a single member at a time. Because this is an easy concept to get tripped up on I’m going to do my best to illustrate it below. Please note that the generator is instantiated by using ‘(…)’ instead of ‘[…]’ notation.

source = [1, 2] #list from which we'll generate 
comp_list = [num + 1 for num in source] #Stores complete result
gen_obj = (num + 1 for num in source) #Stores procedure for generating list

comp_list[0] #Regular list access
2
gen_obj[0] #Doesn't work
Traceback (most recent call last):
  File 'stdin', line 1, in module
TypeError: 'generator' object is unsubscriptable

source.append(3) #alter list

# Comprehension yields the product of unmodified list
for num in comp_list:
  print num
2
3

# Generator yields product of augmented list
for num in gen_obj:
  print num
2
3
4

Obviously, as with all things in life and coding, using list comprehensions vs generator objects is a tradeoff, in this case between flexibility and performance. It’s up to you to apply them judiciously.

Esoterica (Or, The Good, The Bad, and The Ugly)

Let’s start with the good:

almost_csv = [line.split('::') for line in open('my_report.csv')]

I’ve seen mixed opinions on this comprehension but I’ve done enough not-comma-delimited file handling to find this quick and dirty solution useful. Because files are closed when the corresponding file object is deallocated and the file object here is scoped to just the list comprehension, this neatly opens a file, splits it by lines, and then splits lines on ‘,’ before closing the file again. All in the space of just a few characters. Admittedly it means you’re loading the entire file into memory, but hey, sometimes that’s okay. Alternately, if you used a generator instead, you’d get the same concise syntax with non of the memory overhead! Please note that for any case where your delimiter is a single character, the csv module should be used.

Here are a couple more neat little perks, annotated with comments as needed:

# Tuple unpacking works in list comprehensions!
unpacked = ['%s + %s' % (a, b) for a, b in enumerate(range(100, 0, -1))]

# Comprehensions play well with python's built in functions
sorted_owners = sorted(getOwner(dog) for dog in dogs) 
# Note that the above creates a generator object and shares parenthesis with the sorted function

As for the bad, I’m going to unashamedly steal the form of this one from Chris Leary‘s blog post which you should definitely read if you want a deeper and more entertaining analysis of why you shouldn’t try this:

ord_lists = [2,3,4,5,3,4,23,24,25,26,45,46,9,13,14]
sub_seqs = []
[sub_seqs[-1].append(x) if sub_seqs and x == sub_seqs[-1][-1] 
  + 1 else sub_seqs.append([x]) for x in ord_lists]
[None, None, None, None, None, None, None, None, None, None, None, None, None, None, None]
sub_seqs
[[2, 3, 4, 5], [3, 4], [23, 24, 25, 26], [45, 46], [9], [13, 14]]

While this works, and perhaps gets some points for being concise, it’s going strictly against the designated purpose of list comprehensions. List comprehensions are for list construction, not shorthand for loops. All this is doing is saving the developer the inconvenience of writing the proper loop syntax. Because append() does not return a value we’re generating a garbage list of Nones the length of the list that’s being modified; and more importantly any developer who reads this will be forced to re-comprehend the meaning of the code after realizing that the safe and familiar comprehensions that they’re used to are being bent to a completely foreign purpose. List comprehensions are nice, be nice to them too.

The ugly award goes to a list comprehension I saw that used a neat trick that I suppose could be super useful in certain situations, but if you find yourself using it you better have a pretty excellent argument for your case. Python lets you access a dictionary of all local variables, even the unnamed ones, and list comprehensions are assigned the designator ‘_[1]’. This gives us the pretty powerful ability to access and pass to functions the generated list as it’s being generated. For example, here are two comprehensions, the first of which uses a isPrime function that we define to generate the list of prime numbers from zero to fifty, and the latter of which is a concise way of getting the first N numbers in the Fibonacci sequence:

def isPrime(num, primes):
  for prime in primes:
    if prime == 0 or prime == 1:
      pass
    elif num % prime == 0:
      return False
  return True

[num for num in range(50) if isPrime(num, locals()['_[1]'])]
[0, 1, 2, 3, 5, 7, 11, 13, 17, 19, 23, 29, 31, 37, 41, 43, 47]

# Fibonacci Sequence
[locals()['_[1]'][-1] + locals()['_[1]'][-2] if 
  len(locals()['_[1]']) > 1 else 1 for i in range(15)]
[1, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377]

However once again, this is pretty unreadable, and relies on an undocumented feature that has no guarantee of consistency or maintenance. Watch out.

Further Reading

PEP 202, List Comprehensions
A nice warning about abusing list comprehensions, from the desk of Jesse Noller
A guide to Python 3 Comprehensions

Edits

There were a couple of mistakes and a totally untested (and wrong) example. Thank you to all who pointed it out, and I’ll try to continually improve this tutorial as I learn more.

Bootstrap Popover/Tooltip Click Trigger

Navigation

  1. Narration
  2. Breakdown
  3. Demo

Narration

Web development has it’s quirks. And by I mean that those of us like myself who are used to the simple and safe territory of a well defined, well documented, and most of all  consistent server-side programming language usually end up somewhere between daunted and devastated each time we subject ourselves to the tempestuous whims of front-end development. Like a peaceful Vault-dweller, each time I poke my head into the Capital Wasteland of browser-compatibility I either retreat in fear, or venture forth to discover that these badlands are just as much of a deathtrap as I believed them to be.

Unfortunately, my job requires on a daily basis for me to venture out into this dangerous wilderness and test my mettle (and much more so my patience) by working with that special trifecta of HTML, CSS, and Javascript. I also have a terrible memory, so I run into the same problems and reinvestigate them far more often than I should. So I’m making a start to writing down those problems, both for my reference and yours.

Let me begin by saying that I have a confession to make. I use Bootstrap. And I love it. I rely on it. It makes a lot of this terrible business a lot more bearable. When it goes wrong it makes me sad. Not just because I then actually have to do my own work, but because it’s like my close friend has suddenly grown a second, angrier head and has started hurling insults and rocks my way. In short, I feel betrayed.

Most recently this happened with the Twitter Bootstrap popover extension. Normally a lovely tool, I found all of a sudden it would refuse to appear when I attached it to an anchor tag and set it’s trigger event to ‘click’.

For some reason, the ‘click’ trigger maps to an event that requires the element to receive focus. Although this is just fine in Firefox (and IE!), in WebKit based browsers like Chrome and Safari certain tags cannot receive focus from the mouse if they don’t have a tabindex. Oddly enough, the anchor tag is given a place in the tab-order by default in all the browsers I’ve investigated so you can trigger onfocus events by tabbing through to the element, but not by clicking it. It took me a lot longer than it should to figure that out. If you just came here to get your popover working on click you can go home now. Just add “tabindex=’0′” to the elements that you want to have popovers or tooltips and it should work just fine. If you’re more curious about why this is, and how it works, I’ve got a couple more paragraphs for you.

The trouble is, we don’t really have a standard for this. The closest we have is the DOM Level 2 HTML specification that actually defines focus methods for only four tags; HTMLInputElement, HTMLSelectElement, HTMLTextAreaElement and HTMLAnchorElement. What is focusable and how (by keyboard navigation or by mouse clicking) is left somewhat open ended. Because of this, different browsers have chosen to allow focus events by default on a wide variety of elements.

This puts the tabindex attribute in the position of not only dictating the tab order, but also the focusability of elements. Once you get past the now apparent misnaming of the attribute, it’s easy to use in this manner, and should resolve further problems of focus quickly and easily (at least it has for me). For quick analysis of whether this issue is the source of your difficulties I highly recommend referencing this well organized table which details browser behaviour by element in regards to focus.

Breakdown

Problem

Boostrap popovers/tooltips using ‘click’ trigger aren’t working on certain elements in Chrome and Safari.

Root Cause

Many HTML elements can’t obtain focus (in some cases this only applies to gaining focus from a mouse click! -_-) in certain browsers by default.

Solution

If the you want to be able to give the elements focus by mouse, but exclude them from the tab-order (and thereby keyboard focussing), give them a tab index of -1.

If you would like the elements to maintain their tab-order give them a tabindex of 0, and they will be able to gain focus from both mouse and keyboard.

Further Reading

  1. DOM Level 2 HTML Specification
  2. Reference table of browser behaviour
  3. A better written, more technical, and more informative look at much the same topic