Map/Reduce/Filter/Apply¶

Great resource - http://www.u.arizona.edu/~erdmann/mse350/topics/list_comprehensions.html

Other reference (python2): http://www.python-course.eu/lambda.php

Python3 version: http://www.python-course.eu/python3_lambda.php

Map¶

Filter¶

lines = list(filter(lambda x: x.startswith('['), results))

Map, Filter, Lambda, and List Comprehensions in Python Author: R.G. Erdmann

map(), filter(), lambda, and list comprehensions provide compact, elegant, and efficient ways to encode a few common idioms in programming. We often encounter the following scanarios involving for-loops: Some for-loop examples to rewrite more compactly

Building up a list from scratch by looping over a sequence and performing some calculation on each element in the sequence.

    For example, suppose we want to build a list of the squares of the integers from 0 to 9:

    >>> squares = [] # start with an empty list
    >>> for x in range(10): # step over every element in the list of integers from 0 to 9
    ...     squares.append(x**2) # grow the list one element at a time
    ...
    >>> print squares
    [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

    Suppose we want to build a list of the lengths of the names in a list:

    >>> names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach']
    >>> lengths = []
    >>> for name in names:
    ...     lengths.append(len(name))
    ...
    >>> print lengths
    [4, 3, 3, 5, 6, 7, 4]

Building up a list from scratch by looping over nested sequences.

    For example, suppose we want a list of all possible pairs of drink and food from the lists ['water', 'tea', 'juice'] and ['ham', 'eggs', 'spam'], respectively:

    >>> possible_choices = []
    >>> for drink in ['water', 'tea', 'juice']:
    ...     for food in ['ham', 'eggs', 'spam']:
    ...         possible_choices.append([drink,food])
    ...
    >>> print possible_choices
    [['water', 'ham'], ['water', 'eggs'], ['water', 'spam'], ['tea', 'ham'], ['tea', 'eggs'], ['tea', 'spam'], ['juice', 'ham'], ['juice', 'eggs'], ['juice', 'spam']]

    Suppose we want a list of coordinates on a rectangular grid:

    >>> coords = []
    >>> for x in range(5):
    ...     for y in range(3):
    ...         coordinate = (x, y)
    ...         coords.append(coordinate)
    ...
    >>> print coords
    [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2), (3, 0), (3, 1), (3, 2), (4, 0), (4, 1), (4, 2)]

Building a list from scratch by filtering a sequence according to some criterion or criteria.

    For example, suppose we want a list of the squares of the integers from 0 to 9 where the square is greater than 5 and less than 50:

    >>> special_squares = []
    >>> for x in range(10):
    ...     square = x**2
    ...     if square > 5 and square < 50:
    ...         special_squares.append(square)
    ...
    >>> print special_squares
    [9, 16, 25, 36, 49]

    Suppose we want to take a list of names and find only those starting with the letter B:

    >>> names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach']
    >>> b_names = []
    >>> for name in names:
    ...     if name.startswith('B'):
    ...         b_names.append(name)
    ...
    >>> print b_names
    ['Bob', 'Barbara']

Map, Lambda, and Filter

One way to achieve the same goals as in the above examples is to use some of Python’s tools from functional programming: map(), filter(), and lambda(). map()

The map() function applies a function to every member of an iterable and returns the result. Typically, one would use an anonymous inline function as defined by lambda, but it is possible to use any function. The first example above can also be accomplished using map() as follows:

def square(x): ... return x**2 ... squares = map(square, range(10)) print squares [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

In the case of the second example, the len() function already exists, so we can use map() to apply it to every element of the list:

names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach'] lengths = map(len, names) print lengths [4, 3, 3, 5, 6, 7, 4]

lambda

In the first map example above, we created a function, called square, so that map would have a function to apply to the sequence. This is a common occurrence, so Python provides the ability to create a simple (no statements allowed internally) anonymous inline function using a so-called lambda form. Thus, an anonymous function that returns the square of its argument can be written as lambda x: x**2. This means, “Here is an anonymous (nameless) function that takes one arguement, called x, and returns the square of x. Since the lambda form actually evaluates to a function, we can also call it inline. (This is generally a silly thing to do, but we show it here to demonstrate that lambda forms are actually inline function objects.):

print (lambda x: x**2)(5) # first parens contain a lambda form that is a squaring function, second parens represent calling that function 25

Make a function of two arguments (x and y) that returns their product, then call the function with 3 and 4:¶

print (lambda x, y: x*y)(3, 4) 12 print (lambda x: x.startswith('B'))('Bob') True print (lambda x: x.startswith('B'))('Robert') False incrementer = lambda input: input+1

above is same as def incrementer(input): return input+1¶

now we call it¶

print incrementer(3) 4

Using lambda and map together

Lambda forms can be used as the required function argument to the map() function. For example, the first example above can be written as

squares = map(lambda x: x**2, range(10)) print squares [0, 1, 4, 9, 16, 25, 36, 49, 64, 81

In English, this means, “Apply a function that squares its argument (the lambda form) to every member of the list of integers from 0 to 9 (the range()), and store the result in a list called squares. filter()

The fifth and sixth examples above can also be achieved with the filter() built-in function. Filter takes a function returning True or False and applies it to a sequence, returning a list of only those members of the sequence for which the function returned True.

Lambda forms can also be used with the filter function; in fact, they can be used anywhere a function is expected in Python. In the fifth example, the list of squares is filtered according to whether the given entries are greater than 5 and less than 50. A lambda form that returns True when this condition is met is lambda x: x > 5 and x < 50. Thus, we can reproduce the fifth example as follows:

squares = map(lambda x: x**2, range(10)) special_squares = filter(lambda x: x > 5 and x < 50, squares) print special_squares [9, 16, 25, 36, 49]

In English, this means, “Find every member of the squares list for which the member is greater than 5 and less than 50 (every member for which lambda x: x > 5 and x < 50 returns True), and store that in a new variable called special_squares.

Similarly, the sixth example can be reproduced using filter as follows:

names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach'] b_names = filter(lambda s: s.startswith('B'), names) print b_names ['Bob', 'Barbara']

List Comprehensions

All of the six original examples can be achieved using a consistent, clean, and elegant syntax called list comprehensions. Simple list comprehensions

The simplest form of a list comprehension is

[ expression-involving-loop-variable for loop-variable in sequence ]

This will step over every element of sequence, successively setting loop-variable equal to every element one at a time, and will then build up a list by evaluating expression-involving-loop-variable for each one. This eliminates the need to use lambda forms, and thus generally produces a much more readable code than using map() and a more compact code than using a for-loop.

The first example can thus be written compactly as:

squares = [ x**2 for x in range(10) ] print squares [0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

Some other simple examples:

Print the length of each word in the list of names:

>>> print [ len(name) for name in names ]
[4, 3, 3, 5, 6, 7, 4]

Print the last letter of each name in the list of names:

>>> print [ name[-1] for name in names ]
['e', 'y', 'b', 'd', 'e', 'a', 'h']

Print the reverse of each name in the list of names:

>>> print [ name[::-1] for name in names ]
['ennA', 'ymA', 'boB', 'divaD', 'eirraC', 'arabraB', 'hcaZ']

Note that complex expressions can be put in the slot for expression-involving-loop-variable. For example, here we build up a list of names, first letters, and lengths for each name in the list:

print [ [name, name[0], len(name)] for name in names ] [['Anne', 'A', 4], ['Amy', 'A', 3], ['Bob', 'B', 3], ['David', 'D', 5], ['Carrie', 'C', 6], ['Barbara', 'B', 7], ['Zach', 'Z', 4]]

Where [name, name[0], len(name)] occupies the expression-involving-loop-variable slot, so that the list comprehension creates a list of [name, name[0], len(name)] for every name in the names sequence. Nested list comprehensions

List comprehensions can be nested, in which case they take on the following form:

[ expression-involving-loop-variables for outer-loop-variable in outer-sequence for inner-loop-variable in inner-sequence ]

This is equivalent to writing:

results = [] for outer_loop_variable in outer_sequence: for inner_loop_variable in inner_sequence: results.append( expression_involving_loop_variables )

Thus, the third example can be written compactly as:

possible_choices = [ [drink,food] for drink in ['water', 'tea', 'juice'] for food in ['ham', 'eggs', 'spam'] ] print possible_choices [['water', 'ham'], ['water', 'eggs'], ['water', 'spam'], ['tea', 'ham'], ['tea', 'eggs'], ['tea', 'spam'], ['juice', 'ham'], ['juice', 'eggs'], ['juice', 'spam']]

And example 4. can be written as

coords = [ (x,y) for x in range(5) for y in range(3) ] # points on a rectangular grid print coords [(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2), (3, 0), (3, 1), (3, 2), (4, 0), (4, 1), (4, 2)]

Filtered list comprehensions

The final form of list comprehension involves creating a list and filtering it similarly to using filter(). The filtering form of list comprehension takes the following form:

[ expression-involving-loop-variable for loop-variable in sequence if boolean-expression-involving-loop-variable ]

This form is similar to the simple form of list comprehension, but it evaluates boolean-expression-involving-loop-variable for every item and keeps only those members for which the boolean expression is True. Thus we can use list comprehensions to rewrite example 5 as

special_squares = [ x2 for x in range(10) if x2 > 5 and x**2 < 50 ] print special_squares [9, 16, 25, 36, 49]

Note that the above is inefficient, however, since it has to calculate the square of x three separate times for each element in the loop. Thus, the following is an equivalent and more efficient approach:

squares = [ x**2 for x in range(10) ] special_squares = [ s for s in squares if s > 5 and s < 50 ]

Finally, note that the foregoing can be written on a single line using a pair of list comprehensions as follows:

special_squares = [ s for s in [ x**2 for x in range(10) ] if s > 5 and s < 50 ]

Example 6 can be rewritten using a list comprehension as:

names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach'] b_names = [ name for name in names if name.startswith('B') ] print b_names ['Bob', 'Barbara']

Or, again combining the first two lines into one,

b_names = [ name for name in ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach'] if name.startswith('B') ]

Map, Filter, Lambda, and List Comprehensions in Python¶

map(), filter(), lambda, and list comprehensions provide compact, elegant, and efficient ways to encode a few common idioms in programming. We often encounter the following scanarios involving for-loops:

>>> def square(x):
...     return x**2
...
>>> squares = map(square, range(10))
>>> print squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach']
>>> lengths = map(len, names)
>>> print lengths
[4, 3, 3, 5, 6, 7, 4]

>>> print (lambda x: x**2)(5) # first parens contain a lambda form that is a squaring function, second parens represent calling that function
25
>>> # Make a function of two arguments (x and y) that returns their product, then call the function with 3 and 4:
>>> print (lambda x, y: x*y)(3, 4)
12
>>> print (lambda x: x.startswith('B'))('Bob')
True
>>> print (lambda x: x.startswith('B'))('Robert')
False
>>> incrementer = lambda input: input+1
>>> # above is same as def incrementer(input): return input+1
>>> # now we call it
>>> print incrementer(3)
4

>>> squares = map(lambda x: x**2, range(10))
>>> print squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81

>>> squares = map(lambda x: x**2, range(10))
>>> special_squares = filter(lambda x: x > 5 and x < 50, squares)
>>> print special_squares
[9, 16, 25, 36, 49]

>>> names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach']
>>> b_names = filter(lambda s: s.startswith('B'), names)
>>> print b_names
['Bob', 'Barbara']

>>> squares = [ x**2 for x in range(10) ]
>>> print squares
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

>>> print [ [name, name[0], len(name)] for name in names ]
[['Anne', 'A', 4], ['Amy', 'A', 3], ['Bob', 'B', 3], ['David', 'D', 5], ['Carrie', 'C', 6], ['Barbara', 'B', 7], ['Zach', 'Z', 4]]

results = []
for outer_loop_variable in outer_sequence:
    for inner_loop_variable in inner_sequence:
        results.append( expression_involving_loop_variables )

>>> possible_choices = [ [drink,food] for drink in ['water', 'tea', 'juice'] for food in ['ham', 'eggs', 'spam'] ]
>>> print possible_choices
[['water', 'ham'], ['water', 'eggs'], ['water', 'spam'], ['tea', 'ham'], ['tea', 'eggs'], ['tea', 'spam'], ['juice', 'ham'], ['juice', 'eggs'], ['juice', 'spam']]

>>> coords = [ (x,y) for x in range(5) for y in range(3) ] # points on a rectangular grid
>>> print coords
[(0, 0), (0, 1), (0, 2), (1, 0), (1, 1), (1, 2), (2, 0), (2, 1), (2, 2), (3, 0), (3, 1), (3, 2), (4, 0), (4, 1), (4, 2)]

>>> special_squares = [ x**2 for x in range(10) if x**2 > 5 and x**2 < 50 ]
>>> print special_squares
[9, 16, 25, 36, 49]

>>> squares = [ x**2 for x in range(10) ]
>>> special_squares = [ s for s in squares if s > 5 and s < 50 ]

>>> special_squares = [ s for s in [ x**2 for x in range(10) ] if s > 5 and s < 50 ]

>>> names = ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach']
>>> b_names = [ name for name in names if name.startswith('B') ]
>>> print b_names
['Bob', 'Barbara']

>>> b_names = [ name for name in ['Anne', 'Amy', 'Bob', 'David', 'Carrie', 'Barbara', 'Zach'] if name.startswith('B') ]