We closed module 4 with a promise: the two limitations of EasyTask —that it only stores one task and that the data of that task lives in six separate variables— are solved with data structures. This lesson tackles the first and most urgent one: how to store many values under a single name.

The tool is called a list, and it is the data structure you will use most in your life as a programmer: it holds an ordered sequence of values, grows and shrinks on the fly, is traversed with a for loop, and can be sorted and filtered. As soon as you have it, Marta will be able to juggle the book fair poster, the Solé Bakery menu and the Vidal client logo all at once. And along the way we will tie up the loose ends of 04-05: sorted(key=...), map and filter needed a collection to work on, and now we have one.

1. From separate variables to the list
2. Classic array and Python list
3. Creating, measuring and indexing lists
4. Slicing: [start:stop:step]
5. Mutability, aliasing and copies
6. Essential methods and functions
7. Traversing: for, enumerate, zip and in
8. List comprehensions
9. sorted with key, map and filter
10. EasyTask v0.8: parallel lists
11. Common mistakes and tips
12. Exercises
13. Conclusion

To store the estimated days of three tasks, with what you know so far there is only one way out: days_1 = 3, days_2 = 5, days_3 = 2. It works and at the same time it is a dead end: you cannot traverse those variables with a loop (each name is different and they do not form any sequence), you cannot add a fourth task without editing the program, you cannot ask how many there are or sort them. A list turns those three variables into a single object with structure:

The rule that follows from this holds forever: when several variables are named the same except for a number, what you actually want is a list.

Programming literature is full of the word array, which is not exactly the same thing as a Python list. A classic array, as it exists in C, Java or Pascal, is a contiguous block of memory with two strong restrictions —fixed size and a single type of data— in exchange for speed and minimal memory consumption. The Python list is dynamic and heterogeneous.

The fact that ["Book fair poster", 3, True, 4.5] is legal does not mean it is a good idea. A homogeneous list —all elements of the same type and with the same meaning— is much easier to traverse, because inside the for you know what you are holding. If each position means a different thing, what you are looking for is a tuple (05-04) or a dictionary (05-03). And when you need millions of numbers and memory matters, there are the array module of the standard library and the external library NumPy, widely used in numerical computing: we do not develop them here, but it is worth knowing they exist.

There are three usual ways to create a list, and len() to measure it:

The empty list is the starting point of the accumulator pattern from 03-02, only now you accumulate elements instead of adding up numbers. len() is the same function you used with strings: both are sequences. Every element has a position, and that position starts at zero:

A list of three elements has indices 0, 1 and 2: the last valid index is always len(list) - 1. Negative indices count from the end and save you from computing lengths: in this list, -1 is the same element as 2, -2 the same as 1 and -3 the same as 0. That is why titles[-1] is the idiomatic way of saying "the last one", far better than titles[len(titles) - 1]. There is no -0, because -0 is 0.

A slice extracts a chunk and returns a new list. The three numbers are optional. The key rule, and the eternal source of confusion: start is included, stop is NOT. The advantage of the convention is that stop - start gives you directly how many elements come out. The examples start from days = [3, 5, 2, 8, 1, 6].

Two important details. A slice never raises IndexError even if you overshoot: days[2:99] returns whatever is there. And days[:] looks redundant, but it is the classic way of getting a shallow copy.

Lists are mutable: you can change their content without creating a new list, something you could not do with strings (02-01). priorities[1] = "high" replaces the second element and that is that. So far, convenient. Now the part that surprises everyone:

We modified copy and originals changed. The reason is the one you studied in 04-03 with references: copy = originals does not copy the list, it copies the reference; both names point to the same object, and that is called aliasing. To copy for real there are three equivalent ways, which you will see below the diagram.

This has a direct consequence on functions, and it is new with respect to 04-03. There we saw that passing an immutable argument does not let the function touch the caller's variable. With a list it is the other way round:

The function receives a reference to the same list, so its changes are visible outside. It is extremely powerful —that is how v0.8 will add tasks— and also dangerous: a function that silently modifies someone else's list causes surprises. Practical rule: either it modifies and says so in its name (add_task, sort_agenda), or it works on a copy and returns the result. And notice the contrast between clear_wrong and clear_right: reassigning the parameter only changes the local name, whereas calling a method changes the object both of them share.

Methods are called on the list with a dot and almost all of them modify it in place; built-in functions are called with the list inside the parentheses and return a value without touching it.

Decisive difference: pop deletes by position and returns the element; remove deletes by value and returns nothing. And the number one trap of lists is that last line: the methods that modify return None, so never store their result. Built-in functions, on the other hand, return something new and do not touch the list:

The rule for choosing: .sort() if you want the list to end up sorted; sorted() if you need to preserve the original order. Both accept reverse=True. How they sort internally and how much it costs is the subject of Sorting algorithms and Efficiency and Big-O notation; here we use them as tools of the language.

The for loop from 03-02 was made for this. Traversing the list directly —for title in titles:— is the preferred form in Python: on each pass, title takes the value of one element, with no indices, no len() and no chance of IndexError. When you do need the position —to number a listing, for example— use enumerate(), which produces two values per pass:

start=1 makes the numbering the user sees begin at 1 even though the internal indices still start at 0: exactly what a menu needs. When you have two parallel lists, zip() pairs them up:

zip stops with the shortest list without warning: if one has an extra element, that element disappears from the traversal. The last two lines use the in operator from 02-02 to check membership. How in searches internally, and why in a long list it takes longer than in a set, is covered in Search algorithms and in Dictionaries and sets.

This pattern shows up constantly: create an empty list (doubles = []), traverse another sequence and add something on each pass (doubles.append(d * 2)). Python offers a condensed way of writing exactly that, the list comprehension, which reads from the inside out: for each d in days, compute d * 2 and put it in the list.

And now the part that is almost never told: when NOT to use them. Do not use one if the expression does not fit comfortably on a single line, if you need nested conditions, if inside you have to print or ask the user for data, or if you chain two or three for clauses. Something like [t.upper() if len(t) > 10 else t.lower() for t in titles if "e" in t] is legal and it is unreadable: it is crying out for a normal loop. The goal was never to write less, but to read better.

In 04-05 we left higher order over collections pending. We can close it now. sorted(items, key=function) sorts according to what function returns for each element, not according to the element itself:

With a lambda you can invent any criterion you like. Sorting the priorities alphabetically would give high, low, medium, which is not their logical order; instead PRIORITIES.index(p) returns 0 for "high", 1 for "medium" and 2 for "low", and sorting by that number gives the correct order. It is exactly what we saw in 04-05: sorted knows nothing about priorities, only about sorting; the criterion is injected from outside. And key=str.lower sorts ignoring case, while reverse=True flips the result.

map(function, items) applies a function to every element and filter(function, items) keeps those for which it returns True. Both return a lazy object that has to be converted with list():

In modern Python the comprehension is preferred: shorter, without list() and more readable. map and filter are still useful when the function already exists and has a name (map(str.upper, titles) is beautifully clean), and you must recognise them because they abound in other people's code.

v0.7 stored one task in six separate variables. v0.8 stores many tasks in five parallel lists: the element at index i of each list refers to the same task.

The input functions (ask_text, ask_option, ask_integer, confirm), the logic one (classify_urgency) and show_menu/pause are kept untouched. What changes are the action functions and main():

What has changed and why:

• register_task no longer returns anything: it modifies the lists it receives. It can do so because lists are mutable (section 5); the name and the docstring announce it.
• choose_index translates the number the user sees into the internal index by subtracting 1. That - 1 is the border between "first task" for Marta and "index 0" for Python, and it is best kept in a single place.
• enumerate(..., start=1) numbers the listing with no manual counters and completed.count(True) counts the finished ones with no loop. Besides, the has_task flag disappears: len(titles) == 0 answers the same question and on top of that tells you how many there are.

And now the new problem we have created. The five lists must be in sync at all times: if somebody adds a field and forgets an append, they get misaligned and the assignee of one task shows up next to the title of another. There is no error and no warning: the program keeps working and lying. Deleting a task requires five coordinated pop calls, and adding a field forces you to touch the signature of every function. The five parallel lists are, in truth, a single list of tasks that we do not yet know how to write.

Confusing index and human position. The third element is items[2]. This off-by-one gap is the most frequent mistake in the trade: when printing for the user, enumerate(..., start=1); when accessing, subtract 1 in a single place. Expecting .sort() to return something. ordered = items.sort() leaves ordered holding None; if you need a new list, sorted(items). The same goes for append, reverse, insert and clear. And .remove(x) and .index(x) with a value that is not there raise ValueError: check first with if x in items:.

Believing that copy = items copies. It copies the reference. Use .copy(), list() or [:], and check with is if you have doubts. And do not delete elements while you traverse the list: if inside a for you call items.remove(x), the traversal skips elements because the positions shift under your feet. The safe solution is to build a new list with the ones you want to keep, usually with a comprehension.

Tip: use the direct for whenever you can. for t in titles: is clearer and safer than for i in range(len(titles)):. And keep your lists homogeneous: a list contains "many things of the same kind", not "one thing split into pieces".

Say what each print prints and explain why.

With the lists below, print: the number of tasks, the total and the average number of days, the title of the longest task, the titles with priority "high" and the titles sorted by days from highest to lowest. Use comprehensions and sorted(key=...) where they fit.

Write remove_task(titles, assignees, priorities, days, title), which deletes from the four lists the task with that title without misaligning them and warns if it does not exist. Then explain, in two sentences, why this function is awkward to write and to maintain.

Solution 1.

days[2] is the third element, days[-1] the last one and days[1:3] includes indices 1 and 2 but not 3. copy = days is aliasing: the append affects the only list there is and len(days) goes from 6 to 7. other = days[:] really is a copy, so emptying it does not touch the original: 7 and 0. And .sort() sorts in place returning None; after sorting, the first element is the smallest, 1.

Solution 2.

The result is 4 tasks, 18 days in total, an average of 4.50, "Summer flyer" as the longest and two of high priority. days.index(max(days)) gives the position of the maximum, and that position lets you read the corresponding title: that is how parallel lists work. Notice how convoluted the last line ends up: to sort by days you have to look up each title in its own list to find out its index. With a list of dictionaries (05-04) it will be sorted(agenda, key=lambda t: t["days"]) and that is that.

Solution 3.

It is awkward for two reasons. First, it receives four lists just to delete one task: if tomorrow you add a field, you have to change the signature and every call. Second, its correctness depends on the four pop calls always running together and with the same index; forgetting just one is enough for every later task to end up scrambled with no error to give it away.

A list stores many values under a single name and solves what separate variables could not: counting, traversing, adding up, sorting and growing at run time. It is accessed by index from 0, with negative indices to count from the end and IndexError if you overshoot; it is cut up with slices [start:stop:step], where the stop is never included. It is mutable, and hence the aliasing: copy = items copies nothing, and to copy for real there are .copy(), list() and [:]. Its methods modify in place and return None, whereas len, sum, min, max and sorted return a new value. It is traversed with for, with enumerate when you need the position and with zip when there are several lists; in checks membership. Comprehensions condense the "create, traverse, add" pattern as long as the line stays readable. And with sorted(key=...), map and filter we have closed the higher-order chapter from 04-05.

EasyTask has reached v0.8 and at last it manages the studio's real work: it registers as many tasks as needed, lists them numbered and lets you change the priority or complete the nth one. But it has achieved this with five parallel lists that must be synchronised by hand, and you have already seen how fragile that is: one forgotten pop and the listing starts lying silently. The problem is not the list, it is that we are using it for something that is not its job: a list stores many things of the same kind, and what we need is to group the different fields of a single task under one name. That arrives in Dictionaries and sets and is rounded off in Tuples and nested structures. Before that, in Strings, we are going to take a close look at the other sequence you have been using since module 2 without seeing it as such —text— and squeeze its methods dry: the titles, the assignees and the task codes of Alba Studio are all strings, and there is a lot to get out of them.