Silo - Tips - An Introduction To Python Programming For Research

An introduction to Python Programming for Research
James Hetherington
November 4, 2015
Contents
1 Introduction 15
1.1 Why teach Python? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.1.1 Why Python? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.1.2 Why write programs for research? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.1.3 Sensible Input - Reasonable Output . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2 Many kinds of Python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.1 The IPython Notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15
1.2.2 Typing code in the notebook . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.3 Python at the command line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.4 Python scripts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17
1.2.5 Python Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18
2 An example Python data analysis notebook 20

2.1 Why write software to manage your data and plots? . . . . . . . . . . . . . . . . . . . . . . . 20
2.2 The example program: a “graph of green land”. . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.1 Importing Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20
2.2.2 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.3 Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.4 Variables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.5 More complex functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21
2.2.6 Checking our work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.7 Displaying results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
2.2.8 Manipulating Numbers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23
2.2.9 Creating Images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24
2.2.10 Looping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26
2.2.11 Plotting graphs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
2.2.12 Composing Program Elements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31
3 Variables 33
3.1 Variable Assignment . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33
3.2 Reassignment and multiple labels . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.3 Objects and types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35
3.4 Reading error messages. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
3.5 Variables and the notebook kernel . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 36
4 Using Functions 37
4.1 Calling functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.2 Using methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 37
4.3 Functions are just a type of object! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38
4.4 Getting help on functions and methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39
4.5 Operators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 41
1
5 Types 43
5.1 Floats and integers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
5.2 Strings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44
5.3 Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45
5.4 Sequences . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
5.5 Unpacking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 46
6 Containers 47
6.1 Checking for containment. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.2 Mutability . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 47
6.3 Tuples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.4 Memory and containers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48
6.5 Identity vs Equality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 49
7 Dictionaries 51
7.1 The Python Dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.2 Keys and Values . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 51
7.3 Immutable Keys Only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.4 No guarantee of order . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52
7.5 Sets . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
8 Data structures 54
8.1 Nested Lists and Dictionaries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 54
8.2 Exercise: a Maze Model. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
8.2.1 Solution: my Maze Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55
9 Control and Flow 57

9.1 Turing completeness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.2 Conditionality . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.3 Else and Elif . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57
9.4 Comparison . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58
9.5 Automatic Falsehood . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 59
9.6 Indentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
9.7 Pass . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
9.8 Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 60
9.9 Iterables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
9.10 Dictionaries are Iterables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
9.11 Unpacking and Iteration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 61
9.12 Break, Continue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
9.13 Classroom exercise: the Maze Population . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62
9.13.1 Solution: counting people in the maze . . . . . . . . . . . . . . . . . . . . . . . . . . . 63
10 Comprehensions 64
10.1 The list comprehension . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
10.2 Selection in comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 64
10.3 Comprehensions versus building lists with append: . . . . . . . . . . . . . . . . . . . . . . . . 64
10.4 Nested comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.5 Dictionary Comprehensions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.6 List-based thinking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 65
10.7 Classroom Exercise: Occupancy Dictionary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
10.7.1 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 66
2
11 Functions 68
11.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
11.2 Default Parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
11.3 Side effects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 68
11.4 Early Return . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
11.5 Unpacking arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 69
11.6 Sequence Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
11.7 Keyword Arguments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 70
12 Using Libraries 71
12.1 Import . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 71
12.2 Why bother? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
12.3 Importing from modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
12.4 Import and rename . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 73
13 Defining your own classes 74

13.1 User Defined Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
13.2 Methods . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 74
13.3 Constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
13.4 Object-oriented design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 75
13.5 Object oriented design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78
13.6 Exercise: Your own solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 81
14 Python not in the Notebook 82

14.1 Writing Python in Text Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 82
14.2 Loading Our Package . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 84
14.3 The Python Path . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
15 Loading data from files 86

15.1 Loading data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
15.2 An example datafile . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 86
15.3 Path independence and os . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
15.4 The python file type . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
15.5 Working with files. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 88
15.6 Converting Strings to Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 89
15.7 Closing files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
15.8 Writing files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 90
16 Getting data from the Internet 92

16.1 URLs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
16.2 Requests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
16.3 Example: Sunspots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
16.4 Writing our own Parser . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 93
16.5 Writing data to the internet . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 94
17 Field and Record Data 95

17.1 Separated Value Files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
17.2 CSV variants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
17.3 Python CSV readers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 96
17.4 Naming Columns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 97
17.5 Typed Fields . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
3
18 Structured Data 99
18.1 Structured data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
18.2 Json . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99
18.3 Unicode . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
18.4 Yaml . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
18.5 XML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
18.6 Exercise: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
18.7 Solution: Saving and Loading a Maze . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 101
19 Unassessed Exercise: the biggest Earthquake in the UK this Century 103

19.1 The Problem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
19.2 Download the data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 103
19.3 Parse the data as JSON . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
19.4 Investigate the data to discover how it is structured. . . . . . . . . . . . . . . . . . . . . . . . 104
19.5 Find the largest quake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 104
19.6 Get a map at the point of the quake . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
19.7 Display the map . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
20 Plotting with Matplotlib 107

20.1 Importing Matplotlib . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
20.2 Notebook magics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
20.3 A basic plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 107
20.4 Figures and Axes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
20.5 Saving figures. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 113
20.6 Subplots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 114
20.7 Versus plots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
20.8 Learning More . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
21 NumPy 118
21.1 The Scientific Python Trilogy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
21.2 Limitations of Python Lists . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 118
21.3 The NumPy array . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 119
21.4 Elementwise Operations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
21.5 Arange and linspace . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
21.6 Multi-Dimensional Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 121
21.7 Array Datatypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
21.8 Broadcasting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 124
21.9 Newaxis . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
21.10Dot Products . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
21.11Array DTypes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127
21.12Record Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 128
21.13Logical arrays, masking, and selection . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
21.14Numpy memory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
22 The Boids! 130

22.1 Flocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
22.2 Setting up the Boids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 130
22.3 Flying in a Straight Line . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
22.4 Matplotlib Animations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 131
22.5 Fly towards the middle . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133
22.6 Avoiding collisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 134
22.7 Match speed with nearby birds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
4
23 Understanding the “Greengraph” Example 137
23.1 Classes for Greengraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 137
23.2 Invoking our code and making a plot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
23.3 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
23.3.1 What’s version control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
23.3.2 Why use version control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139
23.3.3 Git != GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.3.4 How do we use version control? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.3.5 What is version control? (Team version) . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.3.6 Scope . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.4 Practising with Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.4.1 Example Exercise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.4.2 Programming and documents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 140
23.4.3 Markdown . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
23.4.4 Displaying Text in this Tutorial . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
23.4.5 Setting up somewhere to work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 141
23.5 Solo work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
23.5.1 Configuring Git with your name and email . . . . . . . . . . . . . . . . . . . . . . . . 142
23.5.2 Initialising the repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
23.6 Solo work with Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
23.6.1 A first example file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 142
23.6.2 Telling Git about the File . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
23.6.3 Our first commit . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
23.6.4 Configuring Git with your editor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 143
23.6.5 Git log . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
23.6.6 Hash Codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
23.6.7 Nothing to see here . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 144
23.6.8 Unstaged changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
23.6.9 Staging a file to be included in the next commit . . . . . . . . . . . . . . . . . . . . . 145
23.6.10 The staging area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 145
23.6.11 Message Sequence Charts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146
23.6.12 The Levels of Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
23.6.13 Review of status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 147
23.6.14 Carry on regardless . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
23.6.15 Commit with a built-in-add . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148
23.6.16 Review of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
23.6.17 Git Solo Workflow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 149
23.7 Fixing mistakes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
23.7.1 Referring to changes with HEAD and ˆ . . . . . . . . . . . . . . . . . . . . . . . . . . 151
23.7.2 Reverting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
23.7.3 Conflicted reverts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
23.7.4 Review of changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 151
23.7.5 Antipatch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
23.7.6 Rewriting history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
23.7.7 A new lie . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 152
23.7.8 Using reset to rewrite history . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
23.7.9 Covering your tracks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
23.7.10 Resetting the working area . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 153
23.8 Publishing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
23.8.1 Sharing your work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 156
23.8.2 Creating a repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
23.8.3 Paying for GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
23.8.4 Adding a new remote to your repository . . . . . . . . . . . . . . . . . . . . . . . . . . 157
23.8.5 Remotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 157
5
23.8.6 Playing with GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
23.9 Working with multiple files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
23.9.1 Some new content . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 158
23.9.2 Git will not by default commit your new file . . . . . . . . . . . . . . . . . . . . . . . . 159
23.9.3 Tell git about the new file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 159
23.10Changing two files at once . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 160
23.11Collaboration . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
23.11.1 Form a team . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
23.11.2 Giving permission . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
23.11.3 Obtaining a colleague’s code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 162
23.11.4 Nonconflicting changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 164
23.11.5 Rejected push . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 165
23.11.6 Merge commits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 166
23.11.7 Nonconflicted commits to the same file . . . . . . . . . . . . . . . . . . . . . . . . . . . 167
23.11.8 Conflicting commits . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 173
23.11.9 Resolving conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
23.11.10Commit the resolved file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 175
23.11.11Distributed VCS in teams with conflicts . . . . . . . . . . . . . . . . . . . . . . . . . . 177
23.11.12The Levels of Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 178
23.12Editing directly on GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.12.1 Editing directly on GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.13Social Coding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.13.1 GitHub as a social network . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.14Fork and Pull . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.14.1 Different ways of collaborating . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 179
23.14.2 Forking a repository on GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
23.14.3 Pull Request . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
23.14.4 Practical example - Team up! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 180
23.14.5 Some Considerations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
23.15Git Theory . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
23.15.1 The revision Graph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 182
23.15.2 Git concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
23.15.3 The levels of Git . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
23.16Branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 183
23.16.1 Publishing branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 184
23.16.2 Find out what is on a branch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 185
23.16.3 Merging branches . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
23.16.4 Cleaning up after a branch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 186
23.16.5 A good branch strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
23.16.6 Grab changes from a branch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
23.17Git Stash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 187
23.18Tagging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 188
23.19Working with generated files: gitignore . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 189
23.20Git clean . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191
23.21Hunks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
23.21.1 Git Hunks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
23.21.2 Interactive add . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
23.22GitHub pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
23.22.1 Yaml Frontmatter . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192
23.22.2 The gh-pages branch . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
23.22.3 UCL layout for GitHub pages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
23.23Working with multiple remotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
23.23.1 Distributed versus centralised . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 193
23.23.2 Referencing remotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
6
23.24Hosting Servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
23.24.1 Hosting a local server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 195
23.24.2 Home-made SSH servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
23.25SSH keys and GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
23.26Rebasing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
23.26.1 Rebase vs merge . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
23.26.2 An example rebase . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 197
23.26.3 Fast Forwards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 198
23.26.4 Rebasing pros and cons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
23.27Squashing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
23.27.1 Using rebase to squash . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 199
23.28Debugging With Git Bisect . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
23.28.1 An example repository . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 200
23.28.2 Solving Manually . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 201
23.28.3 Solving automatically . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 202
23.29Testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
23.29.1 A few reasons not to do testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
23.29.2 A few reasons to do testing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
23.29.3 Not a panacea . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 203
23.30Testing primer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
23.30.1 Tests at different scales . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
23.30.2 Legacy code hardening . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
23.30.3 Testing vocabulary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
23.30.4 Branch coverage: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 204
24 How to Test 205

24.0.5 Equivalence partitioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 205
24.1 Using our tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 210
24.1.1 Boundary cases . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
24.2 Positive and negative tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
24.3 Raising exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 212
24.4 Testing frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
24.4.1 Why use testing frameworks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
24.4.2 Common testing frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 213
24.4.3 Nose framework: usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 214
24.5 Testing with floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
24.5.1 Floating points are not reals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 215
24.5.2 Comparing floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216
24.5.3 Comparing vectors of floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
24.6 Classroom exercise: energy calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
24.6.1 Diffusion model in 1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 217
24.6.2 Starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 218
24.6.3 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 219
24.6.4 Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
24.7 Mocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
24.7.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
24.7.2 Mocking frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 221
24.7.3 Recording calls with mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 222
24.8 Using mocks to model test resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 223
24.8.1 Testing functions that call other functions . . . . . . . . . . . . . . . . . . . . . . . . . 225
24.9 Using a debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
24.9.1 Stepping through the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
24.9.2 Using the python debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
24.9.3 Basic navigation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
7
24.9.4 Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226
24.9.5 Post-mortem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 227
24.10Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
24.10.1 Test servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
24.10.2 Memory and profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
24.11Extended TDD Example: Monte-Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
24.11.1 Problem: Implement and test a simple Monte-Carlo algorithm . . . . . . . . . . . . . 231
24.12Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 231
24.13Testing frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
24.13.1 Why use testing frameworks? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
24.13.2 Common testing frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 232
24.13.3 Nose framework: usage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 233
24.14Testing with floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
24.14.1 Floating points are not reals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 234
24.14.2 Comparing floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 235
24.14.3 Comparing vectors of floating points . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
24.15Classroom exercise: energy calculation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
24.15.1 Diffusion model in 1D . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 236
24.15.2 Starting point . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 237
24.15.3 Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 238
24.15.4 Coverage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 239
24.16Mocking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
24.16.1 Definition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
24.16.2 Mocking frameworks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
24.16.3 Recording calls with mock . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 240
24.17Using mocks to model test resources . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241
24.17.1 Testing functions that call other functions . . . . . . . . . . . . . . . . . . . . . . . . . 243
24.18Using a debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
24.18.1 Stepping through the code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
24.18.2 Using the python debugger . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
24.18.3 Basic navigation: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 244
24.18.4 Breakpoints . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
24.18.5 Post-mortem . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 245
24.19Jenkins . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
24.19.1 Test servers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
24.19.2 Memory and profiling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
24.20Extended TDD Example: Monte-Carlo . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
24.20.1 Problem: Implement and test a simple Monte-Carlo algorithm . . . . . . . . . . . . . 249
24.21Solution . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 249
25 Installing Libraries 256

25.1 Installing Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
25.2 Installing Geopy using Pip . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 256
25.3 Installing binary dependencies with Conda . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
25.4 Where do these libraries go? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
25.5 Libraries not in PyPI . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 257
25.6 Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
25.6.1 Libraries are awesome . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
25.7 Careful use of Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
25.7.1 Drawbacks of libraries. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
25.7.2 Contribute, don’t duplicate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 258
25.7.3 How to choose a library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
25.7.4 Sensible Version Numbering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
25.8 Python Libraries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
8
25.8.1 The Python Standard Library . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
25.8.2 The Python Package Index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
25.9 Argparse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 259
25.10Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
25.10.1 Packaging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
25.10.2 Distribution tools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 260
25.10.3 Laying out a project . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
25.10.4 Using setuptools . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 261
25.10.5 Installing from GitHub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
25.10.6 Convert the script to a module . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 262
25.10.7 Write an executable script . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 263
25.10.8 Write an entry point script stub . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
25.10.9 Write a readme file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
25.10.10Write a license file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
25.10.11Write a citation file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
25.10.12Define packages and executables . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 264
25.10.13Write some unit tests . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
25.10.14Developer Install . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
25.10.15Distributing compiled code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 265
25.10.16Homebrew . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
25.10.17Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
25.11Documentation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
25.11.1 Documentation is hard . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 266
25.11.2 Prefer readable code with tests and vignettes . . . . . . . . . . . . . . . . . . . . . . . 266
25.11.3 Comment-based Documentation tools . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25.12Example of using Sphinx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25.12.1 Write some docstrings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25.12.2 Set up sphinx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 267
25.12.3 Define the root documentation page . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.12.4 Run sphinx . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.12.5 Sphinx output . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.13Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.13.1 Software Engineering Stages . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.13.2 Requirements Engineering . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 269
25.13.3 Functional and architectural design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
25.13.4 Waterfall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
25.13.5 Why Waterfall? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
25.13.6 Problems with Waterfall . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
25.13.7 Software is not made of bricks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 270
25.13.10The Agile Manifesto . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.13.11Agile is not absence of process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.13.12Elements of an Agile Process . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.13.13Ongoing Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 271
25.13.14Iterative Development . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.13.15Continuous Delivery . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.13.16Self-organising teams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.13.17Agile in Research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.13.18Conclusion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 272
25.14Exercises . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.14.1 Refactoring to classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.14.2 Refactoring to Inheritance and Polymorphism . . . . . . . . . . . . . . . . . . . . . . . 273
25.14.3 Refactoring to Patterns: Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
9
25.14.4 Refactoring to Patterns: Model/View . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.14.5 Using UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.15Software Licensing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.15.1 Reuse . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.15.2 Disclaimer . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 273
25.15.3 Choose a license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.15.4 Open source doesn’t stop you making money . . . . . . . . . . . . . . . . . . . . . . . 274
25.15.5 Plagiarism vs promotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.15.6 Your code is good enough . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.15.7 Worry about license compatibility and proliferation . . . . . . . . . . . . . . . . . . . . 274
25.15.8 Academic license proliferation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 274
25.15.9 Licenses for code, content, and data. . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
25.15.10Licensing issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
25.15.11Permissive vs share-alike . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
25.15.12Academic use only . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
25.15.13Patents . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 275
25.15.14Use as a web service . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.15.15Library linking . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.15.16Citing software . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.15.17Referencing the license in every file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.15.18Choose a license . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 276
25.15.19Open source does not equal free maintenance . . . . . . . . . . . . . . . . . . . . . . . 276
25.16Managing software issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.1 Issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.2 Some Issue Trackers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.3 Anatomy of an issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.4 Reporting a Bug . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.5 Owning an issue . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.6 Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.7 Resolutions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 277
25.16.8 Bug triage . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
25.16.9 The backlog . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
25.16.10Development cycles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
25.16.11GitHub issues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
25.16.12Exercise - Packaging Greengraph . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 278
26 Construction 279
26.1 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
26.1.1 Construction vs Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
26.1.2 Low-level design decisions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
26.1.3 Algorithms and structures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
26.1.4 Architectural design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 279
26.1.5 Construction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
26.1.6 Literate programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
26.1.7 Programming for humans . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
26.2 Setup . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 280
26.3 Coding Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
26.3.1 One code, many layouts: . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 281
26.3.2 So many choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
26.3.3 Layout . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
26.3.4 Layout choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
26.3.5 Naming Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
26.3.6 Hungarian Notation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 282
26.3.7 Newlines . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
10
26.3.8 Syntax Choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.3.9 Syntax choices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.3.10 Coding Conventions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.3.11 Lint . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.4 Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.4.1 Why comment? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.4.2 Bad Comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283
26.4.3 Comments which are obvious . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
26.4.4 Comments which could be replaced by better style . . . . . . . . . . . . . . . . . . . . 284
26.4.5 Comments vs expressive code . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 284
26.4.6 Comments which belong in an issue tracker . . . . . . . . . . . . . . . . . . . . . . . . 284
26.4.7 Comments which only make sense to the author today . . . . . . . . . . . . . . . . . . 285
26.4.8 Comments which are unpublishable . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.5 Good comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.5.1 Pedagogical comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.5.2 Other good comments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.6 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.6.1 Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.6.2 A word from the Master . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 285
26.6.3 List of known refactorings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
26.6.4 Replace magic numbers with constants . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
26.6.5 Replace repeated code with a function . . . . . . . . . . . . . . . . . . . . . . . . . . . 286
26.6.6 Change of variable name . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
26.6.7 Separate a complex expression into a local variable . . . . . . . . . . . . . . . . . . . . 287
26.6.8 Replace loop with iterator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 287
26.6.9 Replace hand-written code with library code . . . . . . . . . . . . . . . . . . . . . . . 287
26.6.10 Replace set of arrays with array of structures . . . . . . . . . . . . . . . . . . . . . . . 288
26.6.11 Replace constants with a configuration file . . . . . . . . . . . . . . . . . . . . . . . . . 288
26.6.12 Replace global variables with function arguments . . . . . . . . . . . . . . . . . . . . . 288
26.6.13 Merge neighbouring loops . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.6.14 Break a large function into smaller units . . . . . . . . . . . . . . . . . . . . . . . . . . 289
26.6.15 Separate code concepts into files or modules . . . . . . . . . . . . . . . . . . . . . . . . 289
26.6.16 Refactoring is a safe way to improve code . . . . . . . . . . . . . . . . . . . . . . . . . 290
26.6.17 Tests and Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
26.6.18 Refactoring Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 290
26.7 Introduction to Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.1 Classes: User defined types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.2 Declaring a class . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.3 Object instances . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.4 Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.5 Constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.7.6 Member Variable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 291
26.8 Object refactorings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
26.8.1 Replace add-hoc structure with user defined classes . . . . . . . . . . . . . . . . . . . . 292
26.8.2 Replace function with a method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 292
26.8.3 Replace method arguments with class members . . . . . . . . . . . . . . . . . . . . . . 293
26.8.4 Replace global variable with class and member . . . . . . . . . . . . . . . . . . . . . . 293
26.8.5 Object Oriented Refactoring Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.9 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.9.1 Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.9.2 Object-Oriented Design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.9.3 Design processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.9.4 Design and research . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.10More on Objects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
11
26.10.1 Object Based Programming Recap . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.10.2 Class design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 294
26.10.3 UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
26.10.4 YUML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 295
26.11Information Hiding . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 296
26.11.1 Property accessors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
26.11.2 Class Members . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 297
26.11.3 Object-based vs Object-Oriented . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
26.12Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
26.12.1 Ontology and inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
26.12.2 Inheritance in python . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 298
26.12.3 Inheritance terminology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
26.12.4 Inheritance and constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
26.12.5 Inheritance UML diagrams . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
26.12.6 Aggregation vs Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
26.12.7 Aggregation in UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299
26.12.8 Refactoring to inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
26.13Polymorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
26.13.1 Polymorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 300
26.13.2 Polymorphism and Inheritance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 301
26.13.3 Undefined Functions and Polymorphism . . . . . . . . . . . . . . . . . . . . . . . . . . 301
26.13.4 Refactoring to Polymorphism . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
26.13.5 Interfaces and concepts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
26.13.6 Interfaces in UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 302
26.13.7 Further UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.14Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.14.1 Class Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.14.2 Design Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.14.3 Reading a pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.14.4 Introducing Some Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 303
26.15Factory Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
26.15.1 Factory Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
26.15.2 Factory UML . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
26.15.3 Factory Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
26.15.4 Agent model constructor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 304
26.15.5 Agent derived classes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
26.15.6 Refactoring to Patterns . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
26.16Builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
26.16.1 Builder Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 305
26.16.2 Builder example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
26.16.3 Builder preferred to complex constructor . . . . . . . . . . . . . . . . . . . . . . . . . 306
26.16.4 Using a builder . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 306
26.16.5 Avoid staged construction without a builder. . . . . . . . . . . . . . . . . . . . . . . . 306
26.16.6 Builder Message Sequence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
26.17Strategy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
26.18Strategy Pattern . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
26.18.1 Strategy pattern example: sunspots . . . . . . . . . . . . . . . . . . . . . . . . . . . . 307
26.18.2 Sunspot cycle has periodicity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 308
26.18.3 Years are not constant length . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
26.18.4 Uneven time series . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
26.18.5 Uneven time series design . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 309
26.18.6 Strategy Pattern for Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 310
12
26.18.10Strategy Pattern for Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 311
26.18.14Comparison of different algorithms for frequency spectrum of sunspots. . . . . . . . . 312
26.18.15Deviation of year length from average . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
26.19Model-View-Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
26.19.1 Separate graphics from science! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
26.19.2 Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 313
26.19.3 View . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
26.19.4 Controller . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 314
26.20Exercise: Refactoring The Bad Boids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
26.20.1 Bad Boids . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 315
26.20.2 Your Task . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
26.20.3 A regression test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
26.20.4 A regression test . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
26.20.5 Make the regression test fail . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
26.20.6 Start Refactoring . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 317
27 Advanced Python Programming 318

27.1 Avoid Boiler-Plate . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
27.2 Functional programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
27.2.1 Functional Programming . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 318
27.2.2 Closures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
27.2.3 Map and Reduce . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 321
27.3 Lambda Functions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 323
27.4 Using functional programming for numerical methods . . . . . . . . . . . . . . . . . . . . . . 324
27.5 Iterators and Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 325
27.6 Iterators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 326
27.7 Defining Our Own Iterable . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 328
27.7.1 A shortcut to iterables: the iter method. . . . . . . . . . . . . . . . . . . . . . . . 329
27.7.2 Generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 330
27.8 Context managers and decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
27.8.1 Context managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331
27.8.2 Decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 332
27.8.3 Test generators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
27.8.4 Negative test contexts managers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 334
27.8.5 Negative test decorators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 335
27.9 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
27.9.1 Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 336
27.9.2 Design with Exceptions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 341
27.10Operator overloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
27.10.1 Operator overloading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 343
27.11Metaprogramming Example . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 347
27.11.1 Metaprogramming class attributes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 349
27.11.2 Metaprogramming function locals . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
27.11.3 Metaprogramming warning! . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 351
28 Performance programming 352

28.1 Two Mandelbrots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 352
28.2 Many Mandelbrots . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 355
13
29 NumPy 357
29.1 NumPy constructors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 357
29.2 Arraywise Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 359
30 Logical Arrays 363

30.1 Broadcasting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 364
30.2 More Mandelbrot . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 365
31 NumPy Testing 367
32 Arraywise operations are fast 369
33 Indexing with arrays 371
34 Scaling for containers and algorithms 375
35 Dictionary performance 381
36 Cython 385
36.1 Start Coding in Cython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 385
36.2 Cython with C Types . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
36.3 Cython with numpy ndarray . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 387
36.4 Calling C functions from Cython . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 389
14
Chapter 1
Introduction
1.1 Why teach Python?

• In this first session, we will introduce Python.
• This course is about programming for data analysis and visualisation in research.
• It’s not mainly about Python.
• But we have to use some language.
1.1.1 Why Python?

• Python is quick to program in
• Python is popular in research, and has lots of libraries for science
• Python interfaces well with faster languages
• Python is free, so you’ll never have a problem getting hold of it, wherever you go.
1.1.2 Why write programs for research?

• Not just labour saving
• Scripted research can be tested and reproduced
1.1.3 Sensible Input - Reasonable Output

Programs are a rigorous way of describing data analysis for other researchers, as well as for computers.
Computational research suffers from people assuming each other’s data manipulation is correct. By
sharing codes, which are much more easy for a non-author to understand than spreadsheets, we can avoid
the “SIRO” problem. The old saw “Garbage in Garbage out” is not the real problem for science:
• Sensible input
• Reasonable output
1.2 Many kinds of Python

1.2.1 The IPython Notebook
The easiest way to get started using Python, and one of the best for research data work, is the IPython
Notebook.
In the notebook, you can easily mix code with discussion and commentary, and mix code with the results
of that code; including graphs and other data visualisations.
15
In [1]: ### Make plot
%matplotlib inline
import numpy as np
import math
import matplotlib.pyplot as plt
theta=np.arange(0,4*math.pi,0.1)
eight=plt.figure()
axes=eight.add_axes([0,0,1,1])
axes.plot(0.5*np.sin(theta),np.cos(theta/2))
Out[1]: [<matplotlib.lines.Line2D at 0x1061cea10>]
We’re going to be mainly working in the IPython notebook in this course. To get hold of a copy of the
notebook, follow the setup instructions shown on the course website, or use the installation in UCL teaching
cluster rooms.
IPython notebooks consist of discussion cells, referred to as “markdown cells”, and “code cells”, which
contain Python. This document has been created using IPython notebook, and this very cell is a Markdown
Cell.
In [2]: print "This cell is a code cell"
This cell is a code cell
Code cell inputs are numbered, and show the output below.
Markdown cells contain text which uses a simple format to achive pretty layout, for example, to obtain:
bold, italic
• Bullet
16
Quote
We write:
**bold**, *italic*
* Bullet
> Quote
See the Markdown documentation at This Hyperlink
1.2.2 Typing code in the notebook

When working with the notebook, you can either be in a cell, typing its contents, or outside cells, moving
around the notebook.
• When in a cell, press escape to leave it. When moving around outside cells, press return to enter.
• Outside a cell:
• Use arrow keys to move around.
• Press b to add a new cell below the cursor.
• Press m to turn a cell from code mode to markdown mode.
• Press shift+enter to calculate the code in the block.
• Press h to see a list of useful keys in the notebook.
• Inside a cell:
• Press tab to suggest completions of variables. (Try it!)
Supplementary material : Learn more about the notebook here. Try these videos
1.2.3 Python at the command line

Data science experts tend to use a “command line environment” to work. You’ll be able to learn this at our
“Software Carpentry” workshops, which cover other skills for computationally based research.
In [3]: %%bash
# Above line tells Python to execute this cell as *shell code*
# not Python, as if we were in a command line
# This is called a ’cell magic’
python -c "print 2*4"
1.2.4 Python scripts

Once you get good at programming, you’ll want to be able to write your own full programs in Python, which
work just like any other program on your computer. We’ll not cover this in this course, you can learn more
about this in MPHYG001. Here are some examples:
In [4]: %%bash
echo "print 2*4" > eight.py
python eight.py
17
In [5]: %%bash
echo ’#!/usr/bin/env python’ > eight
echo "print 2*4" >> eight
chmod u+x eight
./eight
1.2.5 Python Libraries

We can write our own python libraries, called modules which we can import into the notebook and invoke:
In [6]: %%writefile draw_eight.py

# Above line tells the notebook to treat the rest of this
# cell as content for a file on disk.
import numpy as np
import math
def make_figure():
theta=np.arange(0,4*math.pi,0.1)
eight=plt.figure()
axes=eight.add_axes([0,0,1,1])
axes.plot(0.5*np.sin(theta),np.cos(theta/2))
return eight
Writing draw eight.py
In a real example, we could edit the file on disk using a program such as Notepad++ for windows or
Atom for Mac.
In [7]: import draw_eight # Load the library file we just wrote to disk
In [8]: image=draw_eight.make_figure()
18
19
Chapter 2
An example Python data analysis

notebook
2.1 Why write software to manage your data and plots?

We can use programs for our entire research pipeline. Not just big scientific simulation codes, but also the
small scripts which we use to tidy up data and produce plots. This should be code, so that the whole research
pipeline is recorded for reproducibility. Data manipulation in spreadsheets is much harder to share or check.
You can see another similar demonstration on the software carpentry site at
http://swcarpentry.github.io/python-novice-inflammation/01-numpy.html We’ll try to give links to
other sources of Python training along the way. Part of our approach is that we assume you know how to
use the internet! If you find something confusing out there, please bring it along to the next session. In this
course, we’ll always try to draw your attention to other sources of information about what we’re learning.
Paying attention to as many of these as you need to, is just as important as these core notes.
2.2 The example program: a “graph of green land”.

2.2.1 Importing Libraries
Research programming is all about using libraries: tools other people have provided programs that do many
cool things. By combining them we can feel really powerful but doing minimum work ourselves. The python
syntax to import someone else’s library is “import”.
In [1]: import geopy # A python library for investigating geographic information.

# https://pypi.python.org/pypi/geopy
Now, if you try to follow along on this example in an IPython notebook, you’ll probably find that you
just got an error message.
You’ll need to wait until we’ve covered installation of additional python libraries later in the course, then
come back to this and try again. For now, just follow along and try get the feel for how programming for
data-focused research works.
In [2]: geocoder=geopy.geocoders.GoogleV3(domain="maps.google.co.uk")
geocoder.geocode(’Cambridge’,exactly_one=False)
Out[2]: [Location((52.205337, 0.121817, 0.0)), Location((51.73193, -2.3649, 0.0))]
The results come out as a list inside a list: [Name, [Latitude, Longitude]]. Programs represent data
in a variety of different containers like this.
20
2.2.2 Comments
Code after a # symbol doesn’t get run.
In [3]: print "This runs" # print "This doesn’t"

# print This doesn’t either
This runs
2.2.3 Functions
We can wrap code up in a function, so that we can repeatedly get just the information we want.
In [4]: def geolocate(place):

return geocoder.geocode(place, exactly_one = False)[0][1]
Defining functions which put together code to make a more complex task seem simple from the outside
is the most important thing in programming. The output of the function is stated by “return”; the input
comes in in brackets after the function name:
In [5]: geolocate(’London’)
Out[5]: (51.5073509, -0.1277583)
2.2.4 Variables
We can store a result in a variable:
In [6]: london_location=geolocate("London")
print london_location
(51.5073509, -0.1277583)
2.2.5 More complex functions

The google maps API allows us to fetch a map of a place, given a latitude and longitude. The URLs
look like: http://maps.googleapis.com/maps/api/staticmap?size=400x400&center=51.51,-0.1275&zoom=12
We’ll probably end up working out these URLS quite a bit. So we’ll make ourselves another function to
build up a URL given our parameters.
In [7]: import requests

def request_map_at(lat,long, satellite=True,zoom=10,size=(400,400),sensor=False):
base="http://maps.googleapis.com/maps/api/staticmap?"
params=dict(
sensor= str(sensor).lower(),
zoom= zoom,
size= "x".join(map(str,size)),
center= ",".join(map(str,(lat,long))),
style="feature:all|element:labels|visibility:off"
)
if satellite:
params["maptype"]="satellite"
return requests.get(base,params=params)
In [8]: map_response=request_map_at(51.5072, -0.1275)
21
2.2.6 Checking our work
Let’s see what URL we ended up with:
In [9]: url=map_response.url
print url[0:50]
print url[50:100]
print url[100:]
http://maps.googleapis.com/maps/api/staticmap?styl
e=feature%3Aall%7Celement%3Alabels%7Cvisibility%3A
off&center=51.5072%2C-0.1275&zoom=10&maptype=satellite&sensor=false&size=400x400
We can write automated tests so that if we change our code later, we can check the results are still
valid.
In [10]: from nose.tools import assert_in

assert_in("http://maps.googleapis.com/maps/api/staticmap?",url)
assert_in("center=51.5072%2C-0.1275",url)
assert_in("zoom=10",url)
assert_in("size=400x400",url)
assert_in("sensor=false",url)
Our previous function comes back with an Object representing the web request. In object oriented
programming, we use the . operator to get access to a particular property of the object, in this case, the
actual image at that URL is in the content property. It’s a big file, so I’ll just get the first few chars:
In [11]: map_response.content[0:20]
Out[11]: ’\x89PNG\r\n\x1a\n\x00\x00\x00\rIHDR\x00\x00\x01\x90’
2.2.7 Displaying results

I’ll need to do this a lot, so I’ll wrap up our previous function in another function, to save on typing.
In [12]: def map_at(*args,**kwargs):

return request_map_at(*args,**kwargs).content
I can use a library that comes with IPython notebook to display the image. Being able to work with
variables which contain images, or documents, or any other weird kind of data, just as easily as we can with
numbers or letters, is one of the really powerful things about modern programming languages like Python.
In [13]: import IPython

map_png=map_at(*london_location)
In [14]: print "The type of our map result is actually a: ", type(map_png)
The type of our map result is actually a: <type ’str’>
In [15]: IPython.core.display.Image(map_png)
Out[15]:
22
2.2.8 Manipulating Numbers
Now we get to our research project: we want to find out how urbanised the world is, based on satellite
imagery, along a line between two cites. We expect the satellite image to be greener in the countryside.
We’ll use lots more libraries to count how much green there is in an image.
In [16]: from StringIO import StringIO # A library to convert between files and strings
import numpy as np # A library to deal with matrices
from matplotlib import image as img # A library to deal with images
Let’s define what we count as green:
In [17]: def is_green(pixels):

threshold=1.1
greener_than_red = pixels[:,:,1] > threshold* pixels[:,:,0]
greener_than_blue = pixels[:,:,1] > threshold*pixels[:,:,2]
green = np.logical_and(greener_than_red, greener_than_blue)
return green
23
This code has assumed we have our pixel data for the image as a 400 × 400 × 3 3-d matrix, with each of
the three layers being red, green, and blue pixels.
We find out which pixels are green by comparing, element-by-element, the middle (green, number 1) layer
to the top (red, zero) and bottom (blue, 2)
Now we just need to parse in our data, which is a PNG image, and turn it into our matrix format:
In [18]: def count_green_in_png(data):

pixels=img.imread(StringIO(data)) # Get our PNG image as a numpy array
return np.sum(is_green(pixels))
In [19]: print count_green_in_png(map_at(*london_location))
106725
We’ll also need a function to get an evenly spaced set of places between two endpoints:
In [20]: def location_sequence(start, end, steps):

lats = np.linspace(start[0], end[0], steps) # "Linearly spaced" data
longs = np.linspace(start[1],end[1], steps)
return np.vstack([lats, longs]).transpose()
In [21]: location_sequence(geolocate("London"), geolocate("Cambridge"), 5)
Out[21]: array([[ 5.15073509e+01, -1.27758300e-01],

[ 5.16818474e+01, -6.53644750e-02],
[ 5.18563439e+01, -2.97065000e-03],
[ 5.20308405e+01, 5.94231750e-02],
[ 5.22053370e+01, 1.21817000e-01]])
2.2.9 Creating Images

We should display the green content to check our work:
In [22]: def show_green_in_png(data):

pixels=img.imread(StringIO(data)) # Get our PNG image as rows of pixels
green = is_green(pixels)
out = green[:,:,np.newaxis]*np.array([0,1,0])[np.newaxis,np.newaxis,:]
buffer = StringIO()
result = img.imsave(buffer, out, format=’png’)
return buffer.getvalue()
In [23]: IPython.core.display.Image(
map_at(*london_location, satellite=True)
)
Out[23]:
24
In [24]: IPython.core.display.Image(
show_green_in_png(
map_at(
*london_location,
satellite=True)))
Out[24]:
25
2.2.10 Looping
We can loop over each element in out list of coordinates, and get a map for that place:
In [25]: for location in location_sequence(geolocate("London"), geolocate("Birmingham"),4):

IPython.core.display.display( IPython.core.display.Image(map_at(*location)))
26
27
28
29
So now we can count the green from London to Birmingham!
In [26]: [count_green_in_png(map_at(*location))
for location in location_sequence(geolocate("London"),

geolocate("Birmingham"),
10)]
Out[26]: [106725,
127797,
155996,
158581,
157918,
158665,
158407,
156403,
148491,
138544]
30
2.2.11 Plotting graphs
Let’s plot a graph.
In [27]: import matplotlib.pyplot as plt

%matplotlib inline
In [28]: plt.plot([count_green_in_png(map_at(*location))
for location in location_sequence(geolocate("London"),
geolocate("Birmingham"),
10)])
Out[28]: [<matplotlib.lines.Line2D at 0x1054cb410>]
From a research perspective, of course, this code needs a lot of work. But I hope the power of using
programming is clear.
2.2.12 Composing Program Elements

We built little pieces of useful code, to:
• Find latitude and longitude of a place

• Get a map at a given latitude and longitude
• Decide whether a (red,green,blue) triple is mainly green
• Decide whether each pixel is mainly green
• Plot a new image showing the green places
• Find evenly spaced points between two places
By putting these together, we can make a function which can plot this graph automatically for any two
places:
31
In [29]: def green_between(start, end,steps):
return [count_green_in_png(map_at(*location))
for location in location_sequence(
geolocate(start),
geolocate(end),
steps)]
In [30]: plt.plot(green_between(’New York’, ’Chicago’, 20))
Out[30]: [<matplotlib.lines.Line2D at 0x1025b6150>]
And that’s it! We’ve covered, very very quickly, the majority of the python language, and much of the
theory of software engineering.
Now we’ll go back, carefully, through all the concepts we touched on, and learn how to use them properly
ourselves.
32
Chapter 3
Variables
3.1 Variable Assignment

When we generate a result, the answer is displayed, but not kept anywhere.
In [1]: 2*3
Out[1]: 6
If we want to get back to that result, we have to store it. We put it in a box, with a name on the box.
This is a variable.
In [2]: six=2*3
In [3]: print six
If we look for a variable that hasn’t ever been defined, we get an error.
In [4]: print seven
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<ipython-input-4-cd3a57e315ea> in <module>()
----> 1 print seven
NameError: name ’seven’ is not defined
That’s not the same as an empty box, well labeled:
In [5]: nothing = None
In [6]: print nothing
None
33
(None is the special python value for a no-value variable.)
Supplementary Materials: There’s more on variables at http://swcarpentry.github.io/python-novice-
inflammation/01-numpy.html
Anywhere we could put a raw number, we can put a variable label, and that works fine:
In [7]: print 5*six
30
In [8]: scary = six*six*six
In [9]: print scary
216
3.2 Reassignment and multiple labels

But here’s the real scary thing: it seems like we can put something else in that box:
In [10]: scary = 25
In [11]: print scary
25
Note that the data that was there before has been lost.
No labels refer to it any more - so it has been “Garbage Collected”! We might imagine something pulled
out of the box, and thrown on the floor, to make way for the next occupant.
In fact, though, it is the label that has moved. We can see this because we have more than one label
refering to the same box:
In [12]: name = "James"
In [13]: nom = name
In [14]: print nom
James
In [15]: print name
James
And we can move just one of those labels:
In [16]: nom = "Hetherington"
In [17]: print name
James
In [18]: print nom
Hetherington
34
So we can now develop a better understanding of our labels and boxes: each box is a piece of space (an
address) in computer memory. Each label (variable) is a reference to such a place.
When the number of labels on a box (“variables referencing an address”) gets down to zero, then the
data in the box cannot be found any more.
After a while, the language’s “Garbage collector” will wander by, notice a box with no labels, and throw
the data away, making that box available for more data.
Old fashioned languages like C and Fortran don’t have Garbage collectors. So a memory address with
no references to it still takes up memory, and the computer can more easily run out.
So when I write:
In [19]: name = "Jim"
The following things happen:
1. A new text object is created, and an address in memory is found for it.
2. The variable “name” is moved to refer to that address.
3. The old address, containing “James”, now has no labels.
4. The garbage collector frees the memory at the old address.
Supplementary materials: There’s an online python tutor which is great for visualising memory and
references. Try the scenario we just looked at
Labels are contained in groups called “frames”: our frame contains two labels, ‘nom’ and ‘name’.
3.3 Objects and types

An object, like name, has a type. In the online python tutor example, we see that the objects have type
“str”. str means a text object: Programmers call these ‘strings’.
In [20]: type(name)
Out[20]: str
Depending on its type, an object can have different properties: data fields Inside the object.
Consider a Python complex number for example:
In [21]: z=3+1j
In [22]: z.real
Out[22]: 3.0
In [23]: z.imag
Out[23]: 1.0
A property of an object is accessed with a dot.
The jargon is that the “dot operator” is used to obtain a property of an object.
When we try to access a property that doesn’t exist, we get an error:
In [24]: z.wrong
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
<ipython-input-24-76215e50e85b> in <module>()
----> 1 z.wrong
AttributeError: ’complex’ object has no attribute ’wrong’
35
3.4 Reading error messages.
It’s important, when learning to program, to develop an ability to read an error message and find, from in
amongst all the confusing noise, the bit of the error message which tells you what to change!
We don’t yet know what is meant by AttributeError, or “Traceback”.
In [25]: z2=5-6j
print "Gets to here"
print z.wrong
print "Didn’t get to here"
Gets to here
---------------------------------------------------------------------------
<ipython-input-25-891c7b6126ae> in <module>()
1 z2=5-6j
2 print "Gets to here"
----> 3 print z.wrong
4 print "Didn’t get to here"
AttributeError: ’complex’ object has no attribute ’wrong’
But in the above, we can see that the error happens on the third line of our code cell.
We can also see that the error message: > ‘complex’ object has no attribute ‘wrong’
. . . tells us something important. Even if we don’t understand the rest, this is useful for debugging!
3.5 Variables and the notebook kernel

When I type code in the notebook, the objects live in memory between cells.
In [26]: number = 0
In [27]: print number
0
If I change a variable:
In [28]: number = number +1
In [29]: print number
1
It keeps its new value for the next cell.
But cells are not always evaluated in order.
If I now go back to Input 22, reading number = number + 1, and run it again, with shift-enter. Number
will change from 2 to 3, then from 3 to 4. Try it!
So it’s important to remember that if you move your cursor around in the notebook, it doesn’t always
run top to bottom.
Supplementary material: (1) https://ipython.org/ipython-doc/3/notebook/index.html (2)
http://ipython.org/videos.html
36
Chapter 4
Using Functions
4.1 Calling functions

We often want to do thing to our objects that are more complicated than just assigning them to variables.
In [1]: len("pneumonoultramicroscopicsilicovolcanoconiosis")
Out[1]: 45
Here we have “called a function”.

The function len takes one input, and has one output. The output is the length of whatever the input
was.
Programmers also call function inputs “parameters” or, confusingly, “arguments”.
Here’s another example:
In [2]: sorted("Python")
Out[2]: [’P’, ’h’, ’n’, ’o’, ’t’, ’y’]
Which gives us back a list of the letters in Python, sorted alphabetically.

The input goes in brackets after the function name, and the output emerges wherever the function is
used.
So we can put a function call anywhere we could put a “literal” object or a variable.
In [3]: len(’Jim’)*8
Out[3]: 24
In [4]: x=len(’Mike’)
y=len(’Bob’)
z=x+y
In [5]: print z
4.2 Using methods

Objects come associated with a bunch of functions designed for working on objects of that type. We access
these with a dot, just as we do for data attributes:
In [6]: "shout".upper()
37
Out[6]: ’SHOUT’
These are called methods. If you try to use a method defined for a different type, you get an error:
In [7]: 5.upper()
File "<ipython-input-7-0cdd0df562f5>", line 1

5.upper()
^
SyntaxError: invalid syntax
If you try to use a method that doesn’t exist, you get an error:
In [8]: x=5
x.wrong
---------------------------------------------------------------------------
<ipython-input-8-c914fcddd360> in <module>()
1 x=5
----> 2 x.wrong
AttributeError: ’int’ object has no attribute ’wrong’
Methods and properties are both kinds of attribute, so both are accessed with the dot operator.
Objects can have both properties and methods:
In [9]: z=1+5j
In [10]: z.real
Out[10]: 1.0
In [11]: z.conjugate()
Out[11]: (1-5j)
4.3 Functions are just a type of object!

Now for something that will take a while to understand: don’t worry if you don’t get this yet, we’ll look
again at this in much more depth later in the course.
If we forget the (), we realise that a method is just a property which is a function!
In [12]: z.conjugate
Out[12]: <function conjugate>
In [13]: type(z.conjugate)
38
Out[13]: builtin function or method
In [14]: somefunc=z.conjugate
In [15]: somefunc()
Out[15]: (1-5j)
Functions are just a kind of variable:
In [16]: magic = sorted
In [17]: type(magic)
Out[17]: builtin function or method
In [18]: magic(["Technology", "Advanced"])
Out[18]: [’Advanced’, ’Technology’]
4.4 Getting help on functions and methods

The ‘help’ function, when applied to a function, gives help on it!
In [19]: help(sorted)
Help on built-in function sorted in module builtin :
sorted(...)
sorted(iterable, cmp=None, key=None, reverse=False) --> new sorted list
The ‘dir’ function, when applied to an object, lists all its attributes (properties and methods):
In [20]: dir("Hexxo")
Out[20]: [’ add ’,
’ class ’,
’ contains ’,
’ delattr ’,
’ doc ’,
’ eq ’,
’ format ’,
’ ge ’,
’ getattribute ’,
’ getitem ’,
’ getnewargs ’,
’ getslice ’,
’ gt ’,
’ hash ’,
’ init ’,
’ le ’,
’ len ’,
’ lt ’,
’ mod ’,
’ mul ’,
’ ne ’,
’ new ’,
39
’ reduce ’,
’ reduce ex ’,
’ repr ’,
’ rmod ’,
’ rmul ’,
’ setattr ’,
’ sizeof ’,
’ str ’,
’ subclasshook ’,
’ formatter field name split’,
’ formatter parser’,
’capitalize’,
’center’,
’count’,
’decode’,
’encode’,
’endswith’,
’expandtabs’,
’find’,
’format’,
’index’,
’isalnum’,
’isalpha’,
’isdigit’,
’islower’,
’isspace’,
’istitle’,
’isupper’,
’join’,
’ljust’,
’lower’,
’lstrip’,
’partition’,
’replace’,
’rfind’,
’rindex’,
’rjust’,
’rpartition’,
’rsplit’,
’rstrip’,
’split’,
’splitlines’,
’startswith’,
’strip’,
’swapcase’,
’title’,
’translate’,
’upper’,
’zfill’]
Most of these are confusing methods beginning and ending with , part of the internals of python.
Again, just as with error messages, we have to learn to read past the bits that are confusing, to the bit
we want:
In [21]: "Hexxo".replace("x", "l")
40
Out[21]: ’Hello’
4.5 Operators
Now that we know that functions are a way of taking a number of inputs and producing an output, we
should look again at what happens when we write:
In [22]: x = 2 + 3
In [23]: print x
This is just a pretty way of calling an “add” function. Things would be more symmetrical if add were
actually written
x = +(2,3)
Where ‘+’ is just the name of the name of the adding function.
In python, these functions do exist, but they’re actually methods of the first input: they’re the myste-
rious functions we saw earlier (Two underscores.)
In [24]: x.__add__(7)
Out[24]: 12
We call these symbols, +, - etc, “operators”.

The meaning of an operator varies for different types:
In [25]: [2, 3, 4] + [5, 6]
Out[25]: [2, 3, 4, 5, 6]
Sometimes we get an error when a type doesn’t have an operator:
In [26]: 7-2
Out[26]: 5
In [27]: [2, 3, 4] - [5, 6]
---------------------------------------------------------------------------
TypeError Traceback (most recent call last)
<ipython-input-27-4627195e7799> in <module>()
----> 1 [2, 3, 4] - [5, 6]
TypeError: unsupported operand type(s) for -: ’list’ and ’list’
The word “operand” means “thing that an operator operates on”!

Or when two types can’t work together with an operator:
In [28]: [2, 3, 4] + 5
41
---------------------------------------------------------------------------
<ipython-input-28-84117f41979f> in <module>()
----> 1 [2, 3, 4] + 5
TypeError: can only concatenate list (not "int") to list
Just as in Mathematics, operators have a built-in precedence, with brackets used to force an order of
operations:
In [29]: print 2+3*4
14
In [30]: print (2+3)*4
20
Supplementary material : http://www.mathcs.emory.edu/˜valerie/courses/fall10/155/resources/op precedence.html
42
Chapter 5
Types
We have seen that Python objects have a ‘type’:
In [1]: type(5)
Out[1]: int
5.1 Floats and integers

Python has two core numeric types, int for integer, and float for real number.
In [2]: one=1
ten=10
one_float=1.
ten_float=10.
In [3]: tenth= one_float/ten_float

print type(one), type(one_float)
<type ’int’> <type ’float’>
The meaning of an operator varies depending on the type it is applied to!
In [4]: print one/ten, one_float/ten_float
0 0.1
In [5]: print type(one/ten), type(tenth)
<type ’int’> <type ’float’>
The divided by operator when applied to floats, means divide by for real numbers. But when applied to
integers, it means divide then round down:
In [6]: 10/3
Out[6]: 3
In [7]: 10.0/3
Out[7]: 3.3333333333333335
In [8]: 10/3.0
43
Out[8]: 3.3333333333333335
So if I have two integer variables, and I want the float division, I need to change the type first. There
is a function for every type name, which is used to convert the input to an output of the desired type.
In [9]: x=float(5)
type(x)
Out[9]: float
In [10]: 10/float(3)
Out[10]: 3.3333333333333335
I lied when I said that the float type was a real number. √It’s actually a computer representation of
a real number called a “floating point number”. Representing 2 or 31 perfectly would be impossible in a
computer, so we use a finite amount of memory to do it.
Supplementary material :
• https://docs.python.org/2/tutorial/floatingpoint.html
• http://floating-point-gui.de/formats/fp/
• Advanced: http://docs.oracle.com/cd/E19957-01/806-3568/ncg goldberg.html
5.2 Strings
Python has a built in string type, supporting many useful methods.
In [11]: given = "James"

family = "Hetherington"
full = given + " " + family
So + for strings means “join them together” - concatenate.
In [12]: print full.upper()
JAMES HETHERINGTON
As for float and int, the name of a type can be used as a function to convert between types:
In [13]: print ten+one
11
In [14]: print float( str(ten)+str(one) )
101.0
We can remove extraneous material from the start and end of a string:
In [15]: " Hello ".strip()
Out[15]: ’Hello’
44
5.3 Lists
Python’s basic container type is the list
We can define our own list with square brackets:
In [16]: [1, 3, 7]
Out[16]: [1, 3, 7]
In [17]: type([1, 3, 7])
Out[17]: list
Lists do not have to contain just one type:
In [18]: various_things = [1, 2, "banana", 3.4, [1,2] ]
We access an element of a list with an int in square brackets:
In [19]: various_things[2]
Out[19]: ’banana’
In [20]: index = 0
various_things[index]
Out[20]: 1
Note that list indices start from zero.

We can quickly make a list with numbers counted up:
In [21]: count_to_five = range(5)

print count_to_five
[0, 1, 2, 3, 4]
We can use a string to join together a list of strings:
In [22]: name = ["James", "Philip", "John", "Hetherington"]

print ",".join(name)
James,Philip,John,Hetherington
In [23]: And we can split up a string into a list:
File "<ipython-input-23-a3490b6b76bc>", line 1

And we can split up a string into a list:
^
In [24]: "Ernst Stavro Blofeld".split(" ")
Out[24]: [’Ernst’, ’Stavro’, ’Blofeld’]
45
5.4 Sequences
Many other things can be treated like lists. Python calls things that can be treated like lists sequences.
A string is one such sequence type
In [25]: print count_to_five[1]

print "James"[2]
1
m
In [26]: print count_to_five[1:3]

print "Hello World"[4:8]
[1, 2]
o Wo
In [27]: print len(various_things)

print len("Python")
5
6
In [28]: print "John" in name

print 9 in count_to_five
True
False
5.5 Unpacking
Multiple values can be unpacked when assigning from sequences, like dealing out decks of cards.
In [29]: mylist = [’Hello’, ’World’]

a, b = mylist
print b
World
46
Chapter 6
Containers
6.1 Checking for containment.

The list we saw is a container type: its purpose is to hold other objects. We can ask python whether or
not a container contains a particular item:
In [1]: ’Dog’ in [’Cat’, ’Dog’, ’Horse’]
Out[1]: True
In [2]: ’Bird’ in [’Cat’, ’Dog’, ’Horse’]
Out[2]: False
In [3]: 2 in range(5)
Out[3]: True
In [4]: 99 in range(5)
Out[4]: False
6.2 Mutability
An array can be modified:
In [5]: name = "James Philip John Hetherington".split(" ")

print name
[’James’, ’Philip’, ’John’, ’Hetherington’]
In [6]: name[0] = "Dr"

name[1:3] = ["Griffiths-"]
name.append("PhD")
print " ".join(name)
Dr Griffiths- Hetherington PhD
47
6.3 Tuples
A tuple is an immutable sequence:
In [7]: my_tuple = ("Hello", "World")

my_tuple[0]="Goodbye"
---------------------------------------------------------------------------
<ipython-input-7-d3ad0c7e33f1> in <module>()
1 my tuple = ("Hello", "World")
----> 2 my tuple[0]="Goodbye"
TypeError: ’tuple’ object does not support item assignment
str is immutable too:
In [8]: fish = "Hake"

fish[0] = ’R’
---------------------------------------------------------------------------
<ipython-input-8-fe8069275347> in <module>()
1 fish = "Hake"
----> 2 fish[0] = ’R’
TypeError: ’str’ object does not support item assignment
But note that container reassignment is moving a label, not changing an element:
In [9]: fish = "Rake" ## OK!
Supplementary material : Try the online memory visualiser for this one.
6.4 Memory and containers

The way memory works with containers can be important:
In [10]: x = range(3)
print x
[0, 1, 2]
In [11]: y = x
print y
48
[0, 1, 2]
In [12]: z = x[0:3]
y[1] = "Gotcha!"
print x
print y
print z
[0, ’Gotcha!’, 2]
[0, ’Gotcha!’, 2]
[0, 1, 2]
In [13]: z[2] = "Really?"

print x
print y
print z
[0, ’Gotcha!’, 2]
[0, ’Gotcha!’, 2]
[0, 1, ’Really?’]
Supplementary material : This one works well at the memory visualiser.

The explanation: While y is a second label on the same object, z is a separate object with the same data.
Nested objects make it even more complicated:
In [14]: x=[[’a’,’b’],’c’]
y=x
z=x[0:2]
x[0][1]=’d’
z[1]=’e’
In [15]: x
Out[15]: [[’a’, ’d’], ’c’]
In [16]: y
Out[16]: [[’a’, ’d’], ’c’]
In [17]: z
Out[17]: [[’a’, ’d’], ’e’]
Try the visualiser again.
6.5 Identity vs Equality

Having the same data is different from being the same actual object in memory:
In [18]: print [1, 2] == [1, 2]

print [1, 2] is [1, 2]
True
False
49
The == operator checks, element by element, that two containers have the same data. The is operator
checks that they are actually the same object.
But, and this point is really subtle, for immutables, the python language might save memory by reusing
a single instantiated copy. This will always be safe.
In [19]: print "Hello" == "Hello"

print "Hello" is "Hello"
True
True
50
Chapter 7
Dictionaries
7.1 The Python Dictionary

Python supports a container type called a dictionary.
This is also known as an “associative array”, “map” or “hash” in other languages.
In a list, we use a number to look up an element:
In [1]: names="Martin Luther King".split(" ")

names[1]
Out[1]: ’Luther’
In a dictionary, we look up an element using another object of our choice:
In [2]: me = { "name": "James", "age": 38, "Jobs": ["Programmer", "Teacher"] }
In [3]: print me
{’age’: 38, ’Jobs’: [’Programmer’, ’Teacher’], ’name’: ’James’}
In [4]: print me[’Jobs’]
[’Programmer’, ’Teacher’]
In [5]: print type(me)
<type ’dict’>
7.2 Keys and Values

The things we can use to look up with are called keys:
In [6]: me.keys()
Out[6]: [’age’, ’Jobs’, ’name’]
The things we can look up are called values:
In [7]: me.values()
Out[7]: [38, [’Programmer’, ’Teacher’], ’James’]
When we test for containment on a dict we test on the keys:
51
In [8]: ’Jobs’ in me
Out[8]: True
In [9]: ’James’ in me
Out[9]: False
In [10]: ’James’ in me.values()
Out[10]: True
7.3 Immutable Keys Only

The way in which dictionaries work is one of the coolest things in computer science: the “hash table”. This
is way beyond the scope of this course, but it has a consequence:
You can only use immutable things as keys.
In [11]: good_match = {("Lamb", "Mint"): True, ("Bacon", "Chocolate"): False}
but:
In [12]: illegal = {[1,2]: 3}
---------------------------------------------------------------------------
<ipython-input-12-cca03b227ff4> in <module>()
----> 1 illegal = {[1,2]: 3}
TypeError: unhashable type: ’list’
Supplementary material : You can start to learn about the ‘hash table’ here This material is very ad-
vanced, but, I think, really interesting!
7.4 No guarantee of order

Another consequence of the way dictionaries work is that there’s no guaranteed order among the elements:
In [13]: my_dict = {’0’: 0, ’1’:1, ’2’: 2, ’3’: 3, ’4’: 4}

print my_dict
print my_dict.values()
{’1’: 1, ’0’: 0, ’3’: 3, ’2’: 2, ’4’: 4}

[1, 0, 3, 2, 4]
52
7.5 Sets
A set is a list which cannot contain the same element twice.
In [14]: name = "James Hetherington"

unique_letters = set(name)
print "".join(unique_letters)
a egiHJmonsrth
It has no particular order, but is really useful for checking or storing unique values.
53
Chapter 8
Data structures
8.1 Nested Lists and Dictionaries

In research programming, one of our most common tasks is building an appropriate structure to model our
complicated data. Later in the course, we’ll see how we can define our own types, with their own attributes,
properties, and methods. But probably the most common approach is to use nested structures of lists,
dictionaries, and sets to model our data. For example, an address might be modelled as a dictionary with
appropriately named fields:
In [1]: UCL={
’City’: ’London’,
’Street’: ’Gower Street’,
’Postcode’: ’WC1E 6BT’
}
In [2]: James={
’City’: ’London’,
’Street’: ’Waterson Street’,
’Postcode’: ’E2 8HH’
}
A collection of people’s addresses is then a list of dictionaries:
In [3]: addresses=[UCL, James]
In [4]: addresses
Out[4]: [{’City’: ’London’, ’Postcode’: ’WC1E 6BT’, ’Street’: ’Gower Street’},

{’City’: ’London’, ’Postcode’: ’E2 8HH’, ’Street’: ’Waterson Street’}]
A more complicated data structure, for example for a census database, might have a list of residents or
employees at each address:
In [5]: UCL[’people’]=[’Clare’,’James’, ’Owain’]
In [6]: James[’people’]=[’Sue’, ’James’]
In [7]: addresses
Out[7]: [{’City’: ’London’,

’Postcode’: ’WC1E 6BT’,
’Street’: ’Gower Street’,
54
’people’: [’Clare’, ’James’, ’Owain’]},
{’City’: ’London’,
’Postcode’: ’E2 8HH’,
’Street’: ’Waterson Street’,
’people’: [’Sue’, ’James’]}]
Which is then a list of dictionaries, with keys which are strings or lists.
We can go further, e.g.:
In [8]: UCL[’Residential’]=False
And we can write code against our structures:
In [9]: leaders = [place[’people’][0] for place in addresses]

print leaders
[’Clare’, ’Sue’]
This was an example of a ‘list comprehension’, which have used to get data of this structure, and which
we’ll see more of in a moment. . .
8.2 Exercise: a Maze Model.

Work with a partner to design a data structure to represent a maze using dictionaries and lists.
• Each place in the maze has a name, which is a string.

• Each place in the maze has one or more people currently standing at it, by name.
• Each place in the maze has a maximum capacity of people that can fit in it.
• From each place in the maze, you can go from that place to a few other places, using a direction like
‘up’, ‘north’, or ‘sideways’
Create an example instance, in a notebook, of a simple structure for your maze:
• The front room can hold 2 people. James is currently there. You can go outside to the garden, or
upstairs to the bedroom, or north to the kitchen.
• From the kitchen, you can go south to the front room. It fits 1 person.
• From the garden you can go inside to living room. It fits 3 people. Sue is currently there.
• From the bedroom, you can go downstairs. You can also jump out of the window to the garden. It fits
2 people.
Make sure that your model:
• Allows empty rooms

• Allows you to jump out of the upstairs window, but not to fly back up.
• Allows rooms which people can’t fit in.
8.2.1 Solution: my Maze Model

Here’s one possible solution to the Maze model. Yours will probably be different, and might be just as good.
That’s the artistry of software engineering: some solutions will be faster, others use less memory, while others
will be easier for other people to understand. Optimising and balancing these factors is fun!
In [1]: house = {
’living’ : {
’exits’: {
’north’ : ’kitchen’,
55
’outside’ : ’garden’,
’upstairs’ : ’bedroom’
},
’people’ : [’James’],
’capacity’ : 2
},
’kitchen’ : {
’exits’: {
’south’ : ’living’
},
’people’ : [],
’capacity’ : 1
},
’garden’ : {
’exits’: {
’inside’ : ’living’
},
’people’ : [’Sue’],
’capacity’ : 3
},
’bedroom’ : {
’exits’: {
’downstairs’ : ’living’,
’jump’ : ’garden’
},
’people’ : [],
’capacity’ : 1
}
}
Some important points:
• The whole solution is a complete nested structure.

• I used indenting to make the structure easier to read.
• Python allows code to continue over multiple lines, so long as sets of brackets are not finished.
• There is an Empty person list in empty rooms, so the type structure is robust to potential movements
of people.
• We are nesting dictionaries and lists, with string and integer data.
56
Chapter 9
Control and Flow
9.1 Turing completeness

Now that we understand how we can use objects to store and model our data, we only need to be able to
control the flow of our program in order to have a program that can, in principle, do anything!
Specifically we need to be able to:
• Control whether a program statement should be executed or not, based on a variable. “Conditionality”
• Jump back to an earlier point in the program, and run some statements again. “Branching”
Once we have these, we can write computer programs to process information in arbitrary ways: we are
Turing Complete!
9.2 Conditionality
Conditionality is achieved through Python’s if statement:
In [1]: x = 5
if x < 0:
print x, " is negative"
x=-10
if x < 0:
print x, " is negative"
-10 is negative
The first time through, the print statement never happened.

The controlled statements are indented. Once we remove the indent, the statements will once again
happen regardless.
9.3 Else and Elif

Python’s if statement has optional elif (else-if) and else clauses:
In [2]: x = 5
if x < 0:
print "x is negative"
else:
print "x is positive"
57
x is positive
In [3]: x = 5
if x < 0:
print "x is negative"
elif x == 0:
print "x is zero"
else:
print "x is positive"
x is positive
Try editing the value of x here, and note that other sections are found.
In [4]: choice = ’high’
if choice == ’high’:
print 1
elif choice == ’medium’:
print 2
else:
print 3
9.4 Comparison
True and False are used to represent boolean (true or false) values.
In [5]: 1 > 2
Out[5]: False
Comparison on strings is alphabetical.
In [6]: "UCL" > "KCL"
Out[6]: True
There’s no automatic conversion of the string True to true:
In [7]: True == "True"
Out[7]: False
There are subtle implied order comparisons between types, but it would be bad style to rely on these,
because most human readers won’t remember them:
In [8]: ’1’ < 2
Out[8]: False
In [9]: ’1’ > 2
Out[9]: True
Any statement that evaluates to True or False can be used to control an if Statement.
58
9.5 Automatic Falsehood
Various other things automatically count as true or false, which can make life easier when coding:
In [10]: mytext = "Hello"
if mytext:
print "Mytext is not empty"
mytext2 = ""
if mytext2:
print "Mytext2 is not empty"
Mytext is not empty
We can use logical not and logical and to combine true and false:
In [11]: x=3.2
if not (x>0 and type(x)==int):
print x,"is not a positive integer"
3.2 is not a positive integer
not also understands magic conversion from false-like things to True or False.
In [12]: not not "Who’s there!" # Thanks to Mysterious Student
Out[12]: True
In [13]: bool("")
Out[13]: False
In [14]: bool("James")
Out[14]: True
In [15]: bool([])
Out[15]: False
In [16]: bool([’a’])
Out[16]: True
In [17]: bool({})
Out[17]: False
In [18]: bool({’name’: ’James’})
Out[18]: True
In [19]: bool(0)
Out[19]: False
In [20]: bool(1)
Out[20]: True
But subtly, although these quantities evaluate True or False in an if statement, they’re not themselves
actually True or False under ==:
In [21]: [] == False
Out[21]: False
In [22]: (not not []) == (not not False)
Out[22]: True
59
9.6 Indentation
In Python, indentation is semantically significant. You can choose how much indentation to use, so long as
you are consistent, but four spaces is conventional. Please do not use tabs.
In the notebook, and most good editors, when you press <tab>, you get four spaces.
In [23]: if x>0:
print x
File "<ipython-input-23-b7a20fbd3ce7>", line 2

print x
^
IndentationError: expected an indented block
9.7 Pass
A statement expecting identation must have some indented code. This can be annoying when commenting
things out. (With #)
In [24]: if x>0:
# print x
print "Hello"
File "<ipython-input-24-1d8c0c9cd327>", line 3

print "Hello"
^
IndentationError: expected an indented block
So the pass statement is used to do nothing.

In [25]: if x>0:
pass
print "Hello"
Hello
9.8 Iteration
Our other aspect of control is looping back on ourselves.
We use for . . . in to “iterate” over lists:
In [1]: mylist = [3, 7, 15, 2]
for whatever in mylist:
print whatever**2
9
49
225
4
Each time through the loop, the variable in the value slot is updated to the next element of the sequence.
60
9.9 Iterables
Any sequence type is iterable:
In [2]: vowels="aeiou"
sarcasm = []
for letter in "Okay":
if letter.lower() in vowels:
repetition = 3
else:
repetition = 1
sarcasm.append(letter*repetition)
"".join(sarcasm)
Out[2]: ’OOOkaaay’
The above is a little puzzle, work through it to understand why it does what it does, you have
9.10 Dictionaries are Iterables

All sequences are iterables. Some iterables (things you can for loop over) are not sequences (things with
you can do x[5] to), for example sets and dictionaries.
In [3]: import datetime

now = datetime.datetime.now()
founded = {"James": 1976, "UCL": 1826, "Cambridge": 1209}
current_year = now.year
for thing in founded:

print thing, " is ", current_year - founded[thing], "years old."
James is 39 years old.

UCL is 189 years old.
Cambridge is 806 years old.
9.11 Unpacking and Iteration

Unpacking can be useful with iteration:
In [4]: triples=[
[4,11,15],
[39,4,18]
]
In [5]: for whatever in triples:

print whatever
[4, 11, 15]

[39, 4, 18]
In [6]: for first, middle, last in triples:

print middle
61
11
4
In [7]: # A reminder that the words you use for variable names are arbitrary:
for hedgehog, badger, fox in triples:
print badger
11
4
for example, to iterate over the items in a dictionary as pairs:
In [8]: things = {"James": [1976, ’Kendal’],
"UCL": [1826, ’Bloomsbury’],
"Cambridge": [1209, ’Cambridge’]}
print things.items()
[(’James’, [1976, ’Kendal’]), (’UCL’, [1826, ’Bloomsbury’]), (’Cambridge’, [1209, ’Cambridge’])]
In [9]: for name, year in founded.items():
print name, " is ", current_year - year, "years old."
James is 39 years old.
UCL is 189 years old.
Cambridge is 806 years old.
9.12 Break, Continue

• Continue skips to the next turn of a loop
• Break stops the loop early
In [10]: for n in range(50):
if n==20:
break
if n % 2 == 0:
continue
print n
1
3
5
7
9
11
13
15
17
19
These aren’t useful that often, but are worth knowing about. There’s also an optional else clause on
loops, executed only if you don’t break, but I’ve never found that useful.
9.13 Classroom exercise: the Maze Population

Take your maze data structure. Write a program to count the total number of people in the maze, and also
determine the total possible occupants.
62
9.13.1 Solution: counting people in the maze
With this maze structure:
In [1]: house = {
’living’ : {
’exits’: {
},
’capacity’ : 2
},
’kitchen’ : {
’exits’: {
},
’people’ : [],
’capacity’ : 1
},
’garden’ : {
’exits’: {
},
’capacity’ : 3
},
’bedroom’ : {
’exits’: {
},
’people’ : [],
’capacity’ : 1
}
}
We can count the occupants and capacity like this:
In [2]: capacity = 0
occupancy = 0
for name, room in house.items():
capacity+=room[’capacity’]
occupancy+=len(room[’people’])
print "House can fit", capacity, "people, and currently has:", occupancy, "."
House can fit 7 people, and currently has: 2 .
63
Chapter 10
Comprehensions
10.1 The list comprehension

If you write a for loop inside a pair of square brackets for a list, you magic up a list as defined. This can
make for concise but hard to read code, so be careful.
In [1]: [2**x for x in range(10)]
Out[1]: [1, 2, 4, 8, 16, 32, 64, 128, 256, 512]
You can do quite weird and cool things with comprehensions:
In [2]: [len(str(2**x)) for x in range(10)]
Out[2]: [1, 1, 1, 1, 2, 2, 2, 3, 3, 3]
10.2 Selection in comprehensions

You can write an if statement in comprehensions too:
In [3]: [2**x for x in range(30) if x%3 ==0 ]
Out[3]: [1, 8, 64, 512, 4096, 32768, 262144, 2097152, 16777216, 134217728]
Consider the following, and make sure you understand why it works:
In [4]: "".join([letter for letter in "James Hetherington" if letter.lower() not in ’aeiou’])
Out[4]: ’Jms Hthrngtn’
10.3 Comprehensions versus building lists with append:

This code:
In [5]: result=[]
for x in range(30):
if x%3 == 0:
result.append(2**x)
print result
[1, 8, 64, 512, 4096, 32768, 262144, 2097152, 16777216, 134217728]
64
Does the same as the comprehension above. The comprehension is generally considered more readable.
Comprehensions are therefore an example of what we call ‘syntactic sugar’: they do not increase the
capabilities of the language.
Instead, they make it possible to write the same thing in a more readable way.
Everything we learn from now on will be either syntactic sugar or interaction with something other than
idealised memory, such as a storage device or the internet. Once you have variables, conditionality, and
branching, your language can do anything. (And this can be proved.)
10.4 Nested comprehensions

If you write two for statements in a comprehension, you get a single array generated over all the pairs:
In [6]: [x - y for x in range(4) for y in range(4)]
Out[6]: [0, -1, -2, -3, 1, 0, -1, -2, 2, 1, 0, -1, 3, 2, 1, 0]
You can select on either, or on some combination:
In [7]: [x - y for x in range(4) for y in range(4) if x>=y]
Out[7]: [0, 1, 0, 2, 1, 0, 3, 2, 1, 0]
If you want something more like a matrix, you need to do two nested comprehensions!
In [8]: [[x - y for x in range(4)] for y in range(4)]
Out[8]: [[0, 1, 2, 3], [-1, 0, 1, 2], [-2, -1, 0, 1], [-3, -2, -1, 0]]
Note the subtly different square brackets.

Note that the list order for multiple or nested comprehensions can be confusing:
In [9]: [x+y for x in [’a’,’b’,’c’] for y in [’1’,’2’,’3’]]
Out[9]: [’a1’, ’a2’, ’a3’, ’b1’, ’b2’, ’b3’, ’c1’, ’c2’, ’c3’]
In [10]: [[x+y for x in [’a’,’b’,’c’]] for y in [’1’,’2’,’3’]]
Out[10]: [[’a1’, ’b1’, ’c1’], [’a2’, ’b2’, ’c2’], [’a3’, ’b3’, ’c3’]]
10.5 Dictionary Comprehensions

You can automatically build dictionaries, by using a list comprehension syntax, but with curly brackets and
a colon:
In [11]: { (str(x))*3: x for x in range(3) }
Out[11]: {’000’: 0, ’111’: 1, ’222’: 2}
10.6 List-based thinking

Once you start to get comfortable with comprehensions, you find yourself working with containers, nested
groups of lists and dictionaries, as the ‘things’ in your program, not individual variables.
Given a way to analyse some dataset, we’ll find ourselves writing stuff like:
analysed_data = [analyze(datum) for datum in data]
65
There are lots of built-in methods that provide actions on lists as a whole:
In [12]: any([True, False, True])
Out[12]: True
In [13]: all([True, False, True])
Out[13]: False
In [14]: max([1, 2, 3])
Out[14]: 3
In [15]: sum([1, 2, 3])
Out[15]: 6
My favourite is map, which is syntactic sugar for a simple list comprehension that applies one function to
every member of a list:
In [16]: [str(x) for x in range(10)]
Out[16]: [’0’, ’1’, ’2’, ’3’, ’4’, ’5’, ’6’, ’7’, ’8’, ’9’]
In [17]: map(str, range(10))
Out[17]: [’0’, ’1’, ’2’, ’3’, ’4’, ’5’, ’6’, ’7’, ’8’, ’9’]
So I can write:
analysed_data = map(analyse, data)
10.7 Classroom Exercise: Occupancy Dictionary

Take your maze data structure. Write a program to print out a new dictionary, which gives, for each room’s
name, the number of people in it. Don’t add in a zero value in the dictionary for empty rooms.
The output should look similar to:
In [18]: {’bedroom’: 1, ’garden’: 3, ’kitchen’: 1, ’living’: 2}
Out[18]: {’bedroom’: 1, ’garden’: 3, ’kitchen’: 1, ’living’: 2}
10.7.1 Solution
With this maze structure:
In [1]: house = {
’living’ : {
’exits’: {
},
’capacity’ : 2
},
’kitchen’ : {
66
’exits’: {
},
’people’ : [],
’capacity’ : 1
},
’garden’ : {
’exits’: {
},
’capacity’ : 3
},
’bedroom’ : {
’exits’: {
},
’people’ : [],
’capacity’ : 1
}
}
We can get a simpler dictionary with just capacities like this:
In [2]: {name : room[’capacity’] for name, room in house.items()}
Out[2]: {’bedroom’: 1, ’garden’: 3, ’kitchen’: 1, ’living’: 2}
67
Chapter 11
Functions
11.1 Definition
We use def to define a function, and return to pass back a value:
In [1]: def double(x):

return x*2
print double(5), double([5]), double(’five’)
10 [5, 5] fivefive
11.2 Default Parameters

We can specify default values for parameters:
In [2]: def jeeves(name = "Sir"):

return "Very good, "+ name
In [3]: jeeves()
Out[3]: ’Very good, Sir’
In [4]: jeeves(’James’)
Out[4]: ’Very good, James’
If you have some parameters with defaults, and some without, those with defaults must go later.
11.3 Side effects

Functions can do things to change their mutable arguments, so return is optional.
In [5]: def double_inplace(vec):

vec[:]=[element*2 for element in vec]
z=range(4)
double_inplace(z)
print z
[0, 2, 4, 6]
68
In this example, we’re using [:] to access into the same list, and write it’s data.
vec = [element*2 for element in vec]
would just move a local label, not change the input.

Let’s remind ourselves of this behaviour with a simple array:
In [6]: x=5
x=7
x=[’a’,’b’,’c’]
y=x
In [7]: x
Out[7]: [’a’, ’b’, ’c’]
In [8]: x[:]=["Hooray!", "Yippee"]
In [9]: y
Out[9]: [’Hooray!’, ’Yippee’]
11.4 Early Return

Return without arguments can be used to exit early from a function
In [10]: def extend(to,vec,pad):

if len(vec)>=to:
return
vec[:]=vec+[pad]*(to-len(vec))
In [11]: x=range(3)
extend(6,x,’a’)
print x
[0, 1, 2, ’a’, ’a’, ’a’]
In [12]: z=range(9)
extend(6,z,’a’)
print z
[0, 1, 2, 3, 4, 5, 6, 7, 8]
11.5 Unpacking arguments

If a vector is supplied to a function with a ’*’, its elements are used to fill each of a function’s arguments.
In [13]: def arrow(before, after):

return str(before)+" -> "+str(after)
print arrow(1, 3)
1 -> 3
In [14]: x=[1,-1]
print arrow(*x)
69
1 -> -1
This can be quite powerful:
In [15]: charges={"neutron": 0, "proton": 1, "electron": -1}

for particle in charges.items():
print arrow(*particle)
electron -> -1
proton -> 1
neutron -> 0
11.6 Sequence Arguments

Similiarly, if a * is used in the definition of a function, multiple arguments are absorbed into a list inside
the function:
In [16]: def doubler(*sequence):

return [x*2 for x in sequence]
print doubler(1,2,3)
[2, 4, 6]
11.7 Keyword Arguments

If two asterisks are used, named arguments are supplied as a dictionary:
In [17]: def arrowify(**args):

for key, value in args.items():
print key+" -> "+value
arrowify(neutron="n",proton="p",electron="e")
electron -> e
proton -> p
neutron -> n
70
Chapter 12
Using Libraries
12.1 Import
To use a function or type from a python library, rather than a built-in function or type, we have to import
the library.
In [1]: math.sin(1.6)
---------------------------------------------------------------------------
<ipython-input-1-ecc9cee3d19a> in <module>()
----> 1 math.sin(1.6)
NameError: name ’math’ is not defined
In [2]: import math
In [3]: math.sin(1.6)
Out[3]: 0.9995736030415051
We call these libraries modules:
In [4]: type(math)
Out[4]: module
The tools supplied by a module are attributes of the module, and as such, are accessed with a dot.
In [5]: dir(math)
Out[5]: [’ doc ’,
’ file ’,
’ name ’,
’ package ’,
’acos’,
’acosh’,
71
’asin’,
’asinh’,
’atan’,
’atan2’,
’atanh’,
’ceil’,
’copysign’,
’cos’,
’cosh’,
’degrees’,
’e’,
’erf’,
’erfc’,
’exp’,
’expm1’,
’fabs’,
’factorial’,
’floor’,
’fmod’,
’frexp’,
’fsum’,
’gamma’,
’hypot’,
’isinf’,
’isnan’,
’ldexp’,
’lgamma’,
’log’,
’log10’,
’log1p’,
’modf’,
’pi’,
’pow’,
’radians’,
’sin’,
’sinh’,
’sqrt’,
’tan’,
’tanh’,
’trunc’]
They include properties as well as functions:
In [6]: math.pi
Out[6]: 3.141592653589793
You can always find out where on your storage medium a library has been imported from:
In [7]: print math.__file__[0:50]

print math.__file__[50:]
/usr/local/Cellar/python/2.7.10 2/Frameworks/Pytho
n.framework/Versions/2.7/lib/python2.7/lib-dynload/math.so
72
Note that import does not install libraries from PyPI. It just makes them available to your current
notebook session, assuming they are already installed. Installing libraries is harder, and we’ll cover it later.
So what libraries are available? Until you install more, you might have just the modules that come with
Python, the standard library
Supplementary Materials: Review the list of standard library modules:
https://docs.python.org/2/library/
If you installed via Anaconda, then you also have access to a bunch of modules that are commonly used
in research.
Supplementary Materials: Review the list of modules that are packaged with Anaconda by default:
http://docs.continuum.io/anaconda/pkg-docs.html (The green ticks)
We’ll see later how to add more libraries to our setup.
12.2 Why bother?

Why bother with modules? Why not just have everything available all the time?
The answer is that there are only so many names available! Without a module system, every time I made
a variable whose name matched a function in a library, I’d lose access to it. In the olden days, people ended
up having to make really long variable names, thinking their names would be unique, and they still ended
up with “name clashes”. The module mechanism avoids this.
12.3 Importing from modules

Still, it can be annoying to have to write math.sin(math.pi) instead of sin(pi). Things can be imported
from modules to become part of the current module:
In [8]: import math
math.sin(math.pi)
Out[8]: 1.2246467991473532e-16
In [9]: from math import sin
sin(math.pi)
Out[9]: 1.2246467991473532e-16
Importing one-by-one like this is a nice compromise between typing and risk of name clashes.
It is possible to import everything from a module, but you risk name clashes.
In [10]: from math import *
sin(pi)
Out[10]: 1.2246467991473532e-16
12.4 Import and rename

You can rename things as you import them to avoid clashes or for typing convenience
In [11]: import math as m
m.cos(0)
Out[11]: 1.0
In [12]: pi=3
from math import pi as realpi
print sin(pi), sin(realpi)
0.14112000806 1.22464679915e-16
73
Chapter 13
Defining your own classes
13.1 User Defined Types

A class is a user-programmed Python type.
It is defined like this:
In [1]: class Room(object):

pass
Just as with other python types, you use the name of the type as a function to make a variable of that
type:
In [2]: zero = int()

print type(zero)
<type ’int’>
In [3]: myroom = Room()

print type(myroom)
<class ’ main .Room’>
In the jargon, we say that an object is an instance of a particular class.

Once we have an object with a type of our own devising, we can add properties at will:
In [4]: myroom.name = "Living"
In [5]: print myroom.name
Living
The most common use of a class is to allow us to group data into an object in a way that is easier to
read and understand than organising data into lists and dictionaries.
In [6]: myroom.capacity = 3
myroom.occupants = ["James", "Sue"]
13.2 Methods
So far, our class doesn’t do much!
We define functions inside the definition of a class, in order to give them capabilities, just like the
methods on built-in types.
74
def overfull(self):
return len(self.occupants) > self.capacity
In [8]: myroom = Room()

myroom.capacity = 3
myroom.occupants = ["James", "Sue"]
In [9]: myroom.overfull()
Out[9]: False
In [10]: myroom.occupants.append([’Clare’])
In [11]: myroom.occupants.append([’Bob’])
In [12]: myroom.overfull()
Out[12]: True
When we write methods, we always write the first function argument as self, to refer to the object
instance itself, the argument that goes “before the dot”.
13.3 Constructors
Normally, though, we don’t want to add data to the class attributes on the fly like that. Instead, we define
a constructor that converts input data into an object.

def __init__(self, name, exits, capacity, occupants=[]):
self.name = name
self.occupants = occupants # Note the default argument, occupants start empty
self.exits = exits
self.capacity = capacity
def overfull(self):
return len(self.occupants) > self.capacity
In [14]: living = Room("Living Room", {’north’: ’garden’}, 3)
In [15]: living.capacity
Out[15]: 3
Methods which begin and end with two underscores in their names fulfil special capabilities in Python,
such as constructors.
13.4 Object-oriented design

In building a computer system to model a problem, therefore, we often want to make:
• classes for each kind of thing in our system

• methods for each capability of that kind
• properties (defined in a constructor) for each piece of information describing that kind
For example, the below program might describe our “Maze of Rooms” system:
We define a “Maze” class which can hold rooms:
75
In [16]: class Maze(object):
def __init__(self, name):
self.name = name
self.rooms = {}
def add_room(self, room):

room.maze = self # The Room needs to know which Maze it is a part of
self.rooms[room.name] = room
def occupants(self):
return [occupant for room in self.rooms.values()
for occupant in room.occupants.values()]
def wander(self):
"Move all the people in a random direction"
for occupant in self.occupants():
occupant.wander()
def describe(self):
for room in self.rooms.values():
room.describe()
def step(self):
house.describe()
print
house.wander()
print
def simulate(self, steps):

for _ in range(steps):
self.step()
And a “Room” class with exits, and people:

def __init__(self, name, exits, capacity, maze = None):
self.maze = maze
self.name = name
self.occupants = {} # Note the default argument, occupants start empty
self.exits = exits # Should be a dictionary from directions to room names
def has_space(self):
return len(self.occupants) < self.capacity
def available_exits(self):
return [exit for exit, target in self.exits.items()
if self.maze.rooms[target].has_space() ]
def random_valid_exit(self):
import random
if not self.available_exits():
return None
return random.choice(self.available_exits())
76
def destination(self, exit):
return self.maze.rooms[ self.exits[exit] ]
def add_occupant(self, occupant):

occupant.room = self # The person needs to know which room it is in
self.occupants[occupant.name] = occupant
def delete_occupant(self, occupant):

del self.occupants[occupant.name]
def describe(self):
if self.occupants:
print self.name, ": ", " ".join(self.occupants.keys())
We define a “Person” class for room occupants:
In [18]: class Person(object):

def __init__(self, name, room = None):
self.name=name
def use(self, exit):

self.room.delete_occupant(self)
destination=self.room.destination(exit)
destination.add_occupant(self)
print self.name, "goes", exit, "to the", destination.name
def wander(self):
exit = self.room.random_valid_exit()
if exit:
self.use(exit)
And we use these classes to define our people, rooms, and their relationships:
In [19]: james=Person(’James’)
sue=Person(’Sue’)
bob=Person(’Bob’)
clare=Person(’Clare’)
In [20]: living=Room(’livingroom’, {’outside’:’garden’, ’upstairs’:’bedroom’, ’north’:’kitchen’}, 2)

kitchen=Room(’kitchen’, {’south’:’livingroom’}, 1)
garden=Room(’garden’, {’inside’:’livingroom’}, 3)
bedroom=Room(’bedroom’, {’jump’:’garden’, ’downstairs’: ’livingroom’}, 1)
In [21]: house=Maze(’My House’)
In [22]: for room in [living, kitchen, garden, bedroom]:

house.add_room(room)
In [23]: living.add_occupant(james)
In [24]: garden.add_occupant(sue)
garden.add_occupant(clare)
In [25]: bedroom.add_occupant(bob)
And we can run a “simulation” of our model:
77
In [26]: house.simulate(3)
bedroom : Bob
livingroom : James
garden : Clare Sue
Bob goes jump to the garden

James goes upstairs to the bedroom
Clare goes inside to the livingroom
Sue goes inside to the livingroom
bedroom : James
livingroom : Clare Sue
garden : Bob
James goes jump to the garden

Clare goes north to the kitchen
Sue goes upstairs to the bedroom
Bob goes inside to the livingroom
bedroom : Sue
livingroom : Bob
garden : James
kitchen : Clare
Sue goes downstairs to the livingroom

Bob goes upstairs to the bedroom
James goes inside to the livingroom
13.5 Object oriented design

There are many choices for how to design programs to do this. Another choice would be to separately define
exits as a different class from rooms. This way, we can use arrays instead of dictionaries, but we have to
first define all our rooms, then define all our exits.
In [27]: class Maze(object):

self.name = name
self.rooms = []
self.occupants = []
def add_room(self, name, capacity):

result = Room(name, capacity)
self.rooms.append(result)
return result
def add_exit(self, name, source, target, reverse= None):

source.add_exit(name, target)
if reverse:
target.add_exit(reverse, source)
def add_occupant(self, name, room):

self.occupants.append(Person(name, room))
room.occupancy += 1
78
def wander(self):
for occupant in self.occupants:
occupant.wander()
def describe(self):
occupant.describe()
def step(self):
house.describe()
print
house.wander()
print

self.step()

def __init__(self, name, capacity):
self.name = name
self.occupancy = 0
self.exits = []
return self.occupancy < self.capacity
return [exit for exit in self.exits if exit.valid() ]
import random
return None
def add_exit(self, name, target):

self.exits.append(Exit(name, target))

self.name=name
self.room=room

self.room.occupancy -= 1
destination=exit.target
destination.occupancy +=1
self.room=destination
print self.name, "goes", exit.name, "to the", destination.name
79
def wander(self):
if exit:
self.use(exit)
def describe(self):
print self.name, "is in the", self.room.name
In [30]: class Exit(object):

def __init__(self, name, target):
self.name = name
self.target = target
def valid(self):
return self.target.has_space()
In [31]: house=Maze(’My New House’)
In [32]: living=house.add_room(’livingroom’, 2)
bed = house.add_room(’bedroom’, 1)
garden = house.add_room(’garden’, 3)
kitchen = house.add_room(’kitchen’, 1)
In [33]: house.add_exit(’north’, living, kitchen, ’south’)
In [34]: house.add_exit(’upstairs’, living, bed, ’downstairs’)
In [35]: house.add_exit(’outside’, living, garden, ’inside’)
In [36]: house.add_exit(’jump’,bed, garden)
In [37]: house.add_occupant(’James’, living)

house.add_occupant(’Sue’, garden)
house.add_occupant(’Bob’, bed)
house.add_occupant(’Clare’, garden)
In [38]: house.simulate(3)
James is in the livingroom

Sue is in the garden
Bob is in the bedroom
Clare is in the garden
James goes outside to the garden

Sue goes inside to the livingroom
Bob goes jump to the garden
James is in the garden

Sue is in the livingroom
Bob is in the garden
Clare is in the livingroom
Sue goes north to the kitchen

Bob goes inside to the livingroom
Clare goes outside to the garden
80
James is in the garden
Sue is in the kitchen
Bob is in the livingroom
Clare is in the garden
James goes inside to the livingroom

Bob goes outside to the garden
This is a huge topic, about which many books have been written. The differences between these two
designs are important, and will have long-term consequences for the project. That is the how we start to
think about software engineering, as opposed to learning to program, and is an important part of this
course.
13.6 Exercise: Your own solution

Compare the two solutions above. Discuss with a partner which you like better, and why. Then, starting
from scratch, design your own. What choices did you make that are different from mine?
81
Chapter 14
Python not in the Notebook
We will often want to save our Python classes, for use in multiple Notebooks. We can do this by writing
text files with a .py extension, and then importing them.
14.1 Writing Python in Text Files

You can use a text editor like Atom for Mac or Notepad++ for windows to do this. If you create your own
Python files ending in .py, then you can import them with import just like external libraries.
You can also maintain your library code in a Notebook, and use %%writefile to create your library.
Libraries are usually structured with multiple files, one for each class.
We group our modules into packages, by putting them together into a folder. You can do this with
explorer, or using a shell, or even with Python:
In [1]: import os
if ’mazetool’ not in os.listdir(os.getcwd()):
os.mkdir(’mazetool’)
In [2]: %%writefile mazetool/maze.py

from room import Room
from person import Person
class Maze(object):
self.name = name
self.rooms = []
self.occupants = []
def add_room(self, name, capacity):

result = Room(name, capacity)
self.rooms.append(result)
return result
def add_exit(self, name, source, target, reverse= None):

source.add_exit(name, target)
if reverse:
target.add_exit(reverse, source)
def add_occupant(self, name, room):

self.occupants.append(Person(name, room))
room.occupancy += 1
82
def wander(self):
occupant.wander()
def describe(self):
occupant.describe()
def step(self):
house.describe()
print
house.wander()
print

self.step()
Writing mazetool/maze.py
In [3]: %%writefile mazetool/room.py

from exit import Exit
class Room(object):
def __init__(self, name, capacity):
self.name = name
self.occupancy = 0
self.exits = []
return self.occupancy < self.capacity
return [exit for exit in self.exits if exit.valid() ]
import random
return None
def add_exit(self, name, target):

self.exits.append(Exit(name, target))
Writing mazetool/room.py
In [4]: %%writefile mazetool/person.py
class Person(object):
83
self.name=name
self.room=room

self.room.occupancy -= 1
destination=exit.target
destination.occupancy +=1
self.room=destination
print self.name, "goes", exit.name, "to the", destination.name
def wander(self):
if exit:
self.use(exit)
def describe(self):
print self.name, "is in the", self.room.name
Writing mazetool/person.py
In [5]: %%writefile mazetool/exit.py
class Exit(object):
def __init__(self, name, target):
self.name = name
self.target = target
def valid(self):
return self.target.has_space()
Writing mazetool/exit.py
In order to tell Python that our “mazetool” folder is a Python package, we have to make a special file
called init .py. If you import things in there, they are imported as part of the package:
In [6]: %%writefile mazetool/__init__.py
from maze import Maze
Writing mazetool/ init .py
14.2 Loading Our Package

We just wrote the files, there is no “Maze” class in this notebook yet:
In [7]: myhouse=Maze()
---------------------------------------------------------------------------
<ipython-input-7-fa208048e024> in <module>()
----> 1 myhouse=Maze()
NameError: name ’Maze’ is not defined
84
But now, we can import Maze, (and the other files will get imported via the chained Import statements,
starting from the init .py file.
In [8]: from mazetool import Maze
In [9]: house=Maze(’My New House’)

living=house.add_room(’livingroom’, 2)
Note the files we have created are on the disk in the folder we made:
In [10]: import os
In [11]: os.listdir(os.path.join(os.getcwd(),’mazetool’) )
Out[11]: [’ init .py’,

’ init .pyc’,
’exit.py’,
’exit.pyc’,
’maze.py’,
’maze.pyc’,
’person.py’,
’person.pyc’,
’room.py’,
’room.pyc’]
.pyc files are “Compiled” temporary python files that the system generates to speed things up. They’ll
be regenerated on the fly when your .py files change.
14.3 The Python Path

We want to import these from notebooks elsewhere on our computer: it would be a bad idea to keep all our
Python work in one folder.
Supplementary material The best way to do this is to learn how to make our code into a proper
module that we can install. That’s beyond the scope of this course, but read about setuptools if you want
to know more.
Alternatively, we can add a folder to the “Python Path”, where python searches for modules:
In [12]: import sys

print sys.path[-3]
print sys.path[-2]
print sys.path[-1]
/Library/Python/2.7/site-packages
/usr/local/lib/python2.7/site-packages/IPython/extensions
/Users/ccsprsd/.ipython
In [13]: sys.path.append(os.path.join(’/home/jamespjh/devel/libraries/python’))
In [14]: print sys.path[-1]
/home/jamespjh/devel/libraries/python
I’ve thus added a folder to the list of places searched. If you want to do this permanently, you should set
the PYTHONPATH Environment Variable, which you can learn about in a shell course, or can read about
online for your operating system.
85
Chapter 15
Loading data from files
15.1 Loading data

An important part of this course is about using Python to analyse and visualise data. Most data, of course,
is supplied to us in various formats: spreadsheets, database dumps, or text files in various formats (csv,
tsv, json, yaml, hdf5, netcdf) It is also stored in some medium: on a local disk, a network drive, or on the
internet in various ways. It is important to distinguish the data format, how the data is structured into a
file, from the data’s storage, where it is put.
We’ll look first at the question of data transport: loading data from a disk, and at downloading data
from the internet. Then we’ll look at data parsing: building Python structures from the data. These are
related, but separate questions.
15.2 An example datafile

Let’s write an example datafile to disk so we can investigate it. We’ll just use a plain-text file. IPython
notebook provides a way to do this: if we put %%writefile at the top of a cell, instead of being interpreted
as python, the cell contents are saved to disk.
In [1]: %%writefile mydata.txt

A poet once said, ’The whole universe is in a glass of wine.’
We will probably never know in what sense he meant it,
for poets do not write to be understood.
But it is true that if we look at a glass of wine closely enough we see the entire universe.
There are the things of physics: the twisting liquid which evaporates depending
on the wind and weather, the reflection in the glass;
and our imagination adds atoms.
The glass is a distillation of the earth’s rocks,
and in its composition we see the secrets of the universe’s age, and the evolution of stars.
What strange array of chemicals are in the wine? How did they come to be?
There are the ferments, the enzymes, the substrates, and the products.
There in wine is found the great generalization; all life is fermentation.
Nobody can discover the chemistry of wine without discovering,
as did Louis Pasteur, the cause of much disease.
How vivid is the claret, pressing its existence into the consciousness that watches it!
If our small minds, for some convenience, divide this glass of wine, this universe,
into parts --
physics, biology, geology, astronomy, psychology, and so on --
remember that nature does not know it!
So let us put it all back together, not forgetting ultimately what it is for.
86
Let it give us one more final pleasure; drink it and forget it all!
- Richard Feynman
Writing mydata.txt
Where did that go? It went to the current folder, which for a notebook, by default, is where the notebook
is on disk.
In [2]: import os # The ’os’ module gives us all the tools we need to search in the file system
os.getcwd() # Use the ’getcwd’ function from the ’os’ module to find where we are on disk.
Out[2]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch01data’
Can we see it is there?
In [3]: [x for x in os.listdir(os.getcwd()) if ’.txt’ in x ]
Out[3]: [’mydata.txt’]
Yep! Note how we used a list comprehension to filter all the extraneous files.
15.3 Path independence and os

We can use dirname to get the parent folder for a folder, in a platform independent-way.
In [4]: os.path.dirname(os.getcwd())
Out[4]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher’
We could do this manually using split:
In [5]: "/".join(os.getcwd().split("/")[:-1])
Out[5]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher’
But this would not work on windows, where path elements are separated with a \ instead of a /. So it’s
important to use os.path for this stuff.
Supplementary Materials: If you’re not already comfortable with how files fit into folders, and folders
form a tree, with folders containing subfolders, then look at http://swcarpentry.github.io/shell-novice/01-
filedir.html.
Satisfy yourself that after using %%writedir, you can then find the file on disk with Windows Explorer,
OSX Finder, or the Linux Shell.
We can see how in Python we can investigate the file system with functions in the os module, using just
the same programming approaches as for anything else.
We’ll gradually learn more features of the os module as we go, allowing us to move around the disk, walk
around the disk looking for relevant files, and so on. These will be important to master for automating our
data analyses.
15.4 The python file type

So, let’s read our file:
In [6]: myfile=open(’mydata.txt’)
In [7]: type(myfile)
Out[7]: file
87
We can go line-by-line, by treating the file as an iterable:
In [8]: [x for x in myfile]
Out[8]: ["A poet once said, ’The whole universe is in a glass of wine.’\n",
’We will probably never know in what sense he meant it, \n’,
’for poets do not write to be understood. \n’,
’But it is true that if we look at a glass of wine closely enough we see the entire universe. \
’There are the things of physics: the twisting liquid which evaporates depending\n’,
’on the wind and weather, the reflection in the glass;\n’,
’and our imagination adds atoms.\n’,
"The glass is a distillation of the earth’s rocks,\n",
"and in its composition we see the secrets of the universe’s age, and the evolution of stars. \
’What strange array of chemicals are in the wine? How did they come to be? \n’,
’There are the ferments, the enzymes, the substrates, and the products.\n’,
’There in wine is found the great generalization; all life is fermentation.\n’,
’Nobody can discover the chemistry of wine without discovering, \n’,
’as did Louis Pasteur, the cause of much disease.\n’,
’How vivid is the claret, pressing its existence into the consciousness that watches it!\n’,
’If our small minds, for some convenience, divide this glass of wine, this universe, \n’,
’into parts -- \n’,
’physics, biology, geology, astronomy, psychology, and so on -- \n’,
’remember that nature does not know it!\n’,
’\n’,
’So let us put it all back together, not forgetting ultimately what it is for.\n’,
’Let it give us one more final pleasure; drink it and forget it all!\n’,
’ - Richard Feynman’]
If we do that again, the file has already finished, there is no more data.
In [9]: [x for x in myfile]
Out[9]: []
We need to ‘rewind’ it!
In [10]: myfile.seek(0)
[len(x) for x in myfile if ’ut’ in x]
Out[10]: [94, 94, 64, 78]
It’s really important to remember that a file is a different built in type than a string.
15.5 Working with files.

We can read one line at a time with readline:
first = myfile.readline()
In [12]: first
Out[12]: "A poet once said, ’The whole universe is in a glass of wine.’\n"
In [13]: second=myfile.readline()
In [14]: second
88
Out[14]: ’We will probably never know in what sense he meant it, \n’
We can read the whole remaining file with read:
In [15]: rest=myfile.read()
In [16]: rest
Out[16]: "for poets do not write to be understood. \nBut it is true that if we look at a glass of wine c
Which means that when a file is first opened, read is useful to just get the whole thing as a string:
In [17]: open(’mydata.txt’).read()
Out[17]: "A poet once said, ’The whole universe is in a glass of wine.’\nWe will probably never know in
You can also read just a few characters:
In [19]: myfile.read(15)
Out[19]: ’\n - Richard F’
15.6 Converting Strings to Files

Because files and strings are different types, we CANNOT just treat strings as if they were files:
In [20]: mystring= "Hello World\n My name is James"
In [21]: mystring
Out[21]: ’Hello World\n My name is James’
In [22]: mystring.readline()
---------------------------------------------------------------------------
<ipython-input-22-8c85154fa12a> in <module>()
----> 1 mystring.readline()
AttributeError: ’str’ object has no attribute ’readline’
This is important, because some file format parsers expect input from a file and not a string. We can
convert between them using the StringIO module in the standard library:
In [23]: from StringIO import StringIO
In [24]: mystringasafile=StringIO(mystring)
In [25]: mystringasafile.readline()
Out[25]: ’Hello World\n’
In [26]: mystringasafile.readline()
Out[26]: ’ My name is James’
Note that in a string, \n is used to represent a newline.
89
15.7 Closing files
We really ought to close files when we’ve finished with them, as it makes the computer more efficient. (On
a shared computer, this is particuarly important)
In [27]: myfile.close()
Because it’s so easy to forget this, python provides a context manager to open a file, then close it
automatically at the end of an indented block:
In [28]: with open(’mydata.txt’) as somefile:

content = somefile.read()
print content
A poet once said, ’The whole universe is in a glass of wine.’

We will probably never know in what sense he meant it,
for poets do not write to be understood.
But it is true that if we look at a glass of wine closely enough we see the entire universe.
There are the things of physics: the twisting liquid which evaporates depending
on the wind and weather, the reflection in the glass;
and our imagination adds atoms.
The glass is a distillation of the earth’s rocks,
and in its composition we see the secrets of the universe’s age, and the evolution of stars.
What strange array of chemicals are in the wine? How did they come to be?
There are the ferments, the enzymes, the substrates, and the products.
There in wine is found the great generalization; all life is fermentation.
Nobody can discover the chemistry of wine without discovering,
as did Louis Pasteur, the cause of much disease.
How vivid is the claret, pressing its existence into the consciousness that watches it!
If our small minds, for some convenience, divide this glass of wine, this universe,
into parts --
physics, biology, geology, astronomy, psychology, and so on --
remember that nature does not know it!
So let us put it all back together, not forgetting ultimately what it is for.
Let it give us one more final pleasure; drink it and forget it all!
- Richard Feynman
The code to be done while the file is open is indented, just like for an if statement.
In [29]: print somefile
<closed file ’mydata.txt’, mode ’r’ at 0x10f2b30c0>
You should pretty much always use this syntax for working with files.
15.8 Writing files

We might want to create a file from a string in memory. We can’t do this with the notebook’s %%writefile
– this is just a notebook convenience, and isn’t very programmable.
When we open a file, we can specify a ‘mode’, in this case, ‘w’ for writing. (‘r’ for reading is the default.)
In [30]: with open(’mywrittenfile’, ’w’) as target:

target.write(’Hello’)
target.write(’World’)
90
In [31]: with open(’mywrittenfile’,’r’) as source:
print source.read()
HelloWorld
And we can “append” to a file with mode ‘a’:
In [32]: with open(’mywrittenfile’, ’a’) as target:

target.write(’Hello’)
target.write(’James’)
In [33]: with open(’mywrittenfile’,’r’) as source:

print source.read()
HelloWorldHelloJames
91
Chapter 16
Getting data from the Internet
We’ve seen about obtaining data from our local file system.
The other common place today that we might want to obtain data is from the internet.
It’s very common today to treat the web as a source and store of information; we need to be able to
programmatically download data, and place it in python objects.
We may also want to be able to programmatically upload data, for example, to automatically fill in forms.
This can be really powerful if we want to, for example, do automated metaanalysis across a selection of
research papers.
16.1 URLs
All internet resources are defined by a Uniform Resource Locator.
In [1]: "http://maps.googleapis.com:80/maps/api/staticmap?size=400x400&center=51.51,-0.1275&zoom=12"
Out[1]: ’http://maps.googleapis.com:80/maps/api/staticmap?size=400x400&center=51.51,-0.1275&zoom=12’
A url consists of:
• A scheme (http, https, ssh, . . . )

• A host (maps.googleapis.com, the name of the remote computer you want to talk to)
• A port (optional, most protocols have a typical port associated with them, e.g. 80 for http)
• A path (Like a file path on the machine, here it is maps/api/staticmap)
• A query part after a ?, (optional, usually ampersand-separated parameters e.g. size=400x400, or
zoom=12)
Supplementary materials: These can actually be different for different protocols, the above is a
simplification, you can see more, for example, at https://en.wikipedia.org/wiki/URI scheme
URLs are not allowed to include all characters; we need to, for example, “escape” a space that appears
inside the URL, replacing it with %20, so e.g. a request of http://some example.com/ would need to be
http://some%20example.com/
Supplementary materials: The code used to replace each character is the ASCII code for it.
Supplementary materials: The escaping rules a are quite subtle. See
https://en.wikipedia.org/wiki/Percent-encoding
16.2 Requests
The python requests library can help us manage and manipulate URLs. It is easier to use than the ‘urllib’
library that is part of the standard library, and is included with anaconda and canopy. It sorts out escaping,
parameter encoding, and so on for us.
To request the above URL, for example, we write:
92
In [3]: response = requests.get("http://maps.googleapis.com/maps/api/staticmap",

params={
’size’ : ’400x400’,
’center’ : ’51.51,-0.1275’,
’zoom’ : 12
})
In [4]: response.url
Out[4]: u’http://maps.googleapis.com/maps/api/staticmap?zoom=12&center=51.51%2C-0.1275&size=400x400’
When we do a request, the result comes back as text. For the png image in the above, this isn’t very
readable:
In [5]: response.content[0:10]
Out[5]: ’\x89PNG\r\n\x1a\n\x00\x00’
Just as for file access, therefore, we will need to send the text we get to a python module which understands
that file format.
Again, it is important to separate the transport model, (e.g. a file system, or an “http request” for the
web, from the data model of the data that is returned.)
16.3 Example: Sunspots

Let’s try to get something scientific: the sunspot cycle data from http://sidc.be/silso/home:
In [6]: spots=requests.get(’http://www.sidc.be/silso/INFO/snmtotcsv.php’).text
In [7]: spots[0:80]
Out[7]: u’1749;01;1749.042; 96.7; -1.0; -1;1\n1749;02;1749.123; 104.3; -1.0; -1;1\n1749’
This looks like semicolon-separated data, with different records on different lines. (Line separators come
out as \n)
There are many many scientific datasets which can now be downloaded like this - integrating the download
into your data pipeline can help to keep your data flows organised.
16.4 Writing our own Parser

We’ll need a python library to handle semicolon-separated data like the sunspot data.
You might be thinking: “But I can do that myself!”:
In [8]: lines=spots.split("\n")
lines[0:5]
Out[8]: [u’1749;01;1749.042; 96.7; -1.0; -1;1’,

u’1749;02;1749.123; 104.3; -1.0; -1;1’,
u’1749;03;1749.204; 116.7; -1.0; -1;1’,
u’1749;04;1749.288; 92.8; -1.0; -1;1’,
u’1749;05;1749.371; 141.7; -1.0; -1;1’]
In [9]: years=[line.split(";")[0] for line in lines]
93
In [10]: years[0:15]
Out[10]: [u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1749’,
u’1750’,
u’1750’,
u’1750’]
But don’t: what if, for example, one of the records contains a separator inside it; most computers will
put the content in quotes, so that, for example,
"something; something"; something; something
has three fields, the first of which is
something; something
The naive code above would give four fields, of which the first is
"Something
You’ll never manage to get all that right; so you’ll be better off using a library to do it.
16.5 Writing data to the internet

Note that we’re using requests.get. get is used to receive data from the web. You can also use post to
fill in a web-form programmatically.
Supplementary material: Learn about using post with requests.
Supplementary material: Learn about the different kinds of http request: Get, Post, Put, Delete. . .
This can be used for all kinds of things, for example, to programmatically add data to a web resource.
It’s all well beyond our scope for this course, but it’s important to know it’s possible, and start to think
about the scientific possibilities.
94
Chapter 17
Field and Record Data
17.1 Separated Value Files

Let’s carry on with our sunspots example:
spots=requests.get(’http://www.sidc.be/silso/INFO/snmtotcsv.php’).text
spots.split(’\n’)[0]
Out[1]: u’1749;01;1749.042; 96.7; -1.0; -1;1’
We want to work programmatically with Separated Value files.
These are files which have:
• Each record on a line
• Each record has multiple fields
• Fields are separated by some separator
Typical separators are the space, tab, comma, and semicolon separated values files, e.g.:
• Space separated value (e.g. field1, "field two", field3 )
• Comma separated value (e.g. ‘field1, another field, “wow, another field”)
Comma-separated-value is abbreviated CSV, and tab separated value TSV.
CSV is also used to refer to all the different sub-kinds of separated value files, i.e. some people use csv
to refer to tab, space and semicolon separated files.
CSV is not a particularly superb data format, because it forces your data model to be a list of lists.
Richer file formats describe “serialisations” for dictionaries and for deeper-than-two nested list structures as
well.
Nevertheless, because you can always export spreadsheets as CSV files, (each cell is a field, each row is a
record) CSV files are very popular.
17.2 CSV variants.

Some CSV formats define a comment character, so that rows beginning with, e.g., a #, are not treated as
data, but give a human index.
Some CSV formats define a three-deep list structure, where a double-newline separates records into
blocks.
Some CSV formats assume that the first line defines the names of the fields, e.g.:
name, age, job
James, 38, Programmer
Elizabeth, 89, Queen
95
17.3 Python CSV readers
The Python standard library has a csv module. However, it’s less powerful than the CSV capabilities in
numpy, the main scientific python library for handling data. Numpy is destributed with Anaconda and
Canopy, so we recommend you just use that.
Numpy has powerful capabilities for handling matrices, and other fun stuff, and we’ll learn about these
later in the course, but for now, we’ll just use numpy’s CSV reader, and assume it makes us lists and
dictionaries, rather than it’s more exciting array type.
In [2]: import numpy as np

from StringIO import StringIO
In [3]: sunspots= np.genfromtxt(StringIO(spots), delimiter=’;’)
genfromtxt is a powerful CSV reader. I used the delimiter optional argument to specify the delimeter.
I could also specify names=True if I had a first line naming fields, and comments=# if I had comment lines.
In [4]: sunspots
Out[4]: array([[ 1.74900000e+03, 1.00000000e+00, 1.74904200e+03, ...,

-1.00000000e+00, -1.00000000e+00, 1.00000000e+00],
[ 1.74900000e+03, 2.00000000e+00, 1.74912300e+03, ...,
-1.00000000e+00, -1.00000000e+00, 1.00000000e+00],
[ 1.74900000e+03, 3.00000000e+00, 1.74920400e+03, ...,
-1.00000000e+00, -1.00000000e+00, 1.00000000e+00],
...,
[ 2.01500000e+03, 8.00000000e+00, 2.01562300e+03, ...,
5.00000000e+00, 7.68000000e+02, 0.00000000e+00],
[ 2.01500000e+03, 9.00000000e+00, 2.01570700e+03, ...,
6.60000000e+00, 8.04000000e+02, 0.00000000e+00],
[ 2.01500000e+03, 1.00000000e+01, 2.01579000e+03, ...,
5.30000000e+00, 6.05000000e+02, 0.00000000e+00]])
We can now plot the “Sunspot cycle”:
In [5]: %matplotlib inline

from matplotlib import pyplot as plt
plt.plot(sunspots[:,0], sunspots[:,3]) # Numpy syntax to access all
#rows, specified column.
Out[5]: [<matplotlib.lines.Line2D at 0x1122a0390>]
96
The plot command accepted an array of ‘X’ values and an array of ‘Y’ values. We used a special NumPy
“:” syntax, which we’ll learn more about later.
17.4 Naming Columns

I happen to know that the columns here are defined as follows:
From http://www.sidc.be/silso/infosnmtot:
CSV
Filename: SN m tot V2.0.csv Format: Comma Separated values (adapted for import in spread-
sheets) The separator is the semicolon ‘;’.
Contents: * Column 1-2: Gregorian calendar date - Year - Month * Column 3: Date in fraction
of year. * Column 4: Monthly mean total sunspot number. * Column 5: Monthly mean standard
deviation of the input sunspot numbers. * Column 6: Number of observations used to compute
the monthly mean total sunspot number. * Column 7: Definitive/provisional marker. ‘1’ indicates
that the value is definitive. ‘0’ indicates that the value is still provisional.
I can actually specify this to the formatter:
In [6]: sunspots= np.genfromtxt(StringIO(spots), delimiter=’;’,

names=[’year’,’month’,’date’,
’mean’,’deviation’,’observations’,’definitive’])
In [7]: sunspots
Out[7]: array([(1749.0, 1.0, 1749.042, 96.7, -1.0, -1.0, 1.0),

(1749.0, 2.0, 1749.123, 104.3, -1.0, -1.0, 1.0),
(1749.0, 3.0, 1749.204, 116.7, -1.0, -1.0, 1.0), ...,
(2015.0, 8.0, 2015.623, 64.6, 5.0, 768.0, 0.0),
97
(2015.0, 9.0, 2015.707, 78.1, 6.6, 804.0, 0.0),
(2015.0, 10.0, 2015.79, 61.7, 5.3, 605.0, 0.0)],
dtype=[(’year’, ’<f8’), (’month’, ’<f8’), (’date’, ’<f8’), (’mean’, ’<f8’), (’deviation’,
17.5 Typed Fields

It’s also often good to specify the datatype of each field.
In [8]: sunspots= np.genfromtxt(StringIO(spots), delimiter=’;’,
names=[’year’,’month’,’date’,
’mean’,’deviation’,’observations’,’definitive’],
dtype=[int, int, float, float, float, int, int])
In [9]: sunspots
Out[9]: array([(1749, 1, 1749.042, 96.7, -1.0, -1, 1),
(1749, 2, 1749.123, 104.3, -1.0, -1, 1),
(1749, 3, 1749.204, 116.7, -1.0, -1, 1), ...,
(2015, 8, 2015.623, 64.6, 5.0, 768, 0),
(2015, 9, 2015.707, 78.1, 6.6, 804, 0),
(2015, 10, 2015.79, 61.7, 5.3, 605, 0)],
dtype=[(’year’, ’<i8’), (’month’, ’<i8’), (’date’, ’<f8’), (’mean’, ’<f8’), (’deviation’,
Now, NumPy understands the names of the columns, so our plot command is more readable:
In [10]: sunspots[’year’]
Out[10]: array([1749, 1749, 1749, ..., 2015, 2015, 2015])
In [11]: plt.plot(sunspots[’year’],sunspots[’mean’])
Out[11]: [<matplotlib.lines.Line2D at 0x112371990>]
98
Chapter 18
Structured Data
18.1 Structured data

CSV files can only model data where each record has several fields, and each field is a simple datatype, a
string or number.
We often want to store data which is more complicated than this, with nested structures of lists and
dictionaries. Structured data formats like Json, YAML, and XML are designed for this.
18.2 Json
A very common structured data format is JSON.
This allows us to represent data which is combinations of lists and dictionaries as a text file which looks
a bit like a Javascript (or Python) data literal.
In [1]: import json
Any nested group of dictionaries and lists can be saved:
In [2]: mydata = {’key’: [’value1’, ’value2’], ’key2’: {’key4’:’value3’}}
In [3]: json.dumps(mydata)
Out[3]: ’{"key2": {"key4": "value3"}, "key": ["value1", "value2"]}’
Loading data is also really easy:
In [4]: %%writefile myfile.json
{
"somekey": ["a list", "with values"]
}
Writing myfile.json
In [5]: mydata=json.load(open(’myfile.json’))
In [6]: mydata
Out[6]: {u’somekey’: [u’a list’, u’with values’]}
In [7]: mydata[’somekey’]
Out[7]: [u’a list’, u’with values’]
This is a very nice solution for loading and saving python datastructures.
It’s a very common way of transferring data on the internet, and of saving datasets to disk.
There’s good support in most languages, so it’s a nice inter-language file interchange format.
99
18.3 Unicode
Supplementary Material: Why do the strings come back with ‘u’ everywhere? These are Unicode
Strings, designed to hold hold all the world’s characters.
18.4 Yaml
Yaml is a very similar dataformat to Json, with some nice additions:
• You don’t need to quote strings if they don’t have funny characters in
• You can have comment lines, beginning with a #
• You can write dictionaries without the curly brackets: it just notices the colons.
• You can write lists like this:
In [8]: %%writefile myfile.yaml

somekey:
- a list # Look, this is a list
- with values
Writing myfile.yaml
In [9]: import yaml
In [10]: yaml.load(open(’myfile.yaml’))
Out[10]: {’somekey’: [’a list’, ’with values’]}
Yaml is my favourite format for ad-hoc datafiles, but the library doesn’t ship with default Python,
(though it is part of Anaconda and Canopy) so some people still prefer Json for it’s univerality.
Because Yaml gives the option of serialising a list either as newlines with dashes, or with square brackets,
you can control this choice:
In [11]: yaml.dump(mydata)
Out[11]: "!!python/unicode ’somekey’: [!!python/unicode ’a list’, !!python/unicode ’with values’]\n"
In [12]: yaml.dump(mydata, default_flow_style=False)
Out[12]: "!!python/unicode ’somekey’:\n- !!python/unicode ’a list’\n- !!python/unicode ’with values’\n"
18.5 XML
Supplementary material : XML is another popular choice when saving nested data structures. It’s very
careful, but verbose. If your field uses XML data, you’ll need to learn a python XML parser, (there are a
few), and about how XML works.
18.6 Exercise:
Use YAML or JSON to save your maze datastructure to disk and load it again.
100
18.7 Solution: Saving and Loading a Maze
In [1]: house = {
’living’ : {
’exits’: {
},
’capacity’ : 2
},
’kitchen’ : {
’exits’: {
},
’people’ : [],
’capacity’ : 1
},
’garden’ : {
’exits’: {
},
’capacity’ : 3
},
’bedroom’ : {
’exits’: {
},
’people’ : [],
’capacity’ : 1
}
}
Save the maze with json:
In [2]: import json
In [3]: with open(’maze.json’,’w’) as json_maze_out:

json_maze_out.write(json.dumps(house))
Consider the file on the disk:
In [4]: %%bash
cat ’maze.json’
{"living": {"capacity": 2, "exits": {"outside": "garden", "north": "kitchen", "upstairs": "bedroom"}, "p
and now load it into a different variable:
In [5]: with open(’maze.json’) as json_maze_in:

maze_again = json.load(json_maze_in)
In [6]: maze_again
101
Out[6]: {u’bedroom’: {u’capacity’: 1,
u’exits’: {u’downstairs’: u’living’, u’jump’: u’garden’},
u’people’: []},
u’garden’: {u’capacity’: 3,
u’exits’: {u’inside’: u’living’},
u’people’: [u’Sue’]},
u’kitchen’: {u’capacity’: 1, u’exits’: {u’south’: u’living’}, u’people’: []},
u’living’: {u’capacity’: 2,
u’exits’: {u’north’: u’kitchen’,
u’outside’: u’garden’,
u’upstairs’: u’bedroom’},
u’people’: [u’James’]}}
Or with YAML:
In [7]: import yaml
In [8]: with open(’maze.yaml’,’w’) as yaml_maze_out:

yaml_maze_out.write(yaml.dump(house))
In [9]: %%bash
cat ’maze.yaml’
bedroom:
capacity: 1
exits: {downstairs: living, jump: garden}
people: []
garden:
capacity: 3
exits: {inside: living}
people: [Sue]
kitchen:
capacity: 1
exits: {south: living}
people: []
living:
capacity: 2
exits: {north: kitchen, outside: garden, upstairs: bedroom}
people: [James]
In [10]: with open(’maze.yaml’) as yaml_maze_in:

maze_again = yaml.load(yaml_maze_in)
In [11]: maze_again
Out[11]: {’bedroom’: {’capacity’: 1,

’exits’: {’downstairs’: ’living’, ’jump’: ’garden’},
’people’: []},
’garden’: {’capacity’: 3, ’exits’: {’inside’: ’living’}, ’people’: [’Sue’]},
’kitchen’: {’capacity’: 1, ’exits’: {’south’: ’living’}, ’people’: []},
’living’: {’capacity’: 2,
’exits’: {’north’: ’kitchen’, ’outside’: ’garden’, ’upstairs’: ’bedroom’},
’people’: [’James’]}}
102
Chapter 19
Unassessed Exercise: the biggest

Earthquake in the UK this Century
19.1 The Problem

GeoJSON is a json-based file format for sharing geographic data. One example dataset is the USGS earth-
quake data:

quakes=requests.get("http://earthquake.usgs.gov/fdsnws/event/1/query.geojson",
params={
’starttime’:"2000-01-01",
"maxlatitude":"58.723",
"minlatitude":"50.008",
"maxlongitude":"1.67",
"minlongitude":"-9.756",
"minmagnitude":"1",
"endtime":"2015-07-13",
"orderby":"time-asc"}
)
In [2]: quakes.text[0:50]
Out[2]: u’{"type":"FeatureCollection","metadata":{"generated’
Your exercise: determine the location of the largest magnitude earthquake in the UK this century.
You’ll need to: * Get the text of the web result * Parse the data as JSON * Understand how the data
is structured into dictionaries and lists * Where is the magnitude? * Where is the place description or
coordinates? * Program a search through all the quakes to find the biggest quake. * Find the place of the
biggest quake * Form a URL for Google Maps at that latitude and longitude: look back at the introductory
example * Display that image
19.2 Download the data

quakes_response=requests.get("http://earthquake.usgs.gov/fdsnws/event/1/query.geojson",
params={
’starttime’:"2000-01-01",
"maxlatitude":"58.723",
"minlatitude":"50.008",
103
"maxlongitude":"1.67",
"minlongitude":"-9.756",
"minmagnitude":"1",
"endtime":"2015-07-13",
"orderby":"time-asc"}
)
19.3 Parse the data as JSON

In [2]: import json
In [3]: requests_json = json.loads(quakes_response.text)
19.4 Investigate the data to discover how it is structured.

In [4]: type(requests_json)
Out[4]: dict
In [5]: requests_json.keys()
Out[5]: [u’type’, u’features’, u’bbox’, u’metadata’]
In [6]: len(requests_json[’features’])
Out[6]: 110
In [7]: requests_json[’features’][0].keys()
Out[7]: [u’geometry’, u’type’, u’properties’, u’id’]
In [8]: requests_json[’features’][0][’properties’][’mag’]
Out[8]: 2.6
In [9]: requests_json[’features’][0][’geometry’]
Out[9]: {u’coordinates’: [-2.81, 54.77, 14], u’type’: u’Point’}
19.5 Find the largest quake

In [10]: quakes = requests_json[’features’]
In [11]: largest_so_far = quakes[0]

for quake in quakes:
if quake[’properties’][’mag’] > largest_so_far[’properties’][’mag’]:
largest_so_far = quake
print largest_so_far[’properties’][’mag’]
4.8
In [12]: lat=largest_so_far[’geometry’][’coordinates’][1]
long=largest_so_far[’geometry’][’coordinates’][0]
print "Latitude:", lat, "Longitude:", long
Latitude: 52.52 Longitude: -2.15
104
19.6 Get a map at the point of the quake
def request_map_at(lat,long, satellite=False,zoom=12,size=(400,400),sensor=False):
params=dict(
zoom= zoom,
center= ",".join(map(str,(lat,long)))
)
if satellite:
In [14]: import IPython

map_png=request_map_at(lat, long,zoom=10)
19.7 Display the map

In [15]: IPython.core.display.Image(map_png.content)
Out[15]:
105
106
Chapter 20
Plotting with Matplotlib
20.1 Importing Matplotlib

We import the ‘pyplot’ object from Matplotlib, which provides us with an interface for making figures. We
usually abbreviate it.
In [1]: from matplotlib import pyplot as plt
20.2 Notebook magics

When we write:
We tell the IPython notebook to show figures we generate alongside the code that created it, rather than
in a separate window. Lines beginning with a single percent are not python code: they control how the
notebook deals with python code.
Lines beginning with two percents are “cell magics”, that tell IPython notebook how to interpret the
particular cell; we’ve seen %%writefile, for example.
20.3 A basic plot

When we write:
In [3]: from math import sin, cos, pi

plt.plot([sin(pi*x/100.0) for x in range(100)])
Out[3]: [<matplotlib.lines.Line2D at 0x11265e090>]
107
The plot command returns a figure, just like the return value of any function. The notebook then displays
this.
To add a title, axis labels etc, we need to get that figure object, and manipulate it. For convenience,
matplotlib allows us to do this just by issuing commands to change the “current figure”:
In [4]: plt.plot([sin(pi*x/100.0) for x in range(100)])

plt.title("Hello")
Out[4]: <matplotlib.text.Text at 0x1126f58d0>
108
But this requires us to keep all our commands together in a single cell, and makes use of a “global” single
“current plot”, which, while convenient for quick exploratory sketches, is a bit cumbersome. To produce
from our notebook proper plots to use in papers, Python’s plotting library, matplotlib, defines some types
we can use to treat individual figures as variables, and manipulate this.
20.4 Figures and Axes

We often want multiple graphs in a single figure (e.g. for figures which display a matrix of graphs of different
variables for comparison).
So Matplotlib divides a figure object up into axes: each pair of axes is one ‘subplot’. To make a boring
figure with just one pair of axes, however, we can just ask for a default new figure, with brand new axes
In [5]: sine_graph, sine_graph_axes=plt.subplots();
109
Once we have some axes, we can plot a graph on them:
In [6]: sine_graph_axes.plot([sin(pi*x/100.0) for x in range(100)], label=’sin(x)’)
Out[6]: [<matplotlib.lines.Line2D at 0x112804150>]
We can add a title to a pair of axes:
In [7]: sine_graph_axes.set_title("My graph")
Out[7]: <matplotlib.text.Text at 0x11280da10>
In [8]: sine_graph_axes.set_ylabel("f(x)")
Out[8]: <matplotlib.text.Text at 0x1127c8810>
In [9]: sine_graph_axes.set_xlabel("100 x")
Out[9]: <matplotlib.text.Text at 0x1126f5510>
Now we need to actually display the figure. As always with the notebook, if we make a variable be
returned by the last line of a code cell, it gets displayed:
In [10]: sine_graph
Out[10]:
110
We can add another curve:
In [11]: sine_graph_axes.plot([cos(pi*x/100.0) for x in range(100)], label=’cos(x)’)
Out[11]: [<matplotlib.lines.Line2D at 0x1127e5ad0>]
In [12]: sine_graph
Out[12]:
111
A legend will help us distinguish the curves:
In [13]: sine_graph_axes.legend()
Out[13]: <matplotlib.legend.Legend at 0x1127e5b50>
In [14]: sine_graph
Out[14]:
112
20.5 Saving figures.
We must be able to save figures to disk, in order to use them in papers. This is really easy:
In [15]: sine_graph.savefig(’my_graph.png’)
In order to be able to check that it worked, we need to know how to display an arbitrary image in the
notebook.
The programmatic way is like this:
In [16]: import IPython # Get the notebook’s own library for manipulating itself.
IPython.core.display.Image(open(’my_graph.png’).read())
Out[16]:
113
20.6 Subplots
We might have wanted the sin and cos graphs on separate axes:
In [17]: double_graph=plt.figure()
<matplotlib.figure.Figure at 0x1128edf10>
In [18]: sin_axes=double_graph.add_subplot(2,1,1)
In [19]: cos_axes=double_graph.add_subplot(2,1,2)
In [20]: sin_axes.plot([sin(pi*x/100.0) for x in range(100)])
Out[20]: [<matplotlib.lines.Line2D at 0x112ab9610>]
In [21]: sin_axes.set_ylabel("sin(x)")
In [22]: cos_axes.plot([cos(pi*x/100.0) for x in range(100)])
Out[22]: [<matplotlib.lines.Line2D at 0x112ab9d10>]
In [23]: cos_axes.set_ylabel("cos(x)")
Out[23]: <matplotlib.text.Text at 0x112a2a490>
114
In [24]: cos_axes.set_xlabel("100 x")
Out[24]: <matplotlib.text.Text at 0x112a14650>
In [25]: double_graph
Out[25]:
20.7 Versus plots

When we specify a single list to plot, the x-values are just the array index number. We usually want to
plot something more meaningful:
In [26]: double_graph=plt.figure()
sin_axes=double_graph.add_subplot(2,1,1)
cos_axes=double_graph.add_subplot(2,1,2)
cos_axes.set_ylabel("cos(x)")
sin_axes.set_ylabel("sin(x)")
cos_axes.set_xlabel("x")
115
In [27]: sin_axes.plot([x/100.0 for x in range(100)], [sin(pi*x/100.0) for x in range(100)])
cos_axes.plot([x/100.0 for x in range(100)], [cos(pi*x/100.0) for x in range(100)])
Out[27]: [<matplotlib.lines.Line2D at 0x112c6c8d0>]
In [28]: double_graph
Out[28]:
116
20.8 Learning More
There’s so much more to learn about matplotlib: pie charts, bar charts, heat maps, 3-d plotting, animated
plots, and so on. You can learn all this via the Matplotlib Website. You should try to get comfortable with
all this, so please use some time in class, or at home, to work your way through a bunch of the examples.
117
Chapter 21
NumPy
21.1 The Scientific Python Trilogy

Why is Python so popular for research work?
MATLAB has typically been the most popular “language of technical computing”, with strong built-in
support for efficient numerical analysis with matrices (the mat in MATLAB is for Matrix, not Maths), and
plotting.
Other dynamic languages have cleaner, more logical syntax (Ruby, Scheme)
But Python users developed three critical libraries, matching the power of MATLAB for scientific work:
• Matplotlib, the plotting library created by John D. Hunter

• NumPy, a fast matrix maths library created by Travis Oliphant
• IPython, the precursor of the notebook, created by Fernando Perez
By combining a plotting library, a matrix maths library, and an easy-to-use interface allowing live plotting
commands in a persistent environment, the powerful capabilities of MATLAB were matched by a free and
open toolchain.
We’ve learned about Matplotlib and IPython in this course already. NumPy is the last part of the trilogy.
21.2 Limitations of Python Lists

The normal Python List is just one dimensional. To make a matrix, we have to nest Python arrays:
In [1]: x= [range(5) for N in range(5)]
In [2]: x
Out[2]: [[0, 1, 2, 3, 4],

[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4],
[0, 1, 2, 3, 4]]
Applying an operation to every element is a pain:
In [3]: x+5
---------------------------------------------------------------------------
118
<ipython-input-3-7b83a566c210> in <module>()
----> 1 x+5
TypeError: can only concatenate list (not "int") to list
In [4]: [[elem +5 for elem in row] for row in x]

Out[4]: [[5, 6, 7, 8, 9],
[5, 6, 7, 8, 9],
[5, 6, 7, 8, 9],
[5, 6, 7, 8, 9],
[5, 6, 7, 8, 9]]
Common useful operations like transposing a matrix or reshaping a 10 by 10 matrix into a 20 by 5 matrix
are not easy to code in raw Python lists.
21.3 The NumPy array

NumPy’s array type represents a multidimensional matrix Mi,j,k...n
The NumPy array seems at first to be just like a list:
my_array=np.array(range(5))
In [6]: my_array
Out[6]: array([0, 1, 2, 3, 4])
In [7]: my_array[2]
Out[7]: 2
In [8]: for element in my_array:
print "Hello" * element
Hello
HelloHello
HelloHelloHello
HelloHelloHelloHello
We can also see our first weakness of NumPy arrays versus Python lists:
In [9]: my_array.append(4)
---------------------------------------------------------------------------
<ipython-input-9-ad82621ab44a> in <module>()
----> 1 my array.append(4)
AttributeError: ’numpy.ndarray’ object has no attribute ’append’
For NumPy arrays, you typically don’t change the data size once you’ve defined your array, whereas for
Python lists, you can do this efficiently. However, you get back lots of goodies in return. . .
119
21.4 Elementwise Operations
But most operations can be applied element-wise automatically!
In [10]: my_array + 2
Out[10]: array([2, 3, 4, 5, 6])
These “vectorized” operations are very fast:
In [11]: big_list=range(10000)
big_array=np.arange(10000)
In [12]: %%timeit
[x**2 for x in big_list]
1000 loops, best of 3: 781 µs per loop
In [13]: %%timeit
big_array**2
The slowest run took 8.86 times longer than the fastest. This could mean that an intermediate result is
100000 loops, best of 3: 7.46 µs per loop
21.5 Arange and linspace

NumPy has two easy methods for defining floating-point evenly spaced arrays:
In [14]: x=np.arange(0,10,0.1)
x
Out[14]: array([ 0. , 0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, 0.9, 1. ,
1.1, 1.2, 1.3, 1.4, 1.5, 1.6, 1.7, 1.8, 1.9, 2. , 2.1,
2.2, 2.3, 2.4, 2.5, 2.6, 2.7, 2.8, 2.9, 3. , 3.1, 3.2,
3.3, 3.4, 3.5, 3.6, 3.7, 3.8, 3.9, 4. , 4.1, 4.2, 4.3,
4.4, 4.5, 4.6, 4.7, 4.8, 4.9, 5. , 5.1, 5.2, 5.3, 5.4,
5.5, 5.6, 5.7, 5.8, 5.9, 6. , 6.1, 6.2, 6.3, 6.4, 6.5,
6.6, 6.7, 6.8, 6.9, 7. , 7.1, 7.2, 7.3, 7.4, 7.5, 7.6,
7.7, 7.8, 7.9, 8. , 8.1, 8.2, 8.3, 8.4, 8.5, 8.6, 8.7,
8.8, 8.9, 9. , 9.1, 9.2, 9.3, 9.4, 9.5, 9.6, 9.7, 9.8,
9.9])
We can quickly define non-integer ranges of numbers for graph plotting:
In [15]: import math
In [16]: values=np.linspace(0, math.pi, 100) # Start, stop, number of steps
In [17]: values
Out[17]: array([ 0. , 0.03173326, 0.06346652, 0.09519978, 0.12693304,

0.1586663 , 0.19039955, 0.22213281, 0.25386607, 0.28559933,
0.31733259, 0.34906585, 0.38079911, 0.41253237, 0.44426563,
0.47599889, 0.50773215, 0.53946541, 0.57119866, 0.60293192,
0.63466518, 0.66639844, 0.6981317 , 0.72986496, 0.76159822,
0.79333148, 0.82506474, 0.856798 , 0.88853126, 0.92026451,
0.95199777, 0.98373103, 1.01546429, 1.04719755, 1.07893081,
120
1.11066407, 1.14239733, 1.17413059, 1.20586385, 1.23759711,
1.26933037, 1.30106362, 1.33279688, 1.36453014, 1.3962634 ,
1.42799666, 1.45972992, 1.49146318, 1.52319644, 1.5549297 ,
1.58666296, 1.61839622, 1.65012947, 1.68186273, 1.71359599,
1.74532925, 1.77706251, 1.80879577, 1.84052903, 1.87226229,
1.90399555, 1.93572881, 1.96746207, 1.99919533, 2.03092858,
2.06266184, 2.0943951 , 2.12612836, 2.15786162, 2.18959488,
2.22132814, 2.2530614 , 2.28479466, 2.31652792, 2.34826118,
2.37999443, 2.41172769, 2.44346095, 2.47519421, 2.50692747,
2.53866073, 2.57039399, 2.60212725, 2.63386051, 2.66559377,
2.69732703, 2.72906028, 2.76079354, 2.7925268 , 2.82426006,
2.85599332, 2.88772658, 2.91945984, 2.9511931 , 2.98292636,
3.01465962, 3.04639288, 3.07812614, 3.10985939, 3.14159265])
NumPy comes with ‘vectorised’ versions of common functions which work element-by-element when
applied to arrays:

plt.plot(values,np.sin(values))
Out[18]: [<matplotlib.lines.Line2D at 0x10b283c10>]
So we don’t have to use awkward list comprehensions when using these.
21.6 Multi-Dimensional Arrays

NumPy’s true power comes from multi-dimensional arrays:
In [19]: np.zeros([3,4,2])
121
Out[19]: array([[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]]])
Unlike a list-of-lists in Python, we can reshape arrays:
In [20]: x=np.array(range(40))
y=x.reshape([4,5,2])
y
Out[20]: array([[[ 0, 1],

[ 2, 3],
[ 4, 5],
[ 6, 7],
[ 8, 9]],
[[10, 11],
[12, 13],
[14, 15],
[16, 17],
[18, 19]],
[[20, 21],
[22, 23],
[24, 25],
[26, 27],
[28, 29]],
[[30, 31],
[32, 33],
[34, 35],
[36, 37],
[38, 39]]])
And index multiple columns at once:
In [21]: y[3,2,1]
Out[21]: 35
Including selecting on inner axes while taking all from the outermost:
In [22]: y[:,2,1]
Out[22]: array([ 5, 15, 25, 35])
122
And subselecting ranges:
In [23]: y[2:,:1,:]
Out[23]: array([[[20, 21]],
[[30, 31]]])
And transpose arrays:
In [24]: y.transpose()
Out[24]: array([[[ 0, 10, 20, 30],

[ 2, 12, 22, 32],
[ 4, 14, 24, 34],
[ 6, 16, 26, 36],
[ 8, 18, 28, 38]],
[[ 1, 11, 21, 31],

[ 3, 13, 23, 33],
[ 5, 15, 25, 35],
[ 7, 17, 27, 37],
[ 9, 19, 29, 39]]])
You can get the dimensions of an array with shape
In [25]: y.shape
Out[25]: (4, 5, 2)
In [26]: y.transpose().shape
Out[26]: (2, 5, 4)
Some numpy functions apply by default to the whole array, but can be chosen to act only on certain
axes:
In [27]: x=np.arange(12).reshape(4,3)
x
Out[27]: array([[ 0, 1, 2],

[ 3, 4, 5],
[ 6, 7, 8],
[ 9, 10, 11]])
In [28]: x.sum(1) # Sum along the second axis, leaving the first.
Out[28]: array([ 3, 12, 21, 30])
In [29]: x.sum(0) # Sum along the first axis, leaving the first.
Out[29]: array([18, 22, 26])
In [30]: x.sum() # Sum all axes
Out[30]: 66
123
21.7 Array Datatypes
A Python list can contain data of mixed type:
In [31]: x=[’hello’, 2, 3.4]
In [32]: type(x[2])
Out[32]: float
In [33]: type(x[1])
Out[33]: int
A NumPy array always contains just one datatype:
In [34]: np.array(x)
Out[34]: array([’hello’, ’2’, ’3.4’],

dtype=’|S5’)
NumPy will choose the least-generic-possible datatype that can contain the data:
In [35]: y=np.array([2, 3.4])
In [36]: y
Out[36]: array([ 2. , 3.4])
In [37]: type(y[0])
Out[37]: numpy.float64
21.8 Broadcasting
This is another really powerful feature of NumPy
By default, array operations are element-by-element:
In [38]: np.arange(5)*np.arange(5)
Out[38]: array([ 0, 1, 4, 9, 16])
If we multiply arrays with non-matching shapes we get an error:
In [39]: np.arange(5) * np.arange(6)
---------------------------------------------------------------------------
ValueError Traceback (most recent call last)
<ipython-input-39-66b7c967724c> in <module>()
----> 1 np.arange(5) * np.arange(6)
ValueError: operands could not be broadcast together with shapes (5,) (6,)
124
In [40]: np.zeros([2,3])*np.zeros([2,4])
---------------------------------------------------------------------------
<ipython-input-40-4fb354a381e7> in <module>()
----> 1 np.zeros([2,3])*np.zeros([2,4])
ValueError: operands could not be broadcast together with shapes (2,3) (2,4)
In [41]: m1 = np.arange(100).reshape([10, 10])
In [42]: m2 = np.arange(100).reshape([10, 5, 2])
In [43]: m1+m2
---------------------------------------------------------------------------
<ipython-input-43-e9085a7f6251> in <module>()
----> 1 m1+m2
ValueError: operands could not be broadcast together with shapes (10,10) (10,5,2)
Arrays must match in all dimensions in order to be compatible:
In [44]: np.ones([3,3])*np.ones([3,3]) # Note elementwise multiply, *not* matrix multiply.
Out[44]: array([[ 1., 1., 1.],

[ 1., 1., 1.],
[ 1., 1., 1.]])
Except, that if one array has any Dimension 1, then the data is REPEATED to match the other.
In [45]: m1=np.arange(10).reshape([10,1])
m1
Out[45]: array([[0],
[1],
[2],
[3],
[4],
[5],
[6],
[7],
[8],
[9]])
In [46]: m2=m1.transpose()
m2
125
Out[46]: array([[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]])
In [47]: m1.shape # "Column Vector"
Out[47]: (10, 1)
In [48]: m2.shape # "Row Vector"
Out[48]: (1, 10)
In [49]: m1+m2
Out[49]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],

[ 1, 2, 3, 4, 5, 6, 7, 8, 9, 10],
[ 2, 3, 4, 5, 6, 7, 8, 9, 10, 11],
[ 3, 4, 5, 6, 7, 8, 9, 10, 11, 12],
[ 4, 5, 6, 7, 8, 9, 10, 11, 12, 13],
[ 5, 6, 7, 8, 9, 10, 11, 12, 13, 14],
[ 6, 7, 8, 9, 10, 11, 12, 13, 14, 15],
[ 7, 8, 9, 10, 11, 12, 13, 14, 15, 16],
[ 8, 9, 10, 11, 12, 13, 14, 15, 16, 17],
[ 9, 10, 11, 12, 13, 14, 15, 16, 17, 18]])
In [50]: 10*m1+m2
Out[50]: array([[ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9],

[10, 11, 12, 13, 14, 15, 16, 17, 18, 19],
[20, 21, 22, 23, 24, 25, 26, 27, 28, 29],
[30, 31, 32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47, 48, 49],
[50, 51, 52, 53, 54, 55, 56, 57, 58, 59],
[60, 61, 62, 63, 64, 65, 66, 67, 68, 69],
[70, 71, 72, 73, 74, 75, 76, 77, 78, 79],
[80, 81, 82, 83, 84, 85, 86, 87, 88, 89],
[90, 91, 92, 93, 94, 95, 96, 97, 98, 99]])
This works for arrays with more than one unit dimension.
21.9 Newaxis
Broadcasting is very powerful, and numpy allows indexing with np.newaxis to temporarily create new
one-long dimensions on the fly.
In [51]: x=np.arange(10).reshape(2,5)
y=np.arange(8).reshape(2,2,2)
In [52]: res=x[:,:,np.newaxis,np.newaxis]*y[:,np.newaxis,:,:]
In [53]: res.shape
Out[53]: (2, 5, 2, 2)
In [54]: np.sum(res)
Out[54]: 830
Note that newaxis works because a 3 × 1 × 3 array and a 3 × 3 array contain the same data, differently
shaped:
126
In [55]: threebythree=np.arange(9).reshape(3,3)
threebythree
Out[55]: array([[0, 1, 2],

[3, 4, 5],
[6, 7, 8]])
In [56]: threebythree[:,np.newaxis,:]
Out[56]: array([[[0, 1, 2]],
[[3, 4, 5]],
[[6, 7, 8]]])
21.10 Dot Products

NumPy multiply is element-by-element, not a dot-product:
In [57]: np.arange(9).reshape(3,3) * np.arange(3,12).reshape(3,3)
Out[57]: array([[ 0, 4, 10],

[18, 28, 40],
[54, 70, 88]])
To get a dot-product, do this:
In [58]: np.dot(np.arange(9).reshape(3,3), np.arange(3,12).reshape(3,3))
Out[58]: array([[ 24, 27, 30],

[ 78, 90, 102],
[132, 153, 174]])
21.11 Array DTypes

Arrays have a “dtype” which specifies their datatype:
In [59]: x=[2, 3.4, 7.2, 0]
In [60]: np.array(x)
Out[60]: array([ 2. , 3.4, 7.2, 0. ])
In [61]: np.array(x).dtype
Out[61]: dtype(’float64’)
These are, when you get to know them, fairly obvious string codes for datatypes: NumPy supports all
kinds of datatypes beyond the python basics.
NumPy will convert python type names to dtypes:
In [62]: int_array= np.array(x, dtype=int)
In [63]: float_array=np.array(x, dtype=float)
In [64]: int_array
127
Out[64]: array([2, 3, 7, 0])
In [65]: float_array
Out[65]: array([ 2. , 3.4, 7.2, 0. ])
In [66]: int_array.dtype
Out[66]: dtype(’int64’)
In [67]: float_array.dtype
Out[67]: dtype(’float64’)
21.12 Record Arrays

These are a special array structure designed to match the CSV “Record and Field” model. It’s a very
different structure from the normal numPy array, and different fields can contain different datatypes. We
saw this when we looked at CSV files:
In [68]: x=np.arange(50).reshape([10,5])
In [69]: record_x=x.view(dtype={’names’: ["col1", "col2", "another", "more", "last"],

’formats’: [int]*5 } )
In [70]: record_x
Out[70]: array([[(0, 1, 2, 3, 4)],

[(5, 6, 7, 8, 9)],
[(10, 11, 12, 13, 14)],
[(15, 16, 17, 18, 19)],
[(20, 21, 22, 23, 24)],
[(25, 26, 27, 28, 29)],
[(30, 31, 32, 33, 34)],
[(35, 36, 37, 38, 39)],
[(40, 41, 42, 43, 44)],
[(45, 46, 47, 48, 49)]],
dtype=[(’col1’, ’<i8’), (’col2’, ’<i8’), (’another’, ’<i8’), (’more’, ’<i8’), (’last’, ’<
Record arrays can be addressed with field names like they were a dictionary:
In [71]: record_x[’col1’]
Out[71]: array([[ 0],

[ 5],
[10],
[15],
[20],
[25],
[30],
[35],
[40],
[45]])
We’ve seen these already when we used NumPy’s CSV parser.
128
21.13 Logical arrays, masking, and selection
Numpy defines operators like == and < to apply to arrays element by element
In [72]: x=np.zeros([3,4])
x
Out[72]: array([[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.],
[ 0., 0., 0., 0.]])
In [73]: y=np.arange(-1,2)[:,np.newaxis]*np.arange(-2,2)[np.newaxis,:]
y
Out[73]: array([[ 2, 1, 0, -1],
[ 0, 0, 0, 0],
[-2, -1, 0, 1]])
In [74]: iszero = x==y
iszero
Out[74]: array([[False, False, True, False],
[ True, True, True, True],
[False, False, True, False]], dtype=bool)
A logical array can be used to select elements from an array:
In [75]: y[np.logical_not(iszero)]
Out[75]: array([ 2, 1, -1, -2, -1, 1])
Although when printed, this comes out as a flat list, if assigned to, the selected elements of the array are
changed!
In [76]: y[iszero]=5
In [77]: y
Out[77]: array([[ 2, 1, 5, -1],
[ 5, 5, 5, 5],
[-2, -1, 5, 1]])
21.14 Numpy memory

Numpy memory management can be tricksy:
In [78]: x=np.arange(5)
y=x[:]
In [79]: y[2]=0
x
Out[79]: array([0, 1, 0, 3, 4])
x=x+2
x
Out[80]: array([2, 3, 4, 5, 6])
We must use np.copy to force separate memory. Otherwise NumPy tries it’s hardest to make slices be
views on data.
Now, this has all been very theoretical, but let’s go through a practical example, and see how powerful
NumPy can be.
129
Chapter 22
The Boids!
22.1 Flocking
The aggregate motion of a flock of birds, a herd of land animals, or a school of fish is a beautiful
and familiar part of the natural world. . . The aggregate motion of the simulated flock is created
by a distributed behavioral model much like that at work in a natural flock; the birds choose
their own course. Each simulated bird is implemented as an independent actor that navigates
according to its local perception of the dynamic environment, the laws of simulated physics that
rule its motion, and a set of behaviors programmed into it. . . The aggregate motion of the
simulated flock is the result of the dense interaction of the relatively simple behaviors of the
individual simulated birds.
– Craig W. Reynolds, “Flocks, Herds, and Schools: A Distributed Behavioral Model”, Computer Graphics
21 4 1987, pp 25-34 See the original paper
• Collision Avoidance: avoid collisions with nearby flockmates

• Velocity Matching: attempt to match velocity with nearby flockmates
• Flock Centering: attempt to stay close to nearby flockmates
22.2 Setting up the Boids

Our boids will each have an x velocity and a y velocity, and an x position and a y position.
We’ll build this up in NumPy notation, and eventually, have an animated simulation of our flying boids.
Let’s start with simple flying in a straight line.

Our locations, for each of our N boids, will be an array, shape 2 × N , with the x positions in the first
row, and y positions in the second row.
In [2]: boid_count = 10
We’ll want to be able to seed our Boids in a random position.

We’d better define the edges of our simulation area:
In [3]: limits = np.array([2000,4000])
In [4]: positions = np.random.rand(2,boid_count)*limits[:,np.newaxis]

positions
130
Out[4]: array([[ 710.35809712, 510.16788062, 1486.85845396, 430.77593244,
1185.14988108, 122.60227361, 484.48948249, 52.73997141,
1635.55381156, 1756.18999319],
[ 1909.51584644, 2763.25337086, 1750.37654827, 408.25217478,
3994.47310684, 2390.07042472, 2255.85236995, 2295.40982104,
593.19707991, 159.85521759]])
We used broadcasting with np.newaxis to apply our upper limit to each boid. rand gives us a random
number between 0 and 1. We multiply by our limits to get a number up to that limit.
Let’s put that in a function:
In [5]: def new_flock(count, lower_limits, upper_limits):

width=upper_limits-lower_limits
return (lower_limits[:,np.newaxis] +
np.random.rand(2, count)*width[:,np.newaxis])
But each bird will also need a starting velocity. Let’s make these random too:
In [6]: velocities = new_flock(boid_count, np.array([0, -20]), np.array([10, 20]))

velocities
Out[6]: array([[ 9.75795443, 7.9151025 , 5.93583147, 0.50822314,

4.24600678, 3.21029596, 2.1284636 , 3.86150423,
9.54088974, 1.37615943],
[-14.35986806, 3.39291928, -12.06249557, 8.88649841,
11.37916218, 6.63897802, -12.32173067, 13.71757242,
-13.91071075, -6.57162598]])
22.3 Flying in a Straight Line

Now we see the real amazingness of NumPy: if we want to move our whole flock according to
δx = δt · dv
dt
We just do:
In [7]: delta_t = 1 # Arbitrary choice; defines unit of velocity

positions += velocities * delta_t
22.4 Matplotlib Animations

So now we can animate our Boids using the matplotlib animation tools. All we have to do is import the
relevant libraries:
In [8]: from matplotlib import animation

%matplotlib inline
Then, we make a static plot, showing our first frame:
In [9]: # create a simple animation

positions=new_flock(100, np.array([100,900]), np.array([200,1100]))
velocities=new_flock(100, np.array([0,-20]), np.array([10,20]))
figure = plt.figure()
axes = plt.axes(xlim=(0, limits[0]), ylim=(0, limits[0]))
scatter=axes.scatter(positions[0,:],positions[1,:])
scatter
131
Out[9]: <matplotlib.collections.PathCollection at 0x10ef25950>
/usr/local/lib/python2.7/site-packages/matplotlib/collections.py:571: FutureWarning: elementwise compari
if self. edgecolors == str(’face’):
Then, we define a function which updates the figure for each timestep
In [10]: def update_boids(positions, velocities):
positions += velocities
def animate(frame):
update_boids(positions, velocities)
scatter.set_offsets(positions.transpose())
Call FuncAnimation, and specify how many frames we want:
In [11]: anim=animation.FuncAnimation(figure, animate,
frames=50, interval=50, blit=True)
Save out the figure:
In [12]: anim.save(’boids_1.mp4’)
And download the saved animation
You can even use an external library to view the results directly in the notebook. If you’re on your own
computer, you can download it from https://gist.github.com/gforsyth/188c32b6efe834337d8a (See the notes
on installing libraries. . . )
Unfortunately, if you’re on the teaching cluster, you won’t be able to install it there.
In [13]: from JSAnimation import IPython_display
# Inline animation tool; needs manual install via
# If you don’t have this, you need to save animations as MP4.
anim
132
Out[13]: <matplotlib.animation.FuncAnimation at 0x10edeef50>
22.5 Fly towards the middle

Boids try to fly towards the middle:
In [14]: positions=new_flock(4, np.array([100,900]), np.array([200,1100]))

In [15]: positions
Out[15]: array([[ 133.98680655, 133.46902852, 125.00332456, 141.68189396],

[ 905.01872567, 973.65259734, 1046.94782499, 972.9723188 ]])
In [16]: velocities
Out[16]: array([[ 7.72812511, 2.50819728, 5.43031641, 0.60383818],

[ -5.19483905, -7.32449612, 2.83633313, 11.03471036]])
In [17]: middle=np.mean(positions, 1)
middle
Out[17]: array([ 133.5352634, 974.6478667])
In [18]: direction_to_middle = positions-middle[:, np.newaxis]
In [19]: move_to_middle_strength = 0.01

velocities = velocities - direction_to_middle * move_to_middle_strength
Let’s update our function, and animate that:

move_to_middle_strength = 0.1
middle=np.mean(positions, 1)
direction_to_middle = positions - middle[:, np.newaxis]
velocities -= direction_to_middle * move_to_middle_strength
In [21]: def animate(frame):

In [22]: anim=animation.FuncAnimation(figure, animate,

In [23]: from JSAnimation import IPython_display

anim
Out[23]: <matplotlib.animation.FuncAnimation at 0x10ef3d050>
133
22.6 Avoiding collisions
We’ll want to add our other flocking rules to the behaviour of the Boids.
We’ll need a matrix giving the distances between each bird. This should be N × N .
In [24]: positions=new_flock(4, np.array([100,900]), np.array([200,1100]))

We might think that we need to do the X-distances and Y-distances separately:
In [25]: xpos=positions[0,:]
In [26]: xsep_matrix = xpos[:,np.newaxis] - xpos[np.newaxis,:]
In [27]: xsep_matrix.shape
Out[27]: (4, 4)
In [28]: xsep_matrix
Out[28]: array([[ 0. , -24.0035976 , -27.30156215, -35.17668948],

[ 24.0035976 , 0. , -3.29796454, -11.17309188],
[ 27.30156215, 3.29796454, 0. , -7.87512734],
[ 35.17668948, 11.17309188, 7.87512734, 0. ]])
But in NumPy we can be cleverer than that, and make a 2 by N by N matrix of separations:
In [29]: separations = positions[:,np.newaxis,:] - positions[:,:,np.newaxis]
In [30]: separations.shape
Out[30]: (2, 4, 4)
And then we can get the sum-of-squares δx2 + δy2 like this:
In [31]: squared_displacements = separations * separations
In [32]: square_distances = np.sum(squared_displacements, 0)
In [33]: square_distances
Out[33]: array([[ 0. , 2731.17718748, 3702.52206907, 2840.63621187],

[ 2731.17718748, 0. , 74.20041557, 165.56263795],
[ 3702.52206907, 74.20041557, 0. , 267.63072804],
[ 2840.63621187, 165.56263795, 267.63072804, 0. ]])
Now we need to find birds that are too close:
In [34]: alert_distance = 2000

close_birds = square_distances < alert_distance
close_birds
Out[34]: array([[ True, False, False, False],

[False, True, True, True],
[False, True, True, True],
[False, True, True, True]], dtype=bool)
Find the direction distances only to those birds which are too close:
134
In [35]: separations_if_close = np.copy(separations)
far_away = np.logical_not(close_birds)
In [36]: separations_if_close[0,:,:][far_away] = 0
separations_if_close[1,:,:][far_away] = 0
separations_if_close
Out[36]: array([[[ 0. , 0. , 0. , 0. ],
[ 0. , 0. , 3.29796454, 11.17309188],
[ 0. , -3.29796454, 0. , 7.87512734],
[ 0. , -11.17309188, -7.87512734, 0. ]],
[[ 0. , 0. , 0. , 0. ],
[ 0. , 0. , -7.95762813, 6.38158724],
[ 0. , 7.95762813, 0. , 14.33921537],
[ 0. , -6.38158724, -14.33921537, 0. ]]])
And fly away from them:
In [37]: velocities = velocities + np.sum(separations_if_close,2)
Now we can update our animation:

middle = np.mean(positions, 1)
direction_to_middle = positions - middle[:,np.newaxis]
velocities -= direction_to_middle * move_to_middle_strength
separations = positions[:,np.newaxis,:] - positions[:,:,np.newaxis]

squared_displacements = separations * separations
square_distances = np.sum(squared_displacements, 0)
alert_distance = 100
far_away = square_distances > alert_distance
separations_if_close = np.copy(separations)
separations_if_close[0,:,:][far_away] =0
velocities += np.sum(separations_if_close,1)

anim = animation.FuncAnimation(figure, animate,

from JSAnimation import IPython_display

positions = new_flock(100, np.array([100,900]), np.array([200,1100]))
velocities = new_flock(100, np.array([0,-20]), np.array([10,20]))
anim
Out[39]: <matplotlib.animation.FuncAnimation at 0x10ef9ff10>
135
22.7 Match speed with nearby birds
This is pretty similar:

middle = np.mean(positions, 1)
direction_to_middle = positions-middle[:,np.newaxis]
velocities -= direction_to_middle*move_to_middle_strength
separations = positions[:,np.newaxis,:] - positions[:,:,np.newaxis]

squared_displacements = separations*separations
square_distances = np.sum(squared_displacements, 0)
alert_distance = 100
far_away=square_distances > alert_distance
separations_if_close = np.copy(separations)
velocities += np.sum(separations_if_close,1)
velocity_differences = velocities[:,np.newaxis,:] - velocities[:,:,np.newaxis]

formation_flying_distance = 10000
formation_flying_strength = 0.125
very_far=square_distances > formation_flying_distance
velocity_differences_if_close = np.copy(velocity_differences)
velocity_differences_if_close[0,:,:][very_far] =0
velocity_differences_if_close[1,:,:][very_far] =0
velocities -= np.mean(velocity_differences_if_close, 1) * formation_flying_strength

anim=animation.FuncAnimation(figure, animate,

anim
Out[41]: <matplotlib.animation.FuncAnimation at 0x10eece4d0>
Hopefully the power of NumPy should be pretty clear now. This would be enormously slower and, I
think, harder to understand using traditional lists.
136
Chapter 23
Understanding the “Greengraph”

Example
We now know enough to understand everything we did in the initial example chapter on the “Greengraph”.
Go back to that part of the notes, and re-read the code.
Now, we can even write it up into a class, and save it as a module.
23.1 Classes for Greengraph

In [1]: %%bash
mkdir -p greengraph # Create the folder for the module (on mac or linux)
In [2]: %%writefile greengraph/graph.py

import numpy as np
import geopy
from map import Map
class Greengraph(object):
def __init__(self, start, end):
self.start=start
self.end=end
self.geocoder=geopy.geocoders.GoogleV3(domain="maps.google.co.uk")
def geolocate(self, place):

return self.geocoder.geocode(place, exactly_one=False)[0][1]
def location_sequence(self, start,end,steps):

lats = np.linspace(start[0], end[0], steps)
longs = np.linspace(start[1],end[1], steps)
return np.vstack([lats, longs]).transpose()
def green_between(self, steps):

return [Map(*location).count_green()
for location in self.location_sequence(
self.geolocate(self.start),
self.geolocate(self.end),
steps)]
Writing greengraph/graph.py
137
In [3]: %%writefile greengraph/map.py
import numpy as np
from matplotlib import image as img
import requests
class Map(object):
def __init__(self, lat, long, satellite=True, zoom=10, size=(400,400), sensor=False):
params=dict(
zoom= zoom,
size= "x".join(map(str, size)),
center= ",".join(map(str, (lat, long) )),
style="feature:all|element:labels|visibility:off"
)
if satellite:
self.image = requests.get(base, params=params).content # Fetch our PNG image data

self.pixels= img.imread(StringIO(self.image)) # Parse our PNG image as a numpy array
def green(self, threshold):

# Use NumPy to build an element-by-element logical array
greener_than_red = self.pixels[:,:,1] > threshold* self.pixels[:,:,0]
greener_than_blue = self.pixels[:,:,1] > threshold*self.pixels[:,:,2]
green = np.logical_and(greener_than_red, greener_than_blue)
return green
def count_green(self, threshold = 1.1):

return np.sum(self.green(threshold))
def show_green(data, threshold = 1.1):

green = self.green(threshold)
out = green[:,:,np.newaxis]*array([0,1,0])[np.newaxis,np.newaxis,:]
buffer = StringIO()
result = img.imsave(buffer, out, format=’png’)
return buffer.getvalue()
Writing greengraph/map.py
In [4]: %%writefile greengraph/__init__.py

from graph import Greengraph
Writing greengraph/ init .py
23.2 Invoking our code and making a plot

from greengraph import Greengraph
%matplotlib inline
138
mygraph=Greengraph(’New York’,’Chicago’)
data = mygraph.green_between(20)
In [6]: plt.plot(data)
Out[6]: [<matplotlib.lines.Line2D at 0x104caa6d0>]
23.3 Introduction
23.3.1 What’s version control?
Version control is a tool for managing changes to a set of files.
There are many different version control systems:
• Git
• Mercurial (hg)
• CVS
• Subversion (svn)
• ...
23.3.2 Why use version control?

• Better kind of backup.
• Review history (“When did I introduce this bug?”).
• Restore older code versions.
• Ability to undo mistakes.
• Maintain several versions of the code at a time.
Git is also a collaborative tool:
139
• “How can I share my code?”
• “How can I submit a change to someone else’s code?”
• “How can I merge my work with Sue’s?”
23.3.3 Git != GitHub

• Git: version control system tool to manage source code history.
• GitHub: hosting service for Git repositories.
23.3.4 How do we use version control?

Do some programming, then commit our work:
my vcs commit
Program some more.
Spot a mistake:
my vcs rollback
Mistake is undone.
23.3.5 What is version control? (Team version)
Sue James
my vcs commit ...
... Join the team
... my vcs checkout
... Do some programming
... my vcs commit
my vcs update ...
Do some programming Do some programming
my vcs commit ...
my vcs update ...
my vcs merge ...
my vcs commit ...
23.3.6 Scope
This course will use the git version control system, but much of what you learn will be valid with other
version control tools you may encounter, including subversion (svn) and mercurial (hg).
23.4 Practising with Git

23.4.1 Example Exercise
In this course, we will use, as an example, the development of a few text files containing a description of a
topic of your choice.
This could be your research, a hobby, or something else. In the end, we will show you how to display the
content of these files as a very simple website.
23.4.2 Programming and documents

The purpose of this exercise is to learn how to use Git to manage program code you write, not simple text
website content, but we’ll just use these text files instead of code for now, so as not to confuse matters with
trying to learn version control while thinking about programming too.
140
In later parts of the course, you will use the version control tools you learn today with actual Python
code.
23.4.3 Markdown
The text files we create will use a simple “wiki” markup style called markdown to show formatting. This is
the convention used in this file, too.
You can view the content of this file in the way Markdown renders it by looking on the web, and compare
the raw text.
23.4.4 Displaying Text in this Tutorial

This tutorial is based on use of the Git command line. So you’ll be typing commands in the shell.
To make it easy for me to edit, I’ve built it using IPython notebook.
Commands you can type will look like this, using the %%bash “magic” for the notebook.
In [1]: %%bash
echo some output
some output
with the results you should see below.

In this document, we will show the new content of an edited document like this:
In [2]: %%writefile somefile.md

Some content here
Writing somefile.md
But if you are following along, you should edit the file using a text editor. On windows, we recommend
Notepad++. On mac, we recommend Atom
23.4.5 Setting up somewhere to work

In [3]: %%bash
rm -rf learning_git/git_example # Just in case it’s left over from a previous class; you won’t n
mkdir -p learning_git/git_example
cd learning_git/git_example
I just need to move this IPython notebook’s current directory as well:
In [4]: import os
top_dir = os.getcwd()
top_dir
Out[4]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git’
In [5]: git_dir = os.path.join(top_dir, ’learning_git’)

git_dir
Out[5]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git’
In [6]: working_dir=os.path.join(git_dir, ’git_example’)
In [7]: os.chdir(working_dir)
141
23.5 Solo work
23.5.1 Configuring Git with your name and email
First, we should configure Git to know our name and email address:
In [8]: %%bash
git config --global user.name "James Hetherington"
git config --global user.email "jamespjh@gmail.com"
23.5.2 Initialising the repository

Now, we will tell Git to track the content of this folder as a git “repository”.
In [9]: %%bash
pwd # Note where we are standing-- MAKE SURE YOU INITIALISE THE RIGHT FOLDER
git init
/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git/git example
Initialized empty Git repository in /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/c
As yet, this repository contains no files:
In [10]: %%bash
ls
In [11]: %%bash
git status
On branch master
Initial commit
nothing to commit (create/copy files and use "git add" to track)
23.6 Solo work with Git

So, we’re in our git working directory:
In [1]: import os
git_dir = os.path.join(top_dir, ’learning_git’)
working_dir=os.path.join(git_dir, ’git_example’)
os.chdir(working_dir)
working_dir
Out[1]: ’/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git/git exa
23.6.1 A first example file

So let’s create an example file, and see how to start to manage a history of changes to it.
<my editor> index.md # Type some content into the file.
In [2]: %%writefile index.md
Mountains in the UK
===================
England is not very mountainous.
But has some tall hills, and maybe a mountain or two depending on your definition.
142
Writing index.md
In [3]: cat index.md
Mountains in the UK
===================
23.6.2 Telling Git about the File

So, let’s tell Git that index.md is a file which is important, and we would like to keep track of its history:
In [4]: %%bash
git add index.md
Don’t forget: Any files in repositories which you want to “track” need to be added with git add after
you create them.
23.6.3 Our first commit

Now, we need to tell Git to record the first version of this file in the history of changes:
In [5]: %%bash
git commit -m "First commit of discourse on UK topography"
[master (root-commit) 1561abf] First commit of discourse on UK topography

1 file changed, 4 insertions(+)
create mode 100644 index.md
And note the confirmation from Git.

There’s a lot of output there you can ignore for now.
23.6.4 Configuring Git with your editor

If you don’t type in the log message directly with -m “Some message”, then an editor will pop up, to allow
you to edit your message on the fly.
For this to work, you have to tell git where to find your editor.
In [6]: %%bash
git config --global core.editor vim
You can find out what you currently have with:
In [7]: %%bash
git config --get core.editor
vim
To configure Notepad++ on windows you’ll need something like the below, ask a demonstrator to help
for your machine.
git config --global core.editor "’C:/Program Files (x86)/Notepad++

/notepad++.exe’ -multiInst -nosession -noPlugin"
I’m going to be using vim as my editor, but you can use whatever editor you prefer. (Windows users
could use “Notepad++”, Mac users could use “textmate” or “sublime text”, linux users could use vim, nano
or emacs.)
143
23.6.5 Git log
Git now has one change in its history:
In [8]: %%bash
git log
commit 1561abf7777027fdf3549e56f4f3f1a85e652309
Author: James Hetherington <jamespjh@gmail.com>
Date: Wed Nov 4 17:55:46 2015 +0000
First commit of discourse on UK topography
You can see the commit message, author, and date. . .
23.6.6 Hash Codes

The commit “hash code”, e.g.
c438f1716b2515563e03e82231acbae7dd4f4656
is a unique identifier of that particular revision.
(This is a really long code, but whenever you need to use it, you can just use the first few characters,
however many characters is long enough to make it unique, c438 for example. )
23.6.7 Nothing to see here

Note that git will now tell us that our “working directory” is up-to-date with the repository: there are no
changes to the files that aren’t recorded in the repository history:
In [9]: %%bash
git status
On branch master
nothing to commit, working directory clean
Let’s edit the file again:
vim index.md

Mountains in the UK
===================
Mount Fictional, in Barsetshire, U.K. is the tallest mountain in the world.
Overwriting index.md
Mountains in the UK
===================
144
23.6.8 Unstaged changes
In [12]: %%bash
git status
On branch master
Changes not staged for commit:
(use "git add <file>..." to update what will be committed)
(use "git checkout -- <file>..." to discard changes in working directory)
modified: index.md
no changes added to commit (use "git add" and/or "git commit -a")
We can now see that there is a change to “index.md” which is currently “not staged for commit”. What
does this mean?
If we do a git commit now nothing will happen.
Git will only commit changes to files that you choose to include in each commit.
This is a difference from other version control systems, where committing will affect all changed files.
We can see the differences in the file with:
In [13]: %%bash
git diff
diff --git a/index.md b/index.md
index 4f737f1..263ec81 100644
--- a/index.md
+++ b/index.md
@@ -1,4 +1,6 @@
Mountains in the UK
===================
-But has some tall hills, and maybe a mountain or two depending on your definition.
\ No newline at end of file
+But has some tall hills, and maybe a mountain or two depending on your definition.
+
+Mount Fictional, in Barsetshire, U.K. is the tallest mountain in the world.
Deleted lines are prefixed with a minus, added lines prefixed with a plus.
23.6.9 Staging a file to be included in the next commit

To include the file in the next commit, we have a few choices. This is one of the things to be careful of with
git: there are lots of ways to do similar things, and it can be hard to keep track of them all.
In [14]: %%bash
git add --update
This says “include in the next commit, all files which have ever been included before”.
Note that git add is the command we use to introduce git to a new file, but also the command we use
to “stage” a file to be included in the next commit.
23.6.10 The staging area

The “staging area” or “index” is the git jargon for the place which contains the list of changes which will be
included in the next commit.
You can include specific changes to specific files with git add, commit them, add some more files, and
commit them. (You can even add specific changes within a file to be included in the index.)
145
23.6.11 Message Sequence Charts
In order to illustrate the behaviour of Git, it will be useful to be able to generate figures in Python of a
“message sequence chart” flavour.
There’s a nice online tool to do this, called “Message Sequence Charts”.
Have a look at https://www.websequencediagrams.com
Instead of just showing you these diagrams, I’m showing you in this notebook how I make them. This is
part of our “reproducible computing” approach; always generating all our figures from code.
Here’s some quick code in the Notebook to download and display an MSC illustration, using the Web
Sequence Diagrams API:
In [15]: %%writefile wsd.py
import requests
import re
import IPython
def wsd(code):
response = requests.post("http://www.websequencediagrams.com/index.php", data={
’message’: code,
’apiVersion’: 1,
})
expr = re.compile("(\?(img|pdf|png|svg)=[a-zA-Z0-9]+)")
m = expr.search(response.text)
if m == None:
print "Invalid response from server."
return False
image=requests.get("http://www.websequencediagrams.com/" + m.group(0))
return IPython.core.display.Image(image.content)
Writing wsd.py
In [16]: from wsd import wsd
%matplotlib inline
wsd("Sender->Recipient: Hello\n Recipient->Sender: Message received OK")
Out[16]:
146
23.6.12 The Levels of Git
Let’s make ourselves a sequence chart to show the different aspects of Git we’ve seen so far:
In [17]: message="""
Working Directory -> Staging Area : git add
Staging Area -> Local Repository : git commit
Working Directory -> Local Repository : git commit -a
"""
wsd(message)
Out[17]:
23.6.13 Review of status

In [18]: %%bash
git status
On branch master
Changes to be committed:
(use "git reset HEAD <file>..." to unstage)
modified: index.md
Untracked files:
(use "git add <file>..." to include in what will be committed)
wsd.py
wsd.pyc
In [19]: %%bash
git commit -m "Add a lie about a mountain"
[master 47f2a5c] Add a lie about a mountain

1 file changed, 3 insertions(+), 1 deletion(-)
In [20]: %%bash
git log
147
commit 47f2a5cb7119582d84116d3646286545f93ad967
Date: Wed Nov 4 17:55:48 2015 +0000
Add a lie about a mountain
Date: Wed Nov 4 17:55:46 2015 +0000
Great, we now have a file which contains a mistake.
23.6.14 Carry on regardless

In a while, we’ll use Git to roll back to the last correct version: this is one of the main reasons we wanted to
use version control, after all! But for now, let’s do just as we would if we were writing code, not notice our
mistake and keep working. . .
vim index.md

Mountains and Hills in the UK
===================

===================
23.6.15 Commit with a built-in-add

In [23]: %%bash
git commit -am "Change title"
[master 201110e] Change title

1 file changed, 1 insertion(+), 1 deletion(-)
This last command, git commit -a automatically adds changes to all tracked files to the staging area,
as part of the commit command. So, if you never want to just add changes to some tracked files but not
others, you can just use this and forget about the staging area!
148
23.6.16 Review of changes
In [24]: %%bash
git log | head
commit 201110ec7a142c3590a59700c62bcbdf8574dac1
Date: Wed Nov 4 17:55:49 2015 +0000
Change title
Date: Wed Nov 4 17:55:48 2015 +0000
We now have three changes in the history:
In [25]: %%bash
git log --oneline
201110e Change title

47f2a5c Add a lie about a mountain
1561abf First commit of discourse on UK topography
23.6.17 Git Solo Workflow

We can make a diagram that summarises the above story:
participant "Jim’s repo" as R
participant "Jim’s index" as I
participant Jim as J
note right of J: vim index.md
note right of J: git init

J->I: create
J>R: create
note right of J: git add index.md
J->I: Add content of index.md
note right of J: git commit

J->R: Commit content of index.md
note right of J: git add --update

note right of J: git commit -m "Add a lie"
I->R: Commit change to index.md

note right of J: git commit -am "Change title"
149
J->R: Commit change to index.md
"""
wsd(message)
Out[26]:
150
23.7 Fixing mistakes
We’re still in our git working directory:
In [1]: import os
working_dir
23.7.1 Referring to changes with HEAD and ˆ

The commit we want to revert to is the one before the latest.
HEAD^ refers to the commit before the “head”, which is the latest change. That is, we want to go back
to the change before the current one.
We could have used the hash code to reference this, but you can also refer to the commit before the HEAD
as HEAD^, the one before that as HEAD^^, the one before that as HEAD~3. We could also refer to it with the
hash code, e.g. 73fbeaf
23.7.2 Reverting
Ok, so now we’d like to undo the nasty commit with the lie about Mount Fictional.
In [2]: %%bash
git revert HEAD^
[master b7b8924] Revert "Add a lie about a mountain"

1 file changed, 1 insertion(+), 3 deletions(-)
A commit may pop up, with some default text which you can accept and save.
23.7.3 Conflicted reverts

You may, depending on the changes you’ve tried to make, get an error message here.
If this happens, it is because git could not automagically decide how to combine the change you made
after the change you want to revert, with the attempt to revert the change: this could happen, for example,
if they both touch the same line.
If that happens, you need to manually edit the file to fix the problem. Skip ahead to the section on
resolving conflicts, or ask a demonstrator to help.
23.7.4 Review of changes

The file should now contain the change to the title, but not the extra line with the lie. Note the log:
In [3]: %%bash
git log
commit b7b89246d538956237b8c3308bd6502e2c1ef6ee
Date: Wed Nov 4 17:55:51 2015 +0000
Revert "Add a lie about a mountain"
151
This reverts commit 47f2a5cb7119582d84116d3646286545f93ad967.
commit 201110ec7a142c3590a59700c62bcbdf8574dac1
Date: Wed Nov 4 17:55:49 2015 +0000
Change title
Date: Wed Nov 4 17:55:48 2015 +0000
Add a lie about a mountain
Date: Wed Nov 4 17:55:46 2015 +0000
23.7.5 Antipatch
Notice how the mistake has stayed in the history.
There is a new commit which undoes the change: this is colloquially called an “antipatch”. This is nice:
you have a record of the full story, including the mistake and its correction.
23.7.6 Rewriting history

It is possible, in git, to remove the most recent change altogether, “rewriting history”. Let’s make another
bad change, and see how to do this.
23.7.7 A new lie

===================
Engerland is not very mountainous.
In [5]: %%bash
cat index.md

===================
In [6]: %%bash
git diff
diff --git a/index.md b/index.md

index bb12a24..304eea2 100644
--- a/index.md
+++ b/index.md
152
@@ -1,4 +1,4 @@
===================
-England is not very mountainous.
+Engerland is not very mountainous.
In [7]: %%bash
git commit -am "Add a silly spelling"
[master 94dac44] Add a silly spelling

1 file changed, 1 insertion(+), 1 deletion(-)
In [8]: %%bash
git log | head
commit 94dac4477e6486160a628edf488eaa9eefee24fd
Date: Wed Nov 4 17:55:51 2015 +0000
Add a silly spelling
commit b7b89246d538956237b8c3308bd6502e2c1ef6ee
Date: Wed Nov 4 17:55:51 2015 +0000
23.7.8 Using reset to rewrite history

In [9]: %%bash
git reset HEAD^
Unstaged changes after reset:

M index.md
In [10]: %%bash
git log --oneline
b7b8924 Revert "Add a lie about a mountain"

23.7.9 Covering your tracks

The silly spelling is gone, and it isn’t even in the log. This approach to fixing mistakes, “rewriting history”
with reset, instead of adding an antipatch with revert is dangerous, and we don’t recommend it. But you
may want to do it for small silly mistakes, such as to correct a commit message.
23.7.10 Resetting the working area

When git reset removes commits, it leaves your working directory unchanged – so you can keep the work in
the bad change if you want.
In [11]: %%bash
cat index.md
153
===================
If you want to lose the change from the working directory as well, you can do git reset --hard.
I’m going to get rid of the silly spelling, and I didn’t do --hard, so I’ll reset the file from the working
directory to be the same as in the index:
In [12]: %%bash
git checkout index.md
In [13]: %%bash
cat index.md

===================
We can add this to our diagram:
Staging Area -> Working Directory : git checkout
Local Repository -> Staging Area : git reset
Local Repository -> Working Directory: git reset --hard
"""
from wsd import wsd
%matplotlib inline
wsd(message)
Out[14]:
154
We can add it to Jim’s story:
note right of J: git revert HEAD^
J->R: Add new commit reversing change

R->I: update staging area to reverted version
I->J: update file to reverted version

note right of J: git commit -am "Add another mistake"
J->I: Add mistake
I->R: Add mistake
note right of J: git reset HEAD^
J->R: Delete mistaken commit

R->I: Update staging area to reset commit
note right of J: git checkout index.md
I->J: Update file to reverted version
"""
wsd(message)
Out[15]:
155
23.8 Publishing
We’re still in our working directory:
In [1]: import os
working_dir
23.8.1 Sharing your work

So far, all our work has been on our own computer. But a big part of the point of version control is keeping
your work safe, on remote servers. Another part is making it easy to share your work with the world In this
example, we’ll be using the “GitHub” cloud repository to store and publish our work.
If you have not done so already, you should create an account on GitHub: go to https://github.com/,
fill in a username and password, and click on “sign up for free”.
156
23.8.2 Creating a repository
Ok, let’s create a repository to store our work. Hit “new repository” on the right of the github home screen,
or click here.
Fill in a short name, and a description. Choose a “public” repository. Don’t choose to add a Readme.
23.8.3 Paying for GitHub

For this software carpentry course, you should use public repositories in your personal account for your
example work: it’s good to share! GitHub is free for open source, but in general, charges a fee if you want
to keep your work private.
In the future, you might want to keep your work on GitHub private.
Students can get free private repositories on GitHub, by going to [https://github.com/edu] and filling in
a form.
UCL pays for private GitHub repositories for UCL research groups: you can find the service details on
our web page.
23.8.4 Adding a new remote to your repository

Instructions will appear, once you’ve created the repository, as to how to add this new “remote” server to
your repository, in the lower box on the screen. Mine say:
In [2]: %%bash
git remote add origin git@github.com:UCL/github-example.git
In [3]: %%bash
git push -uf origin master # I have an extra ‘f‘ switch here. You should copy the instructions f
ERROR: Permission to UCL/github-example.git denied to ucl-rsd-ci.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights

and the repository exists.
23.8.5 Remotes
The first command sets up the server as a new remote, called origin.
Git, unlike some earlier version control systems is a “distributed” version control system, which means
you can work with multiple remote servers.
Usually, commands that work with remotes allow you to specify the remote to use, but assume the origin
remote if you don’t.
Here, git push will push your whole history onto the server, and now you’ll be able to see it on the
internet! Refresh your web browser where the instructions were, and you’ll see your repository!
Let’s add these commands to our diagram:
In [4]: message="""
Local Repository -> Remote Repository : git push
"""
from wsd import wsd
%matplotlib inline
wsd(message)
157
Out[4]:
23.8.6 Playing with GitHub

Take a few moments to click around and work your way through the GitHub interface. Try clicking on
‘index.md’ to see the content of the file: notice how the markdown renders prettily.
Click on “commits” near the top of the screen, to see all the changes you’ve made. Click on the commit
number next to the right of a change, to see what changes it includes: removals are shown in red, and
additions in green.
23.9 Working with multiple files

23.9.1 Some new content
So far, we’ve only worked with one file. Let’s add another:
vim lakeland.md
In [5]: %%writefile lakeland.md

Lakeland
========
Cumbria has some pretty hills, and lakes too.
Writing lakeland.md
In [6]: cat lakeland.md
Lakeland
========
Cumbria has some pretty hills, and lakes too.
158
23.9.2 Git will not by default commit your new file
In [7]: %%bash
git commit -am "Try to add Lakeland"
On branch master
Untracked files:
lakeland.md
wsd.py
wsd.pyc
nothing added to commit but untracked files present

This didn’t do anything, because we’ve not told git to track the new file yet.
23.9.3 Tell git about the new file

In [8]: %%bash
git add lakeland.md
git commit -am "Add lakeland"
[master 9d7c86a] Add lakeland
create mode 100644 lakeland.md
Ok, now we have added the change about Cumbria to the file. Let’s publish it to the origin repository.
In [9]: %%bash
git push
warning: push.default is unset; its implicit value has changed in
Git 2.0 from ’matching’ to ’simple’. To squelch this message
and maintain the traditional behavior, use:
git config --global push.default matching
To squelch this message and adopt the new behavior now, use:
git config --global push.default simple
When push.default is set to ’matching’, git will push local branches

to the remote branches that already exist with the same name.
Since Git 2.0, Git defaults to the more conservative ’simple’

behavior, which only pushes the current branch to the corresponding
remote branch that ’git pull’ uses to update the current branch.
See ’git help config’ and search for ’push.default’ for further information.
(the ’simple’ mode was introduced in Git 1.7.11. Use the similar mode
’current’ instead of ’simple’ if you sometimes use older versions of Git)
fatal: The current branch master has no upstream branch.

To push the current branch and set the remote as upstream, use
git push --set-upstream origin master

Visit GitHub, and notice this change is on your repository on the server. We could have said git push
origin to specify the remote to use, but origin is the default.
159
23.10 Changing two files at once
What if we change both files?
In [10]: %%writefile lakeland.md

Lakeland
========
Cumbria has some pretty hills, and lakes too
Mountains:
* Helvellyn
Overwriting lakeland.md

Mountains and Lakes in the UK
===================
In [12]: %%bash
git status
On branch master
modified: index.md
modified: lakeland.md
Untracked files:
wsd.py
wsd.pyc
These changes should really be separate commits. We can do this with careful use of git add, to stage
first one commit, then the other.
In [13]: %%bash
git add index.md
git commit -m "Include lakes in the scope"
[master 8eeb1aa] Include lakes in the scope

1 file changed, 2 insertions(+), 2 deletions(-)
Because we “staged” only index.md, the changes to lakeland.md were not included in that commit.
In [14]: %%bash
git commit -am "Add Helvellyn"
160
[master fa37aa5] Add Helvellyn
In [15]: %%bash
git log --oneline
fa37aa5 Add Helvellyn

8eeb1aa Include lakes in the scope
9d7c86a Add lakeland
In [16]: %%bash
git push




participant "Jim’s remote" as M

note right of J: vim lakeland.md
note right of J: git add index.md

J->I: Add *only* the changes to index.md to the staging area
161
note right of J: git commit -m "Include lakes"
I->R: Make a commit from currently staged changes: index.md only
note right of J: git commit -am "Add Helvellyn"

J->I: Stage *all remaining* changes, (lakeland.md)
I->R: Make a commit from currently staged changes
note right of J: git push

R->M: Transfer commits to Github
"""
wsd(message)
Out[17]:
23.11 Collaboration
23.11.1 Form a team
Now we’re going to get to the most important question of all with Git and GitHub: working with others.
Organise into pairs. You’re going to be working on the website of one of the two of you, together, so
decide who is going to be the leader, and who the collaborator.
23.11.2 Giving permission

The leader needs to let the collaborator have the right to make changes to his code.
In GitHub, go to settings on the right, then collaborators on the left.
Add the user name of your collaborator to the box. They now have the right to push to your repository.
23.11.3 Obtaining a colleague’s code

Next, the collaborator needs to get a copy of the leader’s code. For this example notebook, I’m going to
be collaborating with myself, swapping between my two repositories. Make yourself a space to put it your
work. (I will have two)
162
In [1]: import os
os.chdir(git_dir)
In [2]: %%bash
pwd
rm -rf github-example # cleanup after previous example
rm -rf partner_dir # cleanup after previous example
/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git
Next, the collaborator needs to find out the URL of the repository: they should go to the leader’s
repository’s GitHub page, and note the URL on the top of the screen. Make sure the “ssh” button is pushed,
the URL should begin with git@github.com.
Copy the URL into your clipboard by clicking on the icon to the right of the URL, and then:
In [3]: %%bash
pwd
git clone git@github.com:UCL/github-example.git
mv github-example partner_dir
/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git
Cloning into ’github-example’...
In [4]: partner_dir=os.path.join(git_dir, ’partner_dir’)

os.chdir(partner_dir)
In [5]: %%bash
pwd
ls
/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch02git/learning git/partner dir

Makefile
Pennines.md
Scotland.md
Wales.md
index.md
lakeland.md
Note that your partner’s files are now present on your disk:
In [6]: %%bash
cat lakeland.md
Lakeland
========
Cumbria has some pretty hills, and lakes too
Mountains:
* Helvellyn
163
23.11.4 Nonconflicting changes
Now, both of you should make some changes. To start with, make changes to different files. This will mean
your work doesn’t “conflict”. Later, we’ll see how to deal with changes to a shared file.
Both of you should commit, but not push, your changes to your respective files:
E.g., the leader:
In [8]: %%writefile Wales.md

Mountains In Wales
==================
* Tryfan
* Yr Wyddfa
Writing Wales.md
In [9]: %%bash
ls
Wales.md
index.md
lakeland.md
wsd.py
wsd.pyc
In [10]: %%bash
git add Wales.md
git commit -m "Add wales"
[master 35228fd] Add wales

create mode 100644 Wales.md
And the partner:
In [11]: os.chdir(partner_dir)
In [12]: %%writefile Scotland.md

Mountains In Scotland
==================
* Ben Eighe
* Cairngorm
Overwriting Scotland.md
In [13]: %%bash
ls
Makefile
Pennines.md
Scotland.md
Wales.md
index.md
lakeland.md
164
In [14]: %%bash
git add Scotland.md
git commit -m "Add Scotland"
[master 9ca6f1d] Add Scotland

One of you should now push with git push:
In [15]: %%bash
git push





23.11.5 Rejected push

The other should then push, but should receive an error message:
In [17]: %%bash
git push

165



Do as it suggests:
In [18]: %%bash
git pull
warning: no common commits

From github.com:UCL/github-example
* [new branch] gh-pages -> origin/gh-pages
* [new branch] master -> origin/master
* [new tag] v1.0 -> v1.0
There is no tracking information for the current branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.
git pull <remote> <branch>
If you wish to set tracking information for this branch you can do so with:
git branch --set-upstream-to=origin/<branch> master
23.11.6 Merge commits

A window may pop up with a suggested default commit message. This commit is special: it is a merge
commit. It is a commit which combines your collaborator’s work with your own.
Now, push again with git push. This time it works. If you look on GitHub, you’ll now see that it
contains both sets of changes.
In [19]: %%bash
git push

166



The partner now needs to pull down that commit:
In [21]: %%bash
git pull
Already up-to-date.
In [22]: %%bash
ls
Makefile
Pennines.md
Scotland.md
Wales.md
index.md
lakeland.md
23.11.7 Nonconflicted commits to the same file

Go through the whole process again, but this time, both of you should make changes to a single file, but
make sure that you don’t touch the same line. Again, the merge should work as before:

Mountains In Wales
==================
* Tryfan
* Snowdon
Overwriting Wales.md
In [24]: %%bash
git diff
167
diff --git a/Wales.md b/Wales.md
index 784f1df..e2ca555 100644
--- a/Wales.md
+++ b/Wales.md
@@ -1,9 +1,5 @@
Mountains In Wales
==================
-* Pen y Fan
* Tryfan
-* Snowdon
-* Glyder Fawr
-* Fan y Big
-* Cadair Idris
+* Snowdon
In [25]: %%bash
git commit -am "Translating from the Welsh"
[master 4b2dd03] Translating from the Welsh

In [26]: %%bash
git log --oneline
4b2dd03 Translating from the Welsh

9ca6f1d Add Scotland
ff7d924 Add a makefile and ignore generated files
3f34a69 Add Pennines
670f9d0 Commit Aonach onto master branch
3203223 Merge branch ’master’ of github.com:UCL/github-example
6635705 Add Glyder
c8cbfb6 Add another Beacon
d6549a0 Merge branch ’master’ of github.com:UCL/github-example
b4f45f8 Translating from the Welsh
b93da7b Add a beacon
bebd994 Merge branch ’master’ of github.com:UCL/github-example
50ea306 Add wales
ddac17c Add Scotland
7dc0ebe Add Helvellyn
dde2ca8 Include lakes in the scope
37dc94c Add lakeland
dc4aa48 Revert "Add a lie about a mountain"
e750838 Change title
006b560 Add a lie about a mountain
278d18c First commit of discourse on UK topography

Mountains In Wales
==================
168
* Pen y Fan
* Tryfan
* Snowdon
In [29]: %%bash
git commit -am "Add a beacon"
[master 1e4351d] Add a beacon

In [30]: %%bash
git log --oneline
1e4351d Add a beacon

35228fd Add wales
fa37aa5 Add Helvellyn
8eeb1aa Include lakes in the scope
9d7c86a Add lakeland
In [31]: %%bash
git push




Switching back to the other partner. . .
169
In [33]: %%bash
git push





In [34]: %%bash
git pull
Already up-to-date.
In [35]: %%bash
git push



170


In [36]: %%bash
git log --oneline --graph
* 4b2dd03 Translating from the Welsh

* 9ca6f1d Add Scotland
* ff7d924 Add a makefile and ignore generated files
* 3f34a69 Add Pennines
* 670f9d0 Commit Aonach onto master branch
* 3203223 Merge branch ’master’ of github.com:UCL/github-example
|\
| * c8cbfb6 Add another Beacon
* | 6635705 Add Glyder
|/
* d6549a0 Merge branch ’master’ of github.com:UCL/github-example
|\
| * b93da7b Add a beacon
* | b4f45f8 Translating from the Welsh
|/
* bebd994 Merge branch ’master’ of github.com:UCL/github-example
|\
| * ddac17c Add Scotland
* | 50ea306 Add wales
|/
* 7dc0ebe Add Helvellyn
* dde2ca8 Include lakes in the scope
* 37dc94c Add lakeland
* dc4aa48 Revert "Add a lie about a mountain"
* e750838 Change title
* 006b560 Add a lie about a mountain
* 278d18c First commit of discourse on UK topography
In [38]: %%bash
git pull

171
In [39]: %%bash
git log --graph --oneline
* 1e4351d Add a beacon

* 35228fd Add wales
* fa37aa5 Add Helvellyn
* 8eeb1aa Include lakes in the scope
* 9d7c86a Add lakeland
* b7b8924 Revert "Add a lie about a mountain"
* 201110e Change title
* 47f2a5c Add a lie about a mountain
* 1561abf First commit of discourse on UK topography
participant Sue as S
participant "Sue’s repo" as SR
participant "Shared remote" as M
participant "Jim’s repo" as JR
note left of S: git clone

M->SR: fetch commits
SR->S: working directory as at latest commit
note left of S: edit Scotland.md

note right of J: edit Wales.md
note left of S: git commit -am "Add scotland"

S->SR: create commit with Scotland file
note right of J: git commit -am "Add wales"

J->JR: create commit with Wales file
note left of S: git push

SR->M: update remote with changes

JR-->M: !Rejected change
note right of J: git pull

M->JR: Pull in Sue’s last commit, merge histories
JR->J: Add Scotland.md to working directory

JR->M: Transfer merged history to remote
"""
from wsd import wsd
%matplotlib inline
wsd(message)
Out[40]:
172
23.11.8 Conflicting commits
Finally, go through the process again, but this time, make changes which touch the same line.

Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Fan y Big
In [42]: %%bash
git commit -am "Add another Beacon"
git push
[master c4189aa] Add another Beacon


173




Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Glyder Fawr
In [45]: %%bash
git commit -am "Add Glyder"
git push
[master ae1a72a] Add Glyder




174


When you pull, instead of offering an automatic merge commit message, it says:
In [46]: %%bash
git pull
Already up-to-date.
23.11.9 Resolving conflicts

Git couldn’t work out how to merge the two different sets of changes.
You now need to manually resolve the conflict.
It has marked the conflicted area:
In [47]: %%bash
cat Wales.md
Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Glyder Fawr
Manually edit the file, to combine the changes as seems sensible and get rid of the symbols:

Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Glyder Fawr
* Fan y Big
23.11.10 Commit the resolved file

Now commit the merged result:
In [49]: %%bash
git commit -a --no-edit # I added a No-edit for this non-interactive session. You can edit the
Aborting commit due to empty commit message.
175
In [50]: %%bash
git push





In [52]: %%bash
git pull

In [53]: %%bash
cat Wales.md
Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Fan y Big
176
In [54]: %%bash
* c4189aa Add another Beacon
* 35228fd Add wales
23.11.11 Distributed VCS in teams with conflicts

participant Sue as S
participant "Sue’s repo" as SR
participant "Shared remote" as M
participant "Jim’s repo" as JR
note left of S: edit the same line in wales.md

note right of J: edit the same line in wales.md
note left of S: git commit -am "update wales.md"

S->SR: add commit to local repo
note right of J: git commit -am "update wales.md"

J->JR: add commit to local repo
note left of S: git push

SR->M: transfer commit to remote

JR->M: !Rejected
note right of J: git pull

JR->J: Make confliced file with conflict markers
note right of J: edit file to resolve conflicts

note right of J: git add wales.md
note right of J: git commit
J->JR: Mark conflict as resolved

JR->M: Transfer merged history to remote
note left of S: git pull

M->SR: Download Jim’s resolution of conflict.
"""
wsd(message)
177
Out[55]:
23.11.12 The Levels of Git

Local Repository -> Remote Repository : git push
Remote Repository -> Local Repository : git fetch
Local Repository -> Staging Area : git merge
Remote Repository -> Working Directory: git pull
"""
wsd(message)
Out[56]:
178
23.12 Editing directly on GitHub
23.12.1 Editing directly on GitHub
Note that you can also make changes in the GitHub website itself. Visit one of your files, and hit “edit”.
Make a change in the edit window, and add an appropriate commit message.
That change now appears on the website, but not in your local copy. (Verify this).
Now pull, and check the change is now present on your local version.
23.13 Social Coding

23.13.1 GitHub as a social network
In addition to being a repository for code, and a way to publish code, GitHub is a social network.
You can follow the public work of other coders: go to the profile of your collaborator in your browser,
and it the “follow” button.
Here’s mine : if you want to you can follow me.
Using GitHub to build up a good public profile of software projects you’ve worked on is great for your
CV!
23.14 Fork and Pull

23.14.1 Different ways of collaborating
We have just seen how we can work with others on GitHub: we add them as collaborators on our repositories
and give them permissions to push changes.
Let’s talk now about some other type of collaboration.
Imagine you are a user of an Open Source project like Numpy and find a bug in one of their methods.
179
You can inspect and clone Numpy’s code in GitHub https://github.com/numpy/numpy, play around a
bit and find how to fix the bug.
Numpy has done so much for you asking nothing in return, that you really want to contribute back by
fixing the bug for them.
You make all of the changes but you can’t push it back to Numpy’s repository because you don’t have
permissions.
The right way to do this is forking Numpy’s repository.
23.14.2 Forking a repository on GitHub

By forking a repository, all you do is make a copy of it in your GitHub account, where you will have write
permissions as well.
If you fork Numpy’s repository, you will find a new repository in your GitHub account that is an exact
copy of Numpy. You can then clone it to your computer, work locally on fixing the bug and push the changes
to your fork of Numpy.
Once you are happy with with the changes, GitHub also offers you a way to notify Numpy’s developers
of this changes so that they can include them in the official Numpy repository via starting a Pull Request.
23.14.3 Pull Request

You can create a Pull Request and select those changes that you think can be useful for fixing Numpy’s bug.
Numpy’s developers will review your code and make comments and suggestions on your fix. Then, you
can commit more improvements in the pull request for them to review and so on.
Once Numpy’s developers are happy with your changes, they’ll accept your Pull Request and merge the
changes into their original repository, for everyone to use.
23.14.4 Practical example - Team up!

We will be working in the same repository with one of you being the leader and the other being the collab-
orator.
Collaborators need to go to the leader’s GitHub profile and find the repository we created for that lesson.
Mine is in https://github.com/jamespjh/github-example
1. Fork repository
You will see on the top right of the page a Fork button with an accompanying number indicating how many
GitHub users have forked that repository.
Collaborators need to navigate to the leader’s repository and click the Fork button.
Collaborators: note how GitHub has redirected you to your own GitHub page and you are now looking
at an exact copy of the team leader’s repository.
2. Clone your forked repo

Collaborators: go to your terminal and clone the newly created fork.
git clone git@github.com:jamespjh/github-example.git
3. Create a feature branch

It’s a good practice to create a new branch that’ll contain the changes we want. We’ll learn more about
branches later on. For now, just think of this as a separate area where our changes will be kept not to
interfere with other people’s work.
git checkout -b southwest
180
4. Make, commit and push changes to new branch
For example, let’s create a new file called SouthWest.md and edit it to add this text:
* Exmoor
* Dartmoor
* Bodmin Moor
Save it, and push this changes to your fork’s new branch:
git add SouthWest.md

git commit -m "The South West is also hilly."
git push origin southwest
5. Create Pull Request

Go back to the collaborator’s GitHub site and reload the fork. GitHub has noticed there is a new branch
and is presenting us with a green button to Compare & pull request. Fantastic! Click that button.
Fill in the form with additional information about your change, as you consider necesary to make the
team leader understand what this is all about.
Take some time to inspect the commits and the changes you are submitting for review. When you are
ready, click on the Create Pull Request button.
Now, the leader needs to go to their GitHub site. They have been notified there is a pull request in their
repo awaiting revision.
6. Feedback from team leader

Leaders can see the list of pull requests in the vertical menu of the repo, on the right hand side of the screen.
Select the pull request the collaborator has done, and inspect the changes.
There are three tabs: in one you can start a conversation with the collaborator about their changes, and
in the others you can have a look at the commits and changes made.
Go to the tab labeled as “Files Changed”. When you hover over the changes, a small + button appears.
Select one line you want to make a comment on. For example, the line that contains “Exmoor”.
GitHub allows you to add a comment about that specific part of the change. Your collaborator has
forgotten to add a title at the beginning of the file right before “Exmoor”, so tell them so in the form
presented after clicking the + button.
7. Fixes by collaborator
Collaborators will be notified of this comment by email and also in their profiles page. Click the link
accompanying this notification to read the comment from the team leader.
Go back to your local repository, make the changes suggested and push them to the new branch.
Add this at the beginning of your file:
Hills in the South West:

=======================
Then push the change to your fork:
git add .
git commit -m "Titles added as requested."
git push origin southwest
This change will automatically be added to the pull request you started.
181
8. Leader accepts pull request
The team leader will be notified of the new changes that can be reviewed in the same fashion as earlier.
Let’s assume the team leader is now happy with the changes.
Leaders can see in the “Conversation” tab of the pull request a green button labelled Merge pull
request. Click it and confirm the decission.
The collaborator’s pull request has been accepted and appears now in the original repository owned by
the team leader.
Fork and Pull Request done!
23.14.5 Some Considerations

• Fork and Pull Request are things happening only on the repository’s server side (GitHub in our
case). Consequently, you can’t do things like git fork or git pull-request from the local copy of a
repository.
• You not always need to fork repositories with the intention of contributing. You can fork a library you
use, install it manually on your computer, and add more functionality or customise the existing one,
so that it is more useful for you and your team.
• Numpy’s example is only illustrative. Normally, Open Source projects have in their Wiki a set of
instructions you need to follow if you want to contribute to their software.
• Pull Requests can also be done for merging branches in a non-forked repository. It’s typically used in
teams to merge code from a branch into the master branch and ask team colleagues for code reviews
before merging.
• It’s a good practice before starting a fork and a pull request to have a look at existing forks and pull
requests. You can find the list of pull requests on the vertical menu on right hand side. Try to also
find the network graph displaying all existing forks of a repo, like this example in the NumpyDoc repo:
https://github.com/numpy/numpydoc/network
23.15 Git Theory

23.15.1 The revision Graph
Revisions form a GRAPH
In [1]: import os
In [2]: %%bash
git log --graph --oneline

* 35228fd Add wales
182
23.15.2 Git concepts
• Each revision has a parent that it is based on
• These revisions form a graph
• Each revision has a unique hash code
• In Sue’s copy, revision 43 is ab3578d6
• Jim might think that is revision 38, but it’s still ab3579d6
• Branches, tags, and HEAD are labels pointing at revisions
• Some operations (like fast forward merges) just move labels.
23.15.3 The levels of Git

There are four Separate levels a change can reach in git:
• The Working Copy

• The index (aka staging area)
• The local repository
• The remote repository
Understanding all the things git reset can do requires a good grasp of git theory.
• git reset <commit> <filename> : Reset index and working version of that file to the version in a
given commit
• git reset --soft <commit>: Move local repository branch label to that commit, leave working dir
and index unchanged
• git reset <commit>: Move local repository and index to commit (“–mixed”)
• git reset --hard <commit>: Move local repostiory, index, and working directory copy to that state
23.16 Branches
Branches are increadibly important to why git is cool and powerful.
They are an easy and cheap way of making a second version of your software, which you work on in
parallel, and pull in your changes when you are ready.
In [1]: import os
In [2]: %%bash
git branch # Tell me what branches exist
* master
In [3]: %%bash
git checkout -b experiment # Make a new branch
Switched to a new branch ’experiment’
In [4]: %%bash
git branch
* experiment
master
183
In [5]: %%bash
git commit -am "Add Cadair Idris"
On branch experiment
Untracked files:
wsd.py
wsd.pyc

In [6]: %%bash
git checkout master # Switch to an existing branch
Switched to branch ’master’
In [7]: %%bash
cat Wales.md
Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Fan y Big
In [8]: %%bash
git checkout experiment
Switched to branch ’experiment’
In [9]: cat Wales.md
Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Fan y Big
23.16.1 Publishing branches

To let the server know there’s a new branch use:
In [10]: %%bash
git push -u origin experiment

We use --set-upstream origin (Abbreviation -u) to tell git that this branch should be pushed to and
pulled from origin per default.
If you are following along, you should be able to see your branch in the list of branches in GitHub.
Once you’ve used git push -u once, you can push new changes to the branch with just a git push.
If others checkout your repository, they will be able to do git checkout experiment to see your branch
content, and collaborate with you in the branch.
184
In [11]: %%bash
git branch -r
origin/gh-pages
origin/master
Local branches can be, but do not have to be, connected to remote branches They are said to “track”
remote branches. push -u sets up the tracking relationship.
In [12]: %%bash
git branch -vv
* experiment c4189aa Add another Beacon
master c4189aa Add another Beacon
23.16.2 Find out what is on a branch

In addition to using git diff to compare to the state of a branch, you can use git log to look at lists of
commits which are in a branch and haven’t been merged yet.
In [13]: %%bash
git log master..experiment
Git uses various symbols to refer to sets of commits. The double dot A..B means “ancestor of B and not
ancestor of A”
So in a purely linear sequence, it does what you’d expect.
In [14]: %%bash
git log --graph --oneline HEAD~9..HEAD~5
But in cases where a history has branches, the definition in terms of ancestors is important.
In [15]: %%bash
git log --graph --oneline HEAD~5..HEAD
* 35228fd Add wales
If there are changes on both sides, like this:
In [16]: %%bash
git checkout master
In [17]: %%writefile Scotland.md
Mountains In Scotland
==================
* Ben Eighe
* Cairngorm
* Aonach Eagach
185
Writing Scotland.md
In [18]: %%bash
git diff Scotland.md
In [19]: %%bash
git commit -am "Commit Aonach onto master branch"
On branch master
Untracked files:
Scotland.md
wsd.py
wsd.pyc
Then this notation is useful to show the content of what’s on what branch:
In [20]: %%bash
git log --left-right --oneline master...experiment
Three dots means “everything which is not a common ancestor” of the two commits, i.e. the differences
between them.
23.16.3 Merging branches

We can merge branches, and just as we would pull in remote changes, there may or may not be conflicts.
In [21]: %%bash
git branch
git merge experiment
experiment
* master
Already up-to-date.
In [22]: %%bash
git log --graph --oneline HEAD~3..HEAD

* 35228fd Add wales
23.16.4 Cleaning up after a branch

In [23]: %%bash
git branch
experiment
* master
In [24]: %%bash
git branch -d experiment
Deleted branch experiment (was c4189aa).
In [25]: %%bash
git branch
186
* master
In [26]: %%bash
git branch --remote
origin/gh-pages
origin/master
In [27]: %%bash
git push --delete origin experiment # Remove remote branch - also can use github interface


In [28]: %%bash
git branch --remote
origin/gh-pages
origin/master
23.16.5 A good branch strategy

• A production branch: code used for active work
• A develop branch: for general new code
• feature branches: for specific new ideas
• release branches: when you share code with others
• Useful for isolated bug fixes
23.16.6 Grab changes from a branch

Make some changes on one branch, switch back to another, and use:
git checkout <branch> <path>
To quickly grab a file from one branch into another.

Using git checkout with a path takes the content of files. To grab the content of a specific commit from
another branch, and apply it as a patch to your branch, use:
git cherry-pick <commit>
23.17 Git Stash

If you find you want to pull, but you’re not ready to commit, you can use
In [1]: import os
187
Mountains In Wales
==================
* Pen y Fan
* Tryfan
* Snowdon
* Glyder Fawr
* Fan y Big
* Cadair Idris
In [3]: %%bash
git stash
git pull
Saved working directory and index state WIP on master: c4189aa Add another Beacon
HEAD is now at c4189aa Add another Beacon

In [4]: %%bash
git stash apply
On branch master
modified: Wales.md
Untracked files:
Scotland.md
wsd.py
wsd.pyc
The “Stash” is a way of temporarily saving your working area, and can help out in a pinch.
23.18 Tagging
Tags are easy to read labels for revisions, and can be used anywhere we would name a commit.
Produce real results only with tagged revisions
188
In [5]: %%bash
git tag -a v1.0 -m "Release 1.0"
git push --tags
fatal: tag ’v1.0’ already exists


In [6]: %%writefile Pennines.md
Mountains In the Pennines

========================
* Cross Fell
Writing Pennines.md
In [7]: %%bash
git add Pennines.md
git commit -am "Add Pennines"
[master e218d6b] Add Pennines

2 files changed, 8 insertions(+), 1 deletion(-)
create mode 100644 Pennines.md
In [8]: %%bash
git log v1.0.. --graph --oneline
* e218d6b Add Pennines

* 35228fd Add wales
If .. is used without a following commit name, HEAD is assumed.
23.19 Working with generated files: gitignore

We often end up with files that are generated by our program. It is bad practice to keep these in Git; just
keep the sources.
Examples include .o and .x files for compiled languages, .pyc files in Python.
In our example, we might want to make our .md files into a PDF with pandoc:
In [9]: %%writefile Makefile
MDS=$(wildcard *.md)
189
PDFS=$(MDS:.md=.pdf)
default: $(PDFS)
%.pdf: %.md
pandoc $< -o $@
Writing Makefile
In [10]: %%bash
make
pandoc Pennines.md -o Pennines.pdf

pandoc Scotland.md -o Scotland.pdf
pandoc Wales.md -o Wales.pdf
pandoc index.md -o index.pdf
pandoc lakeland.md -o lakeland.pdf
We now have a bunch of output .pdf files corresponding to each Markdown file.
But we don’t want those to show up in git:
In [11]: %%bash
git status
On branch master
Untracked files:
Makefile
Pennines.pdf
Scotland.md
Scotland.pdf
Wales.pdf
index.pdf
lakeland.pdf
wsd.py
wsd.pyc
nothing added to commit but untracked files present (use "git add" to track)
Use .gitignore files to tell Git not to pay attention to files with certain paths:
In [12]: %%writefile .gitignore

*.pdf
Writing .gitignore
In [13]: %%bash
git status
On branch master
Untracked files:
.gitignore
Makefile
190
Scotland.md
wsd.py
wsd.pyc
nothing added to commit but untracked files present (use "git add" to track)
In [14]: %%bash
git add Makefile
git add .gitignore
git commit -am "Add a makefile and ignore generated files"
git push
[master b298c68] Add a makefile and ignore generated files
2 files changed, 9 insertions(+)
create mode 100644 .gitignore
create mode 100644 Makefile



23.20 Git clean

In [15]: %%bash
git clean -fX
Removing Pennines.pdf
Removing Scotland.pdf
Removing Wales.pdf
Removing index.pdf
Removing lakeland.pdf
In [16]: %%bash
ls
191
Makefile
Pennines.md
Scotland.md
Wales.md
index.md
lakeland.md
wsd.py
wsd.pyc
• With -f: don’t prompt

• with -d: remove directories
• with -x: Also remote .gitignored files
• with -X: Only remove .gitignore files
23.21 Hunks
23.21.1 Git Hunks
A “Hunk” is one git change. This changeset has three hunks:
+import matplotlib
+import numpy as np
from matplotlib import pylab

from matplotlib.backends.backend_pdf import PdfPages
+def increment_or_add(key,hash,weight=1):
+ if key not in hash:
+ hash[key]=0
+ hash[key]+=weight
+
data_path=os.path.join(os.path.dirname(
os.path.abspath(__file__)),
-regenerate=False
+regenerate=True
23.21.2 Interactive add

git add and git reset can be used to stage/unstage a whole file, but you can use interactive mode to stage
by hunk, choosing yes or no for each hunk.
git add -p myfile.py
+import matplotlib
+import numpy as np
#Stage this hunk [y,n,a,d,/,j,J,g,e,?]?
23.22 GitHub pages

23.22.1 Yaml Frontmatter
GitHub will publish repositories containing markdown as web pages, automatically.
You’ll need to add this content:
192
---
---
A pair of lines with three dashes, to the top of each markdown file. This is how GitHub knows which
markdown files to make into web pages. Here’s why for the curious.
---
title: Github Pages Example
---
Mountains and Lakes in the UK
===================

In [18]: %%bash
git commit -am "Add github pages YAML frontmatter"
[master 491044f] Add github pages YAML frontmatter
1 file changed, 6 insertions(+), 2 deletions(-)
23.22.2 The gh-pages branch

GitHub creates github pages when you use a special named branch.
This is best used to create documentation for a program you write, but you can use it for anything.
In [20]: %%bash
git checkout -b gh-pages

git push -u origin gh-pages
Switched to a new branch ’gh-pages’

The first time you do this, GitHub takes a few minutes to generate your pages.
The website will appear at http://username.github.io/repositoryname, for example:
http://UCL.github.io/github-example/
23.22.3 UCL layout for GitHub pages

You can use GitHub pages to make HTML layouts, here’s an example of how to do it, and how it looks. We
won’t go into the detail of this now, but after the class, you might want to try this.
23.23 Working with multiple remotes

23.23.1 Distributed versus centralised
Older version control systems (cvs, svn) were “centralised”; the history was kept only on a server, and all
commits required an internet.
193
Centralised Distributed
Server has history Every user has full history
Your computer has one snapshot Many local branches
To access history, need internet History always available
You commit to remote server Users synchronise histories
cvs, subversion(svn) git, mercurial (hg), bazaar (bzr)
With modern distributed systems, we can add a second remote. This might be a personal fork on github:
In [1]: import os
In [2]: %%bash
git remote add jamespjh git@github.com:jamespjh/github-example.git
git remote -v
jamespjh git@github.com:jamespjh/github-example.git (fetch)

jamespjh git@github.com:jamespjh/github-example.git (push)
origin git@github.com:UCL/github-example.git (fetch)
origin git@github.com:UCL/github-example.git (push)
We can push to a named remote:
In [3]: %%writefile Pennines.md
Mountains In the Pennines

========================
* Cross Fell
* Whernside
Overwriting Pennines.md
In [4]: %%bash
git commit -am "Add Whernside"
[gh-pages f7aff06] Add Whernside

In [5]: %%bash
git push jamespjh

194

ERROR: Permission to jamespjh/github-example.git denied to ucl-rsd-ci.


23.23.2 Referencing remotes

You can always refer to commits on a remote like this:
In [6]: %%bash
git fetch
git log --oneline --left-right jamespjh/master...origin/master
fatal: ambiguous argument ’jamespjh/master...origin/master’: unknown revision or path not in the working
Use ’--’ to separate paths from revisions, like this:
’git <command> [<revision>...] -- [<file>...]’
To see the differences between remotes, for example.

To see what files you have changed that aren’t updated on a particular remote, for example:
In [7]: %%bash
git diff --name-only origin/master
Pennines.md
Scotland.md
index.md
When you reference remotes like this, you’re working with a cached copy of the last time you interacted
with the remote. You can do git fetch to update local data with the remotes without actually pulling. You
can also get useful information about whether tracking branches are ahead or behind the remote breanches
they track:
In [8]: %%bash
git branch -vv
* gh-pages f7aff06 Add Whernside

master 491044f Add github pages YAML frontmatter
23.24 Hosting Servers

23.24.1 Hosting a local server
• Any repository can be a remote for pulls
• Can pull/push over shared folders or ssh
195
• Pushing to someone’s working copy is dangerous
• Use git init --bare to make a copy for pushing
• You don’t need to create a “server” as such, any ‘bare’ git repo will do.
In [9]: bare_dir=os.path.join(git_dir, ’bare_repo’)

os.chdir(git_dir)
In [10]: %%bash
mkdir -p bare_repo
cd bare_repo
git init --bare
Initialized empty Git repository in /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/c
In [12]: %%bash
git remote add local_bare ../bare_repo
git push local_bare



To ../bare repo
* [new branch] gh-pages -> gh-pages
In [13]: %%bash
git remote -v
jamespjh git@github.com:jamespjh/github-example.git (fetch)

jamespjh git@github.com:jamespjh/github-example.git (push)
local bare ../bare repo (fetch)
local bare ../bare repo (push)
origin git@github.com:UCL/github-example.git (fetch)
origin git@github.com:UCL/github-example.git (push)
You can now work with this local repository, just as with any other git server. If you have a colleague
on a shared file system, you can use this approach to collaborate through that file system.
196
23.24.2 Home-made SSH servers
Classroom exercise: Try creating a server for yourself using a machine you can SSH to:
ssh <mymachine>
mkdir mygitserver
cd mygitserver
git init --bare
exit
git remote add <somename> ssh://user@host/mygitserver
git push -u <somename> master
23.25 SSH keys and GitHub

Classroom exercise: If you haven’t already, you should set things up so that you don’t have to keep typing
in your password whenever you interact with GitHub via the command line.
You can do this with an “ssh keypair”. You may have created a keypair in the Software Carpentry shell
training. Go to the ssh settings page on GitHub and upload your public key by copying the content from
your computer. (Probably at .ssh/id rsa.pub)
If you have difficulties, the instructions for this are on the GitHub website.
23.26 Rebasing
23.26.1 Rebase vs merge
A git merge is only one of two ways to get someone else’s work into yours. The other is called a rebase.
In a merge, a revision is added, which brings the branches together. Both histories are retained. In a
rebase, git tries to work out
What would you need to have done, to make your changes, if your colleague had already made
theirs?
Git will invent some new revisions, and the result will be a repository with an apparently linear history.
23.26.2 An example rebase

We’ve built a repository to help visualise the difference between a merge and a rebase, at
https://github.com/UCL-RITS/wocky rebase/blob/master/wocky.md .
The initial state of both collaborators is a text file, wocky.md:
It was clear and cold,

and the slimy monsters
On the master branch, a second commit (‘Dancing’) has been added:
It was clear and cold,

and the slimy monsters
danced and spun in the waves
On the “Carollian” branch, a commit has been added translating the initial state into Lewis Caroll’s
language:
’Twas brillig,
and the slithy toves
So the logs look like this:
197
git log --oneline --graph master
* 2a74d89 Dancing
* 6a4834d Initial state
git log --oneline --graph carollian
* 2232bf3 Translate into Caroll’s language

If we now merge carollian into master, the final state will include both changes:
’Twas brillig,
and the slithy toves
danced and spun in the waves
But the graph shows a divergence and then a convergence:
* b41f869 Merge branch ’carollian’ into master_merge_carollian

|\
| * 2232bf3 Translate into Caroll’s language
* | 2a74d89 Dancing
|/
But if we rebase, the final content of the file is still the same, but the graph is different:
git log --oneline --graph master_rebase_carollian
* df618e0 Dancing
* 2232bf3 Translate into Caroll’s language
To trigger the rebase, we did:
git checkout master

git rebase carollian
If this had been a remote, we would merge it with:
git pull --rebase
23.26.3 Fast Forwards

If we want to continue with the translation, and now want to merge the rebased branch into the carollian
branch, we get:
Updating 2232bf3..df618e0
Fast-forward
wocky.md | 1 +
1 file changed, 1 insertion(+)
The rebased branch was rebased on the carollian branch, so this merge was just a question of updating
metadata to redefine the branch label: a “fast forward”.
198
23.26.4 Rebasing pros and cons
Some people like the clean, apparently linear history that rebase provides.
But rebase rewrites history.
If you’ve already pushed, or anyone else has got your changes, things will get screwed up.
If you know your changes are still secret, it might be better to rebase to keep the history clean. If in
doubt, just merge.
23.27 Squashing
A second use of the git rebase command, is to rebase your work on top of one of your own earlier commits,
in interactive mode, to “squash” several commits that should really be one:
git log
ea15 Some good work

ll54 Fix another typo
de73 Fix a typo
ab11 A great piece of work
cd27 Initial commit
23.27.1 Using rebase to squash

If we type
git rebase -i ab11 #OR HEAD^^
an edit window pops up with:
pick cd27 Initial commit

pick ab11 A great piece of work
pick de73 Fix a typo
pick ll54 Fix another typo
pick ea15 Some good work
# Rebase 60709da..30e0ccb onto 60709da

#
# Commands:
# p, pick = use commit
# e, edit = use commit, but stop for amending
# s, squash = use commit, but meld into previous commit
We can rewrite select commits to be merged, so that the history is neater before we push. This is a great
idea if you have lots of trivial typo commits.
pick cd27 Initial commit

pick ab11 A great piece of work
squash de73 Fix a typo
squash ll54 Fix another typo
pick ea15 Some good work
save the interactive rebase config file, and rebase will build a new history:
git log
199
de82 Some good work
fc52 A great piece of work
cd27 Initial commit
Note the commit hash codes for ‘Some good work’ and ‘A great piece of work’ have changed, as the
change they represent has changed.
23.28 Debugging With Git Bisect

You can use
git bisect
to find out which commit caused a bug.
23.28.1 An example repository

In a nice open source example, I found an arbitrary exemplar on github
In [1]: import os
os.chdir(git_dir)
In [2]: %%bash
rm -rf bisectdemo
git clone git@github.com:shawnsi/bisectdemo.git
Cloning into ’bisectdemo’...
In [3]: bisect_dir=os.path.join(git_dir,’bisectdemo’)
os.chdir(bisect_dir)
In [4]: %%bash
python squares.py 2 # 4
This has been set up to break itself at a random commit, and leave you to use bisect to work out where
it has broken:
In [5]: %%bash
./breakme.sh > break_output
error: branch ’buggy’ not found.

Switched to a new branch ’buggy’
Which will make a bunch of commits, of which one is broken, and leave you in the broken final state
In [6]: python squares.py 2 # Error message
File "<ipython-input-6-69f578907137>", line 1

python squares.py 2 # Error message
^
200
In [7]: ### Bisecting manually
In [8]: %%bash
git bisect start
git bisect bad # We know the current state is broken
git checkout master
git bisect good # We know the master branch state is OK
Your branch is up-to-date with ’origin/master’.

Bisecting: 500 revisions left to test after this (roughly 9 steps)
[719b7e51df8a2f7b66de8a0ba59fffc2d066e851] Comment 499
Bisect needs one known good and one known bad commit to get started
23.28.2 Solving Manually

git bisect good
git bisect good
git bisect good
python squares.py 2 # Crash
git bisect bad
git bisect bad
git bisect bad
python squares.py 2 #Crash
git bisect bad
git bisect good
git bisect good
git bisect good
And eventually:
git bisect good

python squares.py 2
4
git bisect good

2777975a2334c2396ccb9faf98ab149824ec465b is the first bad commit
commit 2777975a2334c2396ccb9faf98ab149824ec465b
Author: Shawn Siefkas <shawn.siefkas@meredith.com>
Date: Thu Nov 14 09:23:55 2013 -0600
Breaking argument type
git bisect end
201
23.28.3 Solving automatically
If we have an appropriate unit test, we can do all this automatically:
In [9]: %%bash
git bisect start
git bisect bad HEAD # We know the current state is broken
git bisect good master # We know master is good
git bisect run python squares.py 2

[719b7e51df8a2f7b66de8a0ba59fffc2d066e851] Comment 499
running python squares.py 2
[0918c3c8da8afc744f5c3ddf1f940094fa69eef1] Comment 249
[b27775cf96b1ee803289d71ae9aae6407832dc55] Comment 124
[f2c4f2b2eac0aea9cee88b5282fc4842ce2fdee7] Comment 61
[860cb6491b2eb040988d7a509229dea0e05376db] Comment 31
4
[9d64f7daf0ca43be200721684c58798a2d631532] Comment 46
4
[f8c592c0ae4806f577962951fcd8bbc57fa4413a] Comment 54
4
[84233913d49cf6017ae7f4690c07d551e626b9f5] Comment 58
4
Bisecting: 1 revision left to test after this (roughly 1 step)
[ca48f57b25ea84c65c66dd86a7a7f6f98fa0c2bb] Breaking argument type
[cd737c00780c8e7a25f613a7699bb190cfc8a8d4] Comment 59
4
ca48f57b25ea84c65c66dd86a7a7f6f98fa0c2bb is the first bad commit
commit ca48f57b25ea84c65c66dd86a7a7f6f98fa0c2bb
Author: Shawn Siefkas <shawn.siefkas@meredith.com>
Date: Thu Nov 14 09:23:55 2013 -0600
Breaking argument type
:100644 100644 6fab6fa23fbb028fed0133de2ec4830970ec5205 b9256a8134240ac2b5c9a80ccf3b0de66850f02e M

bisect run success
202
Previous HEAD position was 719b7e5... Comment 499
Switched to branch ’buggy’
Traceback (most recent call last):
File "squares.py", line 9, in <module>
print(integer**2)
TypeError: unsupported operand type(s) for ** or pow(): ’str’ and ’int’
print(integer**2)
print(integer**2)
print(integer**2)
print(integer**2)
Boom!
23.29 Testing
23.29.1 A few reasons not to do testing
Sensibility Sense
It’s boring Maybe
Code is just a one off throwaway As with most research codes
No time for it A bit more code, a lot less debugging
Tests can be buggy too See above
Not a professional programmer See above
Will do it later See above
23.29.2 A few reasons to do testing

• lazyness testing saves time
• peace of mind tests (should) ensure code is correct
• runnable specification best way to let others know what a function should do and not do
• reproducible debugging debugging that happened and is saved for later reuse
• code structure / modularity since the code is designed for at least two situations
• easier to modify since results can be tested
23.29.3 Not a panacea

Trying to improve the quality of software by doing more testing is like trying to lose weight by
weighting yourself more often. - Steve McConnell
• Testing won’t corrrect a buggy code

• Testing will tell you were the bugs are. . .
203
• . . . if the test cases cover the bugs
23.30 Testing primer

23.30.1 Tests at different scales
Level of test Area covered by test

Unit testing smallest logical block of work (often < 10 lines of code)
Component testing several logical blocks of work together
Integration testing all components together / whole program
Always start at the smallest scale!

If a unit test is too complicated, go smaller.
23.30.2 Legacy code hardening

• Very difficult to create unit-tests for existing code
• Instead we make a regression test
• Run program as a black box:
setup input
run program
read output
check output against expected result
• Does not test correctness of code

• Checks code is a similarly wrong on day N as day 0
23.30.3 Testing vocabulary

• fixture: input data
• action: function that is being tested
• expected result: the output that should be obtained
• actual result: the output that is obtained
• coverage: proportion of all possible paths in the code that the tests take
23.30.4 Branch coverage:

if energy > 0:
! Do this
else:
! Do that
Is there a test for both energy > 0 and energy <= 0?
204
Chapter 24
How to Test
24.0.5 Equivalence partitioning

Think hard about the different cases the code will run under: this is science, not coding!
We can’t write a test for every possible input: this is an infinite amount of work.
We need to write tests to rule out different bugs. There’s no need to separately test equivalent inputs.
Let’s look at an example of this question outside of coding:
• Research Project : Evolution of agricultural fields in Saskatchewan from aerial photography

• In silico translation : Compute overlap of two rectangles
In [1]: import matplotlib.pyplot as plt

from matplotlib.path import Path
import matplotlib.patches as patches
%matplotlib inline
Let’s make a little fragment of matplotlib code to visualise a pair of fields.
In [2]: def show_fields(field1, field2):

def vertices(left, bottom, right, top):
verts = [(left, bottom),
(left, top),
(right, top),
(right, bottom),
(left, bottom)]
return verts
codes = [Path.MOVETO,
Path.LINETO,
Path.LINETO,
Path.LINETO,
Path.CLOSEPOLY]
path1 = Path(vertices(*field1), codes)
path2 = Path(vertices(*field2), codes)
fig = plt.figure()
ax = fig.add_subplot(111)
patch1 = patches.PathPatch(path1, facecolor=’orange’, lw=2)
patch2 = patches.PathPatch(path2, facecolor=’blue’, lw=2)
ax.add_patch(patch1)
ax.add_patch(patch2)
ax.set_xlim(0,5)
ax.set_ylim(0,5)
205
show_fields((1.,1.,4.,4.),(2.,2.,3.,3.))
Here, we can see that the area of overlap, is the same as the smaller field, with area 1.
We could now go ahead and write a subroutine to calculate that, and also write some test cases for our
answer.
But first, let’s just consider that question abstractly, what other cases, not equivalent to this might there
be?
For example, this case, is still just a full overlap, and is sufficiently equivalent that it’s not worth another
test:
In [3]: show_fields((1.,1.,4.,4.),(2.5,1.7,3.2,3.4))
206
But this case is no longer a full overlap, and should be tested separately:
In [4]: show_fields((1.,1.,4.,4.),(2.,2.,3.,4.5))
On a piece of paper, sketch now the other cases you think should be treated as non-equivalent. The
answers are in a separate notebook.
207
In [5]: show_fields((1.,1.,4.,4.),(2,2,4.5,4.5)) # Overlap corner
In [6]: show_fields((1.,1.,4.,4.),(2.,2.,3.,4.)) # Just touching
In [7]: show_fields((1.,1.,4.,4.),(4.5,4.5,5,5)) # No overlap
208
In [8]: show_fields((1.,1.,4.,4.),(2.5,4,3.5,4.5)) # Just touching from outside
In [9]: show_fields((1.,1.,4.,4.),(4,4,4.5,4.5)) # Touching corner
209
24.1 Using our tests
OK, so how might our tests be useful?
Here’s some code that might correctly calculate the area of overlap:
In [10]: def overlap(field1, field2):

left1, bottom1, top1, right1 = field1
overlap_left=max(left1, left2)
overlap_bottom=max(bottom1, bottom2)
overlap_right=min(right1, right2)
overlap_top=min(top1, top2)
overlap_height=(overlap_top-overlap_bottom)
overlap_width=(overlap_right-overlap_left)
return overlap_height*overlap_width
So how do we check our code?

The manual approach would be to look at some cases, and, once, run it and check:
In [11]: overlap((1.,1.,4.,4.),(2.,2.,3.,3.))
Out[11]: 1.0
That looks OK.

But we can do better, we can write code which raises an error if it gets an unexpected answer:
In [12]: assert overlap((1.,1.,4.,4.),(2.,2.,3.,3.)) == 1.0
In [13]: assert overlap((1.,1.,4.,4.),(2.,2.,3.,4.5)) == 2.0
210
In [14]: assert overlap((1.,1.,4.,4.),(2.,2.,4.5,4.5)) == 4.0
In [15]: assert overlap((1.,1.,4.,4.),(4.5,4.5,5,5)) == 0.0
---------------------------------------------------------------------------
AssertionError Traceback (most recent call last)
<ipython-input-15-9b6bffd116ce> in <module>()
----> 1 assert overlap((1.,1.,4.,4.),(4.5,4.5,5,5)) == 0.0
AssertionError:
In [16]: print overlap((1.,1.,4.,4.),(4.5,4.5,5,5))
0.25
What? Why is this wrong?

In our calculation, we are actually getting:
In [17]: overlap_left=4.5
overlap_right=4
overlap_width=-0.5
overlap_height=-0.5
Both width and height are negative, resulting in a positive area. The above code didn’t take into account
the non-overlap correctly.
It should be:
In [18]: def overlap(field1, field2):

overlap_height=max(0, (overlap_top-overlap_bottom))
overlap_width=max(0, (overlap_right-overlap_left))
In [19]: assert overlap((1,1,4,4),(2,2,3,3)) == 1.0

assert overlap((1,1,4,4),(2,2,3,4.5)) == 2.0
assert overlap((1,1,4,4),(2,2,4.5,4.5)) == 4.0
assert overlap((1,1,4,4),(4.5,4.5,5,5)) == 0.0
assert overlap((1,1,4,4),(2.5,4,3.5,4.5)) == 0.0
assert overlap((1,1,4,4),(4,4,4.5,4.5)) == 0.0
Note, we reran our other tests, to check our fix didn’t break something else. (We call that “fallout”)
211
24.1.1 Boundary cases
“Boundary cases” are an important area to test:
• Limit between two equivalence classes: edge and corner sharing fields
• Wherever indices appear, check values at 0, N, N+1
• Empty arrays:
atoms = [read_input_atom(input_atom) for input_atom in input_file]

energy = force_field(atoms)
• What happens if atoms is an empty list?

• What happens when a matrix/data-frame reaches one row, or one column?
24.2 Positive and negative tests

• Positive tests: code should give correct answer with various inputs
• Negative tests: code should crash as expected given invalid inputs, rather than lying
Bad input should be expected and should fail early and explicitly.
Testing should ensure that explicit failures do indeed happen.
24.3 Raising exceptions

In Python, we can signal an error state by raising an error:
In [20]: def I_only_accept_positive_numbers(number):

# Check input
if number < 0:
raise ValueError("Input "+ str(number)+" is negative")
# Do something
In [21]: I_only_accept_positive_numbers(5)
In [22]: I_only_accept_positive_numbers(-5)
---------------------------------------------------------------------------
<ipython-input-22-e283d4657e88> in <module>()
----> 1 I only accept positive numbers(-5)
<ipython-input-20-6e472e931020> in I only accept positive numbers(number)

2 # Check input
3 if number < 0:
----> 4 raise ValueError("Input "+ str(number)+" is negative")
5
6 # Do something
ValueError: Input -5 is negative
212
There are standard “Exception” types, like ValueError we can raise
We would like to be able to write tests like this:
In [23]: assert I_only_accept_positive_numbers(-5) == # Gives a value error
File "<ipython-input-23-af84655f8e47>", line 1

assert I only accept positive numbers(-5) == # Gives a value error
^
But to do that, we need to learn about more sophisticated testing tools, called “test frameworks”.
24.4 Testing frameworks

24.4.1 Why use testing frameworks?
Frameworks should simplify our lives:
• Should be easy to add simple test

• Should be possible to create complex test:
– Fixtures
– Setup/Tear down
– Parameterized tests (same test, mostly same input)
• Find all our tests in a complicated code-base
• Run all our tests with a quick command
• Run only some tests, e.g. test --only "tests about fields
• Report failing tests
• Additional goodies, such as code coverage
24.4.2 Common testing frameworks

• Language agnostic: CTest
• Test runner for executables, bash scripts, etc. . .
• Great for legacy code hardening
• C unit-tests:
– all c++ frameworks,

– Check,
– CUnit
• C++ unit-tests:
– CppTest,
– Boost::Test,
– google-test,
– Catch (best)
• Python unit-tests:
– nose includes test discovery, coverage, etc
213
– unittest comes with standard python library
– py.test, branched off of nose
• R unit-tests:
– RUnit,
– svUnit
– (works with SciViews GUI)
• Fortran unit-tests:
– funit,
– pfunit(works with MPI)
24.4.3 Nose framework: usage

nose is a python testing framework.
We can use its tools in the notebook for on-the-fly tests in the notebook. This, happily, includes the
negative-tests example we were looking for a moment ago.
In [24]: from nose.tools import assert_raises
In [25]: with assert_raises(ValueError):

assert I_only_accept_positive_numbers(-5)
but the real power comes when we write a test file alongside our code files in our homemade packages:
In [26]: %%bash
mkdir -p saskatchewan
touch saskatchewan/__init__.py
In [27]: %%writefile saskatchewan/overlap.py

def overlap(field1, field2):
# Here’s our wrong code again
Writing saskatchewan/overlap.py
In [28]: %%writefile saskatchewan/test_overlap.py

from overlap import overlap
from nose.tools import assert_equal
def test_full_overlap():
assert_equal(overlap((1.,1.,4.,4.),(2.,2.,3.,3.)), 1.0)
def test_partial_overlap():
assert_equal(overlap((1,1,4,4),(2,2,3,4.5)), 2.0)
214
def test_no_overlap():
assert_equal(overlap((1,1,4,4),(4.5,4.5,5,5)), 0.0)
Writing saskatchewan/test overlap.py
In [29]: %%bash
cd saskatchewan
nosetests
..F
======================================================================
FAIL: saskatchewan.test overlap.test no overlap
----------------------------------------------------------------------
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/saskatchewan/test o
assert equal(overlap((1,1,4,4),(4.5,4.5,5,5)), 0.0)
AssertionError: 0.25 != 0.0
----------------------------------------------------------------------
Ran 3 tests in 0.012s
FAILED (failures=1)
Note that it reported which test had failed, how many tests ran, and how many failed.
The symbol ..F means there were three tests, of which the third one failed.
Nose will:
• automagically finds files test *.py

• collects all subroutines called test *
• runs tests and reports results
Some options:
• help: nosetests --help

• test only a given file: nosetests test file.py
• compute coverage: nosetests --with-coverage
24.5 Testing with floating points

24.5.1 Floating points are not reals
Floating points are inaccurate representations of real numbers:
1.0 == 0.99999999999999999 is true to the last bit.
This can lead to numerical errors during calculations: 1000(a − b) 6= 1000a − 1000b
In [30]: >>> 1000.0 * 1.0 - 1000.0 * 0.9999999999999998

2.2737367544323206e-13
>>> 1000.0 * (1.0 - 0.9999999999999998)

2.220446049250313e-13
Out[30]: 2.220446049250313e-13
215
Both results are wrong: 2e-13 is the correct answer.
The size of the error will depend on the magnitude of the floating points:
In [31]: >>> 1000.0 * 1e5 - 1000.0 * 0.9999999999999998e5

1.4901161193847656e-08
Out[31]: 1.4901161193847656e-08
The result should be 2e-8.
24.5.2 Comparing floating points

Comparison can be absolute:
In [32]: from nose.tools import assert_almost_equal

assert_almost_equal( 0.7, 0.7 + 1e-6, delta = 1e-5)
Or relative:

magnitude = 0.7
assert_almost_equal(0.7, 0.7 + 1e-5, delta = magnitude * 1e-5)
---------------------------------------------------------------------------
<ipython-input-33-192d51bb43fc> in <module>()
1 from nose.tools import assert almost equal
2 magnitude = 0.7
----> 3 assert almost equal(0.7, 0.7 + 1e-5, delta = magnitude * 1e-5)
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittes
561 places)
562 msg = self. formatMessage(msg, standardMsg)
--> 563 raise self.failureException(msg)
564
565 def assertNotAlmostEqual(self, first, second, places=None, msg=None, delta=None):
AssertionError: 0.7 != 0.7000099999999999 within 7e-06 delta
Where magnitude should be chosen based on the intrinsic scale of the calculations.
For instance, if calculations is a result of differences between large numbers:
In [34]: (1e15 + 1.4) - (1e15 + 0.7)
Out[34]: 0.625
then magnitude = 1e15 is reasonable.

However, the best choice of scale for comparison in scientific floating point testing is an active area of
research.
216
24.5.3 Comparing vectors of floating points
Numerical vectors are best represented using numpy.
In [35]: from numpy import array, pi
vector_of_reals = array([0.1, 0.2, 0.3, 0.4]) * pi
Numpy ships with a number of assertions (in numpy.testing) to make comparison easy:

from numpy.testing import assert_allclose
expected = array([0.1, 0.2, 0.3, 0.4, 1e-12]) * pi
actual = array([0.1, 0.2, 0.3, 0.4, 2e-12]) * pi
actual[:-1] += 1e-6
assert_allclose(actual, expected, rtol=1e-5, atol=1e-8)
It compares the difference between actual and expected to atol + rtol * abs(expected).
24.6 Classroom exercise: energy calculation

24.6.1 Diffusion model in 1D
Description: A one-dimensional diffusion model. (Could be a gas of particles, or a bunch of crowded people
in a corridor, or animals in a valley habitat. . . )
• Agents are on a 1d axis

• Agents do not want to be where there are other agents
Implementation:
• Given a vector n of positive integers,

P and of arbitrary length
• Compute the energy, E(n) = i ni (ni − 1)
• Later, we will have the likelyhood of an agent moving depend on the change in energy.

density = np.array([0, 0, 3, 5, 8, 4, 2, 1])
fig, ax = plt.subplots()
ax.bar(np.arange(len(density))-0.5, density)
ax.xrange=[-0.5, len(density)-0.5]
ax.set_ylabel("Particle count $n_i$")
ax.set_xlabel("Position $i$")
Out[37]: <matplotlib.text.Text at 0x10e4c4250>
217
Here, the total energy due to position 2 is 3(3 − 1) = 6, and due to column 7 is 1(1 − 1) = 0. We need
to sum these to get the total energy.
24.6.2 Starting point

Create a Python module:
In [38]: %%bash
mkdir -p diffusion
touch diffusion/__init__.py
• Implementation file: diffusion model.py
In [39]: %%writefile diffusion/model.py

def energy(density, coeff=1.0):
""" Energy associated with the diffusion model
Parameters
----------
density: array of positive integers

Number of particles at each position i in the array
coeff: float
Diffusion coefficient.
"""
# implementation goes here
Writing diffusion/model.py
• Testing file: test diffusion model.py
218
In [40]: %%writefile diffusion/test_model.py
from model import energy
def test_energy():
""" Optional description for nose reporting """
# Test something
Writing diffusion/test model.py
Invoke the tests:
In [41]: %%bash
cd diffusion
nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Now, write your code (in model.py), and tests (in test model.py), testing as you do.
24.6.3 Solution
Don’t look until after class!

""" Simplistic 1-dimensional diffusion model """
def energy(density):
:Parameters:
Number of particles at each position i in the array/geometry
"""
from numpy import array, any, sum
# Make sure input is an numpy array

density = array(density)
# ...of the right kind (integer). Unless it is zero length, in which case type does not matte
if density.dtype.kind != ’i’ and len(density) > 0:
raise TypeError("Density should be a array of *integers*.")
# and the right values (positive or null)
if any(density < 0):
raise ValueError("Density should be an array of *positive* integers.")
if density.ndim != 1:
raise ValueError("Density should be an a *1-dimensional* array of positive integers.")
return sum(density * (density - 1))
Overwriting diffusion/model.py

""" Unit tests for a diffusion model """
219
from nose.tools import assert_raises, assert_almost_equal
def test_energy_fails_on_non_integer_density():
with assert_raises(TypeError) as exception:
energy([1.0, 2, 3])
def test_energy_fails_on_negative_density():
with assert_raises(ValueError) as exception: energy([-1, 2, 3])
def test_energy_fails_ndimensional_density():
with assert_raises(ValueError) as exception: energy([[1, 2, 3], [3, 4, 5]])
def test_zero_energy_cases():
# Zero energy at zero density
densities = [ [], [0], [0, 0, 0] ]
for density in densities:
assert_almost_equal(energy(density), 0)
def test_derivative():
from numpy.random import randint
# Loop over vectors of different sizes (but not empty)

for vector_size in randint(1, 1000, size=30):
# Create random density of size N

density = randint(50, size=vector_size)
# will do derivative at this index

element_index = randint(vector_size)
# modified densities
density_plus_one = density.copy()
density_plus_one[element_index] += 1
# Compute and check result

# d(n^2-1)/dn = 2n
expected = 2.0*density[element_index] if density[element_index] > 0 else 0
actual = energy(density_plus_one) - energy(density)
assert_almost_equal(expected, actual)
def test_derivative_no_self_energy():
""" If particle is alone, then its participation to energy is zero """
from numpy import array
density = array([1, 0, 1, 10, 15, 0])

density[1] += 1
expected = 0
Overwriting diffusion/test model.py
220
In [44]: %%bash
cd diffusion
nosetests
...
----------------------------------------------------------------------
OK
24.6.4 Coverage
1. Comment out from exception tests in solution
2. in solution directory, run
In [45]: %%bash
cd diffusion
nosetests --with-coverage --cover-package=diffusion.model -v --cover-html
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module

diffusion.test model.test energy fails on non integer density ... ok
diffusion.test model.test energy fails on negative density ... ok
diffusion.test model.test energy fails ndimensional density ... ok
diffusion.test model.test zero energy cases ... ok
diffusion.test model.test derivative ... ok
If particle is alone, then its participation to energy is zero ... ok
----------------------------------------------------------------------
OK
Look at the coverage results
24.7 Mocking
24.7.1 Definition
Mock: verb,
1. to tease or laugh at in a scornful or contemptuous manner

2. to make a replica or imitation of something
Mocking
• computer science, to simulate the behaviour of real objects in controlled ways.
Stub routine
• A routine that a simulate a more computationally expensive routine, without actually performing any
calculation. Strictly speaking, the term Mocking is reserved for object-oriented approaches
24.7.2 Mocking frameworks

• C: CMocka
• C++: googlemock
• Python: Mock (unittest.mock in python 3.3), doublex
221
24.7.3 Recording calls with mock
Mock objects record the calls made to them:
In [46]: from mock import Mock
function = Mock(name="myroutine", return_value=2)
function(1)
function(5, "hello", a=True)
function.mock_calls
Out[46]: [call(1), call(5, ’hello’, a=True)]
The arguments of each call can be recovered
In [47]: name, args, kwargs = function.mock_calls[1]
args, kwargs
Out[47]: ((5, ’hello’), {’a’: True})
Mock objects can return different values for each call
In [48]: function = Mock(name="myroutine", side_effect=[2, "xyz"])
In [49]: function(1)
Out[49]: 2
In [50]: function(1, "hello", {’a’: True})
Out[50]: ’xyz’
In [51]: function()
---------------------------------------------------------------------------
StopIteration Traceback (most recent call last)
<ipython-input-51-2fcbbbc1fe81> in <module>()
----> 1 function()
/usr/local/lib/python2.7/site-packages/mock.pyc in call ( mock self, *args, **kwargs)

953 # in the signature
954 mock self. mock check sig(*args, **kwargs)
--> 955 return mock self. mock call(*args, **kwargs)
956
957
/usr/local/lib/python2.7/site-packages/mock.pyc in mock call( mock self, *args, **kwargs)

1011
1012 if not callable(effect):
-> 1013 result = next(effect)
1014 if is exception(result):
1015 raise result
StopIteration:
222
24.8 Using mocks to model test resources
Often we want to write tests for code which interacts with remote resources. (E.g. databases, the internet,
or data files.)
We don’t want to have our tests actually interact with the remote resource, as this would mean our tests
failed due to lost internet connections, for example.
Instead, we can use mocks to assert that our code does the right thing in terms of the messages it sends:
the parameters of the function calls it makes to the remote resource.
For example, consider the following code that downloads a map from the internet:
def map_at(lat,long, satellite=False, zoom=12,

size=(400,400), sensor=False):
params=dict(
zoom= zoom,
style="feature:all|element:labels|visibility:off")
if satellite:
In [53]: london_map=map_at(51.5073509, -0.1277583)

import IPython

IPython.core.display.Image(london_map.content)
Out[54]:
223
We would like to test that it is building the parameters correctly. We can do this by mocking the
requests object. We need to temporarily replace a method in the library with a mock. We can use “patch”
to do this:
In [55]: from mock import patch

with patch.object(requests,’get’) as mock_get:
london_map=map_at(51.5073509, -0.1277583)
print mock_get.mock_calls
[call(’http://maps.googleapis.com/maps/api/staticmap?’, params={’style’: ’feature:all|element:labels|vis
Our tests then look like:
In [56]: def test_build_default_params():

default_map=map_at(51.0, 0.0)
mock_get.assert_called_with(
"http://maps.googleapis.com/maps/api/staticmap?",
params={
224
’sensor’:’false’,
’zoom’:12,
’size’:’400x400’,
’center’:’51.0,0.0’,
’style’:’feature:all|element:labels|visibility:off’
}
)
test_build_default_params()
That was quiet, so it passed. When I’m writing tests, I usually modify one of the expectations, to
something ‘wrong’, just to check it’s not passing “by accident”, run the tests, then change it back!
24.8.1 Testing functions that call other functions

In [57]: def partial_derivative(function, at, direction, delta=1.0):
f_x=function(at)
x_plus_delta=at[:]
x_plus_delta[direction]+=delta
f_x_plus_delta=function(x_plus_delta)
return (f_x_plus_delta-f_x)/delta
We want to test that the above function does the right thing. It is supposed to compute the derivative
of a function of a vector in a particular direction.
E.g.:
In [58]: partial_derivative(sum, [0,0,0], 1)
Out[58]: 1.0
How do we assert that it is doing the right thing? With tests like this:
In [59]: from mock import MagicMock
def test_derivative_2d_y_direction():
func=MagicMock()
partial_derivative(func, [0,0], 1)
func.assert_any_call([0, 1.0])
func.assert_any_call([0, 0])
test_derivative_2d_y_direction()
We made our mock a “Magic Mock” because otherwise, the mock results f x plus delta and f x can’t
be subtracted:
In [60]: MagicMock()-MagicMock()
Out[60]: <MagicMock name=’mock. sub ()’ id=’4528870864’>
In [61]: Mock()-Mock()
---------------------------------------------------------------------------
<ipython-input-61-fca1dbe33378> in <module>()
225
----> 1 Mock()-Mock()
TypeError: unsupported operand type(s) for -: ’Mock’ and ’Mock’
24.9 Using a debugger

24.9.1 Stepping through the code
Debuggers are programs that can be used to test other programs. They allow programmers to suspend
execution of the target program and inspect variables at that point.
• Mac - compiled languages: Xcode

• Windows - compiled languages: Visual Studio
• Linux: DDD
• all platforms: eclipse, gdb (DDD and eclipse are GUIs for gdb)
• python: spyder, pdb
• R: RStudio, debug, browser
24.9.2 Using the python debugger

Unfortunately this doesn’t work nicely in the notebook. But from the command line, you can run a python
program with:
python -m pdb my_program.py
24.9.3 Basic navigation:

Basic command to navigate the code and the python debugger:
• help: prints the help

• help n: prints help about command n
• n(ext): executes one line of code. Executes and steps over functions.
• s(tep): step into current function in line of code
• l(ist): list program around current position
• w(where): prints current stack (where we are in code)
• [enter]: repeats last command
• anypythonvariable: print the value of that variable
The python debugger is a python shell: it can print and compute values, and even change the values
of the variables at that point in the program.
24.9.4 Breakpoints
Break points tell debugger where and when to stop We say * b somefunctionname
In [62]: %%writefile energy_example.py

from diffusion.model import energy
print energy([5, 6, 7, 8, 0, 1])
Writing energy example.py
The debugger is, of course, most used interactively, but here I’m showing a prewritten debugger script:
226
In [63]: %%writefile commands
restart # restart session
n
b energy # program will stop when entering energy
c # continue program until break point is reached
print density # We are now "inside" the energy function and can print any variable.
Writing commands
In [64]: %%bash
python -m pdb energy_example.py < commands
> /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/energy example.py(1)<modu

-> from diffusion.model import energy
(Pdb) Restarting energy example.py with arguments:
# restart session
(Pdb) > /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/energy example.py(2
-> print energy([5, 6, 7, 8, 0, 1])
(Pdb) Breakpoint 1 at /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffu
(Pdb) > /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffusion/model.py(
-> from numpy import array, any, sum
(Pdb) [5, 6, 7, 8, 0, 1]
(Pdb)
Alternatively, break-points can be set on files: b file.py:20 will stop on line 20 of file.py.
24.9.5 Post-mortem
Debugging when something goes wrong:
1. Have a crash somewhere in the code

2. run python -m pdb file.py or run the cell with %pdb on
The program should stop where the exception was raised
1. use w and l for position in code and in call stack

2. use up and down to navigate up and down the call stack
3. inspect variables along the way to understand failure
This does work in the notebook.
In [65]: %pdb on
partial_derivative(energy,[5,6,7,8,0,1],5)
Automatic pdb calling has been turned ON
---------------------------------------------------------------------------
<ipython-input-65-cd06eca5c4a7> in <module>()
1 get ipython().magic(u’pdb on’)
2 from diffusion.model import energy
227
----> 3 partial derivative(energy,[5,6,7,8,0,1],5)
<ipython-input-57-aca6cdaead54> in partial derivative(function, at, direction, delta)

3 x plus delta=at[:]
4 x plus delta[direction]+=delta
----> 5 f x plus delta=function(x plus delta)
6 return (f x plus delta-f x)/delta
/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffusion/model.pyc
14 # ...of the right kind (integer). Unless it is zero length, in which case type does not ma
15 if density.dtype.kind != ’i’ and len(density) > 0:
---> 16 raise TypeError("Density should be a array of *integers*.")
17 # and the right values (positive or null)
18 if any(density < 0):
TypeError: Density should be a array of *integers*.
> /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffusion/model.py(16)ene
15 if density.dtype.kind != ’i’ and len(density) > 0:
---> 16 raise TypeError("Density should be a array of *integers*.")
17 # and the right values (positive or null)
---------------------------------------------------------------------------
StdinNotImplementedError Traceback (most recent call last)
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run code(self, code

3050 if result is not None:
3051 result.error in exec = sys.exc info()[1]
-> 3052 self.showtraceback()
3053 else:
3054 outflag = 0
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self,
1854 if self.call pdb:
1855 # drop into debugger
-> 1856 self.debugger(force=True)
1857 return
1858
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in debugger(self, force

1020
1021 with self.readline no record:
-> 1022 pm()
1023
1024 #-------------------------------------------------------------------------
228
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in <lambda>()
1017 else:
1018 # fallback to our internal debugger
-> 1019 pm = lambda : self.InteractiveTB.debugger(force=True)
1020
/usr/local/lib/python2.7/site-packages/IPython/core/ultratb.pyc in debugger(self, force)

1073 etb = etb.tb next
1074 self.pdb.botframe = etb.tb frame
-> 1075 self.pdb.interaction(self.tb.tb frame, self.tb)
1076
1077 if hasattr(self, ’tb’):
/usr/local/lib/python2.7/site-packages/IPython/core/debugger.pyc in interaction(self, frame, tra

276 while True:
277 try:
--> 278 OldPdb.interaction(self, frame, traceback)
279 except KeyboardInterrupt:
280 self.shell.write(’\n’ + self.shell.get exception only())
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.pyc
208 self.setup(frame, traceback)
209 self.print stack entry(self.stack[self.curindex])
--> 210 self.cmdloop()
211 self.forget()
212
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cmd.pyc
128 if self.use rawinput:
129 try:
--> 130 line = raw input(self.prompt)
131 except EOFError:
132 line = ’EOF’
/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.pyc in raw input(self, prom

644 if not self. allow stdin:
645 raise StdinNotImplementedError(
--> 646 "raw input was called, but this frontend does not support input requests."
647 )
648 return self. input request(prompt,
StdinNotImplementedError: raw input was called, but this frontend does not support input request
> /usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.py(646)raw input()

645 raise StdinNotImplementedError(--> 646 "raw input was called, but th
647 )
229
File "/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/ipkernel.py", line 181, in do

shell.run cell(code, store history=store history, silent=silent)
File "/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.py", line 2871, in

interactivity=interactivity, compiler=compiler, result=result)

self.showtraceback()

self.debugger(force=True)

pm()

pm = lambda : self.InteractiveTB.debugger(force=True)
File "/usr/local/lib/python2.7/site-packages/IPython/core/ultratb.py", line 1075, in debugger

self.pdb.interaction(self.tb.tb frame, self.tb)
File "/usr/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 278, in interacti

OldPdb.interaction(self, frame, traceback)
File "/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7

self.cmdloop()

line = raw input(self.prompt)
File "/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.py", line 646, in r

"raw input was called, but this frontend does not support input requests."
230
24.10 Jenkins
24.10.1 Test servers
Goal:
1. run tests nightly

2. run tests after each commit to github (or other)
3. run tests on different platforms
Various groups run servers that can be used to do this automatically.

RITS run a university-wide one.
24.10.2 Memory and profiling

For compiled languages (C, C++, Fortran): * Checking for memory leaks with valgrind: valgrind
--leak-check=full program * Checking cache hits and cache misses with cachegrind: valgrind
--tool=cachegrind program * Profiling the code with callgrind: valgrind --tool=callgrind program
• Python: profile with runsnake

• R: Rprof
24.11 Extended TDD Example: Monte-Carlo

24.11.1 Problem: Implement and test a simple Monte-Carlo algorithm
Given an input function (energy) and starting point (density) and a temperature T :
1. Compute energy at current density.

2. Move randomly chosen agent randomly left or right.
3. Compute second energy.
4. Compare the two energies:
5. If second energy is lower, accept move.
6. β is a parameter which determines how likely the simulation is to move from a ‘less favourable’ situation
to a ‘more favourable’ one.
7. Compute P0 = e−β(E1 −E0 ) and P1 a random number between 0 and 1,
8. If P0 > P1 , do the move anyway.
9. Repeat.
• the algorithm should work for (m)any energy function(s).

• there should be separate tests for separate steps! What constitutes a step?
• tests for the Monte-Carlo should not depend on other parts of code.
• Use matplotlib to plot density at each iteration, and make an animation
24.12 Solution
We need to break our problem down into pieces:
1. A function to generate a random change (random agent, random direction)

2.
231
24.13 Testing frameworks
24.13.1 Why use testing frameworks?
Frameworks should simplify our lives:
• Should be easy to add simple test

• Should be possible to create complex test:
– Fixtures
– Setup/Tear down
– Parameterized tests (same test, mostly same input)
• Find all our tests in a complicated code-base
• Run all our tests with a quick command
• Run only some tests, e.g. test --only "tests about fields
• Report failing tests
• Additional goodies, such as code coverage
24.13.2 Common testing frameworks

• Language agnostic: CTest
• Test runner for executables, bash scripts, etc. . .
• Great for legacy code hardening
• C unit-tests:
– all c++ frameworks,

– Check,
– CUnit
• C++ unit-tests:
– CppTest,
– Boost::Test,
– google-test,
– Catch (best)
• Python unit-tests:
– nose includes test discovery, coverage, etc

– unittest comes with standard python library
– py.test, branched off of nose
• R unit-tests:
– RUnit,
– svUnit
– (works with SciViews GUI)
• Fortran unit-tests:
– funit,
– pfunit(works with MPI)
232
24.13.3 Nose framework: usage
nose is a python testing framework.
We can use its tools in the notebook for on-the-fly tests in the notebook. This, happily, includes the
negative-tests example we were looking for a moment ago.
In [2]: with assert_raises(ValueError):

assert I_only_accept_positive_numbers(-5)
---------------------------------------------------------------------------
<ipython-input-2-ffe48ded21bf> in <module>()
1 with assert raises(ValueError):
----> 2 assert I only accept positive numbers(-5)
NameError: name ’I only accept positive numbers’ is not defined
but the real power comes when we write a test file alongside our code files in our homemade packages:
In [3]: %%bash
mkdir -p saskatchewan
touch saskatchewan/__init__.py
In [4]: %%writefile saskatchewan/overlap.py

def overlap(field1, field2):
# Here’s our wrong code again
Overwriting saskatchewan/overlap.py
In [5]: %%writefile saskatchewan/test_overlap.py

from overlap import overlap
def test_full_overlap():
assert_equal(overlap((1.,1.,4.,4.),(2.,2.,3.,3.)), 1.0)
def test_partial_overlap():
assert_equal(overlap((1,1,4,4),(2,2,3,4.5)), 2.0)
233
def test_no_overlap():
assert_equal(overlap((1,1,4,4),(4.5,4.5,5,5)), 0.0)
Overwriting saskatchewan/test overlap.py
In [6]: %%bash
cd saskatchewan
nosetests
..F
======================================================================
FAIL: saskatchewan.test overlap.test no overlap
----------------------------------------------------------------------
File "/usr/local/lib/python2.7/site-packages/nose/case.py", line 197, in runTest
self.test(*self.arg)
File "/Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/saskatchewan/test o
assert equal(overlap((1,1,4,4),(4.5,4.5,5,5)), 0.0)
AssertionError: 0.25 != 0.0
----------------------------------------------------------------------
FAILED (failures=1)
Note that it reported which test had failed, how many tests ran, and how many failed.
The symbol ..F means there were three tests, of which the third one failed.
Nose will:
• automagically finds files test *.py

• collects all subroutines called test *
• runs tests and reports results
Some options:
• help: nosetests --help

• test only a given file: nosetests test file.py
• compute coverage: nosetests --with-coverage
24.14 Testing with floating points

24.14.1 Floating points are not reals
Floating points are inaccurate representations of real numbers:
1.0 == 0.99999999999999999 is true to the last bit.
This can lead to numerical errors during calculations: 1000(a − b) 6= 1000a − 1000b
In [7]: >>> 1000.0 * 1.0 - 1000.0 * 0.9999999999999998

2.2737367544323206e-13
>>> 1000.0 * (1.0 - 0.9999999999999998)

2.220446049250313e-13
Out[7]: 2.220446049250313e-13
234
Both results are wrong: 2e-13 is the correct answer.
The size of the error will depend on the magnitude of the floating points:
In [8]: >>> 1000.0 * 1e5 - 1000.0 * 0.9999999999999998e5

1.4901161193847656e-08
Out[8]: 1.4901161193847656e-08
The result should be 2e-8.
24.14.2 Comparing floating points

Comparison can be absolute:

assert_almost_equal( 0.7, 0.7 + 1e-6, delta = 1e-5)
Or relative:

magnitude = 0.7
assert_almost_equal(0.7, 0.7 + 1e-5, delta = magnitude * 1e-5)
---------------------------------------------------------------------------
<ipython-input-10-192d51bb43fc> in <module>()
1 from nose.tools import assert almost equal
2 magnitude = 0.7
----> 3 assert almost equal(0.7, 0.7 + 1e-5, delta = magnitude * 1e-5)
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/unittes
561 places)
562 msg = self. formatMessage(msg, standardMsg)
--> 563 raise self.failureException(msg)
564
565 def assertNotAlmostEqual(self, first, second, places=None, msg=None, delta=None):
AssertionError: 0.7 != 0.7000099999999999 within 7e-06 delta
Where magnitude should be chosen based on the intrinsic scale of the calculations.
For instance, if calculations is a result of differences between large numbers:
In [11]: (1e15 + 1.4) - (1e15 + 0.7)
Out[11]: 0.625
then magnitude = 1e15 is reasonable.

However, the best choice of scale for comparison in scientific floating point testing is an active area of
research.
235
24.14.3 Comparing vectors of floating points
Numerical vectors are best represented using numpy.
vector_of_reals = array([0.1, 0.2, 0.3, 0.4]) * pi
Numpy ships with a number of assertions (in numpy.testing) to make comparison easy:

from numpy.testing import assert_allclose
expected = array([0.1, 0.2, 0.3, 0.4, 1e-12]) * pi
actual = array([0.1, 0.2, 0.3, 0.4, 2e-12]) * pi
actual[:-1] += 1e-6
assert_allclose(actual, expected, rtol=1e-5, atol=1e-8)
24.15 Classroom exercise: energy calculation

24.15.1 Diffusion model in 1D
Description: A one-dimensional diffusion model. (Could be a gas of particles, or a bunch of crowded people
in a corridor, or animals in a valley habitat. . . )
• Agents are on a 1d axis

• Agents do not want to be where there are other agents
Implementation:
• Given a vector n of positive integers,

P and of arbitrary length
• Compute the energy, E(n) = i ni (ni − 1)
• Later, we will have the likelyhood of an agent moving depend on the change in energy.

density = np.array([0, 0, 3, 5, 8, 4, 2, 1])
fig, ax = plt.subplots()
ax.bar(np.arange(len(density))-0.5, density)
ax.xrange=[-0.5, len(density)-0.5]
ax.set_ylabel("Particle count $n_i$")
ax.set_xlabel("Position $i$")
---------------------------------------------------------------------------
<ipython-input-1-b05039655391> in <module>()
1 import numpy as np
2 density = np.array([0, 0, 3, 5, 8, 4, 2, 1])
----> 3 fig, ax = plt.subplots()
4 ax.bar(np.arange(len(density))-0.5, density)
5 ax.xrange=[-0.5, len(density)-0.5]
NameError: name ’plt’ is not defined
236
Here, the total energy due to position 2 is 3(3 − 1) = 6, and due to column 7 is 1(1 − 1) = 0. We need
to sum these to get the total energy.
24.15.2 Starting point

Create a Python module:
In [2]: %%bash
mkdir -p diffusion
touch diffusion/__init__.py
• Implementation file: diffusion model.py

def energy(density, coeff=1.0):
Parameters
----------

Number of particles at each position i in the array
coeff: float
Diffusion coefficient.
"""
# implementation goes here
• Testing file: test diffusion model.py

def test_energy():
""" Optional description for nose reporting """
# Test something
Invoke the tests:
In [5]: %%bash
cd diffusion
nosetests
.
----------------------------------------------------------------------
Ran 1 test in 0.000s
OK
Now, write your code (in model.py), and tests (in test model.py), testing as you do.
237
24.15.3 Solution
Don’t look until after class!

""" Simplistic 1-dimensional diffusion model """
def energy(density):
:Parameters:
"""
# Make sure input is an numpy array

# ...of the right kind (integer). Unless it is zero length, in which case type does not matter
raise TypeError("Density should be a array of *integers*.")
return sum(density * (density - 1))

""" Unit tests for a diffusion model """
from nose.tools import assert_raises, assert_almost_equal

def test_energy_fails_on_non_integer_density():
with assert_raises(TypeError) as exception:
energy([1.0, 2, 3])
def test_energy_fails_on_negative_density():
with assert_raises(ValueError) as exception: energy([-1, 2, 3])
def test_energy_fails_ndimensional_density():
with assert_raises(ValueError) as exception: energy([[1, 2, 3], [3, 4, 5]])
def test_zero_energy_cases():
# Zero energy at zero density
densities = [ [], [0], [0, 0, 0] ]
for density in densities:
assert_almost_equal(energy(density), 0)
def test_derivative():
238
# Loop over vectors of different sizes (but not empty)
for vector_size in randint(1, 1000, size=30):
# Create random density of size N

density = randint(50, size=vector_size)
# will do derivative at this index

element_index = randint(vector_size)
# modified densities
density_plus_one[element_index] += 1
# Compute and check result

# d(n^2-1)/dn = 2n
expected = 2.0*density[element_index] if density[element_index] > 0 else 0
def test_derivative_no_self_energy():
""" If particle is alone, then its participation to energy is zero """
from numpy import array
density = array([1, 0, 1, 10, 15, 0])

density[1] += 1
expected = 0
In [8]: %%bash
cd diffusion
nosetests
...
----------------------------------------------------------------------
OK
24.15.4 Coverage
1. Comment out from exception tests in solution
2. in solution directory, run
In [9]: %%bash
cd diffusion
nosetests --with-coverage --cover-package=diffusion.model -v --cover-html
nose.plugins.cover: ERROR: Coverage not available: unable to import coverage module

diffusion.test model.test energy fails on non integer density ... ok
239
diffusion.test model.test energy fails on negative density ... ok
diffusion.test model.test energy fails ndimensional density ... ok
diffusion.test model.test zero energy cases ... ok
diffusion.test model.test derivative ... ok
If particle is alone, then its participation to energy is zero ... ok
----------------------------------------------------------------------
OK
Look at the coverage results
24.16 Mocking
24.16.1 Definition
Mock: verb,
1. to tease or laugh at in a scornful or contemptuous manner

2. to make a replica or imitation of something
Mocking
• computer science, to simulate the behaviour of real objects in controlled ways.
Stub routine
• A routine that a simulate a more computationally expensive routine, without actually performing any
calculation. Strictly speaking, the term Mocking is reserved for object-oriented approaches
24.16.2 Mocking frameworks

• C: CMocka
• C++: googlemock
• Python: Mock (unittest.mock in python 3.3), doublex
24.16.3 Recording calls with mock

Mock objects record the calls made to them:

function = Mock(name="myroutine", return_value=2)
function(1)
function(5, "hello", a=True)
function.mock_calls
Out[1]: [call(1), call(5, ’hello’, a=True)]
The arguments of each call can be recovered
In [2]: name, args, kwargs = function.mock_calls[1]

args, kwargs
Out[2]: ((5, ’hello’), {’a’: True})
Mock objects can return different values for each call
240
In [3]: function = Mock(name="myroutine", side_effect=[2, "xyz"])
In [4]: function(1)
Out[4]: 2
In [5]: function(1, "hello", {’a’: True})
Out[5]: ’xyz’
In [6]: function()
---------------------------------------------------------------------------
<ipython-input-6-2fcbbbc1fe81> in <module>()
----> 1 function()
/usr/local/lib/python2.7/site-packages/mock.pyc in call ( mock self, *args, **kwargs)

953 # in the signature
954 mock self. mock check sig(*args, **kwargs)
--> 955 return mock self. mock call(*args, **kwargs)
956
957
/usr/local/lib/python2.7/site-packages/mock.pyc in mock call( mock self, *args, **kwargs)

1011
1012 if not callable(effect):
-> 1013 result = next(effect)
1014 if is exception(result):
1015 raise result
StopIteration:
24.17 Using mocks to model test resources

Often we want to write tests for code which interacts with remote resources. (E.g. databases, the internet,
or data files.)
We don’t want to have our tests actually interact with the remote resource, as this would mean our tests
failed due to lost internet connections, for example.
Instead, we can use mocks to assert that our code does the right thing in terms of the messages it sends:
the parameters of the function calls it makes to the remote resource.
For example, consider the following code that downloads a map from the internet:
def map_at(lat,long, satellite=False, zoom=12,

size=(400,400), sensor=False):
241
params=dict(
zoom= zoom,
style="feature:all|element:labels|visibility:off")
if satellite:
In [8]: london_map=map_at(51.5073509, -0.1277583)

import IPython

IPython.core.display.Image(london_map.content)
Out[9]:
242
We would like to test that it is building the parameters correctly. We can do this by mocking the
requests object. We need to temporarily replace a method in the library with a mock. We can use “patch”
to do this:
In [10]: from mock import patch

london_map=map_at(51.5073509, -0.1277583)
print mock_get.mock_calls
[call(’http://maps.googleapis.com/maps/api/staticmap?’, params={’style’: ’feature:all|element:labels|vis
Our tests then look like:
In [11]: def test_build_default_params():

default_map=map_at(51.0, 0.0)
mock_get.assert_called_with(
"http://maps.googleapis.com/maps/api/staticmap?",
params={
’sensor’:’false’,
’zoom’:12,
’size’:’400x400’,
’center’:’51.0,0.0’,
’style’:’feature:all|element:labels|visibility:off’
}
)
test_build_default_params()
That was quiet, so it passed. When I’m writing tests, I usually modify one of the expectations, to
something ‘wrong’, just to check it’s not passing “by accident”, run the tests, then change it back!
24.17.1 Testing functions that call other functions

In [12]: def partial_derivative(function, at, direction, delta=1.0):
f_x=function(at)
x_plus_delta=at[:]
x_plus_delta[direction]+=delta
f_x_plus_delta=function(x_plus_delta)
return (f_x_plus_delta-f_x)/delta
We want to test that the above function does the right thing. It is supposed to compute the derivative
of a function of a vector in a particular direction.
E.g.:
In [13]: partial_derivative(sum, [0,0,0], 1)
Out[13]: 1.0
How do we assert that it is doing the right thing? With tests like this:
In [14]: from mock import MagicMock
def test_derivative_2d_y_direction():
func=MagicMock()
243
partial_derivative(func, [0,0], 1)
func.assert_any_call([0, 1.0])
func.assert_any_call([0, 0])
test_derivative_2d_y_direction()
We made our mock a “Magic Mock” because otherwise, the mock results f x plus delta and f x can’t
be subtracted:
In [15]: MagicMock()-MagicMock()
Out[15]: <MagicMock name=’mock. sub ()’ id=’4455339856’>
In [16]: Mock()-Mock()
---------------------------------------------------------------------------
<ipython-input-16-fca1dbe33378> in <module>()
----> 1 Mock()-Mock()
TypeError: unsupported operand type(s) for -: ’Mock’ and ’Mock’
24.18 Using a debugger

24.18.1 Stepping through the code
Debuggers are programs that can be used to test other programs. They allow programmers to suspend
execution of the target program and inspect variables at that point.
• Mac - compiled languages: Xcode

• Windows - compiled languages: Visual Studio
• Linux: DDD
• all platforms: eclipse, gdb (DDD and eclipse are GUIs for gdb)
• python: spyder, pdb
• R: RStudio, debug, browser
24.18.2 Using the python debugger

Unfortunately this doesn’t work nicely in the notebook. But from the command line, you can run a python
program with:
python -m pdb my_program.py
24.18.3 Basic navigation:

Basic command to navigate the code and the python debugger:
• help: prints the help

• help n: prints help about command n
• n(ext): executes one line of code. Executes and steps over functions.
244
• s(tep): step into current function in line of code
• l(ist): list program around current position
• w(where): prints current stack (where we are in code)
• [enter]: repeats last command
• anypythonvariable: print the value of that variable
The python debugger is a python shell: it can print and compute values, and even change the values
of the variables at that point in the program.
24.18.4 Breakpoints
Break points tell debugger where and when to stop We say * b somefunctionname
In [1]: %%writefile energy_example.py

print energy([5, 6, 7, 8, 0, 1])
Overwriting energy example.py
The debugger is, of course, most used interactively, but here I’m showing a prewritten debugger script:
In [2]: %%writefile commands

restart # restart session
n
b energy # program will stop when entering energy
c # continue program until break point is reached
print density # We are now "inside" the energy function and can print any variable.
Overwriting commands
In [3]: %%bash
python -m pdb energy_example.py < commands

(Pdb) Restarting energy example.py with arguments:
# restart session
(Pdb) > /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/energy example.py(2
-> print energy([5, 6, 7, 8, 0, 1])
(Pdb) Breakpoint 1 at /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffu
(Pdb) > /Users/ccsprsd/jenkins/development/workspace/engineering-publisher/ch03tests/diffusion/model.py(
-> from numpy import array, any, sum
(Pdb) [5, 6, 7, 8, 0, 1]
(Pdb)
Alternatively, break-points can be set on files: b file.py:20 will stop on line 20 of file.py.
24.18.5 Post-mortem
Debugging when something goes wrong:
1. Have a crash somewhere in the code

2. run python -m pdb file.py or run the cell with %pdb on
The program should stop where the exception was raised
245
1. use w and l for position in code and in call stack
2. use up and down to navigate up and down the call stack
3. inspect variables along the way to understand failure
This does work in the notebook.
In [4]: %pdb on
partial_derivative(energy,[5,6,7,8,0,1],5)
Automatic pdb calling has been turned ON
---------------------------------------------------------------------------
<ipython-input-4-cd06eca5c4a7> in <module>()
NameError: name ’partial derivative’ is not defined
> <ipython-input-4-cd06eca5c4a7>(3)<module>()
---------------------------------------------------------------------------
StdinNotImplementedError Traceback (most recent call last)
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run code(self, code

3050 if result is not None:
3051 result.error in exec = sys.exc info()[1]
-> 3052 self.showtraceback()
3053 else:
3054 outflag = 0
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in showtraceback(self,
1854 if self.call pdb:
1855 # drop into debugger
-> 1856 self.debugger(force=True)
1857 return
1858
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in debugger(self, force

1020
246
-> 1022 pm()
1023
1024 #-------------------------------------------------------------------------
/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in <lambda>()
1017 else:
1018 # fallback to our internal debugger
-> 1019 pm = lambda : self.InteractiveTB.debugger(force=True)
1020
/usr/local/lib/python2.7/site-packages/IPython/core/ultratb.pyc in debugger(self, force)

1073 etb = etb.tb next
1074 self.pdb.botframe = etb.tb frame
-> 1075 self.pdb.interaction(self.tb.tb frame, self.tb)
1076
1077 if hasattr(self, ’tb’):
/usr/local/lib/python2.7/site-packages/IPython/core/debugger.pyc in interaction(self, frame, tra

276 while True:
277 try:
--> 278 OldPdb.interaction(self, frame, traceback)
279 except KeyboardInterrupt:
280 self.shell.write(’\n’ + self.shell.get exception only())
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/pdb.pyc
208 self.setup(frame, traceback)
209 self.print stack entry(self.stack[self.curindex])
--> 210 self.cmdloop()
211 self.forget()
212
/usr/local/Cellar/python/2.7.10 2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/cmd.pyc
128 if self.use rawinput:
129 try:
--> 130 line = raw input(self.prompt)
131 except EOFError:
132 line = ’EOF’
/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.pyc in raw input(self, prom

644 if not self. allow stdin:
645 raise StdinNotImplementedError(
--> 646 "raw input was called, but this frontend does not support input requests."
647 )
648 return self. input request(prompt,
247
> /usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.py(646)raw input()
645 raise StdinNotImplementedError(--> 646 "raw input was called, but th
647 )
File "/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/ipkernel.py", line 181, in do

shell.run cell(code, store history=store history, silent=silent)

interactivity=interactivity, compiler=compiler, result=result)

self.showtraceback()

self.debugger(force=True)

pm()

pm = lambda : self.InteractiveTB.debugger(force=True)
File "/usr/local/lib/python2.7/site-packages/IPython/core/ultratb.py", line 1075, in debugger

self.pdb.interaction(self.tb.tb frame, self.tb)
File "/usr/local/lib/python2.7/site-packages/IPython/core/debugger.py", line 278, in interacti

OldPdb.interaction(self, frame, traceback)

self.cmdloop()

line = raw input(self.prompt)
File "/usr/local/lib/python2.7/site-packages/IPython/kernel/zmq/kernelbase.py", line 646, in r

"raw input was called, but this frontend does not support input requests."
248
24.19 Jenkins
24.19.1 Test servers
Goal:
1. run tests nightly

2. run tests after each commit to github (or other)
3. run tests on different platforms
Various groups run servers that can be used to do this automatically.

RITS run a university-wide one.
24.19.2 Memory and profiling

For compiled languages (C, C++, Fortran): * Checking for memory leaks with valgrind: valgrind
--leak-check=full program * Checking cache hits and cache misses with cachegrind: valgrind
--tool=cachegrind program * Profiling the code with callgrind: valgrind --tool=callgrind program
• Python: profile with runsnake

• R: Rprof
In [1]:
24.20 Extended TDD Example: Monte-Carlo

24.20.1 Problem: Implement and test a simple Monte-Carlo algorithm
Given an input function (energy) and starting point (density) and a temperature T :
1. Compute energy at current density.

2. Move randomly chosen agent randomly left or right.
3. Compute second energy.
4. Compare the two energies:
5. If second energy is lower, accept move.
6. β is a parameter which determines how likely the simulation is to move from a ‘less favourable’ situation
to a ‘more favourable’ one.
7. Compute P0 = e−β(E1 −E0 ) and P1 a random number between 0 and 1,
8. If P0 > P1 , do the move anyway.
9. Repeat.
• the algorithm should work for (m)any energy function(s).

• there should be separate tests for separate steps! What constitutes a step?
• tests for the Monte-Carlo should not depend on other parts of code.
• Use matplotlib to plot density at each iteration, and make an animation
24.21 Solution
We need to break our problem down into pieces:
1. A function to generate a random change: random agent(), random direction()

2. A function to compute the energy before the change and after it: energy()
249
3. A function to determine the probability of a change given the energy difference (1 if decreases, otherwise
based on exponential): change density()
4. A function to determine whether to execute a change or not by drawing a random num-
beraccept change()
5. A method to iterate the above procedure: step()
Next Step: Think about the possible unit tests
1. Input insanity: e.g. density should non-negative integer; testing by giving negative values etc.
2. change density(): density is change by a particle hopping left or right? Do all positions have an
equal chance of moving?
3. accept change() will move be accepted when second energy is lower?
4. Make a small test case for the main algorithm. (Hint: by using mocking, we can pre-set who to move
where.)
In [1]: %%bash
mkdir -p DiffusionExample
In [2]: %%writefile DiffusionExample/MonteCarlo.py

from numpy import sum, array
from numpy.random import randint, choice
class MonteCarlo(object):
""" A simple Monte Carlo implementation """
def __init__(self, energy, density, temperature=1, itermax=1000):
from numpy import any, array
self.itermax = itermax
if temperature == 0: raise NotImplementedError("Zero temperature not implemented")

if temperature < 0e0: raise ValueError("Negative temperature makes no sense")
if len(density) < 2:
raise ValueError("Density is too short")
# of the right kind (integer). Unless it is zero length, in which case type does not mat
raise TypeError("Density should be an array of *integers*.")
raise ValueError("Density should be an a *1-dimensional* array of positive integers.
if sum(density) == 0:
raise ValueError("Density is empty.")
self.current_energy = energy(density)
self.temperature = temperature
self.density = density
def random_direction(self): return choice([-1, 1])
250
def random_agent(self, density):
#Particle index
particle = randint(sum(density))
current = 0
for location, n in enumerate(density):
current += n
if current > particle: break
return location
def change_density(self, density):

""" Move one particle left or right. """
location = self.random_agent(density)
# Move direction
if(density[location]-1<0): return array(density)
if location == 0: direction = 1
elif location == len(density) - 1: direction = -1
else: direction = self.random_direction()
# Now make change

result = array(density)
result[location] -= 1
result[location + direction] += 1
return result
def accept_change(self, prior, successor):

""" Returns true if should accept change. """
from numpy import exp
from numpy.random import uniform
if successor <= prior: return True
else:
return exp(-(successor - prior) / self.temperature) > uniform()
def step(self):
iteration = 0
while iteration < self.itermax:
new_density = self.change_density(self.density)
new_energy = energy(new_density)
accept = self.accept_change(self.current_energy, new_energy)

if accept: self.density, self.current_energy = new_density, new_energy
iteration+=1
return self.current_energy, self.density
def energy(density, coefficient=1):

:Parameters:
"""
251
# Make sure input is an array

# of the right kind (integer). Unless it is zero length, in which case type does not matter.
raise TypeError("Density should be an array of *integers*.")
return coefficient * 0.5 * sum(density * (density - 1))
Writing DiffusionExample/MonteCarlo.py

import sys
sys.path.append(’DiffusionExample’)
from MonteCarlo import MonteCarlo, energy
import numpy as np
import numpy.random as random
from matplotlib import animation
Temperature = 0.1
density=[np.sin(i) for i in np.linspace(0.1, 3, 100)]
density=np.array(density)*100
density = density.astype(int)
fig = plt.figure()
ax = plt.axes(xlim=(-1,len(density)),ylim=(0,np.max(density)+1))
image = ax.scatter(range(len(density)), density)
txt_energy = plt.text(0, 100, ’Energy = 0’)

plt.xlabel(’Temperature = 0.1’)
plt.ylabel(’Energy Density’)
mc=MonteCarlo(energy, density, temperature = Temperature)
def simulate(step):
energy, density = mc.step()
image.set_offsets(np.vstack((range(len(density)), density)).T)
txt_energy.set_text(’Energy = %f’% energy)
animation.FuncAnimation(fig, simulate, frames=200,

interval=50, blit=True)
252
#anim

Out[3]: <matplotlib.animation.FuncAnimation at 0x10cd246d0>
In [4]: %%writefile DiffusionExample/test_model.py

from MonteCarlo import MonteCarlo
from nose.tools import assert_equal, assert_almost_equal, assert_true, assert_raises
from mock import MagicMock
def test_input_sanity():
""" Check incorrect input do fail """
energy = MagicMock()
with assert_raises(NotImplementedError) as exception: MonteCarlo(sum, [1, 1, 1], 0e0)

with assert_raises(ValueError) as exception: MonteCarlo(energy, [1, 1, 1], temperature=-1e0)
with assert_raises(TypeError) as exception: MonteCarlo(energy, [1.0, 2, 3])

with assert_raises(ValueError) as exception: MonteCarlo(energy, [-1, 2, 3])
with assert_raises(ValueError) as exception: MonteCarlo(energy, [[1, 2, 3], [3, 4, 5]])
with assert_raises(ValueError) as exception: MonteCarlo(energy, [3])
with assert_raises(ValueError) as exception: MonteCarlo(energy, [0, 0])
def test_move_particle_one_over():
""" Check density is change by a particle hopping left or right. """
from numpy import nonzero, multiply
for i in range(100): # Do this n times, to avoid issues with random numbers

# Create density
density = randint(50, size=randint(2, 6))

mc = MonteCarlo(energy, density)
# Change it
new_density = mc.change_density(density)
# Make sure any movement is by one

indices = nonzero(density - new_density)[0]
assert_equal(len(indices), 2, "densities differ in two places")
assert_equal(
multiply.reduce((density - new_density)[indices]),
-1,
"densities differ by + and - 1"
)
def test_equal_probability():
253
""" Check particles have equal probability of movement. """
from numpy import array, sqrt, count_nonzero
density = array([1, 0, 99])

mc = MonteCarlo(energy, density)
changes_at_zero = [(density - mc.change_density(density))[0] != 0 for i in range(10000)]
assert_almost_equal(
count_nonzero(changes_at_zero),
0.01 * len(changes_at_zero),
delta = 0.5 * sqrt(len(changes_at_zero))
)
def test_accept_change():
""" Check that move is accepted if second energy is lower """
from numpy import sqrt, count_nonzero, exp
energy = MagicMock
mc = MonteCarlo(energy, [1, 1, 1], temperature=100.0)
# Should always be true. But do more than one draw, in case randomness incorrectly crept int
# implementation
for i in range(10):
assert_true(mc.accept_change(0.5, 0.4))
assert_true(mc.accept_change(0.5, 0.5))
# This should be accepted only part of the time, depending on exponential distribution
prior, successor = 0.4, 0.5
accepted = [mc.accept_change(prior, successor) for i in range(10000)]
assert_almost_equal(
count_nonzero(accepted) / float(len(accepted)),
exp(-(successor - prior) / mc.temperature),
delta = 3e0 / sqrt(len(accepted))
)
def test_main_algorithm():
import numpy as np
from numpy import testing
from mock import Mock
density = [1, 1, 1, 1, 1]
mc = MonteCarlo(energy, density, itermax = 5)
acceptance = [True, True, True, True, True]

mc.accept_change = Mock(side_effect = acceptance)
mc.random_agent = Mock(side_effect = [0, 1, 2, 3, 4])
mc.random_direction = Mock(side_effect = [1, 1, 1, 1, -1])
np.testing.assert_equal(mc.step()[1], [0,1,1,2,1])
254
Writing DiffusionExample/test model.py
In [5]: %%bash
cd DiffusionExample
nosetests
...
----------------------------------------------------------------------
OK
255
Chapter 25
Installing Libraries
25.1 Installing Libraries

We’ve seen that PyPI carries lots of python libraries. But how do we install them?
In the UCL teaching clusters, we cannot, as these need to be installed by administrators.
So you’ll need to follow this lesson on a computer of your own.
The main problem is this: libraries need other libraries
So you can’t just install a library by copying code to the computer: you’ll find yourself wandering down
a tree of “dependencies”; libraries needed by libraries needed by the library you want.
This is actually a good thing; it means that people are making use of each others’ code. There’s a real
problem in scientific programming, of people who think they’re really clever writing their own twenty-fifth
version of the same thing.
So using other people’s libraries is good.
Why don’t we do it more? Because it can often be quite difficult to install other peoples’ libraries!
Python has developed a good tool for avoiding this: pip.
25.2 Installing Geopy using Pip

On a computer you control, on which you have installed python via Anaconda, you will need to open a
terminal to invoke the library-installer program, pip.
• On windows, go to start->all programs->Anaconda->Anaconda Command Prompt

• On mac, start terminal.
• On linux, open a bash shell.
Into this shell, type:

pip install geopy
The computer will install the package automatically from PyPI.
Now, close IPython notebook if you have it open, and reopen it. Check your new library is installed with:
In [1]: import geopy

geocoder=geopy.geocoders.GoogleV3(domain="maps.google.co.uk")
geocoder.geocode(’Cambridge’,exactly_one=False)
Out[1]: [Location((52.205337, 0.121817, 0.0)), Location((51.73193, -2.3649, 0.0))]
That was actually pretty easy, I hope. This is how you’ll install new libraries when you need them.
Troubleshooting:
On mac or linux, you might get a complaint that you need “superuser”, “root”, or “administrator” access.
If so type:
256
• sudo pip install geopy
and enter your password.
If you get a complaint like: ‘pip is not recognized as an internal or external command’, try the following:
• conda install pip (Windows)
• sudo easy install pip (Mac, Linux)
Ask me over email if you run into trouble.
25.3 Installing binary dependencies with Conda

pip is the usual Python tool for installing libraries. But there’s one area of library installation that is still
awkward: some python libraries depend not on other python libraries, but on libraries in C++ or Fortran.
This can cause you to run into difficulties installing some libraries. Fortunately, for lots of these, Con-
tinuum, the makers of Anaconda, provide a carefully managed set of scripts for installing these awkward
non-python libraries too. You can do this with the conda command line tool, if you’re using Anaconda.
Simply type
• conda install <whatever>
instead of pip install. This will fetch the python package not from PyPI, but from Anaconda’s
distribution for your platform, and manage any non-python dependencies too.
Typically, if you’re using Anaconda, whenever you come across a python package you want, you should
check if Anaconda package it first using this list: http://docs.continuum.io/anaconda/pkg-docs.html. (Or
just by trying conda install and hoping!) If you can conda install it, you’ll likely have less problems.
But Continuum don’t package everything, so you’ll need to pip install from time to time.
25.4 Where do these libraries go?

In [2]: geopy.__path__
Out[2]: [’/usr/local/lib/python2.7/site-packages/geopy’]
Your computer will be configured to keep installed Python packages in a particular place.
Python knows where to look for possible library installations in a list of places, called the “PythonPath”.
It will try each of these places in turn, until it finds a matching library name.
In [3]: import sys
sys.path[-4:] # Just list the last few
Out[3]: [’/Library/Python/2.7/site-packages/parse-1.6.3-py2.7.egg’,
’/Library/Python/2.7/site-packages’,
’/usr/local/lib/python2.7/site-packages/IPython/extensions’,
’/Users/ccsprsd/.ipython’]
25.5 Libraries not in PyPI

Sometimes, such as for the Animation library in the Boids example, you’ll need to download the source
code directly. This won’t automatically follow the dependency tree, but for simple standalone libraries, is
sometimes necessary.
To install these on windows, download and unzip the library into a folder of your choice, e.g.
my python libs.
On windows, a reasonable choice is the folder you end up in when you open the Anaconda terminal. You
can get a graphical view on this folder by typing: explorer .
Make a new folder for your download and unzip the library there.
Now, you need to move so you’re inside your download in the terminal:
257
• cd my python libs
• cd <library name> (e.g. cd JSAnimation-master)
Now, manually install the library in your PythonPath:
• python setup.py install
[You might need to do
• sudo python setup.py install
if you get prompted for ‘root’ or ‘admin’ access.]

This is all pretty awkward, but it is worth practicing this stuff, as most of the power of using programming
for research resides in all the libraries that are out there.
Finally, writing your own libraries is beyond the scope of this course, but we cover it in Our More
Advanced Python Course
25.6 Libraries
25.6.1 Libraries are awesome
The strength of a language lies as much in the set of libraries available, as it does in the language itself.
A great set of libraries allows for a very powerful programming style:
• Write minimal code yourself

• Choose the right libraries
• Plug them together
• Create impressive results
Not only is this efficient with your programming time, it’s also more efficient with computer time.
The chances are any algorithm you might want to use has already been programmed better by someone
else.
25.7 Careful use of Libraries

25.7.1 Drawbacks of libraries.
• Sometimes, libraries are not looked after by their creator: code that is not maintained rots:
– It no longer works with later versions of upstream libraries.
– It doesn’t work on newer platforms or systems.
– Features that are needed now, because the field has moved on, are not added
• Sometimes, libraries are hard to get working:
– For libraries in pure python, this is almost never a problem
– But many libraries involve compiled components: these can be hard to install.
25.7.2 Contribute, don’t duplicate

• You have a duty to the ecosystem of scholarly software:
– If there’s a tool or algorithm you need, find a project which provides it.
– If there are features missing, or problems with it, fix them, don’t create your own library.
258
25.7.3 How to choose a library
• When was the last commit?
• How often are there commits?
• Can you find the lead contributor on the internet?
• Do they respond when approached:
– emails to developer list
– personal emails
– tweets
– irc
– issues raised on GitHub?
• Are there contributors other than the lead contributor?
• Is there discussion of the library on Stack Exchange?
• Is the code on an open version control tool like GitHub?
• Is it on standard package repositories. (PyPI, apt/yum/brew)
• Are there any tests?
• Download it. Can you build it? Do the tests pass?
• Is there an open test dashboard? (Travis/Jenkins/CDash)
• What dependencies does the library itself have? Do they pass this list?
• Are different versions of the library clearly labeled with version numbers?
• Is there a changelog?
25.7.4 Sensible Version Numbering

The best approach to version numbers clearly distinguishes kinds of change:
Given a version number MAJOR.MINOR.PATCH, e.g. 2.11.14 increment the:
• MAJOR version when you make incompatible API changes,

• MINOR version when you add functionality in a backwards-compatible manner, and
• PATCH version when you make backwards-compatible bug fixes.
This is called Semantic Versioning
25.8 Python Libraries

25.8.1 The Python Standard Library
Python comes with a powerful standard library.
Learning python is as much about learning this library as learning the language itself.
You’ve already seen a few packages in this library: math, pdb, pytest, datetime.
25.8.2 The Python Package Index

Python’s real power, however, comes with the Python Package Index: PyPI. This is a huge array of libraries,
with all kinds of capabilities, all easily installable from the command line or through your Python distribution.
25.9 Argparse
This is the standard library for building programs with a command-line interface.
In [1]: %%writefile greeter.py

#!/usr/bin/env python
from argparse import ArgumentParser
if __name__ == "__main__":
259
parser = ArgumentParser(description = "Generate appropriate greetings")
parser.add_argument(’--title’, ’-t’)
parser.add_argument(’--polite’,’-p’, action="store_true")
parser.add_argument(’personal’)
parser.add_argument(’family’)
arguments= parser.parse_args()
greeting= "How do you do, " if arguments.polite else "Hey, "
if arguments.title:
greeting+=arguments.title+" "
greeting+= arguments.personal + " " + arguments.family +"."
print greeting
Writing greeter.py
In [2]: %%bash
#!/usr/bin/env bash
chmod u+x greeter.py
./greeter.py --help
./greeter.py James Hetherington
./greeter.py --polite James Hetherington
./greeter.py James Hetherington --title Dr
usage: greeter.py [-h] [--title TITLE] [--polite] personal family
Generate appropriate greetings
positional arguments:
personal
family
optional arguments:
-h, --help show this help message and exit
--title TITLE, -t TITLE
--polite, -p
Hey, James Hetherington.
How do you do, James Hetherington.
Hey, Dr James Hetherington.
25.10 Packaging
25.10.1 Packaging
Once we’ve made a working program, we’d like to be able to share it with others.
A good cross-platform build tool is the most important thing: you can always have collaborators build
from source.
25.10.2 Distribution tools

Distribution tools allow one to obtain a working copy of someone else’s package.
Language-specific tools: PyPI, Ruby Gems, CPAN, CRAN Platform specific packagers e.g. brew,
apt/yum
Windows doesn’t have anything like brew install or apt-get You have to build an ‘installer’.
260
25.10.3 Laying out a project
When planning to package a project for distribution, defining a suitable project layout is essential.
In [1]: %%bash
tree --charset ascii greetings
greetings
|-- CITATION.md
|-- LICENSE.md
|-- README.md
|-- conf.py
|-- greetings
| |-- init .py
| |-- command.py
| |-- greeter.py
| ‘-- test
| |-- init .py
| |-- fixtures
| | ‘-- samples.yaml
| ‘-- test greeter.py
|-- index.rst
‘-- setup.py
3 directories, 12 files
We can start by making our directory structure
In [2]: %%bash
mkdir -p greetings/greetings/test/fixtures
mkdir -p greetings/scripts
25.10.4 Using setuptools

To make python code into a package, we have to write a setupfile:
In [3]: %%writefile greetings/setup.py
from setuptools import setup, find_packages
setup(
name = "Greetings",
version = "0.1",
packages = find_packages(exclude=[’*test’]),
scripts = [’scripts/greet’],
install_requires = [’argparse’]
)
Overwriting greetings/setup.py
We can now install this code with
sudo python setup.py install
And the package will be then available to use everywhere on the system:
261
In [4]: import greetings
from greetings.greeter import greet
print greetings.greeter.greet("James","Hetherington")
And the scripts are now available as command line commands:
In [5]: %%bash
#!/usr/bin/env bash
greet --help
usage: greet [-h] [--title TITLE] [--polite] personal family
Generate appropriate greetings
positional arguments:
personal
family
optional arguments:
-h, --help show this help message and exit
--title TITLE, -t TITLE
--polite, -p
In [6]: %%bash
greet James Hetherington
greet --polite James Hetherington
greet James Hetherington --title Dr

How do you do, James Hetherington.
Hey, Dr James Hetherington.
25.10.5 Installing from GitHub

We could now submit “greeter” to PyPI for approval, so everyone could pip install it.
However, when using git, we don’t even need to do that: we can install directly from any git URL:
sudo pip install git+git://github.com/jamespjh/greeter
In [7]: %%bash
greet Humphry Appleby --title Sir
Hey, Sir Humphry Appleby.
Try it!
25.10.6 Convert the script to a module

Of course, there’s more to do when taking code from a quick script and turning it into a proper module:
In [8]: %%writefile greetings/greetings/greeter.py
def greet(personal, family, title="", polite=False):
""" Generate a greeting string for a person.
262
Parameters
----------
personal: str
A given name, such as Will or Jean-Luc
family: str
A family name, such as Riker or Picard
title: str
An optional title, such as Captain or Reverend
polite: bool
True for a formal greeting, False for informal.
Returns
-------
string
An appropriate greeting
"""
greeting= "How do you do, " if polite else "Hey, "

if title:
greeting+=title+" "
greeting+= personal + " " + family +"."

return greeting
Overwriting greetings/greetings/greeter.py
The documentation string explains how to use the function; don’t worry about this for now, we’ll consider
this next time.
25.10.7 Write an executable script

In [9]: %%writefile greetings/greetings/command.py
from argparse import ArgumentParser
from greeter import greet
def process():
parser = ArgumentParser(description = "Generate appropriate greetings")
parser.add_argument(’--title’, ’-t’)
parser.add_argument(’--polite’, ’-p’, action="store_true")
parser.add_argument(’personal’)
parser.add_argument(’family’)
arguments= parser.parse_args()
print greet(arguments.personal, arguments.family, arguments.title, arguments.polite)
if __name__ == "__main__":
process()
Overwriting greetings/greetings/command.py
263
25.10.8 Write an entry point script stub
In [10]: %%writefile greetings/scripts/greet
#!/usr/bin/env python
from greetings.command import process
process()
Writing greetings/scripts/greet
25.10.9 Write a readme file

e.g.:
In [11]: %%writefile greetings/README.md
Greetings!
==========
This is a very simple example package used as part of the UCL

[Research Software Engineering with Python](development.rc.ucl.ac.uk/training/engineering) cour
Overwriting greetings/README.md
25.10.10 Write a license file

e.g.:
In [12]: %%writefile greetings/LICENSE.md
(C) University College London 2014
This "greetings" example package is granted into the public domain.
Overwriting greetings/LICENSE.md
25.10.11 Write a citation file

e.g.:
In [13]: %%writefile greetings/CITATION.md
If you wish to refer to this course, please cite the URL

http://development.rc.ucl.ac.uk/training/engineering
Portions of the material are taken from Software Carpentry

http://swcarpentry.org
Overwriting greetings/CITATION.md
25.10.12 Define packages and executables

In [14]: %%bash
touch greetings/greetings/test/__init__.py
touch greetings/greetings/__init__.py
chmod u+x greetings/greetings/command.py
# chmod u+x greetings/scripts/greet
264
25.10.13 Write some unit tests
Separating the script from the logical module made this possible:
In [15]: %%writefile greetings/greetings/test/test_greeter.py

import yaml
import os
from ..greeter import greet
def test_greeter():
with open(os.path.join(os.path.dirname(__file__),’fixtures’,’samples.yaml’)) as fixtures_fi
fixtures=yaml.load(fixtures_file)
for fixture in fixtures:
answer=fixture.pop(’answer’)
assert_equal(greet(**fixture), answer)
Overwriting greetings/greetings/test/test greeter.py
Add a fixtures file:
In [16]: %%writefile greetings/greetings/test/fixtures/samples.yaml

- personal: James
family: Hetherington
answer: "Hey, James Hetherington."
- personal: James
polite: True
answer: "How do you do, James Hetherington."
- personal: James
title: Dr
answer: "Hey, Dr James Hetherington."
Overwriting greetings/greetings/test/fixtures/samples.yaml
25.10.14 Developer Install

If you modify your source files, you would now find it appeared as if the program doesn’t change.
That’s because pip install copies the file.
(On my system to /Library/Python/2.7/site-packages/: this is operating system dependent.)
If you want to install a package, but keep working on it, you can do
sudo python setup.py develop
25.10.15 Distributing compiled code

If you’re working in C++ or Fortran, there is no language specific repository. You’ll need to write platform
installers for as many platforms as you want to support.
Typically:
• dpkg for apt-get on Ubuntu and Debian

• rpm for yum on Redhat and Fedora
• homebrew on OSX (Possibly macports as well)
• An executable msi installer for Windows.
265
25.10.16 Homebrew
Homebrew: A ruby DSL, you host off your own webpage
See my installer for the cppcourse example
If you’re on OSX, do:
brew tap jamespjh/homebrew-reactor

brew install reactor
25.10.17 Exercises
We previously looked at Greengraph.py, a script that enables us to explore how green space varies as we
move from the city to the countryside:
In [17]: ### "geolocation"

import geopy
geocoder=geopy.geocoders.GoogleV3(domain="maps.google.co.uk")
def geolocate(place):
return geocoder.geocode(place,exactly_one=False)[0][1]
london_location=geolocate("London")
print london_location
(51.5073509, -0.1277583)
The Greengraph example is only available as a single large script.

Your task is to transform this into a python package that can be pip installed directly from GitHub.
Remember to include:
• an init .py file

• a setup.py file
• tests
• license and documentation
25.11 Documentation
25.11.1 Documentation is hard
• Good documentation is hard, and very expensive.
• Bad documentation is detrimental.
• Good documentation quickly becomes bad if not kept up-to-date with code changes.
• Professional companies pay large teams of documentation writers.
25.11.2 Prefer readable code with tests and vignettes

If you don’t have the capacity to maintain great documentation, focus on:
• Readable code
• Automated tests
• Small code samples demonstrating how to use the api
266
25.11.3 Comment-based Documentation tools
Documentation tools can produce extensive documentation about your code by pulling out comments near
the beginning of functions, together with the signature, into a web page.
The most popular is Doxygen
Have a look at an example of some Doxygen output
Sphinx is nice for Python, and works with C++ as well. Here’s some Sphinx-generated output and the
corresponding source code Breathe can be used to make Sphinx and Doxygen work together.
Roxygen is good for R.
25.12 Example of using Sphinx

25.12.1 Write some docstrings
We’re going to document our “greeter” example using docstrings with Sphinx.
There are various conventions for how to write docstrings, but the native sphinx one doesn’t look nice
when used with the built in help system.
In writing Greeter, we used the docstring conventions from NumPy. So we use the numpydoc sphinx
extension to support these.
"""
Generate a greeting string for a person.
Parameters
----------
personal: str
A given name, such as Will or Jean-Luc
family: str
A family name, such as Riker or Picard
title: str
An optional title, such as Captain or Reverend
polite: bool
True for a formal greeting, False for informal.
Returns
-------
string
An appropriate greeting
25.12.2 Set up sphinx

Invoke the sphinx-quickstart command to build Sphinx’s configuration file automatically based on questions
at the command line:
sphinx-quickstart
Which responds:
Welcome to the Sphinx 1.2.3 quickstart utility.
Please enter avalues for the following settings (just press Enter to
accept a default value, if one is given in brackets).
Enter the root path for documentation.

> Root path for the documentation [.]:
267
and then look at and adapt the generated config, a file called conf.py in the root of the project. This
contains the project’s Sphinx configuration, as Python variables:
#Add any Sphinx extension module names here, as strings. They can be
#extensions coming with Sphinx (named ’sphinx.ext.*’) or your custom
# ones.
extensions = [
’sphinx.ext.autodoc’, # Support automatic documentation
’sphinx.ext.coverage’ , # Automatically check if functions are documented
’sphinx.ext.mathjax’ , # Allow support for algebra
’sphinx.ext.viewcode’ , # Include the source code in documentation
’numpydoc’ # Support NumPy style docstrings
]
To proceed with the example, we’ll copy a finished conf.py into our folder, though normally you’ll always
use sphinx-quickstart
In [1]: %%writefile ../ch04packaging/greetings/conf.py
import sys
import os
extensions = [
’sphinx.ext.autodoc’, # Support automatic documentation
’sphinx.ext.coverage’, # Automatically check if functions are documented
’sphinx.ext.mathjax’, # Allow support for algebra
’sphinx.ext.viewcode’, # Include the source code in documentation
’numpydoc’ # Support NumPy style docstrings
]
templates_path = [’_templates’]
source_suffix = ’.rst’
master_doc = ’index’
project = u’Greetings’
copyright = u’2014, James Hetherington’
version = ’0.1’
release = ’0.1’
exclude_patterns = [’_build’]
pygments_style = ’sphinx’
html_theme = ’default’
html_static_path = [’_static’]
htmlhelp_basename = ’Greetingsdoc’
latex_elements = {
}
latex_documents = [
(’index’, ’Greetings.tex’, u’Greetings Documentation’,
u’James Hetherington’, ’manual’),
]
man_pages = [
(’index’, ’greetings’, u’Greetings Documentation’,
[u’James Hetherington’], 1)
]
texinfo_documents = [
268
(’index’, ’Greetings’, u’Greetings Documentation’,
u’James Hetherington’, ’Greetings’, ’One line description of project.’,
’Miscellaneous’),
]
Overwriting ../ch04packaging/greetings/conf.py
25.12.3 Define the root documentation page

Sphinx uses RestructuredText another wiki markup format similar to Markdown.
You define an “index.rst” file to contain any preamble text you want. The rest is autogenerated by
sphinx-quickstart
In [2]: %%writefile ../ch04packaging/greetings/index.rst

Welcome to Greetings’s documentation!
=====================================
Simple "Hello, James" module developed to teach research software engineering.
.. autofunction:: greetings.greeter.greet
Overwriting ../ch04packaging/greetings/index.rst
25.12.4 Run sphinx

We can run Sphinx using:
In [3]: %%bash
cd ../session04/greetings/
sphinx-build . doc
bash: line 1: cd: ../session04/greetings/: No such file or directory

Error: Source directory doesn’t contain a conf.py file.
25.12.5 Sphinx output

Sphinx’s output is html. We just created a simple single function’s documentation, but Sphinx will create
multiple nested pages of documentation automatically for many functions.
25.13 Engineering
25.13.1 Software Engineering Stages
• Requirements
• Functional Design
• Architectural Design
• Implementation
• Integration
25.13.2 Requirements Engineering

Requirements capture obviously means describing the things the software needs to be able to do.
A common approach is to write down lots of “user stories”, describing how the software helps the user
achieve something:
269
As a clinician, when I finish an analysis, I want a report to be created on the test results, so that
I can send it to the patient.
As a role, when condition or circumstance applies I want a goal or desire so that benefits occur.
These are easy to map into the Gherkin behaviour driven design test language.
25.13.3 Functional and architectural design

Engineers try to separate the functional design, how the software appears to and is used by the user, from
the architectural design, how the software achieves that functionality.
Changes to functional design require users to adapt, and are thus often more costly than changes to
architectural design.
25.13.4 Waterfall
The Waterfall design philosophy argues that the elements of design should occur in order: first requirements
capture, then functional design, then architectural design. This approach is based on the idea that if a
mistake is made in the design, then programming effort is wasted, so significant effort is spent in trying to
ensure that requirements are well understood and that the design is correct before programming starts.
25.13.5 Why Waterfall?

Without a design approach, programmers resort to designing as we go, typing in code, trying what works,
and making it up as we go along. When trying to collaborate to make software with others this can result in
lots of wasted time, software that only the author understands, components built by colleagues that don’t
work together, or code that the programmer thinks is nice but that doesn’t meet the user’s requirements.
25.13.6 Problems with Waterfall

Waterfall results in a contractual approach to development, building an us-and-them relationship between
users, business types, designers, and programmers.
I built what the design said, so I did my job.
Waterfall results in a paperwork culture, where people spend a long time designing standard forms to
document each stage of the design, with less time actually spent making things.
Waterfall results in excessive adherence to a plan, even when mistakes in the design are obvious to people
doing the work.
25.13.7 Software is not made of bricks

The waterfall approach to software engineering comes from the engineering tradition applied to building
physical objects, where Architects and Engineers design buildings, and builders build them according to the
design.
Software is intrinsically different:

Software is not the same ‘stuff’ as that from which physical systems are constructed. Software
systems differ in material respects from physical systems. Much of this has been rehearsed by
Fred Brooks in his classic ‘No Silver Bullet’ paper. First, complexity and scale are different
in the case of software systems: relatively functionally simple software systems comprise more
independent parts, placed in relation to each other, than do physical systems of equivalent func-
tional value. Second, and clearly linked to this, we do not have well developed components and
composition mechanisms from which to build software systems (though clearly we are working
hard on providing these) nor do we have a straightforward mathematical account that permits
us to reason about the effects of composition.
270
Third, software systems operate in a domain determined principally by arbitrary rules about
information and symbolic communication whilst the operation of physical systems is governed
by the laws of physics. Finally, software is readily changeable and thus is changed, it is used in
settings where our uncertainty leads us to anticipate the need to change.
– Prof. Anthony Finkelstein, UCL Dean of Engineering, and Professor of Software Systems Engineering
25.13.10 The Agile Manifesto

In 2001, authors including Martin Folwer, Ward Cunningham and Kent Beck met in a Utah ski resort, and
published the following manifesto.
Manifesto for Agile Software Development
We are uncovering better ways of developing software by doing it and helping others do it. Through this
work we have come to value:
• Individuals and interactions over processes and tools

• Working software over comprehensive documentation
• Customer collaboration over contract negotiation
• Responding to change over following a plan
That is, while there is value in the items on the right, we value the items on the left more.
25.13.11 Agile is not absence of process

The Agile movement is not anti-methodology, in fact, many of us want to restore credibility to the
word methodology. We want to restore a balance. We embrace modeling, but not in order to file
some diagram in a dusty corporate repository. We embrace documentation, but not hundreds of
pages of never-maintained and rarely-used tomes. We plan, but recognize the limits of planning
in a turbulent environment. Those who would brand proponents of XP or SCRUM or any of the
other Agile Methodologies as “hackers” are ignorant of both the methodologies and the original
definition of the term hacker
– Jim Highsmith.
25.13.12 Elements of an Agile Process

• Continuous delivery
• Self-organising teams
• Iterative development
• Ongoing design
25.13.13 Ongoing Design

Agile development doesn’t eschew design. Design documents should still be written, but treated as living
documents, updated as more insight is gained into the task, as work is done, and as requirements change.
Use of a Wiki or version control repository to store design documents thus works much better than using
Word documents!
Test-driven design and refactoring are essential techniques to ensure that lack of “Big Design Up Front”
doesn’t produce badly constructed spaghetti software which doesn’t meet requirements. By continously
scouring our code for smells, and stopping to refactor, we evolve towards a well-structured design with
weakly interacting units. By starting with tests which describe how our code should behave, we create
executable specifications, giving us confidence that the code does what it is supposed to.
271
25.13.14 Iterative Development
Agile development maintains a backlog of features to be completed and bugs to be fixed. In each iteration,
we start with a meeting where we decide which backlog tasks will be attempted during the development
cycle, estimating how long each will take, and selecting an achievable set of goals for the “sprint”. At the
end of each cycle, we review the goals completed and missed, and consider what went well, what went badly,
and what could be improved.
We try not to add work to a cycle mid-sprint. New tasks that emerge are added to the backlog, and
considered in the next planning meeting. This reduces stress and distraction.
25.13.15 Continuous Delivery

In agile development, we try to get as quickly as possible to code that can be demonstrated to clients. A
regular demo of progress to clients at the end of each development iteration says so much more than sharing
a design document. “Release early, release often” is a common slogan. Most bugs are found by people using
code – so exposing code to users as early as possible will help find bugs quickly.
25.13.16 Self-organising teams

Code is created by people. People work best when they feel ownership and pride in their work. Division of
responsiblities into designers and programmers results in a “Code Monkey” role, where the craftspersonship
and sense of responsibility for code quality is lost. Agile approaches encourage programmers, designers,
clients, and businesspeople to see themselves as one team, working together, with fluid roles. Programmers
grab issues from the backlog according to interest, aptitude, and community spirit.
25.13.17 Agile in Research

Agile approaches, where we try to turn the instincts and practices which emerge naturally when smart
programmers get together into well-formulated best practices, have emerged as antidotes to both the chaotic
free-form typing in of code, and the rigid paperwork-driven approaches of Waterfall.
If these approaches have turned out to be better even in industrial contexts, where requirements for code
can be well understood, they are even more appropriate in a research context, where we are working in
poorly understood fields with even less well captured requirements.
25.13.18 Conclusion
• Don’t ignore design
• See if there’s a known design pattern that will help
• Do try to think about how your code will work before you start typing
• Do use design tools like UML to think about your design without coding straight away
• Do try to write down some user stories
• Do maintain design documents.
BUT
• Do change your design as you work, updating the documents if you have them
• Don’t go dark – never do more than a couple of weeks programming without showing what you’ve
done to colleagues
• Don’t get isolated from the reasons for your code’s existence, stay involved in the research, don’t be a
Code Monkey.
• Do keep a list of all the things your code needs, estimate and prioritise tasks carefully.
272
25.14 Exercises
25.14.1 Refactoring to classes
Complete the exercise on Boids from last week, as far as creating a class for a Boid, if you haven’t already.
25.14.2 Refactoring to Inheritance and Polymorphism

In the Eagle branch in my Boids repository you will find an extension of Boids to support multiple kinds
of Bird. You will see that this suffers from the use of an if (type) statement which would be better
implemented with inheritance and polymorphism.
To access the Eagle branch: cd ¡repository¿ git checkout eagle
25.14.3 Refactoring to Patterns: Builder

The way in which we construct our Boids model’s bird content, specifying each of the model parameters as
a constructor, and add birds using initialise random and initialise from data is rather clunky.
Create a ModelBuilder class with methods to define model parameters and to add a random boid and
boid or boids from data, using the Builder pattern.
You could even create two subclasses of ModelBuilder to build either random boids or boids from a
dataset.
25.14.4 Refactoring to Patterns: Model/View

You can apply Model/View to create a separate class to visualise boid models using Matplotlib, and even
perhaps write another viewer to visualise the boids in a different way: perhaps as graphs over time of
dispersal, distance from flock to eagle and average velocity.
25.14.5 Using UML

You should also use YUML or another UML tool to visualise your class structure.
You don’t have to do all these things to pass the assignment: any minimally object-oriented implemen-
tation of the boids will pass.
25.15 Software Licensing

25.15.1 Reuse
This course is distributed under the Creative Commons By Attribution license, which means you can modify
and reuse the materials, so long as you credit UCL Research IT Services.
25.15.2 Disclaimer
Here we attempt to give some basic advice on choosing a license for your software. But:
• we are NOT lawyers

• opinions differ (and flamewars are boring)
• this training does NOT constitute legal advice.
For an in-depth discussion of software licenses, read the O’Reilly book.

Your department, or UCL, may have policies about applying licenses to code you create while a UCL
employee or student. This training doesn’t address this issue, and does not represent UCL policy – seek
advice from your supervisor or manager if concerned.
273
25.15.3 Choose a license
It is important to choose a license and to create a license file to tell people what it is.
The license lets people know whether they can reuse your code and under what terms. This course has
one, for example.
Your license file should typically be called LICENSE.txt or similar. GitHub will offer to create a license
file automatically when you create a new repository.
25.15.4 Open source doesn’t stop you making money

A common misconception about open source software is the thought that open source means you can’t make
any money. This is wrong.
Plenty of people open source their software and profit from:
• The software under a different license e.g. Saxon

• Consulting. For example: Continuum who help maintain NumPy
• Manuals. For example: VTK
• Add-ons. For example: Puppet
• Server software, which open source client software interacts with. For example: GitHub API clients
25.15.5 Plagiarism vs promotion

Many researchers worry about people stealing their work if they open source their code. But often the
biggest problem is not theft, but the fact no one is aware of your work.
Open source is a way to increase the probability that someone else on the planet will care enough about
your work to cite you.
So when thinking about whether to open source your code, think about whether you’re more worried
about anonymity or theft.
25.15.6 Your code is good enough

New coders worry that they’ll be laughed at if they put their code online. Don’t worry. Everyone, including
people who’ve been coding for decades, writes shoddy code that is full of bugs.
The only thing that will make your code better, is other people reading it.
For small scripts that no one but you will ever use, my recommendation is to use an open repository
anyway. Find a buddy, and get them to comment on it.
25.15.7 Worry about license compatibility and proliferation

Not all open source code can be used in all projects. Some licenses are legally incompatible.
This is a huge and annoying problem. As an author, you might not care, but you can’t anticipate the
exciting uses people might find by mixing your code with someone else’s.
Use a standard license from the small list that are well-used. Then people will understand. Don’t make
up your own.
When you’re about to use a license, see if there’s a more common one which is recommended, e.g.: using
the opensource.org proliferation report
25.15.8 Academic license proliferation

Academics often write their own license terms for their software.
For example:
XXXX NON-COMMERCIAL EDUCATIONAL LICENSE Copyright (c) 2013 Prof. Foo. All
rights reserved.
274
You may use and modify this software for any non-commercial purpose within your educational
institution. Teaching, academic research, and personal experimentation are examples of purpose
which can be non-commercial.
You may redistribute the software and modifications to the software for non-commercial purposes,
but only to eligible users of the software (for example, to another university student or faculty
to support joint academic research).
Please don’t do this. Your desire to slightly tweak the terms is harmful to the future software ecosystem.
Also, Unless you are a lawyer, you cannot do this safely!
25.15.9 Licenses for code, content, and data.

Licenses designed for code should not be used to license data or prose.
Don’t use Creative Commons for software, or GPL for a book.
25.15.10 Licensing issues

• Permissive vs share-alike
• Non-commercial and academic Use Only
• Patents
• Use as a web service
25.15.11 Permissive vs share-alike

Some licenses require all derived software to be licensed under terms that are similarly free. Such licenses
are called “Share Alike” or “Copyleft”.
• Licenses in this class include the GPL.
Those that don’t are called “Permissive”
• These include Apache, BSD, and MIT licenses.
If you want your code to be maximally reusable, use a permissive license If you want to force other people
using your code to make derivatives open source, use a copyleft license.
If you want to use code that has a permissive license, it’s safe to use it and keep your code secret. If you
want to use code that has a copyleft license, you’ll have to release your code under such a license.
25.15.12 Academic use only

Some researchers want to make their code free for ‘academic use only’. None of the standard licenses state
this, and this is a reason why academic bespoke licenses proliferate.
However, there is no need for this, in our opinion.
Use of a standard Copyleft license precludes derived software from being sold without also publishing the
source
So use of a Copyleft license precludes commercial use.
This is a very common way of making a business from open source code: offer the code under GPL for
free but offer the code under more permissive terms, allowing for commercial use, for a fee.
25.15.13 Patents
Intellectual property law distinguishes copyright from patents. This is a complex field, which I am far from
qualified to teach!
People who think carefully about intellectual property law distinguish software licenses based on how
they address patents. Very roughly, if a you want to ensure that contributors to your project can’t then go
off and patent their contribution, some licenses, such as the Apache license, protect you from this.
275
25.15.14 Use as a web service
If I take copyleft code, and use it to host a web service, I have not sold the software.
Therefore, under some licenses, I do not have to release any derivative software. This “loophole” in the
GPL is closed by the AGPL (“Affero GPL”)
25.15.15 Library linking

If I use your code just as a library, without modifying it or including it directly in my own code, does the
copyleft term of the GPL apply?
Yes
If you don’t want it to, use the LGPL. (“Lesser GPL”). This has an exception for linking libraries.
25.15.16 Citing software

Almost all software licenses require people to credit you for what they used (“attribution”).
In an academic context, it is useful to offer a statement as to how best to do this, citing which paper to
cite in all papers which use the software.
This is best done with a CITATION file in your repository.
To cite ggplot2 in publications, please use:
H. Wickham. ggplot2: elegant graphics for data analysis. Springer New York, 2009.
A BibTeX entry for LaTeX users is
@Book{, author = {Hadley Wickham}, title = {ggplot2: elegant graphics for data analy-
sis}, publisher = {Springer New York}, year = {2009}, isbn = {978-0-387-98140-6}, url =
{http://had.co.nz/ggplot2/book}, }
25.15.17 Referencing the license in every file

Some licenses require that you include license information in every file. Others do not.
Typically, every file should contain something like:
In [1]: # (C) University College London 2010-2014

# This software is licensed under the terms of the <foo license>
# See <somewhere> for the license details.
Check your license at opensource.org for details of how to apply it to your software. For example, for the
GPL
25.15.18 Choose a license

See GitHub’s advice on how to choose a license
25.15.19 Open source does not equal free maintenance

One common misunderstanding of open source software is that you’ll automatically get loads of contributors
from around the internets. This is wrong. Most open source projects get no commits from anyone else.
Open source does not guarantee your software will live on with people adding to it after you stop working
on it.
Learn more about these issues from the website of the Software Sustainability Institute
276
25.16 Managing software issues
25.16.1 Issues
Code has bugs. It also has features, things it should do.
A good project has an organised way of managing these. Generally you should use an issue tracker.
25.16.2 Some Issue Trackers

There are lots of good issue trackers.
The most commonly used open source ones are Trac and Redmine.
Cloud based issue trackers include Lighthouse and GitHub.
Commercial solutions include Jira.
In this course, we’ll be using the GitHub issue tracker.
25.16.3 Anatomy of an issue

• Reporter
• Description
• Owner
• Type [Bug, Feature]
• Component
• Status
• Severity
25.16.4 Reporting a Bug

The description should make the bug reproducible:
• Version
• Steps
If possible, submit a minimal reproducing code fragment.
25.16.5 Owning an issue

• Whoever the issue is assigned to works next.
• If an issue needs someone else’s work, assign it to them.
25.16.6 Status
• Submitted
• Accepted
• Underway
• Blocked
25.16.7 Resolutions
• Resolved
• Will Not Fix
• Not reproducible
• Not a bug (working as intended)
277
25.16.8 Bug triage
Some organisations use a severity matrix based on:
• Severity [Wrong answer, crash, unusable, workaround, cosmetic. . . ]

• Frequency [All users, most users, some users. . . ]
25.16.9 The backlog

The list of all the bugs that need to be fixed or features that have been requested is called the “backlog”.
25.16.10 Development cycles

Development goes in cycles.
Cycles range in length from a week to three months.
In a given cycle:
• Decide which features should be implemented

• Decide which bugs should be fixed
• Move these issues from the Backlog into the current cycle. (Aka Sprint)
25.16.11 GitHub issues

GitHub doesn’t have separate fields for status, component, severity etc. Instead, it just has labels, which
you can create and delete.
See for example IPython
25.16.12 Exercise - Packaging Greengraph

We previously looked at Greengraph.py, a script that enables us to explore how green space varies as we
move from the city to the countryside:
The Greengraph example was given earlier in the course as a simple python file.
Your task is to transform this into a python package that can be pip installed directly from GitHub.
Remember to include:
• an init .py file

• a setup.py file
• tests
• license and documentation
• command entry point script
You should, in the end, be able to pip install your code on a clean computer, and do something similar
to
greengraph --from London --to Oxford --steps 10 --out graph.png
278
Chapter 26
Construction
26.1 Construction
Software design gets a lot of press (Object orientation, UML, design patterns)
In this session we’re going to look at advice on software construction
This lecture is available as an IPython Notebook
26.1.1 Construction vs Design

For a given piece of code, there exist several different ways one could write it:
• Choice of variable names
• Choice of comments
• Choice of layout
The consideration of these questions is the area of Software Construction.
26.1.2 Low-level design decisions

We will also look at some of the lower-level software design decisions in the context of this section:
• Division of code into subroutines
• Subroutine access signatures
• Choice of data structures for readability
26.1.3 Algorithms and structures

We will not, in this session, be looking at decisions as to how design questions impact performance:
• Choice of algorithms
• Choice of data structures for performance
• Choice of memory layout
We will consider these in a future session.
26.1.4 Architectural design

We will not, in this session, be looking at the large-scale questions of how program components interact, the
stategic choices that govern how software behaves at the large scale:
• Where do objects get made?
• Which objects own or access other objects?
• How can I hide complexity in one part of the code from other parts of the code?
We will consider these in a future session.
279
26.1.5 Construction
So, we’ve excluded most of the exciting topics. What’s left is the bricks and mortar of software: how letters
and symbols are used to build code which is readable.
26.1.6 Literate programming

In literature, books are enjoyable for different reasons:
• The beauty of stories

• The beauty of plots
• The beauty of characters
• The beauty of paragraphs
• The beauty of sentences
• The beauty of words
Software has beauty at these levels too: stories and characters correspond to architecture and object
design, plots corresponds to algorithms, but the rhythm of sentences and the choice of words corresponds to
software construction.
26.1.7 Programming for humans

• Remember you’re programming for humans as well as computers
• A program is the best, most rigourous way to describe an algorithm
• Code should be pleasant to read, a form of scholarly communication
Read CodeComplete
26.2 Setup
This notebook is based on a number of fragments of code, with an implicit context. We’ve made a library
to set up the context so the examples work:
In [1]: %%writefile context.py

array=[]
agt=[]
ws=[]
agents=[]
counter=0
x=Mock()
y=None
agent=Mock()
value=0
bird_types=["Starling", "Hawk"]
import numpy as np
average=np.mean
hawk=Mock()
starling=Mock()
Mock.__sub__=Mock()
sInput="2.0"
280
input ="2.0"
iOffset=1
offset =1
anothervariable=1
flag1=True
variable=1
flag2=False
def do_something(): pass
Mock.__sub__=Mock()
Mock.__abs__=Mock()
chromosome=None
start_codon=None
subsequence=Mock()
transcribe=Mock()
ribe=Mock()
find=Mock()
hawk=Mock()
starling=Mock()
can_see=Mock()
my_name=""
your_name=""
flag1=False
flag2=False
start=0.0
end=1.0
step=0.1
birds=[Mock()]*2
resolution=100
pi=3.141
result= [0]*resolution
import numpy as np
import math
data= [math.sin(y) for y in np.arange(0,pi,pi/resolution)]
import yaml
import os
Writing context.py
26.3 Coding Conventions

26.3.1 One code, many layouts:
Consider the following fragment of python:
In [1]: from context import *
In [2]: import species

def AddToReaction(name, reaction):
reaction.append(species.Species(name))
this could also have been written:
In [3]: from species import Species
281
def add_to_reaction(a_name,
a_reaction):
l_species = Species(a_name)
a_reaction.append( l_species )
26.3.2 So many choices

• Layout
• Naming
• Syntax choices
26.3.3 Layout
In [4]: reaction= {
"reactants": ["H","H","O"],
"products": ["H2O"]
}
In [5]: reaction2=(
{
"reactants":
[
"H",
"H",
"O"
],
"products":
[
"H2O"
]
}
)
26.3.4 Layout choices

• Brace style
• Line length
• Indentation
• Whitespace/Tabs
26.3.5 Naming Conventions

In [6]: class ClassName(object):
def methodName(variable_name):
instance_variable=variable_name
In [7]: class class_name(object):
def method_name(a_variable):
m_instance_variable=a_variable
26.3.6 Hungarian Notation

Prefix denotes type:
In [8]: fNumber= float(sInput) + iOffset
number = float(input) + offset
282
26.3.7 Newlines
• Newlines make code easier to read
• Newlines make less code fit on a screen
Use newlines to describe your code’s rhythm
26.3.8 Syntax Choices

In [9]: anothervariable+=1
if ((variable==anothervariable) and flag1 or flag2): do_something()
In [10]: anothervariable = anothervariable + 1

variable_equality = (variable == anothervariable);
if ((variable_equality and flag1) or flag2):
do_something()
26.3.9 Syntax choices

• Explicit operator precedence
• Compound expressions
• Package import choices
26.3.10 Coding Conventions

You should try to have an agreed policy for your team for these matters.
If your language sponsor has a standard policy, use that.
E.g. Python PEP8
E.g. Google’s guide for R
E.g. Google’s style guide for C++
26.3.11 Lint
There are automated tools which enforce coding conventions and check for common mistakes.
These are called linters
E.g. pip install pep8
In [11]: %%bash
pep8 species.py
species.py:2:6: E111 indentation is not a multiple of four
It is a good idea to run a linter before every commit, or include it in your CI tests.
26.4 Comments
26.4.1 Why comment?
• You’re writing code for people, as well as computers.
• Comments can help you build code, by representing your design
• Comments explain subtleties in the code which are not obvious from the syntax
• Comments explain why you wrote the code the way you did
26.4.2 Bad Comments

“I write good code, you can tell by the number of comments.”
This is wrong.
283
26.4.3 Comments which are obvious
In [2]: counter=counter+1 # Increment the counter

for element in array: # Loop over elements
pass
26.4.4 Comments which could be replaced by better style

In [3]: for i in range(len(agt)): #for each agent
agt[i].theta+=ws[i] # Increment the angle of each agent
#by its angular velocity
agt[i].x+=r*sin(agt[i].theta) #Move the agent by the step-size
agt[i].y+=r*cos(agt[i].theta) #r in the direction indicated
Is good. But:
In [4]: for agent in agents:

agent.turn()
agent.move()
class Agent(object):
def turn(self):
self.direction+=self.angular_velocity;
def move(self):
self.x+=Agent.step_length*sin(self.direction)
self.y+=Agent.step_length*cos(self.direction)
is probably better.
26.4.5 Comments vs expressive code

The proper use of comments is to compensate for our failure to express yourself in code. Note
that I used the word failure. I meant it. Comments are always failures.
– Robert Martin, Clean Code.

I wouldn’t disagree, but still, writing “self-documenting” code is very hard, so do comment if you’re
unsure!
26.4.6 Comments which belong in an issue tracker

In [5]: x.clear() # Code crashes here sometimes
class Agent(object):
pass
# TODO: Implement pretty-printer method
BUT comments that reference issues in the tracker can be good.

E.g.
In [6]: if x.safe_to_clear(): # Guard added as temporary workaround for #32

x.clear()
is OK.
284
26.4.7 Comments which only make sense to the author today
In [7]: agent.turn() # Turtle Power!
agent.move()
agents[:]=[]# Shredder!
26.4.8 Comments which are unpublishable

In [8]: # Stupid supervisor made me write this code
# So I did it while very very drunk.
26.5 Good comments

26.5.1 Pedagogical comments
Code that is good style, but you’re not familiar with, or that colleagues might not be familiar with
In [9]: # This is how you define a decorator in python
def double(decorated_function):
# Here, the result function forms a closure over
# the decorated function
def result_function(input):
return decorated_function(decorated_function(input))
# The returned result is a function
return result_function
@double
def try_me_twice():
pass
26.5.2 Other good comments

Comments which explain coding definitions or reasons for programming choices.
In [10]: def __init__(self):
self.angle=0 # clockwise from +ve y-axis
nonzero_indices = [] # Use sparse model as memory constrained
26.6 Refactoring
26.6.1 Refactoring
To refactor is to:
• Make a change to the design of some software
• Which improves the structure or readability
• But which leaves the actual behaviour of the program completely unchanged.
26.6.2 A word from the Master

Refactoring is a controlled technique for improving the design of an existing code base. Its essence
is applying a series of small behavior-preserving transformations, each of which “too small to be
worth doing”. However the cumulative effect of each of these transformations is quite significant.
By doing them in small steps you reduce the risk of introducing errors. You also avoid having
the system broken while you are carrying out the restructuring - which allows you to gradually
refactor a system over an extended period of time.
285
– Martin Fowler
26.6.3 List of known refactorings

The next few sections will present some known refactorings
We’ll show before and after code, present any new coding techniques needed to do the refactoring, and
describe code smells: how you know you need to refactor.
26.6.4 Replace magic numbers with constants

Smell: Raw numbers appear in your code
Before:
In [2]: data= [math.sin(x) for x in np.arange(0,3.141,3.141/100)]

result= [0]*100
for i in range(100):
for j in range(i+1, 100):
result[j] += data[i] * data[i-j] / 100
after:
In [3]: resolution=100
pi=3.141
data= [math.sin(x) for x in np.arange(0,pi,pi/resolution)]
result= [0]*resolution
for i in range(resolution):
for j in range(i + 1, resolution):
result[j] += data[i] * data[i-j] / resolution
26.6.5 Replace repeated code with a function

Smell: Fragments of repeated code appear
Before:
In [4]: if abs(hawk.facing-starling.facing)<hawk.viewport:
hawk.hunting()
if abs(starling.facing-hawk.facing)<starling.viewport:
starling.flee()
After:
In [5]: def can_see(source,target):

return (source.facing-target.facing)<source.viewport
if can_see(hawk,starling):
hawk.hunting()
if can_see(starling,hawk):
starling.flee()
286
26.6.6 Change of variable name
Smell: Code needs a comment to explain what it is for
Before:
In [6]: z=find(x,y)
if z:
ribe(x)
After:
In [7]: gene = subsequence(chromosome, start_codon)

if gene:
transcribe(gene)
26.6.7 Separate a complex expression into a local variable

Smell: An expression becomes long
In [8]: if ((my_name==your_name) and flag1 or flag2): do_something()
vs
In [9]: same_names= (my_name==your_name)

flags_OK=flag1 or flag2
if same_names and flags_OK: do_something()
26.6.8 Replace loop with iterator

Smell: Loop variable is an integer from 1 to something
Before:
In [10]: sum=0
for i in range(resolution):
sum+=data[i]
After:
In [11]: sum=0
for value in data:
sum+=value
26.6.9 Replace hand-written code with library code

Smell: It feels like surely someone else must have done this at some point
Before:
In [12]: xcoords=[start+i*step for i in range(int((end-start)/step))]
After:

xcoords=np.arange(start,end,step)
See Numpy, Pandas
287
26.6.10 Replace set of arrays with array of structures
Smell: A function needs to work corresponding indices of several arrays:
Before:
In [14]: def can_see(index,source_angles,target_angles,source_viewports):

return abs(source_angles[i]-target_angles[i])<source_viewports[i]
After:

return (source["facing"]-target["facing"])<source["viewport"]
Warning: this refactoring greatly improves readability but can make code slower, depending on memory
layout. Be careful.
26.6.11 Replace constants with a configuration file

Smell: You need to change your code file to explore different research scenarios
Before:
In [16]: flight_speed=2.0 # mph

bounds=[0,0,100,100]
turning_circle=3.0 # m
bird_counts= {"hawk": 5, "starling": 500}
After:
In [17]: %%writefile config.yaml

bounds: [0,0,100,100]
counts:
hawk: 5
starling: 500
speed: 2.0
turning_circle: 3.0
Writing config.yaml
In [18]: config=yaml.load(open("config.yaml"))
See YAML and PyYaml and Python OS
26.6.12 Replace global variables with function arguments

Smell: A global variable is assigned and then used inside a called function:
In [19]: viewport=pi/4
if hawk.can_see(starling):
hawk.hunt(starling)
class Hawk(object):
def can_see(self,target):
return (self.facing-target.facing)<viewport
Becomes:
288
In [20]: viewport=pi/4
if hawk.can_see(starling,viewport):
hawk.hunt(starling)
class Hawk(object):
def can_see(self,target,viewport):
return (self.facing-target.facing)<viewport
26.6.13 Merge neighbouring loops

Smell: Two neighbouring loops have the same for statement
In [21]: for bird in birds:

bird.build_nest()
for bird in birds:

bird.lay_eggs()
Becomes:
In [22]: for bird in birds:

bird.build_nest()
bird.lay_eggs()
26.6.14 Break a large function into smaller units

Smell: A function or subroutine no longer fits on a page in your editor Smell: A line of code is indented more
than three levels Smell: A piece of code interacts with the surrounding code through just a few variables
Before:
In [23]: def do_calculation():

for predator in predators:
for prey in preys:
if predator.can_see(prey):
predator.hunt(prey)
if predator.can_reach(prey):
predator.eat(prey)
After:
In [24]: def do_calculation():

for predator in predators:
for prey in preys:
predate(predator, prey)
def predate(predator,prey):
if predator.can_see(prey):
predator.hunt(prey)
if predator.can_reach(prey):
predator.eat(prey)
26.6.15 Separate code concepts into files or modules

Smell: You find it hard to locate a piece of code
Smell: You get a lot of version control conflicts
Before:
289
In [25]: class One(object):
pass
class Two(object):
def __init__():
self.child = One()
After:
In [26]: %%writefile anotherfile.py

class One(object):
pass
Writing anotherfile.py
In [27]: from anotherfile import One
class Two(object):
def __init__():
self.child = One()
26.6.16 Refactoring is a safe way to improve code

You may think you can see how to rewrite a whole codebase to be better
However, you may well get lost halfway through the exercise.
By making the changes as small, reversible, incremental steps, you can reach your target design more
reliably.
26.6.17 Tests and Refactoring

Badly structured code cannot be unit tested. There are no “units”.
Before refactoring, ensure you have a robust regression test.
This will allow you to Refactor with confidence
As you refactor, if you create any new units (functions, modules, classes), add new tests for them.
26.6.18 Refactoring Summary

• Replace magic numbers with constants
• Replace repeated code with a function
• Change of variable/function/class name
• Replace loop with iterator
• Replace hand-written code with library code
• Replace set of arrays with array of structures
• Replace constants with a configuration file
• Replace global variables with function arguments
• Break a large function into smaller units
• Separate code concepts into files or modules
And many more

Read The Refactoring Book
290
26.7 Introduction to Objects
26.7.1 Classes: User defined types
def __init__(self,name,age):
self.name=name
self.age=age
def grow_up(self):
self.age+=1
james=Person("James",37)
james.home="London"
26.7.2 Declaring a class

Class: A user-defined type
In [2]: class MyClass(object):

pass
26.7.3 Object instances

Instance: A particular object instantiated from a class.
In [3]: my_object = MyClass()
26.7.4 Method
Method: A function which is “built in” to a class

def someMethod(self, argument):
pass
my_object=MyClass()
my_object.someMethod(value)
26.7.5 Constructor
Constructor: A special method called when instantiating a new object

def __init__(self, argument):
pass
my_object = MyClass(value)
26.7.6 Member Variable

Member variable: a value stored inside an instance of a class.
291
def __init__(self):
self.member = "Value"
my_object = MyClass()
assert(my_object.member == "Value")
26.8 Object refactorings

26.8.1 Replace add-hoc structure with user defined classes
Smell: A data structure made of nested arrays and dictionaries becomes unwieldy
Before:
In [8]: from random import random

birds = [{"position": random(),
"velocity": random(),
"type": kind} for kind in bird_types]
average_position = average([bird["position"] for bird in birds])
After:
In [9]: class Bird(object):

def __init__(self,type):
from random import random
self.type = type
self.position = random()
self.velocity = random()
birds = [Bird(type) for type in bird_types]

average_position = average([bird.position for bird in birds])
26.8.2 Replace function with a method

Smell: A function is always called with the same kind of thing
Before:

return (source.facing-target.facing)<source.viewport
if can_see(hawk,starling):
hawk.hunt()
After:
In [11]: class Bird(object):

def can_see(self,target):
return (self.facing-target.facing)<self.viewport
if hawk.can_see(starling):
hawk.hunt()
292
26.8.3 Replace method arguments with class members
Smell: A variable is nearly always used in arguments to a class.

def __init__(self, genes):
self.genes=genes
def reproduce_probability(self,age): pass
def death_probability(self,age): pass
def emigrate_probability(self,age): pass
After:

def __init__(self, genes, age):
self.age=age
self.genes=genes
def reproduce_probability(self): pass
def death_probability(self): pass
def emigrate_probability(self): pass
26.8.4 Replace global variable with class and member

Smell: A global variable is referenced by a few functions
In [14]: name="James"
birthday=[19,10,76]
today=[29,10]
if today==birthday[0:2]:
print "Happy Birthday, ", name
else:
print "No birthday for you today."
No birthday for you today.

def __init__(self, birthday, name):
self.birth_day=birthday[0]
self.birth_month=birthday[1]
self.birth_year=birthday[2]
self.name=name
def check_birthday(self, today_day, today_month):
if not self.birth_day == today_day:
return False
if not self.birth_month == today_month:
return False
return True
def greet_appropriately(self, today):
if self.check_birthday(*today):
print "Happy Birthday", self.name
else:
print "No birthday for you."
james=Person([19,10,76],"James")
james.greet_appropriately([29,10])
No birthday for you.
293
26.8.5 Object Oriented Refactoring Summary
• Replace ad-hoc structure with a class
• Replace function with a method
• Replace method argument with class member
• Replace global variable with class data
26.9 Design
26.9.1 Design
In this session, we will finally discuss the thing most people think of when they refer to “Software Engineer-
ing”: the deliberate design of software. We will discuss processes and methodologies for planned development
of large-scale software projects: Software Architecture.
26.9.2 Object-Oriented Design

The software engineering community has, in large part, focused on an object-oriented approach to the design
and development of large scale software systems. The basic concepts of object orientation are necessary to
follow much of the software engineering conversation.
26.9.3 Design processes

In addition to object-oriented architecture, software engineers have focused on the development of processes
for robust, reliable software development. These codified ways of working hope to enable organisations to
repeatably and reliably complete complex software projects in a way that minimises both development and
maintainance costs, and meets user requirements.
26.9.4 Design and research

Software engineering theory has largely been developed in the context of commercial software companies.
The extent to which the practices and processes developed for commercial software are applicable in a
research context is itself an active area of research.
26.10 More on Objects

26.10.1 Object Based Programming Recap
We saw last lesson how to
• create our own classes

• store data with member variables
• add functionality with member functions
• initialise objects with constructors
26.10.2 Class design

The concepts we have introduced are common between different object oriented languages. Thus, when we
design our program using these concepts, we can think at an architectural level, independent of language
syntax.
In [1]: class Particle(object):

def __init__(self, position, velocity):
self.position=position
self.velocity=velocity
294
def move(self, delta_t):
self.position+= self.velocity*delta_t
class Particle {
std::vector<double> position;
std::vector<double> velocity;
Particle(std::vector<double> position, std::vector<double> velocity);
void move(double delta_t);
}
type particle
real :: position
real :: velocity
contains
procedure :: init
procedure :: move
end type particle
26.10.3 UML
UML is a conventional diagrammatic notation used to describe “class structures” and other higher level
aspects of software design.
Computer scientists get worked up about formal correctness of UML diagrams and learning the conven-
tions precisely. Working programmers can still benefit from using UML to describe their designs.
26.10.4 YUML
We can see a YUML model for a Particle class with position and velocity data and a move() method
using the YUML online UML drawing tool.
http://yuml.me/diagram/boring/class/[Particle|position;velocity|move%28%29
Here’s how we can use Python code to get an image back from YUML:

import IPython
In [3]: def yuml(model):

result=requests.get("http://yuml.me/diagram/boring/class/"+model)
return IPython.core.display.Image(result.content)
In [4]: yuml("[Particle|position;velocity|move()]")
Out[4]:
295
26.11 Information Hiding
Sometimes, our design for a program would be broken if users start messing around with variables we don’t
want them to change.
Robust class design requires consideration of which subroutines are intended for users to use, and which
are internal. Languages provide features to implement this: access control.
In python, we use leading underscores to control whether member variables and methods can be accessed
from outside the class.

def __init__(self):
self.__private_data=0
self._private_data=0
self.public_data=0
def __private_method(self): pass

def _private_method(self): pass
def public_method(self): pass
def called_inside(self):
self.__private_method()
self._private_method()
self.__private_data=1
self._private_data=1
MyClass().called_inside()
MyClass()._private_method() # Works, but forbidden by convention
MyClass().public_method() # OK
print MyClass()._private_data
print MyClass().public_data
0
0
In [6]: MyClass().__private_method() # Generates error
---------------------------------------------------------------------------
<ipython-input-6-f273bcb10b88> in <module>()
----> 1 MyClass(). private method() # Generates error
AttributeError: ’MyClass’ object has no attribute ’ private method’
In [7]: print MyClass().__private_data # Generates error
---------------------------------------------------------------------------
296
<ipython-input-7-9da94df448bc> in <module>()
----> 1 print MyClass(). private data # Generates error
AttributeError: ’MyClass’ object has no attribute ’ private data’
26.11.1 Property accessors

Python provides a mechanism to make functions appear to be variables. This can be used if you want to
change the way a class is implemented without changing the interface:

def __init__(self):
self.name = "James Hetherington"
assert(Person().name == "James Hetherington")
becomes:

def __init__(self):
self._first = "James"
self._second = "Hetherington"
@property
def name(self):
return self._first + " " + self._second
assert(Person().name == "James Hetherington")
Note that the code behaves the same way to the outside user. The implementation detail is hidden by
private variables. In languages without this feature, such as C++, it is best to always make data private,
and always access data through functions:

def __init__(self):
self._name = "James Hetherington"
def name(self):
return self._name
assert(Person().name() == "James Hetherington")
But in Python this is unnecessary.
26.11.2 Class Members

Class, or static members, belong to the class as a whole, and are shared between instances.
In [11]: class Counted(object):

number_created=0
def __init__(self):
Counted.number_created+=1
@classmethod
def howMany(cls):
return cls.number_created
297
Counted.howMany() # 0
x=Counted()
z=[Counted() for x in range(5)]
Out[11]: 6
26.11.3 Object-based vs Object-Oriented

So far we have seen only object-based programming, not object-oriented programming.
Using Objects doesn’t mean your code is object-oriented.
To understand object-oriented programming, we need to introduce polymorphism and inheritance.
26.12 Inheritance
• Inheritance allows related classes to share code
• Inheritance allows a program to reflect the ontology of kinds of thing in a program.
26.12.1 Ontology and inheritance

• A bird is a kind of animal
• An eagle is a kind of bird
• A starling is also a kind of bird

• All animals can be born and die
• Only birds can fly (Ish.)
• Only eagles hunt

• Only starlings flock
26.12.2 Inheritance in python

In [12]: class Animal(object):
def beBorn(self): print "I exist"
def die(self): print "Argh!"
class Bird(Animal):
def fly(self): print "Whee!"
class Eagle(Bird):
def hunt(self): print "I’m gonna eatcha!"
Eagle().beBorn()
Eagle().hunt()
I exist
I’m gonna eatcha!
298
26.12.3 Inheritance terminology
• A derived class derives from a base class
• A subclass inherits from a superclass
(These are different terms for the same thing.)
• Eagle is a subclass of the Animal superclass.

• Animal is the base class of the Eagle derived class
26.12.4 Inheritance and constructors

def __init__(self, age):
self.age=age
class Person(Animal):
def __init__(self, age, name):
super(Person, self).__init__(age)
self.name=name
26.12.5 Inheritance UML diagrams

UML shows inheritance with an open triangular arrow pointing from subclass to superclass.
In [14]: yuml("[Animal]^-[Bird],[Bird]^-[Eagle],[Bird]^-[Starling]%")
Out[14]:
26.12.6 Aggregation vs Inheritance

If one object has or owns one or more objects, this is not inheritance.
For example, in my solution to the Boids task from last week, the overal Model owned several Boids, and
each Boid owned two 2-vectors, one for position and one for velocity.
26.12.7 Aggregation in UML

The Boids situation can be represented thus:
In [15]: yuml("[Model]<>-*>[Boid],[Boid]position++->[Vector],[Boid]velocity++->[Vector]%")
Out[15]:
299
The open diamond indicates Aggregation, the closed diamond composition. (A given boid might
belong to multiple models, a given position vector is forever part of the corresponding Boid.)
The asterisk represents cardinality, a model may contain multiple Boids.
26.12.8 Refactoring to inheritance

Smell: Repeated code between two classes which are both ontologically subtypes of something
Before:

def __init__(self, age, job):
self.age = age
self.job = job
def birthday(self):
self.age += 1
class Pet(object):
def __init__(self, age, owner):
self.age = age
self.owner = owner
def birthday(self):
self.age += 1
After:

def __init__(self, age):
self.age = age
def birthday(self):
self.age += 1
class Person(Animal):
def __init__(self, age, job):
self.job = job
super(Person, self).__init__(age)
26.13 Polymorphism
26.13.1 Polymorphism
In [18]: class Dog(object):
def noise(self):
return "Bark"
class Cat(object):
def noise(self):
return "Miaow"
300
class Pig(object):
def noise(self): return "Oink"
class Cow(object):
def noise(self): return "Moo"
animals=[Dog(), Dog(), Cat(), Pig(), Cow(), Cat()]

for animal in animals:
print animal.noise()
Bark
Bark
Miaow
Oink
Moo
Miaow
This will print “Bark Bark Miaow Oink Moo Miaow”

If two classes support the same method, but it does different things for the two classes, then if an object is
of an unknown class, calling the method will invoke the version for whatever class the instance is an instance
of.
26.13.2 Polymorphism and Inheritance

Often, polymorphism uses multiple derived classes with a common base class. However, duck typing in
Python means that all that is required is that the types support a common Concept (Such as iterable, or
container, or, in this case, the Noisy concept.)
A common base class is used where there is a likely default that you want several of the derived classes
to have.

def noise(self): return "I don’t make a noise."
class Dog(Animal):
def noise(self): return "Bark"
class Worm(Animal):
pass
class Poodle(Animal):
pass
animals=[Dog(), Worm(), Pig(), Cow(), Poodle()]

for animal in animals:
print animal.noise()
Bark
I don’t make a noise.
Oink
Moo
I don’t make a noise.
26.13.3 Undefined Functions and Polymorphism

In the above example, we put in a dummy noise for Animals that don’t know what type they are.
301
Instead, we can explicitly deliberately leave this undefined, and we get a crash if we access an undefined
method.
In [20]: class Animal(object): pass
class Worm(Animal): pass
In [21]: Worm().noise() # Generates error
---------------------------------------------------------------------------
<ipython-input-21-048812fbd3ee> in <module>()
----> 1 Worm().noise() # Generates error
AttributeError: ’Worm’ object has no attribute ’noise’
26.13.4 Refactoring to Polymorphism

Smell: a function uses a big set of if statements or a case statement to decide what to do:
Before:

def __init__(self,type):
self.type=type
def noise(self):
if self.type=="Dog":
return "Bark"
elif self.type=="Cat":
return "Miaow"
elif self.type=="Cow":
return "Moo"
which is better replaced by the code above.
26.13.5 Interfaces and concepts

In C++, it is common to define classes which declare dummy methods, called “virtual” methods, which
specify the methods which derived classes must implement. Classes which define these methods, which
cannot be instantiated into actual objects, are called “abstract base” classes or “interfaces”.
Python’s Duck Typing approach means explicitly declaring these is unnesssary: any class concept which
implements appropriately named methods will do. These as user-defined concepts, just as “iterable” or
“container” are built-in Python concepts. A class is said to “implement and interface” or “satisfy a concept”.
26.13.6 Interfaces in UML

Interfaces implementation in UML is indicated thus:
In [23]: yuml("[<<Animal>>]^-.-[Dog]")
Out[23]:
302
26.13.7 Further UML
UML is a much larger diagram language than the aspects we’ve shown here.
• Message sequence charts show signals passing back and forth between objects (Web Sequence Diagrams)
• Entity Relationship Diagrams can be used to show more general relationships between things in a
system
26.14 Patterns
26.14.1 Class Complexity
We’ve seen that using object orientation can produce quite complex class structures, with classes owning
each other, instantiating each other, and inheriting from each other.
There are lots of different ways to design things, and decisions to make.
Should I inherit from this class, or own it as a member variable? (“is a” vs “has a”)
26.14.2 Design Patterns

Programmers have noticed that there are certain ways of arranging classes that work better than others.
These are called “design patterns”.
They were first collected on one of the world’s first Wikis, as the Portland Pattern Repository
26.14.3 Reading a pattern

A description of a pattern in a book such as the Gang Of Four book usually includes:
• Intent
• Motivation
• Applicability
• Structure
• Participants
• Collaborations
• Consequences
• Implementation
• Sample Code
26.14.4 Introducing Some Patterns

There are lots and lots of design patterns, and it’s a great literature to get into to read about design questions
in programming and learn from other people’s experience.
We’ll just show a few in this session:
• Factory Method
303
• Builder
• Handle-Body
• Strategy
26.15 Factory Pattern

26.15.1 Factory Pattern
Here’s what the Gang of Four Book says about Factory Method:
Intent: Define an interface for creating an object, but let subclasses decide which class to instantiate.
Factory Method lets a class defer instantiation to subclasses.
Applicability: Use the Factory method pattern when:
• A class can’t anticipate the class of objects it must create
• A class wants its subclasses to specify the objects it creates
This is pretty hard to understand, so let’s look at an example.
26.15.2 Factory UML

In [1]: yuml("[Product]^-[ConcreteProduct], [Creator| (v) FactoryMethod()]^-[ConcreteCreator| FactoryMet
---------------------------------------------------------------------------
<ipython-input-1-8ac5e7e89ac3> in <module>()
----> 1 yuml("[Product]^-[ConcreteProduct], [Creator| (v) FactoryMethod()]^-[ConcreteCreator| Factor
NameError: name ’yuml’ is not defined
26.15.3 Factory Example

An “agent based model” is one like the Boids model from last week: agents act and interact under certain
rules. Complex phenomena can be described by simple agent behaviours.
In [2]: class AgentModel(object):
def simulate(self):
for agent in agents:
for target in agents:
agent.interact(target)
agent.simulate()
26.15.4 Agent model constructor

This logic is common to many kinds of Agent based model, so we can imagine a common class for agent
based models: the constructor could parse a configuration specifying how many agents of each type to create,
their initial conditions and so on.
However, this common constructor doesn’t know what kind of agent to create; as a common base, it
could be a model of boids, or the agents could be remote agents on foreign servers, or they could even be
physical hardware robots connected to the driving model over Wifi!
We need to defer the construction of the agents. We can do this with polymorphism: each derived class
of the ABM can have an appropriate method to create its agents:
304
In [3]: class AgentModel(object):
def __init__(self, config):
self.agents=[]
for agent_config in config:
self.agents.append(self.create(**agent_config))
This is the factory method pattern: a common design solution to the need to defer the construction of
daughter objects to a derived class.
26.15.5 Agent derived classes

In [4]: class BirdModel(AgentModel):
def create(self, species):
return Boid(species)
Agents are the base product, boids or robots are a ConcreteProduct.
In [5]: class WebAgentFactory(AgentModel):

def __init__(self, url):
self.url=url
connection=AmazonCompute.connect(url)
AgentModel.__init__(self)
def create(self, species):
return OnlineAnimal(species, connection)
There is no need to define an explicit base interface for the “Agent” concept in Python: anything that
responds to “simulate” and “interact” methods will do: this is our Agent concept.
26.15.6 Refactoring to Patterns

I personally have got into a terrible tangle trying to make base classes which somehow “promote” themselves
into a derived class based on some code in the base class.
This is an example of an “Antipattern”: like a Smell, this is a recognised Wrong Way of doing things.
What I should have written was a Creator with a FactoryMethod.
26.16 Builder
26.16.1 Builder Pattern

Intent: Separate the steps for constructing a complex object from its final representation.
In [7]: yuml("[Director|Construct()]<>->[Builder| (a) BuildPart()], [Builder]^-[ConcreteBuilder| BuildPa
---------------------------------------------------------------------------
<ipython-input-7-20d53252b1a7> in <module>()
----> 1 yuml("[Director|Construct()]<>->[Builder| (a) BuildPart()], [Builder]^-[ConcreteBuilder| Bui
NameError: name ’yuml’ is not defined
305
26.16.2 Builder example
Let’s continue our Agent Based modelling example.
There’s a lot more to defining a model than just adding agents of different kinds: we need to define
boundary conditions, specify wind speed or light conditions.
We could define all of this for an imagined advanced Model with a very very long constructor, with lots
of optional arguments:
In [8]: class Model(object):
def __init__(self, xsize, ysize,
agent_count, wind_speed,
agent_sight_range, eagle_start_location):
pass
26.16.3 Builder preferred to complex constructor

However, long constructors easily become very complicated. Instead, it can be cleaner to define a Builder
for models. A builder is like a deferred factory: each step of the construction process is implemented as an
individual method call, and the completed object is returned when the model is ready.
In [9]: Model=Mock() # Create a temporary mock so the example works!
In [10]: class ModelBuilder(object):
def start_model(self):
self.model=Model()
def set_bounds(self, xlim, ylim):
self.model.xlim=xlim
self.model.ylim=ylim
def add_agent(self, xpos, ypos):
pass # Implementation here
def finish(self):
return self.model
Inheritance of an Abstract Builder for multiple concrete builders could be used where there might be
multiple ways to build models with the same set of calls to the builder: for example a version of the model
builder yielding models which can be executed in parallel on a remote cluster.
26.16.4 Using a builder

In [11]: builder=ModelBuilder()
builder.start_model()
builder.set_bounds(500,500)
builder.add_agent(40,40)
builder.add_agent(400,100)
model=builder.finish()
model.simulate()
Out[11]: <Mock name=’mock().simulate()’ id=’4429984976’>
26.16.5 Avoid staged construction without a builder.

We could, of course, just add all the building methods to the model itself, rather than having the model be
yielded from a separate builder.
This is an antipattern that is often seen: a class whose init constructor alone is insufficient for it to
be ready to use. A series of methods must be called, in the right order, in order for it to be ready to use.
This results in very fragile code: its hard to keep track of whether an object instance is “ready” or not.
Use the builder pattern to keep deferred construction in control.
306
26.16.6 Builder Message Sequence
Note: Need to add message sequence chart here
26.17 Strategy
In [12]: from numpy import linspace,exp,log,sqrt, array
import math
from scipy.interpolate import UnivariateSpline
from scipy.signal import lombscargle
from scipy.integrate import cumtrapz
from numpy.fft import rfft,fft,fftfreq
import csv
from datetime import datetime
import requests
26.18 Strategy Pattern

Define a family of algorithms, encapsulate each one, and make them interchangeable. Strategy lets the
algorithm vary independently from clients that use it.
26.18.1 Strategy pattern example: sunspots

Consider the sequence of sunspot observations:
In [14]: def load_sunspots():

url_base="http://www.quandl.com/api/v1/datasets/SIDC/SUNSPOTS_A.csv"
x=requests.get(url_base,params={’trim_start’:’1700-12-31’,
’trim_end’:’2014-01-01’,
’sort_order’:’asc’})
data=csv.reader(StringIO(x.text)) #Convert requests result to look
#like a file buffer before
# reading with CSV
data.next() # Skip header row
return [float(row[1]) for row in data]
In [15]: spots=load_sunspots()
plt.plot(spots)
Out[15]: [<matplotlib.lines.Line2D at 0x10b28c3d0>]
307
26.18.2 Sunspot cycle has periodicity
In [16]: spectrum=rfft(spots)
### "FFT figure"

plt.figure()
plt.plot(abs(spectrum))
plt.savefig(’fixed.png’)
308
26.18.3 Years are not constant length
There’s a potential problem with this analysis however:
• Years are not constant length

• Leap years exist
• But, the Fast Fourier Transform assumes evenly spaced intervals
26.18.4 Uneven time series

The Fast Fourier Transform cannot be applied to uneven time series.
We could:
• Ignore this problem, and assume the effect is small

• Interpolate and resample to even times
• Use a method which is robust to unevenly sampled series, such as LSSA
We also want to find the period of the strongest periodic signal in the data, there are various different
methods we could use for this also, such as integrating the fourier series by quadrature to find the mean
frequency, or choosing the largest single value.
26.18.5 Uneven time series design

We could implement a base class for our common code between the different approaches, and define derived
classes for each different algorithmic approach. However, this has drawbacks:
• The constructors for each derived class will need arguments for all the numerical method’s control
parameters, such as the degree of spline for the interpolation method, the order of quadrature for
integrators, and so on.
309
• Where we have multiple algorithmic choices to make (interpolator, periodogram, peak finder. . . ) the
number of derived classes would explode: class SunspotAnalyzerSplineFFTTrapeziumNearMode is
a bit unweildy.
• The algorithmic choices are not then available for other projects
• This design doesn’t fit with a clean Ontology of “kinds of things”: there’s no Abstract Base for
spectrogram generators. . .
26.18.6 Strategy Pattern for Algorithms

• We implement each algorithm for generating a spectrum as its own Strategy class.
• They all implement a common interface
• Arguments to strategy constructor specify parameters of algorithms, such as spline degree
• One strategy instance for each algorithm is passed to the constructor for the overall analysis

First, we’ll define a helper class for our time series.
In [17]: class Series(object):

"""Enhance NumPy N-d array with some helper functions for clarity"""
def __init__(self, data):
self.data=array(data)
self.count=self.data.shape[0]
self.start=self.data[0,0]
self.end=self.data[-1,0]
self.range=self.end-self.start
self.step=self.range/self.count
self.times=self.data[:,0]
self.values=self.data[:,1]
self.plot_data=[self.times,self.values]
self.inverse_plot_data=[1.0/self.times[20:], self.values[20:]]

Then, our class which contains the analysis code, except the numerical methods
In [18]: class AnalyseSunspotData(object):

def format_date(self, date):
date_format="%Y-%m-%d"
return datetime.strptime(date,date_format)
def load_data(self):
start_date_str=’1700-12-31’
end_date_str=’2014-01-01’
self.start_date=self.format_date(start_date_str)
end_date=self.format_date(end_date_str)
url_base=("http://www.quandl.com/api/v1/datasets/"+
"SIDC/SUNSPOTS_A.csv")
x=requests.get(url_base,params={’trim_start’:start_date_str,
’trim_end’:end_date_str,
’sort_order’:’asc’})
secs_per_year=(datetime(2014,1,1)-datetime(2013,1,1)
).total_seconds()
data=csv.reader(StringIO(x.text)) #Convert requests
#result to look
310
#like a file buffer before
#reading with CSV
data.next() # Skip header row
self.series=Series([[
(self.format_date(row[0])-self.start_date
).total_seconds()/secs_per_year
,float(row[1])] for row in data])
def __init__(self, frequency_strategy):

self.load_data()
self.frequency_strategy=frequency_strategy
def frequency_data(self):
return self.frequency_strategy.transform(self.series)

Our existing simple fourier strategy
In [19]: class FourierNearestFrequencyStrategy(object):
def transform(self, series):
transformed=fft(series.values)[0:series.count/2]
frequencies=fftfreq(series.count, series.step)[0:series.count/2]
return Series(zip(frequencies, abs(transformed)/series.count))

A strategy based on interpolation to a spline
In [20]: class FourierSplineFrequencyStrategy(object):
def next_power_of_two(self, value):
"Return the next power of 2 above value"
return 2**(1+int(log(value)/log(2)))
def transform(self, series):

spline=UnivariateSpline(series.times, series.values)
# Linspace will give us *evenly* spaced points in the series
fft_count= self.next_power_of_two(series.count)
points=linspace(series.start,series.end,fft_count)
regular_xs=[spline(point) for point in points]
transformed=fft(regular_xs)[0:fft_count/2]
frequencies=fftfreq(fft_count,
series.range/fft_count)[0:fft_count/2]
return Series(zip(frequencies, abs(transformed)/fft_count))

A strategy using the Lomb-Scargle Periodogram
In [21]: class LombFrequencyStrategy(object):
def transform(self,series):
frequencies=array(linspace(1.0/series.range,
0.5/series.step,series.count))
result= lombscargle(series.times,
series.values,2.0*math.pi*frequencies)
return Series(zip(frequencies, sqrt(result/series.count)))
311
Define our concrete solutions with particular strategies
In [22]: fourier_model=AnalyseSunspotData(FourierSplineFrequencyStrategy())
lomb_model=AnalyseSunspotData(LombFrequencyStrategy())
nearest_model=AnalyseSunspotData(FourierNearestFrequencyStrategy())

Use these new tools to compare solutions
In [23]: comparison=fourier_model.frequency_data().inverse_plot_data+[’r’]
comparison+=lomb_model.frequency_data().inverse_plot_data+[’g’]
comparison+=nearest_model.frequency_data().inverse_plot_data+[’b’]
deviation=365*(fourier_model.series.times-linspace(
fourier_model.series.start,
fourier_model.series.end,
fourier_model.series.count))
26.18.14 Comparison of different algorithms for frequency spectrum of

sunspots.
In [24]: plt.plot(*comparison)
plt.xlim(0,16)
Out[24]: (0, 16)
312
26.18.15 Deviation of year length from average
In [25]: plt.plot(deviation)
Out[25]: [<matplotlib.lines.Line2D at 0x10b626ed0>]
26.19 Model-View-Controller
26.19.1 Separate graphics from science!
Whenever we are coding a simulation or model we want to:
• Implement the maths of the model

• Visualise, plot, or print out what is going on.
We often see scientific programs where the code which is used to display what is happening is mixed up
with the mathematics of the analysis. This is hard to understand.
We can do better by separating the Model from the View, and using a “Controller” to manage them.
26.19.2 Model
class Model(object):
def __init__(self):
self.positions=np.random.rand(100,2)
self.speeds=np.random.rand(100,2)+np.array([-0.5,-0.5])[np.newaxis,:]
self.deltat=0.01
def simulation_step(self):
313
self.positions += self.speeds * self.deltat
def agent_locations(self):
return self.positions
26.19.3 View
In [27]: class View(object):
def __init__(self, model):
self.figure=plt.figure()
axes=plt.axes()
self.model=model
self.scatter=axes.scatter(model.agent_locations()[:,0],
model.agent_locations()[:,1])
def update(self):
self.scatter.set_offsets(self.model.agent_locations())
26.19.4 Controller
In [28]: class Controller(object):
def __init__(self):
self.model=Model() # Or use Builder
self.view=View(self.model)
def animate(frame_number):
self.model.simulation_step()
self.view.update()
self.animator=animate
def go(self):
anim = animation.FuncAnimation(self.view.figure, self.animator, frames=200, interval=50
return anim
In [29]: contl=Controller()

314
%matplotlib inline
contl.go()
Out[30]: <matplotlib.animation.FuncAnimation at 0x10b76fc90>
26.20 Exercise: Refactoring The Bad Boids

26.20.1 Bad Boids
I have written some very bad code implementing our Boids flocking example.
Here’s the Github link.
Please fork it on GitHub, and clone your fork.
git clone git@github.com:yourname/bad-boids.git

# OR git clone https://github.com/yourname/bad-boids.git
For the Exercise, you should start from the GitHub repository, but here’s my terrible code:
In [1]: """
A deliberately bad implementation of [Boids](http://dl.acm.org/citation.cfm?doid=37401.37406)
for use as an exercise on refactoring.
"""

import random
315
# Deliberately terrible code for teaching purposes
boids_x=[random.uniform(-450,50.0) for x in range(50)]

boids_y=[random.uniform(300.0,600.0) for x in range(50)]
boid_x_velocities=[random.uniform(0,10.0) for x in range(50)]
boid_y_velocities=[random.uniform(-20.0,20.0) for x in range(50)]
boids=(boids_x,boids_y,boid_x_velocities,boid_y_velocities)
def update_boids(boids):
xs,ys,xvs,yvs=boids
# Fly towards the middle
for i in range(len(xs)):
for j in range(len(xs)):
xvs[i]=xvs[i]+(xs[j]-xs[i])*0.01/len(xs)
yvs[i]=yvs[i]+(ys[j]-ys[i])*0.01/len(xs)
# Fly away from nearby boids
if (xs[j]-xs[i])**2 + (ys[j]-ys[i])**2 < 100:
xvs[i]=xvs[i]+(xs[i]-xs[j])
yvs[i]=yvs[i]+(ys[i]-ys[j])
# Try to match speed with nearby boids
if (xs[j]-xs[i])**2 + (ys[j]-ys[i])**2 < 10000:
xvs[i]=xvs[i]+(xvs[j]-xvs[i])*0.125/len(xs)
yvs[i]=yvs[i]+(yvs[j]-yvs[i])*0.125/len(xs)
# Move according to velocities
xs[i]=xs[i]+xvs[i]
ys[i]=ys[i]+yvs[i]
figure=plt.figure()
axes=plt.axes(xlim=(-500,1500), ylim=(-500,1500))
scatter=axes.scatter(boids[0],boids[1])
def animate(frame):
update_boids(boids)
scatter.set_offsets(zip(boids[0],boids[1]))
anim = animation.FuncAnimation(figure, animate,

frames=200, interval=50)
If you go into your folder and run the code:
cd bad_boids
python bad_boids.py
You should be able to see some birds flying around, and then disappearing as they leave the window.
316
26.20.2 Your Task
Transform bad boids gradually into better code, while making sure it still works, using a Refactoring
approach.
26.20.3 A regression test

First, have a look at the regression test I made.
To create it, I saved out the before and after state for one iteration of some boids, using ipython:
import yaml
import boids
from copy import deepcopy
before=deepcopy(boids.boids)
boids.update_boids(boids.boids)
after=boids.boids
fixture={"before":before,"after":after}
fixture_file=open("fixture.yml",’w’)
fixture_file.write(yaml.dump(fixture))
fixture_file.close()
26.20.4 A regression test

Then, I used the fixture file to define the test:
from boids import update_boids

import os
import yaml
def test_bad_boids_regression():
regression_data=yaml.load(open(os.path.join(os.path.dirname(__file__),’fixture.yml’)))
boid_data=regression_data["before"]
update_boids(boid_data)
assert_equal(regression_data["after"],boid_data)
26.20.5 Make the regression test fail

Check the tests pass:
nosetests
Edit the file to make the test fail, see the fail, then reset it:
git checkout boids.py
26.20.6 Start Refactoring

Look at the code, consider the list of refactorings, and make changes
Each time, do a git commit on your fork, and write a commit message explaining the refactoring you did.
Try to keep the changes as small as possible.
If your refactoring creates any units, (functions, modules, or classes) write a unit test for the unit: it
is a good idea to get away from regression testing as soon as you can.
317
Chapter 27
Advanced Python Programming
. . . or, how to avoid repeating yourself.
27.1 Avoid Boiler-Plate

Code can often be annoyingly full of “boiler-plate” code: characters you don’t really want to have to type.
Not only is this tedious, it’s also time-consuming and dangerous: unnecessary code is an unnecessary
potential place for mistakes.
There are two important phrases in software design that we’ve spoken of before in this context:
Once And Only Once
Don’t Repeat Yourself (DRY)
All concepts, ideas, or instructions should be in the program in just one place. Every line in the program
should say something useful and important.
We refer to code that respects this principle as DRY code.
In this chapter, we’ll look at some techniques that can enable us to refactor away repetitive code.
Since in many of these places, the techniques will involve working with functions as if they were vari-
ables, we’ll learn some functional programming. We’ll also learn more about the innards of how Python
implements classes.
We’ll also think about how to write programs that generate the more verbose, repetitive program we
could otherwise write. We call this metaprogramming.
27.2 Functional programming

27.2.1 Functional Programming
Understanding to think in a functional programming style is almost as important as object orientation for
building DRY, clear scientific software, and is just as conceptually difficult.
Programs are composed of functions: they take data in (which we call parameters or arguments) and
send data out (through return statements.)
A conceptual trick which is often used by computer scientists to teach the core idea of functional pro-
gramming is this: to write a program, in theory, you only ever need functions with one argument, even
when you think you need two or more. Why?
Let’s define a program to add two numbers:
In [1]: def add(a,b):

return a+b
add(5,6)
318
Out[1]: 11
How could we do this, in a fictional version of Python which only defined functions of one argument? In
order to understand this, we’ll have to understand several of the concepts of functional programming. Let’s
start with a program which just adds five to something:
In [2]: def add_five(a):
return a+5
add_five(6)
Out[2]: 11
OK, we could define lots of these, one for each number we want to add. But that would be infinitely
repetitive. So, let’s try to metaprogram that: we want a function which returns these add N() functions.
Let’s start with the easy case: a function which returns a function which adds 5 to something:
In [3]: def generate_five_adder():
def _add_five(a):
return a+5
return _add_five
coolfunction = generate_five_adder()
coolfunction(7)
Out[3]: 12
OK, so what happened there? Well, we defined a function inside the other function. We can always do
that:
In [4]: def thirty_function():
def times_three(a):
return a*3
def add_seven(a):
return a+7
return times_three(add_seven(3))
thirty_function()
Out[4]: 30
When we do this, the functions enclosed inside the outer function are local functions, and can’t be seen
outside:
In [5]: with assert_raises(NameError):
add_seven
---------------------------------------------------------------------------
<ipython-input-5-d7b364b66955> in <module>()
----> 1 with assert raises(NameError):
2 add seven
NameError: name ’assert raises’ is not defined
319
There’s not really much of a difference between functions and other variables in python. A function is
just a variable which can have () put after it to call the code!
In [6]: print thirty_function

x=[thirty_function, add_five, add]
for fun in x:
print fun
<function thirty function at 0x10c495140>

<function thirty function at 0x10c495140>
<function add five at 0x10c4950c8>
<function add at 0x10c404f50>
And we know that one of the things we can do with a variable is return it. So we can return a function,
and then call it outside:
In [7]: def deferred_greeting():

def greet():
print "Hello"
return greet
friendlyfunction=deferred_greeting()
# Do something else
print "Just passing the time..."
# OK, Go!
friendlyfunction()
Just passing the time...

Hello
So now, to finish this, we just need to return a function to add an arbitrary amount:
In [8]: def define_adder(increment):

def adder(a):
return a+increment
return adder
add_3=define_adder(3)
add_3(9)
Out[8]: 12
We can make this even prettier: let’s make another variable pointing to our define adder() function:
In [9]: add = define_adder
And now we can do the real magic:
In [10]: add(8)(5)
Out[10]: 13
320
27.2.2 Closures
You may have noticed something a bit weird:
In the definition of define adder, increment is a local variable. It should have gone out of scope and
died at the end of the definition. How can the amount the returned adder function is adding still be kept?
This is called a closure. In Python, whenever a function definition references a variable in the surrounding
scope, it is preserved within the function definition.
You can close over global module variables as well:
In [11]: name = "James"

def greet():
print "Hello, ", name
greet()
Hello, James
And note that the closure stores a reference to the variable in the surrounding scope: (“Late Binding”)
In [12]: name="Matt"
greet()
Hello, Matt
27.2.3 Map and Reduce

We often want to apply a function to each variable in an array, to return a new array. We can do this with
a list comprehension:
In [13]: numbers=range(10)
[add_five(i) for i in numbers]
Out[13]: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
But this is sufficiently common that there’s a quick built-in:
In [14]: map(add_five, numbers)
Out[14]: [5, 6, 7, 8, 9, 10, 11, 12, 13, 14]
This map operation is really important conceptually when understanding efficient parallel programming:
different computers can apply the mapped function to their input at the same time. We call this Single
Program, Multiple Data. (SPMD) map is half of the map-reduce functional programming paradigm
which is key to the efficient operation of much of today’s “data science” explosion.
Let’s continue our functional programming mind-stretch by looking at reduce operations.
We very often want to loop with some kind of accumulator, such as when finding a mean, or finding a
maximum:
In [15]: def mean(data):

sum=0.0
for x in data:
sum+=x
return sum/len(data)
mean(range(10))
321
import sys
def my_max(data):
# Start with the smallest possible number
highest=sys.float_info.min
for x in data:
if x>highest:
highest=x
return highest
my_max([2,5,10,-11,-5])
Out[15]: 10
These operations, where we have some variable which is building up a result, and the result is updated
with some operation, can be gathered together as a functional program, taking in the operation to be used
to combine results as an argument:
In [16]: def accumulate(initial, operation, data):

accumulator=initial
for x in data:
accumulator=operation(accumulator, x)
return accumulator
def my_sum(data):
def _add(a,b):
return a+b
return accumulate(0, _add, data)
print my_sum(range(5))
def bigger(a,b):
if b>a:
return b
return a
def my_max(data):
return accumulate(sys.float_info.min, bigger, data)
print my_max([2,5,10,-11,-5])
10
10
Now, because these operations, bigger, and add, are such that e.g. (a+b)+c = a+(b+c) , i.e. they are
associative, we could apply our accumulation to the left half and the right half of the array, each on a
different computer, and then combine the two halves:
1+2+3+4=(1+2)+(3+4)
Indeed, with a bigger array, we can divide-and-conquer more times:
1+2+3+4+5+6+7+8=((1+2)+(3+4))+((5+6)+(7+8))
So with enough parallel computers, we could do this operation on eight numbers in three steps: first, we
use four computers to do one each of the pairwise adds.
Then, we use two computers to add the four totals.
Then, we use one of the computers to do the final add of the two last numbers.
You might be able to do the maths to see that with an N element list, the number of such steps is
proportional to the logarithm of N.
322
We say that with enough computers, reduction operations are O(ln N)
This course isn’t an introduction to algorithms, but we’ll talk more about this O() notation when we
think about programming for performance.
Anyway, this accumulate-under-an-operation process, is so fundamental to computing that it’s usually
in standard libraries for languages which allow functional programming:
In [17]: from functools import reduce
def my_max(data):
return reduce(bigger,data,sys.float_info.min)
my_max([2,5,10,-11,-5])
Out[17]: 10
27.3 Lambda Functions

When doing functional programming, we often want to be able to define a function on the fly:
In [18]: def most_Cs_in_any_sequence(sequences):

def count_Cs(sequence):
return sequence.count(’C’)
counts=map(count_Cs, sequences)
return max(counts)
def most_Gs_in_any_sequence(sequences):
return max(map(lambda sequence: sequence.count(’G’),sequences))
data=[
"CGTA",
"CGGGTAAACG",
"GATTACA"
]
most_Gs_in_any_sequence(data)
Out[18]: 4
The syntax here is that these two definitions are identical:
In [19]: func_name=lambda a,b,c : a+b+c
def func_name(a,b,c):
a+b+c
lambda defines an “anonymous” function.
In [20]: def most_of_given_base_in_any_sequence(sequences, base):

return max(map(lambda sequence: sequence.count(base), sequences))
most_of_given_base_in_any_sequence(data,’A’)
Out[20]: 3
The above fragment defined a lambda function as a closure over base. If you understood that, you’ve
got it!
323
In [21]: def my_max(data): return reduce(lambda a,b: a if a>b else b, data,
sys.float_info.min)
my_max([2,5,10,-11,-5])
Out[21]: 10
27.4 Using functional programming for numerical methods

Probably the most common use in research computing for functional programming is the application of a
numerical method to a function. For example:
In [22]: % matplotlib inline
In [23]: from scipy.optimize import newton

from numpy import linspace, zeros
solve_me=lambda x: x**2-x
print newton(solve_me, 2), newton(solve_me,0.2)
xs=linspace(-1,2,50)
solved=[xs,map(solve_me,xs),xs,zeros(50)]
plt.plot(*solved)
1.0 -3.44190514264e-21
Out[23]: [<matplotlib.lines.Line2D at 0x10ef8ec10>,

<matplotlib.lines.Line2D at 0x10ef8ee90>]
Sometimes such tools return another function:
324
In [24]: def derivative(func, eps):
def _func_derived(x):
return (func(x+eps)-func(x))/eps
return _func_derived
derived=(xs,map(solve_me,xs),xs,map(derivative(solve_me,0.01),xs))
plt.plot(*derived)
print newton(derivative(solve_me,0.01),0)
0.495
Of course, coding your own numerical methods is bad:
In [25]: import scipy.misc
def derivative(func):
def _func_derived(x):
return scipy.misc.derivative(solve_me,x)
return _func_derived
newton(derivative(solve_me),0)
Out[25]: 0.5
If you’ve done a moderate amount of calculus, then you’ll find similarities between functional program-
ming in computer science and Functionals in the calculus of variations.
27.5 Iterators and Generators

We’ve seen that in Python, anything which can be iterated over is called an iterable:
325
In [1]: for key in baskets:
print key.upper()
---------------------------------------------------------------------------
<ipython-input-1-ddc35b8862a9> in <module>()
----> 1 for key in baskets:
2 print key.upper()
NameError: name ’baskets’ is not defined
Surprisingly often, we want to iterate over something that takes a moderately large amount of storage to
store. For example, our map images in the green-graph example.
Our green-graph example involved making an array of all the maps between London and Birmingham.
This kept them all in memory at the same time: first we downloaded all the maps, then we counted the
green pixels in each of them.
This would NOT work if we used more points. We need to use a generator
27.6 Iterators
Consider the basic python range function:
In [2]: print range(10)
[0, 1, 2, 3, 4, 5, 6, 7, 8, 9]
In [3]: total=0
for x in range(int(1e6)): total+= x
total
Out[3]: 499999500000
While this was executing, the range() statement allocated a million integers. This is very inefficient.
We don’t actually need a million integers at once, just each integer in turn up to a million.
xrange is like range, but yields an iterable which is not an array.
In [4]: print xrange(3)
xrange(3)
So we can for loop over it:
In [5]: for i in xrange(3): print i
0
1
2
An generator object, like xrange(3), when we iterate over it, works by defining a next() method which
moves the iterator forward:
326
In [6]: a=iter(xrange(3))
In [7]: a.next()
Out[7]: 0
In [8]: a.next()
Out[8]: 1
In [9]: a.next()
Out[9]: 2
At the end, StopIteration is raised as an exception:
In [10]: print a.next()
---------------------------------------------------------------------------
<ipython-input-10-5aed2ff39c77> in <module>()
----> 1 print a.next()
StopIteration:
In [11]: total=0
for x in xrange(int(1e6)): total+= x
print total
499999500000
Similarly:
In [12]: baskets={
’apples’:5,
’oranges’:3,
’kiwis’:2
}
In [13]: baskets.items()
Out[13]: [(’kiwis’, 2), (’apples’, 5), (’oranges’, 3)]
In [14]: baskets.iteritems()
Out[14]: <dictionary-itemiterator at 0x103350c00>
327
27.7 Defining Our Own Iterable
We can make our own iterators by defining classes that implement next() and iter() methods: this is the
iterator protocol.
For each of the concepts, in Python, like sequence, container, iterable, python defines a protocol, a set
of methods a class must implement, in order to be treated as a member of that concept.
The iterator protocol is the protocol that defines things that support for x in y:.
To define an iterator, the methods that must be supported are next() and iter ().
next() must update the iterator.
We’ll see why we need to define iter in a moment.
In [15]: class fib_iterator(object):

def __init__(self, limit, seed1=1, seed2=1):
self.limit=limit
self.previous=seed1
self.current=seed2
def __iter__(self):
return self
def next(self):
(self.previous, self.current)=(
self.current, self.previous+self.current)
self.limit -=1
if self.limit<0: raise StopIteration() # This will be
# explained in a few slides!
return self.current
In [16]: x=fib_iterator(5)
In [17]: x.next()
Out[17]: 2
In [18]: x.next()
Out[18]: 3
In [19]: x.next()
Out[19]: 5
In [20]: x.next()
Out[20]: 8
In [21]: for x in fib_iterator(5):

print x
2
3
5
8
13
In [22]: sum(fib_iterator(5))
Out[22]: 31
328
27.7.1 A shortcut to iterables: the iter method.
In fact, if, to be iterated over, a class just wants to behave as if it were some other iterable, you can just
implement iter and return iter(some other iterable), without implementing next. For example, an
image class might want to implement some metadata, but behave just as if it were just a 1-d pixel array
when being iterated:
In [23]: from numpy import array

class MyImage(object):
def __init__(self, pixels):
self.pixels=array(pixels,dtype=’uint8’)
self.channels=self.pixels.shape[2]
def __iter__(self):
# return an iterator over the pixels
# See future NumPy lecture for using reshape
return iter(self.pixels.reshape(-1,self.channels))
def show(self):
plt.imshow(self.pixels, interpolation="None")
x=[[[255,255,0],[0,255,0]],[[0,0,255],[255,255,255]]]
image=MyImage(x)

image.show()
In [25]: image.channels
Out[25]: 3
329
In [26]: from webcolors import rgb_to_name
for pixel in image:
print rgb_to_name(pixel)
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
<ipython-input-26-755abc5ac407> in <module>()
----> 1 from webcolors import rgb to name
2 for pixel in image:
3 print rgb to name(pixel)
ImportError: No module named webcolors
The iterator protocol is to implement both iter and next, while the iterable protocol is to imple-
ment iter and return an something iterable.
27.7.2 Generators
There’s a fair amount of “boiler-plate” in the above class-based definition of an iterable.
Python provides another way to specify something which meets the iterator protocol: generators.
In [27]: def my_generator():

yield 5
yield 10
x=my_generator()
print x.next()
print x.next()
with assert_raises(StopIteration):
print x.next()
5
10
---------------------------------------------------------------------------
<ipython-input-27-69305c995a91> in <module>()
7 print x.next()
8
----> 9 with assert raises(StopIteration):
10 print x.next()
330
A function which has yield statements instead of a return statement returns temporarily: it automag-
ically becomes something which implements next.
Each call of next() returns control to the function where it left off.
Control passes back-and-forth between the generator and the caller. Our fibonacci example therefore
becomes a function rather than a class.
In [28]: def yield_fibs(limit, seed1=1,seed2=1):

current=seed1
previous=seed2
while limit>0:
limit-=1
current, previous = current+previous, current
yield current
In [29]: print sum(yield_fibs(5))
31
In [30]: plt.plot(list(yield_fibs(20)))
Out[30]: [<matplotlib.lines.Line2D at 0x1053b1a10>]
27.8 Context managers and decorators

27.8.1 Context managers
We saw that instead of separately opening and closeing a file, we can have the file be automatically closed
using a context manager:
In [31]: with open(’example.yaml’) as foo:

print yaml.load(foo)
331
---------------------------------------------------------------------------
IOError Traceback (most recent call last)
<ipython-input-31-16ce57c9cf6d> in <module>()
----> 1 with open(’example.yaml’) as foo:
2 print yaml.load(foo)
IOError: [Errno 2] No such file or directory: ’example.yaml’
How could we define our own one of these, if we too have clean-up code we always want to run after a
calling function has done its work, or set-up code we want to do first?
We can define a class that meets an appropriate protocol:
In [32]: class verbose_context():

self.name=name
def __enter__(self):
print "Get ready, ", self.name
def __exit__(self, exc_type, exc_value, traceback):
print "OK, done"
with verbose_context("James"):
print "Doing it!"
Get ready, James

Doing it!
OK, done
However, this is pretty verbose! Again, a generator with yield makes for an easier syntax:
In [33]: from contextlib import contextmanager
@contextmanager
def verbose_context(name):
print "Get ready for action, ", name
yield name.upper()
print "You did it"
with verbose_context("James") as shouty:

print "Doing it, ", shouty
Get ready for action, James

Doing it, JAMES
You did it
Again, we use yield to temporarily return from a function.
27.8.2 Decorators
When doing functional programming, we may often want to define mutator functions which take in one
function and return a new function, such as our derivative example earlier.
332
In [34]: def repeater(count):
def wrap_function_in_repeat(func):
def _repeated(x):
counter=count
while counter>0:
counter-=1
x=func(x)
return x
return _repeated
return wrap_function_in_repeat
from math import sqrt
fiftytimes=repeater(50)
fiftyroots=fiftytimes(sqrt)
print fiftyroots(100)
1.0
It turns out that, quite often, we want to apply one of these to a function as we’re defining a class. For
example, we may want to specify that after certain methods are called, data should always be stored:
In [35]: def reset_required(func):

def _with_data_save(self, *args):
func(self,*args)
self.stored_data.append(self.data)
return _with_data_save
class SomeClass(object):
def __init__(self):
self.data=[]
self.stored_data=[]
def _step1(self, ins):
self.data=[x*2 for x in ins]
step1=reset_required(_step1)
In [36]: x=SomeClass()
In [37]: x.step1("Hello")
print x.data
[’HH’, ’ee’, ’ll’, ’ll’, ’oo’]
In [38]: x.step1("World")
print x.data
[’WW’, ’oo’, ’rr’, ’ll’, ’dd’]
In [39]: print x.stored_data
[[’HH’, ’ee’, ’ll’, ’ll’, ’oo’], [’WW’, ’oo’, ’rr’, ’ll’, ’dd’]]
Python provides some “syntactic sugar” to make this kind of coding prettier:
333
In [40]: def reset_required(func):
def _with_data_save(self, *args):
func(self,*args)
self.stored_data.append(self.data)
return _with_data_save
class SomeClass(object):
def __init__(self):
self.data=[]
self.stored_data=[]
@reset_required
def step1(self, ins):
self.data=[x*2 for x in ins]
x=SomeClass()
x.step1("Hello")
x.step1("World")
print x.stored_data
[[’HH’, ’ee’, ’ll’, ’ll’, ’oo’], [’WW’, ’oo’, ’rr’, ’ll’, ’dd’]]
Any function which accepts a function as its first argument and returns a function can be used as a
decorator like this.
Much of Python’s standard functionality is implemented as decorators: we’ve seen @contextmanager,
@classmethod and @attribute. The @contextmanager metafunction, for example, takes in an iterator, and
yields a class conforming to the context manager protocol.
27.8.3 Test generators

A few weeks ago we saw a test which loaded its test cases from a YAML file and asserted each input with
each output. This was nice and concise, but had one flaw: we had just one test, covering all the fixtures, so
we got just one . in the test output when we ran the tests, and if any test failed, the rest were not run. We
can do a nicer job with a test generator:
In [41]: def assert_examplar(**fixture):
answer=fixture.pop(’answer’)
assert_equal(greet(**fixture), answer)
def test_greeter():
with open(os.path.join(os.path.dirname(
__file__),’fixtures’,’samples.yaml’)
) as fixtures_file:
fixtures=yaml.load(fixtures_file)
for fixture in fixtures:
yield assert_exemplar(**fixture)
Each time a function beginning with test does a yield it results in another test.
27.8.4 Negative test contexts managers

We have seen this:
with assert_raises(AttributeError):
x=2
x.foo()
334
We can now see how nose might have implemented this:
In [43]: from contextlib import contextmanager
@contextmanager
def reimplement_assert_raises(exception):
try:
yield
except exception:
pass
else:
raise Exception("Expected,", exception,
" to be raised, nothing was.")
In [44]: with reimplement_assert_raises(AttributeError):

x=2
x.foo()
27.8.5 Negative test decorators

Nose also implements a very nice negative test decorator:
In [45]: from nose.tools import raises
@raises(TypeError, ValueError)
def test_raises_type_error():
raise TypeError("This test passes")
In [46]: test_raises_type_error()
In [47]: @raises(Exception)
def test_that_fails_by_passing():
pass
In [48]: test_that_fails_by_passing()
---------------------------------------------------------------------------
<ipython-input-48-fbd84a070d60> in <module>()
----> 1 test that fails by passing()
/usr/local/lib/python2.7/site-packages/nose/tools/nontrivial.pyc in newfunc(*arg, **kw)

65 else:
66 message = "%s() did not raise %s" % (name, valid)
---> 67 raise AssertionError(message)
68 newfunc = make decorator(func)(newfunc)
69 return newfunc
AssertionError: test that fails by passing() did not raise Exception
335
We could reimplement this ourselves now too:
In [49]: def homemade_raises_decorator(exception):

def wrap_function(func): #Closure over exception
# Define a function which runs another function under the assert_raises context:
def _output(*args): #Closure over func and exception
with assert_raises(exception):
func(*args)
# Return it
return _output
return wrap_function
In [50]: @homemade_raises_decorator(TypeError)
def test_raises_type_error():
raise TypeError("This test passes")
In [51]: test_raises_type_error()
27.9 Exceptions
27.9.1 Exceptions
When we learned about testing, we saw that Python complains when things go wrong by raising an “Excep-
tion” naming a type of error:
In [1]: with assert_raises(ZeroDivisionError):

1/0
---------------------------------------------------------------------------
<ipython-input-1-818314fe40b1> in <module>()
----> 1 with assert raises(ZeroDivisionError):
2 1/0
Exceptions are objects, forming a class hierarchy. We just raised an instance of the ZeroDivisionError
class, making the program crash.
In [2]: import inspect

inspect.getmro(ZeroDivisionError)
Out[2]: (ZeroDivisionError,
ArithmeticError,
StandardError,
Exception,
BaseException,
object)
So we can see that a zero division error is a particular kind of Arithmetic Error.
336
In [3]: x=1
with assert_raises(TypeError):
for y in x: print y
inspect.getmro(TypeError)
---------------------------------------------------------------------------
<ipython-input-3-73048ceb9439> in <module>()
1 x=1
----> 2 with assert raises(TypeError):
3 for y in x: print y
4
5 inspect.getmro(TypeError)
When we were looking at testing, we saw that it is important for code to crash with a meaningful
exception type when something is wrong. We raise an Exception with raise. Often, we can look for an
appropriate exception from the standard set to raise.
However, we may want to define our own exceptions. Doing this is as simple as inheriting from Exception:
In [4]: class MyCustomErrorType(Exception):

pass
with assert_raises(MyCustomErrorType):
raise(MyCustomErrorType("Problem"))
---------------------------------------------------------------------------
<ipython-input-4-c4ea74bc09ae> in <module>()
2 pass
3
----> 4 with assert raises(MyCustomErrorType):
5 raise(MyCustomErrorType("Problem"))
You can add custom data to your exception:
In [5]: class MyCustomErrorType(Exception):

def __init__(self, category=None):
self.category=category
def __str__(self):
return "Error, cateory " + str(self. category)
337
with assert_raises(MyCustomErrorType):
raise(MyCustomErrorType(404))
---------------------------------------------------------------------------
<ipython-input-5-0de47407e54b> in <module>()
5 return "Error, cateory " + str(self. category)
6
----> 7 with assert raises(MyCustomErrorType):
8 raise(MyCustomErrorType(404))
The real power of exceptions comes, however, not in letting them crash the program, but in letting your
program handle them. We say that an exception has been “thrown” and then “caught”.
In [6]: import yaml

try:
config=yaml.load(open("datasource.yaml"))
user=config["userid"]
password=config["password"]
except IOError:
user="anonymous"
password=None
print user
anonymous
Note that we specify only the error we expect to happen and want to handle. Sometimes you see code
that catches everything:
In [7]: try:
config=yaml.lod(open("datasource.yaml"))
except:
user="anonymous"
password=None
print user
anonymous
There was a mistyped function name there, but we did not notice the error, as the generic except caught
it. Therefore, we should catch only the error we want.
In [8]: with open(’datasource2.yaml’,’w’) as outfile:

outfile.write(’userid: jamespjh\n’)
outfile.write(’password: secret\n’)
338
with open(’datasource3.yaml’,’w’) as outfile:
outfile.write(’user: jamespjh\n’)
outfile.write(’password: secret\n’)
def read_credentials(source):
try:
datasource=open(source)
config=yaml.load(datasource)
datasource.close()
except IOError:
user="anonymous"
password=None
return user, password
print read_credentials(’datasource2.yaml’)
print read_credentials(’datasource.yaml’)
with assert_raises(KeyError):
print read_credentials(’datasource3.yaml’)
(’jamespjh’, ’secret’)
(’anonymous’, None)
---------------------------------------------------------------------------
<ipython-input-8-9d772c187db7> in <module>()
23 print read credentials(’datasource.yaml’)
24
---> 25 with assert raises(KeyError):
26 print read credentials(’datasource3.yaml’)
This last code has a flaw: the file was successfully opened, the missing key was noticed, but not explicitly
closed. It’s normally OK, as python will close the file as soon as it notices there are no longer any references
to datasource in memory, after the function exits. But this is not good practice, you should keep a file handle
for as short a time as possible.
In [9]: def read_credentials(source):

try:
except IOError:
user="anonymous"
339
password=None
finally:
datasource.close()
The finally clause is executed whether or not an exception occurs.

The last optional clause of a try statement, an else clause is called only if an exception is NOT raised.
It can be a better place than the try clause to put code other than that which you expect to raise the error,
and which you do not want to be executed if the error is raised. It is executed in the same circumstances
as code put in the end of the try block, the only difference is that errors raised during the else clause are
not caught. Don’t worry if this seems useless to you; most languages implementations of try/except don’t
support such a clause.
In [10]: def read_credentials(source):

try:
except IOError:
user="anonymous"
password=None
else:
finally:
datasource.close()
Exceptions do not have to be caught close to the part of the program calling them. They can be caught
anywhere “above” the calling point in the call stack: control can jump arbitrarily far in the program: up to
the except clause of the “highest” containing try statement.
In [11]: def f4(x):

if x==0:
return
if x==1:
raise ArithmeticError()
if x==2:
raise SyntaxError()
if x==3:
raise TypeError()
In [12]: def f3(x):

try:
print "F3Before"
f4(x)
print "F3After"
except ArithmeticError:
print "F3Except"
In [13]: def f2(x):

try:
print "F2Before"
f3(x)
print "F2After"
except SyntaxError:
print "F2Except"
340
In [14]: def f1(x):
try:
print "F1Before"
f2(x)
print "F1After"
except TypeError:
print "F1Except"
In [15]: f1(0)
F1Before
F2Before
F3Before
F3After
F2After
F1After
In [16]: f1(1)
F1Before
F2Before
F3Before
F3Except
F2After
F1After
In [17]: f1(2)
F1Before
F2Before
F3Before
F2Except
F1After
In [18]: f1(3)
F1Before
F2Before
F3Before
F1Except
27.9.2 Design with Exceptions

Now we know how exceptions work, we need to think about the design implications. . . How best to use
them.
Traditional software design theory will tell you that they should only be used to describe and recover
from exceptional conditions: things going wrong. Normal program flow shouldn’t use them.
Python’s designers take a different view: use of exceptions in normal flow is considered OK. For example,
all iterators raise a StopIteration exception to indicate the iteration is complete.
A commonly recommended python design pattern is to use exceptions to determine whether an object
implments a protocol (concept/interface), rather than testing on type.
For example, we might want a function which can be supplied either a data series or a path to a location
on disk where data can be found. We can examine the type of the supplied content:
In [19]: import yaml

def analysis(source):
341
if type(source)==dict:
name=source[’modelname’]
else:
content=open(source)
source=yaml.load(content)
print name
In [20]: analysis({’modelname’:’Super’})
Super
In [21]: with open(’example.yaml’,’w’) as outfile:

outfile.write(’modelname: brilliant\n’)
In [22]: analysis(’example.yaml’)
brilliant
However, we can also use the try-it-and-handle-exceptions approach to this.
In [23]: def analysis(source):

try:
except TypeError:
print name
analysis(’example.yaml’)
brilliant
This approach is more extensible, and behaves properly if we give it some other data-source
which responds like a dictionary or string.
In [24]: def analysis(source):

try:
except TypeError:
# Source was not a dictionary-like object
# Maybe it is a file path
try:
except IOError:
# Maybe it was already raw YAML content
source=yaml.load(source)
print name
analysis("modelname: Amazing")
Amazing
342
Sometimes we want to catch an error, partially handle it, perhaps add some extra data to the exception,
and then re-raise to be caught again further up the call stack.
The keyword “raise” with no argument in an except: clause will cause the caught error to be re-thrown.
Doing this is the only circumstance where it is safe to do except: without catching a specfic type of error.
In [25]: try:
# Something
pass
except:
# Do this code here if anything goes wrong
raise
It can be useful to catch and re-throw an error as you go up the chain, doing any clean-up needed for
each layer of a program.
The error will finally be caught and not re-thrown only at a higher program layer that knows how to
recover. This is known as the “throw low catch high” principle.
27.10 Operator overloading

Warning: Advanced Topic!
27.10.1 Operator overloading

We need to use a metaprogramming trick to make this teaching notebook work. I want to be able to put
explanatory text in between parts of a class definition, so I’ll define a decorator to help me build up a class
definition gradually.
In [1]: def extend(class_to_extend):

""" Metaprogramming to allow gradual implementation
of class during notebook. Thanks to
http://www.ianbicking.org/blog/2007/08/opening-python-classes.html """
def decorator(extending_class):
for name, value in extending_class.__dict__.iteritems():
if name in [’__dict__’,’__module__’, ’__weakref__’, ’__doc__’]:
continue
setattr(class_to_extend,name,value)
return class_to_extend
return decorator
Imagine we wanted to make a library to describe some kind of symbolic algebra system:
In [2]: class Term(object):

def __init__(self, symbols=[], powers=[], coefficient=1):
self.coefficient=coefficient
self.data={symbol: exponent for symbol,exponent
in zip(symbols, powers)}
class Expression(object):
def __init__(self, terms): self.terms=terms
So that 5x2 y + 7x + 2 might be constructed as:
In [3]: first=Term([’x’,’y’],[2,1],5)
second=Term([’x’],[1],7)
third=Term([],[],2)
result=Expression([first, second, third])
343
This is pretty cumbersome.
What we’d really like is to have 2x+y give an appropriate expression.
First, we’ll define things so that we can construct our terms and expressions in different ways.
In [4]: class Term(object):

def __init__(self, *args):
lead=args[0]
if type(lead)==type(self):
# Copy constructor
self.data=dict(lead.data)
self.coefficient=lead.coefficient
elif type(lead)==int:
self.from_constant(lead)
elif type(lead)==str:
self.from_symbol(*args)
elif type(lead)==dict:
self.from_dictionary(*args)
else:
self.from_lists(*args)
def from_constant(self, constant):

self.coefficient=constant
self.data={}
def from_symbol(self, symbol, coefficient=1, power=1):

self.data={symbol:power}
def from_dictionary(self, data, coefficient=1):

self.data=data
def from_lists(self, symbols=[], powers=[], coefficient=1):

self.data={symbol: exponent for symbol,exponent
in zip(symbols, powers)}
In [5]: class Expression(object):

def __init__(self, terms=[]):
self.terms=list(terms)
We could define add() and multiply() operations on expressions and terms:
In [6]: @extend(Term)
class Term(object):
def add(self, *others):
return Expression((self,)+others)
class Term(object):
def multiply(self, *others):
result_data=dict(self.data)
result_coeff=self.coefficient
# Convert arguments to Terms first if they are
344
# constants or integers
others=map(Term,others)
for another in others:
for symbol, exponent in another.data.iteritems():
if symbol in result_data:
result_data[symbol]+=another.data[symbol]
else:
result_data[symbol]=another.data[symbol]
result_coeff*=another.coefficient
return Term(result_data,result_coeff)
In [8]: @extend(Expression)
def add(self, *others):
result=Expression(self.terms)
for another in others:
if type(another)==Term:
result.terms.append(another)
else:
result.terms+=another.terms
return result
We can now construct the above expression as:
In [9]: x=Term(’x’)
y=Term(’y’)
first=Term(5).multiply(Term(’x’),Term(’x’),Term(’y’))
second=Term(7).multiply(Term(’x’))
third=Term(2)
expr=first.add(second,third)
This is better, but we still can’t write the expression in a ‘natural’ way.
However, we can define what * and + do when applied to Terms!:
class Term(object):
def __add__(self, other):
return self.add(other)
def __mul__(self, other):
return self.multiply(other)
def multiply(self, another):
# Distributive law left as exercise
pass
def __add__(self, other):

return self.add(other)
In [12]: x_plus_y=Term(’x’)+’y’
x_plus_y.terms[0].data
Out[12]: {’x’: 1}
345
In [13]: five_x_ysq=Term(’x’)*5*’y’*’y’
print five_x_ysq.data, five_x_ysq.coefficient
{’y’: 2, ’x’: 1} 5
This is called operator overloading. We can define what add and multiply mean when applied to our
class.
Note that this only works so far if we multiply on the right-hand-side! However, we can define a multi-
plication that works backwards, which is used as a fallback if the left multiply raises an error:
def __radd__(self, other):
return self.__add__(other)
class Term(object):
def __rmul__(self, other):
return self.__mul__(other)
def __radd__(self, other):
return self.__add__(other)
In [16]: print 5*Term(’x’)
< main .Term object at 0x103a3b510>
It’s not easy at the moment to see if these things are working!
In [17]: fivex=5*Term(’x’)
print fivex.data, fivex.coefficient
{’x’: 1} 5
We can add another operator method str , which defines what happens if we try to print our class:
class Term(object):
def __str__(self):
def symbol_string(symbol, power):
if power==1:
return symbol
else:
return symbol+’^’+str(power)
symbol_strings=[symbol_string(symbol, power)
for symbol, power in self.data.iteritems()]
prod=’*’.join(symbol_strings)
if not prod:
return str(self.coefficient)
if self.coefficient==1:
return prod
else:
return str(self.coefficient)+’*’+prod
346
def __str__(self):
return ’+’.join(map(str,self.terms))
In [20]: first=Term(5)*’x’*’x’*’y’
second=Term(7)*’x’
third=Term(2)
expr=first+second+third
In [21]: print expr
5*y*x^2+7*x+2
We can add lots more operators to classes. eq to determine if objects are equal. getitem to apply
[1] to your object. Probably the most exciting one is call , which overrides the () operator; allows us to
define classes that behave like functions! We call these callables.
In [22]: class Greeter(object):

def __init__(self, greeting):
self.greeting = greeting
def __call__(self, name):
print self.greeting, name
greeter_instance = Greeter("Hello")
greeter_instance("James")
Hello James
We’ve now come full circle in the blurring of the distinction between functions and objects! The full
power of functional programming is really remarkable.
If you want to know more about the topics in this lecture, using a different language syntax, I recommend
you watch the Abelson and Sussman “Structure and Interpretation of Computer Programs” lectures. These
are the Computer Science equivalent of the Feynman Lectures!
27.11 Metaprogramming Example

Warning: Advanced topic!
Consider a bunch of variables, each of which need initialising and incrementing:
In [1]: bananas=0
apples=0
oranges=0
bananas+=1
apples+=1
oranges+=1
The right hand side of these assignments doesn’t respect the DRY principle. We could of course define
a variable for our initial value:
In [2]: initial_fruit_count=0
bananas=initial_fruit_count
apples=initial_fruit_count
oranges=initial_fruit_count
347
However, this is still not as DRY as it could be: what if we wanted to replace the assignment with, say,
a class constructor and a buy operation:
In [3]: class Basket:

def __init__(self):
self.count=0
def buy(self):
self.count+=1
bananas=Basket()
apples=Basket()
oranges=Basket()
bananas.buy()
apples.buy()
oranges.buy()
We had to make the change in three places. Whenever you see a situation where a refactoring or change
of design might require you to change the code in multiple places, you have an opportunity to make the code
DRYer.
In this case, metaprogramming for incrementing these variables would involve just a loop over all the
variables we want to initialise:
In [4]: baskets=[bananas, apples, oranges]

for basket in baskets: basket.buy()
However, this trick doesn’t work for initialising a new variable:

with assert_raises(NameError):
baskets=[bananas, apples, oranges, kiwis]
So can we declare a new variable programmatically? Given a list of the names of fruit baskets we want,
initialise a variable with that name?
In [6]: basket_names=[’bananas’, ’apples’, ’oranges’, ’kiwis’]
globals()[’apples’]
Out[6]: < main .Basket instance at 0x1059ef9e0>
Wow, we can! Every module or class in Python, is, under the hood, a special dictionary, storing the
values in its namespace. So we can create new variables by assigning to this dictionary. globals() gives a
reference to the attribute dictionary for the current module
In [7]: for name in basket_names:

globals()[name]=Basket()
print kiwis.count
This is metaprogramming.
I would NOT recommend using it for an example as trivial as the one above. A better, more Pythonic
choice here would be to use a data structure to manage your set of fruit baskets:
348
In [8]: baskets={}
for name in basket_names:
baskets[name]=Basket()
print baskets[’kiwis’].count
Or even, using a dictionary comprehension:
In [9]: baskets={name : Basket() for name in baskets}

print baskets[’kiwis’].count
Which is the nicest way to do this, I think. Code which feels like metaprogramming is needed to make
it less repetitive can often instead be DRYed up using a refactored data structure, in a way which is cleaner
and more easy to understand. Nevertheless, metaprogramming is worth knowing.
27.11.1 Metaprogramming class attributes

We can metaprogram the attributes of a module using the globals() function.
We will also want to be able to metaprogram a class, by accessing its attribute dictionary.
This will allow us, for example, to programmatically add members to a class.
In [10]: class Boring(object):

pass
If we are adding our own attributes, we can just do so directly:
In [11]: x=Boring()
x.name="James"
In [12]: x.name
Out[12]: ’James’
And these turn up, as expected, in an attribute dictionary for the class:
In [13]: x.__dict__
Out[13]: {’name’: ’James’}
We can use getattr to access this special dictionary:
In [14]: getattr(x, ’name’)
If we want to add an attribute given it’s name as a string, we can use setattr:
In [15]: setattr(x,’age’,38)
x.age
Out[15]: 38
349
And we could do this in a loop to programmatically add many attributes.
The real power of accessing the attribute dictionary comes when we realise that there is very little
difference between member data and member functions.
Now that we know, from our functional programming, that a function is just a variable that can
be called with (), we can set an attribute to a function, and it becomes a member function!
In [16]: setattr(Boring, ’describe’, lambda self: self.name+ " is "+str(self.age))
In [17]: x.describe()
Out[17]: ’James is 38’
In [18]: x.describe
Out[18]: <bound method Boring.<lambda> of < main .Boring object at 0x106085350>>
In [19]: Boring.describe
Out[19]: <unbound method Boring.<lambda>>
Note that we set this method as an attribute of the class, not the instance, so it is available to other
instances of Boring:
In [20]: y=Boring()
y.name = ’Jim’
y.age = 99
In [21]: y.describe()
Out[21]: ’Jim is 99’
We can define a standalone function, and then bind it to the class. It’s first argument automagically
becomes self.
In [22]: def broken_birth_year(b_instance):

import datetime
current=datetime.datetime.now().year
return current-b_instance.age
In [23]: Boring.birth_year=broken_birth_year
In [24]: x.birth_year()
Out[24]: 1977
In [25]: x.birth_year
Out[25]: <bound method Boring.broken birth year of < main .Boring object at 0x106085350>>
In [26]: x.birth_year.__name__
Out[26]: ’broken birth year’
350
27.11.2 Metaprogramming function locals
We can access the attribute dictionary for the local namespace inside a function with locals() but this
cannot be written to.
Lack of safe programmatic creation of function-local variables is a flaw in Python.

def __init__(self, name, age, job, children_count):
for name,value in locals().iteritems():
if name==’self’: continue
print "Setting self.", name, " to ", value
setattr(self, name, value)
In [28]: me=Person("James", 38, "Scientific Programmer", 0)
Setting self. job to Scientific Programmer

Setting self. children count to 0
Setting self. age to 38
Setting self. name to James
In [29]: me.name
27.11.3 Metaprogramming warning!

Use this stuff sparingly!
The above example worked, but it produced Python code which is not particularly understandable.
Remember, your objective when programming is to produce code which is descriptive of what it does.
The above code is definitely less readable, less maintainable and more error prone than:

def __init__(self, name, age, job, children_count):
self.name=name
self.age=age
self.job=job
self.children_count=children_count
Sometimes, metaprogramming will be really helpful in making non-repetitive code, and you should have
it in your toolbox, which is why I’m teaching you it. But doing it all the time overcomplicated matters.
We’ve talked a lot about the DRY principle, but there is another equally important principle:
KISS: Keep it simple, Stupid!
Whenever you write code and you think, “Gosh, I’m really clever”,you’re probably doing it wrong. Code
should be about clarity, not showing off.
351
Chapter 28
Performance programming
We’ve spent most of this course looking at how to make code readable and reliable. For research work, it is
often also important that code is efficient: that it does what it needs to do quickly.
It is very hard to work out beforehand whether code will be efficient or not: it is essential to Profile code,
to measure its performance, to determine what aspects of it are slow.
When we looked at Functional programming, we claimed that code which is conceptualised in terms of
actions on whole data-sets rather than individual elements is more efficient. Let’s measure the performance
of some different ways of implementing some code and see how they perform.
28.1 Two Mandelbrots

You’re probably familiar with a famous fractal called the Mandelbrot Set.
For a complex number c, c is in the Mandelbrot set if the series zi+1 = zi2 + c (With z0 = c) stays close
to 0. Traditionally, we plot a color showing how many steps are needed for |zi | > 2, whereupon we are sure
the series will diverge.
Here’s a trivial python implementation:
In [1]: def mandel1(position,limit=50):

value=position
while abs(value)<2:
limit-=1
value=value**2+position
if limit<0:
return 0
return limit
In [2]: xmin=-1.5
ymin=-1.0
xmax=0.5
ymax=1.0
resolution=300
xstep=(xmax-xmin)/resolution
ystep=(ymax-ymin)/resolution
xs=[(xmin+(xmax-xmin)*i/resolution) for i in range(resolution)]
ys=[(ymin+(ymax-ymin)*i/resolution) for i in range(resolution)]
In [3]: %%timeit
data=[[mandel1(complex(x,y)) for x in xs] for y in ys]
1 loops, best of 3: 913 ms per loop
352
In [4]: data1=[[mandel1(complex(x,y)) for x in xs] for y in ys]

plt.imshow(data1,interpolation=’none’)
Out[5]: <matplotlib.image.AxesImage at 0x10432fb10>
We will learn this lesson how to make a version of this code which works Ten Times faster:

def mandel_numpy(position,limit=50):
value=position
diverged_at_count=np.zeros(position.shape)
while limit>0:
limit-=1
diverging=value*np.conj(value)>4
first_diverged_this_time=np.logical_and(diverging, diverged_at_count==0)
diverged_at_count[first_diverged_this_time]=limit
value[diverging]=2
return diverged_at_count
In [7]: ymatrix,xmatrix=np.mgrid[ymin:ymax:ystep,xmin:xmax:xstep]
values=xmatrix+1j*ymatrix
data_numpy=mandel_numpy(values)

plt.imshow(data_numpy,interpolation=’none’)
353
Out[8]: <matplotlib.image.AxesImage at 0x10537e110>
In [9]: %%timeit
data_numpy=mandel_numpy(values)
10 loops, best of 3: 51.3 ms per loop
Note we get the same answer:
In [10]: sum(sum(abs(data_numpy-data1)))
Out[10]: 0.0
In [1]: xmin=-1.5
ymin=-1.0
xmax=0.5
ymax=1.0
resolution=300
value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
354
28.2 Many Mandelbrots
Let’s compare our naive python implementation which used a list comprehension, taking 662ms, with the
following:
In [4]: %%timeit
data2=[]
for y in ys:
row=[]
for x in xs:
row.append(mandel1(complex(x,y)))
data2.append(row)
1 loops, best of 3: 1.03 s per loop
In [5]: data2=[]
for y in ys:
row=[]
for x in xs:
data2.append(row)
Interestingly, not much difference. I would have expected this to be slower, due to the normally high cost
of appending to data.

Out[6]: <matplotlib.image.AxesImage at 0x112335550>
We ought to be checking if these results are the same by comparing the values in a test, rather than
re-plotting. This is cumbersome in pure Python, but easy with NumPy, so we’ll do this later.
Let’s try a pre-allocated data structure:
In [7]: data3=[[0 for i in range(resolution)] for j in range(resolution)]
In [8]: %%timeit
for j,y in enumerate(ys):
for i,x in enumerate(xs):
data3[j][i]=mandel1(complex(x,y))
In [9]: for j,y in enumerate(ys):

for i,x in enumerate(xs):
data3[j][i]=mandel1(complex(x,y))
In [10]: plt.imshow(data3,interpolation=’none’)
Out[10]: <matplotlib.image.AxesImage at 0x10f4047d0>
Nope, no gain there.

Let’s try using functional programming approaches:
In [11]: %%timeit
data4=[]
for y in ys:
bind_mandel=lambda x: mandel1(complex(x,y))
data4.append(map(bind_mandel,xs))
355
In [12]: data4=[]
for y in ys:
bind_mandel=lambda x: mandel1(complex(x,y))
data4.append(map(bind_mandel,xs))
Out[13]: <matplotlib.image.AxesImage at 0x110df4850>
That was a tiny bit slower.

So, what do we learn from this? Our mental image of what code should be faster or slower is often wrong,
or doesn’t make much difference. The only way to really improve code performance is empirically, through
measurements.
In [1]: xmin=-1.5
ymin=-1.0
xmax=0.5
ymax=1.0
resolution=300
Let’s try using numpy
356
Chapter 29
NumPy
Numerical Python, NumPy, is a library that enables us to do much faster work with floating point data than
ordinary python.
29.1 NumPy constructors

NumPy’s core type is the ndarray, or N-Dimensional Array:
In [3]: np.zeros([3,4,2])
Out[3]: array([[[ 0., 0.],

[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]],
[[ 0., 0.],
[ 0., 0.],
[ 0., 0.],
[ 0., 0.]]])
We rarely construct an ndarray directly:
In [4]: np.ndarray([2,2,2],dtype=’int’)
Out[4]: array([[[-3458764513820540928, -3458764513820540928],

[ 140264375058447, 140264375108176]],
[[ 140264375108176, 4523476336],
[ 4551329752, 0]]])
We can convert any Python iterable into an ndarray with the array magic constructor:
In [5]: x=np.array(xrange(5))
print x
357
[0 1 2 3 4]
But NumPy arrays can only contain one type of data, unlike Python lists, which is one source of their
speed:
In [6]: np.array([1,1.0,’one’])
Out[6]: array([’1’, ’1.0’, ’one’],

dtype=’|S32’)
NumPy decided to make them all strings.

The real magic of numpy arrays is that most python operations are applied, quickly, on an elementwise
basis:
In [7]: x*2
Out[7]: array([0, 2, 4, 6, 8])
Ndarray multiplication is element wise, not matrix multiplication or vector dot product:
In [8]: x*x
Out[8]: array([ 0, 1, 4, 9, 16])
Numpy’s mathematical functions also happen this way, and are said to be “vectorized” functions.
In [9]: np.sqrt(x)
Out[9]: array([ 0. , 1. , 1.41421356, 1.73205081, 2. ])
Numpy contains many useful functions for creating matrices. In our earlier lectures we’ve seen linspace
and arange for evenly spaced numbers.
In [10]: np.linspace(0,10,21)
Out[10]: array([ 0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. ,

4.5, 5. , 5.5, 6. , 6.5, 7. , 7.5, 8. , 8.5,
9. , 9.5, 10. ])
In [11]: np.arange(0,10,0.5)
Out[11]: array([ 0. , 0.5, 1. , 1.5, 2. , 2.5, 3. , 3.5, 4. , 4.5, 5. ,

5.5, 6. , 6.5, 7. , 7.5, 8. , 8.5, 9. , 9.5])
Here’s one for creating matrices like coordinates in a grid:
In [12]: xmin=-1.5
ymin=-1.0
xmax=0.5
ymax=1.0
resolution=300
ymatrix, xmatrix=np.mgrid[ymin:ymax:ystep,xmin:xmax:xstep]
In [13]: print ymatrix
358
[[-1. -1. -1. ..., -1. -1. -1. ]
[-0.99333333 -0.99333333 -0.99333333 ..., -0.99333333 -0.99333333
-0.99333333]
[-0.98666667 -0.98666667 -0.98666667 ..., -0.98666667 -0.98666667
-0.98666667]
...,
[ 0.98 0.98 0.98 ..., 0.98 0.98 0.98 ]
[ 0.98666667 0.98666667 0.98666667 ..., 0.98666667 0.98666667
0.98666667]
[ 0.99333333 0.99333333 0.99333333 ..., 0.99333333 0.99333333
0.99333333]]
We can add these together to make a grid containing the complex numbers we want to test for membership
in the Mandelbrot set.
In [14]: values=xmatrix+1j*ymatrix
In [15]: print values
[[-1.50000000-1.j -1.49333333-1.j -1.48666667-1.j ...,

0.48000000-1.j 0.48666667-1.j 0.49333333-1.j ]
[-1.50000000-0.99333333j -1.49333333-0.99333333j -1.48666667-0.99333333j
..., 0.48000000-0.99333333j 0.48666667-0.99333333j
0.49333333-0.99333333j]
[-1.50000000-0.98666667j -1.49333333-0.98666667j -1.48666667-0.98666667j
..., 0.48000000-0.98666667j 0.48666667-0.98666667j
0.49333333-0.98666667j]
...,
[-1.50000000+0.98j -1.49333333+0.98j -1.48666667+0.98j ...,
0.48000000+0.98j 0.48666667+0.98j 0.49333333+0.98j ]
[-1.50000000+0.98666667j -1.49333333+0.98666667j -1.48666667+0.98666667j
..., 0.48000000+0.98666667j 0.48666667+0.98666667j
0.49333333+0.98666667j]
[-1.50000000+0.99333333j -1.49333333+0.99333333j -1.48666667+0.99333333j
..., 0.48000000+0.99333333j 0.48666667+0.99333333j
0.49333333+0.99333333j]]
29.2 Arraywise Algorithms

We can use this to apply the mandelbrot algorithm to whole ARRAYS
In [16]: z0=values
z1=z0*z0+values
z2=z1*z1+values
z3=z2*z2+values
In [17]: print z3
[[ 24.06640625+20.75j 23.16610231+20.97899073j
22.27540349+21.18465854j ..., 11.20523832 -1.88650846j
11.57345330 -1.6076251j 11.94394738 -1.31225596j]
[ 23.82102149+19.85687829j 22.94415031+20.09504528j
22.07634812+20.31020645j ..., 10.93323949 -1.5275283j
11.28531994 -1.24641067j 11.63928527 -0.94911594j]
[ 23.56689029+18.98729242j 22.71312709+19.23410533j
359
21.86791017+19.4582314j ..., 10.65905064 -1.18433756j
10.99529965 -0.90137318j 11.33305161 -0.60254144j]
...,
[ 23.30453709-18.14090998j 22.47355537-18.39585192j
21.65061048-18.62842771j ..., 10.38305264 +0.85663867j
10.70377437 +0.57220289j 11.02562928 +0.27221042j]
[ 23.56689029-18.98729242j 22.71312709-19.23410533j
21.86791017-19.4582314j ..., 10.65905064 +1.18433756j
10.99529965 +0.90137318j 11.33305161 +0.60254144j]
[ 23.82102149-19.85687829j 22.94415031-20.09504528j
22.07634812-20.31020645j ..., 10.93323949 +1.5275283j
11.28531994 +1.24641067j 11.63928527 +0.94911594j]]
So can we just apply our mandel1 function to the whole matrix?

value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
In [19]: mandel1(values)
---------------------------------------------------------------------------
<ipython-input-19-34d5142e7f61> in <module>()
----> 1 mandel1(values)
<ipython-input-18-a590bb4450c8> in mandel1(position, limit)

1 def mandel1(position,limit=50):
2 value=position
----> 3 while abs(value)<2:
4 limit-=1
5 value=value**2+position
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or
No. The logic of our current routine would require stopping for some elements and not for others.
We can ask numpy to vectorise our method for us:
In [20]: mandel2=np.vectorize(mandel1)
In [21]: data5=mandel2(values)

%matplotlib inline
360
Out[22]: <matplotlib.image.AxesImage at 0x1110120d0>
Is that any faster?
In [23]: %%timeit
data5=mandel2(values)
This is not significantly faster. When we use vectorize it’s just hiding an plain old python for loop under
the hood. We want to make the loop over matrix elements take place in the “C Layer”.
What if we just apply the Mandelbrot algorithm without checking for divergence until the end:
In [24]: def mandel_numpy_explode(position,limit=50):

value=position
while limit>0:
limit-=1
diverging=abs(value)>2
return abs(value)<2
In [25]: data6=mandel_numpy_explode(values)
/usr/local/lib/python2.7/site-packages/IPython/kernel/ main .py:5: RuntimeWarning: overflow encountered

/usr/local/lib/python2.7/site-packages/IPython/kernel/ main .py:5: RuntimeWarning: invalid value encoun
/usr/local/lib/python2.7/site-packages/IPython/kernel/ main .py:6: RuntimeWarning: overflow encountered
OK, we need to prevent it from running off to ∞
361
In [26]: def mandel_numpy(position,limit=50):
value=position
while limit>0:
limit-=1
# Avoid overflow
value[diverging]=2
return abs(value)<2
In [27]: data6=mandel_numpy(values)
In [28]: %%timeit
data6=mandel_numpy(values)

%matplotlib inline
Out[29]: <matplotlib.image.AxesImage at 0x111aee810>
Wow, that was TEN TIMES faster.

There’s quite a few NumPy tricks there, let’s look at them:
362
Chapter 30
Logical Arrays
In [30]: diverging=abs(z3)>2
print diverging[30]
[ True True True True True True True True True True True True
True True True True True True True True True True True True
True True True True True True False False False False False False
False False False False False False False False False False False False
False False False False False False False False True True True True
True True True True True True True True True True True True]
When we apply a logical condition to a NumPy array, we get a logical array.
In [31]: x=np.arange(10)*1.0
y=np.ones([10])*5
z=x>y
print z
[False False False False False False True True True True]
Logical arrays can be used to index into arrays:
In [32]: x[x>80]
363
Out[32]: array([], dtype=float64)
In [33]: x[np.logical_not(z)]
Out[33]: array([ 0., 1., 2., 3., 4., 5.])
And you can use such an index as the target of an assignment:
In [34]: x[z]=5
print x
[ 0. 1. 2. 3. 4. 5. 5. 5. 5. 5.]
30.1 Broadcasting
In our example above, we didn’t compare two arrays to get our logical array, but an array to a scalar integer.
When we apply an operation to things of different shapes, NumPy will broadcast the smaller index:
In [35]: print x>5
[False False False False False False False False False False]
This can be used quite creatively:
In [36]: row=np.array([[1,2,3]])
column=np.array([[0],[2],[4]])
print row.shape
(1, 3)
In [37]: print column.shape
(3, 1)
In [38]: row*column
Out[38]: array([[ 0, 0, 0],

[ 2, 4, 6],
[ 4, 8, 12]])
In [39]: x=np.ones([4,1,2])
y=np.ones([1,4,1])
print (x+y).shape
print x+y
(4, 4, 2)
[[[ 2. 2.]
[ 2. 2.]
[ 2. 2.]
[ 2. 2.]]
[[ 2. 2.]
[ 2. 2.]
[ 2. 2.]
[ 2. 2.]]
[[ 2. 2.]
364
[ 2. 2.]
[ 2. 2.]
[ 2. 2.]]
[[ 2. 2.]
[ 2. 2.]
[ 2. 2.]
[ 2. 2.]]]
30.2 More Mandelbrot

Of course, we didn’t calculate the number-of-iterations-to-diverge, just whether the point was in the set.
Let’s correct our code to do that:
value=position
while limit>0:
limit-=1
first_diverged_this_time=np.logical_and(diverging,
diverged_at_count==0)
value[diverging]=2
Out[42]: <matplotlib.image.AxesImage at 0x1118fd110>
365
In [43]: %%timeit
Note that here, all the looping over mandelbrot steps was in Python, but everything below the loop-over-
positions happened in C. The code was amazingly quick compared to pure Python.
Can we do better by avoiding a square root?

value=position
while limit>0:
limit-=1
diverging=value*np.conj(value)>4
first_diverged_this_time=np.logical_and(diverging, diverged_at_count==0)
value[diverging]=2
In [45]: %%timeit
Probably not worth the time I spent thinking about it!
366
Chapter 31
NumPy Testing
Now, let’s look at calculating those residuals, the differences between the different datasets.
In [47]: np.sum((data8-data5)**2)
Out[47]: 0.0
For our non-numpy datasets, numpy knows to turn them into arrays:
sum(sum((data1-data7)**2))
Out[48]: 0.0
But this doesn’t work for pure non-numpy arrays
In [49]: data2=[]
for y in ys:
row=[]
for x in xs:
data2.append(row)
In [50]: data2-data1
---------------------------------------------------------------------------
<ipython-input-50-f2646eae3452> in <module>()
----> 1 data2-data1
TypeError: unsupported operand type(s) for -: ’list’ and ’list’
So we have to convert to NumPy arrays explicitly:
In [51]: sum(sum((np.array(data2)-np.array(data1))**2))
367
Out[51]: 0
NumPy provides some convenient assertions to help us write unit tests with NumPy arrays:
In [52]: x = [1e-5, 1e-3, 1e-1]

y = np.arccos(np.cos(x))
np.testing.assert_allclose(x, y, rtol=1e-6, atol=1e-20)
In [53]: np.testing.assert_allclose(data7, data1)
368
Chapter 32
Arraywise operations are fast
Note that we might worry that we carry on calculating the mandelbrot values for points that have already
diverged.

value=np.zeros(position.shape)+position
calculating=np.ones(position.shape,dtype=’bool’)
while limit>0:
limit-=1
value[calculating]=value[calculating]**2+position[calculating]
diverging_now=np.zeros(position.shape,dtype=’bool’)
diverging_now[calculating]=value[calculating
]*np.conj(value[calculating])>4
calculating=np.logical_and(calculating,
np.logical_not(diverging_now))
diverged_at_count[diverging_now]=limit
In [56]: %%timeit
Out[57]: <matplotlib.image.AxesImage at 0x1122282d0>
369
This was not faster even though it was doing less work
This often happens: on modern computers, branches (if statements, function calls) and memory access
is usually the rate-determining step, not maths.
Complicating your logic to avoid calculations sometimes therefore slows you down. The only way to know
is to measure
370
Chapter 33
Indexing with arrays
We’ve been using Boolean arrays a lot to get access to some elements of an array. We can also do this with
integers:
y=x.reshape([8,8])
y
Out[58]: array([[ 0, 1, 2, 3, 4, 5, 6, 7],

[ 8, 9, 10, 11, 12, 13, 14, 15],
[16, 17, 18, 19, 20, 21, 22, 23],
[24, 25, 26, 27, 28, 29, 30, 31],
[32, 33, 34, 35, 36, 37, 38, 39],
[40, 41, 42, 43, 44, 45, 46, 47],
[48, 49, 50, 51, 52, 53, 54, 55],
[56, 57, 58, 59, 60, 61, 62, 63]])
In [59]: y[[0,5,2]]
Out[59]: array([[ 0, 1, 2, 3, 4, 5, 6, 7],

[40, 41, 42, 43, 44, 45, 46, 47],
[16, 17, 18, 19, 20, 21, 22, 23]])
In [60]: y[[0,2,5],[1,2,7]]
Out[60]: array([ 1, 18, 47])
We can use a : to indicate we want all the values from a particular axis:
In [61]: y[0:8:2,[0,2]]
Out[61]: array([[ 0, 2],

[16, 18],
[32, 34],
[48, 50]])
We can mix array selectors, boolean selectors, :s and ordinary array seqeuencers:
In [62]: z=x.reshape([4,4,4])
print z
371
[[[ 0 1 2 3]
[ 4 5 6 7]
[ 8 9 10 11]
[12 13 14 15]]
[[16 17 18 19]
[20 21 22 23]
[24 25 26 27]
[28 29 30 31]]
[[32 33 34 35]
[36 37 38 39]
[40 41 42 43]
[44 45 46 47]]
[[48 49 50 51]
[52 53 54 55]
[56 57 58 59]
[60 61 62 63]]]
In [63]: z[:,[1,3],0:3]
Out[63]: array([[[ 4, 5, 6],

[12, 13, 14]],
[[20, 21, 22],

[28, 29, 30]],
[[36, 37, 38],

[44, 45, 46]],
[[52, 53, 54],

[60, 61, 62]]])
We can manipulate shapes by adding new indices in selectors with np.newaxis:
In [64]: z[:,np.newaxis,[1,3],0].shape
Out[64]: (4, 1, 2)
When we use basic indexing with integers and : expressions, we get a view on the matrix so a copy is
avoided:
In [65]: a=z[:,:,2]
a[0,0]=-500
z
Out[65]: array([[[ 0, 1, -500, 3],

[ 4, 5, 6, 7],
[ 8, 9, 10, 11],
[ 12, 13, 14, 15]],
[[ 16, 17, 18, 19],

[ 20, 21, 22, 23],
[ 24, 25, 26, 27],
[ 28, 29, 30, 31]],
372
[[ 32, 33, 34, 35],
[ 36, 37, 38, 39],
[ 40, 41, 42, 43],
[ 44, 45, 46, 47]],
[[ 48, 49, 50, 51],

[ 52, 53, 54, 55],
[ 56, 57, 58, 59],
[ 60, 61, 62, 63]]])
We can also use . . . to specify “: for as many as possible intervening axes”:
In [66]: z[1]
Out[66]: array([[16, 17, 18, 19],

[20, 21, 22, 23],
[24, 25, 26, 27],
[28, 29, 30, 31]])
In [67]: z[...,2]
Out[67]: array([[-500, 6, 10, 14],

[ 18, 22, 26, 30],
[ 34, 38, 42, 46],
[ 50, 54, 58, 62]])
However, boolean mask indexing and array filter indexing always causes a copy.
Let’s try again at avoiding doing unnecessary work by using new arrays containing the reduced data
instead of a mask:

positions=np.zeros(position.shape)+position
value=np.zeros(position.shape)+position
indices=np.mgrid[0:values.shape[0],0:values.shape[1]]
while limit>0:
limit-=1
value=value**2+positions
diverging_now=value*np.conj(value)>4
diverging_now_indices=indices[:,diverging_now]
carry_on=np.logical_not(diverging_now)
value=value[carry_on]
indices=indices[:,carry_on]
positions=positions[carry_on]
diverged_at_count[diverging_now_indices[0,:],
diverging_now_indices[1,:]]=limit
Out[70]: <matplotlib.image.AxesImage at 0x1120d54d0>
373
In [71]: %%timeit
Still slower. Probably due to lots of copies.
374
Chapter 34
Scaling for containers and algorithms
We’ve seen that NumPy arrays are really useful. Why wouldn’t we always want to use them for data which
is all the same type?

from timeit import repeat
%matplotlib inline
In [2]: def time_append_to_ndarray(count):

return repeat(’np.append(before,[0])’,
’import numpy as np; before=np.ndarray(’+str(count)+’)’,
number=10000)
In [3]: def time_append_to_list(count):

return repeat(’before.append(0)’,
’before=[0]*’+str(count),
number=10000)
In [4]: counts=np.arange(1,100000,10000)
In [5]: plt.plot(counts,map(time_append_to_list,counts))
plt.ylim(ymin=0)
Out[5]: (0, 0.0013000000000000002)
375
In [6]: plt.plot(counts,map(time_append_to_ndarray,counts))
plt.ylim(ymin=0)
Out[6]: (0, 0.60000000000000009)
Both scale well for accessing the middle element:
376
In [7]: def time_lookup_middle_element_in_list(count):
before=[0]*count
def totime():
x=before[count/2]
return repeat(totime,number=10000)
In [8]: def time_lookup_middle_element_in_ndarray(count):

before=np.ndarray(count)
def totime():
x=before[count/2]
In [9]: plt.plot(counts,map(time_lookup_middle_element_in_list,counts))
plt.ylim(ymin=0)
Out[9]: (0, 0.0045500000000000002)
In [10]: plt.plot(counts,map(time_lookup_middle_element_in_ndarray,counts))
plt.ylim(ymin=0)
Out[10]: (0, 0.0074999999999999997)
377
But a list performs badly for insertions at the beginning:
In [11]: x=range(5)
In [12]: x
Out[12]: [0, 1, 2, 3, 4]
In [13]: x[0:0]=[-1]
In [14]: x
Out[14]: [-1, 0, 1, 2, 3, 4]
In [15]: def time_insert_to_list(count):

return repeat(’before[0:0]=[0]’,’before=[0]*’+str(count),number=10000)
In [16]: plt.plot(counts,map(time_insert_to_list,counts))
plt.ylim(ymin=0)
Out[16]: (0, 0.40000000000000002)
378
There are containers in Python that work well for insertion at the start:
In [17]: from collections import deque
In [18]: def time_insert_to_deque(count):
return repeat(’before.appendleft(0)’,’from collections import deque; before=deque([0]*’+str
In [19]: plt.plot(counts,map(time_insert_to_deque,counts))
plt.ylim(ymin=0)
Out[19]: (0, 0.0014499999999999999)
379
But looking up in the middle scales badly:
In [20]: def time_lookup_middle_element_in_deque(count):

before=deque([0]*count)
def totime():
x=before[count/2]
In [21]: plt.plot(counts,map(time_lookup_middle_element_in_deque,counts))
plt.ylim(ymin=0)
Out[21]: (0, 0.059999999999999998)
What is going on here?

Arrays are stored as contiguous memory. Anything which changes the length of the array requires the
whole array to be copied elsewhere in memory.
This copy takes time proportional to the array size.
The Python list type is also an array, but it is allocated with extra memory. Only when that memory
is exhausted is a copy needed.
If the extra memory is typically the size of the current array, a copy is needed every 1/N appends, and
costs N to make, so on average copies are cheap. We call this amortized constant time.
The deque type works differently: each element contains a pointer to the next. Inserting elements is
therefore very cheap, but looking up the Nth element requires traversing N such pointers.
380
Chapter 35
Dictionary performance
For another example, let’s consider the performance of a dictionary versus a couple of other ways in which
we could implement an associative array.
In [22]: class evildict(object):

def __init__(self,data):
self.data=data
def __getitem__(self,akey):
for key,value in self.data:
if key==akey:
return value
raise KeyError()
In [23]: me=[["Name","James"],["Job","Programmer"],["Home","London"]]
In [24]: me_evil=evildict(me)
In [25]: me_evil["Job"]
Out[25]: ’Programmer’
In [26]: me_dict=dict(me)
In [27]: me_evil["Job"]
In [28]: class sorteddict(object):

def __init__(self,data):
self.data=sorted(data,key=lambda x:x[0])
self.keys=map(lambda x:x[0],self.data)
def __getitem__(self,akey):
from bisect import bisect_left
loc=bisect_left(self.keys,akey)
if loc!=len(self.data):
return self.data[loc][1]
raise KeyError()
In [29]: me_sorted=sorteddict(me)
In [30]: me_sorted["Job"]
381
In [31]: def time_dict_generic(ttype,count,number=10000):
from random import randrange
keys=range(count)
values=[0]*count
data=ttype(zip(keys,values))
def totime():
x=data[keys[count/2]]
In [32]: time_dict=lambda count: time_dict_generic(dict,count)

time_sorted=lambda count: time_dict_generic(sorteddict,count)
time_evil=lambda count: time_dict_generic(evildict,count)
In [33]: counts=np.arange(1,1000,100)
plt.plot(counts,map(time_dict,counts))
plt.ylim(ymin=0)
Out[33]: (0, 0.0050500000000000007)
In [34]: plt.plot(counts,map(time_sorted,counts))
plt.ylim(ymin=0)
Out[34]: (0, 0.041999999999999996)
382
In [35]: plt.plot(counts,map(time_evil,counts))
plt.ylim(ymin=0)
Out[35]: (0, 0.35000000000000003)
383
We can’t really see what’s going on here for the sorted example as there’s too much noise, but theoretically
we should get logarithmic asymptotic performance.
We write this down as O(ln N ). This doesn’t mean there isn’t also a constant term, or a term proportional
to something that grows slower (such as ln(ln N )): we always write down just the term that is dominant for
large N . Similarly, the hash-table based solution used by dict is O(1) and the simple check-each-in-turn
solution is O(N ). We saw before that list is O(1) for appends, O(N ) for inserts. Numpy’s array is O(N )
for appends.
Exercise: determine what the asymptotic peformance for the Boids model in terms of the number of Boids.
Make graphs to support this. Bonus: how would the performance scale with the number of dimensions?
384
Chapter 36
Cython
Cython can be viewed as an extension of Python where variables and functions are annotated with extra
information, in particular types. The resulting Cython source code will be compiled into optimized C or
C++ code, and thereby yielding substantial speed-up of slow Python code. In other word, cython provides
a way of writting Python with comparable performance to that of C/C++.
36.1 Start Coding in Cython

Cython code must, unlike Python, be compiled. This happens in the following stages:
• The cython code in .pyx file will be translated to a C file.

• The C file will be compiled by a C compiler into a shared library, which will be directely loaded into
Python.
In ipython notebook, everything is a lot easier. One need only to load Cython extension (%load ext
Cython) at the beginning and put %%cython mark in front of cells of cython code. Cells with cython mark
will be treated as a .pyx code and consequently, compiled into C.
For details, please see Building Cython Code.
Pure python Mandelbrot set:
In [1]: xmin=-1.5
ymin=-1.0
xmax=0.5
ymax=1.0
resolution=300
In [2]: def mandel(position,limit=50):

value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
Compiled by Cython:
385
In [3]: %load_ext Cython
In [4]: %%cython
def mandel_cython(position,limit=50):
value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
Let’s verify the result

%matplotlib inline
f, axarr = plt.subplots(1,2)
axarr[0].imshow([[mandel(complex(x,y)) for x in xs] for y in ys],interpolation=’none’)
axarr[0].set_title(’Pure Python’)
axarr[1].imshow([[mandel(complex(x,y)) for x in xs] for y in ys],interpolation=’none’)
axarr[1].set_title(’Cython’)
Out[5]: <matplotlib.text.Text at 0x106755b90>
In [6]: %timeit [[mandel(complex(x,y)) for x in xs] for y in ys] # pure python

%timeit [[mandel_cython(complex(x,y)) for x in xs] for y in ys] # cython

We have improved the performance of a factor of 1.5 by just using the cython compiler, without chang-
ing the code!
386
36.2 Cython with C Types
But we can do better by telling Cython what C data type we would use in the code. Note we’re not actually
writing C, we’re writing Python with C types.
typed variable
In [7]: %%cython
def var_typed_mandel_cython(position,limit=50):
cdef double complex value # typed variable
value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
typed function + typed variable
In [8]: %%cython
cpdef call_typed_mandel_cython(double complex position,int limit=50): # typed function
cdef double complex value # typed variable
value=position
while abs(value)<2:
limit-=1
if limit<0:
return 0
return limit
performance of one number:
In [9]: # pure python

%timeit a=mandel(complex(0,0))
In [10]: # premitive cython

%timeit a=mandel_cython(complex(0,0))
In [11]: # cython with C type variable

%timeit a=var_typed_mandel_cython(complex(0,0))
100000 loops, best of 3: 15 µs per loop
In [12]: # cython with typed variable + function

%timeit a=call_typed_mandel_cython(complex(0,0))
36.3 Cython with numpy ndarray

You can use NumPy from Cython exactly the same as in regular Python, but by doing so you are losing
potentially high speedups because Cython has support for fast access to NumPy arrays.
387
ymatrix,xmatrix=np.mgrid[ymin:ymax:ystep,xmin:xmax:xstep]
values=xmatrix+1j*ymatrix
In [14]: %%cython
import numpy as np
cimport numpy as np
cpdef numpy_cython_1(np.ndarray[double complex,ndim=2] position, int limit=50):

cdef np.ndarray[long,ndim=2] diverged_at
cdef double complex value
cdef int xlim
cdef int ylim
cdef double complex pos
cdef int steps
cdef int x, y
xlim=position.shape[1]
ylim=position.shape[0]
diverged_at=np.zeros([ylim, xlim], dtype=int)
for x in xrange(xlim):
for y in xrange(ylim):
steps=limit
value=position[y,x]
pos=position[y,x]
while abs(value)<2 and steps>=0:
steps-=1
value=value**2+pos
diverged_at[y,x]=steps
return diverged_at
Note the double import of numpy: the standard numpy module and a Cython-enabled version of numpy
that ensures fast indexing of and other operations on arrays. Both import statements are necessary in code
that uses numpy arrays. The new thing in the code above is declaration of arrays by np.ndarray.
In [15]: %timeit data_cy=[[mandel(complex(x,y)) for x in xs] for y in ys] # pure python
In [16]: %timeit data_cy=[[call_typed_mandel_cython(complex(x,y)) for x in xs] for y in ys] # typed cyth
In [17]: %timeit numpy_cython_1(values) # ndarray
A trick of using np.vectorize
In [18]: numpy_cython_2=np.vectorize(call_typed_mandel_cython)
In [19]: %timeit numpy_cython_2(values) # vectorize
388
36.4 Calling C functions from Cython
Example: compare sin() from Python and C library
In [20]: %%cython
import math
cpdef py_sin():
cdef int x
cdef double y
for x in xrange(1e7):
y=math.sin(x)
In [21]: %%cython
from libc.math cimport sin as csin # import from C library
cpdef c_sin():
cdef int x
cdef double y
for x in xrange(1e7):
y=csin(x)
In [22]: %timeit [math.sin(i) for i in xrange(int(1e7))] # python
In [23]: %timeit py_sin() # cython call python library
In [24]: %timeit c_sin() # cython call C library
389

Silo - Tips - An Introduction To Python Programming For Research

Uploaded by

Silo - Tips - An Introduction To Python Programming For Research

Uploaded by

An introduction to Python Programming for Research

2 An example Python data analysis notebook 20

9 Control and Flow 57

13 Defining your own classes 74

14 Python not in the Notebook 82

15 Loading data from files 86

16 Getting data from the Internet 92

17 Field and Record Data 95

19 Unassessed Exercise: the biggest Earthquake in the UK this Century 103

20 Plotting with Matplotlib 107

22 The Boids! 130

24 How to Test 205

25 Installing Libraries 256

27 Advanced Python Programming 318

28 Performance programming 352

30 Logical Arrays 363

31 NumPy Testing 367

32 Arraywise operations are fast 369

33 Indexing with arrays 371

34 Scaling for containers and algorithms 375

35 Dictionary performance 381

1.1 Why teach Python?

1.1.1 Why Python?

1.1.2 Why write programs for research?

1.1.3 Sensible Input - Reasonable Output

1.2 Many kinds of Python

Out[1]: [<matplotlib.lines.Line2D at 0x1061cea10>]

In [2]: print "This cell is a code cell"

This cell is a code cell

See the Markdown documentation at This Hyperlink

1.2.2 Typing code in the notebook

1.2.3 Python at the command line

python -c "print 2*4"

1.2.4 Python scripts

1.2.5 Python Libraries

In [6]: %%writefile draw_eight.py

Writing draw eight.py

An example Python data analysis

2.1 Why write software to manage your data and plots?

2.2 The example program: a “graph of green land”.

In [1]: import geopy # A python library for investigating geographic information.

Out[2]: [Location((52.205337, 0.121817, 0.0)), Location((51.73193, -2.3649, 0.0))]

In [3]: print "This runs" # print "This doesn’t"

In [4]: def geolocate(place):

Out[5]: (51.5073509, -0.1277583)

2.2.5 More complex functions

In [7]: import requests

In [8]: map_response=request_map_at(51.5072, -0.1275)

In [10]: from nose.tools import assert_in

2.2.7 Displaying results

In [12]: def map_at(*args,**kwargs):

In [13]: import IPython

The type of our map result is actually a: <type ’str’>

Let’s define what we count as green:

In [17]: def is_green(pixels):

In [18]: def count_green_in_png(data):

In [19]: print count_green_in_png(map_at(*london_location))

In [20]: def location_sequence(start, end, steps):

In [21]: location_sequence(geolocate("London"), geolocate("Cambridge"), 5)

Out[21]: array([[ 5.15073509e+01, -1.27758300e-01],

2.2.9 Creating Images

In [22]: def show_green_in_png(data):

In [25]: for location in location_sequence(geolocate("London"), geolocate("Birmingham"),4):

for location in location_sequence(geolocate("London"),

In [27]: import matplotlib.pyplot as plt

Out[28]: [<matplotlib.lines.Line2D at 0x1054cb410>]

2.2.12 Composing Program Elements

• Find latitude and longitude of a place

In [30]: plt.plot(green_between(’New York’, ’Chicago’, 20))

Out[30]: [<matplotlib.lines.Line2D at 0x1025b6150>]

3.1 Variable Assignment

In [3]: print six

In [4]: print seven

NameError Traceback (most recent call last)

NameError: name ’seven’ is not defined

That’s not the same as an empty box, well labeled:

In [8]: scary = sixsixsix