Key Points

This post originally appeared on the Software Carpentry website.

On the flight back from Vancouver yesterday, I finally did what I should have done eight months ago and compiled the key points from our core lesson content. The results are presented below, broken down by lesson and topic; going forward, we're going to use something like this as a basis for defining what Software Carpentry is, and what workshop attendees can expect to learn.

The Shell

What and Why

The shell is a program whose primary purpose is to read commands, run programs, and display results.

Files and Directories

The file system is responsible for managing information on disk.
Information is stored in files, which are stored in directories (folders).
Directories can also store other directories, which forms a directory tree.
/ on its own is the root directory of the whole filesystem.
A relative path specifies a location starting from the current location.
An absolute path specifies a location from the root of the filesystem.
Directory names in a path are separated with '/' on Unix, but '\' on Windows.
'..' means "the directory above the current one"; '.' on its own means "the current directory".
Most files' names are something.extension; the extension isn't required, and doesn't guarantee anything, but is normally used to indicate the type of data in the file.
cd path changes the current working directory.
ls path prints a listing of a specific file or directory; ls on its own lists the current working directory.
pwd prints the user's current working directory (current default location in the filesystem).
whoami shows the user's current identity.
Most commands take options (flags) which begin with a '-'.

Creating Things

Unix documentation uses '^A' to mean "control-A".
The shell does not have a trash bin: once something is deleted, it's really gone.
mkdir path creates a new directory.
cp old new copies a file.
mv old new moves (renames) a file or directory.
nano is a very simple text editor—please use something else for real work.
rm path removes (deletes) a file.
rmdir path removes (deletes) an empty directory.

Pipes and Filters

'*' is a wildcard pattern that matches zero or more characters in a pathname.
'?' is a wildcard pattern that matches any single character.
The shell matches wildcards before running commands.
command > file redirects a command's output to a file.
first | second is a pipeline: the output of the first command is used as the input to the second.
The best way to use the shell is to use pipes to combine simple single-purpose programs (filters).
cat displays the contents of its inputs.
head displays the first few lines of its input.
sort sorts its inputs.
tail displays the last few lines of its input.
wc counts lines, words, and characters in its inputs.

Loops

Use a for loop to repeat commands once for every thing in a list.
Every for loop needs a variable to refer to the current "thing".
Use $name to expand a variable (i.e., get its value).
Do not use spaces, quotes, or wildcard characters such as '*' or '?' in filenames, as it complicates variable expansion.
Give files consistent names that are easy to match with wildcard patterns to make it easy to select them for looping.
Use the up-arrow key to scroll up through previous commands to edit and repeat them.
Use history to display recent commands, and !number to repeat a command by number.
Use ^C (control-C) to terminate a running command.

Shell Scripts

Save commands in files (usually called shell scripts) for re-use.
Use bash filename to run saved commands.
$* refers to all of a shell script's command-line arguments.
$1, $2, etc., refer to specified command-line arguments.
Letting users decide what files to process is more flexible and more consistent with built-in Unix commands.

Finding Things

Everything is stored as bytes, but the bytes in binary files do not represent characters.
Use nested loops to run commands for every combination of two lists of things.
Use '\' to break one logical line into several physical lines.
Use parentheses '()' to keep things combined.
Use $(command) to insert a command's output in place.
find finds files with specific properties that match patterns.
grep selects lines in files that match patterns.
man command displays the manual page for a given command.

Version Control with Subversion

Version control is a better way to manage shared files than email or shared folders.
The master copy is stored in a repository.
Nobody ever edits the master directory: instead, each person edits a local working copy.
People share changes by committing them to the master or updating their local copy from the master.
The version control system prevents people from overwriting each other's work by forcing them to merge concurrent changes before committing.
It also keeps a complete history of changes made to the master so that old versions can be recovered reliably.
Version control systems work best with text files, but can also handle binary files such as images and Word documents.

Basic Use

Every repository is identified by a URL.
Working copies of different repositories may not overlap.
Each changed to the master copy is identified by a unique revision number.
Revisions identify snapshots of the entire repository, not changes to individual files.
Each change should be commented to make the history more readable.
Commits are transactions: either all changes are successfully committed, or none are.
The basic workflow for version control is update-change-commit.
svn add things tells Subversion to start managing particular files or directories.
svn checkout url checks out a working copy of a repository.
svn commit -m "message" things sends changes to the repository.
svn diff compares the current state of a working copy to the state after the most recent update.
svn diff -r HEAD compares the current state of a working copy to the state of the master copy.
svn history shows the history of a working copy.
svn status shows the status of a working copy.
svn update updates a working copy from the repository.

Merging Conflicts

Conflicts must be resolved before a commit can be completed.
Subversion puts markers in text files to show regions of conflict.
For each conflicted file, Subversion creates auxiliary files containing the common parent, the master version, and the local version.
svn resolve files tells Subversion that conflicts have been resolved.

Recovering Old Versions

Old versions of files can be recovered by merging their old state with their current state.
Recovering an old version of a file does not erase the intervening changes.
Use branches to support parallel independent development.
svn merge merges two revisions of a file.
svn revert undoes local changes to files.

Setting up a Repository

Repositories can be hosted locally, on local (departmental) servers, on hosting services, or on their owners' own domains.
svnadmin create name creates a new repository.

Provenance

$Keyword:$ in a file can be filled in with a property value each time the file is committed.
Put version numbers in programs' output to establish provenance for data.
svn propset svn:keywords property files tells Subversion to start filling in property values.

Basic Programming

Basic Operations

Use '=' to assign a value to a variable.
Assigning to one variable does not change the values associated with other variables.
Use print to display values.
Variables are created when values are assigned to them.
Variables cannot be used until they have been created.
Addition ('+'), subtraction ('-'), and multiplication ('*') work as usual in Python.
Use meaningful, descriptive names for variables.

Creating Programs

Store programs in files whose names end in .py and run them with python name.py.

Types

The most commonly used data types in Python are integers (int), floating-point numbers (float), and strings (str).
Strings can start and end with either single quote (') or double quote (").
Division ('/') produces an int result when given int values: one or both arguments must be float to get a float result.
"Adding" strings concatenates them, multiplying strings by numbers repeats them.
Strings and numbers cannot be added because the behavior is ambiguous: convert one to the other type first.
Variables do not have types, but values do.

Reading Files

Data is either in memory, on disk, or far away.
Most things in Python are objects, and have attached functions called methods.
When lines are read from files, Python keeps their end-of-line characters.
Use str.strip to remove leading and trailing whitespace (including end-of-line characters).
Use file(name, mode) to open a file for reading ('r'), writing ('w'), or appending ('a').
Opening a file for writing erases any existing content.
Use file.readline to read a line from a file.
Use file.close to close an open file.
Use print >> file to print to a file.

Standard Input and Output

The operating system automatically gives every program three open "files" called standard input, standard output, and standard error.
Standard input gets data from the keyboard, from a file when redirected with '<', or from the previous stage in a pipeline with '|'.
Standard output writes data to the screen, to a file when redirected with '>', or to the next stage in a pipeline with '|'.
Standard error also writes data to the screen, and is not redirected by '>' or '|'.
Use import library to import a library.
Use library.thing to refer to something imported from a library.
The sys library provides open "files" called sys.stdin and sys.stdout for standard input and output.

Repeating Things

Use for variable in something: to loop over the parts of something.
The body of a loop must be indented consistently.
The parts of a string are its characters; the parts of a file are its lines.

Making Choices

Use if test to do something only when a condition is true.
Use else to do something when a preceding if test is not true.
The body of an if or else must be indented consistently.
Combine tests using and and or.
Use '<', '<=', '>=', and '>' to compare numbers or strings.
Use '==' to test for equality and '!=' to test for inequality.
Use variable += expression as a shorthand for variable = variable + expression (and similarly for other arithmetic operations).

Flags

The two Boolean values True and False can be assigned to variables like any other values.
Programs often use Boolean values as flags to indicate whether something has happened yet or not.

Reading Data Files

Use str.split() to split a string into pieces on whitespace.
Values can be assigned to any number of variables at once.

Provenance Revisited

Put version numbers in programs' output to establish provenance for data.

Lists

Use [value, value, ...] to create a list of values.
for loops process the elements of a list, in order.
len(list) returns the length of a list.
[] is an empty list with no values.

More About Lists

Lists are mutable: they can be changed in place.
Use list.append(value) to append something to the end of a list.
Use list[index] to access a list element by location.
The index of the first element of a list is 0; the index of the last element is len(list)-1.
Negative indices count backward from the end of the list, so list[-1] is the last element.
Trying to access an element with an out-of-bounds index is an error.
range(number) produces the list of numbers [0, 1, ..., number-1].
range(len(list)) produces the list of legal indices for list.

Checking and Smoothing Data

range(start, end) creates the list of numbers from start up to, but not including, end.
range(start, end, stride) creates the list of numbers from start up to end in steps of stride.

Nesting Loops

Use nested loops to do things for combinations of things.
Make the range of the inner loop depend on the state of the outer loop to automatically adjust how much data is processed.
Use min(...) and max(...) to find the minimum and maximum of any number of values.

Nesting Lists

Use nested lists to store multi-dimensional data or values that have regular internal structure (such as XYZ coordinates).
Use list_of_lists[first] to access an entire sub-list.
Use list_of_lists[first][second] to access a particular element of a sub-list.
Use nested loops to process nested lists.

Aliasing

Several variables can alias the same data.
If that data is mutable (e.g., a list), a change made through one variable is visible through all other aliases.

Functions and Libraries

How Functions Work

Define a function using def name(...)
The body of a function must be indented.
Use name(...) to call a function.
Use return to return a value from a function.
The values passed into a function are assigned to its parameters in left-to-right order.
Function calls are recorded on a call stack.
Every function call creates a new stack frame.
The variables in a stack frame are discarded when the function call completes.
Grouping operations in functions makes code easier to understand and re-use.

Global Variables

Every function always has access to variables defined in the global scope.
Programmers often write constants' names in upper case to make their intention easier to recognize.
Functions should not communicate by modifying global variables.

Multiple Arguments

A function may take any number of arguments.
Define default values for parameters to make functions more convenient to use.
Defining default values only makes sense when there are sensible defaults.

Returning Values

A function may return values at any point.
A function should have zero or more return statements at its start to handle special cases, and then one at the end to handle the general case.
"Accidentally" correct behavior is hard to understand.
If a function ends without an explicit return, it returns None.

Aliasing

Values are actually passed into functions by reference, which means that they are aliased.
Aliasing means that changes made to a mutable object like a list inside a function are visible after the function call completes.

Libraries

Any Python file can be imported as a library.
The code in a file is executed when it is imported.
Every Python file is a scope, just like every function.

Standard Libraries

Use from library import something to import something under its own name.
Use from library import something as alias to import something under the name alias.
from library import * imports everything in library under its own name, which is usually a bad idea.
The math library defines common mathematical constants and functions.
The system library sys defines constants and functions used in the interpreter itself.
sys.argv is a list of all the command-line arguments used to run the program.
sys.argv[0] is the program's name.
sys.argv[1:] is everything except the program's name.

Building Filters

If a program isn't told what files to process, it should process standard input.
Programs that explicitly test values' types are more brittle than ones that rely on those values' common properties.
The variable __name__ is assigned the string '__main__' in a module when that module is the main program, and the module's name when it is imported by something else.
If the first thing in a module or function is a string that isn't assigned to a variable, that string is used as the module or function's documentation.
Use help(name) to display the documentation for something.

Functions as Objects

A function is just another kind of data.
Defining a function creates a function object and assigns it to a variable.
Functions can be assigned to other variables, put in lists, and passed as parameters.
Writing higher-order functions helps eliminate redundancy in programs.
Use filter to select values from a list.
Use map to apply a function to each element of a list.
Use reduce to combine the elements of a list.

Databases

A relational database stores information in tables with fields and records.
A database manager is a program that manipulates a database.
The commands or queries given to a database manager are usually written in a specialized language called SQL.

Selecting

SQL is case insensitive.
The rows and columns of a database table aren't stored in any particular order.
Use SELECT fields FROM table to get all the values for specific fields from a single table.
Use SELECT * FROM table to select everything from a table.

Removing Duplicates

Use SELECT DISTINCT to eliminate duplicates from a query's output.

Calculating New Values

Use expressions in place of field names to calculate per-record values.

Filtering

Use WHERE test in a query to filter records based on logical tests.
Use AND and OR to combine tests in filters.
Use IN to test whether a value is in a set.
Build up queries a bit at a time, and test them against small data sets.

Sorting

Use ORDER BY field ASC (or DESC) to order a query's results in ascending (or descending) order.

Aggregation

Use aggregation functions like SUM MAX to combine many query results into a single value.
Use the COUNT function to count the number of results.
If some fields are aggregated, and others are not, the database manager displays an arbitrary result for the unaggregated field.
Use GROUP BY to group values before aggregation.

Database Design

Each field in a database table should store a single atomic value.
No fact in a database should ever be duplicated.

Combining Data

Use JOIN to create all possible combinations of records from two or more tables.
Use JOIN tables ON test to keep only those combinations that pass some test.
Use table.field to specify a particular field of a particular table.
Use aliases to make queries more readable.
Every record in a table should be uniquely identified by the value of its primary key.

Self Join

Use a self join to combine a table with itself.

Missing Data

Use NULL in place of missing information.
Almost every operation involving NULL produces NULL as a result.
Test for nulls using IS NULL and IS NOT NULL.
Most aggregation functions skip nulls when combining values.

Nested Queries

Use nested queries to create temporary sets of results for further querying.
Use nested queries to subtract unwanted results from all results to leave desired results.

Creating and Modifying Tables

Use CREATE TABlE name(...) to create a table.
Use DROP TABLE name to erase a table.
Specify field names and types when creating tables.
Specify PRIMARY KEY, NOT NULL, and other constraints when creating tables.
Use INSERT INTO table VALUES(...) to add records to a table.
Use DELETE FROM table WHERE test to erase records from a table.
Maintain referential integrity when creating or deleting information.

Transactions

Place operations in a transaction to ensure that they appear to be atomic, consistent, isolated, and durable.

Programming With Databases

Most applications that use databases embed SQL in a general-purpose programming language.
Database libraries use connections and cursors to manage interactions.
Programs can fetch all results at once, or a few results at a time.
If queries are constructed dynamically using input from users, malicious users may be able to inject their own commands into the queries.
Dynamically-constructed queries can use SQL's native formatting to safeguard against such attacks.

Number Crunching with NumPy

High-level libraries are usually more efficient for numerical programming than hand-coded loops.
Most such libraries use a data-parallel programming model.
Arrays can be used as matrices, as physical grids, or to store general multi-dimensional data.

Basics

NumPy is a high-level array library for Python.
import numpy to import NumPy into a program.
Use numpy.array(values) to create an array.
Initial values must be provided in a list (or a list of lists).
NumPy arrays store homogeneous values whose type is identified by array.dtype.
Use old.astype(newtype) to create a new array with a different type rather than assigning to dtype.
numpy.zeros creates a new array filled with 0.
numpy.ones creates a new array filled with 1.
numpy.identity creates a new identity matrix.
numpy.empty creates an array but does not initialize its values (which means they are unpredictable).
Assigning an array to a variable creates an alias rather than copying the array.
Use array.copy to create a copy of an array.
Put all array indices in a single set of square brackets, like array[i0, i1].
array.shape is a tuple of the array's size in each dimension.
array.size is the total number of elements in the array.

Storage

Arrays are stored using descriptors and data blocks.
Many operations create a new descriptor, but alias the original data block.
Array elements are stored in row-major order.
array.transpose creates a transposed alias for an array's data.
array.ravel creates a one-dimensional alias for an array's data.
array.reshape creates an arbitrarily-shaped alias for an array's data.
array.resize resizes an array's data in place, filling with zero as necessary.

Indexing

Arrays can be sliced using start:end:stride along each axis.
Values can be assigned to slices as well as read from them.
Arrays can be used as subscripts to select items in arbitrary ways.
Masks containing True and False can be used to select subsets of elements from arrays.
Use '&' and '|' (or logical_and and logical_or) to combine tests when subscripting arrays.
Use where, choose, or select to select elements or alternatives in a single step.

Linear Algebra

Addition, multiplication, and other arithmetic operations work on arrays element-by-element.
Operations involving arrays and scalars combine the scalar with each element of the array.
array.dot performs "real" matrix multiplication.
array.sum calculates sums or partial sums of array elements.
array.mean calculates array averages.

Making Recommendations

Getting data in the right format for processing often requires more code than actually processing it.
Data with many gaps should be stored in sparse arrays.
numpy.cov calculates variancess and covariances.

The Game of Life

Padding arrays with fixed elements is an easy way to implement boundary conditions.
scipy.signal.convolve applies a weighted mask to each element of an array.

Quality

Defensive Programming

Design programs to catch both internal errors and usage errors.
Use assertions to check whether things that ought to be true in a program actually are.
Assertions help people understand how programs work.
Fail early, fail often.
When bugs are fixed, add assertions to the program to prevent their reappearance.

Handling Errors

Use raise to raise exceptions.
Raise exceptions to report errors rather than trying to handle them inline.
Use try and except to handle exceptions.
Catch exceptions where something useful can be done about the underlying problem.
An exception raised in a function may be caught anywhere in the active call stack.

Unit Testing

Testing cannot prove that a program is correct, but is still worth doing.
Use a unit testing library like Nose to test short pieces of code.
Write each test as a function that creates a fixture, executes an operation, and checks the result using assertions.
Every test should be able to run independently: tests should not depend on one another.
Focus testing on boundary cases.
Writing tests helps us design better code by clarifying our intentions.

Numbers

Floating point numbers are approximations to actual values.
Use tolerances rather than exact equality when comparing floating point values.
Use integers to count and floating point numbers to measure.
Most tests should be written in terms of relative error rather than absolute error.
When testing scientific software, compare results to exact analytic solutions, experimental data, or results from simpler or previously-tested programs.

Coverage

Use a coverage analyzer to see which parts of a program have been tested and which have not.

Debugging

Use an interactive symbolic debugger instead of print statements to diagnose problems.
Set breakpoints to halt the program at interesting points instead of stepping through execution.
Try to get things right the first time.
Make sure you know what the program is supposed to do before trying to debug it.
Make sure the program is actually running the test case you think it is.
Make the program fail reliably.
Simplify the test case or the program in order to localize the problem.
Change one thing at a time.
Be humble.

Designing Testable Code

Separating interface from implementation makes code easier to test and re-use.
Replace some components with simplified versions of themselves in order to simplify testing of other components.
Do not create arbitrary, variable, or random results, as they are extremely hard to test.
Isolate interactions with the outside world when writing tests.

Sets and Dictionaries

Sets

Use sets to store distinct unique values.
Create sets using set() or {v1, v2, ...}.
Sets are mutable, i.e., they can be updated in place like lists.
A loop over a set produces each element once, in arbitrary order.
Use sets to find unique things.

Storage

Sets are stored in hash tables, which guarantee fast access for arbitrary keys.
The values in sets must be immutable to prevent hash tables misplacing them.
Use tuples to store multi-part elements in sets.

Dictionaries

Use dictionaries to store key-value pairs with distinct keys.
Create dictionaries using {k1:v1, k2:v2, ...}
Dictionaries are mutable, i.e., they can be updated in place.
Dictionary keys must be immutable, but values can be anything.
Use tuples to store multi-part keys in dictionaries.
dict[key] refers to the dictionary entry with a particular key.
key in dict tests whether a key is in a dictionary.
len(dict) returns the number of entries in a dictionary.
A loop over a dictionary produces each key once, in arbitrary order.
dict.keys() creates a list of the keys in a dictionary.
dict.values() creates a list of the keys in a dictionary.

Simple Examples

Use dictionaries to count things.
Initialize values from actual data instead of trying to guess what values could "never" occur.

Phylogenetic Trees

Problems that are described using matrices can often be solved more efficiently using dictionaries.
When using tuples as multi-part dictionary keys, order the tuple entries to avoid accidental duplication.

Development

The Grid

Get something simple working, then start to add features, rather than putting everything in the program at the start.
Leave FIXME markers in programs as you are developing them to remind yourself what still needs to be done.

Aliasing

Draw pictures of data structures to aid debugging.

Randomness

Use a well-tested random number generation library to generate pseudorandom values.
If a random number generation library is given the same seed, it will produce the same sequence of values.

Neighbors

and and or stop evaluating arguments as soon as they have an answer.

Bugs

Test programs with successively more complex cases.

Refactoring

Refactor programs as necessary to make testing easier.
Replace randomness with predictability to make testing easier.

Performance

Scientists want faster programs both to handle bigger problems and to handle more problems with available resources.
Before speeding a program up, ask, "Does it need to be faster?" and, "Is it correct?"
Recording start and end times is a simple way to measure performance.
Analyze algorithms to predict how a program's performance will change with problem size.

Profiling

Use a profiler to determine which parts of a program are responsible for most of its running time.

A New Beginning

Better algorithms are better than better hardware.