These are the notes for Data Analytics with Python, episode 9. In episode 8 we built NumPy arrays and did maths on whole arrays at once. Here we learn to pick exactly the data we want out of an array (indexing, slicing, boolean masks, fancy indexing), to change its shape (reshape, flatten, transpose), and to summarise it along rows or columns (axis, sum, mean, min, max, std, argmax, argmin), plus np.where, np.sort and np.unique. Every term is defined the first time it appears and collected in the Glossary. Every program below was run for real (Python 3.13, NumPy 2.x); outputs are unedited.
The problem: a month of sales in a grid
Ramesh's chai stall now has a whole month of sales: 4 weeks, 7 days each. In his notebook he writes one line per week. That is a grid (a table of numbers):
- A row is one line going across: here, one week.
- A column is one line going down: here, one weekday (every Monday, every Tuesday, ...).
- An array with rows and columns is two-dimensional (2-D). A plain list of numbers is one-dimensional (1-D).
- The shape (from episode 8) lists the size in each direction:
(4, 7)means 4 rows and 7 columns.
Every example below starts from this array (rows = weeks 1 to 4, columns = Mon to Sun):
import numpy as np
cups = np.array([[320, 280, 410, 300, 350, 390, 450],
[310, 295, 380, 330, 360, 420, 470],
[290, 260, 400, 310, 340, 405, 440],
[330, 300, 430, 320, 370, 415, 480]])
Ramesh asks: how many cups on Wednesday of week 2? We need a way to point at exactly one cell.
Indexing: cups[row, column]
Real-life example: a cinema ticket. "Row C, seat 5": two pieces of information find exactly one seat.
Indexing means getting a value by its position. A position (or index) is an item's place number, and NumPy, like Python lists (episode 5), counts from 0. So week 2 is row 1, and Wednesday is column 2. Negative indexes count from the end: -1 is the last.
print(cups.shape)
print(cups[1][2])
print(cups[1, 2])
print(cups[-1, -1])
(4, 7)
380
380
480
cups[1][2] works (it first takes row 1, then item 2 of that row), but the NumPy way is one bracket with a comma: cups[1, 2]. It is faster and it is the form used everywhere (pandas' .iloc uses it too).
Asking for a row that doesn't exist:
print(cups[4, 0])
Traceback (most recent call last):
File "main.py", line 6, in <module>
print(cups[4, 0])
~~~~^^^^^^
IndexError: index 4 is out of bounds for axis 0 with size 4
Out of bounds means outside the grid: only rows 0 to 3 exist. Note the word axis in the message: it is explained below.
Slicing rows and columns
Slicing takes a range of positions: start:stop, where stop is not included (exactly like lists). In 2-D you slice rows and columns separately, with a comma: cups[rows, columns]. A colon on its own, :, means all rows or all columns. An optional third number is the step.
print(cups[1])
print(cups[:, 0])
print(cups[0:2, 5:7])
print(cups[0, ::2])
[310 295 380 330 360 420 470]
[320 310 290 330]
[[390 450]
[420 470]]
[320 410 350 450]
| Code | Means | Result |
|---|---|---|
cups[1] |
row 1 (week 2), all columns | 7 numbers |
cups[:, 0] |
all rows, column 0 | every Monday |
cups[0:2, 5:7] |
rows 0-1, columns 5-6 | weekends of weeks 1-2 (2 x 2) |
cups[0, ::2] |
row 0, every 2nd column | Mon, Wed, Fri, Sun of week 1 |
Comparison arrays and boolean masks
New problem: which days sold more than 400 cups? This is about values, not positions: we don't know where they are.
The loop way from episode 6 works, but it is long, and the obvious shortcut fails on a plain list:
week = [320, 280, 410, 300, 350, 390, 450]
busy = []
for c in week:
if c > 400:
busy.append(c)
print(busy)
print(week > 400)
[410, 450]
Traceback (most recent call last):
File "main.py", line 7, in <module>
print(week > 400)
^^^^^^^^^^
TypeError: '>' not supported between instances of 'list' and 'int'
On a NumPy array, comparisons are element-wise (applied to each element, one by one, as in episode 8):
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week > 400)
print(week[week > 400])
big = week > 400
print(big.sum(), "busy days")
[False False True False False False True]
[410 450]
2 busy days
week > 400is a comparison array: a boolean array (boolean = True or False) with one answer per element.- Put a boolean array inside the brackets and NumPy keeps only the positions marked
True. That is a boolean mask. Truecounts as 1 andFalseas 0, somask.sum()counts the matches.
Real-life example: a stencil. A card with holes cut in it. The True/False array is the card (True = a hole); laid on the data, you only see the values under the holes.
Masks work on 2-D arrays too; the result is a flat 1-D array of the matching values:
print(cups > 440)
print(cups[cups > 440])
[[False False False False False False True]
[False False False False False False True]
[False False False False False False False]
[False False False False False False True]]
[450 470 480]
All three are Sundays.
Combining conditions: &, |, ~ and the parentheses rule
Normal days: more than 300 and less than 400. Two wrong ways first.
Without brackets:
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[week > 300 & week < 400])
Traceback (most recent call last):
File "main.py", line 3, in <module>
print(week[week > 300 & week < 400])
^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
With the word and:
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[(week > 300) and (week < 400)])
Traceback (most recent call last):
File "main.py", line 3, in <module>
print(week[(week > 300) and (week < 400)])
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all()
Why: and, or and not need one True or False. An array holds many, so Python can't decide whether the whole array is "true" (ambiguous = it could mean two things). And Python evaluates & before > and <, so without brackets week > 300 & week < 400 is read as week > (300 & week) < 400, which again needs a single True/False.
The fix: NumPy's element-wise operators, with brackets around every condition:
| Operator | Means | Example |
|---|---|---|
& (ampersand) |
and, cell by cell | (week > 300) & (week < 400) |
| (pipe) |
or, cell by cell | (week < 300) | (week > 400) |
~ (tilde) |
not: flips each True/False | ~(week > 400) |
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week[(week > 300) & (week < 400)])
print(week[(week < 300) | (week > 400)])
print(week[~(week > 400)])
[320 350 390]
[280 410 450]
[320 280 300 350 390]
.any() (is at least one True?) and .all() (are all True?), the methods the error suggests, are for when you really want a single answer about the whole array.
Fancy indexing: a list of positions
Just Monday and Sunday, positions 0 and 6: not a range, not a condition. Pass a list of positions inside the brackets. That is fancy indexing. Positions can repeat and be in any order.
week = cups[0]
print(week[[0, 6]])
print(week[[6, 0, 0]])
print(cups[[0, 3]])
[320 450]
[450 320 320]
[[320 280 410 300 350 390 450]
[330 300 430 320 370 415 480]]
On a 2-D array, a list of row numbers picks whole rows (weeks 1 and 4).
Reshaping: reshape, flatten, transpose
New problem: the billing app exports the month as one long line of 28 numbers (shape (28,)). With no rows or columns, "every Monday" means position arithmetic by hand (0, 7, 14, 21).
Reshaping changes an array's shape without changing its data. The values fill the new shape row by row.
import numpy as np
log = np.array([320, 280, 410, 300, 350, 390, 450,
310, 295, 380, 330, 360, 420, 470,
290, 260, 400, 310, 340, 405, 440,
330, 300, 430, 320, 370, 415, 480])
print(log.shape)
cups = log.reshape(4, 7)
print(cups)
print(cups.shape)
(28,)
[[320 280 410 300 350 390 450]
[310 295 380 330 360 420 470]
[290 260 400 310 340 405 440]
[330 300 430 320 370 415 480]]
(4, 7)
The rule: the new shape must hold exactly the same number of items (the size, from episode 8). np.arange(28) is 0 to 27.
import numpy as np
log = np.arange(28)
print(log.reshape(5, 7))
Traceback (most recent call last):
File "main.py", line 3, in <module>
print(log.reshape(5, 7))
~~~~~~~~~~~^^^^^^
ValueError: cannot reshape array of size 28 into shape (5,7)
5 x 7 = 35, not 28.
Shortcuts:
-1in a shape means "work this one out for me":reshape(-1, 7)gives(4, 7).flatten()turns any array back into one line (a 1-D copy).- Transpose (
.T) swaps rows and columns: row 1 becomes column 1. Shape(4, 7)becomes(7, 4).
import numpy as np
log = np.arange(1, 29)
print(log.reshape(-1, 7).shape)
print(log.reshape(2, -1).shape)
small = np.array([[1, 2, 3],
[4, 5, 6]])
print(small.flatten())
print(small.T)
print(small.shape, small.T.shape)
(4, 7)
(2, 14)
[1 2 3 4 5 6]
[[1 4]
[2 5]
[3 6]]
(2, 3) (3, 2)
Real-life example: an egg tray. Reshape = putting 28 eggs into a 4 x 7 tray, one row at a time (a 5 x 7 tray would have 7 empty holes, so NumPy refuses). Flatten = tipping the tray back into one line. Transpose = turning the tray on its side.
Axis: totals per row or per column
New problem: two questions that sound alike: what did I sell each week? and what did I sell on each weekday (all Mondays together...)?
Real-life example: a school marks register. Rows = students, columns = subjects. The class teacher totals each student by adding across a row; the subject teacher totals each subject by adding down a column.
Each direction of an array is an axis. For a 2-D array, axis 0 runs down the rows and axis 1 runs across the columns; the shape (4, 7) lists them in that order.
sum(axis=1): squashes the columns → one number per row (per week).sum(axis=0): squashes the rows → one number per column (per weekday).- No axis: one number for the whole array.
Memory trick: the axis you name is the one that disappears.
axis 1 (columns) →
Mon Tue Wed Thu Fri Sat Sun sum(axis=1)
axis 0 wk1 320 280 410 300 350 390 450 → 2500
(rows) wk2 310 295 380 330 360 420 470 → 2565
↓ wk3 290 260 400 310 340 405 440 → 2445
wk4 330 300 430 320 370 415 480 → 2645
↓ ↓ ↓ ↓ ↓ ↓ ↓
sum(axis=0) 1250 1135 1620 1260 1420 1630 1840
print(cups.sum())
print(cups.sum(axis=1))
print(cups.sum(axis=0))
print(cups.mean(axis=0))
10155
[2500 2565 2445 2645]
[1250 1135 1620 1260 1420 1630 1840]
[312.5 283.75 405. 315. 355. 407.5 460. ]
10,155 cups this month; week 4 was best (2,645); Sunday averages 460 cups and Tuesday under 284.
Aggregation: sum, mean, min, max, std
Aggregation turns many numbers into one summary number:
| Function | Gives | Example on the month |
|---|---|---|
sum() |
total | 10155 |
mean() |
average (total ÷ count) | about 362.7 |
min() |
smallest value | 260 |
max() |
largest value | 480 |
std() |
standard deviation (spread) | about 60.7 |
All of them accept axis=.
Standard deviation measures spread: roughly, how far the values typically sit from the mean. Two stalls both average 350 cups; one sells exactly 350 every day (std 0), the other swings between 200 and 500 (std about 127.5). Same mean, very different business.
print(cups.min(), cups.max())
print(cups.max(axis=1))
print(round(cups.mean(), 1), round(cups.std(), 1))
steady = np.array([350, 350, 350, 350])
wild = np.array([200, 500, 250, 450])
print(steady.mean(), steady.std())
print(wild.mean(), wild.std().round(1))
260 480
[450 470 440 480]
362.7 60.7
350.0 0.0
350.0 127.5
How it is computed: subtract the mean from each value, square the differences, average them (that average is the variance), then take the square root. NumPy's std() divides by the number of values n (the population formula, ddof=0). The sample version divides by n - 1 (std(ddof=1), and pandas' default); it comes back in the statistics episodes.
argmax and argmin: which position?
max() says the best number was 450, but Ramesh wants to know which day. argmax() returns the position of the largest value; argmin() the position of the smallest. Then use the position to index a labels array (a label is a human-readable name for a position, like "Sun").
import numpy as np
days = np.array(["Mon", "Tue", "Wed", "Thu",
"Fri", "Sat", "Sun"])
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(week.max(), week.argmax())
print("Best day:", days[week.argmax()])
print("Worst day:", days[week.argmin()])
450 6
Best day: Sun
Worst day: Tue
On a 2-D array:
print(cups.argmax())
print(cups.argmax(axis=1))
print(cups.argmin(axis=0))
27
[6 6 6 6]
[2 2 1 0 2 0 2]
- No axis: the position as if the array were flattened (27 = the last cell).
axis=1: each week's best day (column 6, Sunday, every week).axis=0: for each weekday, the weakest week (row number).
np.where: labels, positions and replacing values
np.where(condition, value_if_true, value_if_false) is the array version of episode 6's conditional expression. With only a condition, it returns the positions where it is True.
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
labels = np.where(week >= 400, "busy", "ok")
print(labels)
print(np.where(week >= 400))
print(np.where(week < 300, 300, week))
['ok' 'ok' 'busy' 'ok' 'ok' 'ok' 'busy']
(array([2, 6]),)
[320 300 410 300 350 390 450]
The last line replaces every value under 300 with 300 and keeps the rest: a common cleaning step (capping).
np.sort and np.unique
np.sort(a)returns a sorted copy (smallest first); the original is unchanged.[::-1]reverses it (largest first).np.unique(a)returns each distinct value once, sorted. Withreturn_counts=Trueit also returns how many times each appears: a frequency table (pandas calls thisvalue_counts, episode 11).
import numpy as np
week = np.array([320, 280, 410, 300, 350, 390, 450])
print(np.sort(week))
print(np.sort(week)[::-1])
print(week)
snacks = np.array(["samosa", "chai", "chai", "bun",
"chai", "samosa", "chai"])
print(np.unique(snacks))
names, counts = np.unique(snacks, return_counts=True)
print(names, counts)
[280 300 320 350 390 410 450]
[450 410 390 350 320 300 280]
[320 280 410 300 350 390 450]
['bun' 'chai' 'samosa']
['bun' 'chai' 'samosa'] [1 4 2]
Mini project: the monthly report
From the flat export to a full report, without a single loop.
import numpy as np
log = np.array([320, 280, 410, 300, 350, 390, 450,
310, 295, 380, 330, 360, 420, 470,
290, 260, 400, 310, 340, 405, 440,
330, 300, 430, 320, 370, 415, 480])
days = np.array(["Mon", "Tue", "Wed", "Thu",
"Fri", "Sat", "Sun"])
cups = log.reshape(4, 7)
weekly = cups.sum(axis=1)
print("Weekly totals:", weekly)
print("Best week:", weekly.argmax() + 1)
avg = cups.mean(axis=0)
print("Best weekday:", days[avg.argmax()])
print("Slowest weekday:", days[avg.argmin()])
print("Busy days (400+):", (cups >= 400).sum())
print("Std per weekday:", cups.std(axis=0).round(1))
Weekly totals: [2500 2565 2445 2645]
Best week: 4
Best weekday: Sun
Slowest weekday: Tue
Busy days (400+): 10
Std per weekday: [14.8 15.6 18. 11.2 11.2 11.5 15.8]
+ 1 because people count weeks from 1. Thursday and Friday have the smallest spread: the most predictable days.
Common mistakes
| Code | Real error | Why | Fix |
|---|---|---|---|
cups[4, 0] (only 4 rows) |
IndexError: index 4 is out of bounds for axis 0 with size 4 |
Positions start at 0: rows are 0-3 | cups[3, 0] or cups[-1, 0] |
week > 400 on a plain list |
TypeError: '>' not supported between instances of 'list' and 'int' |
Lists don't compare element-wise | np.array(week) > 400 |
week[week > 300 & week < 400] |
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() |
& runs before > and < |
week[(week > 300) & (week < 400)] |
week[(week > 300) and (week < 400)] |
ValueError: The truth value of an array with more than one element is ambiguous. Use a.any() or a.all() |
and needs one True/False |
Use &, |, ~ |
np.arange(28).reshape(5, 7) |
ValueError: cannot reshape array of size 28 into shape (5,7) |
New shape must hold the same count | reshape(4, 7) or reshape(-1, 7) |
cups.sum(axis=0) for weekly totals |
(no error: 7 numbers instead of 4) | The named axis disappears | cups.sum(axis=1) |
week.max() to find the best day |
(no error: a value, not a day) | max gives the value |
days[week.argmax()] |
Practice
Use this step-count array (3 weeks x 7 days):
import numpy as np
steps = np.array([[6200, 4800, 9100, 7500, 3900, 11000, 8600],
[5400, 7200, 8800, 4100, 6600, 12500, 9900],
[7000, 4500, 9300, 6800, 5200, 10400, 4700]])
days = np.array(["Mon", "Tue", "Wed", "Thu", "Fri", "Sat", "Sun"])
1. Total steps for each week, and the best weekday on average.
Solution
print(steps.sum(axis=1))
print(days[steps.mean(axis=0).argmax()])
[51100 54500 47900]
Sat
2. List the days under 5,000 steps and count them, two ways.
Solution
low = steps[steps < 5000]
print(low, len(low), (steps < 5000).sum())
[4800 3900 4100 4500 4700] 5 5
3. Label every day "active" (8,000+) or "rest".
Solution
print(np.where(steps >= 8000, "active", "rest"))
[['rest' 'rest' 'active' 'rest' 'rest' 'active' 'active']
['rest' 'rest' 'active' 'rest' 'rest' 'active' 'active']
['rest' 'rest' 'active' 'rest' 'rest' 'active' 'rest']]
4. With week = np.array([320, 280, 410, 300, 350, 390, 450]): the days from 300 to 350 inclusive, the 3 largest values, and their positions (largest first).
Solution
print(week[(week >= 300) & (week <= 350)])
print(np.sort(week)[-3:])
print(week.argsort()[::-1][:3])
[320 300 350]
[390 410 450]
[6 2 5]
argsort() returns the positions that would sort the array.
5. On the month cups: total weekend cups (Sat + Sun), the shape of the transpose, and the weeks with a total over 2,500.
Solution
print(cups[:, 5:].sum())
print(cups.T.shape)
print(cups[cups.sum(axis=1) > 2500])
3470
(7, 4)
[[310 295 380 330 360 420 470]
[330 300 430 320 370 415 480]]
The last one is a boolean mask over rows: one True/False per week.
Quick quiz
a = np.array([[1, 2, 3],
[4, 5, 6]])
print(a.sum(axis=0))
A. [ 6 15] B. [5 7 9] C. 21 D. [1 4]
Answer
B. axis=0 squashes the rows: one total per column (1+4, 2+5, 3+6). [ 6 15] would be axis=1, and 21 is no axis.
Interview questions
- What is a boolean mask? A True/False array of the same shape as the data; indexing with it keeps only the True positions (
a[a > 400]). - Why does
a[a > 1 and a < 5]fail?andneeds a single True/False, but an array comparison gives many:ValueError: The truth value of an array ... is ambiguous. Use(a > 1) & (a < 5). - Why are parentheses needed with
&?&has higher precedence than>/<, soa > 1 & a < 5is evaluated asa > (1 & a) < 5. - What does
axis=0mean insum? Aggregate down the rows (axis 0 disappears): one result per column. argmaxvsmax?maxreturns the largest value;argmaxreturns its position (flattened position if no axis).- What does
reshape(-1, 7)do? Makes 7 columns and works out the number of rows; the total size must divide evenly. np.where(cond)vsnp.where(cond, x, y)? One argument: the positions wherecondis True. Three: a new array withxwhere True,ywhere False.- Does slicing copy the data? Basic slicing returns a view (it shares memory with the original); boolean masks and fancy indexing return copies. Episode 10 covers copy vs view in detail.
Glossary
| Term | Meaning | Example |
|---|---|---|
| Grid / table | Numbers arranged in rows and columns | 4 weeks x 7 days |
| Row | One line going across | week 2: cups[1] |
| Column | One line going down | every Monday: cups[:, 0] |
| 1-D array | An array with one direction, like a list | np.array([320, 280]) |
| 2-D array | An array with rows and columns | cups, shape (4, 7) |
| Shape | Size in each direction | (4, 7) |
| Size | Total number of items | 28 |
| Index / position | An item's place number, counted from 0 | Wednesday = 2 |
| Indexing | Getting a value by its position | cups[1, 2] → 380 |
| Negative index | Counts from the end | cups[-1, -1] → 480 |
| Out of bounds | A position outside the array | cups[4, 0] → IndexError |
| Slicing | Taking a range of positions, start:stop:step (stop excluded) |
cups[0:2, 5:7] |
: (alone) |
All rows or all columns | cups[:, 0] |
| Element-wise | Applied to each element, one by one | week > 400 |
| Comparison array | Result of comparing an array: one True/False per element | [False False True ...] |
| Boolean | A True-or-False value | True |
| Boolean mask | A True/False array used inside [] to keep the True positions |
week[week > 400] |
& / | / ~ |
Element-wise and / or / not | (a > 1) & (a < 5) |
| Parentheses rule | Put brackets around every condition when using &, | |
(week > 300) & (week < 400) |
| Ambiguous | Could mean two things | "truth value of an array is ambiguous" |
.any() / .all() |
Is at least one / every element True? | (week > 400).any() → True |
| Fancy indexing | Indexing with a list of positions | week[[0, 6]] |
| reshape | Same data, new shape (same size) | log.reshape(4, 7) |
-1 in reshape |
"Work this dimension out" | reshape(-1, 7) |
| flatten | Any array → a 1-D copy | small.flatten() |
Transpose (.T) |
Swap rows and columns | (4, 7) → (7, 4) |
| Axis | One direction of an array: 0 = down the rows, 1 = across the columns | sum(axis=1) → per week |
| Aggregation | Turning many values into one summary value | sum, mean |
| sum | Total | cups.sum() → 10155 |
| mean | Average: total ÷ count | cups.mean(axis=0) |
| min / max | Smallest / largest value | 260 / 480 |
Standard deviation (std) |
How far values typically sit from the mean (spread) | steady stall 0, wild stall 127.5 |
| Variance | The average squared distance from the mean (std squared) | wild.var() |
| argmax / argmin | Position of the largest / smallest value | week.argmax() → 6 |
| Label | A human-readable name for a position | days[6] → "Sun" |
| np.where | Pick values by a condition, or find True positions | np.where(week >= 400, "busy", "ok") |
| np.sort | A sorted copy of an array | np.sort(week) |
| np.unique | Distinct values, sorted (optionally with counts) | np.unique(snacks, return_counts=True) |
| Frequency table | Each distinct value with how often it appears | bun 1, chai 4, samosa 2 |
| argsort | Positions that would sort the array | week.argsort() |
| View | An array that shares memory with another (from slicing) | cups[1] |
Summary
- Indexing
a[row, col]picks one cell (positions from 0, negatives from the end); slicinga[rows, cols]picks blocks,:means all. - A comparison gives a boolean array; used inside
[]it is a mask that keeps the True cells.mask.sum()counts matches. - Combine masks with
&,|,~, with brackets around each condition;and/orraise the "ambiguous"ValueError. - Fancy indexing picks a list of positions.
- reshape changes shape (same size;
-1= work it out), flatten makes it 1-D,.Tswaps rows and columns. - axis: the axis you name disappears.
axis=1→ per row,axis=0→ per column. - Aggregation:
sum,mean,min,max,std(spread). argmax/argmin give where, not what. - np.where labels, finds positions and replaces values; np.sort sorts a copy; np.unique gives distinct values and counts.
- Next episode: NumPy 3: broadcasting, random numbers and NaN.