Machine Learning Full Course · Episode 71. Every code sample below was run with Python 3.13, scikit-learn and NLTK 3.10.4 (NLTK data packages movie_reviews, stopwords, wordnet and punkt_tab, installed with nltk.download(...)), and the output shown is the real output.
The big idea: to a computer, text is just characters, so a hand-made list of "happy words" fails: it misses capital letters, repeated letters, word forms and negation, and reached only 65.3% on 2,000 real movie reviews. Natural language processing turns text into numbers in steps: normalise and clean it (regex), cut it into tokens, optionally drop stop words and reduce words to stems or lemmas, add n-grams to keep some order, weigh the words with TF-IDF, and let logistic regression learn a weight for every word. That pipeline reached 88.4% on reviews it had never seen.
Sources: the corpus is the polarity dataset v2.0 of Bo Pang and Lillian Lee (1,000 positive and 1,000 negative movie reviews; first used in "A Sentimental Education: Sentiment Analysis Using Subjectivity Summarization Based on Minimum Cuts", Proceedings of the ACL, 2004, according to the README shipped with NLTK's movie_reviews corpus). Martin F. Porter, "An algorithm for suffix stripping", Program 14(3):130-137, 1980 (tartarus.org: the stemmer removes "the commoner morphological and inflexional endings from words in English").
Words you need first
- One-hot vector and bag of words (Episodes 16, 28): a column per word; a review becomes its word counts, order forgotten.
- TF-IDF (Episode 39): term frequency × inverse document frequency: a word's count in this text, down-weighted if it appears in many texts.
- Logistic regression (Episodes 22-23): a linear model with a sigmoid; one weight per feature;
Cis the inverse regularisation strength. - Pipeline and leakage (Episodes 15, 17): chain preprocessing and the model so preprocessing is fitted on training data only.
- Stratified split (Episode 15): keeps the same share of each class in train and test.
The story: a rule that reads reviews
Imagine a food delivery app with thousands of reviews a day. Natural language processing (NLP) is getting computers to work with human language. Sentiment analysis is deciding whether a text is positive or negative. A corpus is the collection of texts we learn from.
from nltk.corpus import movie_reviews as mr # 2,000 real movie reviews, labelled
def rule(text): # obvious attempt: look for happy words
happy = sum(text.count(w) for w in ["good", "great", "loved", "best"])
sad = sum(text.count(w) for w in ["bad", "worst", "boring", "awful"])
return "pos" if happy > sad else "neg"
for r in ["The biryani was good", "Not good at all, cold food", "GREAT taste!!!",
"Bad packing but the best paneer I've had", "goooood"]:
print(f"{r!r:45} -> {rule(r)}")
ids = mr.fileids()
right = sum(rule(mr.raw(f)) == mr.categories(f)[0] for f in ids)
print(f"rule on {len(ids)} real reviews: {right / len(ids):.1%} right")
Output:
'The biryani was good' -> pos
'Not good at all, cold food' -> pos
'GREAT taste!!!' -> neg
"Bad packing but the best paneer I've had" -> pos
'goooood' -> neg
rule on 2000 real reviews: 65.3% right
- "Not good at all" is positive (the rule saw good and missed not).
- "GREAT taste!!!" is negative (
"great"in lower case is not in"GREAT"), and "goooood" too. - "Bad packing but the best paneer" is right only by luck: the rule never saw Bad with a capital B.
- 65.3% on 2,000 real reviews.
Why: text is just characters (GREAT ≠ great ≠ goooood), one word has many forms (love, loved, loving), the rule ignores order, and eight hand-picked words miss most of the signal.
Real-life example: preparing vegetables
| Kitchen | NLP step |
|---|---|
| Wash off mud and stickers | Normalisation + regex cleaning: lower case, remove links, squash repeated letters |
| Chop into pieces | Tokenisation: text → a list of tokens |
| Throw away the peels (not the useful bits!) | Stop-word removal: drop very common words |
| Diced, sliced or grated: still potato | Stemming / lemmatisation: loved, loving → love |
Normalisation, regex cleaning and tokenisation
- Text normalisation: making different ways of writing the same thing identical: lower case, every number →
num, "goooood" → "good". - Regular expression (regex): a small pattern language for finding text.
\d+= one or more digits;https?://\S+=httporhttps,://, then non-space characters;(.)\1{2,}= a character followed by two or more copies of itself.re.sub(pattern, replacement, text)replaces every match. - Token: one unit of text, usually a word or a punctuation mark. Word tokenisation: turning text into a list of tokens.
import re
from nltk.tokenize import word_tokenize
review = "LOVED it!!! Delivery in 20 mins :) but the dal wasn't hot... 5/5 https://t.co/xyz"
print("split() :", review.split())
text = review.lower() # normalise case
text = re.sub(r"https?://\S+", " ", text) # regex: remove links
text = re.sub(r"\d+", " num ", text) # numbers -> one token
text = re.sub(r"(.)\1{2,}", r"\1\1", text) # "!!!" -> "!!", "goooood" -> "good"
print("cleaned :", text)
print("regex :", re.findall(r"[a-z']+", text)) # words only
print("tokenize :", word_tokenize(text)) # NLTK's word tokeniser
print("goooood ->", re.sub(r"(.)\1{2,}", r"\1\1", "goooood"))
Output:
split() : ['LOVED', 'it!!!', 'Delivery', 'in', '20', 'mins', ':)', 'but', 'the', 'dal', "wasn't", 'hot...', '5/5', 'https://t.co/xyz']
cleaned : loved it!! delivery in num mins :) but the dal wasn't hot.. num / num
regex : ['loved', 'it', 'delivery', 'in', 'num', 'mins', 'but', 'the', 'dal', "wasn't", 'hot', 'num', 'num']
tokenize : ['loved', 'it', '!', '!', 'delivery', 'in', 'num', 'mins', ':', ')', 'but', 'the', 'dal', 'was', "n't", 'hot', '..', 'num', '/', 'num']
goooood -> good
split() keeps punctuation glued on ("it!!!", "hot...") and the whole link as one word. NLTK's word_tokenize separates punctuation and splits "wasn't" into was + n't, so the negation becomes its own token.
Stop words, stemming and lemmatisation
- Stop words: the most common words (the, is, and), often removed to save space. NLTK's English list has 198 words; scikit-learn's built-in
stop_words="english"list has 318. Both contain "not" and "no". - Stemming: chopping endings with fixed rules (Porter, 1980). Fast; the stem may not be a real word: studies → studi.
- Lemmatisation: looking up the dictionary form, the lemma, in a dictionary (WordNet). It needs the part of speech (noun, verb, adjective); NLTK's default is noun.
from nltk.corpus import stopwords
from nltk.stem import PorterStemmer, WordNetLemmatizer
stop = set(stopwords.words("english")) # NLTK's English stop-word list
print("stop words:", len(stop), "| e.g.", sorted(stop)[:5])
words = "the food was not good and the delivery was not fast".split()
print("kept:", [w for w in words if w not in stop]) # "not" is a stop word!
print("'not' in list?", "not" in stop, "| 'no' in list?", "no" in stop)
stem, lem = PorterStemmer(), WordNetLemmatizer()
for w in ["studies", "studying", "better", "running", "ran", "cookies", "university"]:
v, n = lem.lemmatize(w, "v"), lem.lemmatize(w) # as a verb, as a noun
print(f"{w:10} stem {stem.stem(w):7} verb {v:10} noun {n}")
print("better as an adjective:", lem.lemmatize("better", "a"))
Output:
stop words: 198 | e.g. ['a', 'about', 'above', 'after', 'again']
kept: ['food', 'good', 'delivery', 'fast']
'not' in list? True | 'no' in list? True
studies stem studi verb study noun study
studying stem studi verb study noun studying
better stem better verb better noun better
running stem run verb run noun running
ran stem ran verb run noun ran
cookies stem cooki verb cookies noun cooky
university stem univers verb university noun university
better as an adjective: good
"the food was not good and the delivery was not fast" became ['food', 'good', 'delivery', 'fast']: a complaint turned into praise. For sentiment, never drop negations blindly. Note the lemmatiser quirks: as a noun, cookies → cooky and ran stays ran; as a verb, ran → run; as an adjective, better → good.
N-grams
An n-gram is n tokens in a row: "good" is a 1-gram (unigram), "not good" a 2-gram (bigram), "was not good" a 3-gram (trigram). A sentence of n words has n − 1 bigrams. Adding bigrams puts some order back into a bag of words, at the cost of many more columns.
from nltk.util import ngrams
from sklearn.feature_extraction.text import CountVectorizer
a, b = "good not bad", "bad not good"
for n in (1, 2):
print(f"{n}-grams of {a!r}:", [" ".join(g) for g in ngrams(a.split(), n)])
print(f"{n}-grams of {b!r}:", [" ".join(g) for g in ngrams(b.split(), n)])
for rng in [(1, 1), (1, 2)]:
cv = CountVectorizer(ngram_range=rng).fit([a, b])
X = cv.transform([a, b]).toarray()
print(f"ngram_range={rng}: columns {list(cv.get_feature_names_out())}")
print(" rows:", X.tolist(), "| same row?", (X[0] == X[1]).all())
Output:
1-grams of 'good not bad': ['good', 'not', 'bad']
1-grams of 'bad not good': ['bad', 'not', 'good']
2-grams of 'good not bad': ['good not', 'not bad']
2-grams of 'bad not good': ['bad not', 'not good']
ngram_range=(1, 1): columns ['bad', 'good', 'not']
rows: [[1, 1, 1], [1, 1, 1]] | same row? True
ngram_range=(1, 2): columns ['bad', 'bad not', 'good', 'good not', 'not', 'not bad', 'not good']
rows: [[1, 0, 1, 1, 1, 1, 0], [1, 1, 1, 0, 1, 0, 1]] | same row? False
A TF-IDF + logistic regression sentiment classifier
TfidfVectorizer(sublinear_tf=True): TF-IDF with1 + log(count)instead of the raw count, so a word repeated 20 times is not 20 times louder.LogisticRegression(C=10): one weight per column; positive weights push towards "pos".make_pipeline: the vectorizer learns its vocabulary from the training reviews only.
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]
y = [mr.categories(f)[0] for f in mr.fileids()] # "pos" or "neg"
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
for name, kw in [("words", {}), ("words, stop words removed", {"stop_words": "english"}),
("words + 2-grams", {"ngram_range": (1, 2), "min_df": 2})]:
model = make_pipeline(TfidfVectorizer(sublinear_tf=True, **kw), LogisticRegression(C=10))
model.fit(Xtr, ytr) # fit on TRAIN text only
print(f"{name:26}: test accuracy {model.score(Xte, yte):.1%}")
for r in ["the acting was not good and the plot was a mess",
"a beautiful, moving film with a brilliant cast"]:
print(f"{r!r} -> {model.predict([r])[0]}")
Output:
words : test accuracy 88.4%
words, stop words removed : test accuracy 87.6%
words + 2-grams : test accuracy 87.8%
'the acting was not good and the plot was a mess' -> neg
'a beautiful, moving film with a brilliant cast' -> pos
The same three settings with 5-fold cross-validation (Episode 26) on all 2,000 reviews: plain words 87.8% (± 0.6%), stop words removed 87.0% (± 1.1%), words + 2-grams 87.7% (± 0.9%). No real winner: measure, don't assume.
What the model learned
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
docs = [mr.raw(f) for f in mr.fileids()]
y = [mr.categories(f)[0] for f in mr.fileids()]
vec = TfidfVectorizer(sublinear_tf=True)
X = vec.fit_transform(docs) # rows = reviews, columns = words
print("matrix:", X.shape, "| non-zero:", X.nnz, f"({X.nnz / (X.shape[0] * X.shape[1]):.2%})")
clf = LogisticRegression(C=10).fit(X, y)
w, words = clf.coef_[0], vec.get_feature_names_out() # weight > 0 pushes towards "pos"
order = w.argsort()
print("most negative:", [words[i] for i in order[:8]])
print("most positive:", [words[i] for i in order[-8:][::-1]])
Output:
matrix: (2000, 39659) | non-zero: 666842 (0.84%)
most negative: ['bad', 'worst', 'boring', 'supposed', 'nothing', 'unfortunately', 'plot', 'script']
most positive: ['great', 'fun', 'well', 'hilarious', 'very', 'also', 'memorable', 'excellent']
The matrix is sparse: only 0.84% of its cells are non-zero, so scikit-learn stores only those. Nobody gave the model word lists; it found bad, worst, boring and great, fun, hilarious in the data.
Common mistakes
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
vec.fit("The dal was hot and the roti was soft") # one review, not a list
Traceback (most recent call last):
...
ValueError: Iterable over raw text documents expected, string object received.
from sklearn.feature_extraction.text import TfidfVectorizer
vec = TfidfVectorizer()
X = vec.transform(["the dal was hot"]) # transform before fit
Traceback (most recent call last):
...
sklearn.exceptions.NotFittedError: The TF-IDF vectorizer is not fitted
| Mistake | Exact message / symptom | Why | Fix |
|---|---|---|---|
vec.fit("one review") |
ValueError: Iterable over raw text documents expected, string object received. |
it needs a list of documents | vec.fit(["one review"]) |
transform before fit |
NotFittedError: The TF-IDF vectorizer is not fitted |
no vocabulary learned yet | fit on training text, then transform |
| Removing stop words for sentiment | "not good" → "good", silently | not and no are on the lists | keep negations, and measure |
| Fitting the vectorizer on all the text | a test score that is too optimistic | vocabulary and IDF leak from the test set | a Pipeline, fitted on train only |
| Lemmatising without the part of speech | cookies → cooky, ran → ran | the default is noun | pass pos, or use a part-of-speech tagger |
| Keyword rules | 65.3% | case, spelling, negation, few words | learn weights from labelled data |
Mini project: explain each prediction
Clean the text inside the vectorizer (preprocessor=), train on all 2,000 reviews, and for a new review print the probability and the words that pushed most for and against (TF-IDF value × weight).
import re
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
def clean(t): # the cleaning steps from the video
t = re.sub(r"https?://\S+", " ", t.lower())
t = re.sub(r"\d+", " num ", t)
return re.sub(r"(.)\1{2,}", r"\1\1", t)
model = make_pipeline(TfidfVectorizer(sublinear_tf=True, preprocessor=clean),
LogisticRegression(C=10)).fit(docs, y)
vec, clf = model[0], model[1]
def explain(review): # which words pushed the decision?
row = vec.transform([review]); words = vec.get_feature_names_out()
parts = sorted(((row[0, j] * clf.coef_[0][j], words[j]) for j in row.indices), reverse=True)
p = model.predict_proba([review])[0][list(clf.classes_).index("pos")]
print(f"{review!r}\n P(pos) = {p:.2f} | for: {[w for _, w in parts[:3]]} | against: {[w for _, w in parts[-3:]]}")
for r in ["What a brilliant, funny film. Loved it!!!",
"The plot was boring and the ending was the worst I have seen",
"Not bad at all, actually quite good"]:
explain(r)
Output:
'What a brilliant, funny film. Loved it!!!'
P(pos) = 0.90 | for: ['brilliant', 'loved', 'film'] | against: ['it', 'what', 'funny']
'The plot was boring and the ending was the worst I have seen'
P(pos) = 0.01 | for: ['seen', 'ending', 'and'] | against: ['plot', 'boring', 'worst']
'Not bad at all, actually quite good'
P(pos) = 0.19 | for: ['quite', 'good', 'not'] | against: ['at', 'actually', 'bad']
The third review shows the limit of single words: "Not bad at all, actually quite good" gets only 0.19, because bad has a strong negative weight and not is just another word. Try ngram_range=(1, 2) and see whether "not bad" helps; the next episodes give the model a real sense of word meaning and order.
Practice
1. Remove NLTK's stop words except the negations. Does it beat removing all of them?
Solution
from nltk.corpus import movie_reviews as mr, stopwords
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
keep = {"not", "no", "nor", "don't", "didn't", "wasn't", "isn't"}
stop = sorted(set(stopwords.words("english")) - keep) # NLTK list minus negations
for name, sw in [("no removal", None), ("NLTK list", sorted(stopwords.words("english"))),
("NLTK list minus negations", stop)]:
m = make_pipeline(TfidfVectorizer(sublinear_tf=True, stop_words=sw), LogisticRegression(C=10))
print(f"{name:26}: {m.fit(Xtr, ytr).score(Xte, yte):.1%}")
no removal : 88.4%
NLTK list : 88.0%
NLTK list minus negations : 88.4%
Removing the full list costs 0.4 points; keeping the negations brings it back to the no-removal score.
2. Add a Porter stemmer to the pipeline.
Solution
import re
from nltk.corpus import movie_reviews as mr
from nltk.stem import PorterStemmer
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import train_test_split
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
Xtr, Xte, ytr, yte = train_test_split(docs, y, test_size=0.25, random_state=71, stratify=y)
stem = PorterStemmer()
def stem_tokens(text): return [stem.stem(w) for w in re.findall(r"[a-z']+", text.lower())]
for name, kw in [("words", {}), ("Porter stems", {"tokenizer": stem_tokens, "token_pattern": None})]:
m = make_pipeline(TfidfVectorizer(sublinear_tf=True, **kw), LogisticRegression(C=10)).fit(Xtr, ytr)
print(f"{name:13}: {len(m[0].vocabulary_):6} columns, test accuracy {m.score(Xte, yte):.1%}")
words : 35327 columns, test accuracy 88.4%
Porter stems : 25784 columns, test accuracy 86.6%
Stemming cut the vocabulary by about a quarter but lowered accuracy here; with plenty of data, TF-IDF + logistic regression often does fine without it.
3. Compare ngram_range (1, 1), (1, 2) and (1, 3) with 5-fold cross-validation.
Solution
from nltk.corpus import movie_reviews as mr
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import cross_val_score
from sklearn.pipeline import make_pipeline
docs = [mr.raw(f) for f in mr.fileids()]; y = [mr.categories(f)[0] for f in mr.fileids()]
for rng in [(1, 1), (1, 2), (1, 3)]:
m = make_pipeline(TfidfVectorizer(sublinear_tf=True, ngram_range=rng, min_df=2), LogisticRegression(C=10))
s = cross_val_score(m, docs, y, cv=5)
print(f"ngram_range={rng}: 5-fold accuracy {s.mean():.1%} ± {s.std():.1%}")
ngram_range=(1, 1): 5-fold accuracy 88.0% ± 0.7%
ngram_range=(1, 2): 5-fold accuracy 87.7% ± 0.9%
ngram_range=(1, 3): 5-fold accuracy 87.3% ± 1.4%
On 2,000 reviews, more n-grams added columns faster than useful signal.
4. How many 3-grams does a 10-word sentence have?
Solution
10 − 3 + 1 = 8. In general n words have n − k + 1 k-grams.
Quick quiz
from nltk.util import ngrams
grams = list(ngrams("the food was cold".split(), 2))
# len(grams) = ?
Answer
3: ("the", "food"), ("food", "was"), ("was", "cold").
Interview questions
- Stemming vs lemmatisation? Stemming chops endings by rules (fast, may produce non-words like "studi"); lemmatisation maps to the dictionary form using a vocabulary and the part of speech (slower, real words).
- Should you always remove stop words? No. For topic tasks it can save space; for sentiment it can delete negations. Measure both.
- Why TF-IDF instead of raw counts? Words common to every document (the, movie) carry little information; IDF down-weights them.
- Why is the TF-IDF matrix sparse and why does it matter? Each document uses a tiny fraction of the vocabulary; storing only non-zeros saves memory and makes linear models fast.
- How do n-grams help, and what do they cost? They capture short phrases and negation ("not good"); the number of columns grows quickly, so use
min_dfand validation. - Where can leakage happen in a text pipeline? Fitting the vectorizer (vocabulary and IDF) on test documents; fit it inside a pipeline on training folds only.
Glossary
| Term | Meaning, with an example |
|---|---|
| Natural language processing (NLP) | Getting computers to work with human language; classifying reviews |
| Corpus | A collection of texts used for learning; 2,000 movie reviews |
| Sentiment analysis | Deciding whether a text is positive or negative |
| Text normalisation | Making different spellings identical: "GREAT" → "great", "20" → "num" |
| Regular expression (regex) | A text-search pattern; \d+ matches "20" |
| Regex cleaning | Using re.sub to remove or replace patterns such as links |
| Token | One unit of text, usually a word: "wasn't" → "was", "n't" |
| Word tokenisation | Splitting text into tokens (word_tokenize) |
| Stop words | Very common words often removed: the, is, and (and also not, no!) |
| Stemming | Chopping word endings by rules: studies → studi (Porter, 1980) |
| Stem | What is left after stemming; may not be a real word |
| Lemmatisation | Mapping a word to its dictionary form: ran → run (WordNet) |
| Lemma | A word's dictionary form |
| Part of speech | A word's grammatical role: noun, verb, adjective |
| WordNet | An English dictionary database used by NLTK's lemmatiser |
| N-gram | n tokens in a row; "not good" is a 2-gram (bigram) |
ngram_range |
Which n-gram lengths a vectorizer counts; (1, 2) = words and pairs |
sublinear_tf |
Use 1 + log(count) as the term frequency |
| TF-IDF | Term frequency × inverse document frequency (Episode 39) |
| Sparse matrix | A matrix stored without its zeros; 0.84% non-zero here |
| Text classifier | A model that assigns a label to a text; TF-IDF + logistic regression |
Summary
- A keyword rule reached 65.3% because text is just characters: case, spelling, word forms and negation break it.
- Normalise and clean with regex, tokenise (NLTK separates "n't"), and be careful: stop-word lists contain not and no.
- Stemming chops (studi), lemmatisation looks up (study) and needs the part of speech.
- N-grams keep a little order; "good not bad" and "bad not good" get different rows.
- TF-IDF + logistic regression reached 88.4% on held-out reviews; its learned weights found bad/worst/boring and great/fun/hilarious by itself.
Next: Episode 72, word embeddings: dense word vectors where good and great are close, word2vec from scratch, GloVe and subword tokenisation.