01 / Context & question
A spam filter can fail in two ways, and they do not cost the same.
Every incoming SMS must be classified as ham (a legitimate message) or spam (promotions, fake prizes, premium-rate numbers).
A filter can let a spam through: annoying. It can also block a real message, such as an appointment or a login code: much worse for the user. Everything that follows is therefore built to measure these two errors separately, not just an overall percentage of correct answers.
On this set of messages, does a model that follows word order really beat a much simpler classic method?
Before choosing a model, I first needed to know what these messages look like.
02 / Exploratory data analysis (EDA)
Exploration reveals three things that shape everything after: imbalance, duplicates and a recognisable spam style.
Exploratory data analysis (EDA) means looking at the data before modelling: how many messages, what share of spam, which anomalies, which visible differences between the two classes. It is what drives the right preparation and evaluation choices.
- counted messages per class and looked for missing values;
- looked for duplicated messages;
- compared the length of ham and spam messages;
- listed the most frequent words in each class.
4,825 ham and 747 spam, no missing values.
Removed before any split.
4,516 ham and 641 spam.
The imbalance means tracking spam precision and recall rather than accuracy. Duplicates must be removed before splitting, otherwise the same message could be learned and then “recognised” at test time. Links and sales vocabulary show that cleaning must not erase these clues.
These observations directly guided how the text was prepared.
03 / Text preparation
Deliberately light cleaning, to keep the clues the filter needs.
- lowercase (“FREE” and “free” become the same word);
- each link replaced by the word
urltoken: the exact address changes from one spam to another, but the presence of a link is a signal; - punctuation and repeated spaces removed;
- labels encoded: ham = 0, spam = 1.
I did not remove short words or numbers: in SMS, “txt”, “150p” or a short code are precisely typical of spam, as the exploration showed.
Once the messages were prepared, the evaluation had to be organised before training anything.
04 / Training, validation and test split
Three separate sets of messages, each with a precise role.
The BiLSTM vocabulary and the TF-IDF weights are learned only from the 3,609 training messages. Otherwise, words seen only in the test set would influence the model, and the result would be too optimistic.
With this frame in place, I could train two very different approaches under the same conditions.
05 / Two models, two ways of reading an SMS
A model that follows word order, against a method that weighs important words.
A · BiLSTM
B · TF-IDF + Logistic Regression
- each SMS is brought to 33 words, the length of 95% of training messages (shorter ones padded, longer ones cut);
- class weights give more importance to errors on spam, the minority class;
- training stops when the validation error stops decreasing (early stopping), before the model memorises the training set.
Both models output a probability; what remained was deciding when a message is declared spam.
06 / Choosing the decision threshold
The threshold sets the trade-off between spam let through and real messages blocked.
I tested thresholds on the validation set and kept the one with the best spam F1: 0.83 for the BiLSTM and 0.56 for TF-IDF + Logistic Regression. The BiLSTM’s high threshold suggests very clear-cut probabilities on obvious spam: it can demand strong certainty before blocking.
Both thresholds were frozen before any evaluation on the test set, so the final result is not tuned on the messages used to measure it.
With models and thresholds fixed, the test set could finally be used.
07 / Test results
The BiLSTM wins, but only by a short margin.
The test set contains 774 messages: 678 real messages and 96 spam.
These numbers are close; the real question becomes which one to choose, and why.
08 / Interpretation & real-world value
The best score does not always mean the best production choice.
The BiLSTM blocks two fewer real messages and stops one more spam: it is the best classifier in this experiment. But TF-IDF stays very close, and even gets slightly higher areas under the ROC and precision-recall curves, meaning it ranks messages just as well when the threshold varies.
The BiLSTM if every blocked real message matters; TF-IDF if speed, easy updates and explainability come first, since the words driving each decision can be read directly.
With 13 errors out of 774 messages, the BiLSTM is rarely wrong. But 9 of the 96 spam still get through, almost one in ten: that is where more recent data or a richer model would help most.
It remains to be clear about what this experiment proves, and what it does not.
Decision: the BiLSTM is the best tested classifier, but the gap is small enough that TF-IDF + Logistic Regression remains a credible choice when simplicity, speed and explainability matter more.
09 / Limits & lessons
A solid result on this set of messages, to be confirmed on recent SMS.
- An old, English-only set of messages (corpus): spam vocabulary has changed since.
- Training and test come from the same source: no validation on another dataset.
- No speed or memory measurement, which would matter for deployment.
Comparing a complex model with a simple method, on the same split and with a threshold chosen on validation, teaches more than an isolated score. Here the BiLSTM wins, but the small gap shows that extra complexity is only worthwhile when the gain justifies the operational cost.
Going further
I would extend the study by testing a recent SMS dataset, trying a transformer-type model and setting the threshold from the real cost of each error. The notebook details every step and its outputs.