Machine Learning · Security · MS Capstone · 2026

Phishing Detection System

An explainable phishing detector I built solo for my MS capstone, then rebuilt so every number is reproducible. It scores any URL on demand and corroborates evidence across five layers before it flags a site: on 136 real brand sites it never trained on, the full system flagged 4, where the URL model alone would have flagged 40%.

PythonLightGBMXGBoostRandom ForestLogistic RegressionPlaywrightStreamlitDockerpytest
View on GitHubTry itAll projects
0.80
F1 of the URL model on 147,702 test URLs from websites it never saw
4 / 136
real brand sites flagged as phishing by the full system
385
automated tests, run in CI on every push

At a glance

A URL goes through five layers and comes out as one of three verdicts. The layers exist because a URL model alone raises too many false alarms on real sites.

Figure 1System diagram
Diagram: a URL passes through the URL model, live capture, page analysis, brand and trust checks and the adjudication layer, ending in one of three verdicts.
How a URL becomes a verdict

What it showsOne URL moving left to right through the five layers and ending in one of three verdicts. The arrow over the top is URL-only mode, where no page is loaded and layers 2 and 3 are skipped.

Why it mattersThis is the map for every chart below. It also explains why the browser demo mostly answers uncertain: it only has layer 1 and the rules that need no page.

Figure 2Headline result
Chart: of 136 real brand URLs, the URL model alone flags 55; the full system calls 4 phishing, 24 uncertain and 108 legitimate.
False alarms on real brand sites: 55 of 136 becomes 4

What it showsThe same 136 official brand URLs (PayPal, Microsoft, Chase, Amazon and others) at three stages. The URL model alone flags 55. Add the adjudication layer without a page and it declines to accuse any of them. With the page captured, the full system calls 4 phishing, 24 uncertain and 108 legitimate.

Why it mattersThis is the reason the project is a layered system and not just a classifier. A model score alone would bury an analyst in false alarms on the sites people use most.

Live demoRuns in your browser

Try the detector yourself

Paste any URL and the URL model, plus the rules that can run without visiting the page, will score it in your browser. The same window has replays of the full system on real brand sites and real phishing pages, including the ones it gets wrong.

Try it
Case study

How the system is built, what was measured, and the decisions behind it.

Overview01

URL-based machine learning is fast, but on modern, JavaScript-heavy sites it tends to over-alert, flagging legitimate pages as threats. This project pairs an ML triage step with a deterministic evidence review, so every verdict is corroborated across multiple independent signals and comes with a reason a security analyst can read.

It runs as a Streamlit dashboard in Docker and returns one of three verdicts: likely-phishing, uncertain (routed to review), or likely-legitimate, rather than an opaque score.

How it works02

A URL flows through five coordinated layers before a verdict is issued:

  1. ML + model agreement. Four models (LightGBM, XGBoost, Random Forest, Logistic Regression) are trained on 54 URL and host features from 794,563 deduplicated URLs. Train and test are split by registered domain, so no website appears in both, and the features ignore http vs https (a quirk of the dataset the model would otherwise learn). XGBoost was picked on a validation split, never the test set; the other three vote as supporting evidence.
  2. Live capture with Playwright. The page is actually loaded to collect final URLs, redirect chains, cross-domain form targets, and TLS / certificate state.
  3. HTML / DOM and behavior. Structural tells are analyzed: wrapper pages, interstitials, credential-harvest layouts, and network patterns that hint at exfiltration.
  4. Brand and trust context. Rule-based brand-impersonation checks test brand-to-domain coherence against a registry used as a weak prior, never a whitelist.
  5. Evidence Adjudication Layer. A deterministic engine weighs all of the above into likely-phishing, uncertain, or likely-legitimate, returning uncertain rather than guessing when signals conflict.

Results03

Every number below is regenerated by one script (seed 42) and saved with the commit, data hash and environment that produced it, in VERIFIED_METRICS.md.

What was measuredResult
URL model (XGBoost), 147,702 test URLs from unseen websitesF1 0.801 (95% CI 0.799 to 0.803), ROC-AUC 0.880, precision 0.772, recall 0.832
136 real brand sites (PayPal, Microsoft, Chase, Amazon and others), URL model alone40.4% flagged
Same 136 sites, full system with live page capture4 phishing, 24 uncertain, 108 legitimate
99 fresh phishing URLs from the PhishStats feed, full system39 phishing, 20 uncertain, 40 legitimate
Time per URL15.7 ms URL model only; 6.8 s median with live page capture
URL model vs a local LLM (llama3.2:3b via Ollama), 2,000 held-out test URLs, same 2-core CPUF1 0.802 vs 0.635 (difference +0.167, 95% CI +0.134 to +0.202); 11.4 ms vs 741.6 ms median, 65x faster per URL
Cascade on the same 2,000 URLs, example band: the LLM only sees URLs the model scores 0.30 to 0.7081.6% of URLs never reach the LLM; false alarms 24.6% to 16.1%, recall 0.860 to 0.770, F1 0.802 to 0.787; 5.4x faster than the LLM on every URL

The live-capture rows replay one saved snapshot of real pages, captured by GitHub Actions. I wrote the tuning rules down before changing anything, tuned only on a validation half, and scored the test half once.

Compared with an existing tool. I ran a free open-source LLM on the same URLs as a baseline. It raised far fewer false alarms (6.6% vs 24.6%) but caught only half of the phishing (recall 0.501 vs 0.860). This is one small open model with one fixed prompt, so it says nothing about large hosted models. 8 of its 2,000 answers were neither label and were counted as legitimate. The cascade is an offline measurement, not how the app runs by default, and no band width was picked as the setting; every band is in VERIFIED_METRICS.md.

The honest numbers

4 / 136
real brand sites the full system calls phishing (7 before the rule tuning). The URL model alone flags 40.4% of them.
test half, frozen snapshot replay
39 / 99
fresh phishing pages caught by the full system; 40 were called legitimate. Recall on fresh phishing is the known weakness.
PhishStats feed, captured pages, test half
69 / 284
PhishStats URLs that ML-only mode (the live demo) calls likely phishing; the other 215 stay uncertain, because without the page it never says legitimate.
ML-only dashboard, no page capture

What this means: the system is good at not crying wolf on real sites, and it is not yet good at catching brand-new phishing. Across all 212 legitimate pages in the test replay, 19 were labeled phishing. Every number here is generated from VERIFIED_METRICS.md and the replay results in metrics/results/tuning, not typed by hand.

The live demo was checked against the Python app on 148,746 URLs (147,714 from the held-out test file plus the curated evaluation sets): identical features for 148,746 of 148,746, the same calibrated score (to the 6 decimals the app reports) for 148,746, and the same 4-model vote for 148,746. On 21,032 of them the real dashboard was also run, and the ML-only verdict matched for 21,032 of 21,032. Largest probability difference: 1.2e-07 (tolerance 1e-6). Parity test and export code.

What it is good at, and what it is not. The layers do their job of not crying wolf on real sites. Catching brand-new phishing is the weak spot: the URL model rates many famous homepages as suspicious as phishing pages, so no rule can separate them without adding false alarms. Improving the URL model is the next step.
Figure 3Error analysis
Chart: final verdicts per evaluation set. With page capture, 39 of 99 fresh phishing pages are called phishing and 40 legitimate.
Final verdicts on every evaluation set

What it showsWhere the full system lands on each test set, with and without page capture. Brand sites are mostly cleared, 15 of 76 popular homepages are still called phishing, and only 39 of 99 fresh phishing pages are caught while 40 are called legitimate.

Why it mattersIt shows the weak spot as measured instead of hiding it. Recall on brand-new phishing is where the next round of work goes.

The URL model in detail

Six charts on the URL model. Swipe or use the arrows.

1 of 6
Figure 4Model performance
Chart: ROC and precision-recall curves for XGBoost, LightGBM, Random Forest and Logistic Regression on 147,702 held-out URLs.
ROC and precision-recall curves, four models

What it showsEvery threshold trade-off for the four models on 147,702 URLs from websites never seen in training. The dot marks XGBoost at the 0.5 threshold: it catches 83.2% of phishing and flags 21.3% of legitimate URLs.

Why it mattersA single F1 number hides the trade-off; the curves show all of it. The three tree models sit almost on top of each other, so the choice of XGBoost is not a fragile one.

Figure 5Model comparison
Chart: F1, ROC-AUC and PR-AUC for the four models and a majority-class baseline.
Four models against a baseline

What it showsF1, ROC-AUC and PR-AUC side by side for XGBoost, LightGBM, Random Forest, Logistic Regression and a majority-class baseline, all on the same held-out URLs.

Why it mattersIt proves the models learned something real (every one beats the baseline) and that tree models beat the linear one. XGBoost was picked on validation data, so these test scores were never used to choose it.

Figure 6Error counts
Confusion matrix for XGBoost: 62,253 correctly passed, 16,881 false alarms, 11,499 missed, 57,069 correctly caught.
Confusion matrix of the primary model

What it showsThe XGBoost score turned into four counts at the 0.5 threshold: 62,253 legitimate URLs correctly passed, 16,881 false alarms, 11,499 phishing URLs missed and 57,069 correctly caught.

Why it mattersIt turns an abstract F1 of 0.80 into real mistakes you can count. That 21.3% false alarm rate is exactly what layers 2 to 5 exist to fix.

Figure 7Explainability
Chart: the 12 URL features with the largest average contribution to the XGBoost score, led by path length.
What the URL model looks at

What it showsThe 12 features (of 54) with the largest average push on the XGBoost score. Path length carries more than twice the weight of anything else, and these 12 make up 88% of the total.

Why it mattersIt opens the black box. The model leans on the shape of the URL, which explains why long but legitimate addresses get flagged and why page evidence is needed to overrule it.

Figure 8Calibration
Calibration chart: predicted probability against the observed phishing rate, raw and after isotonic calibration.
Can the probability be trusted?

What it showsPredicted probability against how often those URLs really were phishing, before and after isotonic calibration. At the top the model says 97% and is right 85% of the time, and calibration made the Brier score slightly worse (0.1408 to 0.1455).

Why it mattersThe adjudication rules use score thresholds, so knowing where the score is overconfident matters. It is reported as an open item, not smoothed over.

Figure 9Controlled experiment
Chart: rule tuning lowered legitimate pages called phishing on validation and test splits while phishing detection stayed the same.
Rule tuning, before and after

What it showsThe same saved page captures replayed with the old and the new rules. Legitimate pages called phishing fall from 15.0% to 10.3% on validation and from 10.4% to 9.0% on test, while fresh phishing caught stays identical.

Why it mattersIt shows the tuning was done properly: rules changed on the validation half only, the test half scored once, and no recall was traded away to get the improvement.

Design decisions04

Deterministic at runtime. I prototyped LLM brand inference, then replaced it with deterministic rules for reproducibility. No external AI is called when scoring a URL, so verdicts are reproducible and auditable.
Conservative by default. When evidence disagrees the system says uncertain and routes the URL to review, instead of forcing a clean-looking answer.
Rebuilt to be verifiable. An audit of my first version found a leaky train/test split, so I rebuilt training and evaluation: domain-grouped splits, guards that stop a leaky run, and golden-output tests that pin the dashboard's behavior. 385 tests (pytest) run in GitHub Actions on every push, alongside a Docker build.
Figure 10Audit result
Chart: ROC-AUC and F1 of the four models before and after the rebuild. Every score went down, for example XGBoost ROC-AUC from 0.950 to 0.880.
Before and after the rebuild

What it showsROC-AUC and F1 for all four models before and after I fixed the evaluation. Every score went down, for example XGBoost ROC-AUC from 0.950 to 0.880 and F1 from 0.876 to 0.801.

Why it mattersThe drop is the evidence that the leak was real. The earlier numbers were inflated by a data artifact and by test websites sitting in the training set; the lower numbers are the ones I can defend.

Tech

Pythonscikit-learnXGBoostLightGBMRandom ForestLogistic RegressionPlaywrightBeautifulSoupStreamlitDocker & Composepytest