PhishNChips: 12 decision models on phishing detection

Precision and recall

Precision is the share of flagged emails that were phishing, so it measures how far an alert can be trusted. For a mail gateway or a cloud email security tool it comes first: every false alarm quarantines real mail or costs an analyst time, and real inboxes hold little phishing, so a model that is not precise buries the genuine attacks. Recall is the share of phishing that was flagged. A model can be precise yet miss most phishing, so read the two together. Precision here is measured with 67% phishing in the test set. See "In a real inbox" for what it falls to at realistic volumes.

F1

One score that combines precision and recall. It sits closer to the lower of the two, so a model cannot score well by being strong on one and weak on the other. For example, 100% precision with 40% recall gives an F1 of only 57%. It is only shown on sets that contain both phishing and legitimate email, and it depends on the phishing share of the test set.

Accuracy

Share of decisions that were correct. On the main set 2,011 emails are phishing and 1,000 are legitimate, so flagging everything would already score 67%. It hides which kind of mistake was made, so it ranks below precision and recall.

False-positive rate

Share of legitimate emails wrongly flagged. Reported only where the set contains legitimate emails.

95% interval

The Wilson score interval around a rate. With 3,011 emails, scores a point or two apart often overlap and should not be read as a ranking.

Real-inbox precision

Precision recalculated from each model's recall and false-positive rate at a chosen share of phishing. Those two rates do not depend on the mix of the test set, but precision does.

Brier score

Mean squared error of the model's probability against the true label. Lower is better and rewards well-calibrated confidence.

Cost and latency

Cost is the reported spend divided by the decisions with cost data, scaled to 1,000 decisions. Latency is the mean time per decision.

Scores come from the 3,011-email unique set plus its four source sets. The repeated all-emails rows in the raw export are identical to the main set and are counted once. Rates are recomputed from the confusion counts.

Data

PhishNChips v5.2 at revision 89afcc3: 3,387 rows across four files, reduced to 3,011 unique emails (2,011 phishing, 1,000 legitimate) after dropping 2 rows with a blank required field and 374 exact duplicates. The dataset's own labels are the ground truth and were not independently checked.

Question

Each model answered one binary question, PHISHING or LEGITIMATE, with the same written definitions every time. This is not the PhishNChips baseline prompt, which asks whether to click a link. Only the sender, subject, body, link text and URL strings were sent, never the label, and no link was visited.

Run

One run of one prompt on 2026-10-18, through the OpenRouter decisions endpoint. The three Clef models were pinned to the Cloudflare provider with fallback off. Failed calls were retried up to three times. Latency is client-observed and includes retries. Cost is what OpenRouter reported per decision.

Limits

Hosted models change, so a repeat run can differ. The main set holds twice as much phishing as legitimate email, so use the core set for a balanced comparison. Cost and latency depend on routing and load on the day.

The scripts, the recorded predictions (without email text) and step-by-step instructions are in the project repository. You can reproduce the results at three levels:

python3 eval/summarize_results.py --check   # rebuild every score from the recorded predictions
python3 eval/prepare_data.py                # rebuild the 3,011-email corpus from the pinned files
python3 eval/run_models.py --run-models --confirm-send   # repeat the run (needs OPENROUTER_API_KEY, costs money)