Bastion — Scam-Call & SMS Classifier (Gemma 4 E2B + LoRA)

Fine-tuned Gemma 4 E2B for on-device scam-call and SMS classification. Shipped inside Bastion, the on-device phone-scam shield for seniors built for the Gemma 4 Good Hackathon (Impact Track · Safety & Trust, and LiteRT Special Technology Track).

TL;DR

A LoRA adapter that takes Gemma 4 E2B from F1 0.305 to F1 0.915 on a 100-sample stratified BothBosu test set (3-class scam / scam_partial / ham), and ships in three formats for two on-device runtimes — llama.cpp (Q4_K_M GGUF) and Google AI Edge LiteRT-LM (.litertlm, QAT-v2).

Artefact	Format	Size	Runtime	Purpose
bastion-text-lora-v1.Q4_K_M.gguf	GGUF (Gemma 4 E2B + LoRA, merged, Q4_K_M)	~3.2 GB	llama.cpp	Reference deployment artefact, used by the shipped Samsung A54 demo
bastion-qat-v2-gemma-4-E2B-it.litertlm	LiteRT-LM v2 (QAT)	~2.4 GB	litertlm-android 0.10.2 / LiteRT-LM v0.11	Google AI Edge runtime artefact (LiteRT Special Technology Track)
bastion-mmproj.BF16.gguf	GGUF mm-projector	~215 MB	llama.cpp (multimodal)	Optional: pairs with stock Gemma 4 E2B for the multimodal-direct experiment baseline (D11/D13)

Why this exists

Live phone scams target older adults and cost USD ~3B/year in the US alone. Cloud assistants cannot intervene fast enough — by the time a transcript uploads, the senior has read out the code. Bastion runs the model on the phone that is ringing. This adapter is the part that decides whether the call gets ended.

The base model is good at spotting scam patterns (binary F1 ≈ 0.99 on BothBosu/100) but ships its verdicts as the cautious scam_partial label, which never triggers Bastion's interrupt — so it is, in practice, useless for an intervention system. The LoRA fine-tune fixes exactly that gap.

Model details

• Base model: google/gemma-4-E2B-it
• Adapter: LoRA, rank 16, language layers only, dropout 0.05
• Training framework: Unsloth
• Training data: synthetic (transcript, label) pairs produced by a Gemini 3.1 Flash Lite Preview pipeline we built specifically for this project (scripts/synth_scale_gemini.py + scripts/chunk_and_filter.py in the Bastion repo). Source scripts come from BothBosu/scam-dialogue (Apache-2.0) plus Gemini-generated ham counterparts. Both sides are spoken aloud via Gemini multi-speaker TTS, then degraded from studio quality to phone-call audio (8 kHz, codec artefacts) so the audio distribution matches what the host application sees on the wire. Each WAV is chunked into 15-second windows, transcribed by Gemma 4 E2B, and only the chunks whose transcript a Gemini label-judge still agrees with the source dialog label are kept. The result is a clean, on-distribution text training set without manual transcription cost. We additionally hold out UCI SMS Spam (CC-BY 4.0) and the SMS phishing subset of DIFrauD (MIT) as evaluation references; they are not part of the v1 training mix.
• Output schema (JSON, tool-call style):

  {
    "verdict": "ham | scam_partial | scam_clear",
    "reason": "<≤ 25 words>",
    "intervene": true | false
  }
  

• Intervention contract: verdict == "scam_clear" && intervene == true triggers TelecomManager.endCall() on Android. Anything else is a banner warning at most.

Intended use

• On-device scam-call screening in conjunction with an OEM call recorder
  • an ASR front-end (Bastion uses Sherpa-ONNX Whisper Tiny).
• On-device SMS scam classification (text mode, same model + adapter).
• Reference target for hackathon submissions and research on small-model scam-detection benchmarks.

This adapter is not a general assistant — it is single-purpose. Outside the scam / not-scam decision, behaviour falls back to base Gemma 4 E2B.

Results

Evaluation set: BothBosu test, 100 samples, stratified 50 scam / 50 ham, 3-class schema. Full eval log in the Bastion repo.

Setting	F1 (binary)	F1 (3-class)	Precision	Recall	scam_clear recall (k/50)	Parse rate
Gemma 4 E2B base (prompt only)	0.667	0.305	1.000	0.180	9 / 50	100 / 100
Gemma 4 E2B + bastion_text_lora_v1 (Q4_K_M, merged)	0.985	0.915	0.977	0.860	43 / 50	100 / 100

+61 absolute-point lift on the 3-class metric that controls intervention. The one false positive at this threshold was a real bank-verification call labelled scam_partial, which would not cross Bastion's interrupt gate (scam_clear only, confidence ≥ 0.8).

Latency on the CPU reference build (Q4_K_M GGUF, llama.cpp, x86 CPU): p50 5.8 s, p95 8.8 s per 15-second window.

Limitations

• Trained on English, but the scam-classification layer generalises. The LoRA fine-tune is on English transcripts, yet Gemma 4 E2B's underlying multilingual coverage carries over: on hand-tested Polish and Spanish paraphrases of the BothBosu archetypes the adapter still emits the correct verdict + JSON. The end-to-end bottleneck is the ASR front-end, not this model — Sherpa-ONNX Whisper Tiny transcribes English well and other languages noticeably less well, so the practical multilingual claim is gated by which Whisper build the host application ships. A formal Polish eval split is in progress and not part of this release.
• Quantisation sensitivity. Q2_K and IQ2_M variants destroy the LoRA signal (F1 drops to 0.00–0.73). Ship Q4_K_M or higher, or the LiteRT-LM QAT-v2 build.
• Single-turn classification. The adapter classifies one rolling window at a time. Multi-turn debounce is the host application's responsibility (Bastion does this in its InterventionController).
• Adversarial robustness untested. Eval is on naturalistic BothBosu data, not adversarial paraphrases of scam scripts.

How to use

Via llama.cpp (Q4_K_M, merged)

huggingface-cli download 3sk4p3/bastion bastion-text-lora-v1.Q4_K_M.gguf --local-dir ./bastion
./llama-cli -m ./bastion/bastion-text-lora-v1.Q4_K_M.gguf \
  --temp 0.0 --json-schema-file ml/prompts/tool_schema.json \
  -p "$(cat ml/prompts/system_prompt.md)\n<transcript>$TRANSCRIPT</transcript>"

Via LiteRT-LM (.litertlm, on Android)

val runner = RealLiteRtGemmaRunner(
    context = appContext,
    modelPath = "/sdcard/Android/data/<pkg>/files/bastion-qat-v2-gemma-4-E2B-it.litertlm",
)
runner.warmUp()
val verdict = runner.classifyText(transcript)  // JSON parsed into the schema above

The Kotlin runner is in android/inference/ in the Bastion repo and ships with the APK.

Training reproducibility

• Unsloth notebook: ml/notebooks/train_text_lora_v1.ipynb
• Prompts and JSON tool schema: ml/prompts/
• Eval scripts: scripts/eval_bothbosu_100.py, scripts/eval_litertlm.py
• Eval log (append-only): ml/evals/RESULTS.md

License & attribution

• This repository (LoRA artefacts + model card): Apache-2.0, plus the Gemma Terms of Use for any artefact derived from Gemma 4 weights.
• Base model: Gemma 4 E2B by Google DeepMind, used under the Gemma Terms of Use.
• Training data: see each dataset's own licence (Apache-2.0, MIT, CC-BY 4.0 as listed above).
• Naming: bastion_text_lora_v1 follows the Gemma variant naming guidelines — variant name precedes Gemma identifier, no stand-alone "Gemma" branding.

Citation

@misc{bastion2026,
  title  = {Bastion: On-Device Scam-Call Shield for Seniors},
  author = {Szczepanik, Kamil and Arkik, Mohamed},
  year   = {2026},
  howpublished = {\url{https://gitlab.com/3sk4p3/bastion}},
  note   = {Gemma 4 Good Hackathon submission, Impact Track / Safety \& Trust}
}