X/Twitter Threat Analysis Pipeline
Capstone for DTSA 5506 Data Mining at CU Boulder. The pipeline queries the X API v2 over a rolling 24-hour window, filters spam and non-English posts, scores sentiment with VADER, classifies negative posts with a zero-shot transformer (facebook/bart-large-mnli), and writes results to a three-table SQLite store.
What's built
- Ingestion: X API v2 recent search, 24-hour rolling windows, English-only, retweets excluded
- Spam filtering: keyword-based removal of promotional and giveaway content
- Sentiment scoring: VADER compound scores with standard ±0.05 thresholds for positive / neutral / negative
- Threat classification:
facebook/bart-large-mnlizero-shot acrossnon_threat,implied_threat,harassment,incitement, run only on negative-sentiment posts - Context flagging: post-processing heuristic that tags gaming and streamer slang for human review without overwriting the original label
- Persistence: normalized SQLite schema across three tables joined by
tweet_id
Two findings worth surfacing
API sentiment diverges sharply from the algorithmic feed. Brave Browser launch coverage came in at 92.5% positive in the raw API stream, against an impression of widespread negative reception from algorithmically curated personal feeds the same day. The algorithm amplifies engagement, not representativeness.
Spam materially distorts sentiment. Unfiltered Elon Musk tweets returned 63.75% positive, inflated by giveaway-and-prize scam content. After keyword-based filtering removed 25 of 80 tweets, the positive share dropped to 50.9%.
Stack
Python 3.13, requests, pandas, vaderSentiment, Hugging Face transformers (facebook/bart-large-mnli), SQLite. Managed with uv.
Full evaluation across five datasets (politics, tech, gaming, consumer brands) is in the paper. Sections 4.2–4.3 cover the sentiment and threat distributions.