Text classification · 2026
Craft Forum Discourse Classification
Can a small labeled dataset support reliable discourse classification—and does DistilBERT actually help?
Classify discourse with very little labeled data
The dataset contained four kinds of conversation from art and craft forums. Because posts in the same thread share language and context, a random row-level split could make performance look better than it really was.
Label the data and keep every conversation intact
- Manually labeled 212 posts from 28 forum threads.
- Placed each entire thread in either training or testing.
- Compared TF-IDF logistic regression with unweighted and class-weighted DistilBERT.
- Used macro F1 so minority categories mattered in the evaluation.
The simpler model performed best
TF-IDF with logistic regression led the four-category comparison. Unweighted DistilBERT predicted the same category for every test post; class weighting raised macro F1 only from 0.151 to 0.172.
212labeled posts
28separate threads
0.172best DistilBERT macro F1
Model complexity did not compensate for limited labeled data
On this dataset, TF-IDF with logistic regression was both stronger and easier to interpret. Keeping whole threads together also kept shared conversational context out of the test set.