A new framework pre-trains the Tsetlin Machine using semantic clusters from language models, avoiding embeddings. The method groups text samples into coherent clusters via K-means or Top2Vec, then uses cluster-sample pairs to train a non-negated TM with Type I feedback. Results show superior performance across five datasets, matching BERT-level accuracy while maintaining full interpretability.