User helloadhavan has created a dataset called CC-FilteredCorpus designed to filter Common Crawl and extract content as clean Markdown. The project aims to preserve useful document structure, including headings, lists, links, code blocks, and tables, rather than flattening everything into plain text.

  • The dataset is hosted on Hugging Face at helloadhavan/CC-FilteredCorpus.
  • It targets LLM pretraining and NLP research where preserving web document structure is important.

This resource provides a structured alternative to raw Common Crawl data for researchers needing formatted text.