Discussion on building a local AI expert for complex software platforms

A user seeks architectural guidance for creating an AI assistant that functions as a true expert on a complex software platform, rather than a simple documentation chatbot. The goal is to leverage hundreds of pages of docs, tutorial videos, and existing automation scripts to provide accurate, reliable answers and workflow assistance.

• The solution must run locally on modest hardware without requiring a GPU for inference.
• Models under 3B parameters are considered realistic for deployment but their expert capabilities are uncertain.
• Potential approaches include Continued Pretraining (CPT), Supervised Fine-Tuning (SFT), RAG, and agent-based systems with LangGraph.

The author asks the community to share experiences on which architecture works best, whether small local models can realistically achieve expertise, and if RAG is sufficient or if fine-tuning is necessary.