A criminal defense lawyer is seeking technical advice for running a local large language model on an RTX 4060 to adapt Brazilian legal documents without sending confidential data to cloud APIs. The user reports that the current setup, using Qwen3 35B-A3B via llama.cpp, frequently hallucinates statute citations and case IDs with high confidence, even when the correct information is present in the source template.
- Hardware constraints limit the model to an RTX 4060 (8GB VRAM) with CPU offload, requiring efficient quantization and context settings.
- The workflow involves adapting existing `.docx` templates by swapping facts while preserving accurate legal reasoning and citations.
- Current mitigation attempts include strict "edit-only" system prompts that reduce but do not eliminate hallucinations.
- The user requests guidance on model selection for citation faithfulness, constrained generation via GBNF grammars, RAG grounding, and `.docx`-preserving edit pipelines.
The author aims to identify reliable techniques for local, offline legal document processing that prevent the invention of false legal references.