local llm recommendations for questions
Run a low-quantized 20B model via llama.cpp instead of Ollama to handle documentation reading and question answering efficiently.
- Models used purely for answering questions run slowly and experience heavy hallucination issues
- A 20B model with a low quantization like iq3_xxs provides acceptable document comprehension and question answering
- Avoiding Ollama is suggested due to perceived bloat and degraded performance compared to llama.cpp


