Skip to main content
Discovery icon for Luduvo Discord server

Luduvo

729 Online
3,553 Members
# general6 people

local llm recommendations for questions

Run a low-quantized 20B model via llama.cpp instead of Ollama to handle documentation reading and question answering efficiently.

  • Models used purely for answering questions run slowly and experience heavy hallucination issues
  • A 20B model with a low quantization like iq3_xxs provides acceptable document comprehension and question answering
  • Avoiding Ollama is suggested due to perceived bloat and degraded performance compared to llama.cpp
Summarized from recent messages, might be inaccurate

More conversations

Talk, Play, &
Hang Out

Find your space to hang out, discuss the latest updates with the party chat, and see who's down to join your next match.
Get App