Thanks for the GitHub link, I want to try this in a M2 with 32G memory. It will “waste” the GPU and neural cores, but might still run fast.
Off topic: I so much appreciate everyone’s work Getting LLMs running on inexpensive hardware! I have been having crazy amounts of fun with Ollama and a wide variety of tuned models. And, so fast!
I might be missing something but it seems like most of the speed up is from quantization which is commonly used already, and the CPU instance used here isn't that much cheaper (~10-15%?) than a GPU instance that could run the model. For high utilization workloads the extra throughput might be useful though.
If we observe the performance comparison graph (1), 2.8->9 tok/s is achieved via quantization, but the remaining jumps from 9->16.6->24.6 tok/s are achieved from the sparse fine-tuning.
Hetzner do cheap servers, but don't have many GPUs. If you're looking at saving money by running on CPU, you shouldn't be looking at one of the most expensive server providers.
With two/three instances, you can probably fit a 70B model into RAM, and you don't need super low latency between models to be able to do inference split layerwise between machines.
Don't think so. I suspect it would require quite in-depth surgery of llamacpp to add in the ability to send activations over the internet and pipeline stuff to keep all the cores busy.
I'd like to know the answer to that too. I shouldn't have implied that CPU inference on Hetzner is cheaper than GPU on AWS, when I don't have any idea on the cost of either.
Is it compared anywhere with a smaller model fine-tuned on the same thing? Edit: I didn't see any comparisons skimming the paper. More specifically fine tuned smaller models can often be pretty good. I'd want to see how 3B and 1B etc llama models fine-tuned on the same dataset perform. Is sparsity the key here or is it just fewer parameters?
The quantization part was interesting - they also quantized activations and have some adaptive quantization that accommodates outliers.
What "no drop in accuracy" means? Is it like they do some kind of lossless compression and it is guaranteed to behave same? Or are they claiming that as a result of a (subjective?) test that measures accuracy? If so how does such test works?
This seems like a big step forward in terms of running specific use-case trained inference locally, right? At least given current hardware generally deployed in business.
DeepSparse is on GitHub: https://github.com/neuralmagic/deepsparse