Ollama for Linux – Run LLMs on Linux with GPU Acceleration

Sep 26, 2023 ·

Hi HN,

Over the last few months I've been working with some folks on a tool named Ollama (https://github.com/jmorganca/ollama) to run open-source LLMs like Llama 2, Code Llama and Falcon locally, starting with macOS.

The biggest ask since then has been "how can I run Ollama on Linux?" with GPU support out of the box. Setting up and configuring CUDA and then compiling and running llama.cpp (which is a fantastic library and runs under the hood) can be quite painful on different combinations of linux distributions and Nvidia GPUs. The goal for Ollama's linux version was to automate this process to make it easy to get up and running.

The is the first Linux release! There's still lots to do, but I wanted to share it here for to see what everyone thinks. Thanks for anyone who has given it a try and sent feedback!

Comments URL: https://news.ycombinator.com/item?id=37661755

Points: 173

# Comments: 54