> For the complete documentation index, see [llms.txt](https://docs.release.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.release.com/release.ai/release.ai-serving-inference.md).

# Serving Inference

Using Release.ai to serve inference APIs.

Once you have tuned a model, you need a way to serve it to the rest of your apps and ultimately your users. An inference server such as Ollama is one way to achieve this. In this doc we will show you how to serve your models using Release.

## Creating and running the Ollama service

Create an Ollama server by selecting `Create Application` in the Applications page then click `Create from template`. Select Execution type of `Server`, then pick `Ollama` from the list of available templates. Hit `Finish` then click `Create Environment`. If you want this environment to stay up persistently, select `Permanent` from the `Environment Type` and give it a name like `production` or `staging`.

The `Llm` field can be populated with the name of a pre-build Ollama model. You can browse available models on the [Ollama library](https://ollama.com/library/).

Make sure to select a cluster that has GPU resources available in the `Advanced Options` section.

Now click `Create Environment`. Once the environment is up and running, you can begin sending queries to it.

## Querying the Ollama server

You can query your new server using CURL and the hostname from the environment page.

```bash
curl https://${OLLAMA_BASE_URL}/api/chat -d '{
  "model": "${YOUR_MODEL_NAME}",
  "stream": false,
  "messages": [
    { "role": "user", "content": "why is the sky blue" }
  ]
}'
```

## Loading a new model

When you tune a new version of a model and convert it to HuggingFace format, you can load it into Ollama by using CURL.

```bash
curl https://${OLLAMA_BASE_URL}/api/create -d '{
    "name": "${YOUR_MODEL_NAME}",
    "modelfile": "FROM ${YOUR_MODEL_PATH}/ollama.bin\nTEMPLATE \"[INST] <<SYS>>{{ .System }}<</SYS>>\n{{ .Prompt }} [/INST]\""
  }'
```

Now you can query it with the same command given above and the new name.


---

# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.

## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation by asking a question.

Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters:

```
GET https://docs.release.com/release.ai/release.ai-serving-inference.md?ask=<question>&goal=<user_goal>
```

`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case.

The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.