Deploying a model

OpenShift AI can deploy models using vLLM with single-model serving.

Prerequisites

A user with admin privileges has enabled the single-model serving platform on your OpenShift cluster.

Procedure

In the OpenShift AI dashboard, navigate to the project details page and click the Models tab.
In the Single-model serving platform tile, click Select single-model.

Depending on how model serving has been configured on your cluster, you might see only one model serving platform option.
Click the Deploy model button, then provide the following values in the form:
1. For Model deployment name, type Granite.
2. For Serving runtime, select vLLM NVIDIA GPU ServingRuntime for KServe for deployment on NVIDIA GPUs. If you are using Intel or AMD GPUs, select vLLM Intel or AMD GPU.
3. For Model framework (name - version), select vLLM.
4. For Deployment mode, select Standard.
5. For Number of model server replicas to deploy, select 1.
6. For Model server size, select Custom.
7. For CPUs requested select 1
8. For CPU limit, select 4
9. For Memory requested, select 16GiB
10. For Memory limit, select 24GiB
11. For Accelerator, select NVIDIA GPU or the appropriate accelerator for your cluster.
12. For Number of accelerators, select 1.
13. For Model route, select the checkbox for Make deployed models available through an external route.
14. For Token Authentication, select the checkbox for Require token authentication.
15. For Source model location, select Create connection.
16. For Connection type, select URI - v1.
17. For Connection name, type Red Hat Catalog Granite.
18. For URI, enter the following URI:
  oci://registry.redhat.io/rhelai1/modelcar-granite-3-1-8b-lab-v1:1.4
For Configuration Parameters - Additional serving runtime arguments, enter the following argument:
```
--max-model-len=4096
```
Click Deploy.

Verification

Notice the loading symbol under the Status section. The symbol changes to a green checkmark when the deployment completes successfully.

Next step

Creating a workbench