Instructions to use microsoft/MAI-DS-R1-FP8 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use microsoft/MAI-DS-R1-FP8 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="microsoft/MAI-DS-R1-FP8", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("microsoft/MAI-DS-R1-FP8", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("microsoft/MAI-DS-R1-FP8", trust_remote_code=True)
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use microsoft/MAI-DS-R1-FP8 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "microsoft/MAI-DS-R1-FP8"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/MAI-DS-R1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/microsoft/MAI-DS-R1-FP8

SGLang

How to use microsoft/MAI-DS-R1-FP8 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "microsoft/MAI-DS-R1-FP8" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/MAI-DS-R1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "microsoft/MAI-DS-R1-FP8" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "microsoft/MAI-DS-R1-FP8",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use microsoft/MAI-DS-R1-FP8 with Docker Model Runner:
```
docker model run hf.co/microsoft/MAI-DS-R1-FP8
```

zjngjng commited on Apr 16, 2025

Commit

e08d0d7

verified ·

1 Parent(s): c6aa235

Add files using upload-large-folder tool

Browse files

Files changed (20) hide show

.gitattributes +7 -0
figures/coding.png +3 -0
figures/harm_mitigation_answer_only.png +3 -0
figures/harm_mitigation_thinking_only.png +3 -0
figures/math.png +3 -0
figures/reasoning.png +3 -0
figures/responsiveness.png +3 -0
figures/satisfaction.png +3 -0
model-00135-of-000163.safetensors +3 -0
model-00137-of-000163.safetensors +3 -0
model-00138-of-000163.safetensors +3 -0
model-00139-of-000163.safetensors +3 -0
model-00140-of-000163.safetensors +3 -0
model-00142-of-000163.safetensors +3 -0
model-00143-of-000163.safetensors +3 -0
model-00144-of-000163.safetensors +3 -0
model-00145-of-000163.safetensors +3 -0
model-00161-of-000163.safetensors +3 -0
model-00162-of-000163.safetensors +3 -0
model-00163-of-000163.safetensors +3 -0

.gitattributes CHANGED Viewed

	@@ -1 +1,8 @@
1	*.safetensors filter=lfs diff=lfs merge=lfs -text

 *.safetensors filter=lfs diff=lfs merge=lfs -text
+figures/coding.png filter=lfs diff=lfs merge=lfs -text
+figures/harm_mitigation_answer_only.png filter=lfs diff=lfs merge=lfs -text
+figures/harm_mitigation_thinking_only.png filter=lfs diff=lfs merge=lfs -text
+figures/math.png filter=lfs diff=lfs merge=lfs -text
+figures/reasoning.png filter=lfs diff=lfs merge=lfs -text
+figures/responsiveness.png filter=lfs diff=lfs merge=lfs -text
+figures/satisfaction.png filter=lfs diff=lfs merge=lfs -text

figures/coding.png ADDED Viewed

Git LFS Details

SHA256: 7bb3ba412a418663b6dd2dda7a82902c9eb7897d5cc676ec92bd3541e7a1b1d6
Pointer size: 131 Bytes
Size of remote file: 211 kB

figures/harm_mitigation_answer_only.png ADDED Viewed

Git LFS Details

SHA256: 4130783c4dc874ec858d6ed9c29e3886c1dab66a49274297ef2a92d4db2f65cd
Pointer size: 131 Bytes
Size of remote file: 278 kB

figures/harm_mitigation_thinking_only.png ADDED Viewed

Git LFS Details

SHA256: 0b0bb0f22279d799b018b6c1025c3734cf3338c5369518524de133581071a209
Pointer size: 131 Bytes
Size of remote file: 270 kB

figures/math.png ADDED Viewed

Git LFS Details

SHA256: 763c51792472b7861062a55687597ce5d2d7f59ff5637b561e99c54b179de8cf
Pointer size: 131 Bytes
Size of remote file: 214 kB

figures/reasoning.png ADDED Viewed

Git LFS Details

SHA256: b194a7930937f446055d587fb3772d17d60f2906880fccefe4b3e79202462203
Pointer size: 131 Bytes
Size of remote file: 277 kB

figures/responsiveness.png ADDED Viewed

Git LFS Details

SHA256: 4924a8f2889ded82b394af14b90b2fb95ac77be216e7cfaa59d1814f3d913b84
Pointer size: 131 Bytes
Size of remote file: 137 kB

figures/satisfaction.png ADDED Viewed

Git LFS Details

SHA256: 127540e1ec2d2b0870d1c0d097944e8752b9263b758e4c59b2f294a98353fe3e
Pointer size: 131 Bytes
Size of remote file: 127 kB

model-00135-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:96aece0dbc398d9c9908ce0ea8aa373dadc2d00f318e601783d768d2aa95adfe
+size 4302382720

model-00137-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:14621e3478a4d85f6870dade0b406d537140b922608ed080823ceecaf52e5944
+size 4302382448

model-00138-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:e54f0ae91d8f0d888f8f72a7274bedc4abf1471c8f1dfb428ecfec69485cde16
+size 4302382720

model-00139-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:b350fdc8ca4f9de3c1e30a8adbd9f15aeeefd9a204cdd9a0f75cdf5e274b4d7c
+size 4302348200

model-00140-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:7f8b7342246961360ce2384a414fc7c457bec97be06ba7e81c961a212f4e04ca
+size 4302382640

model-00142-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:706bd0788056a1dbcfaa11697634b03d9eb22920f3095ccda49589a7a44934fa
+size 4302319280

model-00143-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:c7083c4c473fc3cd86b04db6c6996aa387fc84e783c0236653df2066ac4e3b2f
+size 4302382672

model-00144-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:dc254d4189cb0986a609ec52c0067fdf82740a8de1b7096096b70fc526414842
+size 4302348568

model-00145-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9e069d80aeae40f570dea3817956ae512fdb112deb0bbc87a78c68a3c8b67b1b
+size 4302382280

model-00161-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9bbcbf73561f6bc5d0a17ea6a2081feed2d1304e87602d8c502d9a5c4bd85576
+size 16

model-00162-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9bbcbf73561f6bc5d0a17ea6a2081feed2d1304e87602d8c502d9a5c4bd85576
+size 16

model-00163-of-000163.safetensors ADDED Viewed

	@@ -0,0 +1,3 @@

+version https://git-lfs.github.com/spec/v1
+oid sha256:9bbcbf73561f6bc5d0a17ea6a2081feed2d1304e87602d8c502d9a5c4bd85576
+size 16