Instructions to use princeton-nlp/gemma-2-9b-it-DPO with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use princeton-nlp/gemma-2-9b-it-DPO with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="princeton-nlp/gemma-2-9b-it-DPO")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("princeton-nlp/gemma-2-9b-it-DPO")
model = AutoModelForCausalLM.from_pretrained("princeton-nlp/gemma-2-9b-it-DPO")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Inference
Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use princeton-nlp/gemma-2-9b-it-DPO with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "princeton-nlp/gemma-2-9b-it-DPO"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/gemma-2-9b-it-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/princeton-nlp/gemma-2-9b-it-DPO

SGLang

How to use princeton-nlp/gemma-2-9b-it-DPO with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "princeton-nlp/gemma-2-9b-it-DPO" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/gemma-2-9b-it-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "princeton-nlp/gemma-2-9b-it-DPO" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "princeton-nlp/gemma-2-9b-it-DPO",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use princeton-nlp/gemma-2-9b-it-DPO with Docker Model Runner:
```
docker model run hf.co/princeton-nlp/gemma-2-9b-it-DPO
```
Browse Quantizations to use this model in llama.cpp, Ollama, LM Studio, or any compatible app.

gemma-2-9b-it-DPO Model Card

This model was trained under the same setup as gemma-2-9b-it-SimPO, with the DPO objective.

SimPO (Simple Preference Optimization) is an offline preference optimization algorithm designed to enhance the training of large language models (LLMs) with preference optimization datasets. SimPO aligns the reward function with the generation likelihood, eliminating the need for a reference model and incorporating a target reward margin to boost performance. Please refer to our preprint and github repo for more details.

Model Details

Model Description

We fine-tuned google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm with the DPO objective.

Developed by: Yu Meng, Mengzhou Xia, Danqi Chen
Model type: Causal Language Model
License: gemma
Finetuned from model: google/gemma-2-9b-it

Model Sources

Repository: https://github.com/princeton-nlp/SimPO
Paper: https://arxiv.org/pdf/2405.14734

How to Get Started with the Model

import torch
from transformers import pipeline

model_id = "princeton-nlp/gemma-2-9b-it-DPO"

generator = pipeline(
    "text-generation",
    model=model_id,
    model_kwargs={"torch_dtype": torch.bfloat16},
    device="cuda",
)
outputs = generator([{"role": "user", "content": "What's the difference between llamas and alpacas?"}], do_sample=False, max_new_tokens=200)
print(outputs[0]['generated_text'])

Training Details

Training Data

We use princeton-nlp/gemma2-ultrafeedback-armorm as the preference optimization dataset.

Training Hyperparameters

We used the following hyperparameters:

learning rate: 5e-7
batch size: 128
beta: 0.01

The other hyperparameters are kept the same with our SimPO recipe.

Speeds, Sizes, Times

Fine-tuning the google/gemma-2-9b-it on princeton-nlp/gemma2-ultrafeedback-armorm takes around 150 mins to finish on 8xH100 GPUs.

Evaluation Results

models	AE2 LC	AE2 WR	AE2 Length	AH	AH Length	GSM	GSM Length	MMLU	MMLU Length
google/gemma-2-9b-it	51.1	38.1	1571	40.8	545	87.4	395	72.7	515
princeton-nlp/gemma-2-9b-it-DPO	67.8	65.4	2016	58.9	717	88.5	392	72.2	624
princeton-nlp/gemma-2-9b-it-SimPO	72.4	65.9	1833	59.1	693	88.0	341	72.2	441

Technical Specifications

Model Architecture and Objective

The model architecture is based on google/gemma-2-9b-it. We use the DPO training objective.

Hardware

We used 8xH100 GPUs for model training.

Software

Training was done using the alignment-handbook library.

Citation

gemma model:

@article{gemma_2024,
    title={Gemma},
    url={https://www.kaggle.com/m/3301},
    DOI={10.34740/KAGGLE/M/3301},
    publisher={Kaggle},
    author={Gemma Team},
    year={2024}
}

DPO paper:

@article{rafailov2024direct,
  title={Direct Preference Optimization: Your language model is secretly a reward model},
  author={Rafailov, Rafael and Sharma, Archit and Mitchell, Eric and Manning, Christopher D and Ermon, Stefano and Finn, Chelsea},
  journal={Advances in Neural Information Processing Systems},
  volume={36},
  year={2024}
}

SimPO paper:

@article{meng2024simpo,
  title={{SimPO}: Simple preference optimization with a reference-free reward},
  author={Meng, Yu and Xia, Mengzhou and Chen, Danqi},
  journal={arXiv preprint arXiv:2405.14734},
  year={2024}
}

UltraFeedback paper:

@article{cui2023ultrafeedback,
  title={{UltraFeedback}: Boosting language models with high-quality feedback},
  author={Cui, Ganqu and Yuan, Lifan and Ding, Ning and Yao, Guanming and Zhu, Wei and Ni, Yuan and Xie, Guotong and Liu, Zhiyuan and Sun, Maosong},
  journal={arXiv preprint arXiv:2310.01377},
  year={2023}
}

ArmoRM paper:

@article{wang2024interpretable,
  title={Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts},
  author={Wang, Haoxiang and Xiong, Wei and Xie, Tengyang and Zhao, Han and Zhang, Tong},
  journal={arXiv preprint arXiv:2406.12845},
  year={2024}
}