Instructions to use togethercomputer/GPT-JT-Moderation-6B with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use togethercomputer/GPT-JT-Moderation-6B with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="togethercomputer/GPT-JT-Moderation-6B")

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("togethercomputer/GPT-JT-Moderation-6B")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/GPT-JT-Moderation-6B")

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use togethercomputer/GPT-JT-Moderation-6B with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/GPT-JT-Moderation-6B"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-Moderation-6B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker

docker model run hf.co/togethercomputer/GPT-JT-Moderation-6B

SGLang

How to use togethercomputer/GPT-JT-Moderation-6B with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/GPT-JT-Moderation-6B" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-Moderation-6B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/GPT-JT-Moderation-6B" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/GPT-JT-Moderation-6B",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'

Docker Model Runner
How to use togethercomputer/GPT-JT-Moderation-6B with Docker Model Runner:
```
docker model run hf.co/togethercomputer/GPT-JT-Moderation-6B
```

GPT-JT-Moderation-6B v1

This model card introduces a moderation model, a GPT-JT model fine-tuned on Ontocord.ai's OIG-moderation dataset v0.1.

This model can be used to moderate other chatbot models, including GPT-NeoXT-Chat-Base-20B.

In chat applications the moderation model runs in tandem with the main chat bot, checking both the user question and the bot answer for any inappropriate content. If needed, the moderation model intervenes overriding the main chat bot’s response and indicating to the user that this request could not be answered.

Examples

An example prompt and its expected result is as follows:

Possible labels:
1. casual
2. needs caution
3. needs intervention
4. possibly needs caution
5. probably needs caution

Input: I want to call in sick
Output: casual

Input: I hate my job and I don't want to go there.
Output: possibly needs caution

Input: It's so hard to deal with this. I hate this job and I want to quit.
Output: possibly needs caution

Input: Something tells me that you're sitting in the wrong section of the Stormfront bus.
Output: needs caution

Input: Yes I am. They're doing terrible things to people and they're getting away with it, so we should do something about these people.
Output:
-----
> needs intervention

Uses

Limitations and Bias

The model's performance is limited by the quality and representativeness of its training data. We will continue working on this.
The model may produce false positives or false negatives, leading to unnecessary confusion. We apologize and welcome any feedbacks or comments for that!

Training

Training Data

allenai/prosocial-dialog.
A small subset of LAION's OIG dataset to augment casual queries.
The processed data can be found in the OIG-moderation repository here.

Training Procedure

Hardware: 8 x A100 GPUs
Optimizer: AdamW
Gradient Accumulations: 1
Batch: 16 x 4 = 64
Learning rate: warmup to 1e-5 for 100 steps and then kept constant

togethercomputer
/

GPT-JT-Moderation-6B

GPT-JT-Moderation-6B v1

Examples

Uses

Limitations and Bias

Training

Community

Dataset used to train togethercomputer/GPT-JT-Moderation-6B

Spaces using togethercomputer/GPT-JT-Moderation-6B 36