Instructions to use ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("text-generation", model="ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
pipe(messages)

# Load model directly
from transformers import AutoTokenizer, AutoModelForCausalLM

tokenizer = AutoTokenizer.from_pretrained("ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16")
model = AutoModelForCausalLM.from_pretrained("ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16")
messages = [
    {"role": "user", "content": "Who are you?"},
]
inputs = tokenizer.apply_chat_template(
	messages,
	add_generation_prompt=True,
	tokenize=True,
	return_dict=True,
	return_tensors="pt",
).to(model.device)

outputs = model.generate(**inputs, max_new_tokens=40)
print(tokenizer.decode(outputs[0][inputs["input_ids"].shape[-1]:]))

Notebooks
Google Colab
Kaggle
Local Apps

vLLM

How to use ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16

SGLang

How to use ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Docker Model Runner
How to use ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16 with Docker Model Runner:
```
docker model run hf.co/ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16
```

Model Card for LLaMA-3.1-8B-Instruct (SmoothQuant FP16)

This repository contains a SmoothQuant-processed FP16 version of Meta LLaMA-3.1-8B-Instruct.
The model is not quantized. SmoothQuant is applied only as an outlier-suppression preprocessing step to improve downstream low-bit quantization.

Model Details

Model Description

This model applies SmoothQuant (activation-aware weight smoothing) to the FP16 weights of LLaMA-3.1-8B-Instruct.
No architectural changes or precision reduction are performed.

The purpose of this model is to serve as an intermediate checkpoint in a deployment pipeline targeting CPU and edge devices using llama.cpp and mixed-precision quantization.

Developed by: Muhammad Arslan Rafiq
Funded by: Academic coursework (AI on Edge Devices)
Shared by: ArslanRobo
Model type: Decoder-only Transformer (Causal Language Model)
Language(s) (NLP): English
License: LLaMA 3.1 Community License
Finetuned from model: meta-llama/Llama-3.1-8B-Instruct

Model Sources

Repository: https://huggingface.co/ArslanRobo
Paper: SmoothQuant – https://arxiv.org/abs/2211.10438
Base Model: https://huggingface.co/meta-llama/Llama-3.1-8B-Instruct

Uses

Direct Use

Research on outlier suppression techniques
Preprocessing before post-training quantization
Analysis of quantization robustness
Academic experiments on edge deployment

Downstream Use

Conversion to GGUF format
Mixed-precision or low-bit quantization using llama.cpp
CPU-only and embedded device inference (e.g., Raspberry Pi)

Out-of-Scope Use

Production deployment without quantization
Safety-critical or regulated applications
Fine-tuning without further evaluation

Bias, Risks, and Limitations

Inherits all biases and limitations of the original LLaMA-3.1 model
SmoothQuant does not reduce model size or latency by itself
Quality improvements appear only after quantization
No safety fine-tuning applied beyond the base model

Recommendations

Users should:

Apply quantization after SmoothQuant
Evaluate perplexity and downstream task performance
Use appropriate safety filters for real-world deployment

How to Get Started with the Model

from transformers import AutoModelForCausalLM, AutoTokenizer
import torch

model = AutoModelForCausalLM.from_pretrained(
    "ArslanRobo/llama-3.1-8b-instruct-smoothquant-fp16",
    torch_dtype=torch.float16,
    device_map="auto"
)

tokenizer = AutoTokenizer.from_pretrained(
    "ArslanRobo/llama-3.1-8b-instruct-smoothquant-fp16"
)
Training Details
Training Data
No additional training data was used.
The model weights are derived directly from the base LLaMA-3.1-8B-Instruct model.

Training Procedure
Preprocessing
Activation statistics collected on WikiText-2

SmoothQuant applied with α = 0.5

FP16 precision preserved

Training Hyperparameters
Training regime: FP16 (no training, preprocessing only)

Speeds, Sizes, Times
Model size: ~15 GB (FP16)

SmoothQuant calibration: ~5–10 minutes on NVIDIA T4

Evaluation
Testing Data, Factors & Metrics
Testing Data
WikiText-2 (used for calibration and analysis)

Factors
Activation outliers

Weight distribution

Quantization sensitivity

Metrics
Perplexity (evaluated post-quantization)

Quantization MSE and SNR (simulated)

Results
SmoothQuant reduces quantization error

Improves robustness for 4-bit and mixed-precision quantization

Minimal impact on FP16 perplexity

Model Examination
Weight distribution smoothing

Reduced activation outliers

Improved numerical stability for low-bit quantization

Environmental Impact
Hardware Type: NVIDIA T4 GPU

Hours used: ~1 hour

Cloud Provider: Google Colab / Kaggle

Compute Region: Not specified

Carbon Emitted: Not measured

Technical Specifications
Model Architecture and Objective
Decoder-only Transformer

Causal language modeling objective

Compute Infrastructure
Hardware
NVIDIA T4 GPU (16 GB VRAM)

Software
PyTorch

Hugging Face Transformers

Accelerate

Datasets

Citation
BibTeX
bibtex
Copy code
@article{xiao2023smoothquant,
  title={SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models},
  author={Xiao, Guangxuan and others},
  journal={arXiv preprint arXiv:2211.10438},
  year={2023}
}
APA
Xiao, G., et al. (2023). SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models. arXiv:2211.10438.

Model Card Authors
Muhammad Arslan Rafiq

Downloads last month: 2

Safetensors

Model size

6B params

Tensor type

F16

Paper for ArslanRobo/llama-3.1-8b-instruct-smoothquant-Pruned30Tailor-fp16

SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models

Paper • 2211.10438 • Published Nov 18, 2022 • 6