Instructions to use bond005/meno-lite-0.1-gguf with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use bond005/meno-lite-0.1-gguf with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="bond005/meno-lite-0.1-gguf",
	filename="meno-lite-0.1-IQ3_M.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use bond005/meno-lite-0.1-gguf with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M
# Run inference directly in the terminal:
llama-cli -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M
# Run inference directly in the terminal:
./llama-cli -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M
# Run inference directly in the terminal:
./build/bin/llama-cli -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Use Docker

docker model run hf.co/bond005/meno-lite-0.1-gguf:Q4_K_M

LM Studio
Jan

vLLM

How to use bond005/meno-lite-0.1-gguf with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "bond005/meno-lite-0.1-gguf"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "bond005/meno-lite-0.1-gguf",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/bond005/meno-lite-0.1-gguf:Q4_K_M

Ollama
How to use bond005/meno-lite-0.1-gguf with Ollama:
```
ollama run hf.co/bond005/meno-lite-0.1-gguf:Q4_K_M
```

Unsloth Studio new

How to use bond005/meno-lite-0.1-gguf with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bond005/meno-lite-0.1-gguf to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for bond005/meno-lite-0.1-gguf to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for bond005/meno-lite-0.1-gguf to start chatting

Pi new

How to use bond005/meno-lite-0.1-gguf with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "bond005/meno-lite-0.1-gguf:Q4_K_M"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use bond005/meno-lite-0.1-gguf with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf bond005/meno-lite-0.1-gguf:Q4_K_M

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default bond005/meno-lite-0.1-gguf:Q4_K_M

Run Hermes

hermes

Docker Model Runner
How to use bond005/meno-lite-0.1-gguf with Docker Model Runner:
```
docker model run hf.co/bond005/meno-lite-0.1-gguf:Q4_K_M
```

Lemonade

How to use bond005/meno-lite-0.1-gguf with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull bond005/meno-lite-0.1-gguf:Q4_K_M

Run and chat with the model

lemonade run user.meno-lite-0.1-gguf-Q4_K_M

List all available models

lemonade list

Meno-Lite-0.1 GGUF

This repository contains quantized GGUF versions of Meno-Lite-0.1. All variants were produced using an importance matrix computed on the train split of the ru_llm_calibration dataset, and are intended to be run with llama.cpp.

Available Formats

Quantization type	File size	Quality	Recommendation
Q8_0	~8.05 GB	Virtually identical to FP16	Best quality. Ideal for CPU inference when memory is not a constraint.
Q5_K_M	~5.41 GB	Minimal degradation	Recommended balance. Excellent speed and quality, fits most consumer GPUs.
Q4_K_M	~4.65 GB	Moderate degradation	"Golden standard". Best trade-off between size and quality.
IQ3_M	~3.54 GB	Noticeable degradation	Maximum memory savings. Quality drops visibly; suited for highly constrained devices.

Quality Evaluation

Quality was measured on the test split of the Ru LLM Calibration dataset using the llama-perplexity utility. The original FP16 model served as the reference.

Metric	Q8_0	Q5_K_M	Q4_K_M	IQ3_M
Mean PPL (Q) ↓	9.047	9.075	9.135	9.689
PPL correlation ↑	99.97%	99.87%	99.69%	98.64%
Mean KLD ↓	0.0020	0.0077	0.0174	0.0804
Same top p ↑	96.71%	94.36%	92.16%	84.58%

↑ – higher is better; ↓ – lower is better

How to interpret these metrics:

Mean PPL (Q): Lower is better. Shows the average perplexity of the quantized model.
PPL correlation: Closer to 100% indicates the quantized model behaves almost identically to FP16. Values above 99.5% are considered excellent.
Mean KLD: Measures the divergence between the output probability distributions. Lower is better; 0 means identical distributions.
Same top p: The percentage of tokens where the quantized model's top prediction matches the FP16 model. Higher is better – it reflects how often the model's first-choice token remains unchanged.

Usage

1. Install `llama.cpp`

Follow the official build instructions.

2. Run the model

# CLI
./llama-cli -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf -p "Привет, как дела?"

# Server with WebUI (default http://127.0.0.1:8080)
./llama-server -hf bond005/meno-lite-0.1-gguf -m meno-lite-0.1-Q4_K_M.gguf --host 0.0.0.0 --port 8080

For more details on available parameters, see the llama.cpp documentation.

About Meno-Lite-0.1

Meno-Lite-0.1 is a 7B model based on Qwen2.5, fine-tuned for RAG, document QA, information extraction, and knowledge graph construction. Read more about its capabilities, training procedure, and limitations in the main model card.

License

All quantized variants inherit the license of the original model (Apache 2.0).

Downloads last month: 829

GGUF

Model size

8B params

Architecture

qwen2

Hardware compatibility

3-bit

4-bit

5-bit

8-bit

View +1 variant

Model tree for bond005/meno-lite-0.1-gguf

Base model

Qwen/Qwen2.5-7B

Finetuned

RefalMachine/RuadaptQwen2.5-7B-Lite-Beta

Finetuned

bond005/meno-lite-0.1

Quantized

(1)

this model

Collection including bond005/meno-lite-0.1-gguf

Meno v0.1

Collection

Russian LLMs for RAG, Knowledge Graphs etc. • 5 items • Updated 12 days ago