Svngoku/kongo-llama

Kongo Llama Experiment

Model Details

• Tokenizer

from transformers import PreTrainedTokenizerFast

# Assuming your custom tokenizer is `tokenizer`
wrapped_tokenizer = PreTrainedTokenizerFast(
    tokenizer_object=tokenizer,
    bos_token="[BOS]",  # Replace with your special tokens
    eos_token="[EOS]",  # Replace with your special tokens
    unk_token="[UNK]",
    pad_token="[PAD]"
)

# Ensure padding is applied to the right side (used in causal language modeling)
wrapped_tokenizer.padding_side = "right"

• Model

from transformers import LlamaConfig, LlamaForCausalLM

config = LlamaConfig(
    vocab_size=len(wrapped_tokenizer),  # Get vocab size from the wrapped tokenizer
    hidden_size=512,                    # Adjust model size as needed
    intermediate_size=1024,
    num_hidden_layers=8,                # Set number of layers and heads
    num_attention_heads=8,
    max_position_embeddings=512,
    rms_norm_eps=1e-6,
    initializer_range=0.02,
    use_cache=True,
    pad_token_id=wrapped_tokenizer.pad_token_id,
    bos_token_id=wrapped_tokenizer.bos_token_id,
    eos_token_id=wrapped_tokenizer.eos_token_id,
)

model = LlamaForCausalLM(config)

• Trainer

from transformers import TrainingArguments, Trainer

# Define training arguments
training_args = TrainingArguments(
    output_dir="kongo-llama",           # Output directory for model and checkpoints
    num_train_epochs=1,
    per_device_train_batch_size=8,
    learning_rate=5e-5,
    warmup_steps=500,
    weight_decay=0.01,
    logging_dir="./logs",
    logging_steps=10,
    save_steps=1000,
)


trainer = Trainer(
    model=model,                          # Your model instance
    args=training_args,                   # Training arguments
    train_dataset=dataset,                # Tokenized dataset with input_ids and labels
    tokenizer=wrapped_tokenizer,          # Wrapped tokenizer
    data_collator=data_collator,          # Data collator for causal language modeling
)

Model Description

This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.

• Developed by: [More Information Needed]
• Funded by [optional]: [More Information Needed]
• Shared by [optional]: [More Information Needed]
• Model type: [More Information Needed]
• Language(s) (NLP): [More Information Needed]
• License: [More Information Needed]
• Finetuned from model [optional]: [More Information Needed]

Model Sources [optional]

• Repository: [More Information Needed]
• Paper [optional]: [More Information Needed]
• Demo [optional]: [More Information Needed]

Uses

# Use a pipeline as a high-level helper
from transformers import pipeline
pipe = pipeline("text-generation", model="Svngoku/kongo-llama")

pipe(
  "Mbote, mono ",     
  max_length=150, 
  num_beams=5,     
  temperature=0.7, 
  do_sample=True,
  top_p=0.95 
)

[{'generated_text': 'Mbote, mono  na ngambu ya mpila ya bo ke monisa nde bantu yonso zole yina kaka na kati ya bo ke sadilaka yo mosi ve kana bo ke vandaka ti yo yina, to bima ya nkaka ya bo ke salaka sambu na bana ya zulu.'}]