sartifyllc/MultiLinguSwahili-bge-small-en-v1.5-nli-matryoshka

SentenceTransformer based on BAAI/bge-small-en-v1.5

This is a sentence-transformers model finetuned from BAAI/bge-small-en-v1.5 on the Mollel/swahili-n_li-triplet-swh-eng dataset. It maps sentences & paragraphs to a 384-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

• Model Type: Sentence Transformer
• Base model: BAAI/bge-small-en-v1.5
• Maximum Sequence Length: 512 tokens
• Output Dimensionality: 384 tokens
• Similarity Function: Cosine Similarity
• Training Dataset:
  • Mollel/swahili-n_li-triplet-swh-eng

Model Sources

• Documentation: Sentence Transformers Documentation
• Repository: Sentence Transformers on GitHub
• Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 512, 'do_lower_case': True}) with Transformer model: BertModel 
  (1): Pooling({'word_embedding_dimension': 384, 'pooling_mode_cls_token': True, 'pooling_mode_mean_tokens': False, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sartifyllc/MultiLinguSwahili-bge-small-en-v1.5-nli-matryoshka")
# Run inference
sentences = [
    'Mwanamume na mwanamke wachanga waliovaa mikoba wanaweka au kuondoa kitu kutoka kwenye mti mweupe wa zamani, huku watu wengine wamesimama au wameketi nyuma.',
    'mwanamume na mwanamke wenye mikoba',
    'tai huruka',
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 384]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities.shape)
# [3, 3]

Evaluation

Metrics

Semantic Similarity

• Dataset: sts-test-256
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.6832
spearman_cosine	0.6771
pearson_manhattan	0.6892
spearman_manhattan	0.6892
pearson_euclidean	0.6917
spearman_euclidean	0.6917
pearson_dot	0.6418
spearman_dot	0.6286
pearson_max	0.6917
spearman_max	0.6917

Semantic Similarity

• Dataset: sts-test-128
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.6753
spearman_cosine	0.6731
pearson_manhattan	0.6907
spearman_manhattan	0.6928
pearson_euclidean	0.6934
spearman_euclidean	0.6941
pearson_dot	0.6004
spearman_dot	0.5858
pearson_max	0.6934
spearman_max	0.6941

Semantic Similarity

• Dataset: sts-test-64
• Evaluated with EmbeddingSimilarityEvaluator

Metric	Value
pearson_cosine	0.6546
spearman_cosine	0.6524
pearson_manhattan	0.6837
spearman_manhattan	0.6797
pearson_euclidean	0.6861
spearman_euclidean	0.6816
pearson_dot	0.5121
spearman_dot	0.4914
pearson_max	0.6861
spearman_max	0.6816

Training Details

Training Dataset

Mollel/swahili-n_li-triplet-swh-eng

• Dataset: Mollel/swahili-n_li-triplet-swh-eng
• Size: 1,115,700 training samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 7 tokens mean: 15.18 tokens max: 80 tokens	min: 5 tokens mean: 18.53 tokens max: 52 tokens	min: 5 tokens mean: 17.8 tokens max: 53 tokens

• Samples:

  anchor	positive	negative
  A person on a horse jumps over a broken down airplane.	A person is outdoors, on a horse.	A person is at a diner, ordering an omelette.
  Mtu aliyepanda farasi anaruka juu ya ndege iliyovunjika.	Mtu yuko nje, juu ya farasi.	Mtu yuko kwenye mkahawa, akiagiza omelette.
  Children smiling and waving at camera	There are children present	The kids are frowning

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Evaluation Dataset

Mollel/swahili-n_li-triplet-swh-eng

• Dataset: Mollel/swahili-n_li-triplet-swh-eng
• Size: 13,168 evaluation samples
• Columns: anchor, positive, and negative
• Approximate statistics based on the first 1000 samples:

  	anchor	positive	negative
  type	string	string	string
  details	min: 6 tokens mean: 26.43 tokens max: 94 tokens	min: 5 tokens mean: 13.37 tokens max: 65 tokens	min: 5 tokens mean: 14.7 tokens max: 54 tokens

• Samples:

  anchor	positive	negative
  Two women are embracing while holding to go packages.	Two woman are holding packages.	The men are fighting outside a deli.
  Wanawake wawili wanakumbatiana huku wakishikilia vifurushi vya kwenda.	Wanawake wawili wanashikilia vifurushi.	Wanaume hao wanapigana nje ya duka la vyakula vitamu.
  Two young children in blue jerseys, one with the number 9 and one with the number 2 are standing on wooden steps in a bathroom and washing their hands in a sink.	Two kids in numbered jerseys wash their hands.	Two kids in jackets walk to school.

• Loss: MatryoshkaLoss with these parameters:

  {
      "loss": "MultipleNegativesRankingLoss",
      "matryoshka_dims": [
          256,
          128,
          64
      ],
      "matryoshka_weights": [
          1,
          1,
          1
      ],
      "n_dims_per_step": -1
  }
  

Training Hyperparameters

Non-Default Hyperparameters

• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• learning_rate: 2e-05
• num_train_epochs: 1
• warmup_ratio: 0.1
• bf16: True
• batch_sampler: no_duplicates

All Hyperparameters

Click to expand

• overwrite_output_dir: False
• do_predict: False
• prediction_loss_only: True
• per_device_train_batch_size: 64
• per_device_eval_batch_size: 64
• per_gpu_train_batch_size: None
• per_gpu_eval_batch_size: None
• gradient_accumulation_steps: 1
• eval_accumulation_steps: None
• learning_rate: 2e-05
• weight_decay: 0.0
• adam_beta1: 0.9
• adam_beta2: 0.999
• adam_epsilon: 1e-08
• max_grad_norm: 1.0
• num_train_epochs: 1
• max_steps: -1
• lr_scheduler_type: linear
• lr_scheduler_kwargs: {}
• warmup_ratio: 0.1
• warmup_steps: 0
• log_level: passive
• log_level_replica: warning
• log_on_each_node: True
• logging_nan_inf_filter: True
• save_safetensors: True
• save_on_each_node: False
• save_only_model: False
• no_cuda: False
• use_cpu: False
• use_mps_device: False
• seed: 42
• data_seed: None
• jit_mode_eval: False
• use_ipex: False
• bf16: True
• fp16: False
• fp16_opt_level: O1
• half_precision_backend: auto
• bf16_full_eval: False
• fp16_full_eval: False
• tf32: None
• local_rank: 0
• ddp_backend: None
• tpu_num_cores: None
• tpu_metrics_debug: False
• debug: []
• dataloader_drop_last: False
• dataloader_num_workers: 0
• dataloader_prefetch_factor: None
• past_index: -1
• disable_tqdm: False
• remove_unused_columns: True
• label_names: None
• load_best_model_at_end: False
• ignore_data_skip: False
• fsdp: []
• fsdp_min_num_params: 0
• fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
• fsdp_transformer_layer_cls_to_wrap: None
• accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'gradient_accumulation_kwargs': None}
• deepspeed: None
• label_smoothing_factor: 0.0
• optim: adamw_torch
• optim_args: None
• adafactor: False
• group_by_length: False
• length_column_name: length
• ddp_find_unused_parameters: None
• ddp_bucket_cap_mb: None
• ddp_broadcast_buffers: False
• dataloader_pin_memory: True
• dataloader_persistent_workers: False
• skip_memory_metrics: True
• use_legacy_prediction_loop: False
• push_to_hub: False
• resume_from_checkpoint: None
• hub_model_id: None
• hub_strategy: every_save
• hub_private_repo: False
• hub_always_push: False
• gradient_checkpointing: False
• gradient_checkpointing_kwargs: None
• include_inputs_for_metrics: False
• eval_do_concat_batches: True
• fp16_backend: auto
• push_to_hub_model_id: None
• push_to_hub_organization: None
• mp_parameters:
• auto_find_batch_size: False
• full_determinism: False
• torchdynamo: None
• ray_scope: last
• ddp_timeout: 1800
• torch_compile: False
• torch_compile_backend: None
• torch_compile_mode: None
• dispatch_batches: None
• split_batches: None
• include_tokens_per_second: False
• include_num_input_tokens_seen: False
• neftune_noise_alpha: None
• optim_target_modules: None
• batch_sampler: no_duplicates
• multi_dataset_batch_sampler: proportional

Training Logs

Epoch	Step	Training Loss	sts-test-128_spearman_cosine	sts-test-256_spearman_cosine	sts-test-64_spearman_cosine
0.0115	100	9.6847	-	-	-
0.0229	200	8.5336	-	-	-
0.0344	300	7.768	-	-	-
0.0459	400	7.2049	-	-	-
0.0574	500	6.9425	-	-	-
0.0688	600	7.029	-	-	-
0.0803	700	6.259	-	-	-
0.0918	800	6.0939	-	-	-
0.1032	900	5.991	-	-	-
0.1147	1000	5.39	-	-	-
0.1262	1100	5.3214	-	-	-
0.1377	1200	5.1469	-	-	-
0.1491	1300	4.901	-	-	-
0.1606	1400	5.2725	-	-	-
0.1721	1500	5.077	-	-	-
0.1835	1600	4.8006	-	-	-
0.1950	1700	4.5318	-	-	-
0.2065	1800	4.48	-	-	-
0.2180	1900	4.5752	-	-	-
0.2294	2000	4.427	-	-	-
0.2409	2100	4.4021	-	-	-
0.2524	2200	4.5903	-	-	-
0.2639	2300	4.4561	-	-	-
0.2753	2400	4.372	-	-	-
0.2868	2500	4.2698	-	-	-
0.2983	2600	4.3954	-	-	-
0.3097	2700	4.2697	-	-	-
0.3212	2800	4.125	-	-	-
0.3327	2900	4.3611	-	-	-
0.3442	3000	4.2527	-	-	-
0.3556	3100	4.1892	-	-	-
0.3671	3200	4.0417	-	-	-
0.3786	3300	3.9434	-	-	-
0.3900	3400	3.9797	-	-	-
0.4015	3500	3.9611	-	-	-
0.4130	3600	4.04	-	-	-
0.4245	3700	3.965	-	-	-
0.4359	3800	3.778	-	-	-
0.4474	3900	4.0624	-	-	-
0.4589	4000	3.8972	-	-	-
0.4703	4100	3.7882	-	-	-
0.4818	4200	3.8048	-	-	-
0.4933	4300	3.9253	-	-	-
0.5048	4400	3.9832	-	-	-
0.5162	4500	3.6644	-	-	-
0.5277	4600	3.7353	-	-	-
0.5392	4700	3.7768	-	-	-
0.5506	4800	3.796	-	-	-
0.5621	4900	3.875	-	-	-
0.5736	5000	3.7856	-	-	-
0.5851	5100	3.8898	-	-	-
0.5965	5200	3.6327	-	-	-
0.6080	5300	3.7727	-	-	-
0.6195	5400	3.8582	-	-	-
0.6310	5500	3.729	-	-	-
0.6424	5600	3.7088	-	-	-
0.6539	5700	3.8414	-	-	-
0.6654	5800	3.7624	-	-	-
0.6768	5900	3.8816	-	-	-
0.6883	6000	3.7483	-	-	-
0.6998	6100	3.7759	-	-	-
0.7113	6200	3.6674	-	-	-
0.7227	6300	3.6441	-	-	-
0.7342	6400	3.7779	-	-	-
0.7457	6500	3.6691	-	-	-
0.7571	6600	3.7636	-	-	-
0.7686	6700	3.7424	-	-	-
0.7801	6800	3.4943	-	-	-
0.7916	6900	3.5399	-	-	-
0.8030	7000	3.3658	-	-	-
0.8145	7100	3.2856	-	-	-
0.8260	7200	3.3702	-	-	-
0.8374	7300	3.3121	-	-	-
0.8489	7400	3.2322	-	-	-
0.8604	7500	3.1577	-	-	-
0.8719	7600	3.1873	-	-	-
0.8833	7700	3.1492	-	-	-
0.8948	7800	3.2035	-	-	-
0.9063	7900	3.1607	-	-	-
0.9177	8000	3.1557	-	-	-
0.9292	8100	3.0915	-	-	-
0.9407	8200	3.1335	-	-	-
0.9522	8300	3.14	-	-	-
0.9636	8400	3.1422	-	-	-
0.9751	8500	3.1923	-	-	-
0.9866	8600	3.1085	-	-	-
0.9980	8700	3.089	-	-	-
1.0	8717	-	0.6731	0.6771	0.6524

Framework Versions

• Python: 3.11.9
• Sentence Transformers: 3.0.1
• Transformers: 4.40.1
• PyTorch: 2.3.0+cu121
• Accelerate: 0.29.3
• Datasets: 2.19.0
• Tokenizers: 0.19.1

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MatryoshkaLoss

@misc{kusupati2024matryoshka,
    title={Matryoshka Representation Learning}, 
    author={Aditya Kusupati and Gantavya Bhatt and Aniket Rege and Matthew Wallingford and Aditya Sinha and Vivek Ramanujan and William Howard-Snyder and Kaifeng Chen and Sham Kakade and Prateek Jain and Ali Farhadi},
    year={2024},
    eprint={2205.13147},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

MultipleNegativesRankingLoss

@misc{henderson2017efficient,
    title={Efficient Natural Language Response Suggestion for Smart Reply}, 
    author={Matthew Henderson and Rami Al-Rfou and Brian Strope and Yun-hsuan Sung and Laszlo Lukacs and Ruiqi Guo and Sanjiv Kumar and Balint Miklos and Ray Kurzweil},
    year={2017},
    eprint={1705.00652},
    archivePrefix={arXiv},
    primaryClass={cs.CL}
}