SentenceTransformer based on intfloat/multilingual-e5-large

This is a sentence-transformers model finetuned from intfloat/multilingual-e5-large. It maps sentences & paragraphs to a 1024-dimensional dense vector space and can be used for semantic textual similarity, semantic search, paraphrase mining, text classification, clustering, and more.

Model Details

Model Description

Model Type: Sentence Transformer
Base model: intfloat/multilingual-e5-large
Maximum Sequence Length: 256 tokens
Output Dimensionality: 1024 dimensions
Similarity Function: Cosine Similarity

Model Sources

Documentation: Sentence Transformers Documentation
Repository: Sentence Transformers on GitHub
Hugging Face: Sentence Transformers on Hugging Face

Full Model Architecture

SentenceTransformer(
  (0): Transformer({'max_seq_length': 256, 'do_lower_case': False, 'architecture': 'XLMRobertaModel'})
  (1): Pooling({'word_embedding_dimension': 1024, 'pooling_mode_cls_token': False, 'pooling_mode_mean_tokens': True, 'pooling_mode_max_tokens': False, 'pooling_mode_mean_sqrt_len_tokens': False, 'pooling_mode_weightedmean_tokens': False, 'pooling_mode_lasttoken': False, 'include_prompt': True})
  (2): Normalize()
)

Usage

Direct Usage (Sentence Transformers)

First install the Sentence Transformers library:

pip install -U sentence-transformers

Then you can load this model and run inference.

from sentence_transformers import SentenceTransformer

# Download from the 🤗 Hub
model = SentenceTransformer("sentence_transformers_model_id")
# Run inference
sentences = [
    'How do I effectively navigate between high-end shopping districts and traditional markets in Hawaii?',
    "friends in your own privacy and do this until you are blue in the face or you can't speak anymore then head back home and chill out because that was a pretty chill day but definitely a fun and eventful one as well. so hopefully you're not too hungover. it's day six. you can sleep in a little bit, but we will do something in the morning, which is something you can only do on saturday. so it's really important to follow this itinerary this way if possible, which is to visit the kaka'ako farmer's market. now this is located near the ala moana area, which you can definitely take the bus to. and i believe the farmer market ends at noon, which isn't too late for a farmer's market. this is a really cool experience because there are so many vendors where you can get some fresh fruit some fresh produce some small businesses that make some delicious food whether it's hawaiian some pan-asian japanese american bakeries it's a little bit pricey i'm not gonna lie but hey we've been pretty frugal and cheap so far so it's okay to splurge a little bit on food here eat until your heart's content until the market closes hopefully and then while you're in the area, if you want to do some shopping, maybe you do, maybe you don't, that's okay, head to the ala moana mall, which is actually, i think, the biggest outlet mall, outdoor outlet mall, whatever you want to call it, in the united states. there's lots of good shopping options here. now, if shopping's not so much of your thing, or maybe it is, i still recommend afterwards, or instead of, heading to don quixote, which is actually a japanese sort of department store, i guess you can call it, but it's completely different than like walmart, and the one in hawaii is no exception. this is a really cool place to get souvenirs for really cheap price and to see some really interesting either asian or japanese or hawaiian goods that you can just look at and kind of be like, huh, i've never seen that before. or man, that advertisement's a little bit out of left field compared to what we've seen in america because hawaii is in america, but one of my favorite things about it is it has that sort of asian influence, which makes it feel definitely a little bit more exotic than let's say idaho or somewhere like that. and that can be it for day six if you'd like. if you're more of the adventurous type and you don't want to shop and maybe you get up early and then you stop at don quixote and you still have a lot of time in the day, you can do another hike. i really recommend doing three peaks. it's one of my favorite hikes i've ever done in my life. it's about an hour or so on the bus so you have to incorporate that time as well but you'll not regret it because the views are amazing. it's a little bit sketchy so be careful and only do the first peak of three if you feel comfortable but it is certainly an adventure so if you're an adventurous person, that's something to consider as well. if you're not and you're wiped from our itinerary so far, i don't blame you. chill out, catch another sunset, maybe enjoy the pool at your hotel, eat some food, and call it a day for day six. so finally, sadly, we are going to begin day seven, which is your last day here in hawaii, your last full day. here we are not going to do anything too crazy. we're just going to relax and take in the local area of waikiki. so to start the day, we're going to get an acai bowl, which in my opinion is the national food of hawaii. it's not poke, it's acai bowls, everybody. two of my recommendations are going to be aloe, aiello, that is, and island vintage, which is definitely very popular.",
    "you can time it at the right time. now, if you still have some more room left over, i really recommend going to ice monster, which is all this is in waikiki, everybody, so it's all walking. no buses for today. thank the lord. and get some delicious taiwanese shaved ice. it's definitely different than the snow cone, but it's very smooth and it just feels effortlessly like it's melting on your tongue like it's snow itself. very delicious, really good, definitely recommend it if you've never tried it. and it also fits the tropical feel perfectly. now while you're still exploring and enjoying waikiki, i really recommend going to the international marketplace where they have lots of cool stores, some good shopping as well.",
]
embeddings = model.encode(sentences)
print(embeddings.shape)
# [3, 1024]

# Get the similarity scores for the embeddings
similarities = model.similarity(embeddings, embeddings)
print(similarities)
# tensor([[1.0000, 0.6860, 0.3253],
#         [0.6860, 1.0000, 0.3919],
#         [0.3253, 0.3919, 1.0000]])

Training Details

Training Dataset

Unnamed Dataset

Size: 7,264 training samples
Columns: anchor, positive, and negative

Approximate statistics based on the first 1000 samples:

	anchor	positive	negative
type	string	string	string
details	min: 10 tokens mean: 24.3 tokens max: 79 tokens	min: 26 tokens mean: 229.31 tokens max: 256 tokens	min: 3 tokens mean: 165.35 tokens max: 256 tokens

Samples:

anchor	positive	negative
`How to use Hawaii’s local transportation?`	to watch it. it really is. yeah, i gotta say waikiki gets a bad rap for sometimes being a bit too crowded, but sunset, the vibe is great. there's often live music, everybody's just enjoying watching that sun go down, so it's a great place to start your vacation. you can even hop in one of the catamarans that has beach loading and go out sailing and watch the sunset that way. that's a great way to start. there you go. gosh, we're good. oh my gosh, you're lucky you found this channel. day number two, and guess what? you are up early, my friend, because you have jet lag. real bad. real bad, especially if you have kids with you. wow, good luck. so we are going to take advantage of that today, and you are going to head up to the north shore to go snorkeling at kualima cove, or just enjoy a nice beach day up there. kualima cove is at turtle bay resort, which is like five-star luxury resort that's been remodeled, but all beaches in hawaii are public, and this is a great place to go snorkeling...	and stay tuned because you're about to open the air traffic control tower after being closed for decades. click the link in the description below for more information on the pearl harbor aviation museum and the official place to purchase tickets. day number four. you have done a lot on this trip so far. oh my goodness, you must be tired. so let's hang out in waikiki for the day. there's plenty to do in waikiki. you can visit the different beaches in waikiki. there's eight of yes, wow. and each one is a little bit different. you can enjoy some shopping because there are world-class stores here, or you can hop in the water and try some surfing, which jordan really
`How to use Hawaii’s local transportation?`	so are you going to oahu, hawaii and you're either too cheap or broke to get a rental car for your time there? there's no judgment in that question because that was my mentality and you definitely can have an amazing time in oahu, hawaii even for like a week or so without any rental car whatsoever. i was there for a month and mostly got by without a rental car so i'm definitely very qualified to give you an amazing seven-day one-week itinerary for oahu, hawaii with no rental car. that being said, let's start off with day number zero or arrival day. so unless you're a very strong swimmer or taking a very fast speedboat, you'll be arriving at the daniel k. inouye international airport. probably a little bit jet lag. it's definitely a little bit tired. so let's just get to your hotel and check in. i do recommend staying in waikiki, which people might kind of roll their eyes or scoff at because it's very touristic. it's like a little bit of a concrete jungle in the middle of hawaii, built ...	warming up your taste buds then if you head to the link in the description below we have a full first time to hawaii article and that think of it like a starter yeah i like this i think she's hungry i had nothing to do with writing this. this is all her, but that's pretty good. that's a good analogy. hold on, but we have a main course. if you want more, head to the hawaiivacationguide.com slash itineraries, and that is our main course itineraries. but today, here in this video, we are going to give you a taste of what your first time to hawaii itinerary could look like, and i think this is
`How to use Hawaii’s local transportation?`	walk which is about 50 minutes. which, hey, do you want to walk there and burn out your legs a little bit? it's up to you, depends on how much energy you have to spare. either way, the hike isn't so difficult. it costs about $5 to enter, walking that is, which you'll be doing because you don't have a car of course, and i believe nowadays you have to either reserve a time slot or something like that. it's also very busy so the earlier you go the better. the hike itself is really cool it's not too difficult like i said you go through a little bit of a tunnel and at the end you will get some amazing views of waikiki, waikiki beach and some other parts of the island which you're probably super excited to explore which we'll definitely do on other days besides today. so once you finish hiking diamond head return back down the way you came it's an out and back trail and head to the nearby neighborhood called kaimuki. now kaimuki is a really cool and trendy sort of hip and sort of like a coll...	`slash itineraries because we have some awesome itineraries with great excursion days to take you all over the islands and you will have a great time would you call it the main course i would you call the main course and don't forget to check out the pearl harbor aviation museum because that will make your trip to honolulu all the more special thank you so much for watching thanks a lot.`

Loss: MultipleNegativesRankingLoss with these parameters:

{
    "scale": 20.0,
    "similarity_fct": "cos_sim",
    "gather_across_devices": false,
    "directions": [
        "query_to_doc"
    ],
    "partition_mode": "joint",
    "hardness_mode": null,
    "hardness_strength": 0.0
}

Training Hyperparameters

Non-Default Hyperparameters

num_train_epochs: 4
learning_rate: 2e-05
warmup_steps: 90
gradient_accumulation_steps: 4
fp16: True
dataloader_drop_last: True

All Hyperparameters

Click to expand

per_device_train_batch_size: 8
num_train_epochs: 4
max_steps: -1
learning_rate: 2e-05
lr_scheduler_type: linear
lr_scheduler_kwargs: None
warmup_steps: 90
optim: adamw_torch_fused
optim_args: None
weight_decay: 0.0
adam_beta1: 0.9
adam_beta2: 0.999
adam_epsilon: 1e-08
optim_target_modules: None
gradient_accumulation_steps: 4
average_tokens_across_devices: True
max_grad_norm: 1.0
label_smoothing_factor: 0.0
bf16: False
fp16: True
bf16_full_eval: False
fp16_full_eval: False
tf32: None
gradient_checkpointing: False
gradient_checkpointing_kwargs: None
torch_compile: False
torch_compile_backend: None
torch_compile_mode: None
use_liger_kernel: False
liger_kernel_config: None
use_cache: False
neftune_noise_alpha: None
torch_empty_cache_steps: None
auto_find_batch_size: False
log_on_each_node: True
logging_nan_inf_filter: True
include_num_input_tokens_seen: no
log_level: passive
log_level_replica: warning
disable_tqdm: False
project: huggingface
trackio_space_id: trackio
eval_strategy: no
per_device_eval_batch_size: 8
prediction_loss_only: True
eval_on_start: False
eval_do_concat_batches: True
eval_use_gather_object: False
eval_accumulation_steps: None
include_for_metrics: []
batch_eval_metrics: False
save_only_model: False
save_on_each_node: False
enable_jit_checkpoint: False
push_to_hub: False
hub_private_repo: None
hub_model_id: None
hub_strategy: every_save
hub_always_push: False
hub_revision: None
load_best_model_at_end: False
ignore_data_skip: False
restore_callback_states_from_checkpoint: False
full_determinism: False
seed: 42
data_seed: None
use_cpu: False
accelerator_config: {'split_batches': False, 'dispatch_batches': None, 'even_batches': True, 'use_seedable_sampler': True, 'non_blocking': False, 'gradient_accumulation_kwargs': None}
parallelism_config: None
dataloader_drop_last: True
dataloader_num_workers: 0
dataloader_pin_memory: True
dataloader_persistent_workers: False
dataloader_prefetch_factor: None
remove_unused_columns: True
label_names: None
train_sampling_strategy: random
length_column_name: length
ddp_find_unused_parameters: None
ddp_bucket_cap_mb: None
ddp_broadcast_buffers: False
ddp_backend: None
ddp_timeout: 1800
fsdp: []
fsdp_config: {'min_num_params': 0, 'xla': False, 'xla_fsdp_v2': False, 'xla_fsdp_grad_ckpt': False}
deepspeed: None
debug: []
skip_memory_metrics: True
do_predict: False
resume_from_checkpoint: None
warmup_ratio: None
local_rank: -1
prompts: None
batch_sampler: batch_sampler
multi_dataset_batch_sampler: proportional
router_mapping: {}
learning_rate_mapping: {}

Training Logs

Epoch	Step	Training Loss
0.2203	50	1.3365
0.4405	100	0.5632
0.6608	150	0.4383
0.8811	200	0.3778
1.1013	250	0.3399
1.3216	300	0.2640
1.5419	350	0.2533
1.7621	400	0.2160
1.9824	450	0.2270
2.2026	500	0.1697
2.4229	550	0.1717
2.6432	600	0.1603
2.8634	650	0.1552
3.0837	700	0.1374
3.3040	750	0.1195
3.5242	800	0.1085
3.7445	850	0.1114
3.9648	900	0.1298

Framework Versions

Python: 3.10.12
Sentence Transformers: 5.3.0
Transformers: 5.3.0
PyTorch: 2.10.0+cu128
Accelerate: 1.13.0
Datasets: 4.7.0
Tokenizers: 0.22.2

Citation

BibTeX

Sentence Transformers

@inproceedings{reimers-2019-sentence-bert,
    title = "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks",
    author = "Reimers, Nils and Gurevych, Iryna",
    booktitle = "Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing",
    month = "11",
    year = "2019",
    publisher = "Association for Computational Linguistics",
    url = "https://arxiv.org/abs/1908.10084",
}

MultipleNegativesRankingLoss

@misc{oord2019representationlearningcontrastivepredictive,
      title={Representation Learning with Contrastive Predictive Coding},
      author={Aaron van den Oord and Yazhe Li and Oriol Vinyals},
      year={2019},
      eprint={1807.03748},
      archivePrefix={arXiv},
      primaryClass={cs.LG},
      url={https://arxiv.org/abs/1807.03748},
}