rodrigomt

/

s2-pro-gguf

like 24

Hey! This was happening because on Windows large files weren’t being read properly, and on NVIDIA GPUs the model could sometimes miss the stop token and just keep generating until it hit the limit. That’s been fixed now just pull the latest commit and rebuild. I only tested the Windows fix on CPU, I don't have a Vulkan setup to test on Windows, so let me know if it works for you with -v 0!

the sound appeared, but now she says the same thing, regardless of the request

I'll download the updated models now and write later.

(base) PS R:\fishaudio_gguf\s2.cpp> build\Release\s2.exe -m models/s2-pro-q6_k.gguf -t models/tokenizer.json -text "Hello, this is a test." -v 0 -o test_fixed.wav
--- Pipeline Init ---
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
[Model] Reading metadata from models/s2-pro-q6_k.gguf
[Model] Architecture: fish-speech
[GGUF] fish-speech.context_length = 32768
[GGUF] fish-speech.vocab_size = 155776
[GGUF] fish-speech.embedding_length = 2560
[GGUF] fish-speech.feed_forward_length = 9728
[GGUF] fish-speech.block_count = 36
[GGUF] fish-speech.attention.head_count = 32
[GGUF] fish-speech.attention.head_count_kv = 8
[GGUF] fish-speech.rope.freq_base = 1e+06
[GGUF] fish-speech.attention.layer_norm_rms_epsilon = 1e-06
[GGUF] fish_speech.codebook_size = 4096
[GGUF] fish_speech.num_codebooks = 10
[GGUF] fish_speech.semantic_begin_id = 151678
[GGUF] fish_speech.semantic_end_id = 155773
[GGUF] fish_speech.tie_word_embeddings = true
[GGUF] fish_speech.attention_qk_norm = true
[GGUF] fish_speech.scale_codebook_embeddings = true
[GGUF] fish_speech.fast_context_length = 11
[GGUF] fish_speech.fast_embedding_length = 2560
[GGUF] fish_speech.fast_feed_forward_length = 9728
[GGUF] fish_speech.fast_block_count = 4
[GGUF] fish_speech.fast_head_count = 32
[GGUF] fish_speech.fast_head_count_kv = 8
[GGUF] fish_speech.fast_head_dim = 128
[GGUF] fish_speech.fast_rope_freq_base = 1e+06
[GGUF] fish_speech.fast_layer_norm_rms_eps = 1e-06
[GGUF] fish_speech.fast_attention_qk_norm = false
[GGUF] fish_speech.fast_project_in = false
[Model] Layers: 36, Dim: 2560, Vocab: 155776, head_count: 32, has_fast_decoder: 1
[Model] Weights loaded. Total tensors: 813
--- Pipeline Synthesize ---
Text: Hello, this is a test.
[Generate] Prefilling 27 tokens...
[Generate] Generating (max 512 tokens)...
[Generate] 50 / 512 tokens...
[Generate] Done: 78 frames generated.
Saved audio to: test_fixed.wav

Great, I updated the model, the sound appeared, but for some reason the model says a phrase before the text, and then voices the text itself.

(base) PS R:\fishaudio_gguf\s2.cpp> build\Release\s2.exe -m models/s2-pro-q8_02.gguf -t models/tokenizer2.json -text "Hello, this is a test. I'll download the updated models now and write later." -v 0 -o test_fixed4.wav
--- Pipeline Init ---
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
[Model] Reading metadata from models/s2-pro-q8_02.gguf
[Model] Architecture: fish-speech
[GGUF] fish-speech.context_length = 32768
[GGUF] fish-speech.vocab_size = 155776
[GGUF] fish-speech.embedding_length = 2560
[GGUF] fish-speech.feed_forward_length = 9728
[GGUF] fish-speech.block_count = 36
[GGUF] fish-speech.attention.head_count = 32
[GGUF] fish-speech.attention.head_count_kv = 8
[GGUF] fish-speech.rope.freq_base = 1e+06
[GGUF] fish-speech.attention.layer_norm_rms_epsilon = 1e-06
[GGUF] fish_speech.codebook_size = 4096
[GGUF] fish_speech.num_codebooks = 10
[GGUF] fish_speech.semantic_begin_id = 151678
[GGUF] fish_speech.semantic_end_id = 155773
[GGUF] fish_speech.tie_word_embeddings = true
[GGUF] fish_speech.attention_qk_norm = true
[GGUF] fish_speech.scale_codebook_embeddings = true
[GGUF] fish_speech.fast_context_length = 11
[GGUF] fish_speech.fast_embedding_length = 2560
[GGUF] fish_speech.fast_feed_forward_length = 9728
[GGUF] fish_speech.fast_block_count = 4
[GGUF] fish_speech.fast_head_count = 32
[GGUF] fish_speech.fast_head_count_kv = 8
[GGUF] fish_speech.fast_head_dim = 128
[GGUF] fish_speech.fast_rope_freq_base = 1e+06
[GGUF] fish_speech.fast_layer_norm_rms_eps = 1e-06
[GGUF] fish_speech.fast_attention_qk_norm = false
[GGUF] fish_speech.fast_project_in = false
[Model] Layers: 36, Dim: 2560, Vocab: 155776, head_count: 32, has_fast_decoder: 1
[Model] Weights loaded. Total tensors: 813
--- Pipeline Synthesize ---
Text: Hello, this is a test. I'll download the updated models now and write later.
[Generate] Prefilling 38 tokens...
[Generate] Generating (max 512 tokens)...
[Generate] 150 / 512 tokens...
[Generate] Done: 151 frames generated.
Saved audio to: test_fixed4.wav

Does it only support 8 languages, not 50 like the original? It doesn't speak any other languages.

Hi @Ayuy , thank you for testing and reporting the issue!

Regarding the phrase before the text "You are a helpful assistant," this issue was a lingering system warning in the code. It has already been identified and a fix will be released with the next open PR merge. After implementation, the model will only speak the text you provide.

Regarding language support, the model itself (S2 Pro) supports over 80 languages, as documented by Fish Audio. The "8 languages" label on the Hugging Face website only reflects Level 1 and Level 2 languages. Level 1: Japanese, English, Chinese. Level 2: Korean, Spanish, Portuguese, Arabic, Russian, French, German. The other languages ​​listed in the Fish Audio documentation should also work, but I haven't personally tested them. Feel free to try them out.

Strangely enough, the model speaks German but not Russian, and this is where the problems lie. She doesn't understand coding (probably): Text: ╧ЁштхЄ, ¤Єю ЄхёЄ. and generation does not work in this language (Russian)

(base) PS R:\fishaudio_gguf\s2.cpp> build\Release\s2.exe -m models/s2-pro-q8_02.gguf -t models/tokenizer2.json -text "Привет, это тест." -v 0 -o test_fixed12.wav
--- Pipeline Init ---
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
[Model] Reading metadata from models/s2-pro-q8_02.gguf
[Model] Architecture: fish-speech
[GGUF] fish-speech.context_length = 32768
[GGUF] fish-speech.vocab_size = 155776
[GGUF] fish-speech.embedding_length = 2560
[GGUF] fish-speech.feed_forward_length = 9728
[GGUF] fish-speech.block_count = 36
[GGUF] fish-speech.attention.head_count = 32
[GGUF] fish-speech.attention.head_count_kv = 8
[GGUF] fish-speech.rope.freq_base = 1e+06
[GGUF] fish-speech.attention.layer_norm_rms_epsilon = 1e-06
[GGUF] fish_speech.codebook_size = 4096
[GGUF] fish_speech.num_codebooks = 10
[GGUF] fish_speech.semantic_begin_id = 151678
[GGUF] fish_speech.semantic_end_id = 155773
[GGUF] fish_speech.tie_word_embeddings = true
[GGUF] fish_speech.attention_qk_norm = true
[GGUF] fish_speech.scale_codebook_embeddings = true
[GGUF] fish_speech.fast_context_length = 11
[GGUF] fish_speech.fast_embedding_length = 2560
[GGUF] fish_speech.fast_feed_forward_length = 9728
[GGUF] fish_speech.fast_block_count = 4
[GGUF] fish_speech.fast_head_count = 32
[GGUF] fish_speech.fast_head_count_kv = 8
[GGUF] fish_speech.fast_head_dim = 128
[GGUF] fish_speech.fast_rope_freq_base = 1e+06
[GGUF] fish_speech.fast_layer_norm_rms_eps = 1e-06
[GGUF] fish_speech.fast_attention_qk_norm = false
[GGUF] fish_speech.fast_project_in = false
[Model] Layers: 36, Dim: 2560, Vocab: 155776, head_count: 32, has_fast_decoder: 1
[Model] Weights loaded. Total tensors: 813
--- Pipeline Synthesize ---
Text: ╧ЁштхЄ, ¤Єю ЄхёЄ.
[Generate] Prefilling 37 tokens...
[Generate] Generating (max 512 tokens)...
[Generate] 50 / 512 tokens...
[Generate] Done: 86 frames generated.
Saved audio to: test_fixed12.wav

This is a file with the working Russian language (s2.cpp\src\main.cpp)

#include "s2_pipeline.h"
#include
#include
#include

#ifdef _WIN32
#include <windows.h>
#include <shellapi.h>
#endif

void print_uso() {
std::cout << "Usage: s2 [options]\n";
std::cout << "Options:\n";
std::cout << " -m, --model

Path to reference audio for cloning\n";
std::cout << " -pt, --prompt-text Text of the reference audio for cloning\n";
std::cout << " -o, --output

if (argc < 2) {
    print_uso();
    return 1;
}

s2::PipelineParams params;
// Default paths
params.model_path = "model.gguf";
params.tokenizer_path = "tokenizer.json";
params.output_path = "out.wav";
params.text = "Hello world";
params.vulkan_device = -1;

for (int i = 1; i < argc; ++i) {
    std::string arg = argv[i];
    if (arg == "-m" || arg == "--model") {
        if (i + 1 < argc) params.model_path = argv[++i];
    } else if (arg == "-t" || arg == "--tokenizer") {
        if (i + 1 < argc) params.tokenizer_path = argv[++i];
    } else if (arg == "-text") {
        if (i + 1 < argc) params.text = argv[++i];
    } else if (arg == "-pa" || arg == "--prompt-audio") {
        if (i + 1 < argc) params.prompt_audio_path = argv[++i];
    } else if (arg == "-pt" || arg == "--prompt-text") {
        if (i + 1 < argc) params.prompt_text = argv[++i];
    } else if (arg == "-o" || arg == "--output") {
        if (i + 1 < argc) params.output_path = argv[++i];
    } else if (arg == "-v" || arg == "--vulkan") {
        if (i + 1 < argc) params.vulkan_device = std::stoi(argv[++i]);
    } else if (arg == "-threads") {
        if (i + 1 < argc) params.gen.n_threads = std::stoi(argv[++i]);
    } else if (arg == "-max-tokens") {
        if (i + 1 < argc) params.gen.max_new_tokens = std::stoi(argv[++i]);
    } else if (arg == "-temp") {
        if (i + 1 < argc) params.gen.temperature = std::stof(argv[++i]);
    } else if (arg == "-top-p") {
        if (i + 1 < argc) params.gen.top_p = std::stof(argv[++i]);
    } else if (arg == "-top-k") {
        if (i + 1 < argc) params.gen.top_k = std::stoi(argv[++i]);
    } else if (arg == "-h" || arg == "--help") {
        print_uso();
        return 0;
    }
}

// If tokenizer path was not explicitly set, search for tokenizer.json in:
//   1. Same directory as the model file
//   2. Parent directory of the model file
//   3. Working directory (default fallback)
if (params.tokenizer_path == "tokenizer.json") {
    std::string model_path = params.model_path;
    size_t slash = model_path.find_last_of("/\\");
    if (slash != std::string::npos) {
        std::string model_dir = model_path.substr(0, slash + 1);
        // Check same dir as model
        std::string candidate = model_dir + "tokenizer.json";
        if (FILE * f = std::fopen(candidate.c_str(), "r")) {
            std::fclose(f);
            params.tokenizer_path = candidate;
        } else {
            // Check parent dir
            size_t parent_slash = model_dir.find_last_of("/\\", slash - 1);
            if (parent_slash != std::string::npos) {
                candidate = model_dir.substr(0, parent_slash + 1) + "tokenizer.json";
                if (FILE * f2 = std::fopen(candidate.c_str(), "r")) {
                    std::fclose(f2);
                    params.tokenizer_path = candidate;
                }
            }
        }
    }
}

s2::Pipeline pipeline;
if (!pipeline.init(params)) {
    std::cerr << "Pipeline initialization failed." << std::endl;
    return 1;
}

if (!pipeline.synthesize(params)) {
    std::cerr << "Synthesis failed." << std::endl;
    return 1;
}

return 0;

and how to insert these tags?: [pause] [emphasis] [laughing] [inhale] [chuckle] [tsk] [singing] [excited] [laughing tone] [interrupting] [chuckling] [excited tone] [volume up] [echo] [angry] [low volume] [sigh] [low voice] [whisper] [screaming] [shouting] [loud] [surprised] [short pause] [exhale] [delight] [panting] [audience laughter] [with strong accent] [volume down] [clearing throat] [sad] [moaning] [shocked]

@Ayuy ,
I think this Russian issue is pretty clearly a Windows encoding problem, especially in the text I/O pipeline. That garbled text in the log (╧ЁштхЄ, ¤Єю ЄхёЄ.) is a classic symptom of UTF-8 bytes being misread under the wrong code page. Since your main.cpp already forces UTF-8 and rebuilds argv using GetCommandLineW(), that lines up perfectly with what's happening and it also explains why German works fine while Russian breaks: Latin characters have enough overlap with ASCII in Western code pages to survive, but Cyrillic has zero overlap and gets completely mangled.

Your fix with SetConsoleOutputCP(CP_UTF8) + WideCharToMultiByte(CP_UTF8) is definitely on the right track. My guess is the real problem is happening further down the pipeline somewhere like a cout / printf call or a conversion step before tokenization. If the text hits a non-UTF-8 code page at any point in that chain, it's already corrupted by the time it reaches the model.

I'll dig into this more carefully to make sure everything is truly end-to-end UTF-8, especially console output and tokenizer input. The model itself should handle Russian just fine as long as the encoding is consistent throughout.

As for tags like [pause], [laughs], [angry], etc.: from what I've seen, S2 Pro does support that kind of bracket hint (per Fish Audio's docs), but I wouldn't rely on it too heavily just yet. I've seen people on Hugging Face mention it works maybe 1 in 10 times with cloned voices, and even with a standard voice it's pretty inconsistent. So I'd treat it as "works occasionally" for now, especially in voice cloning scenarios.

I'm also running tests across other languages to validate everything more thoroughly. Once I wrap that up and have the "Added experimental CUDA support #1" PR ready, I'll post an update here.

the longer the output, the more the voice fades out (the voice becomes quieter)

This generation consumes 7 GB of VRAM and 30 GB of RAM.

R:\fishaudio_gguf\s2.cpp>build\Release\s2.exe -m models/s2-pro-q8_02.gguf -t models/tokenizer2.json -pa x_var.wav -pt "Диалог между людьми это сложный но удивительный процесс мы не просто обмениваемся информацией, мы передаём эмоции, тонкие оттенки смысла и даже то, что остаётся за словами, именно это делает общение живым." -text "One fine evening, a no less fine government clerk called Ivan Dmitritch Tchervyakov was sitting in the second row of the stalls, gazing through an opera glass at the Cloches de Corneville. He gazed and felt at the acme of bliss. But suddenly. . . . In stories one so often meets with this But suddenly. The authors are right: life is so full of surprises! But suddenly his face puckered up, his eyes disappeared, his breathing was arrested . . . he took the opera glass from his eyes, bent over and . . . "Aptchee!!" he sneezed as you perceive. It is not reprehensible for anyone to sneeze anywhere. Peasants sneeze and so do police superintendents, and sometimes even privy councillors. All men sneeze. Tchervyakov was not in the least confused, he wiped his face with his handkerchief, and like a polite man, looked round to see whether he had disturbed any one by his sneezing. But then he was overcome with confusion. He saw that an old gentleman sitting in front of him in the first row of the stalls was carefully wiping his bald head and his neck with his glove and muttering something to himself. In the old gentleman, Tchervyakov recognised Brizzhalov, a civilian general serving in the Department of Transport. Something seemed to give way in Tchervyakov's stomach. Seeing nothing and hearing nothing he reeled to the door, went out into the street, and went staggering along...Reaching home mechanically, without taking off his uniform, he lay down on the sofa and died." -v 0 -max-tokens 2000 -temp 0.7 -top-p 0.7 -top-k 30 -o clone_result2000_x.wav
--- Pipeline Init ---
ggml_vulkan: Found 1 Vulkan devices:
ggml_vulkan: 0 = NVIDIA GeForce RTX 2060 (NVIDIA) | uma: 0 | fp16: 1 | bf16: 1 | warp size: 32 | shared memory: 49152 | int dot: 1 | matrix cores: NV_coopmat2
[Model] Reading metadata from models/s2-pro-q8_02.gguf
[Model] Architecture: fish-speech
[GGUF] fish-speech.context_length = 32768
[GGUF] fish-speech.vocab_size = 155776
[GGUF] fish-speech.embedding_length = 2560
[GGUF] fish-speech.feed_forward_length = 9728
[GGUF] fish-speech.block_count = 36
[GGUF] fish-speech.attention.head_count = 32
[GGUF] fish-speech.attention.head_count_kv = 8
[GGUF] fish-speech.rope.freq_base = 1e+06
[GGUF] fish-speech.attention.layer_norm_rms_epsilon = 1e-06
[GGUF] fish_speech.codebook_size = 4096
[GGUF] fish_speech.num_codebooks = 10
[GGUF] fish_speech.semantic_begin_id = 151678
[GGUF] fish_speech.semantic_end_id = 155773
[GGUF] fish_speech.tie_word_embeddings = true
[GGUF] fish_speech.attention_qk_norm = true
[GGUF] fish_speech.scale_codebook_embeddings = true
[GGUF] fish_speech.fast_context_length = 11
[GGUF] fish_speech.fast_embedding_length = 2560
[GGUF] fish_speech.fast_feed_forward_length = 9728
[GGUF] fish_speech.fast_block_count = 4
[GGUF] fish_speech.fast_head_count = 32
[GGUF] fish_speech.fast_head_count_kv = 8
[GGUF] fish_speech.fast_head_dim = 128
[GGUF] fish_speech.fast_rope_freq_base = 1e+06
[GGUF] fish_speech.fast_layer_norm_rms_eps = 1e-06
[GGUF] fish_speech.fast_attention_qk_norm = false
[GGUF] fish_speech.fast_project_in = false
[Model] Layers: 36, Dim: 2560, Vocab: 155776, head_count: 32, has_fast_decoder: 1
[Model] Weights loaded. Total tensors: 813
--- Pipeline Synthesize ---
Text: One fine evening, a no less fine government clerk called Ivan Dmitritch Tchervyakov was sitting in the second row of the stalls, gazing through an opera glass at the Cloches de Corneville. He gazed and felt at the acme of bliss. But suddenly. . . . In stories one so often meets with this But suddenly. The authors are right: life is so full of surprises! But suddenly his face puckered up, his eyes disappeared, his breathing was arrested . . . he took the opera glass from his eyes, bent over and . . . Aptchee!! he sneezed as you perceive. It is not reprehensible for anyone to sneeze anywhere. Peasants sneeze and so do police superintendents, and sometimes even privy councillors. All men sneeze. Tchervyakov was not in the least confused, he wiped his face with his handkerchief, and like a polite man, looked round to see whether he had disturbed any one by his sneezing. But then he was overcome with confusion. He saw that an old gentleman sitting in front of him in the first row of the stalls was carefully wiping his bald head and his neck with his glove and muttering something to himself. In the old gentleman, Tchervyakov recognised Brizzhalov, a civilian general serving in the Department of Transport. Something seemed to give way in Tchervyakov's stomach. Seeing nothing and hearing nothing he reeled to the door, went out into the street, and went staggering along...Reaching home mechanically, without taking off his uniform, he lay down on the sofa and died.
Loading reference audio: x_var.wav
[Generate] Prefilling 821 tokens...
[Generate] Generating (max 2000 tokens)...
[Generate] 2000 / 2000 tokens...
[Generate] Done: 2000 frames generated.
Saved audio to: clone_result2000_x.wav

I'm not sure, but it seems to work. The stress (stress in a word) is placed like this: Соски^ ; повали^в.

Вставьте символ ^ непосредственно перед ударной гласной в слове. Модель будет произносить эту гласную с усилением, а остальную часть слова — без изменения.

Insert the ^ symbol immediately before a stressed vowel in a word. The model will pronounce this vowel with emphasis and the rest of the word without change.

Can you re^cord this pre^sent? I understa^nd it's important.