IntJudge-8B

Introduction

We're excited to unveil IntJudge-v1, the public beta version of our IntJudge model.

IntJudge is a judge model for evaluating open-ended multimodal generation methods. We use the Dev Set of OpenING to train the proposed IntJudge. Trained with a novel data pipeline, our IntJudge achieves an agreement rate of 82.42% with human judgments, outperforming GPT-based evaluators by 11.34%.

The IntJudge-8B is built based on the Qwen2-VL-7B-Instruct.

Requirements

We advise you to build from source with command pip install git+https://github.com/huggingface/transformers, or you might encounter the KeyError.

You should also install qwen-vl-utils.

pip install qwen-vl-utils

The code of using IntJudge for pairwise evaluation on interleaved image-text generation can be found at https://github.com/LanceZPF/OpenING.

Citation

If you find our work helpful, feel free to give us a cite.

@misc{zhou2024GATE,
      title={GATE OpenING: A Comprehensive Benchmark for Judging Open-ended Interleaved Image-Text Generation}, 
      author={Pengfei Zhou, Xiaopeng Peng, Jiajun Song, Chuanhao Li, Zhaopan Xu, Yue Yang, Ziyao Guo, Hao Zhang, Yuqi Lin, Yefei He, Lirui Zhao, Shuo Liu, Tianhua Li, Yuxuan Xie, Xiaojun Chang, Yu Qiao, Wenqi Shao, and Kaipeng Zhang},
      year={2024},
      eprint={2411.18499},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}