Skip to content

Issues: open-compass/VLMEvalKit

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

45 Open 217 Closed
45 Open 217 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Assignee
Filter by who’s assigned
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Issues list

本地评估模型相关疑问 awaiting confirm
#693 opened Dec 24, 2024 by Leke-G
@FangXinyu-0913
2
ChartQA/DocVQA 分数异常
#683 opened Dec 23, 2024 by OrlandoBloom16
@PhoenixZ810
1
MME-realworld benchmark如何设置local path？ awaiting confirm
#676 opened Dec 19, 2024 by ZihaoZheng98
2
AUTO_SPLIT=1 如何搭配CUDA_VISIBLE_DEVICES=使用
#674 opened Dec 19, 2024 by ZihaoZheng98
@FangXinyu-0913
2
The image quality in VLMEvalKit is compressed and lower than that provided by Benchmark itself.
#665 opened Dec 13, 2024 by Sync-yxh
1
Local judge LLM?
#664 opened Dec 12, 2024 by lyzhongcrd
3
报错
#661 opened Dec 12, 2024 by 0704highlight
@PhoenixZ810
1
AutoScoringJudge文件
#651 opened Dec 5, 2024 by wksuixin
@PhoenixZ810
1
LLAVA-one-vision MMMU 性能评测差别很大
#645 opened Dec 4, 2024 by linxid
2
用Chartqa测评LLama3.2-11b-vision时，结果全部为0
#644 opened Dec 4, 2024 by GoogleAlphaZero
@PhoenixZ810
1
Does VLMEvalKit support Paged Attention evaluation?
#641 opened Dec 2, 2024 by weiaicunzai
@PhoenixZ810
1
Discrepancy in MM Math Evaluation Results: Possible Issue with Answer Extraction in VLMEvalkit
#638 opened Dec 1, 2024 by mantle2048
6
The evaluation device_map setting is not optimized for quantized models using the ExLlamaV2 backend.
#629 opened Nov 26, 2024 by WeiweiZhang1
Ovis1.5-Llama3-8B在Hallusion Bench上的指标和榜单上的指标差距过大
#595 opened Nov 13, 2024 by LIRENDA621
1
How soon is the 'News' section in the README updated after a new Benchmark or Model is added?
#590 opened Nov 10, 2024 by Baiqi-Li
1
Evaluations get stuck on the last 4 questions awaiting confirm
#588 opened Nov 9, 2024 by XuGW-Kevin
4
ChartQA eval bug
#584 opened Nov 7, 2024 by lemonliu1992
Corrupted MathVista testmini data WIP
#578 opened Nov 5, 2024 by LeoDu0314
1
AI2d gpt和claude3.5官方分数非常高 awaiting confirm
#577 opened Nov 5, 2024 by Violettttee
3
Qwen-VL-Max-0809 MME中celebrity测出来和榜单结果差距有10分左右 awaiting confirm
#571 opened Nov 2, 2024 by lihua8848
7
LLaVA v1.5 7b very low performance on TextVQA
#554 opened Oct 28, 2024 by wufeim
The result of gpt4o on MathVista is 61.6, which is lower than the 62.7 on the list. awaiting confirm
#553 opened Oct 28, 2024 by kkk123698745
3
How to eval with multigpu when using device_map='auto'
#528 opened Oct 18, 2024 by qianwangn
@FangXinyu-0913
1
Different evaluate result between different VLMEvalKit version
#523 opened Oct 16, 2024 by terry-for-github
@kennymckormick
5
Are images compressed in tsv files?
#517 opened Oct 12, 2024 by ywwynm
3
ProTip! Type g p on any issue or pull request to go back to the pull request listing page.