Evaluation results of Gemini-2.5-Pro

[jwwang424]

Thanks for the interesting work!

I tested gemini-2.5-pro-preview-05-06 on the released Video-Holmes-test benchmark, and got the overall avg score of 62.3, which seems differ from your reported Gemini-2.5-Pro result (51.3). I just the raw video (with audio) to gemini's api and use your provided prompt:

Based on the given video, reason and answer the single-choice question. Provide your reasoning between the <think> and </think> tags, and then give your final answer between the <answer> and </answer> tags. The question is: {question}. The options are: {options}. Your answer:

What could be the reasons for the differences in test results?

[donahowe]

Hi, thank you for sharing this information.

I believe the gap may be due to differences in model versions—we are currently using gemini-2.5-pro-preview-03-25. We will update to the latest version and refresh our leaderboard accordingly. You can also review and compare our results here:

Results-gemini-2.5-pro.json

[jwwang424]

Thanks for the quick reply. To my knowledge, this two versions of gemini-2.5-pro should not have such big capacity gap. After checking the json file your provided, I think the most likely reason should be in response parsing, as there are many "WRONG" in "Predict Answer" and "Thinking" in the json file.

Actually I also found that gemini-2.5-pro often failed to follow the output format of <think>*</think><answer>*<answer>, so I prompt it to summarize the answer in Final answer: X for easy prediction extraction. My updated prompt:

Based on the given video, reason and answer the single-choice question. Provide your reasoning between the <think> and </think> tags, and then give your final answer between the <answer> and </answer> tags. The question is: {question}. The options are: {options}.

Output the final answer in the format "Final answer: X", where X is the correct option's letter.

Here is my code for prediction extraction:

TEMPLATES = [['{}\.', ' {} -', '{}。', '\'{}\'', '\"{}\"'], ['{} ', '{}_', '({})', '（{}）']]
CHOICES = ['A', 'B', 'C', 'D', 'E', 'F']
COMBINES = list()
for temp in TEMPLATES:
    COMBINES.append([[choice, t.format(choice)] for t in temp for choice in CHOICES])

def remove_bold(text):
    # rm **text** 或 __text__
    text = re.sub(r'\*\*(.*?)\*\*', r'\1', text)
    text = re.sub(r'__(.*?)__', r'\1', text)
    return text

def extract_boxed_answer(text):
    match = re.findall(r'\\boxed\{(.+?)\}', text)
    if len(match):
        return match[-1]
    else:
        return text

def get_last_predict(text):
    answer_prefixes = [
        "The best answer is",
        "The correct answer is",
        "The answer is",
        "The answer",
        "The best option is",
        "The correct option is",
        "Best answer:",
        "Best option:",
        "Answer:",
        "Option:",
        "The correct answer",
        "The correct option",
        "the best answer is",
        "the correct answer is",
        "the answer is",
        "the answer",
        "the best option is",
        "the correct option is",
        "best answer:",
        "best option:",
        "answer:",
        "option:",
        "the correct answer",
        "the correct option",
        'final answer',
        'Final answer',
    ]
    answer_str = text
    for prefix in answer_prefixes:
        last_answer_pos = answer_str.rfind(prefix)
        if last_answer_pos != -1:
            # Extract the string after "Answer:"
            answer_str = answer_str[last_answer_pos + len(prefix):].strip()
    return answer_str

def extract_option(predict):
    predict = re.sub(r'\s+', ' ', predict)
    predict = remove_bold(predict)
    predict = get_last_predict(predict)
    predict = extract_boxed_answer(predict)
    bais = 10
    fixed_predict = predict
    if len(predict) == 1: # output is A, B, C, D...
        pass
    else:
        count = 0
        min_index = [len(predict) + bais] * len(COMBINES)
        tmp_ans = [''] * len(COMBINES)
        for j, comb in enumerate(COMBINES):
            for cb in comb:
                if comb[1] in [' A ', ' A', 'A ']:
                    continue
                ret = re.search(cb[1], predict)
                if ret is not None:
                    count += 1
                    #print(min_index, ret.span()[0])
                    if min_index[j] > ret.span()[0]:
                        min_index[j] = ret.span()[0]
                        #print(comb[0])
                        tmp_ans[j] = cb[0]
                else:
                    pass
        for k, item in enumerate(min_index):
            if item < len(predict) + bais:
                fixed_predict = tmp_ans[k]
                break
    return fixed_predict

[donahowe]

Great, your observation is correct. We recommend using the official evaluation code, as its logic is consistent with the existing video reasoning benchmark: we consider the ability of models to follow instructions as a fundamental requirement. If a model fails to produce outputs according to the instructions, it will be directly marked as incorrect.