EgoCoT-Bench

LLM-Assisted Question Generation

Candidate question-answer pairs are generated from verified spatio-temporal scene graph (STSG) evidence rather than free-form video descriptions. For each subtask, we first traverse task-specific evidence paths to identify grounded targets, associated temporal and spatial cues, interaction relations, and local action history. We then use an LLM to render these verified structural facts into a multiple-choice question, answer options, and step-by-step rationale in a fixed JSON format.

Across subtasks, the generation prompts follow a shared principle: each sample must be directly answerable from the provided structured evidence, contain a unique and unambiguous answer, include plausible distractors, and remain traceable to explicit temporal, spatial, and interaction cues. In this way, the LLM serves as a renderer of verified structural evidence rather than a source of unconstrained content invention.

Generic generation template:

You are generating benchmark QA for egocentric video understanding.

    Generate ONE multiple-choice QA pair grounded only in the provided structured egocentric evidence.

    General requirements:
    1. The question must be answerable using only the provided evidence.
    2. The answer must be unique and unambiguous.
    3. Distractors should be plausible but contradicted by the evidence.
    4. The reasoning must be step-by-step and grounded in timestamps, object states,
      spatial localization, and interaction relations when available.
    5. Return valid JSON only.

    Output format:
    {
      "question": "...",
      "choices": {"A": "...", "B": "...", "C": "...", "D": "..."},
      "answer": "A/B/C/D",
      "reasoning": {"step1": "...", "step2": "...", "step3": "..."}
    }

LLM Judge for Reasoning Quality

We use a separate LLM judge to score model-generated reasoning on a strict 0–5 scale. The judge takes as input the question, the ground-truth answer, the annotated reference reasoning, the predicted answer, and the predicted reasoning. It evaluates the predicted reasoning in terms of logical soundness, coherence, and consistency with the ground-truth answer, and returns a JSON object containing an integer score and a brief explanation.

Empty or missing reasoning outputs are assigned a score of 0. In implementation, we use deterministic decoding with temperature set to 0.0 and require strict JSON output.

Judge prompt:

You are an expert evaluator for video question answering reasoning.

    Given:
    - a question,
    - the ground-truth answer,
    - the annotated reference reasoning,
    - the model-predicted answer, and
    - the model-predicted reasoning,

    score the predicted reasoning on a strict 0–5 scale.

    Evaluate the reasoning based on:
    1. logical soundness,
    2. coherence,
    3. consistency with the ground-truth answer, and
    4. whether the reasoning supports the predicted answer in a faithful and non-contradictory manner.

    Scoring rubric:
    5: logically sound, coherent, and fully supports the correct answer.
    4: mostly correct and coherent, with only minor redundancy or phrasing issues.
    3: largely reasonable, but contains minor mistakes or incomplete support.
    2: partially relevant, but includes major logical flaws or unsupported steps.
    1: mostly incoherent, weakly relevant, or poorly connected to the answer.
    0: empty, irrelevant, or fundamentally incorrect.

    Return strict JSON:
    {
      "reasoning_score": <0-5 integer>,
      "analysis": "<brief explanation>"
    }