EAS Resilience logo EAS Resilience

RUNBOOK

How to run Resilience EAS

EAS Resilience Evaluation is based on Habitat-Sim 3.0; its dataset and benchmark formats follow the PARTNR Planner. A separate LLM Judge acts as the critic, while stage baselines are calculated from EAS execution experience.

LinuxBashCondaHydra
On this page

01. Environment installation

Refer to the installation instructions from both projects: PARTNR Planner Installation and Habitat-Sim Installation. The commands below retain the PARTNR-compatible pinned versions.

  1. Create the environment

    conda create -n habitat-llm python=3.9.2 cmake=3.14.0 -y
conda activate habitat-llm

  2. Install the simulator stack

    Match the CUDA build to the target machine. PARTNR currently pins PyTorch 2.4.1 and Habitat-Sim 0.3.3.

    conda install pytorch==2.4.1 torchvision==0.19.1 \
  torchaudio==2.4.1 pytorch-cuda=12.4 -c pytorch -c nvidia -y
conda install habitat-sim=0.3.3 withbullet headless \
  -c conda-forge -c aihabitat -y

  3. Install the project

    Run this inside the full PARTNR-compatible Resilience EAS checkout containing third_party/ and requirements.txt.

    git submodule sync
git submodule update --init --recursive
pip install -e ./third_party/habitat-lab/habitat-lab
pip install -e ./third_party/habitat-lab/habitat-baselines
pip install -e ./third_party/transformers-CFG
pip install -r requirements.txt -e .

  4. Download and verify assets

    Use the PARTNR guide to download HSSD, PARTNR episodes, OVMM objects, and Habitat rearrangement assets.

    test -f data/datasets/partnr_episodes/v0_0/val_mini.json.gz
test -f data/datasets/partnr_episodes/v0_0/val.json.gz
test -d data/hssd-hab
test -d data/objects_ovmm
test -f evaluation/perturbation_banks/instruction_context_v6/manifest.json

    Expected: all five checks exit with status 0. The frozen stress bank is required only for Resilience Case Analysis.

  5. Run the upstream smoke test

    The official PARTNR Quickstart includes a heuristic baseline that does not require an LLM.

    python -m habitat_llm.examples.planner_demo \
  --config-name baselines/heuristic_full_obs.yaml \
  habitat.dataset.data_path="data/datasets/partnr_episodes/v0_0/val_mini.json.gz"

02. Planner runtime configuration

  1. Set planner variables

    cd /path/to/resilience-eas
export DATASET=data/datasets/partnr_episodes/v0_0/val_mini.json.gz
export PARTNR_BENCHMARK=data/datasets/partnr_episodes/v0_0/val.json.gz
export SERVED_MODEL_NAME="your-served-model-name"
# Example (Qwen): SERVED_MODEL_NAME="Qwen3.5-9B-YOYO-Instruct"
export QWEN35_SERVED_MODEL="$SERVED_MODEL_NAME"  # config compatibility

  2. Check the Hydra config chain

    test -f habitat_llm/conf/baselines/qwen35_centralized_zero_shot_react_summary_vllm.yaml
test -f habitat_llm/conf/baselines/qwen35_centralized_zero_shot_react_summary_vllm_state.yaml
test -f habitat_llm/conf/evaluation/resilience_config.yaml
test -f habitat_llm/conf/experiments/resilience_joint_qwen35.yaml
test -f habitat_llm/conf/instruct/qwen_few_shot_centralized_motoronly_new.yaml

  3. Start the vLLM server

    Run the generic OpenAI-compatible server in a dedicated GPU terminal. See the vLLM execution notes.

    conda activate vllm
export MODEL_PATH="/path/to/your/model"
export SERVED_MODEL_NAME="your-served-model-name"
# Example (Qwen): MODEL_PATH="/path/to/Qwen3.5-9B-YOYO-Instruct"
# Example (Qwen): SERVED_MODEL_NAME="Qwen3.5-9B-YOYO-Instruct"
python -m vllm.entrypoints.openai.api_server \
  --model "$MODEL_PATH" --served-model-name "$SERVED_MODEL_NAME" \
  --host 127.0.0.1 --port 8000 --trust-remote-code

  4. Check the planner service

    python -c "import hydra, habitat, habitat_sim, habitat_llm"
curl -fsS http://127.0.0.1:8000/v1/completions \
  -H 'Content-Type: application/json' \
  --data "{\"model\":\"$SERVED_MODEL_NAME\",\"prompt\":\"ping\",\"max_tokens\":1}" >/dev/null

    Expected: the Habitat imports succeed and the local planner completion request exits with status 0.

03. How to run EAS Case Analysis

Run one ordinary PARTNR episode. This validates the simulator, planner, task result, and trace path.

  1. Run episode 78

    EAS_RUN="outputs/habitat_llm/eas-78-$(date +%Y%m%d-%H%M%S)"
python -m habitat_llm.examples.planner_demo_mp_new \
  --config-name baselines/qwen35_centralized_zero_shot_react_summary_vllm \
  hydra.run.dir="$EAS_RUN" habitat.dataset.data_path="$DATASET" \
  '++episode_ids=[78]'

    Config: baselines/qwen35_centralized_zero_shot_react_summary_vllm.yaml → evaluation/centralized_evaluation_runner_motortoolsonly_multiagent.yaml → instruct/qwen_few_shot_centralized_motoronly_new.yaml.

  2. Check the result

    test -s "$EAS_RUN/results/episode_result_log.csv"
test -s "$EAS_RUN/results/$(basename "$DATASET")/stats/78.json"

    Pass: both files exist. Task failure is still a valid measured outcome.

04. How to run Resilience EAS Case Analysis

Run the frozen 10 episodes × 3 seeds × 6 intensity levels (λ = 0, .2, .4, .6, .8, 1): 180 rollout cells. Complete the separate LLM Judge preflight first. See the Case Analysis execution notes.

  1. Audit the frozen stress bank

    python -m dataset_generation.benchmark_generation.audit_perturbation_bank \
  --bank-path evaluation/perturbation_banks/instruction_context_v6 \
  --dataset-path "$DATASET" \
  --report-path evaluation/perturbation_bank_audit_v6.json

    Pass: status=complete, 30 valid entries, 180 resolved levels, formal_valid=true, and no errors.

  2. Run the 10 × 3 × 6 grid

    RES_RUN="outputs/habitat_llm/resilience-case-$(date +%Y%m%d-%H%M%S)"
python -u -m habitat_llm.examples.planner_demo_expq1 \
  --config-name experiments/resilience_joint_qwen35 \
  hydra.run.dir="$RES_RUN" habitat.dataset.data_path="$DATASET"

    Config: experiments/resilience_joint_qwen35.yaml → baselines/qwen35_centralized_zero_shot_react_summary_vllm_state.yaml + evaluation/resilience_config.yaml + instruct/qwen_few_shot_centralized_motoronly_new.yaml.

  3. Check 180 rollout cells

    CASE="$RES_RUN/results/resilience_joint_qwen35/joint_resilience_evaluation"
python - "$CASE/raw_rollouts.csv" <<'PY'
import csv, sys
rows = list(csv.DictReader(open(sys.argv[1])))
cols = ("episode_id", "perturbation_seed", "perturbation_type", "perturbation_intensity")
cells = {tuple(row[col] for col in cols) for row in rows}
assert len(rows) == len(cells) == 180
print("180/180 unique cells")
PY
python -c 'import json,sys; assert json.load(open(sys.argv[1]))["subexperiments"][0]["row_count"]==180' \
  "$RES_RUN/results/resilience_joint_qwen35/suite_manifest.json"

    Expected: 180/180 unique cells and a 180-row suite manifest.

  4. Check lifecycle workers

    python - "$CASE/conditions/default/runner/stress_sweep" <<'PY'
import glob, json, sys
paths = glob.glob(sys.argv[1] + "/**/execution_manifest.json", recursive=True)
counts = [b["status_counts"] for path in paths for b in json.load(open(path))["batches"]]
assert paths and all(x["failed"] == x["not_started"] == 0 for x in counts)
print(f"{len(paths)} worker manifest(s) complete")
PY

    Expected: no failed or unstarted workers. Calibration and critic-training workers may make the completed count greater than 180.

05. How to run Resilience EAS Benchmark

Run the project-defined first 400 entries of the installed PARTNR v0_0/val split. This is an upstream task benchmark, not the 10 × 3 × 6 resilience grid. The linked PARTNR execution notes provide the execution pattern.

  1. Check the source split

    python -c 'import gzip,json,sys; n=len(json.load(gzip.open(sys.argv[1],"rt"))["episodes"]); print(n); assert n>=400' \
  "$PARTNR_BENCHMARK"

    Pass: the installed source split contains at least 400 entries.

  2. Run the first 400 entries (indices 0–399)

    BENCH_RUN="outputs/habitat_llm/benchmark-400-$(date +%Y%m%d-%H%M%S)"
EPISODES="[$(seq -s, 0 399)]"
python -m habitat_llm.examples.planner_demo_mp_new \
  --config-name baselines/qwen35_centralized_zero_shot_react_summary_vllm \
  hydra.run.dir="$BENCH_RUN" habitat.dataset.data_path="$PARTNR_BENCHMARK" \
  num_proc=4 "++episode_indices=$EPISODES"

    Config: the same baseline, evaluation runner, and few-shot instruction chain used by EAS Case Analysis.

    Default: num_proc=4 enables isolated episode workers; admission remains capped at one worker with a 48 GiB available-memory gate. Raise the cap only after a target-host memory smoke test.

  3. Check 400 episode results

    python -c 'import csv,json,sys; r=list(csv.DictReader(open(sys.argv[1]))); m=json.load(open(sys.argv[2])); assert len(r)==400==len({x["episode_id"] for x in r}) and m["status"]=="completed"; print("400/400 completed")' \
  "$BENCH_RUN/results/episode_result_log.csv" \
  "$BENCH_RUN/results/mp_workers/execution_manifest.json"

    Expected: 400/400 completed from unique merged episode rows and a completed worker manifest.

06. References

  1. PARTNR Planner Quickstart
  2. PARTNR environment and asset installation
  3. Habitat-Sim installation and source builds
  4. Habitat-Lab
  5. PARTNR project page
  6. Resilience EAS Case Analysis execution notes
  7. PARTNR benchmark execution notes
  8. Generic vLLM server setup
  9. Qwen vLLM server and client example
  10. PARTNR full-method run examples
  11. Terminal-Bench runbook structure reference

SEPARATE SERVICE

LLM Judge

The planner and Judge are independent model services. The planner selects actions; the Judge never executes actions. During Resilience Case Analysis, the Judge supplies structured critic scores, while clean EAS execution evidence is aggregated into StageBaselines scoped by Judge and episode.

ServicePurposeAPIUsed by
PlannerSelects Habitat actions127.0.0.1:8000/v1/completionsAll runs
LLM JudgeScores critic transitions$HABITAT_LLM_BASE_URL/chat/completionsResilience only

  1. Set the Judge endpoint

    The canonical joint config uses evaluation.critic.llm_model=gpt-5.1 with deterministic inference. Set the endpoint explicitly so the Judge cannot fall back to the planner vLLM URL.

    export HABITAT_LLM_BASE_URL="https://your-judge-host.example/v1"
export OPENAI_API_KEY="your-judge-api-key"

    Required: an OpenAI-compatible /v1/chat/completions endpoint with JSON response mode. Keep the API key out of files and version control.

  2. Run the structured Judge preflight

    This calls the same RewardShaper preflight used by the project and consumes one Judge request.

    python - <<'PY'
import json, os
from habitat_llm.evaluation.core.critic_lifecycle import preflight_reward_shaper
result = preflight_reward_shaper({
    "use_llm_shaping": True, "reward_shaper_json_mode": True,
    "llm_model": "gpt-5.1",
    "llm_base_url": os.environ["HABITAT_LLM_BASE_URL"],
})
print(json.dumps(result, indent=2))
assert result["valid"], result["reason"]
PY

    Pass: valid=true with numeric goal_progress_score, rationality_score, and efficiency_score.

  3. Validate Judge and StageBaseline lineage

    CASE="$RES_RUN/results/resilience_joint_qwen35/joint_resilience_evaluation"
python - "$CASE/raw_rollouts.csv" <<'PY'
import csv, sys
rows = list(csv.DictReader(open(sys.argv[1])))
truthy = lambda value: str(value).lower() in {"1", "true", "yes"}
fields = ("reward_shaper_valid", "critic_lifecycle_valid",
          "stage_baseline_available", "stage_baseline_episode_match")
assert {row["judge_model"] for row in rows} == {"gpt-5.1"}
assert {row["stage_baseline_alignment_scope"] for row in rows} == {"judge_model_episode"}
assert all(all(truthy(row[field]) for field in fields) for row in rows)
print("Judge lineage valid")
PY

    Expected: every formal row uses the same Judge and an episode-matched StageBaseline. A different Judge requires its own clean calibration evidence.