[WIP] Regression for 3.6-EA1 by tosokin · Pull Request #331 · openshift-psap/forge

tosokin · 2026-10-14T08:37:47Z

No description provided.

openshift-ci · 2026-10-14T08:37:51Z

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

openshift-ci · 2026-10-14T08:37:52Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign ashtarkb for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

coderabbitai · 2026-10-14T08:37:57Z

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

tosokin · 2026-10-14T08:40:05Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-distributed-defrault-benchmark
/cluster janus
/pipeline forge-test-only

psap-forge-bot · 2026-10-14T11:52:29Z

✅ Execution of llm_d | cpt-release-testing-gpt-oss-120b completed with success after 3h 2m 31s ✅

forge-llm-d-20261014-084058 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — success → Preflight checks completed successfully
01__test — success

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-distributed-defrault-benchmark
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: Create remove-before-merge.txt
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      runtime.deployment_profile: release-distributed-default
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results
- nested run 001__llmd__concurrent-1k-1k__release-distributed-defa
- nested run 002__llmd__heavy-heterogeneous__release-distributed-defa
- nested run 003__llmd__multi-turn__release-distributed-defa

Pipeline Step Details

✅ 00__preflight `9 seconds`

✅ 01__test `3 hours, 2 minutes, 22 seconds`

Test Description

This FORGE run tests the llm_d project using the cpt-release-testing-gpt-oss-120b preset, deploying openai/gpt-oss-120b on the Janus H200 cluster with the release-distributed-default deployment profile. It runs the concurrent-1k-1k, heavy-heterogeneous, and multi-turn benchmark workloads to validate RHOAI 3.6 release performance.

📊 Test directory: 001__llmd__concurrent-1k-1k__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=concurrent-1k-1k
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 1.0h, benchmark: 53.0m
📊 Test directory: 002__llmd__heavy-heterogeneous__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=heavy-heterogeneous
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 59.6m, benchmark: 52.1m
📊 Test directory: 003__llmd__multi-turn__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=multi-turn
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 15.1m, benchmark: 7.1m
✅ 004__postprocessing: ✔️ parse | ✔️ visualize | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ✔️ s3_import | ⚪ analyse_kpis | ✔️ s3_export

📤 02__export-artifacts

🔒 Censoring Report
- 📊 865 files scanned
- ✅ Clean files: 865

✅ Post-processing Status /workspace/artifacts/01__test/004__postprocessing

✅ parse: success
✅ visualize: success
- 📄 reports/report_00_guidellm_performance_analysis.html
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
✅ s3_import: success
⚠️ analyse_kpis: warning
- all KPIs were skipped, no regression test performed
- 🌐 kpi_analyze.html
- 📊 kpi_analyze.json
- No regression in 21 KPIs
✅ s3_export: success
- 📤 Exported to: s3://psap-dashboard-data/forge-cpt/llm-d/release/26-10-14_114450_985/
- ✅ Uploaded files: 3

psap-forge-bot · 2026-10-14T11:53:03Z

🟢 Submission of llm_d cpt-release-testing-gpt-oss-120b succeeded after 3 hours, 12 minutes, 4 seconds 🟢

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-distributed-defrault-benchmark
/cluster janus
/pipeline forge-test-only

tosokin · 2026-10-21T05:59:23Z

/test fournos rhaiis nvidia
/cluster janus
/pipeline forge-full
/var tests.rhaiis.model_key: gpt-oss-120b
/var tests.rhaiis.workload_keys: ["debug-guidellm-1k-1k","debug-profile1-reverse","profile1"]
/var rhaiis.engines.vllm.args.tensor-parallel-size: 2
/var tests.rhaiis.version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm

psap-forge-bot · 2026-10-21T08:02:05Z

✅ Execution of rhaiis | nvidia completed with success after 1h 52m 17s ✅

forge-rhaiis-20261021-060154 -- rhaiis nvidia

00__pre-cleanup — success
01__prepare — success
02__preflight — success
03__test — success
04__post-cleanup — success

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - nvidia
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      rhaiis.engines.vllm.args.tensor-parallel-size: 2
      tests.rhaiis.model_key: gpt-oss-120b
      tests.rhaiis.version: RHOAI-3.6
      tests.rhaiis.workload_keys:
      - debug-guidellm-1k-1k
      - debug-profile1-reverse
      - profile1
    project: rhaiis
hardware:
  gpuCount: 2
  gpuType: nvidia
owner: tosokin
pipeline: forge-full

MLFlow links

mlflow results
- nested run 005__benchmark_debug-guidellm-1k-1k
- nested run 006__benchmark_debug-profile1-reverse
- nested run 007__benchmark_profile1

Pipeline Step Details

✅ 00__pre-cleanup `8 seconds`

✅ 01__prepare `5 seconds`

✅ 02__preflight `4 seconds`

✅ 03__test `1 hour, 51 minutes, 53 seconds`

Test Description

Regression test for the rhaiis project on RHOAI 3.6, benchmarking vLLM serving gpt-oss-120b with tensor-parallel-size 2 on a 2-GPU NVIDIA janus cluster. It runs GuideLLM workloads debug-guidellm-1k-1k, debug-profile1-reverse, and profile1 to capture/export performance metrics.

📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/005__benchmark_debug-guidellm-1k-1k
- model_key=gpt-oss-120b, workload_key=debug-guidellm-1k-1k, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=3abbaf45-4a6e-4e03-9077-22821884cf07
📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/006__benchmark_debug-profile1-reverse
- model_key=gpt-oss-120b, workload_key=debug-profile1-reverse, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=3abbaf45-4a6e-4e03-9077-22821884cf07
📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/007__benchmark_profile1
- model_key=gpt-oss-120b, workload_key=profile1, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=3abbaf45-4a6e-4e03-9077-22821884cf07
✅ 002__postprocessing: ✔️ parse | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ⚪ s3_import | ⚪ analyse_kpis | ⚪ s3_export

✅ 04__post-cleanup `7 seconds`

📤 05__export-artifacts

🔒 Censoring Report
- 📊 226 files scanned
- ✅ Clean files: 226

✅ Post-processing Status /workspace/artifacts/03__test/002__postprocessing

✅ parse: success
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
⏭️ s3_import: disabled
- s3_import disabled
⏭️ analyse_kpis: disabled
- analyze disabled
⏭️ s3_export: disabled
- s3_export disabled

psap-forge-bot · 2026-10-21T08:02:37Z

🟢 Submission of rhaiis nvidia succeeded after 2 hours, 42 seconds 🟢

/test fournos rhaiis nvidia
/var tests.rhaiis.model_key: gpt-oss-120b
/var tests.rhaiis.workload_keys: ["debug-guidellm-1k-1k","debug-profile1-reverse","profile1"]
/var rhaiis.engines.vllm.args.tensor-parallel-size: 2
/var tests.rhaiis.version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm
/cluster janus
/pipeline forge-full

tosokin · 2026-10-21T08:32:54Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-benchmark
/cluster janus
/pipeline forge-test-only

psap-forge-bot · 2026-10-21T11:41:45Z

⛔ Execution of llm_d | cpt-release-testing-gpt-oss-120b was aborted (user abort) after 9s ⛔

forge-llm-d-20261021-083346 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — success → Preflight checks completed successfully
01__test — signal_abort → Aborted by signal SIGTERM

⛔ JOB ABORTED - spec.shutdown=Stop

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-benchmark
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      runtime.deployment_profile: release-distributed-default
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results
- nested run 001__llmd__concurrent-1k-1k__release-distributed-defa
- nested run 002__llmd__heavy-heterogeneous__release-distributed-defa

Pipeline Step Details

✅ 00__preflight `9 seconds`

❌ 01__test — `signal_abort`: Aborted by signal SIGTERM

Test Description

This FORGE run tests the llm_d project’s RHOAI 3.6-EA2 release benchmarking for openai/gpt-oss-120b on the Janus H200 cluster using the release-distributed-default deployment profile. It executes the concurrent-1k-1k, heavy-heterogeneous, and multi-turn completion benchmarks and exports KPIs for regression analysis.

📊 Test directory: 001__llmd__concurrent-1k-1k__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=concurrent-1k-1k
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 1.4h, benchmark: 52.9m
📊 Test directory: 002__llmd__heavy-heterogeneous__release-distributed-defa
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=heavy-heterogeneous
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 59.3m, benchmark: 52.2m

📤 02__export-artifacts

Censoring Applied

Caliper Export: 5 file(s) contained sensitive content

5 file(s) had sensitive content sanitized and included in export:

01__test/001__llmd__concurrent-1k-1k__release-distributed-defa/artifacts/namespace.events.txt

01__test/001__llmd__concurrent-1k-1k__release-distributed-defa/004__deploy_llmisvc/artifacts/pod_descriptions.txt

01__test/001__llmd__concurrent-1k-1k__release-distributed-defa/008__capture_llmisvc_state/artifacts/workload_descriptions.txt

01__test/001__llmd__concurrent-1k-1k__release-distributed-defa/008__capture_llmisvc_state/artifacts/llminferenceservice.pods.describe.txt

01__test/002__llmd__heavy-heterogeneous__release-distributed-defa/artifacts/namespace.events.txt
Sensitive content (passwords, API keys, tokens) was replaced with placeholder text.

For details, see: $ARTIFACT_DIR/censoring_report.yaml

🔒 Censoring Report
- 📊 480 files scanned
- ✅ Clean files: 475
- ⚠️ Unexpected censoring: 5
- contains keyword pattern: AKIA[0-9A-Z]{16}: 5 file(s)

tosokin · 2026-10-21T12:08:04Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/cluster janus
/pipeline forge-test-only
/var runtime.deployment_profile: release-distributed-default
/var workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas: 1
/var workloads.benchmarks.heavy-heterogeneous.deployment_overrides.replicas: 1
/var workloads.benchmarks.multi-turn.deployment_overrides.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica

psap-forge-bot · 2026-10-21T12:09:45Z

❌ Execution of llm_d | cpt-release-testing-gpt-oss-120b failed (pipeline step failure) after 2s ❌

forge-llm-d-20261021-120840 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — internal_error → Config key 'workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas' does not exist, and cannot create it at the moment :/

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      runtime.deployment_profile: release-distributed-default
      workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas: 1
      workloads.benchmarks.heavy-heterogeneous.deployment_overrides.replicas: 1
      workloads.benchmarks.multi-turn.deployment_overrides.replicas: 1
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results

Pipeline Step Details

❌ 00__preflight `2 seconds` — `internal_error`: Config key 'workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas' does not exist, and cannot create it at the moment :/ (🔴 4E)

📤 02__export-artifacts (🔴 6E)

🔒 Censoring Report
- 📊 28 files scanned
- ✅ Clean files: 28

psap-forge-bot · 2026-10-21T12:10:11Z

🔴 Submission of llm_d cpt-release-testing-gpt-oss-120b failed after 1 minute, 29 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20261021-120840' failed: Tasks Completed: 2 (Failed: 1, Cancelled 0), Skipped: 1

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas: 1
/var workloads.benchmarks.heavy-heterogeneous.deployment_overrides.replicas: 1
/var workloads.benchmarks.multi-turn.deployment_overrides.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
/cluster janus
/pipeline forge-test-only

tosokin · 2026-10-21T13:24:26Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
/cluster janus
/pipeline forge-test-only

psap-forge-bot · 2026-10-21T14:42:13Z

❌ Execution of llm_d | cpt-release-testing-gpt-oss-120b failed (pipeline step failure) after 1h 12m 27s ❌

forge-llm-d-20261021-132509 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — success → Preflight checks completed successfully
01__test — test_failure → Tests failed with exit code 1

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      deployments.profiles.release-distributed-default.replicas: 1
      runtime.deployment_profile: release-distributed-default
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results
- nested run 001__llmd__concurrent-1k-1k__release-distributed-defa
- nested run 002__llmd__heavy-heterogeneous__release-distributed-defa
- nested run 003__llmd__multi-turn__release-distributed-defa

Pipeline Step Details

✅ 00__preflight `10 seconds`

❌ 01__test `1 hour, 12 minutes, 17 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

Test Description

This test runs the llm_d project’s RHOAI 3.6 release benchmark for openai/gpt-oss-120b on the Janus H200 cluster using the cpt-release-testing-gpt-oss-120b preset. It focuses on the release-distributed-default deployment with 1 replica, running the concurrent-1k-1k, heavy-heterogeneous, and multi-turn workloads to collect regression KPIs.

Failure Review 001 Llmd Concurrent-1K-1K Release-Distributed-Defa

001__llmd__concurrent-1k-1k__release-distributed-defa

The FORGE test failed during dependency installation when the pip install command for benchconf from the Git branch toskin-llm-d-profiles-align-with-rhaiis returned a non-zero exit status of 1. Because the install was run with --quiet and no task log was available, the exact pip/Git/network error is not visible, but the direct blocker was the unsuccessful installation of benchconf from the specified Git URL.

Failure Review 002 Llmd Heavy-Heterogeneous Release-Distributed-Defa

002__llmd__heavy-heterogeneous__release-distributed-defa

The FORGE run failed during a dependency-installation step when python3 -m pip install --quiet 'benchconf @ git+https://github.com/tosokin/benchconf@toskin-llm-d-profiles-align-with-rhaiis' returned a non-zero exit status of 1. This blocked the test because the required benchconf package could not be installed from the specified GitHub branch. The available logs do not include the underlying pip error, so the exact pip or Git failure reason is not evidenced.

❌ Test directory: 001__llmd__concurrent-1k-1k__release-distributed-defa
- Test failed with exception: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--quiet', 'benchconf @ git+https://github.com/tosokin/benchconf@toskin-llm-d-profiles-align-with-rhaiis']' returned non-zero exit status 1.
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=concurrent-1k-1k
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 5.0m, benchmark: 1.7s
❌ Test directory: 002__llmd__heavy-heterogeneous__release-distributed-defa
- Test failed with exception: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--quiet', 'benchconf @ git+https://github.com/tosokin/benchconf@toskin-llm-d-profiles-align-with-rhaiis']' returned non-zero exit status 1.
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=heavy-heterogeneous
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 4.2m, benchmark: 1.3s
📊 Test directory: 003__llmd__multi-turn__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=multi-turn
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 29.7m, benchmark: 22.4m
❌ 004__postprocessing: ✔️ parse | ✔️ visualize | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ❌ s3_import | ⚪ analyse_kpis | ❌ s3_export

📤 02__export-artifacts

🔒 Censoring Report
- 📊 738 files scanned
- ✅ Clean files: 738

❌ Post-processing Status /workspace/artifacts/01__test/004__postprocessing

✅ parse: success
✅ visualize: success
- 📄 reports/report_00_guidellm_performance_analysis.html
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
❌ s3_import: failed
- ⚠️ An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
⚠️ analyse_kpis: warning
- no historical KPI found for regression testing
- 🌐 kpi_analyze.html
- 📊 kpi_analyze.json
- No regression in 7 KPIs
❌ s3_export: failed
- ⚠️ All uploads failed. Failed files: ['/workspace/artifacts/01__test/004__postprocessing/kpis/kpis.json', '/workspace/artifacts/01__test/004__postprocessing/kpis/dashboard.csv', '/workspace/artifacts/01__test/004__postprocessing/regression_analyze/kpi_analyze.json']

psap-forge-bot · 2026-10-21T14:42:47Z

🔴 Submission of llm_d cpt-release-testing-gpt-oss-120b failed after 1 hour, 17 minutes, 36 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20261021-132509' failed: Tasks Completed: 3 (Failed: 1, Cancelled 0), Skipped: 0

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
/cluster janus
/pipeline forge-test-only

tosokin · 2026-10-21T15:04:07Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
/cluster janus
/pipeline forge-test-only

psap-forge-bot · 2026-10-21T16:38:26Z

❌ Execution of llm_d | cpt-release-testing-gpt-oss-120b failed (pipeline step failure) after 1h 15m 56s ❌

forge-llm-d-20261021-151754 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — success → Preflight checks completed successfully
01__test — test_failure → Tests failed with exit code 1

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      deployments.profiles.release-distributed-default.replicas: 1
      runtime.deployment_profile: release-distributed-default
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results
- nested run 001__llmd__concurrent-1k-1k__release-distributed-defa
- nested run 002__llmd__heavy-heterogeneous__release-distributed-defa
- nested run 003__llmd__multi-turn__release-distributed-defa

Pipeline Step Details

✅ 00__preflight `9 seconds`

❌ 01__test `1 hour, 15 minutes, 47 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

Test Description

This FORGE test runs the llm_d release-regression test for openai/gpt-oss-120b on RHOAI 3.6, narrowing the standard rhoai-release multi-deployment preset to only the release-distributed-default profile with 1 replica. It then performs the default smoke check and runs the concurrent-1k-1k, heavy-heterogeneous, and multi-turn benchmarks.

Failure Review 001 Llmd Concurrent-1K-1K Release-Distributed-Defa

001__llmd__concurrent-1k-1k__release-distributed-defa

The FORGE test failed during Python dependency installation when pip install --quiet 'benchconf @ git+https://github.com/tosokin/benchconf@tosokin-llm-d-profiles-align-with-rhaiis' returned non-zero exit status 1. The evidence does not capture the underlying pip/git error, so the definitive failure is that the Git-based benchconf installation command failed.

Failure Review 002 Llmd Heavy-Heterogeneous Release-Distributed-Defa

002__llmd__heavy-heterogeneous__release-distributed-defa

The FORGE failure occurred during a Python dependency installation step, not during Kubernetes, GPU, KServe, or LLMInferenceService deployment. The specific failing command was:

/usr/bin/python3 -m pip install --quiet 'benchconf @ git+https://github.com/tosokin/benchconf@tosokin-llm-d-profiles-align-with-rhaiis'

It returned non-zero exit status 1, meaning the required benchconf dependency could not be installed, and no detailed pip/Git error was available to identify the underlying cause.

❌ Test directory: 001__llmd__concurrent-1k-1k__release-distributed-defa
- Test failed with exception: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--quiet', 'benchconf @ git+https://github.com/tosokin/benchconf@tosokin-llm-d-profiles-align-with-rhaiis']' returned non-zero exit status 1.
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=concurrent-1k-1k
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 4.1m, benchmark: 1.5s
❌ Test directory: 002__llmd__heavy-heterogeneous__release-distributed-defa
- Test failed with exception: Command '['/usr/bin/python3', '-m', 'pip', 'install', '--quiet', 'benchconf @ git+https://github.com/tosokin/benchconf@tosokin-llm-d-profiles-align-with-rhaiis']' returned non-zero exit status 1.
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=heavy-heterogeneous
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 4.1m, benchmark: 1.2s
📊 Test directory: 003__llmd__multi-turn__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=multi-turn
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 29.2m, benchmark: 22.4m
❌ 004__postprocessing: ✔️ parse | ✔️ visualize | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ❌ s3_import | ⚪ analyse_kpis | ❌ s3_export

📤 02__export-artifacts

🔒 Censoring Report
- 📊 738 files scanned
- ✅ Clean files: 738

❌ Post-processing Status /workspace/artifacts/01__test/004__postprocessing

✅ parse: success
✅ visualize: success
- 📄 reports/report_00_guidellm_performance_analysis.html
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
❌ s3_import: failed
- ⚠️ An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
⚠️ analyse_kpis: warning
- no historical KPI found for regression testing
- 🌐 kpi_analyze.html
- 📊 kpi_analyze.json
- No regression in 7 KPIs
❌ s3_export: failed
- ⚠️ All uploads failed. Failed files: ['/workspace/artifacts/01__test/004__postprocessing/kpis/kpis.json', '/workspace/artifacts/01__test/004__postprocessing/kpis/dashboard.csv', '/workspace/artifacts/01__test/004__postprocessing/regression_analyze/kpi_analyze.json']

psap-forge-bot · 2026-10-21T16:38:49Z

🔴 Submission of llm_d cpt-release-testing-gpt-oss-120b failed after 1 hour, 20 minutes, 53 seconds 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20261021-151754' failed: Tasks Completed: 3 (Failed: 1, Cancelled 0), Skipped: 0

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica
/cluster janus
/pipeline forge-test-only

tosokin · 2026-10-21T16:50:40Z

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var runtime.benchmark_key: concurrent-1k-1k
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica-1k-1k
/cluster janus
/pipeline forge-test-only

psap-forge-bot · 2026-10-21T18:15:05Z

❌ Execution of llm_d | cpt-release-testing-gpt-oss-120b failed (pipeline step failure) after 1h 19m 33s ❌

forge-llm-d-20261021-165150 -- llm_d cpt-release-testing-gpt-oss-120b

00__preflight — success → Preflight checks completed successfully
01__test — test_failure → Tests failed with exit code 1

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - cpt-release-testing-gpt-oss-120b
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica-1k-1k
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      cpt.kpi.labels.product_version: RHOAI-3.6
      deployments.profiles.release-distributed-default.replicas: 1
      runtime.benchmark_key: concurrent-1k-1k
      runtime.deployment_profile: release-distributed-default
    project: llm_d
owner: tosokin
pipeline: forge-test-only

MLFlow links

mlflow results

Pipeline Step Details

✅ 00__preflight `13 seconds`

❌ 01__test `1 hour, 19 minutes, 20 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 2E)

Test Description

This llm_d FORGE run tests RHOAI 3.6-EA2 performance regression for openai/gpt-oss-120b on the Janus H200 cluster, narrowing the release-testing preset to a single release-distributed-default deployment with 1 replica. It runs the concurrent-1k-1k benchmark to collect KPIs and export results for regression analysis.

📊 Test directory: 001__llmd__concurrent-1k-1k__release-distributed-defa
- Test completed successfully
- model_name=openai/gpt-oss-120b, deployment_profile=release-distributed-default, guidellm_loadshape=concurrent-1k-1k
- KPI extra labels: platform=OCP, test_harness=rhoai-release, product_version=RHOAI-3.6, gpu_type=NVIDIA-H200, model_name=openai/gpt-oss-120b
- Duration test: 58.9m, benchmark: 52.1m
❌ 002__postprocessing: ✔️ parse | ✔️ visualize | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ❌ s3_import | ⚪ analyse_kpis | ❌ s3_export

📤 02__export-artifacts

🔒 Censoring Report
- 📊 311 files scanned
- ✅ Clean files: 311

❌ Post-processing Status /workspace/artifacts/01__test/002__postprocessing

✅ parse: success
✅ visualize: success
- 📄 reports/report_00_guidellm_performance_analysis.html
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
❌ s3_import: failed
- ⚠️ An error occurred (InvalidAccessKeyId) when calling the ListObjectsV2 operation: The AWS Access Key Id you provided does not exist in our records.
⚠️ analyse_kpis: warning
- no historical KPI found for regression testing
- 🌐 kpi_analyze.html
- 📊 kpi_analyze.json
- No regression in 7 KPIs
❌ s3_export: failed
- ⚠️ All uploads failed. Failed files: ['/workspace/artifacts/01__test/002__postprocessing/kpis/kpis.json', '/workspace/artifacts/01__test/002__postprocessing/kpis/dashboard.csv', '/workspace/artifacts/01__test/002__postprocessing/regression_analyze/kpi_analyze.json']

psap-forge-bot · 2026-10-21T18:15:52Z

🔴 Submission of llm_d cpt-release-testing-gpt-oss-120b failed after 1 hour, 24 minutes 🔴

Error: FournosJobFailureError: FOURNOS Job 'forge-llm-d-20261021-165150' failed: Tasks Completed: 3 (Failed: 1, Cancelled 0), Skipped: 0

/test fournos llm_d cpt-release-testing-gpt-oss-120b
/var runtime.deployment_profile: release-distributed-default
/var runtime.benchmark_key: concurrent-1k-1k
/var deployments.profiles.release-distributed-default.replicas: 1
/var cpt.kpi.labels.product_version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-EA2-gpt-oss-120b-janus-distributed-default-1-replica-1k-1k
/cluster janus
/pipeline forge-test-only

tosokin · 2026-10-22T06:05:09Z

/test fournos rhaiis nvidia
/cluster janus
/pipeline forge-full
/var tests.rhaiis.model_key: gpt-oss-120b
/var tests.rhaiis.workload_keys: ["debug-guidellm-1k-1k","debug-profile1-reverse","profile1"]
/var rhaiis.engines.vllm.args.tensor-parallel-size: 2
/var tests.rhaiis.version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm-full-prom-metrics

psap-forge-bot · 2026-10-22T08:39:40Z

✅ Execution of rhaiis | nvidia completed with success after 2h 24m 33s ✅

forge-rhaiis-20261022-060600 -- rhaiis nvidia

00__pre-cleanup — success
01__prepare — success
02__preflight — success
03__test — success
04__post-cleanup — success

Execution Engine Configuration

cluster: janus
exclusive: true
executionEngine:
  forge:
    args:
    - nvidia
    configOverrides:
      caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
      caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm-full-prom-metrics
      caliper.export.backend.mlflow.config.workspace: forge-llmd
      ci_job.gh.from_gh: true
      ci_job.gh.pr.num: 331
      ci_job.gh.pr.title: '[WIP] Regression for 3.6-EA1'
      ci_job.gh.repo.name: forge
      ci_job.gh.repo.owner: openshift-psap
      rhaiis.engines.vllm.args.tensor-parallel-size: 2
      tests.rhaiis.model_key: gpt-oss-120b
      tests.rhaiis.version: RHOAI-3.6
      tests.rhaiis.workload_keys:
      - debug-guidellm-1k-1k
      - debug-profile1-reverse
      - profile1
    project: rhaiis
hardware:
  gpuCount: 2
  gpuType: nvidia
owner: tosokin
pipeline: forge-full

MLFlow links

mlflow results
- nested run 006__benchmark_debug-guidellm-1k-1k
- nested run 007__benchmark_debug-profile1-reverse
- nested run 008__benchmark_profile1

Pipeline Step Details

✅ 00__pre-cleanup `8 seconds`

✅ 01__prepare `5 seconds`

✅ 02__preflight `5 seconds`

✅ 03__test `2 hours, 24 minutes, 8 seconds`

Test Description

This FORGE test runs the rhaiis project on a 2-GPU NVIDIA Janus cluster, overriding vLLM to use tensor-parallel-size=2 for the gpt-oss-120b model under RHOAI-3.6. It benchmarks/debugs GuideLLM workloads debug-guidellm-1k-1k, debug-profile1-reverse, and profile1, exporting full prompt metrics to MLflow.

📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/006__benchmark_debug-guidellm-1k-1k
- model_key=gpt-oss-120b, workload_key=debug-guidellm-1k-1k, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=18d45037-5d84-46cb-a3a6-6e3885b60b5d
📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/007__benchmark_debug-profile1-reverse
- model_key=gpt-oss-120b, workload_key=debug-profile1-reverse, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=18d45037-5d84-46cb-a3a6-6e3885b60b5d
📊 Test directory: 001__test_gpt-oss-120b_debug-guidellm-1k-1k_debug-profile1-reverse_profile1/008__benchmark_profile1
- model_key=gpt-oss-120b, workload_key=profile1, accelerator=NVIDIA, tensor_parallel_size=2, hf_model_id=openai/gpt-oss-120b, version=RHOAI-3.6, image_tag=v0.24.0, cluster_tag=janus, runtime_args=tensor-parallel-size: 2; gpu-memory-utilization: 0.92; trust-remote-code: True; no-enable-log-requests: True; no-enable-prefix-caching: True; uvicorn-log-level: debug, env_vars=, run_uuid=18d45037-5d84-46cb-a3a6-6e3885b60b5d
✅ 002__postprocessing: ✔️ parse | ✔️ artifacts_to_kpis | ✔️ kpis_to_mlflow | ✔️ dashboard_csv | ⚪ artifacts_to_ai_data | ⚪ s3_import | ⚪ analyse_kpis | ⚪ s3_export

✅ 04__post-cleanup `7 seconds`

📤 05__export-artifacts

🔒 Censoring Report
- 📊 307 files scanned
- ✅ Clean files: 307

✅ Post-processing Status /workspace/artifacts/03__test/002__postprocessing

✅ parse: success
✅ artifacts_to_kpis: success
- 🌐 kpis.html
- 📄 kpis.json
✅ kpis_to_mlflow: success
✅ dashboard_csv: success
- 📊 dashboard.csv
⏭️ artifacts_to_ai_data: disabled
- kpi.artifacts_to_ai_data disabled
⏭️ s3_import: disabled
- s3_import disabled
⏭️ analyse_kpis: disabled
- analyze disabled
⏭️ s3_export: disabled
- s3_export disabled

psap-forge-bot · 2026-10-22T08:39:56Z

🟢 Submission of rhaiis nvidia succeeded after 2 hours, 33 minutes, 55 seconds 🟢

/test fournos rhaiis nvidia
/var tests.rhaiis.model_key: gpt-oss-120b
/var tests.rhaiis.workload_keys: ["debug-guidellm-1k-1k","debug-profile1-reverse","profile1"]
/var rhaiis.engines.vllm.args.tensor-parallel-size: 2
/var tests.rhaiis.version: RHOAI-3.6
/var caliper.export.backend.mlflow.config.experiment: llm-d-3.6-regression
/var caliper.export.backend.mlflow.config.workspace: forge-llmd
/var caliper.export.backend.mlflow.config.run_name: 3.6-gpt-oss-120b-janus-rhaiis-1k-1k-debug-guidellm-full-prom-metrics
/cluster janus
/pipeline forge-full

This reverts commit 4a5c908.

Create remove-before-merge.txt

191dd66

openshift-ci Bot added the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Oct 14, 2026

kpouget changed the title ~~Create remove-before-merge.txt~~ [llm-d] Create remove-before-merge.txt Oct 14, 2026

tosokin changed the title ~~[llm-d] Create remove-before-merge.txt~~ [WIP] Regression for 3.6-EA1 Oct 14, 2026

tosokin added 4 commits October 19, 2026 08:10

Merge branch 'main' into 3.6-regression

3622193

Merge branch 'main' into 3.6-regression

39a3cc7

Add debug profile wothout warmup or rampup

de01e2d

debug profile1 with reverse order

7886a13

tosokin added 2 commits October 21, 2026 14:36

Merge branch 'main' into 3.6-regression

a4b91a8

Use RHAIIS-aligned benchconf branch for llm-d

57d2a0c

fix typo in branch name

1a3176f

fix another typo

dd7874d

Add Prometheus capture to RHAIIS tests temporarily

4a5c908

Revert "Add Prometheus capture to RHAIIS tests temporarily"

a921923

This reverts commit 4a5c908.

Conversation

tosokin commented Oct 14, 2026

Uh oh!

openshift-ci Bot commented Oct 14, 2026

Uh oh!

openshift-ci Bot commented Oct 14, 2026

Uh oh!

coderabbitai Bot commented Oct 14, 2026

Draft PR not reviewed

Uh oh!

tosokin commented Oct 14, 2026

Uh oh!

psap-forge-bot Bot commented Oct 14, 2026

✅ 00__preflight 9 seconds

✅ 01__test 3 hours, 2 minutes, 22 seconds

Test Description

📤 02__export-artifacts

Uh oh!

psap-forge-bot Bot commented Oct 14, 2026

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

✅ 00__pre-cleanup 8 seconds

✅ 01__prepare 5 seconds

✅ 02__preflight 4 seconds

✅ 03__test 1 hour, 51 minutes, 53 seconds

Test Description

✅ 04__post-cleanup 7 seconds

📤 05__export-artifacts

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

✅ 00__preflight 9 seconds

❌ 01__test — signal_abort: Aborted by signal SIGTERM

Test Description

📤 02__export-artifacts

Censoring Applied

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

❌ 00__preflight 2 seconds — internal_error: Config key 'workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas' does not exist, and cannot create it at the moment :/ (🔴 4E)

📤 02__export-artifacts (🔴 6E)

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

✅ 00__preflight 10 seconds

❌ 01__test 1 hour, 12 minutes, 17 seconds — test_failure: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

Test Description

Failure Review 001 Llmd Concurrent-1K-1K Release-Distributed-Defa

Failure Review 002 Llmd Heavy-Heterogeneous Release-Distributed-Defa

📤 02__export-artifacts

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

✅ 00__preflight 9 seconds

❌ 01__test 1 hour, 15 minutes, 47 seconds — test_failure: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

Test Description

Failure Review 001 Llmd Concurrent-1K-1K Release-Distributed-Defa

Failure Review 002 Llmd Heavy-Heterogeneous Release-Distributed-Defa

📤 02__export-artifacts

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

Uh oh!

tosokin commented Oct 21, 2026

Uh oh!

psap-forge-bot Bot commented Oct 21, 2026

✅ 00__preflight 13 seconds

❌ 01__test 1 hour, 19 minutes, 20 seconds — test_failure: Tests failed with exit code 1 (🔴 2E)

Test Description

✅ 00__preflight `9 seconds`

✅ 01__test `3 hours, 2 minutes, 22 seconds`

✅ 00__pre-cleanup `8 seconds`

✅ 01__prepare `5 seconds`

✅ 02__preflight `4 seconds`

✅ 03__test `1 hour, 51 minutes, 53 seconds`

✅ 04__post-cleanup `7 seconds`

✅ 00__preflight `9 seconds`

❌ 01__test — `signal_abort`: Aborted by signal SIGTERM

❌ 00__preflight `2 seconds` — `internal_error`: Config key 'workloads.benchmarks.concurrent-1k-1k.deployment_overrides.replicas' does not exist, and cannot create it at the moment :/ (🔴 4E)

✅ 00__preflight `10 seconds`

❌ 01__test `1 hour, 12 minutes, 17 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

✅ 00__preflight `9 seconds`

❌ 01__test `1 hour, 15 minutes, 47 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 6E, 🟡 4W)

✅ 00__preflight `13 seconds`

❌ 01__test `1 hour, 19 minutes, 20 seconds` — `test_failure`: Tests failed with exit code 1 (🔴 2E)

✅ 00__pre-cleanup `8 seconds`

✅ 01__prepare `5 seconds`

✅ 02__preflight `5 seconds`

✅ 03__test `2 hours, 24 minutes, 8 seconds`

✅ 04__post-cleanup `7 seconds`