AssetOpsBench — Inference Optimization

A performance-profiling framework for IBM Research's AssetOpsBench, a multi-agent system for industrial asset maintenance. I built it to measure where the agents spend their time, tokens, and tool calls: the groundwork for an inference-optimization study in Columbia's High Performance Machine Learning course (Spring 2026).

What I built

Everything lives in profiling/:

• Instrumented orchestrators: wrap both orchestration styles, MetaAgent (PlanExecuteRunner) and AgentHive, and time every MCP tool call.
• Token tracking: a wrapper around the LiteLLM backend that records prompt and completion tokens for every LLM call.
• Public MCP tool adapters: replace AgentHive's private IBM tool packages so it runs against the public MCP servers (IoT, FMSR, TSFM, Utilities).
• Tool-call accuracy: Jaccard similarity between the agents an orchestrator actually called and the agents each scenario expects.
• Deep profiling: torch.profiler traces for time-series (TSFM) scenarios, with all metrics logged to Weights & Biases.
• Benchmark suite: 20 scenarios (5 each across IoT, TSFM, FMSR, and work orders), run on both orchestrators, producing 40 per-scenario result files and comparison charts.

Results

Run it

uv sync
uv run profiling-benchmark                    # full benchmark: both orchestrators, 20 scenarios
uv run profiling-benchmark --n-per-domain 2   # quicker run
uv run profiling-benchmark --charts-only      # regenerate charts from saved results
uv run profiling-benchmark --dry-run          # list scenarios without running

The default model is Claude Haiku 4.5 through LiteLLM. Set the matching API key, or choose another model with --llm-model or the PROFILING_LLM_MODEL environment variable.

The full optimization study

The optimizations built on this profiling (parallelization, caching, and INT4 quantization) were developed with Yassine Jebbouri, Darief Rida Maes, and Vivek G. Iyer, and live in the team repository: yassinejebbouri/AssetOpsBench.

---

The original AssetOpsBench README follows below.

---

AI Agents for Industrial Asset Operations & Maintenance

📘 Tutorials: Learn more from our detailed guides —
ReActXen IoT Agent (EMNLP 2025) | FailureSensorIQ (NeurIPS 2025) | AssetOpsBench Lab (AAAI 2026) | Spiral (AAAI 2026) | AssetOpsBench Technical Material

📄 Paper | 🤗 HF-Dataset | 📢 IBM Blog | 🤗 HF Blog | Contributors

---

📢 Call for Scenario Contribution

We are expanding AssetOpsBench to cover a broader range of industrial challenges. We invite researchers and practitioners to contribute new scenarios, particularly in the following areas:

• Asset Classes: Turbines, HVAC Systems, Pumps, Transformers, CNC Machines, Robotics, Engines, and so on.
• Task Domains: Prognostics and Health Management, Remaining Useful Life (RUL) estimation, or Root Cause Analysis (RCA), Diagnostic Analysis and Predictive Maintenance.

How to contribute:

1. Define your scenario following our Utterance Guideline, Ground Truth Guideline

2. Explore the Hugging Face dataset as examples.

3. Submit a Pull Request or open an Issue with the tag new-scenario.

4. Contact us via email if any question:

   • Dhaval Patel (pateldha@us.ibm.com)
   • Nianjun Zhou (jzhou@us.ibm.com)

---

Resources

• Video Overview: AssetOpsBench - AI Agents for Industrial Asset Operations & Maintenance by Reliability Odyssey.

---

📑 Table of Contents

1. Announcements
2. Introduction
3. Datasets
4. AI Agents
5. Multi-Agent Frameworks
6. System Diagram
7. Leaderboards
8. Docker Setup
9. Talks & Events
10. External Resources
11. Contributors

---

Announcements (Papers, Invited Talks, etc)

• 📊 Dataset Update: AssetOpsBench expanded to cover wider variety of 9 Asset classes (Chiller, AHU, Pump, Motor, Bearing, Engine, Rotors, Boilers, Turbine, etc.) and various Tasks (Remaining Useful Life, Fault Classification, Rule Monitoring, etc.)
  
  Special Thanks to primary Contributors: 👥 @DeveloperMindset123, @ChathurangiShyalika, @Fabio-Lorenzi1

• 📰 AAAI-2026: SPIRAL: Symbolic LLM Planning via Grounded and Reflective Search
  

• 🎯 AAAI-2026 Lab: From Inception to Productization: Hands-on Lab for the Lifecycle of Multimodal Agentic AI in Industry 4.0
  

• 📰 AABA4ET/AAAI-2026: Agentic Code Generation for Heuristic Rules in Equipment Monitoring

• 📰 IAAI/AAAI-2026: Diversity Meets Relevancy: Multi-Agent Knowledge Probing for Industry 4.0 Applications

• 📰 IAAI/AAAI-2026: Deployed AI Agents for Industrial Asset Management: CodeReAct Framework for Event Analysis and Work Order Automation

• 📰 AAAI-2026 Demo: AssetOpsBench-Live: Privacy-Aware Online Evaluation of Multi-Agent Performance in Industrial Operations
  

• 📰 NeurIPS-2025 Social — Evaluating Agentic Systems
  Talk: Building Reliable Agentic Benchmarks: Insights from AssetOpsBench Total Registered Users: 2000+
  
  

• 🕓 Past Event: 2025-10-03 – 2-Hour Workshop: AI Agents and Their Role in Industry 4.0 Applications
  

• 🏆 Accepted Papers: Parts of papers are accepted at NeurIPS 2025, EMNLP 2025 Research Track, and EMNLP 2025 Industry Track.

• 🚀 2025-09-01: CODS 2025 Competition launched – Access AI Agentic Challenge AssetOpsBench-Live.

• 📦 2025-06-01: AssetOpsBench v1.0 released with 141 industrial Scenarios.

✨ Stay tuned for new tracks, competitions, and community events.

---

Introduction

AssetOpsBench is a unified framework for developing, orchestrating, and evaluating domain-specific AI agents in industrial asset operations and maintenance.

It provides:

• 4 domain-specific agents
• 2 multi-agent orchestration frameworks

Designed for maintenance engineers, reliability specialists, and facility planners, it allows reproducible evaluation of multi-step workflows in simulated industrial environments.

---

Datasets: 141 Scenarios

AssetOpsBench scenarios span multiple domains:

Domain	Example Task
IoT	"List all sensors of Chiller 6 in MAIN site"
FSMR	"Identify failure modes detected by Chiller 6 Supply Temperature"
TSFM	"Forecast 'Chiller 9 Condenser Water Flow' for the week of 2020-04-27"
WO	"Generate a work order for Chiller 6 anomaly detection"

Some tasks focus on a single domain, others are multi-step end-to-end workflows.
Explore all scenarios HF-Dataset.

---

AI Agents

Domain-Specific Agents (Important tools)

• IoT Agent: get_sites, get_history, get_assets, get_sensors
• FMSR Agent: get_sensors, get_failure_modes, get_failure_sensor_mapping
• TSFM Agent: forecasting, timeseries_anomaly_detection
• WO Agent: generate_work_order

Multi-Agent Frameworks (Blue Prints)

• MetaAgent: reAct-based single-agent-as-tool orchestration
• AgentHive: plan-and-execute sequential workflow

MCP Environment

The src/ directory contains MCP servers and a plan-execute runner built on the Model Context Protocol. See INSTRUCTIONS.md for setup, usage, and testing.

---

Leaderboards

• Evaluated with 7 Large Language Models
• Trajectories scored using LLM Judge (Llama-4-Maverick-17B)
• 6-dimensional criteria measure reasoning, execution, and data handling

Example: MetaAgent leaderboard

---

Run AssetOpsBench in Docker

• Please Refer to the
• Pre-built Docker Images: assetopsbench-basic (minimal) & assetopsbench-extra (full)
• Conda environment: assetopsbench
• Full setup guide

cd /path/to/AssetOpsBench
chmod +x benchmark/entrypoint.sh
docker-compose -f benchmark/docker-compose.yml build
docker-compose -f benchmark/docker-compose.yml up

---

External Resources

• 📄 Paper: AssetOpsBench: Benchmarking AI Agents for Industrial Asset Operations
• 🤗 HuggingFace: Scenario & Model Hub
• 📢 Blog: Insights, Tutorials, and Updates
• 🎥 Recorded Talks: Link coming soon.

---

---

Contributors

Thanks goes to these wonderful people ✨

DhavalRepo18 💻 📖	ShuxinLin 💻 📖	jtrayfield 💻 📖	nianjunz 💻 📖	ChathurangiShyalika 💻 📖	PUSHPAK-JAISWAL 💻 📖	bradleyjeck 💻 📖
florenzi002 💻 📖	kushwaha001 💻	Mohit Gupta 📖	Ayan Das 📖 💻

---