mbpp

Reference implementation of Double Ratchet: co-evolving an inspectable evaluation metric with a lifecycle-managed skill library for self-improving LLM agents (arXiv:2607.12790)

Reference implementation of Ratchet: a minimal hygiene recipe for self-evolving LLM agents (arXiv:2605.22148)

Benchmark suite for evaluating LLMs and SLMs on coding and SE tasks. Features HumanEval, MBPP, SWE-bench, and BigCodeBench with an interactive Streamlit UI. Supports cloud APIs (OpenAI, Anthropic, Google) and local models via Ollama. Tracks pass rates, latency, token usage, and costs.

LLM Collaboration for Code Generation

Lightweight LLM evaluation tool for OpenAI-compatible endpoints (Chat Completions / Responses) with CLI and local Web UI. MMLU, GSM8K, HumanEval, TruthfulQA, HellaSwag — bring your own BASE_URL + MODEL_NAME.

A lightweight, configuration-driven evaluation framework for LLM code generation & reasoning tasks (MBPP, HumanEval, GSM8K). Supports multi-provider (DeepSeek, OpenAI, ZhipuAI) and concurrent execution.

Resample or reroute after a weak-verifier stop? Pre-registered measurements of recoverable stopping debt on MBPP+, a two-sided action-support gate on BigCodeBench that fails closed, a LiveCodeBench observability ladder, and the exchangeable-actions reference showing realized-maximum gaps carry no selector signal. Artifacts for arXiv:2607.08665v3.

LLM code-agent framework — Thought→Code→Observation loop, process-isolated sandbox, MBPP + SWE-bench tracks, MCP tools, multi-model OpenRouter benchmarks. 42 School project.

Autonomous code agent built entirely from scratch (agent frameworks prohibited): a Thought → Code → Observation loop where the LLM writes and executes real Python to solve programming tasks, using code-based tool calling instead of JSON.

mbpp-tr üzerinde Qwen3-1.7B LoRA fine-tune deneyleri; üretilen kod testlerle çalıştırılarak değerlendirildi (negatif sonuç raporu)

MBPP (sanitized) veri setinin Türkçe çevirisi; çeviri, inceleme ve doğrulama araçları

🧪 Automated LLM coding benchmarks with Ollama - HumanEval & MBPP evaluation suite with safe execution, comprehensive logging, and detailed analysis tools

Diagnostic toolkit for self-hosted LLM inference: failure-taxonomic stress harness + 4-phase orchestrator + parametric vLLM launchers

Fine-tuning CodeT5 for Python code generation on the MBPP dataset. Features custom TensorFlow training loops, mixed precision, XLA optimization, and distributed multi-GPU strategies.

Asking clarification questions before writing code, using Expected Information Gain (EIG). Based on "20 Questions for Code."

Study #4: FP8+MTP{3,5} speed on repne/vllm:v2 + max_tokens=8192 quality re-runs for BF16+DFlash n=8 and FP8+MTP=3. Follow-up to studies #2 and #3.

Dissertation: a multi-agent (Programmer→Tester) pipeline benchmarking self-hosted small language models against Claude for automated unit-test generation on HumanEval and MBPP, with coverage and mutation metrics.

BDD_Dataset: 1,137 execution-verified Python/Behave BDD problems translated from HumanEval and MBPP, paired with original canonical reference solutions. Companion dataset for the MSR 2027 paper "A Behavior-Driven Dataset for Python Code Generation Benchmarks."

Controlled comparison of boosted SFT vs. GRPO at matched compute. i.e., which training loop escapes the problems a base model never solves. Reproduces Amin et al. (NeurIPS 2025) as the baseline. In progress.

Preregistered AI-safety study of sandbagging model organisms: trigger type sets the sign of cross-capability alignment (task-local locks dismantle it, situational locks amplify it) and cue-sharing sets its size. All five predictions failed, four reversed.