MOEModels.ai
Open workbench

DEPLOYBENCH / ENDPOINT RUNNER V0.1

SOURCE AVAILABLE · NOT PUBLISHED TO NPM

THE MEASUREMENT SURFACE FOR DEPLOYMENT ASSURANCE

Measure the deployment.
Keep the receipts.

DeployBench runs a bounded workload against an endpoint you control, captures request-level systems evidence, and emits a privacy-conscious JSON result with its limitations still attached.

Run from source Verify a passport
RUNNERv0.1OpenAI-compatible streaming endpoints
EVIDENCE CLASSMeasuredDirect endpoint observations
ELIGIBILITYNot comparableEvery standalone v0.1 result
CANONICAL CONTROLLED RUNS0No MOEModels-controlled run is published
01

WHAT EXISTS NOW

A benchmark result is only useful when its boundary survives the chart.

DeployBench is narrower than a leaderboard and more operational than a model-card score. It measures a specific served endpoint under a declared request shape. It does not infer a universal best model, runtime, or GPU.

The current package exists in the source repository. It is not an npm release, and no output has been admitted to the canonical MOEModels evaluation registry yet.

MEASURED / V0.1

Endpoint behavior,
observed directly.

Warmup requests remain outside the measured set. Successful and failed requests remain visible. Missing token usage or first-token events become explicit gaps instead of generated values.

01

Time to first token

Client-observed TTFT for successful content-bearing streams, summarized with count and distribution statistics.

MEASURED ENDPOINT EVIDENCE
02

Request latency

End-to-end request duration, including failures and timeouts rather than silently discarding them.

MEASURED ENDPOINT EVIDENCE
03

Success and failure

Attempted, successful, and failed requests with safe HTTP or structured error classifications.

MEASURED ENDPOINT EVIDENCE
04

Observed throughput

Request throughput and output-token throughput when the endpoint returns complete usage metadata.

MEASURED ENDPOINT EVIDENCE

STILL OPEN

Measured does not mean complete.

These dimensions remain unsupported or unknown in the current runner. They are requirements for later protocol and adapter releases, not implied capabilities.

  • Accelerator and host memory telemetry
  • Expert routing and all-to-all communication
  • Quality retention and task correctness
  • Power, energy, and infrastructure cost
  • Failure recovery and availability behavior
  • Artifact-byte verification behind a remote endpoint

PRIVACY CONTRACT

The useful evidence leaves.
The sensitive content does not.

The runner sends requests directly from your machine to the endpoint you specify. Its result stores prompt identity and size—not prompt text—plus request measurements and token usage when the endpoint supplies it.

NEVER STOREDAPI keyPrompt textResponse text
RETAINEDPrompt SHA-256UTF-8 byte countTimings + safe usage

DeployBench does not upload a result unless a future explicit publish or organization-ingest workflow is invoked. The current v0.1 runner has no canonical publication step.

EVIDENCE ADMISSION

Measured locally.
Not automatically comparable.

DEFAULT CLASSIFICATIONcomparisonEligible: falsePreserved in every standalone v0.1 result
01

Exact artifact identity and immutable revision

REQUIRED
02

Pinned runtime, version, precision, and launch configuration

REQUIRED
03

Hardware, node, accelerator, and interconnect topology

REQUIRED
04

Repeated trials with retained incomplete and failed observations

REQUIRED
05

Raw evidence review and a complete methodology fingerprint

REQUIRED
02

LOCAL WORKFLOW

Plan. Run. Verify. Share.

One evidence contract connects the web decision surface to local measurement and a portable receipt.

01PLAN

Define the deployment envelope

Use the workbench to bind an artifact, runtime target, hardware topology, workload, and service objective.

Build an assurance plan
02RUN

Measure in your environment

Point DeployBench at an OpenAI-compatible streaming endpoint you control. No weights or prompts are uploaded to MOEModels.

Open runner guide
03VERIFY + SHARE

Make the receipt portable

Pack compatible trials under one content address, recompute every summary, inspect completeness, and verify optional operator authorship.

Open Passport verifier
DEPLOYBENCH / SOURCE CHECKOUTNO NPM RELEASE
# Build from the public GitHub source
git clone https://github.com/SamSnead85/moemodels.git
cd moemodels
npm install
npm run build --workspace @moemodels/bench

# Run against an endpoint you control
MOEMODELS_BENCH_API_KEY=... node packages/bench/dist/cli.js run \
  --endpoint http://127.0.0.1:8000/v1/chat/completions \
  --model <served-model> \
  --artifact-repository <owner/repository> \
  --artifact-revision <immutable-commit> \
  --runtime <runtime> --runtime-version <version> \
  --hardware <accelerators> --topology <topology> \
  --requests 32 --concurrency 8 --output trial-01.json

# Repeat the identical configuration, then create one portable receipt
node packages/bench/dist/cli.js pack   trial-01.json trial-02.json trial-03.json   --output passport.json

# Recompute integrity, summaries, signatures, and evidence gates offline
node packages/bench/dist/cli.js verify passport.json --json

THE COMPOUNDING ASSET

Every verified run can make the next decision better.

The open runner creates distribution and inspectability. The durable platform value emerges from reviewed configurations, historical outcomes, calibration, and repeat deployment workflows. That corpus has not been created yet.

  1. 01Assurance planDefine what must be proven
  2. 02DeployBench runProduce local observations
  3. 03Passport verificationRecompute integrity and method gates
  4. 04Controlled corpusAccumulate eligible configurations
  5. 05CalibrationCompare estimates with outcomes
  6. 06Better decisionReduce the next proof gap
CURRENT CANONICAL CORPUS0 controlled configurations

The flywheel starts only when the first complete, reviewed bundle is admitted.

START WITH THE DECISION

Know the static floor.
Measure the rest.

Build the validation plan in the workbench, use DeployBench from the GitHub source checkout, then verify the resulting Passport locally.

Build an assurance plan Verify a Deployment Passport

Install from the public GitHub source; the npm package is not published yet.

PROTOCOL REFERENCE

DeployBench v0.1 draft · evidence bundle and measurement method

Read the source specification