Batching and concurrency: define the service envelope

Capacity is the workload that can be served while meeting accepted latency, quality and reliability limits. The paper requires declared allocation so several reservations do not silently claim the same unrestricted resources.

Updated

Computraq is preparing for launch. These guides explain readiness and the proposed framework; they do not establish a live service, integration or partnership.

Prepare these inputs

  • Request length distributions, burst patterns and latency objectives.
  • GPU allocation, scheduler settings, maximum sequences and token limits.
  • A reproducible load test and agreed error/timeout behavior.

Decisions and sequence

  • Measure throughput and latency together under representative mixtures of requests.
  • Include cold start, long prefill, retries and overload rather than testing only steady short prompts.
  • Set admitted concurrency and backpressure to the tested envelope; reserve additional capacity explicitly.

The first useful deliverable

A load-test report and admission policy showing supported concurrency, latency distribution and saturation behavior.

Failure modes to catch

More concurrent requests can improve utilization while worsening accepted-work latency. A tokens-per-second figure without the workload and queue conditions is incomplete.

How this follows the paper

Read The Independent Institution, sections 5. The role and sequence above are practical preparation derived from those sections; any newly proposed commercial arrangement still needs its own confirmed terms.

Computraq participation status

Computraq is in prelaunch. This guide interprets the proposed operating model in The Independent Institution; it does not establish current hosting, paid booking, provider integration, partnership or a guarantee of service. Use the paper, original slides and expression-of-interest form to prepare a concrete contribution.

Sources and verification