Tune LLM Inference Performance on Cloud GPU

What you will build

A benchmark harness and a written study, so that someone else could reproduce your numbers and trust your recommendation.

Starting point: you only need a Vultr account. This challenge is self-contained, so you can do it without having done any other.

The requirements

• A load generator you wrote or configured, producing repeatable results
• A baseline run with the exact configuration recorded
• At least four single-variable experiments, each compared against the baseline
• Metrics: time to first token, inter-token latency, tokens per second and p95 latency at three concurrency levels
• A quality check, such as a fixed prompt set, to show tuning did not quietly break answers

Rules of the study

1. Change one thing at a time
2. Warm up before measuring
3. Run each configuration at least three times and report the spread
4. Record GPU memory and utilisation alongside latency

vllm serve MODEL_NAME --max-num-seqs 64 --gpu-memory-utilization 0.90

How you will be judged

• Methodology a reviewer can repeat
• Conclusions supported by the data, including the results that disappointed you
• A final recommendation with its cost per million tokens

GPUs are billed while they exist. Script your runs so you can stop the instance between sessions.