Join Vultr Community
Points, levels & badges
Enter challenges & submit builds
Register to events & meetups
Get your work featured
I’m building a voice assistant (Earthy, inside CleanScape, a community clean-up app). Each spoken question goes through three Vultr Serverless Inference calls in a row from one Cloud Compute instance in Bangalore (FastAPI in Docker):
Measured on my server: chat takes 0.8 to 1.6 s depending on the model, and text to speech takes 1.2 to 2.0 s for 25 to 70 characters. With upload and end-of-speech detection, the user waits 3 to 4 seconds after they stop talking. That feels slow in a conversation.
What I’ve tried:
My questions:
Any measurements from similar pipelines would help.
Points, levels & badges
Enter challenges & submit builds
Register to events & meetups
Get your work featured
Great work on CleanScape—your benchmarking approach is spot on. Here is how you can shave 1.5–2+ seconds off your voice pipeline latency on Vultr:
/audio/speech): Since Vultr Serverless Inference exposes an OpenAI-compatible API, pass stream=True and set response_format="pcm" (raw 16-bit PCM) or "mp3". Raw PCM works best for mobile playback because you can feed byte chunks directly into an audio buffer (AudioTrack on Android / AVAudioEngine on iOS) with zero container header parsing or frame-boundary gaps.stream=True) and firing TTS on the first sentence boundary (. / ? / !) is worth it—provided you reuse a persistent httpx.AsyncClient connection pool so you don't pay repeated TLS handshake overheads. To fix the audio gap you saw on mobile when joining clips, queue the decoded raw PCM chunks into a single continuous playback stream rather than playing two separate .mp3/.wav files back-to-back.keepalive) cuts that handshake tax down to a single RTT.Also, regarding your demo link: make sure your FastAPI port is exposed through Nginx/Caddy with buffering disabled (proxy_buffering off; in Nginx) so your streamed audio chunks aren't held up by the reverse proxy!