CleanScape (HackGlobal 2026): voice, vision and chat on Vultr, and a question about our demo link

View profile

I’m building a voice assistant (Earthy, inside CleanScape, a community clean-up app). Each spoken question goes through three Vultr Serverless Inference calls in a row from one Cloud Compute instance in Bangalore (FastAPI in Docker):

  1. whisper-large-v3-turbo: speech to text
  2. deepseek-v4.1-flash: the answer, about 30 words
  3. qwen3-tts-12hz-1.7b: the voice

Measured on my server: chat takes 0.8 to 1.6 s depending on the model, and text to speech takes 1.2 to 2.0 s for 25 to 70 characters. With upload and end-of-speech detection, the user waits 3 to 4 seconds after they stop talking. That feels slow in a conversation.

What I’ve tried:

  • Picked the fastest chat model by timing six of them with the same prompt.
  • Capped spoken answers at two short sentences so there is less to voice.
  • Pre-generated answers and audio for common questions at start-up, which makes those instant.
  • Voiced only the first sentence, then the rest in parallel. It started sooner, but joining two clips on the phone caused gaps, so I went back to one clip.
  • Recorded at 16 kHz mono to keep the upload small.

My questions:

  1. Does the /audio/speech endpoint support streaming, so playback can start before the whole clip is generated? If so, which response format works best for that?
  2. Is it sound to stream the chat completion and send each finished sentence to text to speech as it arrives, or do the per-request overheads cancel the gain for answers this short?
  3. Does the region of my Compute instance affect inference latency, or is inference served from a fixed location?
  4. Is there a speech-to-speech or realtime model on Serverless Inference that would replace the three-step chain?

Any measurements from similar pipelines would help.

Answer
MG
1 answer

Join Vultr Community

Points, levels & badges

Enter challenges & submit builds

Register to events & meetups

Get your work featured

1 Answer

0
Answered: 13 hours ago

Great work on CleanScape—your benchmarking approach is spot on. Here is how you can shave 1.5–2+ seconds off your voice pipeline latency on Vultr:

  1. Streaming TTS (/audio/speech): Since Vultr Serverless Inference exposes an OpenAI-compatible API, pass stream=True and set response_format="pcm" (raw 16-bit PCM) or "mp3". Raw PCM works best for mobile playback because you can feed byte chunks directly into an audio buffer (AudioTrack on Android / AVAudioEngine on iOS) with zero container header parsing or frame-boundary gaps.
  2. Pipelining Chat Stream → TTS: Yes, streaming chat (stream=True) and firing TTS on the first sentence boundary (. / ? / !) is worth it—provided you reuse a persistent httpx.AsyncClient connection pool so you don't pay repeated TLS handshake overheads. To fix the audio gap you saw on mobile when joining clips, queue the decoded raw PCM chunks into a single continuous playback stream rather than playing two separate .mp3/.wav files back-to-back.
  3. Compute Region vs. Inference Latency: Vultr Serverless Inference runs on dedicated GPU clusters hosted in select core US/EU data centers rather than every regional edge location (like Bangalore). While keeping your FastAPI instance in Bangalore minimizes the initial user-to-server audio upload latency, your server-to-inference hop crosses international backbone routes (~150–250 ms per round trip). Reusing persistent HTTP/2 connections (keepalive) cuts that handshake tax down to a single RTT.
  4. Native Speech-to-Speech Models: Vultr Serverless Inference currently focuses on discrete OpenAI-compatible endpoints (ASR, Chat Completions, and TTS) rather than stateful WebSocket Realtime speech-to-speech models. Keeping your decoupled 3-step streaming pipeline—with persistent connection pooling and PCM audio streaming—is the fastest production pattern on the platform right now.

Also, regarding your demo link: make sure your FastAPI port is exposed through Nginx/Caddy with buffering disabled (proxy_buffering off; in Nginx) so your streamed audio chunks aren't held up by the reverse proxy!

0

Post Answer