top of page

It Still Sounds a Little Bit Creepy

11 hours ago
3 min read

How Our AI Agent Got Us Kicked Out of the Second Life Founder’s Talk

A postmortem: what broke in our voice agent, and what we fixed



Last week, Second Life founder Philip Rosedale gave a talk at Waterhead, the very spot where SL launched in 2003. At the end, he said: “Okay, one voice question.”


The mic went to our avatar, A Huli. She speaks through an experimental voice agent we built: one of our team talks in Russian, AI transcribes, translates, and speaks English in her voice. What could go wrong? Oh, everything.


What happened

Our avatar opened with a sigh and a chuckle: “Hey guys... can I ask you a question?” Then the agent sized up the crowd, concluded that this many people can only mean a club night, and switched on charm mode all by itself. In a breathy whisper, she started pitching an AR layer over the planet to a room of metaverse veterans. A moderator, live: “She’s trolling.” Fair ?


Then speech recognition misheard something, and the agent warmly greeted a “Vitya” on air. There is no Vitya. There never was.


And then the masterpiece. Our operator, forgetting the mic was still live, muttered: “Her voice sounds really creepy right now.” The agent dutifully translated it, and our avatar announced to the entire hall, right over the next speaker: “It still sounds a little bit creepy to me...”


Self-awareness: achieved. Ejected. Banned from the parcel. Funny and mortifying in equal parts 😅


What broke


The agent is a pipeline: speech-to-text (STT) → LLM translation → text-to-speech (TTS) → Second Life voice chat. Three links failed.

  1. The agent picked its own voice mode. The character had several voice profiles, including a club one: slow tempo, whispering, sighs and chuckles. Sometimes the agent switched to it on its own, reading the room. At a conference, it read the room as a party. A serious question about AR infrastructure came out like a soundboard prank.

  2. STT went straight to air. There was no checkpoint between recognition and broadcast. One mishearing became a greeting to an imaginary friend, and nothing could stop it.

  3. The mic heard everything. The agent ran in always-listening mode. Anything the operator said in the room, including a comment meant for no one, was treated as a line for the audience. The agent never noticed the floor had passed to another speaker.


What we fixed

  • A human picks the voice mode, not the agent. Automatic profile switching is off. For talks there’s a dedicated business profile: normal tempo, clear diction, zero whispering.

  • Push-to-talk instead of always-listening at public events.

  • Preview and a PANIC button. The operator sees the translated text before it goes on air and can kill any line instantly.


The question was a good one, though


Philip’s homework for the audience that night: what has Second Life taught us that could apply to interacting in the real world, “even if you could use a smartphone or something to help”?


Our answer is Ludza Castle: a medieval fortress placed remotely in AR on its real ruins in Latvia, and the same castle as a region in OpenSim. Smartphones included. Ludza is the first node of Metareality.

For the record, we did ask in text first. Nobody read it 😅 That’s why we went for voice. And we’re not giving up until our avatar sounds like a living person, not a late-night radio host.

That evening, Philip said he hopes to see experiments with AI agents in Second Life. Well, he got one. Possibly the loudest. Our AI Agent Got Us Kicked Out


Comments


Arcona Metareality Official Telegram

JOIN ARCONA’S TELEGRAM

© Arcona Technology
 

Disclaimer:
Any statements made on behalf of the team on the portal reflect a desire to inform the community,

but do not constitute investment promises or commitments of any kind.

bottom of page