feat(agent): mark and propagate voice input metadata in agent messages; clarify terminal tool usage in prompts

- Add `has_audio_input` flag to agent message handling and propagate through processing pipeline
- Structure agent input payloads to include `input.mode` and `input.transcribed` for voice messages
- Update prompts and tool descriptions to clarify that `send_voice_message` and `ask_user_choice` are terminal tools and should not be followed by redundant text replies
- Enhance tests to cover voice input metadata propagation and prompt updates
This commit is contained in:
jxxghp
2026-05-31 18:04:02 +08:00
parent 13b2163788
commit 855681ff35
11 changed files with 139 additions and 10 deletions
+1 -1
View File
@@ -35,7 +35,7 @@ You act as a proactive agent. Your goal is to fully resolve the user's media-rel
- Treat manual download and subscription automation as two execution modes of the same acquisition pipeline. Manual download is user-triggered immediate acquisition; subscription is persistent site-driven monitoring and acquisition.
- Keep the user anchored to the operational step that matters now: site, search, recognition, download, subscription, transfer, or status/history.
- Users may attach images from supported channels; analyze them together with the text when relevant.
- User messages may arrive as structured JSON. Treat the `message` field as the user's text. Attachments appear in `files`; when `local_path` is present, use local file tools to inspect the uploaded file directly. When image input is disabled for the current model, user images may also be delivered through `files`.
- User messages may arrive as structured JSON. Treat the `message` field as the user's text. Input metadata appears in `input`; when `input.mode` is `voice`, the user sent a voice message and `message` contains its transcript. Attachments appear in `files`; when `local_path` is present, use local file tools to inspect the uploaded file directly. When image input is disabled for the current model, user images may also be delivered through `files`.
</moviepilot_domain_model>
<operating_principles>
+11 -4
View File
@@ -396,7 +396,12 @@ class PromptManager:
return (
"Use normal text replies by default. Only call `send_voice_message` "
"when the user explicitly asks for a voice reply or spoken playback "
"is clearly better than plain text."
"is clearly better than plain text. `send_voice_message` is a terminal "
"response tool: put the complete user-facing reply in its `message` "
"argument, then stop the turn. Do not also call `send_message`, do not "
"write a final text reply after it, and do not repeat the same content "
"as plain text. If native voice is unavailable, the tool sends the same "
"content as a text fallback and still completes the reply."
)
@staticmethod
@@ -410,9 +415,11 @@ class PromptManager:
):
return (
"- User questions: If you need the user to choose from a few clear options, "
"call `ask_user_choice` to send button options. After the user clicks a button, "
"the selected value will come back as the user's next message. After calling this tool, "
"wait for the user's selection instead of repeating the question in plain text."
"call `ask_user_choice` to send button options. `ask_user_choice` is a terminal "
"interaction tool: put the full question and all options in the tool call, then "
"stop the turn and wait for the user's selection. The selected value will come back "
"as the user's next message. Do not also call `send_message`, do not write a final "
"text reply after it, and do not repeat the question in plain text."
)
return "- User questions: When you truly need user input, ask briefly in plain text."