Commit Graph

17 Commits

Author SHA1 Message Date
lilong.129
ba43e9fd0e refactor: enhance JSON handling and improve request retry logic in DriverSession 2025-06-27 11:52:53 +08:00
lilong.129
0491299245 merge master 2025-06-25 12:07:38 +08:00
徐聪
470bde97d7 fix: failed to return err 2025-06-25 11:13:56 +08:00
lilong.129
d0ceeb6c51 refactor: update AI result handling to differentiate content and thought based on result types in report generation 2025-06-24 16:01:50 +08:00
lilong.129
8fc8d06604 feat: unify AI action handling with detailed execution results and enhanced UI integration 2025-06-24 13:42:08 +08:00
lilong.129
fc32b5d874 feat: enhance AI query handling with detailed result structure and improved UI display 2025-06-24 11:50:37 +08:00
lilong.129
58befd6eae refactor: rename buildMCPCallToolRequest to BuildMCPCallToolRequest for consistency across the codebase 2025-06-22 22:54:12 +08:00
lilong.129
b271e655b1 feat: add MCP plugin support and optimize AI service configuration
- Add UIXT runner with MCP plugin support
   - Refactor AI service options handling
   - Optimize configuration parsing for LLM and CV services
   - Update dependencies to latest versions
2025-06-13 20:24:57 +08:00
lilong.129
f6e7e970f8 feat: 实现 AIQuery 功能并支持 OutputSchema
- 新增 AIQuery 方法到 StepMobile,支持使用自然语言从屏幕中提取信息
- 实现 AIQuery 在 driver_ext_ai.go 中的完整功能,包括屏幕截图和 LLM 查询
- 添加 OutputSchema 支持,允许用户定义自定义输出格式进行结构化查询
- 新增 ToolAIQuery MCP 工具,完整集成到 MCP 服务器中
- 在 ActionOptions 中添加 OutputSchema 字段和 WithOutputSchema 选项函数
- 添加 ACTION_Query 的配置支持和字段映射
- 完善测试覆盖:
  * 添加 TestAIQuery 单元测试,包含多种 OutputSchema 使用场景
  * 添加 TestToolAIQuery MCP 工具测试
  * 定义 GameInfo、UIElementInfo 等结构体用于测试
- 更新文档:
  * 在 docs/uixt/ai.md 中添加完整的 AIQuery 使用指南
  * 包含基本用法、OutputSchema 示例、最佳实践等
- 支持复杂的嵌套结构体和数组类型的 OutputSchema
- 与现有 AIAction、AIAssert 功能保持一致的 API 设计
2025-06-13 10:27:08 +08:00
lilong.129
b9de3cf7a3 refactor: simplify AI action execution and improve sub-action handling 2025-06-08 19:16:37 +08:00
lilong.129
ec4f1eb68a refactor: unify action execution interface and merge AI action handling 2025-06-07 23:59:07 +08:00
lilong.129
6e1bd5bbe2 feat: optimize MCP tools response format with automatic schema generation
- Remove all manual ReturnSchema() methods from tools
- Implement automatic schema generation using reflection
- Unify response format to flat structure with action/success/message fields
- Simplify tool implementation by removing MCPResponse embedding
- Update documentation to reflect new architecture
- Achieve ~70% code reduction while maintaining type safety
2025-06-05 23:17:06 +08:00
lilong.129
56831845ca change: fix logs 2025-06-05 20:26:18 +08:00
lilong.129
5f400735fc fix: 修复 StartToGoal 命令无法通过 CTRL+C 中断的问题
- 为 AI 相关方法添加 context.Context 参数支持中断

- 在重试循环中添加上下文取消检查

- 创建可取消的上下文并监听中断信号

- 更新 MCP 工具调用使用带上下文的方法

现在用户可以通过 CTRL+C 正常中断长时间运行的 AI 自动化任务
2025-06-05 20:00:20 +08:00
lilong.129
c4e7ab00a7 feat: implement ToolStartToGoal and fix LLM service initialization
- Add ToolStartToGoal implementation with AI-driven goal automation
- Fix LLM service not initialized issue by applying global AI config to XTDriver creation
- Ensure XTDriver is created with proper AI services from the first initialization
- Add StartToGoal method to StepMobile for goal-oriented automation
- Register ToolStartToGoal in MCP server and add corresponding action type
- Add comprehensive test case for StartToGoal functionality
- Fix ReturnSchema consistency across AI tools (StartToGoal, AIAction, Finished)
- Extract AI service options in MCP argument processing

This resolves the root cause where XTDriver was created without AI services
in runStepMobileUI, ensuring only one XTDriver initialization with complete
AI service configuration.
2025-06-05 16:52:11 +08:00
lilong.129
bd8cb5abf4 refactor: move MobileAction to option package and update imports
- Move MobileAction struct from uixt package to uixt/option package
- Delete uixt/driver_action.go file as MobileAction is now in option package
- Update all import statements across the codebase to use option.MobileAction
- Update ActionTool interface to use option.MobileAction in ConvertActionToCallToolRequest method
- Maintain backward compatibility while improving package organization
- Clean up code structure by consolidating action-related types in option package

Files affected:
- server/uixt.go: Updated imports and type references
- step.go: Updated imports and ActionResult struct
- step_ui.go: Updated all MobileAction references to option.MobileAction
- uixt/mcp_server.go: Updated ActionTool interface and removed detailed comments
- uixt/mcp_server_test.go: Updated all test cases to use option.MobileAction
- uixt/mcp_tools_*.go: Updated ConvertActionToCallToolRequest method signatures
- uixt/option/action.go: Added MobileAction struct definition
- uixt/sdk.go: Updated ExecuteAction method signature
2025-06-03 18:15:28 +08:00
lilong.129
1cc4b1cf5b refactor: modularize MCP server tools by functionality
- Split large mcp_server.go into modular files by functionality
- Create dedicated files for each tool category:
  - mcp_tools_device.go: Device management tools
  - mcp_tools_touch.go: Touch operation tools  
  - mcp_tools_swipe.go: Swipe and drag operation tools
  - mcp_tools_input.go: Input and IME tools
  - mcp_tools_button.go: Button operation tools
  - mcp_tools_app.go: Application management tools
  - mcp_tools_screen.go: Screen operation tools
  - mcp_tools_utility.go: Utility tools (sleep, popups)
  - mcp_tools_web.go: Web operation tools
  - mcp_tools_ai.go: AI-driven operation tools
- Update mcp_server.md documentation to reflect modular architecture
- Maintain pure ActionTool architecture with complete tool decoupling
- Improve code organization and maintainability
2025-06-03 15:45:42 +08:00