Commit Graph

46 Commits

Author SHA1 Message Date
lilong.129
ba43e9fd0e refactor: enhance JSON handling and improve request retry logic in DriverSession 2025-06-27 11:52:53 +08:00
lilong.129
8fc8d06604 feat: unify AI action handling with detailed execution results and enhanced UI integration 2025-06-24 13:42:08 +08:00
lilong.129
fc32b5d874 feat: enhance AI query handling with detailed result structure and improved UI display 2025-06-24 11:50:37 +08:00
lilong.129
f4b60f4d86 fix: update error handling in runStepMobileUI to store error messages as strings 2025-06-20 19:36:01 +08:00
lilong.129
0c9dac95a1 feat: enhance report generation by integrating session data and improving AI query display 2025-06-20 17:38:36 +08:00
lilong.129
e3432a192d refactor: improve validation process by adding variable substitution and enhancing error handling 2025-06-18 23:14:43 +08:00
lilong.129
e40db65287 feat: enhance report generation with new AI query and validation display features 2025-06-18 22:35:19 +08:00
lilong.129
a890981e2d fix: update StartTime to use UnixMilli for better precision across step functions 2025-06-18 13:51:44 +08:00
lilong.129
64ed72057f fix: update step failed in ParseStep 2025-06-18 11:31:28 +08:00
lilong.129
1145f424b1 feat: implement two-level auto popup handler configuration
- Add AutoPopupHandler field to both TConfig and StepConfig
- Support testcase-level global configuration via TConfig.EnableAutoPopupHandler()
- Support step-level specific configuration via StepMobile.EnableAutoPopupHandler()
- Priority: testcase config > step config > default disabled
- Simplify Loops field type from *types.IntOrString to int in StepConfig
- Update documentation to reflect new structure
2025-06-14 12:11:04 +08:00
lilong.129
b271e655b1 feat: add MCP plugin support and optimize AI service configuration
- Add UIXT runner with MCP plugin support
   - Refactor AI service options handling
   - Optimize configuration parsing for LLM and CV services
   - Update dependencies to latest versions
2025-06-13 20:24:57 +08:00
lilong.129
f6e7e970f8 feat: 实现 AIQuery 功能并支持 OutputSchema
- 新增 AIQuery 方法到 StepMobile,支持使用自然语言从屏幕中提取信息
- 实现 AIQuery 在 driver_ext_ai.go 中的完整功能,包括屏幕截图和 LLM 查询
- 添加 OutputSchema 支持,允许用户定义自定义输出格式进行结构化查询
- 新增 ToolAIQuery MCP 工具,完整集成到 MCP 服务器中
- 在 ActionOptions 中添加 OutputSchema 字段和 WithOutputSchema 选项函数
- 添加 ACTION_Query 的配置支持和字段映射
- 完善测试覆盖:
  * 添加 TestAIQuery 单元测试,包含多种 OutputSchema 使用场景
  * 添加 TestToolAIQuery MCP 工具测试
  * 定义 GameInfo、UIElementInfo 等结构体用于测试
- 更新文档:
  * 在 docs/uixt/ai.md 中添加完整的 AIQuery 使用指南
  * 包含基本用法、OutputSchema 示例、最佳实践等
- 支持复杂的嵌套结构体和数组类型的 OutputSchema
- 与现有 AIAction、AIAssert 功能保持一致的 API 设计
2025-06-13 10:27:08 +08:00
lilong.129
dd52faef57 refactor: move Call function 2025-06-09 20:52:32 +08:00
lilong.129
96da4515a1 feat: optimize test report UI and add LLM usage tracking 2025-06-09 17:04:55 +08:00
lilong.129
b9de3cf7a3 refactor: simplify AI action execution and improve sub-action handling 2025-06-08 19:16:37 +08:00
lilong.129
ec4f1eb68a refactor: unify action execution interface and merge AI action handling 2025-06-07 23:59:07 +08:00
lilong.129
5f400735fc fix: 修复 StartToGoal 命令无法通过 CTRL+C 中断的问题
- 为 AI 相关方法添加 context.Context 参数支持中断

- 在重试循环中添加上下文取消检查

- 创建可取消的上下文并监听中断信号

- 更新 MCP 工具调用使用带上下文的方法

现在用户可以通过 CTRL+C 正常中断长时间运行的 AI 自动化任务
2025-06-05 20:00:20 +08:00
lilong.129
c4e7ab00a7 feat: implement ToolStartToGoal and fix LLM service initialization
- Add ToolStartToGoal implementation with AI-driven goal automation
- Fix LLM service not initialized issue by applying global AI config to XTDriver creation
- Ensure XTDriver is created with proper AI services from the first initialization
- Add StartToGoal method to StepMobile for goal-oriented automation
- Register ToolStartToGoal in MCP server and add corresponding action type
- Add comprehensive test case for StartToGoal functionality
- Fix ReturnSchema consistency across AI tools (StartToGoal, AIAction, Finished)
- Extract AI service options in MCP argument processing

This resolves the root cause where XTDriver was created without AI services
in runStepMobileUI, ensuring only one XTDriver initialization with complete
AI service configuration.
2025-06-05 16:52:11 +08:00
lilong.129
bd8cb5abf4 refactor: move MobileAction to option package and update imports
- Move MobileAction struct from uixt package to uixt/option package
- Delete uixt/driver_action.go file as MobileAction is now in option package
- Update all import statements across the codebase to use option.MobileAction
- Update ActionTool interface to use option.MobileAction in ConvertActionToCallToolRequest method
- Maintain backward compatibility while improving package organization
- Clean up code structure by consolidating action-related types in option package

Files affected:
- server/uixt.go: Updated imports and type references
- step.go: Updated imports and ActionResult struct
- step_ui.go: Updated all MobileAction references to option.MobileAction
- uixt/mcp_server.go: Updated ActionTool interface and removed detailed comments
- uixt/mcp_server_test.go: Updated all test cases to use option.MobileAction
- uixt/mcp_tools_*.go: Updated ConvertActionToCallToolRequest method signatures
- uixt/option/action.go: Added MobileAction struct definition
- uixt/sdk.go: Updated ExecuteAction method signature
2025-06-03 18:15:28 +08:00
lilong.129
d3011d467e feat: enhance signal handling and graceful shutdown for MCP integration 2025-05-29 00:59:17 +08:00
lilong.129
c5fb391ef5 feat: add global AntiRisk configuration support
- Add AntiRisk field to TConfig struct for global anti-risk switch
- Add SetAntiRisk method to configure global anti-risk setting
- Implement automatic AntiRisk application in mobile UI steps
- Global AntiRisk setting applies to all actions unless explicitly disabled
- Maintains backward compatibility with existing action-level AntiRisk settings
2025-05-29 00:11:34 +08:00
lilong.129
4ea08b0198 feat: integrate MCP tools with UI actions and improve environment variable inheritance 2025-05-28 22:58:59 +08:00
lilong.129
2fe5b14d63 refactor: integrate and optimize MCP tool calling methods 2025-05-27 21:39:17 +08:00
lilong.129
866cc0e4d2 feat: implement MCP hooks integration with anti_risk option 2025-05-27 19:46:08 +08:00
lilong.129
404865ba6b refactor: complete ActionOptions unification and pointer type optimization 2025-05-27 13:34:12 +08:00
lilong.129
9a5e0849de fix: handle GetOrCreateXTDriver when serial is empty 2025-05-26 21:25:25 +08:00
lilong.129
2569670c7f feat: implement unified XTDriver cache 2025-05-26 19:39:46 +08:00
lilong.129
778344c826 change: remove call function tool 2025-05-26 00:43:01 +08:00
lilong.129
2e17d9df16 refactor: merge DoAction to mcp server tools 2025-05-25 23:53:07 +08:00
lilong.129
7986c4899f refactor: move DoAction to MCP tools call 2025-05-25 08:10:57 +08:00
lilong.129
4ff2692f02 refactor: move action options 2025-05-25 00:15:18 +08:00
lilong.129
0722e92e87 change: remove options for AppLaunch/AppTerminate 2025-05-12 17:36:16 +08:00
lilong.129
7a6890a160 feat: set timeout for Call function 2025-05-12 08:48:54 +08:00
lilong.129
9bafea53af feat: support action options for AppLaunch/AppTerminate 2025-05-10 00:01:30 +08:00
徐聪
dff083e93f fix: delete useless codes 2025-05-07 21:50:27 +08:00
徐聪
2c206697d6 fix: update ACTION_WebCloseTab 2025-05-07 21:41:43 +08:00
徐聪
2b06e4a280 update: fucntion call 2025-05-07 21:39:22 +08:00
徐聪
0bd621ad3c fix: XTDriver 2025-05-07 20:21:56 +08:00
徐聪
6cce5e3c5b fix: web ui test 2025-05-07 20:12:06 +08:00
lilong.129
d2976844fc fix: load testcase panic caused by config options 2025-04-27 11:50:50 +08:00
lilong.129
7d5fdfe490 change: AIAssert 2025-03-25 14:33:07 +08:00
lilong.129
8125191ffb feat: add AIAction, do actions with LLM 2025-03-22 13:20:06 +08:00
lilong.129
a80c8af3f6 fix: unittests 2025-03-05 18:10:41 +08:00
lilong.129
e107389d6e refactor: move uixt pkg 2025-03-05 11:04:02 +08:00
lilong.129
29c0e1cbef refactor: DriverSession 2025-02-17 17:11:18 +08:00
lilong.129
17292cb112 refactor: add tests 2025-02-12 15:16:01 +08:00