Commit Graph

40 Commits

Author SHA1 Message Date
lilong.129
70471d2fb4 fix: enhance logging for interrupted processes and ensure step results are saved in failfast mode 2025-06-25 11:57:09 +08:00
lilong.129
53fad4edc5 refactor: streamline AI assertion result handling by consolidating error management and improving result structure 2025-06-24 23:10:46 +08:00
lilong.129
d0ceeb6c51 refactor: update AI result handling to differentiate content and thought based on result types in report generation 2025-06-24 16:01:50 +08:00
lilong.129
b1719344c0 feat: enhance AI result handling with model name and usage statistics for query, action, and assertion types 2025-06-24 15:25:12 +08:00
lilong.129
8fc8d06604 feat: unify AI action handling with detailed execution results and enhanced UI integration 2025-06-24 13:42:08 +08:00
lilong.129
fc32b5d874 feat: enhance AI query handling with detailed result structure and improved UI display 2025-06-24 11:50:37 +08:00
lilong.129
0c9dac95a1 feat: enhance report generation by integrating session data and improving AI query display 2025-06-20 17:38:36 +08:00
lilong.129
ed5d3127cb fix: add missing action options 2025-06-19 21:57:26 +08:00
lilong.129
e40db65287 feat: enhance report generation with new AI query and validation display features 2025-06-18 22:35:19 +08:00
lilong.129
a890981e2d fix: update StartTime to use UnixMilli for better precision across step functions 2025-06-18 13:51:44 +08:00
lilong.129
b271e655b1 feat: add MCP plugin support and optimize AI service configuration
- Add UIXT runner with MCP plugin support
   - Refactor AI service options handling
   - Optimize configuration parsing for LLM and CV services
   - Update dependencies to latest versions
2025-06-13 20:24:57 +08:00
lilong.129
409cd693f0 refactor: GetScreenshotBase64WithSize 2025-06-13 12:01:21 +08:00
lilong.129
f6e7e970f8 feat: 实现 AIQuery 功能并支持 OutputSchema
- 新增 AIQuery 方法到 StepMobile,支持使用自然语言从屏幕中提取信息
- 实现 AIQuery 在 driver_ext_ai.go 中的完整功能,包括屏幕截图和 LLM 查询
- 添加 OutputSchema 支持,允许用户定义自定义输出格式进行结构化查询
- 新增 ToolAIQuery MCP 工具,完整集成到 MCP 服务器中
- 在 ActionOptions 中添加 OutputSchema 字段和 WithOutputSchema 选项函数
- 添加 ACTION_Query 的配置支持和字段映射
- 完善测试覆盖:
  * 添加 TestAIQuery 单元测试,包含多种 OutputSchema 使用场景
  * 添加 TestToolAIQuery MCP 工具测试
  * 定义 GameInfo、UIElementInfo 等结构体用于测试
- 更新文档:
  * 在 docs/uixt/ai.md 中添加完整的 AIQuery 使用指南
  * 包含基本用法、OutputSchema 示例、最佳实践等
- 支持复杂的嵌套结构体和数组类型的 OutputSchema
- 与现有 AIAction、AIAssert 功能保持一致的 API 设计
2025-06-13 10:27:08 +08:00
lilong.129
81a92ae155 docs: update AI module README with latest features
- Add comprehensive documentation for the new Query functionality
- Update interface method names from Call to Plan for consistency
- Add OpenAI GPT-4O model support documentation
- Include detailed usage examples for basic and custom schema queries
- Add configuration examples for multiple model services
- Document new features like ResetHistory, Usage statistics, and automatic type conversion
- Expand advanced features section with custom output format examples
- Update all code examples to reflect the latest API changes

The documentation now reflects the current state of the AI module with all three core capabilities:
- Planning (renamed from Call)
- Assertion
- Query (new feature)

All examples and configurations are updated to match the latest implementation.
2025-06-10 20:52:44 +08:00
lilong.129
88ae8faee1 feat: enhance VLM response parsing and DOUBAO model support
- Fix JSON extraction logic by prioritizing brace counting method
- Add support for DOUBAO string array coordinate format
- Introduce IS_UI_TARS helper function for model type checking
- Add comprehensive tests for JSON parsing and coordinate handling
- Improve error handling with retry delays for LLM service failures
2025-06-10 15:56:13 +08:00
lilong.129
96da4515a1 feat: optimize test report UI and add LLM usage tracking 2025-06-09 17:04:55 +08:00
lilong.129
14cef72f5a feat: add model name display in AI actions and optimize HTML report
- Add ModelName field to PlanningResult and SubActionResult
- Update HTML report with improved layout and model name display
- Fix elapsed time setting bug and enhance mobile responsiveness
2025-06-08 22:08:51 +08:00
lilong.129
b9de3cf7a3 refactor: simplify AI action execution and improve sub-action handling 2025-06-08 19:16:37 +08:00
lilong.129
ec4f1eb68a refactor: unify action execution interface and merge AI action handling 2025-06-07 23:59:07 +08:00
lilong.129
b642ea004e feat: implement UI automation test history isolation
- Add ResetHistory option to PlanningOptions and ActionOptions
- Implement task completion detection with isTaskFinished() method
- Add executeActions() method to separate action execution logic
- Modify ConversationHistory.Clear() to completely clear all messages including system message
- Refactor StartToGoal() to automatically reset history on first attempt
- Add WithResetHistory() option function for consistent API
- Consolidate test files into driver_ext_ai_test.go with comprehensive test coverage
2025-06-06 15:29:42 +08:00
lilong.129
56831845ca change: fix logs 2025-06-05 20:26:18 +08:00
lilong.129
5f400735fc fix: 修复 StartToGoal 命令无法通过 CTRL+C 中断的问题
- 为 AI 相关方法添加 context.Context 参数支持中断

- 在重试循环中添加上下文取消检查

- 创建可取消的上下文并监听中断信号

- 更新 MCP 工具调用使用带上下文的方法

现在用户可以通过 CTRL+C 正常中断长时间运行的 AI 自动化任务
2025-06-05 20:00:20 +08:00
lilong.129
8cdc71d90b change: RoundToOneDecimal 2025-06-05 17:47:29 +08:00
lilong.129
866cc0e4d2 feat: implement MCP hooks integration with anti_risk option 2025-05-27 19:46:08 +08:00
lilong.129
2e17d9df16 refactor: merge DoAction to mcp server tools 2025-05-25 23:53:07 +08:00
lilong.129
02611d3d5a refactor: uixt MCP Server 2025-05-24 23:22:00 +08:00
lilong.129
b639b4473f test: update unittests 2025-05-24 01:00:30 +08:00
lilong.129
009bfa4ecb refactor: replace ui-tars parser with https://github.com/bytedance/UI-TARS/blob/main/codes/ui_tars/action_parser.py 2025-05-22 22:52:47 +08:00
lilong.129
0c20fe7b02 feat: add argument --with-uixt to start built-in uixt MCP server 2025-05-20 22:36:46 +08:00
lilong.129
037e69315e change: remove unused code 2025-05-20 18:03:54 +08:00
lilong.129
3f1ee03529 refactor: mcphost planner 2025-05-18 21:55:01 +08:00
lilong.129
9c735bd46a feat: save screenshot after action 2025-05-12 18:04:56 +08:00
lilong.129
7fa4155390 refactor: move code 2025-04-27 22:37:48 +08:00
lilong.129
9bcdd5d19a feat: add AIAsert 2025-04-27 22:25:06 +08:00
lilong.129
7d5fdfe490 change: AIAssert 2025-03-25 14:33:07 +08:00
lilong.129
5ebfca7f62 refactor: add AIAction 2025-03-22 12:08:42 +08:00
lilong.129
12e0f7f9a2 feat: save screenshots for PlanNextAction 2025-03-22 01:07:28 +08:00
lilong.129
8a3b6b5c4c feat: appendConversationHistory for ai planner 2025-03-22 00:06:30 +08:00
lilong.129
bbc05513f9 fix: get screen size for planner 2025-03-20 20:52:59 +08:00
lilong.129
55acaceb09 feat: add TapByLLM/PlanNextAction for XTDriver 2025-03-19 21:16:21 +08:00