lilong.129
739fb42d27
fix: unittest
2025-07-21 16:53:38 +08:00
lilong.129
4890c098fa
change: remove addDeviceContextForWings
2025-07-21 15:14:11 +08:00
lilong.129
318c942860
change: addDeviceContextForWings
2025-07-21 14:52:53 +08:00
lilong.129
9890588ca7
fix: support combined LLMService with wings service
2025-07-20 23:33:49 +08:00
余泓铮
9c7f4b9275
merge master
2025-07-09 15:53:52 +08:00
余泓铮
813a4b7646
feat: 支持wings断言和自动化
2025-07-09 15:45:56 +08:00
lilong.129
d329fb610f
fix: display ai assert in html report
2025-07-06 11:08:52 +08:00
lilong.129
7fac0dcc00
fix: cannot unmarshal object into Go struct field SubActionResult error
2025-07-04 20:46:11 +08:00
lilong.129
147020cbe5
feat: add time limit for StartToGoal
2025-07-04 14:23:09 +08:00
lilong.129
7393f7aab3
refactor: BuildMCPCallToolRequest
2025-07-03 18:18:34 +08:00
lilong.129
3cd4e5b836
change: update examples
2025-06-30 16:11:55 +08:00
lilong.129
ae0d28c26d
feat: add action timeout for StartToGoal
2025-06-30 13:55:17 +08:00
lilong.129
16bb91a098
feat: add action timeout for StartToGoal
2025-06-30 13:26:11 +08:00
lilong.129
0b6e764c9f
refactor: enhance context cancellation handling in mobile UI and driver extensions
2025-06-30 13:04:00 +08:00
lilong.129
7aa4ad652c
fix: wait 3 seconds for tool calls to complete
2025-06-29 21:42:07 +08:00
lilong.129
a040b53ea9
refactor: enhance screenshot handling by introducing base64 encoding and updating related methods
2025-06-29 20:38:33 +08:00
lilong.129
5baabee89c
fix: update invokeToolCall to accept options and refactor action type handling in MarkUIOperation
2025-06-29 15:59:48 +08:00
lilong.129
70471d2fb4
fix: enhance logging for interrupted processes and ensure step results are saved in failfast mode
2025-06-25 11:57:09 +08:00
lilong.129
53fad4edc5
refactor: streamline AI assertion result handling by consolidating error management and improving result structure
2025-06-24 23:10:46 +08:00
lilong.129
d0ceeb6c51
refactor: update AI result handling to differentiate content and thought based on result types in report generation
2025-06-24 16:01:50 +08:00
lilong.129
b1719344c0
feat: enhance AI result handling with model name and usage statistics for query, action, and assertion types
2025-06-24 15:25:12 +08:00
lilong.129
8fc8d06604
feat: unify AI action handling with detailed execution results and enhanced UI integration
2025-06-24 13:42:08 +08:00
lilong.129
fc32b5d874
feat: enhance AI query handling with detailed result structure and improved UI display
2025-06-24 11:50:37 +08:00
lilong.129
0c9dac95a1
feat: enhance report generation by integrating session data and improving AI query display
2025-06-20 17:38:36 +08:00
lilong.129
ed5d3127cb
fix: add missing action options
2025-06-19 21:57:26 +08:00
lilong.129
e40db65287
feat: enhance report generation with new AI query and validation display features
2025-06-18 22:35:19 +08:00
lilong.129
a890981e2d
fix: update StartTime to use UnixMilli for better precision across step functions
2025-06-18 13:51:44 +08:00
lilong.129
b271e655b1
feat: add MCP plugin support and optimize AI service configuration
...
- Add UIXT runner with MCP plugin support
- Refactor AI service options handling
- Optimize configuration parsing for LLM and CV services
- Update dependencies to latest versions
2025-06-13 20:24:57 +08:00
lilong.129
409cd693f0
refactor: GetScreenshotBase64WithSize
2025-06-13 12:01:21 +08:00
lilong.129
f6e7e970f8
feat: 实现 AIQuery 功能并支持 OutputSchema
...
- 新增 AIQuery 方法到 StepMobile，支持使用自然语言从屏幕中提取信息
- 实现 AIQuery 在 driver_ext_ai.go 中的完整功能，包括屏幕截图和 LLM 查询
- 添加 OutputSchema 支持，允许用户定义自定义输出格式进行结构化查询
- 新增 ToolAIQuery MCP 工具，完整集成到 MCP 服务器中
- 在 ActionOptions 中添加 OutputSchema 字段和 WithOutputSchema 选项函数
- 添加 ACTION_Query 的配置支持和字段映射
- 完善测试覆盖：
* 添加 TestAIQuery 单元测试，包含多种 OutputSchema 使用场景
* 添加 TestToolAIQuery MCP 工具测试
* 定义 GameInfo、UIElementInfo 等结构体用于测试
- 更新文档：
* 在 docs/uixt/ai.md 中添加完整的 AIQuery 使用指南
* 包含基本用法、OutputSchema 示例、最佳实践等
- 支持复杂的嵌套结构体和数组类型的 OutputSchema
- 与现有 AIAction、AIAssert 功能保持一致的 API 设计
2025-06-13 10:27:08 +08:00
lilong.129
81a92ae155
docs: update AI module README with latest features
...
- Add comprehensive documentation for the new Query functionality
- Update interface method names from Call to Plan for consistency
- Add OpenAI GPT-4O model support documentation
- Include detailed usage examples for basic and custom schema queries
- Add configuration examples for multiple model services
- Document new features like ResetHistory, Usage statistics, and automatic type conversion
- Expand advanced features section with custom output format examples
- Update all code examples to reflect the latest API changes
The documentation now reflects the current state of the AI module with all three core capabilities:
- Planning (renamed from Call)
- Assertion
- Query (new feature)
All examples and configurations are updated to match the latest implementation.
2025-06-10 20:52:44 +08:00
lilong.129
88ae8faee1
feat: enhance VLM response parsing and DOUBAO model support
...
- Fix JSON extraction logic by prioritizing brace counting method
- Add support for DOUBAO string array coordinate format
- Introduce IS_UI_TARS helper function for model type checking
- Add comprehensive tests for JSON parsing and coordinate handling
- Improve error handling with retry delays for LLM service failures
2025-06-10 15:56:13 +08:00
lilong.129
96da4515a1
feat: optimize test report UI and add LLM usage tracking
2025-06-09 17:04:55 +08:00
lilong.129
14cef72f5a
feat: add model name display in AI actions and optimize HTML report
...
- Add ModelName field to PlanningResult and SubActionResult
- Update HTML report with improved layout and model name display
- Fix elapsed time setting bug and enhance mobile responsiveness
2025-06-08 22:08:51 +08:00
lilong.129
b9de3cf7a3
refactor: simplify AI action execution and improve sub-action handling
2025-06-08 19:16:37 +08:00
lilong.129
ec4f1eb68a
refactor: unify action execution interface and merge AI action handling
2025-06-07 23:59:07 +08:00
lilong.129
b642ea004e
feat: implement UI automation test history isolation
...
- Add ResetHistory option to PlanningOptions and ActionOptions
- Implement task completion detection with isTaskFinished() method
- Add executeActions() method to separate action execution logic
- Modify ConversationHistory.Clear() to completely clear all messages including system message
- Refactor StartToGoal() to automatically reset history on first attempt
- Add WithResetHistory() option function for consistent API
- Consolidate test files into driver_ext_ai_test.go with comprehensive test coverage
2025-06-06 15:29:42 +08:00
lilong.129
56831845ca
change: fix logs
2025-06-05 20:26:18 +08:00
lilong.129
5f400735fc
fix: 修复 StartToGoal 命令无法通过 CTRL+C 中断的问题
...
- 为 AI 相关方法添加 context.Context 参数支持中断
- 在重试循环中添加上下文取消检查
- 创建可取消的上下文并监听中断信号
- 更新 MCP 工具调用使用带上下文的方法
现在用户可以通过 CTRL+C 正常中断长时间运行的 AI 自动化任务
2025-06-05 20:00:20 +08:00
lilong.129
8cdc71d90b
change: RoundToOneDecimal
2025-06-05 17:47:29 +08:00
lilong.129
866cc0e4d2
feat: implement MCP hooks integration with anti_risk option
2025-05-27 19:46:08 +08:00
lilong.129
2e17d9df16
refactor: merge DoAction to mcp server tools
2025-05-25 23:53:07 +08:00
lilong.129
02611d3d5a
refactor: uixt MCP Server
2025-05-24 23:22:00 +08:00
lilong.129
b639b4473f
test: update unittests
2025-05-24 01:00:30 +08:00
lilong.129
009bfa4ecb
refactor: replace ui-tars parser with https://github.com/bytedance/UI-TARS/blob/main/codes/ui_tars/action_parser.py
2025-05-22 22:52:47 +08:00
lilong.129
0c20fe7b02
feat: add argument --with-uixt to start built-in uixt MCP server
2025-05-20 22:36:46 +08:00
lilong.129
037e69315e
change: remove unused code
2025-05-20 18:03:54 +08:00
lilong.129
3f1ee03529
refactor: mcphost planner
2025-05-18 21:55:01 +08:00
lilong.129
9c735bd46a
feat: save screenshot after action
2025-05-12 18:04:56 +08:00
lilong.129
7fa4155390
refactor: move code
2025-04-27 22:37:48 +08:00
