Commit Graph

4465 Commits

Author SHA1 Message Date
lilong.129
a47d65bc4e feat: migrate all remaining MCP tools to use UnifiedActionRequest
- Migrated 39 remaining MCP tools from individual Request structures to UnifiedActionRequest
- All tools now use unifiedReq.GetMCPOptions(ACTION_*) instead of option.NewMCPOptions(Request{})
- Completed the unification of parameter definitions across all 40 MCP tools
- Eliminated duplication between ActionOptions and Request structures
- All tests pass, confirming successful migration

Tools migrated:
- Basic operations: TapAbsXY, TapByOCR, TapByCV, DoubleTapXY
- Device management: ListPackages, ScreenShot, GetScreenSize, PressButton
- App management: LaunchApp, TerminateApp, AppInstall, AppUninstall, AppClear
- Swipe operations: SwipeDirection, SwipeCoordinate, SwipeToTapApp, SwipeToTapText, SwipeToTapTexts
- Input/Navigation: Input, Home, Back, Drag
- Web operations: WebLoginNoneUI, SecondaryClick, HoverBySelector, TapBySelector, SecondaryClickBySelector, WebCloseTab
- System utilities: SetIme, GetSource, ClosePopups
- Sleep operations: SleepMS, SleepRandom
- AI/Task management: AIAction, Finished

This completes the ActionOptions and Request structures integration initiative.
2025-05-26 23:10:58 +08:00
lilong.129
8181e4244a refactor ToolSwipe to delegate to existing tools instead of duplicating logic
- Modified ToolSwipe.ConvertActionToCallToolRequest to delegate to ToolSwipeDirection and ToolSwipeCoordinate
- Removed duplicate parameter handling logic in favor of reusing existing implementations
- Fixed linter error by removing unused variable
- Maintained backward compatibility while reducing code duplication
- All tests pass, confirming the refactoring is successful
2025-05-26 22:42:50 +08:00
lilong.129
6ae4c300c1 add generic swipe tool with auto-detection of direction vs coordinate params
- Added ACTION_Swipe to option/action.go for generic swipe functionality
- Implemented ToolSwipe in mcp_server.go that automatically detects parameter type:
  - String params (up/down/left/right) use direction-based swipe logic
  - Array params [fromX, fromY, toX, toY] use coordinate-based swipe logic
- Added comprehensive test coverage for ToolSwipe in mcp_server_test.go
- Updated tool registration to include the new generic swipe tool
- All tests pass, confirming backward compatibility with existing tools
2025-05-26 22:39:23 +08:00
lilong.129
8895e9e970 merge mcp_tools_test.go into mcp_server_test.go
- Merged all individual MCP tool test functions from mcp_tools_test.go into mcp_server_test.go
- Added require import for additional test assertions
- Removed duplicate TestMCPServer4XTDriver function
- Deleted the original mcp_tools_test.go file
- All 39 MCP tools now have comprehensive unit tests in a single file
- Tests cover tool name, description, options, and request conversion functionality
2025-05-26 22:32:26 +08:00
lilong.129
77f5683f9a fix: remove unnecessary IgnoreNotFoundError and MaxRetryTimes from coordinate-based tap tools
- Removed IgnoreNotFoundError and MaxRetryTimes parameters from TapRequest, TapAbsXYRequest, and DoubleTapXYRequest structures
- Updated corresponding tool implementations to remove references to these non-existent fields
- These parameters are not applicable to coordinate-based operations as they don't involve element searching
- Only OCR/CV-based operations need these error handling parameters

This ensures that only relevant tools have the ignore_NotFoundError functionality,
making the API more consistent and avoiding confusion.
2025-05-26 22:10:08 +08:00
lilong.129
df65f9a828 fix: MCP server ignore_NotFoundError option not working
- Fixed TapByOCR and TapByCV tools to properly handle ignore_NotFoundError option
- Added option parameters to all MCP tool request structures
- Fixed ConvertActionToCallToolRequest methods to extract action options
- Added extractActionOptionsToArguments helper function for consistent option handling
- Extended fix to all MCP tools: SwipeToTapApp, SwipeToTapText, SwipeToTapTexts, TapXY, TapAbsXY
- Added comprehensive tests for option parameter handling
- Updated test expectations to match actual registered tools

This ensures that when ignore_NotFoundError is set to true, OCR/CV operations
will return nil instead of throwing errors when target elements are not found,
allowing tests to continue execution as expected.
2025-05-26 22:02:01 +08:00
lilong.129
9a5e0849de fix: handle GetOrCreateXTDriver when serial is empty 2025-05-26 21:25:25 +08:00
lilong.129
2569670c7f feat: implement unified XTDriver cache 2025-05-26 19:39:46 +08:00
lilong.129
1bd2b1ba5e change: move code 2025-05-26 16:08:27 +08:00
lilong.129
7045a9d452 change: check call tool result error 2025-05-26 15:30:51 +08:00
lilong.129
5eafcc8a2b merge master 2025-05-26 14:53:36 +08:00
lilong.129
f20fdd51bc feat: Validate model type and model name compatibility 2025-05-26 09:40:28 +08:00
lilong.129
4e74247cab fix: miss tool call ID 2025-05-26 09:28:46 +08:00
lilong.129
36c5044402 feat: add mcp tool finished 2025-05-26 09:05:48 +08:00
lilong.129
e60c362257 change: rename function 2025-05-26 08:49:06 +08:00
lilong.129
778344c826 change: remove call function tool 2025-05-26 00:43:01 +08:00
lilong.129
a888022cbc refactor: split driver cache 2025-05-26 00:35:56 +08:00
lilong.129
2e17d9df16 refactor: merge DoAction to mcp server tools 2025-05-25 23:53:07 +08:00
lilong.129
7986c4899f refactor: move DoAction to MCP tools call 2025-05-25 08:10:57 +08:00
lilong.129
4ff2692f02 refactor: move action options 2025-05-25 00:15:18 +08:00
lilong.129
97dad38b7b refactor: move tool request types to option 2025-05-24 23:51:58 +08:00
lilong.129
f65d8aebbd refactor: move model struct to types 2025-05-24 23:32:10 +08:00
lilong.129
02611d3d5a refactor: uixt MCP Server 2025-05-24 23:22:00 +08:00
lilong.129
0a68701faa refactor: move uixt mcp server 2025-05-24 10:46:50 +08:00
lilong.129
014140ccc7 change: append tool call message for planner 2025-05-24 10:28:55 +08:00
lilong.129
b639b4473f test: update unittests 2025-05-24 01:00:30 +08:00
lilong.129
81c854f963 refactor: merge ai parser 2025-05-24 00:25:44 +08:00
lilong.129
19ddcb40cc change: update ui-tars prompt 2025-05-23 22:05:21 +08:00
lilong.129
009bfa4ecb refactor: replace ui-tars parser with https://github.com/bytedance/UI-TARS/blob/main/codes/ui_tars/action_parser.py 2025-05-22 22:52:47 +08:00
lilong.129
3b77ade24f refactor: json asserter 2025-05-22 18:22:12 +08:00
lilong.129
c377664518 refactor: add LLMServiceTypeDoubaoVL 2025-05-22 15:34:11 +08:00
lilong.129
269fe2de23 fix: tap_xy, swipe handler 2025-05-21 22:51:51 +08:00
lilong.129
bb592548b4 fix: chat with screenshot 2025-05-21 22:35:16 +08:00
lilong.129
d58bbaeb5f fix: uixt tool take_screenshot 2025-05-21 18:05:12 +08:00
lilong.129
60e608f101 feat: add uixt tool swipe 2025-05-21 17:47:53 +08:00
lilong.129
7724cf0062 feat: add uixt tool press_button 2025-05-21 17:25:17 +08:00
lilong.129
5c68760cca feat: add uixt tool get_screen_size 2025-05-21 16:51:39 +08:00
lilong.129
0d6a37ecef feat: add uixt tool terminate_app 2025-05-21 16:42:53 +08:00
lilong.129
03553a4962 feat: add uixt tool launch_app 2025-05-21 16:37:42 +08:00
lilong.129
495443a2c4 feat: add uixt tool list_packages 2025-05-21 16:24:54 +08:00
lilong.129
044eb07a35 feat: add uixt tool select_device 2025-05-20 23:11:22 +08:00
lilong.129
1fa87819ae feat: add uixt tool list_available_devices 2025-05-20 22:58:16 +08:00
lilong.129
0c20fe7b02 feat: add argument --with-uixt to start built-in uixt MCP server 2025-05-20 22:36:46 +08:00
lilong.129
037e69315e change: remove unused code 2025-05-20 18:03:54 +08:00
lilong.129
83434cca1e feat: load uixt mcp server in mcphost 2025-05-20 17:07:33 +08:00
lilong.129
5066c64368 refactor: move server models to uixt/types 2025-05-20 14:21:18 +08:00
lilong.129
2f48a92f7f feat: add mcp server for uixt tap/swipe/screenshot actions 2025-05-20 13:26:55 +08:00
徐聪
e4ab10351e Merge 'fix/xucong.053/ai_assert' into 'master'
fix: failed to use ai_assert

See merge request: !89
2025-05-19 07:46:17 +00:00
徐聪
46dbf37db7 fix: failed to use ai_assert 2025-05-19 15:31:36 +08:00
lilong.129
b2ab14efcc refactor: rename to AssertionResult 2025-05-19 11:51:49 +08:00