Commit Graph

27 Commits

Author SHA1 Message Date
lilong.129
d883aa6a21 change: rename VLM name 2025-06-05 18:09:25 +08:00
lilong.129
0add3231ff refactor: merge ActionSummary and Thought fields to eliminate duplication
- Remove redundant ActionSummary field from PlanningResult struct
- Update parsers to use unified Thought field instead of duplicate fields
- Modify chat interface to display Thought instead of ActionSummary
- Update planner logging to use thought instead of summary
- Adjust prompt templates to use thought field consistently
- Switch test LLM service from UI-TARS to DoubaoVL
- Add default parameter handling for sleep tool
2025-06-05 14:19:09 +08:00
lilong.129
c204542f1f feat: optimize UI-TARS parser with coordinate conversion and action mapping
- Add action mapping for UI-TARS parser to convert action names to option.ActionName
- Implement bounding box to center point coordinate conversion for better accuracy
- Update coordinate normalization to handle coordinates > 1000 properly
- Enhance test cases to verify coordinate scaling and center point conversion
- Improve action argument processing with proper coordinate transformation
- Add comprehensive test coverage for coordinate conversion edge cases

Key improvements:
- Bounding box [x1,y1,x2,y2] now converts to center point [cx,cy] for actions
- Coordinate scaling properly handles different screen resolutions
- Action names are mapped through doubao_1_5_ui_tars_action_mapping
- Enhanced error handling for invalid coordinate formats
2025-06-04 23:16:14 +08:00
lilong.129
4e74247cab fix: miss tool call ID 2025-05-26 09:28:46 +08:00
lilong.129
014140ccc7 change: append tool call message for planner 2025-05-24 10:28:55 +08:00
lilong.129
81c854f963 refactor: merge ai parser 2025-05-24 00:25:44 +08:00
lilong.129
19ddcb40cc change: update ui-tars prompt 2025-05-23 22:05:21 +08:00
lilong.129
009bfa4ecb refactor: replace ui-tars parser with https://github.com/bytedance/UI-TARS/blob/main/codes/ui_tars/action_parser.py 2025-05-22 22:52:47 +08:00
lilong.129
3f1ee03529 refactor: mcphost planner 2025-05-18 21:55:01 +08:00
lilong.129
fcddcfb630 refactor: GetModelConfig 2025-04-30 15:21:17 +08:00
lilong.129
2ae252b52a refactor: merge planner 2025-04-30 14:07:48 +08:00
lilong.129
4d7c7e8aaf refactor: ai asserter 2025-04-29 20:08:22 +08:00
lilong.129
7132eec39e feat: add status code for llm 2025-04-28 21:06:53 +08:00
lilong.129
68dbeb368a refactor: adds a message to the conversation history 2025-04-28 20:12:08 +08:00
lilong.129
7fa4155390 refactor: move code 2025-04-27 22:37:48 +08:00
lilong.129
9bcdd5d19a feat: add AIAsert 2025-04-27 22:25:06 +08:00
lilong.129
84ff75c3b1 change: add tests 2025-04-27 19:13:55 +08:00
lilong.129
70a8ee01f7 refactor: llm planner 2025-04-21 21:33:30 +08:00
lilong.129
ebeae596a7 stash 2025-04-21 14:39:37 +08:00
lilong.129
2ad5c4f6db fix: load env 2025-03-23 10:06:50 +08:00
lilong.129
12e0f7f9a2 feat: save screenshots for PlanNextAction 2025-03-22 01:07:28 +08:00
lilong.129
8a3b6b5c4c feat: appendConversationHistory for ai planner 2025-03-22 00:06:30 +08:00
lilong.129
868acd45ac fix: load jpeg image 2025-03-20 20:39:32 +08:00
lilong.129
da0bdc4fe5 fix: convertCoordinateAction 2025-03-20 18:02:35 +08:00
lilong.129
3801ffb744 feat: load .env file from current working directory upward recursively 2025-03-20 14:23:56 +08:00
lilong.129
b5f3e7ff96 change: remove unused code 2025-03-19 22:47:10 +08:00
lilong.129
55acaceb09 feat: add TapByLLM/PlanNextAction for XTDriver 2025-03-19 21:16:21 +08:00