feat: implement ToolStartToGoal and fix LLM service initialization

- Add ToolStartToGoal implementation with AI-driven goal automation
- Fix LLM service not initialized issue by applying global AI config to XTDriver creation
- Ensure XTDriver is created with proper AI services from the first initialization
- Add StartToGoal method to StepMobile for goal-oriented automation
- Register ToolStartToGoal in MCP server and add corresponding action type
- Add comprehensive test case for StartToGoal functionality
- Fix ReturnSchema consistency across AI tools (StartToGoal, AIAction, Finished)
- Extract AI service options in MCP argument processing

This resolves the root cause where XTDriver was created without AI services
in runStepMobileUI, ensuring only one XTDriver initialization with complete
AI service configuration.
This commit is contained in:
lilong.129
2025-06-05 16:52:11 +08:00
parent 0add3231ff
commit c4e7ab00a7
7 changed files with 199 additions and 19 deletions

View File

@@ -279,9 +279,23 @@ func setupXTDriver(_ context.Context, args map[string]any) (*XTDriver, error) {
platform, _ := args["platform"].(string)
serial, _ := args["serial"].(string)
// Extract AI service options from arguments if provided
var aiOpts []option.AIServiceOption
// Check for LLM service type
if llmService, ok := args["llm_service"].(string); ok && llmService != "" {
aiOpts = append(aiOpts, option.WithLLMService(option.LLMServiceType(llmService)))
}
// Check for CV service type
if cvService, ok := args["cv_service"].(string); ok && cvService != "" {
aiOpts = append(aiOpts, option.WithCVService(option.CVServiceType(cvService)))
}
config := DriverCacheConfig{
Platform: platform,
Serial: serial,
Platform: platform,
Serial: serial,
AIOptions: aiOpts,
}
return GetOrCreateXTDriver(config)
}