Microsoft finetuned Qwen 27B for computer use (huggingface.co)

Microsoft Research has introduced Fara1.5-27B, a sophisticated multimodal computer use agent (CUA) designed to automate web browser tasks by interpreting screenshots and executing user requests through structured tool calls. This vision-only agent operates by observing browser activity through images, predicting actions based on the current state and prior history, and performing tasks like filling forms, making reservations, and conducting searches. The model leverages a supervised fine-tuning process from the Qwen3.5-27B base, utilizing a unique multi-agent data pipeline called FaraGen1.5 that synthesizes web tasks and verifies their execution outcomes. The significance of Fara1.5-27B lies in its potential to enhance productivity and efficiency in web interactions while addressing safety concerns through a critical-points framework. This framework requires user confirmation before executing sensitive actions, such as entering personal information or conducting irreversible tasks, effectively keeping the user in control. The architecture supports a lengthy context of 262,144 tokens, allowing for comprehensive action tracking over multi-step tasks. With capabilities grounded at the pixel level and execution recommended in secure environments, Fara1.5-27B represents a significant step towards reliable web automation, albeit with warnings about potential risks and the need for human oversight to mitigate errors and ensure safe usage.