An Android UI dump for LLMs (10× fewer tokens, same actions) (elliotgao2.github.io)

A new tool called Handsets has been developed to optimize how large language models (LLMs) interact with Android UIs, significantly reducing the token count required for actions. By converting UI data from the traditional uiautomator XML dump format to a streamlined tabular format using `hs ui -i`, the tool achieves approximately 10× fewer tokens for equivalent tasks. This is pivotal as it conserves context while maintaining decision quality, ultimately lowering costs associated with LLM usage. The Handsets tool efficiently drops non-essential elements from the UI data, such as empty layout containers and default attribute values, while retaining only the information that LLMs need to make decisions. By offering a concise representation of interactive elements, the format promotes quicker processing and allows agents to fit more UI interactions within the same context window. This advancement could enhance the performance of AI applications navigating Android interfaces and could serve as a template for similar optimizations across various input scenarios in AI systems.