AI Agent, AI Spy - Udbhav Tiwari and Meredith Whittaker [video] (media.ccc.de)

In a recent talk, Udbhav Tiwari and Meredith Whittaker explored the implications of "Agentic AI," which refers to AI systems that operate autonomously within operating systems and applications, such as web browsers. They highlighted how this shift transforms these technologies from neutral tools into proactive entities that engage in surveillance without user consent. A key example discussed was Microsoft's "Recall" feature, which creates a comprehensive memory of user activity, posing significant privacy risks and potentially undermining the security of privacy-centric applications like Signal. This emerging paradigm raises alarms about the erosion of personal agency, as users face automated recommendations that may prioritize corporate interests over individual choice. The speakers proposed an actionable framework for mitigating these risks, which includes empowering developers with the ability to mark sensitive applications, granting users granular control over AI access, mandating transparency from OS vendors on data practices, and supporting adversarial research to expose vulnerabilities in these systems. By advocating for proactive measures, Tiwari and Whittaker aim to safeguard user privacy and maintain developer agency in an increasingly automated digital landscape, urging the community to advocate for structural changes before it's too late.