How Many Bits Can an Adapter Write? Capacity and Memorization of LoRA (arxiv.org)

Recent research delves into the capacity and memorization aspects of Low-Rank Adaptors (LoRA), highlighting their efficiency in parameter-efficient fine-tuning. Traditionally viewed as mere "skills," these adapters have been quantitatively analyzed to reveal that they write significantly fewer bits into a model than full fine-tuning would imply. The study reveals that the actual memory capacity of LoRA adapters depends more on the location of their parameters within the network architecture rather than the number of parameters alone. For instance, shifting parameters from attention layers to the Multi-Layer Perceptron (MLP) effectively doubles the storage capacity, while a disorganized parameter structure leads to diminished capacity. This finding is crucial for the AI/ML community as it provides measurable insights into how fine-tuning processes can be designed to minimize privacy leakage. The research establishes a clear distinction between how supervised and reinforcement learning approaches store information, further emphasizing the need to focus on what is being written during fine-tuning. This quantitative approach transforms a previously anecdotal understanding into actionable data, empowering developers to make informed decisions about model training techniques and privacy concerns tied to data retention.