Shuffling Is Not Enough: Breaking Permutation-Based Model Confidentiality (arxiv.org)

Recent research has revealed significant vulnerabilities in permutation-based model confidentiality within hybrid fully homomorphic encryption (FHE) inference systems. While hybrid FHE aims to secure private inference by performing linear computations on the server side and allowing clients to apply nonlinearities, the study demonstrates that returning merely noisy, shuffled responses fails to adequately safeguard model confidentiality. Specifically, the researchers pinpoint that a minimal number of queries can lead to the exact recovery of sensitive model outputs, demonstrating that as few as $d+1$ queries can allow attackers to ascertain significant information about permutation-invariant layers, thus breaching confidentiality. This finding has crucial implications for the AI/ML community, signaling that relying solely on techniques like shuffle-model differential privacy does not provide the necessary protection against model information leaks. The ability to flawlessly recover linear layers from a ResNet-20 model and confirm the same for other architectures like ImageNet-scale CNNs indicates that current privacy mechanisms might enable attackers to perform fingerprinting and lineage attribution. As a result, the research calls into question the efficacy of existing privacy-preserving practices and highlights the need for more robust strategies to protect sensitive model information in secure AI applications.