Speculative Sampling Explained (saibo-creator.github.io)

A new technique known as speculative sampling has been introduced, designed to align draft sampling distributions $q(x)$ with target sampling distributions $p(x)$. This method tackles the issue of oversampling and undersampling tokens, which occurs when directly sampling from $q(x)$. By implementing a clever rejection mechanism, speculative sampling allows for down-sampling over-represented tokens and up-sampling under-represented ones, thereby approximating the target distribution. The key innovation lies in the introduction of a residual distribution, which captures the under-sampled probabilities and offers them a second chance in the sampling process. This dual approach—rejecting over-sampled tokens with a probability defined as $p(x_i) / q(x_i)$ and re-sampling from the residual distribution when a rejection occurs—ensures that the final output effectively mirrors the target distribution. The validity of this method is supported by mathematical proofs, signifying its potential to enhance sampling efficiency and accuracy in AI and machine learning applications. This advancement promises significant implications for tasks involving generative models and other probabilistic frameworks, where precise sampling is critical.