Teaching My AI to Sleep: Continual Learning with Llama 3 on a Mac Mini (medium.com)

A new project, "Circadia," explores the concept of providing a local language model (LLM), specifically Llama 3.2, with a “circadian rhythm” on a Mac Mini M2. The creator aimed to replicate the human memory process of consolidating and pruning information during sleep. By implementing a dual approach using Low-Rank Adaptation (LoRA) for style and personality adjustments and Retrieval-Augmented Generation (RAG) for factual memory retention, the model trains during nighttime based on conversational logs from the day while avoiding catastrophic forgetting. The experiment revealed surprising insights about dataset size and model performance. Contrary to expectations, a small synthetic dataset allowed the model to adapt its style effectively, preserving performance better than a larger professional dataset. Key takeaways include the importance of a gentle learning rate for LoRA training to maintain intelligence, the necessity of routing factual information, and the observation that while personalization can come at a slight cost to reasoning capability, focused training yields more effective results. Future enhancements may involve smarter classifiers and more sophisticated memory organization during the model's “sleep” cycles.