Teaching an LLM a Niche Diagraming Language (huy.rocks)

A recent project focused on teaching a smaller language model (LLM), specifically Qwen2.5-Coder-7B, to generate and edit diagrams using the niche Pintora diagramming language. While popular languages like Mermaid and PlantUML have been well-integrated into existing LLMs, Pintora presents a unique challenge due to limited training data. The training strategy involved two phases: Continued Pretraining (CPT) where the model learned Pintora's syntax and an Instruction Finetuning (IFT) phase that adapted it for specific diagramming tasks. Throughout the process, the developer faced challenges such as finding enough training data and optimizing resource use, ultimately generating about 1,500 comprehensive data entries through creative techniques, including using AI to assist with data generation. For the AI/ML community, this project highlights the potential of adapting smaller models for specialized tasks, particularly in niche programming domains. The outcomes, with an 86% accuracy in generating correct syntax diagrams, demonstrate the feasibility of using resource-efficient models to achieve significant results in learning complex languages. The datasets and model are available on Hugging Face, offering a valuable resource for other developers interested in advancing LLM capabilities within niche applications. This experiment not only shows the adaptability of LLMs to diverse programming languages but also sets the stage for future projects tackling other lesser-known languages like Strudel.