We introduce Library X, a free, open-source PyTorch library for training deep learning models with differential privacy. Library X is designed for simplicity, flexibility, and speed. It provides a simple and user-friendly API, and enables machine learning practitioners to make a training pipeline private by adding as little as two lines to their code. In this paper, we first provide a brief overview of Library X, and then reveal the challenges posed by the prevalence of large language models (LLMs). To tackle these challenges, we propose several new features, either recently added or planned for the next version, including Fast Gradient and Ghost Clipping, model parallelism, parameter-efficient fine-tuning (PEFT), and mixed precision training.