Accelerating Generative AI with PyTorch (ISC 2024)

Information

Generative Artificial Intelligence (Generative AI) has emerged as a transformative force in creating realistic data, images, and content. By leveraging deep learning (DL) techniques, generative AI models learn patterns and structures from large datasets and autonomously produce novel outputs without explicit programming for each possible outcome. Even though these models are usually pre-trained on massive datasets, fine-tuning for specific tasks or performing inference still requires considerable number of computational resources, necessitating strategic optimizations to unlock their full potential.

This tutorial, powered by EuroCC 2, aims to equip participants with a comprehensive understanding of the computational challenges inherent in Generative AI. Moreover, it seeks to present strategies for identifying and mitigating bottlenecks, exclusively leveraging PyTorch’s native features without resorting to additional languages like C++ or CUDA. The tutorial encompasses a diverse array of topics, including hands-on sessions on setting up and optimizing a generic DL pipeline on a supercomputer, an exploration of mainstream Generative AI models, and optimization strategies for inference specifically tailored to Generative AI. Additionally, it investigates advanced subjects such as sparsification and model parallelism for models with a large number of parameters. By the end of this tutorial, participants will be capable at navigating the challenges of accelerating Generative AI using the powerful tools and techniques inherent in PyTorch.

Presenters

• Chris Stylianou
  • Research Engineer, The Cyprus Institute, EuroCC Cyprus
• Robert Jan Schlimbach
  • Technical Advisor ML/HPC, SURF BV, EuroCC Netherlands
• Ivan Gentile
  • Data Scientist, IFAB, EuroCC Italy
• Charalambos Chrysostomou
  • Research Scientist, The Cyprus Institute, EuroCC Cyprus
• Constantine Dovrolis
  • Professor; Director of CaSToRC, The Cyprus Institute, EuroCC Cyprus
• Boris Velichkov
  • Researcher, Sofia University, EuroCC Cyprus

Agenda

Time Slot	Topic	Presenter(s)
09:00 – 09:30	Introduction and setup	Boris Velichkov
09:30 – 10:15	Efficient Scaling of Machine Learning Models with PyTorch: Distributed Learning	Charalambos Chrysostomou
10:30 – 11:00	PyTorch and Profiling	Robert Jan Schlimbach, Boris Velichkov
11:00 – 11:30	Coffee Break	N/A
11:30 - 11:45	A gentle introduction to LLMs and LLaMa	Ivan Gentile
11:45 – 12:30	Optimizing Llama: Enhancing Efficiency and Scalability of Large Language Models with PyTorch	Chris Stylianou
12:30 – 13:00	An overview of methods for efficient generative AI training & inference	Constantine Dovrolis