Celery is a distributed task queue system that allows you to run time-consuming operations asynchronously in the background.
It’s particularly useful for I/O-bound operations, periodic tasks, and long-running processes.
The Launchpad uses Celery because of the unique characteristics of GenAI workflows that rely heavily on LLM calls. Unlike traditional APIs with predictable response times, LLM calls can vary significantly in duration depending on factors like prompt complexity, model load, and request volume.Running these workflows as background tasks ensures the main application remains responsive to receive new events and serve other endpoints, while Celery workers handle the time-intensive AI processing separately. This architecture allows us to scale processing capacity easily by spawning additional workers when demand increases.Celery provides robust capabilities including task distribution, automatic retries, monitoring, and horizontal scaling, making it perfect for the unpredictable nature of GenAI workflow processing.
