One big advantage of Next.js is that it allows developers to run their entire app - both frontend and backend - in the same repo, which streamlines development. However, platforms designed to host Next.js applications like Vercel are primarily focused on frontend use cases that expect snappy response times. Because of this, Vercel enforces a hard timeout limit of under 60 seconds for all processes.

Unfortunately, applications commonly have back-end tasks that require extended runtimes, such as data aggregation, API syncing, or scheduled maintenance. Even if these tasks can finish in just under 60 seconds, slight variations in runtime might come across strict timeout limits resulting in 504 errors.

In this guide, we'll explore common use cases for background jobs and discuss current hosting solutions. We'll introduce a service called Webrunner, specially optimized for Next.js background jobs, which addresses these limitations and ensures smoother execution of time-consuming tasks.

What are background jobs?

Background jobs are tasks that require more than a few seconds to complete and can be triggered either by a predefined schedule or through user interaction. For example, an app might sync data from an external service (e.g. Twitter API), which can be set to occur regularly (e.g. every hour for each user) or on-demand (e.g. initiated by the user clicking a "import data" button). Regardless of the trigger, these asynchronous processes need more than a few seconds to run, making it essential to delegate them to background workers.

Let's delve into some more example use cases where background jobs are vital:

1. Sending emails and notifications: Sending personalized notifications or scheduled emails to users is a common feature in many applications. However, looping over every user and processing custom notification parameters can take a long time.
2. Data aggregation and analytics: For applications that perform data aggregation or complex analytics tasks, executing these operations directly within the main application thread can lead to slow responses and hinder user interactions. Putting these tasks into a background job enables the app to cache data in advnace so that users can access insights without waiting for time-consuming computation.
3. Integrations and API Calls: Applications often interact with external services such as downloading or syncing data via APIs. Making synchronous API calls during user interactions can result in slower response times and potential failures due to network issues. Background jobs can handle these API calls asynchronously, preventing bottlenecks and allowing users to see up-to-date information immediately.

Limitations of Web Hosting Platforms

Popular web hosting platforms like Vercel offer remarkable scalability and cost-effectiveness with serverless functions, but they are designed for requests that render data directly to the end user. To make sure requests are handled quickly, Vercel has timeout limits of 60 seconds (10 seconds for Hobby projects). Unfortunately, this limitation poses a significant challenge when running lengthy tasks that require additional processing time. For Next.js applications dealing with long-lived background jobs, this can result in 504 timeout errors.

Setting Up a Task Queue on AWS or GCP

To overcome the limitations imposed by web hosting platforms, developers can resort to setting up a task queue and worker processes on cloud providers like AWS or GCP. However, this approach requires considerable effort and maintenance:

1. Building and maintaining Docker images: Designing and building Docker images to encapsulate background job code can be time-consuming and must be managed in a continuous integration workflow.
2. Queue management: Implementing an efficient task queue management system with proper observability and monitoring can be complex.
3. Infrastructure configuration: Configuring and managing cloud infrastructure for scalable background workers requires expertise and constant monitoring.

Introducing Webrunner

Webrunner enables Next.js developers to run background jobs that can last up to 24 hours with minimial (even no-code) setup.

How Webrunner Works

Webrunner deploys background workers running the same Next.js source code as your regular app. This unique feature allows you to define tasks using native Next.js API endpoints, enabling seamless migration of processes from Vercel to Webrunner without any modifications.

Background tasks are triggered by the Webrunner scheduler or by ad-hoc calls to the Webrunner API. Initiating a background job is as simple as calling the same api endpoints already defined in your Next.js app.

With Webrunner, each request kicks off a background task that is added to a queue. As workers begin and process each task, its status is updated in real-time. Webrunner workers have up to 24 hours to complete each task, making sure your processes don't timeout.

Learn more about Webrunner and supercharge your Next.js application with background jobs that deploy seamlessly today.