Skip to content

Generate large files in the background ​

An ordinary endpoint call has to answer quickly. That is not enough time to build a large report or a ZIP of many files. A background route answers the visitor at once with a job id, then runs the function for up to five minutes with 30 times the usual fuel. The result is kept for seven days.

The usual pattern has three parts. The function writes the file with env.file and returns a link to it. Meanwhile, the visitor's page polls the job until the link is ready.

This page assumes the following:

  • the distribution acme-marketing (d3f7a1c4-90b2-4e6a-8f15-6c2b7e0a9d38);
  • a function runtime, script 01M3RD5T2C8V7X1N4K6P9Q2W5Z;
  • a private file grant on reports/* for that script.

Store and serve files from a function shows how to set up the grant.

1. Write the handler ​

The handler is an ordinary function runtime. What it returns becomes the job's result once the run finishes.

javascript
function handler(request, env) {
  const key = `reports/${Date.now()}.csv`;
  const writer = env.file.create(key, { contentType: 'text/csv' });
  const encoder = new TextEncoder();
  writer.append(encoder.encode('id,total\n'));
  for (let page = 0; page < 200; page += 1) {
    const lines = [];
    for (let i = 0; i < 5000; i += 1) lines.push(`${page * 5000 + i},${i % 97}`);
    writer.append(encoder.encode(lines.join('\n') + '\n'));
  }
  const file = writer.commit();
  const link = env.file.link(key, { ttlSeconds: 3600 });
  return {
    headers: { 'content-type': 'application/json' },
    body: JSON.stringify({ size: file.size, url: link.url }),
  };
}

On a background route, one run can write up to 5 GiB through append. Test the handler with edge_function_invoke before you map it. A direct invoke runs with the normal limits, so test with a smaller output.

2. Map a background route ​

This step changes what visitors can reach. Include every path you already have.

text
On acme-marketing, map POST /_api/reports/build to script 01M3RD5T2C8V7X1N4K6P9Q2W5Z
as a background route. Keep the existing endpoints.
bash
edge api PATCH /v1/distributions/d3f7a1c4-90b2-4e6a-8f15-6c2b7e0a9d38 --input - <<'EOF'
{
  "runtime": {
    "api": {
      "routes": [
        { "match": "/_api/reports/build", "wasm": "01M3RD5T2C8V7X1N4K6P9Q2W5Z", "methods": ["POST"], "background": true }
      ]
    }
  }
}
EOF
bash
curl -s -X PATCH \
  https://api.edge.deca-dev.com/v1/distributions/d3f7a1c4-90b2-4e6a-8f15-6c2b7e0a9d38 \
  -H "Authorization: Bearer $EDGE_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"runtime":{"api":{"routes":[{"match":"/_api/reports/build","wasm":"01M3RD5T2C8V7X1N4K6P9Q2W5Z","methods":["POST"],"background":true}]}}}'

The agent runs edge_distribution_update. The path goes live as soon as the update compiles.

3. Start a job and poll it ​

bash
curl -s -X POST https://acme-marketing.edge.deca-dev.com/_api/reports/build \
  -H "x-amz-content-sha256: $(printf '' | sha256sum | cut -d' ' -f1)"
json
{ "job_id": "job_5f0c2a9e7b1d4c3f8a6e0b2d9c7f1a34", "status": "queued", "status_url": "/_api/_jobs/job_5f0c2a9e7b1d4c3f8a6e0b2d9c7f1a34" }

Poll the status_url every few seconds. status moves from queued to running. When the run finishes, it becomes succeeded and the function's response appears in result:

json
{
  "job_id": "job_5f0c2a9e7b1d4c3f8a6e0b2d9c7f1a34",
  "status": "succeeded",
  "result": {
    "status_code": 200,
    "headers": { "content-type": "application/json" },
    "body": "{\"size\":7388890,\"url\":\"https://…\"}"
  },
  …
}

If the run fails, status is failed and error_code gives the reason, for example fuel_exhausted or deadline_exceeded. The function's own output is available from edge_function_logs.

A visitor's page does the same with fetch. It posts to the route, polls status_url until the status is succeeded, and then sends the visitor to the link in the result.

A request body can be at most 192 KiB. At most five background runs per distribution can be active at the same time. A sixth request gets 429; try again when one of the five has finished.

Next ​