Logpush HTTP Delivery Jobs fail to creatre

[stefan21]

Hello,

I’m trying to setup a Logpush HTTP Delivery job to get my Worker logs to my custom endpoint.
The endpoints works and when creating this endpoint with the Cloudflare API I do get a test call from Cloudflare.
However the logpush create API call returns this error:

{
  "errors": [
    {
      "code": 1004,
      "message": "creating a new job is not allowed: exceeded max jobs allowed"
    }
  ],
  "messages": [],
  "result": null,
  "success": false
}

There are no other jobs created and we are on the paid worker plan.
What might cause this error? Did anyone else run into this before?

Thanks in advance.

[Chaika]

What dataset are you trying to use? The only one you can use is workers_trace_events. If you could share the full (but cleaned of any secrets/sensitive information) request, it would be helpful.
There are docs here about getting started with Worker Trace Events logpush:

Logpush · Cloudflare Workers docs

[stefan21]

Thanks for the quick reply @Chaika
I tried using using dataset workers_trace_events but got this error in return:

{
  "errors": [
    {
      "code": 1002,
      "message": "dataset not supported"
    }
  ],
  "messages": [],
  "result": null,
  "success": false
}

So Instead I’m using http_requests like this:

$ curl -s https://api.cloudflare.com/client/v4/zones/$ZONE_TAG/logpush/jobs -X POST -d '
{
  "name": "theburritobot.com-https",
  "logpull_options": "fields=RayID,EdgeStartTimestamp&timestamps=rfc3339",
  "destination_conf": "https://logs.example.com?header_Authorization=Basic%20REDACTED&tags=host:theburritobot.com,dataset:http_requests",
  "max_upload_bytes": 5000000,
  "max_upload_records": 1000,
  "dataset": "http_requests",
  "enabled": true
}' \
-H "X-Auth-Email: <EMAIL>" \
-H "X-Auth-Key: <API_KEY>"

[Chaika]

You can’t use the http_requests dataset, that’s enterprise only (as is any other dataset then workers_trace_events).

The reason why you can’t use the worker trace dataset is because it’s account-scoped, not zone-scoped like the endpoint you are using is.
Check out the Workers Logpush guide I linked above

The curl request you want is going to look something like this:

$ curl -s https://api.cloudflare.com/client/v4/accounts/<ACCOUNT_ID>/logpush/jobs -X POST -d '
{
  "name": "theburritobot.com-workers_trace_events",
  "logpull_options": "fields=Event,EventTimestampMs,Outcome,Exceptions,Logs,ScriptName",
  "destination_conf": "https://logs.example.com?header_Authorization=Basic%20REDACTED&tags=host:theburritobot.com,dataset:workers_trace_events",
  "dataset": "workers_trace_events",
  "enabled": true
}' \
-H "X-Auth-Email: <EMAIL>" \
-H "X-Auth-Key: <API_KEY>"

ps. The HTTP destinations are not available in the GUI, but you can use the GUI (Account → Analytics & Logs → Logs) to make all other types. HTTP Destinations get sent the logs gzip encoded as well, might be worth pointing out.

[stefan21]

I got it working. The resource you mentioned was very helpful.
My issue was indeed that I tried creating zone logpush job, but I needed a account logpush job.

Thanks for the GZIP encoding mention as well. Issue is solved now!