KeyValue Bulk Insert Fails

Developers Workers
#1

I am trying to write 7MB of data using KV bulk insert. Theoretically, the two bits of code below should do the same thing.

The first code just puts in one item using a for loop and the second tries to bulk insert the same item. The code was modified to support debugging, I am obviously trying to insert the entire array of data. When using the entire array, the first fails for load issues after several hundred inserts.

The second one fails with the error shown at the end for just one item in the batch.

    for(const item of json) {
		try {
			const result = await fetch(`https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/storage/kv/namespaces/$KV/values/${item.key}`,
				{
					method: "PUT",
					headers: {
				        "Content-Type": "application/json",
				        "X-Auth-Key": "$KEY",
				        "X-Auth-Email": "$EMAIL"
				      },
				      body: JSON.stringify(item.value)
				}
			);
			console.log(result);
		} catch(e) {
			console.log(e);
			break;
		}
		break; // just break after first since we are debugging
	}
	fetch("https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/storage/kv/namespaces/$KV/bulk",
		{
			method: "PUT",
			headers: {
		        "Content-Type": "application/json",
		        "X-Auth-Key": "$KEY",
		        "X-Auth-Email": "$EMAIL"
		      },
		      body: JSON.stringify([json[0]])
		}
	)
	.then((response) => console.log(response))
	.catch((e) => console.log(e))

// Always throws Response {
  url: "https://api.cloudflare.com/client/v4/accounts/92dcaefc91ea9f8eb9632c01148179af/storage/kv/namespaces...",
  status: 400,
  statusText: "Bad Request",
  type_: "default",
  trailer: Promise { <pending> },
  headers: Headers { date: Sat, 23 May 2020 15:01:41 GMT, content-type: application/json; charset=UTF-8, set-cookie: __cfduid=dde913c46802ba06261c26ca0640b11fa1590246101; expires=Mon, 22-Jun-20 15:01:41 GMT; path=/; domain=.api.cloudflare.com; HttpOnly; SameSite=Lax, cf-ray: 597fa754fc3fe35e-SEA, cache-control: no-store, no-cache, must-revalidate, cf-request-id: 02e3a6e9160000e35e4c93e200000001, vary: Accept-Encoding, cf-cache-status: DYNAMIC, expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct", x-envoy-upstream-service-time: 8, server: cloudflare },
  body: Body { contentType: "application/json; charset=UTF-8", locked: false, body: [Circular] },
  type: "default",
  redirected: false
}
#3

So json is an array containing objects with key and value fields respectively? I’d print everything before you send it to make sure the request actually follows https://api.cloudflare.com/#workers-kv-namespace-write-multiple-key-value-pairs

#4

Also, the variables in your two examples are just stand-ins, right? In your actual code there are the actual values. That would not expand.

#5

The form of the data in the array is {“key”:“some key”,“value”:“some value”}.

When used as its 2 parts it works in the for loop, i.e. the key as part of the curl url and the value as the data.

#6

Again, add the mentioned debug information and check the URLs.

#7

I do print the entire array to a file before sending. It contents of the array appear to be fine. Not sure what you mean about other debug info.

I have also tried just batch putting the example data provided by Cloudflare and it fails.

#8

Can you post the output?

Also, you still didnt address the URL part.

#9

What does this return?

fetch("https://api.cloudflare.com/client/v4/accounts/$ACCOUNT/storage/kv/namespaces/$KV/bulk",
	{
		method: "PUT",
		headers: {
			"Content-Type": "application/json",
			"X-Auth-Key": "$KEY",
			"X-Auth-Email": "$EMAIL"
		},
		body: '[{ "key": "key1", "value": "value1"}, { "key": "key2", "value": "value2"} ]'
	}
)
.then((response) => console.log(response))
.catch((e) => console.log(e));
#10

Thanks so much for your quick responses!

It always results in:

 Response {
  url: "https://api.cloudflare.com/client/v4/accounts/92dcaefc91ea9f8eb9632c01148179af/storage/kv/namespaces...",
  status: 400,
  statusText: "Bad Request",
  type_: "default",
  trailer: Promise { <pending> },
  headers: Headers { date: Sat, 23 May 2020 15:01:41 GMT, content-type: application/json; charset=UTF-8, set-cookie: __cfduid=dde913c46802ba06261c26ca0640b11fa1590246101; expires=Mon, 22-Jun-20 15:01:41 GMT; path=/; domain=.api.cloudflare.com; HttpOnly; SameSite=Lax, cf-ray: 597fa754fc3fe35e-SEA, cache-control: no-store, no-cache, must-revalidate, cf-request-id: 02e3a6e9160000e35e4c93e200000001, vary: Accept-Encoding, cf-cache-status: DYNAMIC, expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct", x-envoy-upstream-service-time: 8, server: cloudflare },
  body: Body { contentType: "application/json; charset=UTF-8", locked: false, body: [Circular] },
  type: "default",
  redirected: false
}
#11

Even with the example I posted? In that case it does not necessarily have to be the body but something in the request is wrong. Again, the URL is correct, as I mentioned a few times already?

If it is, I’d try next with the example provided with the API reference.

#12

sandro thanks for your assistance, I had thought just stringifying my array of objects with key and value properties was the right thing to to. It turns out I had to stringify the value of the value properties so:

{“key”:“my key”,“value”:JSON.stringify(myData)} are the elements of the array

not

{“key”:“my key”,“value”:myData}

Not very convenient, but understandable given the way the KV store works inside workers