Cloudflare KV 100x faster with limit

I have (a still) small KV with less then a 100 keys. The format of the key is user:$userid:item.

The following two queries for looking up keys , the first one being consistently 10-100x faster:

await USER_DATA.list({ prefix: prefix, limit: 10 });

13ms

await USER_DATA.list({ prefix: prefix });

1300ms.

There are only 3 keys that match in both scenarios.

How is this possible?