Aws-sdk-v3 hangs after ~50 getObject requests

Developers Storage
1

I’m currently updating all the objects on my bucket to have a content encoding of brotli, as when I transferred my files from an old storage provider, the content encoding was not transfered properly. I wrote a script using aws-sdk/client-s3 to go through every object in the bucket and put the same object with the correct content-encoding. However, for some reason, the getObject method hangs after 50 requests or so, and only returns back to normal after a few minutes of idle, which then hangs again after 35 or so requests.

I assume this is a bot protection or some sort, so is there a way to temporarily disable this or another workaround?

2

I was able to work around this by initializing a new S3 object everytime I want to get an object.

const recursive = async (params) => {
    const objects = await objectClient.listObjectsV2(params)

    for (const object of objects.Contents) {
        const actObject = await new S3({
            credentials: {
                accessKeyId: objectAccessKey,
                secretAccessKey: objectSecretKey
            },
            endpoint: objectEndpoint,
            region: objectRegion,
        }).getObject({
            Bucket: objectName,
            Key: object.Key
        })
        

        if (actObject.ContentEncoding != 'br') {
            await objectClient.putObject({
                Bucket: objectName,
                Key: object.Key,
                ContentEncoding: 'br',
                ContentType: 'application/json',
                Body: await actObject.Body.transformToByteArray(),
            })
        }
        console.log(`${object.Key} | ${++i}/16584`)
    }

    if (objects.IsTruncated) {
        recursive(Object.assign(params, { ContinuationToken: objects.ContinuationToken }))
    }

    return
}

await recursive({
    Bucket: objectName,
    MaxKeys: 1000
})
3

Recreating the S3 client doesn’t actually talk to the storage provider at all so I assume it’s some issue with the SDK itself if that fixes it.