Backup directly from Google Drive to R2

Developers Storage
#1

I am trying to backup my Google account to Cloudflare’s R2.

However, I do not have proper bandwidth to download all my Google data to my PC and then upload it to R2 so I am trying to find a way to do that directly (server to server).

Recently Google added an option to send the backup files directly to cloud storage:

image
image1642×843 56.7 KB

So, I could potentially tell Google Takeout to store the data to Google Drive. The question then is how do I transfer the data from Google Drive to R2 without going through my PC?

One way I can think of is using rclone, but I would need a cloud server to run rclone somewhere (otherwise, running it on my PC means data will go through my PC).

Is it possible to run rclone in a Cloudflare Worker perhaps?

#2

R2 is not yet included in it as of this doc:

#3

That doc is for transferring into and from Google Cloud Storage, Google Takeout is more consumer oriented and can put the backup files in Google Drive, which is something and cannot be accessed the same way as Cloud Storage.

R2 will probably not be included in the list in the future either because Takeout is a consumer facing product…

#4

Now that I think of it, I can probably write a script to access Google Drive API via google JS SDK…

Worker will then download the file / stream it directly to R2. Might be possible, I am not sure…

I can embed Google Picker in a Worker:
The Google Picker API | Google Developers

Then somehow stream it to R2

#5

So I was able to hook it up and the files are from uploading from Google Drive → S3

import { drive_v3 } from "googleapis";
import { Upload } from "@aws-sdk/lib-storage";
import { S3 } from "@aws-sdk/client-s3";

const s3Client = new S3({
  endpoint: ENDPOINT,
  region: REGION,
  credentials: {
    accessKeyId: ACCESS_KEY_ID,
    secretAccessKey: SECRET_ACCESS_KEY,
  },
});

interface Env {}

export const onRequest: PagesFunction<Env> = async (context) => {
  const accessToken = context.request.headers.get("X-Access-Token");
  const data: drive_v3.Schema$File = await context.request.json();

  // Get the file from Google Drive
  const driveRequest = new Request(
    `https://www.googleapis.com/drive/v3/files/${data.id}?alt=media`,
    {
      headers: {
        Authorization: `Bearer ${accessToken}`,
      },
    }
  );

  // fetch file from Google Drive
  const resp = await fetch(driveRequest);

  // Send file to S3 using multipart upload
  try {
    const parallelUploads3 = new Upload({
      client: s3Client,
      params: {
        Bucket: BUCKET,
        Key: data.name,
        Body: resp.body,
        ContentMD5: hexToBase64(data.md5Checksum),
      },
      queueSize: 4, // optional concurrency configuration
      partSize: 1024 * 1024 * 5, // optional size of each part, in bytes, at least 5MB
      leavePartsOnError: false, // optional manually handle dropped parts
    });

    await parallelUploads3.done();
    console.log(`Upload complete: ${data.name}`);
  } catch (e) {
    console.log(e);
  }

  return new Response('{"res": "Hello world"}');
};

function hexToBase64(hexStr) {
  let base64 = "";
  for (let i = 0; i < hexStr.length; i++) {
    base64 += !((i - 1) & 1)
      ? String.fromCharCode(parseInt(hexStr.substring(i - 1, i + 1), 16))
      : "";
  }

  return btoa(base64);
}

But with large files, I am getting

✘ [ERROR] Error: Worker exceeded CPU time limit.

After some time.
I am not sure why, any idea ?