Hi,
today I tested airbyte s3 destination with r2, and airbyte does internal checks
and throws exception below:
"2022-08-08 04:44:55 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:55 \u001b[32mINFO\u001b[m i.a.i.d.s.S3Destination(testSingleUpload):81 - Started testing if all required credentials assigned to user for single file uploading"
28: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.i.d.s.S3Destination(testSingleUpload):91 - Finished checking for normal upload mode"
29: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.i.d.s.S3Destination(testMultipartUpload):95 - Started testing if all required credentials assigned to user for multipart upload"
30: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m a.m.s.StreamTransferManager(getMultiPartOutputStreams):329 - Initiated multipart upload to xxxx/test/path/test_1659933896033 with full ID XXXXXXXXXXXXXXX
31: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000]"
32: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m a.m.s.MultiPartOutputStream(close):158 - Called close() on [MultipartOutputStream for parts 1 - 10000]"
33: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[33mWARN\u001b[m a.m.s.MultiPartOutputStream(close):160 - [MultipartOutputStream for parts 1 - 10000] is already closed"
34: "2022-08-08 04:44:56 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:56 \u001b[32mINFO\u001b[m a.m.s.StreamTransferManager(complete):367 - [Manager uploading to xxxx/test/path/test_1659933896033 with id ABhCqE4iX...0+1c2gw==]: Uploading leftover stream [Part number 1 containing 3.34 MB]"
35: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:57 \u001b[1;31mERROR\u001b[m a.m.s.StreamTransferManager(abort):432 - Aborting [Manager uploading to xxxx/test/path/test_1659933896033 with id ABhCqE4iX...0+1c2gw==] due to error: java.lang.IllegalArgumentException: Invalid base 16 character: 'P'"
36: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:57 \u001b[32mINFO\u001b[m a.m.s.StreamTransferManager(abort):470 - [Manager uploading to xxxx/test/path/test_1659933896033 with id ABhCqE4iX...0+1c2gw==]: Aborted"
37: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - 2022-08-08 04:44:57 \u001b[1;31mERROR\u001b[m i.a.i.d.s.S3Destination(check):72 - Exception attempting to access the S3 bucket: "
38: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - java.lang.IllegalArgumentException: Invalid base 16 character: 'P'"
39: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - \tat com.amazonaws.util.Base16Codec.pos(Base16Codec.java:100) ~[aws-java-sdk-core-1.12.6.jar:?]"
40: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - \tat com.amazonaws.util.Base16Codec.decode(Base16Codec.java:87) ~[aws-java-sdk-core-1.12.6.jar:?]"
41: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - \tat com.amazonaws.util.Base16Lower.decode(Base16Lower.java:53) ~[aws-java-sdk-core-1.12.6.jar:?]"
42: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - \tat com.amazonaws.util.BinaryUtils.fromHex(BinaryUtils.java:48) ~[aws-java-sdk-core-1.12.6.jar:?]"
43: "2022-08-08 04:44:57 \u001b[32mINFO\u001b[m i.a.w.i.DefaultAirbyteStreamFactory(lambda$create$0):61 - \tat com.amazonaws.services.s3.AmazonS3Client.doUploadPart(AmazonS3Client.java:3881) ~[aws-java-sdk-s3-1.12.6.jar:?]"
Please check r2 server’s compatibilities with s3 api.
airbyte s3 destination uses this part code to do multipart upload check and fails in this part of code:
here
s3Client.uploadPart(uploadRequest)