You’re asking if you can use S3 with the Custom Domain, but not enable public access? If so, the S3 Compatability API is only available through r2.cloudflarestorage.com. Custom Domains are just a simple way to publicly serve the contents of a bucket. You can’t use Presigned URLs or s3 compact through them.
If your question is just if you can restrict the accessibility of assets on a Custom Domain, the easiest way would probably be to use a Worker, bound to the R2 Bucket, using Worker Custom Domains, and only serve index.html/a few specific assets you want (or just make a separate bucket).