Next.js with react-router-dom with rewrites getting 404

For one reason or another, I ended up using Next.js with react-router-dom for my single page application website. The entire page does not use ssr.
The only pages for Next.js is index.tsx, and the rest of the routing is handled client side by react-router-dom.
I have a rewrites from /:any* to / in next.config.js which has been working great on netlify, but both the Next.js and Next.js static html preset in Cloudflare results in a 404 when visiting any page that’s not /.

Note: I have a custom 404 page in react-router-dom but the default next.js 404 page is been shown.

I tried to use the _redirects file with /:any* / but that didn’t help either.

Could someone please provide some suggestions or guidance on how to fix the routing issue on Cloudflare for my Next.js and react-router-dom single page application