Cloudflare Access JWT token is invalid

I’m trying to set up Cloudflare Access using JWT tokens. When I follow the instructions and copy the token from my CF_Authorization cookie and the public key into jwt.io I always get “invalid signature”. What am I doing wrong?