Worker request.url value strips of trailing & character

#1

We were having a weird issue with a passive worker on one of our API endpoints that does HMAC auth using the request data as part of the signature.

In our case for various reason one of our requests contained an & character at the end of the URL string. We found that these request were suddenly failing when the worker was activated although it was a passive worker that was not modifying the request.

Turns out that for some reason all &'s at the end of the URL are being stripped as seen in the screenshot below.

has anyone else encountered this? anything we can do about it?

Workers___sageonestaging_com___Sage_One___Cloudflare_-_Web_Performance___Security
Workers___sageonestaging_com___Sage_One___Cloudflare_-_Web_Performance___Security.jpg988x435 62.3 KB

Update: Here is a easy to reproduce version of the failure - https://cloudflareworkers.com/#35ac553cdca2d02017d12e109e7e1709:https://tutorial.cloudflareworkers.com

Another reproduction

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})


async function handleRequest(request) {
  console.log('Got request', request)
  let re = 'http://postb.in/YOUR_BIN_ID_HERE' +'/a?url='+ request.url
  const response = await fetch(re,request)
  console.log('Got response', response)
  return response
}

then do

curl "https://YOUR_URL_WITH_WORKER.com/a?business_guid=f0c13f5a2e4cea8aeab7d2fe25df80a2&&&"

When you review the contents of the bin located at http://postb.in/b/YOUR_BIN_ID_HERE you will notice that the query is missing the &s

#2

try like this.
const parsedUrl = new URL(request.url);

#3

Your code works, problem is that you are testing it in builtin Worker Editor / Console in dashboard and sometimes it’s acting weird. You will get different results in this editor compared to actual website.

Replace your code with following:

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request));
})

async function handleRequest(request) {
  console.log('Requested URL: ' + request.url);
  return new Response('Requested URL: ' + request.url, {status: 200});
}

Save and test https://yourdomain.com/test?imtest&&&&& it in editor first… Both in console and preview page you will get:

Requested URL: https://yourdomain.com/test?imtest

(same issue as in your example)

Now open https://yourdomain.com/test?imtest&&&&& in your browser directly and you will see that this issue doesn’t exists anymore… You will get:

Requested URL: https://yourdomain.com/test?imtest&&&&&

So, real problem is builtin editor / console, not your code.

Never trust this editor, always test it on actual URL, instead of console.log(), add output to response body like I did in my example (it’s not perfect solution but it works).

There is many other issues with editor, like request.cf is undefined and many other problems, hopefully they will fix it in future.

Also, next time please include your code in your post, so we can copy it instead of rewriting it from screenshot you provide :sweat_smile:

#4

Thanks for the help Nikoloz, your point about the editor is a very good one and one that I did not know. When using your example I am able to see the & characters at the end.

However, when a full request object that included a url with an & at the end is passed to the fetch function what the origin sees is missing the &

The issue only seems to happen when query strings are present.

curl "https://URL_OF_APP.com/a?business_guid=f0c13f5a2e4cea8aeab7d2fe25df80a2&&&"
turns into 
"GET /a?business_guid=f0c13f5a2e4cea8aeab7d2fe25df80a2"

but

curl "https://URL_OF_APP.com/business_guid=f0c13f5a2e4cea8aeab7d2fe25df80a2&&&"
come through as expected
"GET /business_guid=f0c13f5a2e4cea8aeab7d2fe25df80a2&&&

When I turn the worker off, these requests go back to normal

#5

According to support this bug has now been fixed with the following release - 2018/11/29 Workers Runtime Release Notes

#6

Note: The bug is fixed in production, but we’ve discovered that there is a second instance of the same bug in the preview code. So in the preview, you may still see this bug happening until our next release in a few days. But, in production, it should be fixed.

Unfortunately fixing this bug did actually break some users who were accidentally relying on it. In the future we will have to be much more cautious about making any change to behavior even if it’s fixing bugs. :confused:

#7

Kenton, is there any chance to run multiple versions of V8 on your servers so customer can choose version to avoid this kind of breaking changes? There is always a chance that something like this will go wrong and some developers might find it challenging to fully trust Cloudflare Workers. Imagine running serverless functions in Node.js or PHP environment without knowing their version and automatic version updates in production.

Also if it’s possible please add cf.colo and cf.country support in preview, different behavior in preview and production is very confusing as you can see from this topic.