Fetch.json() and lack of unicode support

Developers Workers
#1

The following works fine in a browser (by creating a service worker)

addEventListener('fetch', event => {
    event.respondWith(fetchAndApply(event.request))
})

async function fetchAndApply(request) {
  try {

    const LanguageRes = await fetch('https://cdn.jsdelivr.net/gh/collinbarrett/FilterLists/data/Language.json')
    const Language = await LanguageRes.text()
    return new Response(JSON.stringify(JSON.parse(Language)))

    // This throws an internal error
    //const Language = await LanguageRes.json()
    //return new Response(JSON.stringify(Language))

  } catch (err) {
    return new Response(err.stack || err)
  }
}

content-length: 126
content-type: text/plain;charset=UTF-8

SyntaxError: Unexpected token  in JSON at position 0
    at JSON.parse (<anonymous>)
    at fetchAndApply (worker.js:10:45)
#2

Are you doing the parse and stringify on purpose? This combination would be somewhat redundant.

But as for the syntax error, the returned JSON does not appear to be valid. It seems to start with a byte order mark and that is not appreciated by JavaScript’s parser.

If you call trim on Language before passing it to parse it should work.

1 Like
#3

Are you doing the parse and stringify on purpose? This combination would be somewhat redundant.

Yes, I simplified the code to share with the community. I plan to manipulate the json.

Thanks, yes trimming the string fixed it. Hm Google Chrome is more forgiving…

#4

Are you sure you have the BOM as well when testing it in the browser? I’d be surprised if there was a major difference, as Cloudflare uses the same JavaScript engine as Chrome.

#5

Yea fairly sure. there are 0 modifications to have it run in the browser. (so you need to reload the page to actually get the results.)

This is the html I used, not very elegant since on every update of the file you need to go to devtools and unregister it. (I only use it for debugging)

This needs to be on a webserver (I used statikk npm -g install statikk)

<!DOCTYPE html>
<html>
<head>
    <title></title>
</head>
<body>
</body>
<script type="text/javascript">
    if ('serviceWorker' in navigator) {
      navigator.serviceWorker.register('/index.js').then(function(registration) {
      }, function(error) {
        console.log('Service worker registration failed:', error)
      })
    }
</script>
</html>
#6

Just tested it, Firefox and Chrome throw the same error

JSON.parse(String.fromCodePoint(65279) + '"helloworld"');

IMHO a good thing, as there is a defined JSON format and invalid strings shouldnt be simply “sanitised”.

2 Likes
#7

Ah so then I would surmise that statikk or Google Chrome’s network stack trimmed the response.

strings shouldnt be simply “sanitised”.

Yea agreed :slight_smile:

1 Like