Workers fetch()/POST with text/plain, multiline body?

#1

All,

I’ve been using the following to send log messages to my private server. Each POST contains one log entry, sent as text/plain.

var rayid  = ""
var colo   = ""

function logit(msg) {
    let headers = {
      'X-Log-Auth': LOG_AUTH_KEY,
      'Content-Type': 'application/text'
    }
    let ts     = new Date

    let m = ts.toISOString() + ": " + colo + ": " + rayid + ": " + msg
    return fetch(LOG_URL, {
      method: "POST",
      body: m,
      headers: headers
    })
}

This works great as long as the messages are a single line. Some Lua on my nginx server appends the line to a physical file in my /var/log/ and I’m a happy camper.

However, I would like to send multiple lines in one fetch(). Basically, I want to take e.stack, prepend my prefix to each line, and send it. I’ve found a couple of ways to prepend my prefix, so that’s not an issue. The issue is the damn newlines. No matter what I try, Buffer.from(0x0A) and other crazy concoctions, I can’t seem to get JS to write a real newline into the String. Also, using Buffer.* seems to blow up in Workers. :-/ . I’ve also tried [line1, line2, lin3].join('\n') which also injects the string \x0A instead of real newline.

Does anyone know how I can inject a real newline into the string?

EDIT: Here’s what I get:

2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:  Stack:\x0A2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:  Error: This is a message?\x0A2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:      at failsafe (worker.js:87:13)\x0A2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:      at worker.js:1371:23

And this is what I would prefer it to look like:

2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:  Stack:
2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:  Error: This is a message?
2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:      at failsafe (worker.js:87:13)
2019-09-23T16:20:25.160Z: ORF: 51add889392fc098: OOPS:      at worker.js:1371:23
#2

Ok, I found the problem. It appears that when you’re using nginx’s logging facility, it’s the one mangling the log entries and preventing multi-line messages. As a result, my Worker JS looks like this:

async function failsafe(event) {
  try {
    var response = await handleRequest(event)
  } catch (e) {
    let headers = {
      'X-Log-Auth': LOG_AUTH_KEY,
      'Content-Type': 'application/json'
    }
    let ts = new Date

    const prefix = ts.toISOString() + ": " + colo + ": " + rayid + ": OOPS: "
    var lines = [`${prefix} Stack:`]
    var stack = e.stack.split(/\r?\n/)
    for (var i = 0; i < stack.length; i++) {
      lines.push(`${prefix} ${stack[i]}`)
    }
    const output = await JSON.stringify(lines, null, '\t')
    event.waitUntil(fetch(LOG_URL, {
      method: "POST",
      body: output,
      headers: headers
    }))
    return bail(event, 500, "Internal server error", null)
  }

  return response
}

And then the quick hack in my nginx-config to test it out:

log_format json_log '$log_body';
# ...
        location /log-dev {
                access_log /var/log/dev-access.log json_log;
                error_log /var/log/dev-error.log info;

                set $log_body "";
                set $A_LOG "/var/log/dev-access.log";
                echo_read_request_body;
                lua_need_request_body on;
                access_by_lua_block {
                        -- handle JSON arrays of messages
                        local h, err = ngx.req.get_headers()
                        if h.content_type and h.content_type == "application/json" then
                                ngx.status = ngx.HTTP_OK
                                local cjson = require("cjson")
                                local t = cjson.decode(ngx.var.request_body)
                                local log_body = table.concat(t,"\n")
                                local file = io.open(ngx.var.A_LOG, "a")
                                file:write(log_body)
                                file:close()
                        else
                                ngx.status = ngx.HTTP_OK
                                ngx.var.log_body = ngx.var.request_body
                        end
                }
        }

There’s still some cleanup to do on the nginx side. I don’t like having multiple processes / threads writing to the same file. For testing, it’s fine, but I need a less racy solution.