We have a external compliance scan tool hosted internally. It scans for headers, robot.txt, CSP etc… We have 2 identical applications that returned the same exact results before we added one of them into Access. We have a robot.txt file set at the application level for both DNS The DNS using Access is now failing the robot.txt value and not presenting the headers we set in a worker. This same worker is showing the proper values for the DNS not using Access, Ideally we’d like use the worker to resolve the robot.txt.
A post was merged into an existing topic: Adding robot.txt to a worker?