Teams/Access no longer shows log entries and allows unauthorized user access

I’ve been using CF Teams and Access to manage policies for internal applications and was used to troubleshooting any policy issues by using the Access logs for the past 24 hours.

As of a couple days ago however, no entries appear in my Access logs despite continuing to have users including myself login. What could be causing these logs to be entirely empty?

Even stranger still, is that when I look at my applications in the regular Cloudflare dashboard vs the Cloudflare Teams dashboard, only one application appears in the regular CF dashboard however all my applications appear in the Teams dashboard.

The last log entries appear to be from two days ago despite active users on applications since then.

Another concerning element. In the Cloudflare Launcher, all users are able to see and access applications that they are not permitted to access. These users do not have any policies that would grant them access to some of these applications and the policies were working fine up until two days ago.

The error when I attempt to view the logs says access requests will show up here once an application has been added to access

I have numerous applications already.

Even stranger behavior: when I delete a self-hosted application that uses cloudflared from Teams dashboard, it remains accessible.

The only thing protecting my applications at this point is the Google Auth key which limits logins to accounts within my domain. If that protection was not there at the Google layer, internal services would be publicly accessible. Quite a serious flaw.

Performed the following actions on a server with a tunnel that serves up two applications:

  • Updated Cloudflared to the latest version (2022.1.0)
  • Deleted one of the two applications on that server in Teams dashboard
  • Rebooted the server

Result: Both applications (including the deleted one) are still accessible through the tunnel.

Expected behavior: The untouched application would work as expected, the deleted application would not be accessible.

I’m baffled at this point.

EDIT: I re-added the deleted application and added a policy to only allow a specific email and it finally successfully denied access to all other emails except the one I specified which was the expected behavior. Unfortunately there is still no activity reflected in the Access Log.

Looking into this further, there is something broken with groups and logs in Access.

To test, I took one application that has one policy attached to it. That policy gave include permissions to one group (let’s call it Group1)

Group1 was created months ago and sometimes gets updated via an API integration that I have. This group should restrict access to allow only my email address however it seems to be allowing any user with an account in my domain to access applications that should only be accessible by myself.

I then decided to create Group2. Group2 contains exactly the same rules as Group1.

I then remove Group1 from the application’s policy and add Group2.

After closing my incognito window and logging back in, the group restriction for Group2 works as it should. It allows my email address to access it but denies any others.

When I switch the application’s policy to use Group1 again, it allows access to other accounts again.

TLDR: Two Groups that have the same rules do not behave the same way. Logging is still broken regardless of which group is specified in the policy.

To verify some of my findings, I used another Cloudflare Teams account to test the logging and it is also broken on that account (does not show logins or denials to applications).

As the setup is different in regards to auth, I was not able to test the issue with the Groups.

Not sure why this is strange. A website is accessible by default to the internet. An access policy restricts the ability to reach a resource based on successful authentication and a JWT.

Same as above. A Cloudflare tunnel is a way to expose a resource without opening an inbound port on a firewall, it maps a hostname to an internal origin connection. It doesn’t require an Access policy to make it accessible (many public …e.g. www.example.com) are exposed via Cloudflare tunnels.

The only conclusion one can draw is that the policies / conditions defined in the group are not identical based on the information provided. Since you appear to be familiar with the API I’d suggest using it to list the details of each group and compare along with the associated policy.

I’m rechecking my work previously and I see inconsistencies in the UI vs what is returned by the API.

Immediately upon making a change to a policy, the UI still shows the previous values however the API response shows the change as expected.

Not sure if there is aggressive caching being done on the UI side for policy data but it is certainly throwing off my testing.

Logging still is completely gone however.

The way in which the policies are interpreted does seem to have changed however.

E.g. if you wanted specific email addresses to be allowed to access an application only from specific countries, wouldn’t the structure be:

"include": [
    {
        "email": {
            "email": "[email protected]"
        }
    },
    {
        "email": {
            "email": "[email protected]"
        }
    },
    {
        "geo": {
            "country_code": "US"
        }
    },
    {
        "geo": {
            "country_code": "CA"
        }
    }
],
"require": [],
"exclude": []

Been a while since I played with policies but pretty sure that allows either of 2 email addresses or anyone from the US or CA to login (those are a series of OR statements).

2 Likes

Yeah I see that possibility now however it’s partially driven by some front end errors:

  1. When attempting to make all those statements required and leave include empty, I get the following error which prevents me from saving it: Please add at least one rule to the group. (I can’t remove the include block at all, even when other rules are defined)

  2. When I add one random include rule to save the policy (e.g. the Warp rule) and then include the rest of the statements in the required block, the following warning appears but allows me to save the policy: There is an error with one or more of your rules. Reconfigure by selecting an invalid rule and try again.

So I can’t save a policy with only required statements and I am met with a warning when I add a random include statement to satisfy the include requirement.

EDIT: It’s also not quite clear if multiple selection options in the required statements (like having multiple email addresses defined) function as AND or OR internally. Perhaps a possible improvement to the docs could help clear that up.

So I believe I have the policy side figured out now.

  1. If you have multiple components that you want to be required to enable access, put only one as an include and the rest in the required block in order to be able to save it.

  2. Do not trust the UI after updating an existing policy. Create an API token and use the /access/groups endpoint to ensure the policy was saved properly. It usually is saved properly but you won’t see the update reflected in the dashboard even with a refresh and you’ll tear your hair out wondering why your changes aren’t showing (and make more unncessary changes to your configurations as a result) Also, ignore the warnings you see and trust your gut.

  3. Logs are definitely broken this weekend. fingers crossed it’s resolved shortly on Monday.

  4. If a require rule has multiple selections (e.g. multiple email addresses), Cloudflare interprets rules with the same property name as OR. For example:

 "include": [
    {
        "ip": {
            "ip": "192.168.0.2/32"
        }
    }
],
"require": [
    {
        "geo": {
            "country_code": "US"
        }
    },
    {
        "email": {
            "email": "[email protected]"
        }
    },
    {
        "email": {
            "email": "[email protected]"
        }
    }
]

In the above example, [email protected] OR [email protected] may be used to login even though they reside within the same JSON level and require usually interprets rules in this fashion as AND (I would have thought it would make more sense to nest them within an emails array but that’s just me)

I feel like I’m going a bit mad. Now those rules are not working this morning (or were they ever even after testing yesterday? It’s not clear how caching works as the results of policy changes are not immediately implemented)

As a hypothetical, if I wanted to create a policy/group to ONLY allow [email protected] OR [email protected] to login from the US OR CA, what would the JSON representation of that rule look like? My guess would be the following however this is not working:

"include": [
    {
        "geo": {
            "country_code": "US"
        }
    },
    {
        "geo": {
            "country_code": "CA"
        }
    }
],
"require": [
    {
        "email": {
            "email": "[email protected]"
        }
    },
    {
        "email": {
            "email": "[email protected]"
        }
    }
]

As I test further, it seems like having multiple emails as a requirement is causing it to fail and the failure would apply to any property where multiple options can be entered.

I don’t believe this was an issue before as I’ve been running teams for 3+ months now and only now has this issue occurred. It appears as though Access is requiring that the user have all email addresses specified in the rule but, since a user can only have one email, it will fail 100% of the time. My observation indicates that Access is now parsing that above hypothetical ruleset as:

(ALLOW if request is from US OR CA) AND (ALLOW if request is from [email protected] AND [email protected])

The expected way of parsing the ruleset would be:

(ALLOW if request is from US OR CA) AND (ALLOW if request is from [email protected] OR [email protected])

When I remove all but one of the email addresses, the permissions begin to work for that remaining email address.

The new rule says US OR CA and user has an email for a AND b … as you determined. I believe that jives with the documentation.

That is intended. AFAIK it hasn’t changed in the last 4 years.

You can add multiple emails to the same include IIRC. Look at the common configuration examples.

So I completely understand being able to add multiple emails to the same include but how would I enforce those emails AND more than one country?

So, as mentioned in the hypothetical scenario (two different users only being allowed to login from the US or Canada), I shouldn’t put both constraints as include because then it theoretically allows anyone in the US or Canada to login OR those particular users from anywhere.

I also can’t put the Country constraint in a require block because it will only allow requests from those users if they’re simultaneously in BOTH countries (which isn’t possible)

I also can’t put the emails in a require block due to the same reason as the countries (a user can’t be two people at the same time)

That’s where things seem a bit stuck.

Plea visit the common configurations from the developers docs I referenced above. It’s got a lot of detail that. I believe addresses the question you’re asking.

I finally see.

So if you have multiple conditions that are required, break up the statements into simpler groups and then, for each application, you have to create a policy that combines those groups together on a per-application level.

I suppose my misconception with groups was thinking they allowed you to structure detailed permissions based on your own user/org structures and reuse those groups across applications (and have updates to those group cascade to all applications). Seems like that only works up to a point however and then multiple policies have to be involved.

Thank you! That clears up my questions regarding groups and policies.

In the meantime, I suppose I’ll have to wait until Access logs work again as they’re still broken unfortunately.

Update: Access logs are back up and running again. Seems like that issue was fixed so I’ll close this thread. Hopefully others will benefit from my struggles :slight_smile:

Thank you @cscharff!

This topic was automatically closed 3 days after the last reply. New replies are no longer allowed.