Load balancing DNS queries if traffic originates within CF

I would like to use the load balancer feature, but we were charged for an enormous number of DNS queries when I set it up initially, so I need to try to avoid that issue.

Our traffic all originates from another customer within Cloudflare, so I hoped that if we can get the load balancer properly set up with orange cloud proxy, that the number of DNS queries will be much lower.

Our traffic is all websockets over port 80, so it uses a supported port and protocol, and the default cross-user restriction has been removed. I configured the LB as gray cloud, with 1 DNS-based origin, and 1 IP-based fallback. Traffic was routing just fine in this configuration.

As soon as I enabled orange cloud on the load balancer, all DNS routing stopped functioning immediately. I waited for some time, but it never restored to service, so I had to revert. In the meantime, we were charged for 350 million DNS queries, because I had set the TTL extremely low, so we could fall back to gray cloud routing during the test. It was a painful lesson that cost me $500 in about a week and a half of traffic. CF billing has told me to pound sand when I asked for a refund, so it is what it is.

Can anyone help me out? I’d really like to get this set up properly, and also not pay like 2 grand a month for a simple load balancing feature, which is ridiculous.

Thanks!

I have to say, Cloudflare has a beautiful product, but their support is bad, period. They need to up their game, bigtime. I’ve gotten really responsive and helpful support from many other service providers, but CF really stands out for their poor performance in this regard. For example, this issue here - I received one response from support about 2 1/2 weeks after I submitted the ticket. Now, I get it, we are small potatoes, but that doesn’t stop the other service providers we use from responding promptly and helpfully. Another case in point is this post to the “Community” - crickets… Anyone? There is zero support for this product. If you have an issue, figure it out yourself. If you have a problem with billing, get ready to be ignored or have your ticket closed without further consideration.

Like I said, love the product, but wow what a huge disconnect between that and the support. If you have an issue, you’re on your own unless you’re paying for an Enterprise account, I have to presume!

Curious how low did you set TTL for you to generate 350 million DNS queries and what traffic requests amount was pushed through the Load Balancers?

Probably best advice I can give if you want to continue working on your Cloudflare Load Balancer setup without unexpected costs, is to setup a staging instance of your setup - so a separate test CF Load balancer/pool/origins which don’t pass production level traffic through them. Then test away until you get it working as expected before switching to production usage.

And generally, I use IP based origin setups not DNS based origins.

Hi @tom.gnade

Can you elaborate more about what you need the load-balancing to accomplish? Are you needing to split traffic between servers per request type? or load balance traffic between multiple origin servers with monitoring? Or use advanced load balancing features like multi-origin in geographically distinct regions?

If you need only basic load balancing to randomly split traffic between any number of origin servers in the same physical location than you can do this for free in Cloudflare.

Thanks, yes I agree that would be ideal. We had to meet a deadline so I just had to use our prod environment to implement. In the future, I will try to make sure we have a test environment.

I now realize that the load balancer wants to use IP addresses only. I have been able to get it working by setting it up that way, and using orange cloud for the balancer. When it wasn’t working before, I had the TTL set short so I could fail back from orange to gray quickly and have the records propagate, so I wouldn’t have an extended outage.

When I tested the LB set up correctly a few days ago, it apparently again generated over 1.5 million queries somehow in the course of a few hours. Something is just wrong, because the number of requests was nowhere near that number, and the TTL was set to automatic.

The LB function just doesn’t work for our use-case. They charge way too much money for the traffic, and so it makes the feature basically useless for real-world volumes of traffic. Also, there is no way for me to observe the billable unit. The DNS section of the analytics screen only shows instantaneous rates of queries, not the sum total, and is severely limited in lookback range. There is no possible way for me to check or observe the billable unit that CF is using to charge for this feature, which is ridiculous.

@user2765 I wanted to test the load balancing feature because it would allow me to spread load between several servers, it implements health checks against the origins, and it would allow me to bypass my edge load balancer by using a direct IP-address reference to my server as the fallback pool.

Just those features were the ones enabled when I was charged for 430M DNS queries, so I can assure you it is far from a “free” feature. I don’t know of any other way to split traffic.

I also wanted to be able to offer regional failover as a possible setup for our clients, so that was something I was going to explore. Unfortunately, I never got that far because they are somehow counting a number of DNS queries that is disconnected from reality. It’s just not a usable feature, if they are going to try to charge several thousand dollars a month for a single stream of traffic to one or two backend servers. That’s several times the cost of the servers. It makes no economic sense. I’ve come to the conclusion that I have to just stick with their basic DNS services for now. I already have an edge load balancer, so their “nice to have” feature is now a no-go for me, unless they can somehow explain to me how they are measuring millions of DNS queries for a few thousand requests.

Not sure how you managed that - checked on one of my CF Load Balanced sites and seeing from 530K LB requests, I made ~400K DNS requests for that hostname.

One possibility is health checking on CF and 3rd parties driving up requests/DNS requests?

But yeah I agree billable consumption should be better shown.

Yes, the actual Load Balancer product does implement desirable features for a load balancing such as health checks, failover, customisable rules, and importantly geographic region “intelligent routing”; but that all comes at a cost.

My comment yesterday was to say that if you only want to distribute traffic between multiple servers then you can accomplish that for free aside from the actual Load Balancer product.

If all your servers are in the same datacenter or relatively close physically then I can assume you don’t need geographic region “intelligent routing” to your origin pools.

The “free” method I refer to is not the actual Load Balancer product but instead a virtual “pool” by way of how Cloudflare will randomly pick an IP when you have multiple for the same record and is :orange: proxied.

Simply add all your IP’s to the DNS and switch on the :orange: proxy. From the public facing side the DNS will only show between 1 & 3 of Cloudflare’s IPv4/IPv6 depending where in the world you are. From the Cloudflare side, your traffic will be fairly evenly distributed across all the origin IP’s.

Indeed this is very simple, but if simple is what you want then solved for free!

Here’s an example:

WX20220314-091214

WX20220314-092133

https://developers.cloudflare.com/load-balancing/

As for the health checks, you can implement a 3rd party tool or create one yourself. When you get an alert you can either manually fix it or you can tie in the Cloudflare API to update your DNS and drop the respective IP from your “pool”.

Cloudflare also offers a standalone Health Check product.

Now that was an excellent answer, thank you so much! I still don’t have a satisfactory explanation from CF for why they are recording such a high volume of DNS queries, I will keep pursuing that with them. The trick with multiple matching records is excellent, I will use that, I already have multiple monitors implemented so not concerned about that so much.

Thanks again!

Glad to head it makes more sense to you now and is the solution to your issue. I would have explained in more detail in my first comment but I just assumed you’d know what I was saying.

The actual Load Balancer product is very good and I don’t think its overpriced for the intended use-case. I’m sure Cloudflare could have explained this to you but to be fair it’s not really their job to evaluate if you are using their appropriate services or not for your use case.

This DNS behaviour I described is standard for almost all DNS host providers, and its standard behaviour in almost all self-hosted linux DNS software, so to be fair its not Cloudflare’s job to give personalised answers on support tickets to educate their customers on basic stuff like this either.

Hope this points you in the right direction, let me know if you need further help on this.

This topic was automatically closed 15 days after the last reply. New replies are no longer allowed.