Clamav-freshclam cannot download any file using Cloudflare Inc ECC CA-3 certificate due to a 'cookie-alert'

References due to the number of URLs constraint:

  • clamav_url=https://database.clamav.net
  • clamav_file_url=https://database.clamav.net/daily-26108.cdiff

In Ubuntu, clamav-freshclam is a systemd service which tries to periodically update its database by downloading some files from clamav_url. This site uses Cloudflare Inc ECC CA-3 certificate.
The following errors appear:
● clamav-freshclam.service - ClamAV virus database updater

Mar 15 08:58:13 hostname freshclam[403486]: Mon Mar 15 08:58:13 2021 → ^Download failed (77) Mon Mar 15 08:58:13 2021 → ^ Message: Problem with the SSL CA cert (path? access rights?)
Mar 15 08:58:13 hostname freshclam[403486]: Mon Mar 15 08:58:13 2021 → ^getpatch: Can’t download daily-26108.cdiff from clamav_file_url
Mar 15 08:58:13 hostname freshclam[403486]: Mon Mar 15 08:58:13 2021 → ^Incremental update failed, trying to download daily.cvd

The recurring error message “Problem with the SSL CA cert” appears to be misleading because:

  • I have downloaded the official Cloudflare CA public certificates from:
    115000479507-Managing-Cloudflare-Origin-CA-certificates#h_30cc332c-8f6e-42d8-9c59-6c1f06650639
  • I also downloaded Digicert CA certificates used to sign Cloudflare’s ones from:
    203041594-Cloudflare-SSL-cipher-browser-and-protocol-support#h_2YLgMUktyQ0fhxN3VfzLFY
  • I have added these CA certificates into the ssl certs store with:
    sudo dpkg-reconfigure ca-certificates
  • I successfully tested them by downloading clamav_url :
    wget $clamav_url
    –2021-03-15 09:29:49-- ```clamav_url``
    Resolving clamav_url (clamav_url)… 104.16.219.84, 104.16.218.84, 2606:4700::6810:da54, …
    Connecting to clamav_url (clamav_url)|104.16.219.84|:443… connected.
    HTTP request sent, awaiting response… 200 OK
    Length: unspecified [text/html]
    Saving to: ‘index.htm

However, downloading a file from clamav.net remains impossible:

wget --debug $clamav_file_url
DEBUG output created by Wget 1.21 on linux-gnu.

URI encoding = ‘UTF-8’
Converted file name 'daily-26108.cdiff' (UTF-8) -> 'daily-26108.cdiff' (UTF-8)
--2021-03-15 09:32:09--  ```clamav_file_url```
Certificates loaded: 121
Resolving clamav_url (clamav_url)... 104.16.219.84, 104.16.218.84, 2606:4700::6810:da54, ...
Caching clamav_url => 104.16.219.84 104.16.218.84 2606:4700::6810:da54 2606:4700::6810:db54
Connecting to clamav_url (clamav_url)|104.16.219.84|:443... connected.
Created socket 3.
Releasing 0x000055eccbc64710 (new refcount 1).

---request begin---
GET /daily-26108.cdiff HTTP/1.1
User-Agent: Wget/1.21
Accept: */*
Accept-Encoding: identity
Host: clamav_url
Connection: Keep-Alive

---request end---
HTTP request sent, awaiting response... 
---response begin---
HTTP/1.1 403 Forbidden
Date: Mon, 15 Mar 2021 08:29:42 GMT
Content-Type: text/html; charset=UTF-8
Transfer-Encoding: chunked
Connection: keep-alive
X-Frame-Options: SAMEORIGIN
Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Thu, 01 Jan 1970 00:00:01 GMT
Set-Cookie: __cfduid=dee3461814d0bc5359cb32c4fc6b276391615796982; expires=Wed, 14-Apr-21 08:29:42 GMT; path=/; domain=.clamav.net; HttpOnly; SameSite=Lax
cf-request-id: 08d69a6a530000ee5c5ba46000000001
Expect-CT: max-age=604800, report-uri="```report-uri_cloudflare_url```"
Vary: Accept-Encoding
Strict-Transport-Security: max-age=15552000
X-Content-Type-Options: nosniff
Server: cloudflare
CF-RAY: 63046023bcc7ee5c-CDG

---response end---
403 Forbidden
cdm: 1

Stored cookie clamav.net -1 (ANY) / <permanent> <insecure> [expiry 2021-04-14 10:29:42] __cfduid dee3461814d0bc5359cb32c4fc6b276391615796982
Registered socket 3 for persistent reuse.
Parsed Strict-Transport-Security max-age = 15552000, includeSubDomains = false
Updated HSTS host: clamav_url:443 (max-age: 15552000, includeSubdomains: false)
URI content encoding = ‘UTF-8’
Skipping 512 bytes of body: [<!DOCTYPE html>
<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->
<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->
<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->Skipping 512 bytes of body: [
<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->
<head>
<title>Access denied | clamav_url used Cloudflare to restrict access</title>
<meta charset="UTF-8" />
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />
<meta name="robotsSkipping 512 bytes of body: [" content="noindex, nofollow" />
<meta name="viewport" content="width=device-width,initial-scale=1" />
<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/main.css" type="text/css" media="screen,projection" />


</head>
<body>
  <div id="cf-wrapper">
    <div class="cf-alert cf-alert-error cf-cookie-error hidden" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>
    <div id="cf-error-details" class="p-0">
      <header class="mx-auto pt-10 lg:pt-6 lg:px-8 w-240 lg:w-Skipping 512 bytes of body: [full mb-15 antialiased">
         <h1 class="inline-block md:block mr-2 md:mb-2 font-light text-60 md:text-3xl text-black-dark leading-tight">
           <span data-translate="error">Error</span>
           <span>1020</span>
         </h1>
         <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">Ray ID: 63046023bcc7ee5c &bull;</span>
         <span class="inline-block md:block heading-ray-id font-mono text-15 lg:text-sm lg:leading-relaxed">2021-03-15 08:29:Skipping 512 bytes of body: [42 UTC</span>
        <h2 class="text-gray-600 leading-1.3 text-3xl lg:text-2xl font-light">Access denied</h2>
      </header>

      <section class="w-240 lg:w-full mx-auto mb-8 lg:px-8">
          <div id="what-happened-section" class="w-1/2 md:w-full">
            <h2 class="text-3xl leading-tight font-normal mb-4 text-black-dark antialiaseSkipping 512 bytes of body: [d" data-translate="what_happened">What happened?</h2>
            <p>This website is using a security service to protect itself from online attacks.</p>
            
          </div>

          
      </section>

      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">
  <p class="text-13">
    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">63046023bcc7eSkipping 512 bytes of body: [e5c</strong></span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 176.139.106.168</span>
    <span class="cf-footer-separator sm:hidden">&bull;</span>
    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="```error-landing_cloudflare_url```" id="brand_link" target="_blank">Cloudflare</a></span>
    
  </p>
</div><!-- /.error-footer -->


 Skipping 363 bytes of body: [   </div><!-- /#cf-error-details -->
  </div><!-- /#cf-wrapper -->

  <script type="9e597140c3fb152be366936d-text/javascript">
  window._cf_translation = {};
  
  
</script>

<script src="```rocket-loader_cloudflare_url```" data-cf-settings="9e597140c3fb152be366936d-|49" defer=""></scripSkipping 18 bytes of body: [t></body>
</html>
] done.
2021-03-15 09:32:09 ERROR 403: Forbidden

This very strange ‘cookie-alert’ message appears.
Any suggestion about how to solve this issue?

For the record, I have no such issue when downloading the same file from chrome.

Hi @actionmystique!

Loading https://database.clamav.net/daily-26108.cdiff in a web browser we observe a Cloudflare Challenge that wget does not support.

How about using curl to download it?

@amayorga already replied the answer to it.

clamav_url=https://database.clamav.net

With curl, the error pops up with a different code and even less explanation:
curl --verbose clamav_url/daily-26108.cdiff

  • Trying 104.16.219.84:443…
  • Connected to clamav_url (104.16.219.84) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
    CN=sni.cloudflaressl.com
  • start date: Aug 15 00:00:00 2020 GMT
  • expire date: Aug 15 12:00:00 2021 GMT
  • subjectAltName: host “clamav_url” matched cert’s “clamav_url”
  • issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x563792c483a0)

GET /daily-26108.cdiff HTTP/2
Host: clamav_url
user-agent: curl/7.74.0
accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 256)!
    < HTTP/2 403
    < date: Tue, 16 Mar 2021 19:13:23 GMT
    < content-type: text/plain; charset=UTF-8
    < content-length: 16
    < x-frame-options: SAMEORIGIN
    < cache-control: private, max-age=0, no-store, no-cache,
    must-revalidate, post-check=0, pre-check=0
    < expires: Thu, 01 Jan 1970 00:00:01 GMT
    < set-cookie: __cfduid=ddcb7b193b610184915d632a777cc506e1615922003;
    expires=Thu, 15-Apr-21 19:13:23 GMT; path=/; domain=.clamav.net;
    HttpOnly; SameSite=Lax
    < cf-request-id: 08de0e16a200000165631ec000000001
    < expect-ct: max-age=604800, report-uri=“cloudflare_report_url”
    < strict-transport-security: max-age=15552000
    < x-content-type-options: nosniff
    < server: cloudflare
    < cf-ray: 63104c6a9e3a0165-CDG
    <
  • Connection #0 to host clamav_url left intact
    error code: 1020

Is there any way to avoid/skip the challenge when using a CLI tool
since it is used to detect the client browser (“Checking your browser
before accessing clamav.net”)?

Wait a bit.
Do you use the method from above (using wget or curl) to update your ClamAV virus database?

Shoudln’t t that be installed by a package like if Debian OS - which I have and use, it has automatic updates and checking for newer version (via cronjob).

Or I am missing the point here?

Have you tried installing ca-certificates package at your host/origin?

Or you are trying to download a file daily-26108.cdiff from your Website because of …?

Cannot you compress/archive or “zip” it and download like that? - oh, ok, not your website, so you cannot do it like that.

yes.

  • clamav-freshclam is installed and tries to perform the updates regularly
  • wget/curl CLI methods are just used to demonstrate and debug what happens
  • downloading the file from a browser is just way to rule out some potential cause(s)

Already installed.

There is no such issue on another Ubuntu device with the exact same SSL and freshclam configurations and located on the same private network as the failing device sharing the same IP public address.

Is it possible that cloudflare enforces a limit on the number of devices which are allowed to download from https://database.clamav.net/daily.cvd?

Hmmm, I thought something might be off. Thanks @actionmystique for the detail. and everyone for the comments.

As to your last though, this is where I am headed and there is nothing else wrong, either a Captcha Wall or a ‘429 Too Many Requests’ as per the log below.

Not sure but at this stage I am looking at other mirrors as it seems that CF might have made this impossible for standard clamav users.

< HTTP/1.1 429 Too Many Requests
< Date: Sun, 21 Mar 2021 04:54:35 GMT
< Content-Type: text/plain; charset=UTF-8
< Content-Length: 16
< Connection: close
< Retry-After: 10053
< X-Frame-Options: SAMEORIGIN
< Cache-Control: private, max-age=0, no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< Expires: Thu, 01 Jan 1970 00:00:01 GMT
< Set-Cookie: __cfduid=d3baba18d53cfbe3a6e57a29a6037157f1616302475; expires=Tue, 20-Apr-21 04:54:35 GMT; path=/; domain=.clamav.net; HttpOnly; SameSite=Lax
< cf-request-id: 08f4bba1f00000027bfe1a2000000001
< Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
< Strict-Transport-Security: max-age=15552000
< X-Content-Type-Options: nosniff
< Server: cloudflare
< CF-RAY: 633495498866027b-SJC
< 
Time:    0.0s, ETA:    0.0s [========================>]        16B/16B
* Closing connection 11
ERROR: downloadFile: Unexpected response (429) from https://database.clamav.net/daily.cvd
ERROR: getcvd: Can't download daily.cvd from https://database.clamav.net/daily.cvd
Giving up on https://database.clamav.net...
ERROR: Update failed for database: daily
WARNING: fc_update_databases: fc_update_database failed: HTTP GET failed (11)
ERROR: Database update process failed: HTTP GET failed (11)
ERROR: Update failed.

I have found the cause of the issue: clamav does not support symlinks for any of the following:

  • /etc/clamav
  • /etc/ssl
  • /etc/ssl/certs
  • /var/lib/clamav

If I make sure there is no symlink anymore for any of the above folders, then the issue is worked around:

freshclam --debug --verbose

  • Trying 104.16.219.84:443…
  • Connected to clamav_url (104.16.219.84) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: C=US; ST=CA; L=San Francisco; O=Cloudflare, Inc.; CN=sni.cloudflaressl.com
  • start date: Aug 15 00:00:00 2020 GMT
  • expire date: Aug 15 12:00:00 2021 GMT
  • subjectAltName: host “clamav_url” matched cert’s “clamav_url”
  • issuer: C=US; O=Cloudflare, Inc.; CN=Cloudflare Inc ECC CA-3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55631ca7a1e0)

GET /safebrowsing.cvd HTTP/2
Host: clamav_url
user-agent: ClamAV/0.103.0 (OS: linux-gnu, ARCH: x86_64, CPU: x86_64)
accept: /

I’m not sure whether this symlink sensitivity is by design or a bug.

1 Like