Periodic Data Updates for D1 Database

Developers D1
1

Hi All.

I have a Question regarding the need to update D1 Data every few Seconds.

The Upstash Database as written in the cloudflare documentation allows me to continuously fetch every second/s fresh data from 3rd party API Providers and store it in a distributed low latency database with the possibility to avoid make same call to the api endpoint while a api call is allready processing aka possibility to discard further parallel calls till the ongoing call is finished.
That way i am able and have the possibility when a visitor hit my cloudflare worker website to deliver the data like real time cache without to be exposed to a bottleneck becouse of bad 3rd party API Provider performance.

https://qstash-examples-periodic-data-updates.vercel.app/

My Question is How can i update D1 Database Data every few Seconds from a 3rd party API Provider with the D1 Distributed Database and cloudflare workers ?

The Workers Cron Triggers don’t allow to call a Worker every few seconds.
At the Moment the shortest period is 60 Seconds aka every 1 Minute.

If Workers Cron Triggers could be executed every second this could help a lot but at the moment this is not possible sadly with Cloudflare Workers Cron Triggers.

Is there a SERVERLESS Cloudflare possibility to periodical call workers every few seconds for fetching fresh data from 3rd party API Providers and store it in the D1 distributed database ready for when the visitor hit the cloudflare workers to deliver the data to him to avoid bottleneck problems at the 3rd party API Provider ?

Thanks in Advance for the helpful Answers and Tips.

4

Sadly nobody Answered how is it possible to update a D1 Database every few seconds as the Cron Triggers allow only a Update Trigger every Minute.

As it looks like this is not yet possible with the D1 Database yet,
I have a new Question.

Is it possible to do Hybrid Replication from a External SQL Lite Origin Datbase to the Cloudflare D1 Database ?

Google as a example gives it’s Customers the Possibility to Replicate SQL Data from a External Origin SQL Database to their SQL Cloud Product Solution.

This page describes a configuration that replicates data from a source database server to MySQL replicas. This configuration is sometimes referred to as an external server configuration.

Use cases for external server configuration

External server configuration helps you achieve the following goals:

  1. Migrate your data from your self-managed MySQL server to Google Cloud with a minimum of downtime.
  2. Retain colocation and control of your server while off-loading the administration of the replicas to Cloud SQL.This use case is sometimes called a hybrid cloud. Replication between your self-managed server and the Cloud SQL replica continues indefinitely.

Having the possibility to replicate Data from a existing external SQL Lite Origin Database to the D1 Cloudflare Database would help a lot to quick jump start and use the Cloudflare D1 Database as no new Programm Application Code needs to be written by a Computer Programmer for the D1 Database to keep writing, importing and updating the Data.

A search on the internet for SQL Lite replication proved that it exist this functionality.

SQLite DATABASE REPLICATION made easy

Each master database can have one or many replicas. These replicas can be handled by the same application or another one, in the same device or a distant one. Each connection can use a different communication protocol, and a different connection direction.

no changes in your code

In the master application you just need to change the URI filename to something like this:

“file:/path/to/file.db?replica=master&slave=tcp://server.ip:1234”

It would be great if it is possible to just give D1 a url where it can replicate the data from and then only
reading from the D1 Databse is done inside the cloudflare workers.
This would simplify a lot and a lot of customers i am sure would start use D1 additionaly as their data can easy be replicated from their origin server to the cloud and then from their used.

Also based on the Cloudflare D1 Database Documentation
it is possible to create Indexes for specific Table Keys for a much faster querying!

Use indexes

Indexes enable D1 to improve query performance over the indexed columns for common (popular) queries by reducing the amount of data (number of rows) the database has to scan when running a query.

When i try do this in the Dashboard this possibility does not exist however.
It would be great if the index functionality would be availble in the Dash board to have the possibility to create additional indexes in the Dashboard.

Thanks a lot.