What is the name of the modal you’re running?
baai/bge-large-en-v1.5
What is the error number?
N/A
What is the error message?
N/A
What is the issue or error you’re encountering
The results of the baai/bge-large-en-v1.5 model are completely different from my local instance
What steps have you taken to resolve the issue?
I’ve been debugging to make sure that this is not related to my code and I’ve made a sample where I compare the the direct similarity of the same hard coded piece of text and the results are completely off. I did normalize the vectors already.
What are the steps to reproduce the issue?
- Take a long piece of text, in my case about 2,300 tokens.
- Generate embeddings using Cloudflare AI using bge-large-en-v1.5.
- Generate embeddings using Ollama using bge-large-en-v1.5.
- Take 1 word which is definitly inside the text.
- Convert the 1 word into a token embedding.
- Generate cosine similarity for the Cloudflare embeddings and the single word embedding.
- Generate cosine similarity for the Ollama embeddings and the single word embedding.
The results are off completely between the two models, and it seems that the local (non cloudflare) model is working less good.
To me it seems like Cloudflare is maybe using a different model or is doing something else, or possibly expects the prompt in a different format. Either way, this is stopping me from moving to Cloudflare because the results are just much worse.
Is there any insights into what Cloudflare is doing extra?
