Best way to restrict access to my worker for an extension?

I just created my first worker. I am accessing this worker from a Chrome extension (e.g., chrome-extension:///popup.html). If I want to only allow my extension to call this worker, what would be the best way to go about doing this?