Stream usages spikes drastically when videos are being processed, then returns to an expected value when finished.
What are the steps to reproduce the issue?
We recently implemented Cloudflare stream to handle videos on our platform. We’ve found that when videos are first uploading and begin processing, our account usage will multiply by many orders of magnitude. Then, after the videos finish processing, the account usage returns a value that is to be expected based on the true amount of content we have uploaded.
For example, when uploading ~20 videos for a total of ~120 minutes in rapid succession, our dashboard shows almost 6,000 minutes in use. Then, after videos have finished processing, it drops to just over 1,000 minutes, which is much more realistic of what we have uploaded. This is a fairly extreme example, but the same behavior occurs in smaller scale as well.
This is causing a big headache, because uploads will start failing due to our account being out of space, but when I go to check our usage, there’s more than enough room. I’ve also had to up our Stream plan multiple times to try and get around this which is getting costly.
Is this expected behavior? Anyway to get around this besides trying to throttle uploads in our platform?
@jturner Are you using the direct creator upload API? If so, what is the maxDurationSeconds value being sent? Additionally, is this behavior also happening when there’s only one video? Lastly, it would be helpful if you could share one (or more) video ID’s where this behavior was observed with us to inspect more thoroughly.
@kkipp Yes, we are using direct creator uploads. Currently we are not specifying a maxDurationSeconds, but I can play around with this as needed.
The behavior does exist even when uploading only one video. For example, uploading a 9 minute video resulted in the following usage changes: 1,247min → 1,007 min.
I think the more likely cause of what you’ve observed is ultimately timing. I believe setting maxDurationSeconds is required for creator uploads regardless of upload method (file or TUS), but if you aren’t setting that, it’ll get a default value of 4 hours. That storage duration is “reserved” in your account until the end-user’s upload is complete and encoded. Thus, if you provision a bunch of direct URLs at once with a really high maxDurationSeconds, you may see a short period of very high storage usage before those uploads resolve. I’ve seen this trip up folks before. Would you share the code you’re using to create the direct upload URL?
The second, less common case is something about the encoding/packaging of an original file leads our initial inspection to think it’s longer than it really is. I’ll take a look at the two IDs you’ve specified just to be sure, but I’ve only seen this a few times. — edit: I checked the original files, their estimated duration matches the output, so this doesn’t seem to apply here.
@tsmith512 apologies for the terribly delayed reply. After some light testing, it appears your suspicions were correct - the lack of a maxDurationSeconds value was causing the 4 hour default to be set and reserving major amounts of time. Thanks so much!
