- Item seller digitalpoint
Move files into R2
- Thread starter fly
- Start date
I haven’t seen a particularly great way to do it because of how WordPress stores media meta data internally.
Having a tool built-in to do it (both for an individual item as well as bulk) does seem like it would be a good addition to the plugin though, so consider it added to the “to do” list.
There would definitely be some gotchas though (for example the existing media URLs would change), so if they are spidered by Google, those URLs wouldn’t be valid any longer. Someone would also need to edit existing posts/pages that were using the old URL as well.
Having a tool built-in to do it (both for an individual item as well as bulk) does seem like it would be a good addition to the plugin though, so consider it added to the “to do” list.
There would definitely be some gotchas though (for example the existing media URLs would change), so if they are spidered by Google, those URLs wouldn’t be valid any longer. Someone would also need to edit existing posts/pages that were using the old URL as well.
Just out of curiosity, how many media items (and approx total size if you know it) are you looking to move if it was an option? Do you have access to your server's CLI interface?
Looking at the internals of WordPress to see what exactly would be needed. Like if someone wants to do it all in a single "click", it might need to be with WordPress's CLI interface simply because the underlying HTTP request is certainly going to timeout if you are trying to process/upload 10,000 media items in a single pass.
And replacing old media URLs with the new URLs is be a little hacky internally (it might just have to be a find/replace database query with how WordPress works internally).
Looking at the internals of WordPress to see what exactly would be needed. Like if someone wants to do it all in a single "click", it might need to be with WordPress's CLI interface simply because the underlying HTTP request is certainly going to timeout if you are trying to process/upload 10,000 media items in a single pass.
And replacing old media URLs with the new URLs is be a little hacky internally (it might just have to be a find/replace database query with how WordPress works internally).
Not sure, as this is for a prospective client. They're getting raked over the coals at WPEngine for an 11GB media gallery. No idea yet of average object size.
WP Offload Media has worked fantastically in the past for S3, so maybe that helps with a roadmap? IIRC, they do it all over HTTP.
WP Offload Media has worked fantastically in the past for S3, so maybe that helps with a roadmap? IIRC, they do it all over HTTP.
Ya, was thinking we could probably queue up a background process via WordPress' cron system to allow a long running process like that to happen without needing to get into the WordPress CLI. Although I'm not 100% sold on that idea because it would be nice to get feedback on the progress rather than queue it up and it happens blindly on the backend.
I've already built the same functionality for the XenForo version of this plugin, and it looks like so:
xenforo.com
Doing it with the CLI tools would also allow additional options like doing multiple transfers in parallel so it would be *much* faster (helpful for a site with a ton of media).
That being said, I'm already working on it. The idea is to make it bi-directional (there may be cases where someone wants to move off R2 as well). Want to be able to do it for individual media as well as a bulk option to move everything on (or off) R2. Tackling the individual move first, then will move on to the bulk. The
I've already built the same functionality for the XenForo version of this plugin, and it looks like so:
[DigitalPoint] App for Cloudflare®
Yeah, it's easier to serve them outside the "assets" directory structure and keep them out of the bucket. (Just tested it that way, and the errors are gone.) I did try configuring CORS with my server URL but still got the errors. With the feature in beta, it might not be working at 100%...
xenforo.com
Doing it with the CLI tools would also allow additional options like doing multiple transfers in parallel so it would be *much* faster (helpful for a site with a ton of media).
That being said, I'm already working on it. The idea is to make it bi-directional (there may be cases where someone wants to move off R2 as well). Want to be able to do it for individual media as well as a bulk option to move everything on (or off) R2. Tackling the individual move first, then will move on to the bulk. The
Move to from R2 to local button becomes Move to from local to R2 if it's not already in R2 (see below).
Not sure what WP Engine's pricing is for storage (or maybe bandwidth), but ya... R2 is probably a good choice considering the first 10GB are free and each GB stored is $0.015 thereafter. So the "out the door" cost for 11GB total would be $0.17 per month if the free 10GB was being used by something else on the Cloudflare account, or $0.02 per month if it uses the 10GB free for that bucket. And no bandwidth costs. I assume "raked over the coals" is something more than $0.17 per month. 😂
Wow crazy it would be that much. WP Engine probably uses something like R2 on the backend and it's all just pure profit for them. I suppose purely as a business, it's not a terrible idea because apparently some people will pay it.To the tune of thousands.
Just depends on the site I guess. I know some sites using in the 100s of TB range worth of R2 media (although not using WordPress).
Just for some background, the way the R2 system works currently is newly uploaded media will end up in R2, but you can have a mix of media locations (for example if a site had media locally before they enabled R2 it doesn't try to move the existing stuff... which I know you know at this point). I've made some changes for the next version that allows a user with an administrator role (it's not allowed for lower level roles like editors) to move an individual media item to R2 if it's stored locally as well as the reverse (someone could move an individual item stored in R2 back to the local filesystem). So that's working at this point.
One thing to be aware of is the URLs of the underlying media change. For example something stored in the local filesystem might have a URL of:
Moving it to R2 and it would have something more along the lines of:
My concern with people moving already existing/in-use media back and forth like that is if the underlying URLs were used (for example if someone was making a post with HTML or using an image on a completely different site)... changing the actual URL where the image is at becomes a problem. The good news is the internal media ID doesn't change, so if WordPress posts/pages are using standard WordPress image/media blocks, the underlying URL change doesn't matter because the URL is generated on demand by the media block.
On to the next part (building a tool that can loop through existing media and use the [now] existing migration function).
I've been thinking more about maybe using the WordPress cron system to schedule tasks. It couldn't really transfer files in parallel, but maybe sites wouldn't want to do that in a real world scenario anyway. Depending on the traffic of the site, you might not even want to migrate a ton (for example 11GB) of data as fast as you can. If you do it too fast you could effectively use a ton of server resources and bandwidth and making the site slower for end users. Like would it be better to transfer the data slowly over 24 hours and end users not know anything is going on, or better to blast through it as fast as you can and make the site terribly slow for end users for an hour or two while all that data is being processed and transferred?
Open to any thoughts/ideas...
One thing to be aware of is the URLs of the underlying media change. For example something stored in the local filesystem might have a URL of:
https://shawnhogan.com/wp-content/uploads/2023/11/some-image.jpgMoving it to R2 and it would have something more along the lines of:
https://media.shawnhogan.com/2023/11/some-image.jpgMy concern with people moving already existing/in-use media back and forth like that is if the underlying URLs were used (for example if someone was making a post with HTML or using an image on a completely different site)... changing the actual URL where the image is at becomes a problem. The good news is the internal media ID doesn't change, so if WordPress posts/pages are using standard WordPress image/media blocks, the underlying URL change doesn't matter because the URL is generated on demand by the media block.
On to the next part (building a tool that can loop through existing media and use the [now] existing migration function).
I've been thinking more about maybe using the WordPress cron system to schedule tasks. It couldn't really transfer files in parallel, but maybe sites wouldn't want to do that in a real world scenario anyway. Depending on the traffic of the site, you might not even want to migrate a ton (for example 11GB) of data as fast as you can. If you do it too fast you could effectively use a ton of server resources and bandwidth and making the site slower for end users. Like would it be better to transfer the data slowly over 24 hours and end users not know anything is going on, or better to blast through it as fast as you can and make the site terribly slow for end users for an hour or two while all that data is being processed and transferred?
Open to any thoughts/ideas...
Yep… you could do rewrites without issues since you can know what the new URL would be, based on the old URL. That definitely would resolve any issues where users were utilizing the underlying URL directly in cases of posts/pages being written with HTML editing or other sites referencing the media. I’m going to add some notes for people using the migration tool with info about them setting up redirects.
So the bulk migration seems to be working now, so the plan is that it will be part of the next version (along with the individual media item migration options). Still want to test it on a few more sites, but it looks like this:
I ran it through tests on a couple sites, going so far as to transfer everything from R2 into the local filesystem, then transferred them all from local to R2 and went back and forth a few times and it works.
I ended up not going the cron/background task route because WordPress crons are triggered by page views, so not that great when you have to rely on traffic for the task to actually happen. There's also situations where some WordPress installations disable WordPress crons.
So the way it works is you can see it progressing as it happens (someone could leave a browser window open to monitor it). It will process however many media items (and their variations) base on the number to process per batch, show you items were done, how many are remaining, etc. It will then refresh the page automatically to start the next batch. Doing it this way seems to be better than a blind, "hope it's happening behind the scenes...". It also gives someone the ability to abort the migration (for whatever reason) simply by closing the browser window it's running in.
I ran it through tests on a couple sites, going so far as to transfer everything from R2 into the local filesystem, then transferred them all from local to R2 and went back and forth a few times and it works.
I ended up not going the cron/background task route because WordPress crons are triggered by page views, so not that great when you have to rely on traffic for the task to actually happen. There's also situations where some WordPress installations disable WordPress crons.
So the way it works is you can see it progressing as it happens (someone could leave a browser window open to monitor it). It will process however many media items (and their variations) base on the number to process per batch, show you items were done, how many are remaining, etc. It will then refresh the page automatically to start the next batch. Doing it this way seems to be better than a blind, "hope it's happening behind the scenes...". It also gives someone the ability to abort the migration (for whatever reason) simply by closing the browser window it's running in.
Last edited:
Rolled out now. Part of the 1.8.1 update.
Similar threads
- Sticky
- Discussion