App for Cloudflare® Pro

App for Cloudflare® Pro 1.8.4

Move files into R2

I haven’t seen a particularly great way to do it because of how WordPress stores media meta data internally.

Having a tool built-in to do it (both for an individual item as well as bulk) does seem like it would be a good addition to the plugin though, so consider it added to the “to do” list.

There would definitely be some gotchas though (for example the existing media URLs would change), so if they are spidered by Google, those URLs wouldn’t be valid any longer. Someone would also need to edit existing posts/pages that were using the old URL as well.
 
Just out of curiosity, how many media items (and approx total size if you know it) are you looking to move if it was an option? Do you have access to your server's CLI interface?

Looking at the internals of WordPress to see what exactly would be needed. Like if someone wants to do it all in a single "click", it might need to be with WordPress's CLI interface simply because the underlying HTTP request is certainly going to timeout if you are trying to process/upload 10,000 media items in a single pass.

And replacing old media URLs with the new URLs is be a little hacky internally (it might just have to be a find/replace database query with how WordPress works internally).
 
Not sure, as this is for a prospective client. They're getting raked over the coals at WPEngine for an 11GB media gallery. No idea yet of average object size.

WP Offload Media has worked fantastically in the past for S3, so maybe that helps with a roadmap? IIRC, they do it all over HTTP.
 
Ya, was thinking we could probably queue up a background process via WordPress' cron system to allow a long running process like that to happen without needing to get into the WordPress CLI. Although I'm not 100% sold on that idea because it would be nice to get feedback on the progress rather than queue it up and it happens blindly on the backend.

I've already built the same functionality for the XenForo version of this plugin, and it looks like so:


Doing it with the CLI tools would also allow additional options like doing multiple transfers in parallel so it would be *much* faster (helpful for a site with a ton of media).

That being said, I'm already working on it. The idea is to make it bi-directional (there may be cases where someone wants to move off R2 as well). Want to be able to do it for individual media as well as a bulk option to move everything on (or off) R2. Tackling the individual move first, then will move on to the bulk. The Move to from R2 to local button becomes Move to from local to R2 if it's not already in R2 (see below).

image.png
 
They're getting raked over the coals at WPEngine for an 11GB media gallery.
Not sure what WP Engine's pricing is for storage (or maybe bandwidth), but ya... R2 is probably a good choice considering the first 10GB are free and each GB stored is $0.015 thereafter. So the "out the door" cost for 11GB total would be $0.17 per month if the free 10GB was being used by something else on the Cloudflare account, or $0.02 per month if it uses the 10GB free for that bucket. And no bandwidth costs. I assume "raked over the coals" is something more than $0.17 per month. 😂
 
To the tune of thousands. ;)

edit: Just to be clear, this is obviously not a normal situation.
 
To the tune of thousands. ;)
Wow crazy it would be that much. WP Engine probably uses something like R2 on the backend and it's all just pure profit for them. I suppose purely as a business, it's not a terrible idea because apparently some people will pay it.

edit: Just to be clear, this is obviously not a normal situation.
Just depends on the site I guess. I know some sites using in the 100s of TB range worth of R2 media (although not using WordPress).
 
  • Like
Reactions: fly
Just for some background, the way the R2 system works currently is newly uploaded media will end up in R2, but you can have a mix of media locations (for example if a site had media locally before they enabled R2 it doesn't try to move the existing stuff... which I know you know at this point). I've made some changes for the next version that allows a user with an administrator role (it's not allowed for lower level roles like editors) to move an individual media item to R2 if it's stored locally as well as the reverse (someone could move an individual item stored in R2 back to the local filesystem). So that's working at this point.

One thing to be aware of is the URLs of the underlying media change. For example something stored in the local filesystem might have a URL of:

https://shawnhogan.com/wp-content/uploads/2023/11/some-image.jpg

Moving it to R2 and it would have something more along the lines of:

https://media.shawnhogan.com/2023/11/some-image.jpg

My concern with people moving already existing/in-use media back and forth like that is if the underlying URLs were used (for example if someone was making a post with HTML or using an image on a completely different site)... changing the actual URL where the image is at becomes a problem. The good news is the internal media ID doesn't change, so if WordPress posts/pages are using standard WordPress image/media blocks, the underlying URL change doesn't matter because the URL is generated on demand by the media block.

On to the next part (building a tool that can loop through existing media and use the [now] existing migration function).

I've been thinking more about maybe using the WordPress cron system to schedule tasks. It couldn't really transfer files in parallel, but maybe sites wouldn't want to do that in a real world scenario anyway. Depending on the traffic of the site, you might not even want to migrate a ton (for example 11GB) of data as fast as you can. If you do it too fast you could effectively use a ton of server resources and bandwidth and making the site slower for end users. Like would it be better to transfer the data slowly over 24 hours and end users not know anything is going on, or better to blast through it as fast as you can and make the site terribly slow for end users for an hour or two while all that data is being processed and transferred?

Open to any thoughts/ideas...
 
This all makes great sense. In theory, I could rewrite URLs though to prevent issues, right? I'm not sure the client cares, just thinking out loud. Obviously, this would only work when the entire library has been moved.

And I agree that I don't think speed is an issue.
 
Yep… you could do rewrites without issues since you can know what the new URL would be, based on the old URL. That definitely would resolve any issues where users were utilizing the underlying URL directly in cases of posts/pages being written with HTML editing or other sites referencing the media. I’m going to add some notes for people using the migration tool with info about them setting up redirects.
 
So the bulk migration seems to be working now, so the plan is that it will be part of the next version (along with the individual media item migration options). Still want to test it on a few more sites, but it looks like this:

image.png


I ran it through tests on a couple sites, going so far as to transfer everything from R2 into the local filesystem, then transferred them all from local to R2 and went back and forth a few times and it works.

I ended up not going the cron/background task route because WordPress crons are triggered by page views, so not that great when you have to rely on traffic for the task to actually happen. There's also situations where some WordPress installations disable WordPress crons.

So the way it works is you can see it progressing as it happens (someone could leave a browser window open to monitor it). It will process however many media items (and their variations) base on the number to process per batch, show you items were done, how many are remaining, etc. It will then refresh the page automatically to start the next batch. Doing it this way seems to be better than a blind, "hope it's happening behind the scenes...". It also gives someone the ability to abort the migration (for whatever reason) simply by closing the browser window it's running in.
 
Last edited:
So these client projects got moved to next year, but I should have a bunch of them for you. Just wanted to say thanks again for your work on this.
 
So these client projects got moved to next year, but I should have a bunch of them for you. Just wanted to say thanks again for your work on this.
No worries… it’s there if you need it, and if you don’t, you don’t. 😀
 
Back
Top