The life and times of Ranger Pretzel - WordPress WXR File Splitter (RSS XML)

Main

Stuff

Code

WordPress WXR File Splitter (RSS XML)

Written by Ranger Pretzel

Saturday, 20 February 2010

Hey all you WordPress fans. Ever tried to move your blog from one server to another, only to find out that the WXR file that you're trying to import is too big? Well, you could try fiddling with PHP variables to increase the upload size limit, but even then you might run into some sort of execution time-out. Heck, your server admin may not even let you change these settings.

The way around this problem is to split your humongous WXR data file into smaller chunks. Sure you could do it "by hand", but wow, would that take forever.

Have I got a tool for you! I present to you:

The WXR File Splitter v1.02

This software requires a PC running Windows XP, Vista, or Windows 7 with the .NET 2.0 framework installed. (Over 70% of Windows users can run .Net 2.0 applications -- If you don't have it installed, you can download .NET 3.5 directly from Microsoft.)

Just download the file, unzip the file, and run it. It is a very simple 2-step process to break the large file into smaller, more manageable files (as shown in the image above.)

Also, you should note that if any of your imports hang or fail, you can just hit the "reload" button on your browser and click the "resend" button and it should pick-up where it left off...

Comments

Add New

Jim Bowman - Using this splitter

| 99.177.131.xxx | 2010-02-25 22:02:03

Just did so, it worked like a charm. Congratulations on this excellent workaround of downloads too big for WPress to swallow.

dave - problems

| 24.82.177.xxx | 2010-03-02 20:54:42

the splitter seems to have worked, i chose 7 files so each is about 1.5Mb at the moment, however when I do the import, it doesn't import all the stuff in the files, any idea why? I checked the files and there's lots of posts, should be over 1000 but after importing all the xml files I have only 7 posts. confused!

Ranger Pretzel

| Super Administrator | 2010-03-03 00:36:28

Dave: I had a similar problem with my friend's blog. I used the splitter to break the massive WXR file into 27 smaller ones (100 items per file), but even then, each Import job would sometimes hang or fail. So I found if I hit the "reload" button on my browser and pressed the "resend" button, it would pick up where it left off.

dave - oh really?

| 24.82.177.xxx | 2010-03-03 00:35:50

oh really???? I have been trying for hours and hours with no luck, I will try your suggestion! Thank you!

Ranger Pretzel

| Super Administrator | 2010-03-03 00:42:41

Also, one thing to note, (like many people) we were trying to use the "download and import" feature since her site had a lot of images on it. And there was one XML file that just wouldn't finish/complete and I noticed that the file it was trying to download from the WordPress servers was over 10MB. To work around this, I located the single troublesome ITEM in the XML file and deleted it. All the other items imported fine. I then manually recreated that entry via the WP Dashboard/Admin panel.

dave - wow

| 24.82.177.xxx | 2010-03-03 00:43:48

it does appear to be working. . .

Ranger Pretzel

| Super Administrator | 2010-03-03 00:45:25

Sweet. Glad I could help out!

dave - it worked

| 24.82.177.xxx | 2010-03-03 05:08:18

got it done, thanks for the help!

although reloading the page does effectively continue on, i suspect it actually just restarts the process but because the first 'items' already exist it can skip those and get further on each iteration. it doesn't matter really, but at first i tried to do a 7Mb file but it wasn't working because it still has to skip tons of stuff first and can timeout there.

best bet is to follow the advice above, split xml into 100 items chunks and just keep hitting reload on the page when it stops.

i had to upload 46 files it took several hours, but it was better than doing several hundred. hopefully i never have to do that again!

cheers!

dave - local install

| 206.108.31.xxx | 2010-03-03 18:34:05

as someone mentioned elsewhere a good idea is to perform the import on a local install, then export to sql from there, wish i thought of that

Ranger Pretzel - too much work

| Super Administrator | 2010-03-03 20:19:49

Heh, ya know, I thought of doing that, but then realized I would have to install a copy of Linux, Apache, MySQL, and PHP, and THEN install WordPress and THEN import the XML file. (Seemed like a lot of work.) Granted writing an app to split the WXR file wasn't easy either, but it only took me a few hours to get it right... :woohoo:

dave - xamp

| 206.108.31.xxx | 2010-03-03 20:26:14

hehe yeah

can't you install it on windows with xamp fairly quickly? I think I've done it before, but my host is on linux so perhaps this wouldn't work? no idea

http://geeksaresexy.blogspot.com/2006/06/installing-wordpress-locally-under.html

dave - local import problems

| 207.216.177.xxx | 2010-03-13 20:18:53

hey, after all i'm back and trying to import to a local install but i keep getting this error (edited url):

Importing attachment http://xxxxxxxxxxx.files.wordpress.com/2009/08/img_0222.jpg... Remote file error: Remote server did not respond

but when i actually go to
http://xxxxxxxxxxx.files.wordpress.com/2009/08/img_0222.jpg

it works fine, any idea why the import would be failing like this? i get the same error with every single attachment, thanks a lot for any help really appreciated!

dave

Ranger Pretzel

| Super Administrator | 2010-03-13 20:30:30

Actually, yes, I had a similar problem with my friend's site. Just like you, I could download the image in my browser, but the import function would fail. So instead, I downloaded the image directly to the same server we were transferring over to. Then in the XML file, wherever that one file URL path existed (usually in about 2 or 3 places in the XML file), I replaced it with the local server path.

My conclusion is the the remote Wordpress servers are so slow, the the import script times-out or something. But when the file is local, it's able to pull it in immediately without any delay. I only had to do this for 2 files.

Give it a shot and let me know how it works for you...

dave - hmmm

| 207.216.177.xxx | 2010-03-13 20:42:18

the problem is that there are thousands of posts and images there's no way i can do it manually

argh this is frustrating! i guess maybe i could use some kind of script to download all the images if i can isolate all the urls

you're right it's got to be a timeout problem, there has to be a setting somewhere for the timeout that i'm missing, i tried these directions http://nick-hoffman.blogspot.com/2009/09/importing-large-wordpress-blogs.html and then restarted the servers but im still seeing the same problem, wordpress decides in about 1 second that the server didn't respond and moves on to the next file, hmmmmm

Ranger Pretzel

| Super Administrator | 2010-03-13 21:32:47

well, typically, it's only 1 or 2 posts, so you don't have to do all 1000 or so. Just figure out which 1 is causing the problem and manually edit the XML for that one case... Good luck.

dave - :(

| 207.216.177.xxx | 2010-03-13 21:39:53

the posts/categories/tags and all that are importing nicely but every single attachment throws the remote server did not respond error

i changed all the timeout values i could find even digging into the wordpress core like functions.php and http.php but no luck, same problem

i'll wait for some help on the wordpress forums hopefully

but thanks for the suggestion of manually downloading all the images that's a good idea, im trying to think of how to write a script to download a few thousand images using http urls? maybe a batch/shell script could do it? i'm not amazing with scripting and i can't find a script that would do this already unfortunately!

dave	\| 207.216.177.xxx \| 2010-03-14 07:08:50

the cause was that the wordpress.com installation was set to private, even though the files were accessible directly

Todd J. List

| 69.245.72.xxx | 2010-03-19 02:16:47

Thanks so much!

This looks like a cool tool. I will keep this in mind the next time I need to move a big site. Last time I was able to hack the PHP upload limitations. I may not be so lucky next time.

Dave, did you get your issue sorted?

dave	\| 207.216.177.xxx \| 2010-03-19 02:28:26

"kind of" i'm actually running the local import again as i write this, i can't seem to get all the images, or they are going into the wrong upload folders for some reason, for example during the import it will import:
/2009/02/test.jpg

and then when i search for the file on my hard drive, it is under /2009/10/ and i don't even have an /2009/02/ folder so basically a lot of the images don't work right now

dave - done

| 206.108.31.xxx | 2010-03-19 15:33:24

finally got it done, for no reason the images pre 2009/10 were all going into 2009/10 so i had to run sql updates to fix it

UPDATE wp_posts SET post_content = REPLACE ( post_content, 'oldurl/2009/02', 'newurl/wp-content/uploads/2009/10')

and then again for 2009/03 etc.

and then to fix the newer images to point to the new domain:

UPDATE wp_posts SET post_content = REPLACE ( post_content, 'oldurl', 'newurl/wp-content/uploads')

cheers for the help!

mark k. - Thanks

| 84.228.83.xxx | 2010-04-03 03:04:36

That was realy helpfull. The only suggestion, if you ever going to change anything, is to change the option from number of files to maximal size of file (dump the previous items when the current one bypass this limit) maybe the result will be less files to import...

Ranger Pretzel

| Super Administrator | 2010-04-06 23:41:45

That's not a bad idea. It would require a good bit more programming, though. Perhaps I could put an additional line in there noting the estimated file size. That would be pretty easy.

Rich - Doesnt work

| 81.141.101.xxx | 2010-04-09 19:44:15

Hi,
Seems like a great tool, but it doesn't work for me. I've got a 32mb XML file and when I try and split it, it jams on Step 2, Searching for items. Any ideas?

Ranger Pretzel

| Super Administrator | 2010-04-14 08:47:08

Hey Rich,
If you're willing to work with me, I can probably help you out. Please send me a message via the "Contact Me" link at the top of this page. Enter your email address in the email field so that I can contact you back.
Cool?

kizta - Hi

| 115.73.232.xxx | 2010-04-22 11:30:11

Thank you for the great tool, I struggle all day trying to import the file.

Your software did make my life lot easier! :cheer:

Jo Jordan - Thanks.Works perfectly

| 86.31.205.xxx | 2010-05-05 15:53:44

Worked perfectly for me. Thanks very much.

For other noobes like me, I've written up what I did in more detail here.
http://www.flowingmotion.jojordan.org/2010/05/05/step-5-consolidating-my-online-strategy-moving-my-content-from-wordpress-com-to-self-hosted-dreamhost/

OSnapple - Good Looks

| 98.189.183.xxx | 2010-05-10 16:29:25

i have a couple of huge hip hop blogs i run and just want to say thanks for this prog!! it made my life sooo much easier

Sanjay M - Very neat

| 122.167.16.xxx | 2010-05-21 02:49:18

This really made my transition much easier, thank you!
http://msanjay.in

ElJay Arem (IMC OnAir - IMCRad - Number and Size of splitted WX

| 95.118.41.xxx | 2010-06-28 21:57:24

Hi, Ranger !

looks like you did a great job. Yet I have not tested your tool. Better to ask before...

I had the same problem as many. My last backup with smaller size than 8 MB I was able to do in November 2009. It had 7.3 MB. The Export/Import worked fine.

Suddenly and unexpected now the exported file of my blog at wordpress.com has a bigger size of nearby 9 MB.

As I found some tips on the web 1st I changed the datas in wp-config.pho or php.ini, nothing changed; I cannot import that file size locally on my own server (I use WAMP server (very cool freeware !!) on a local Vista mashine, to have my blog working even offline and backupped). Actually I am planning an online version with wordpress.org, too as it gives me more option with plugins Google Analytics etc. ... so I need to know how to import an XML size of bigger > 8 MB without too many try & errors.

The question is how to proceed best migrating the datas and how to split (???). By sure, I dont want import 20 little file...

ElJay Arem (IMC OnAir - IMCRad - Number and Size of splitted WX

| 95.118.41.xxx | 2010-06-28 21:59:23

part 2...

... by sure, I dont want import 20 little files of each 100 Items. - Or must it be like that ??? (sounds like a nightmare)

Shall I empty 1st my existing second blog on WAMP server ? - And then importing the splitted files ?

How big should be the splitted files. The 9 MB big exported XML has 2100 items with nearby 800 postings.

Is it OK to split it in 3x700 Items ? If I do, each file has 2.75 (1st), 2.83 (2nd) and 3.077 MB (3rd part). Or is that size too big ? How big is the risk to loose one of the postings or image links ?

I have a 2nd blog which runs online already, same I like to backup it "offline" with WAMP Server. As it is connected with RSS news syndication, it already has nearby 4000 postings with an exported XML file size of nearby 42 MB.
Do I have to split the 4100 items with each max. 1000 items ? - That means 40 files... isnt a better way possible ? 6x 7 MB file size would be more easier...

Hope I do not bother you with such questions. I am a total...

ElJay Arem (IMC OnAir - IMCRad - Number and Size of splitted WX

| 95.118.41.xxx | 2010-06-28 22:02:08

last part 3...

Hope I do not bother you with such questions. I am a total computer / internet newbee. Just like to use such tools for a better online communitcation with my users... :-)

Warm regards from Europe/North Germany, Hamburg.
ElJay.
------------
Blog addresses:

9 MB backup size: http://radioblog.india-meets-classic.net (Wordpress.com)
42 MB backup size: http://www.imcradio.net/electronics (Wordpress.org system)

Pratik Bagaria - Setting up a new site with exi

| 210.211.139.xxx | 2010-07-09 04:31:27

Hi there,

Firstly thanks for such a good program.
I am redesigning a new site using WP. The existing site is also using WP and i kinda 2 yrs old. There is tons of posts, tags and images.

So I had designed the new site on my mac locally usinf MAMP. All design work is done... Now the most imp. part was getting the data. I thought a simple export - import wud do the work... but i was wring

The exported XML file was 49MB in size.. and while importing locally also it crashed... and only half of the data got imported...

Now I will try splitting the XML file into many parts and then import. Hoping it goes well...!!

Only confusion is how can i delete the half imported data from my wordpress setup???

blueamulet - Having trouble with split file

| 206.162.192.xxx | 2010-08-01 14:48:45

Okay, I thought I had really found the answer to my headaches, but not yet, it seems... I can easily use the splitter, it only takes a few seconds, but my split files are still too big! :pinch:

Well, one was too big while the second was small. I have one xlr file that is about 2.5 MB and conceivably should be split into 3 parts, not just two. But I can't use the last option to make as many parts as I want. How to do this?

Thanks