Out of memory error. Weird. Ok, you can try a slightly different way, though i think it should be the same. Reinstall Openfire on new server (clean install, btw, how you tried running new Openfire on this new server to see if it launches ok and runs, well at least with only the admin user?).
Then copy Openfire\conf\openfire.xml, Openfire\embedded-db\, and if you use SSL Openfire\resources\security\ from the old installation to the same places in the new installation, overwriting. Also, make sure new installation isn't running while copying.