Migrating From a Static Magento Site to AWS

I have been in the computer industry in various jobs for well over 30 years. During this time, I have seen it come and go and have been part of exciting new technological breakthroughs. Every time a new technology, operating system, or programming environment made its way into the hands of us IT professionals, it was always exciting. But nothing has been as exciting as the advent of the Internet, the World Wide Web, and all the breakthroughs and changes that have occurred as a result.

One of the most exciting breakthroughs to occur in recent years has been the rise of the Cloud. The Cloud concept has allowed for the scalability of any type of application. It has allowed for website businesses to shed the shackles of maintaining and owning hardware.

There are a number of Cloud providers, each of them with their own pros and cons, but the leader in the space hands-down, in my humble opinion, would be Amazon. Of course, people can disagree and claim Azure, Google, Rackspace, etc. offers the best solution. But judging by their sheer volume of applications, time in the space, ease of accessibility, and security I would say Amazon is the leader. Of course, I am open to any thoughts on this.

As for my experience, I have a Masters in Computer Science and have built everything from financial system web platforms, to social media file sharing and collaboration platforms, to e-commerce, to simple vanity sites, and most recently Magento and WordPress sites. For many of these jobs, I had to set up the environment to run either Linux or Windows. Recently though, much of my work has been in the Linux environment.

A number of my clients (Linux-based systems) have thought about or already made the decision to move to the cloud and, most of the time, the choice is Amazon. Hence, this article (Part 1) will be about migrating a single Magento storefront from a locally-hosted solution to AWS. Later articles will be about setting up multiple stores on AWS.

So, what is the current setup?

• A very straight forward two server set-up: One web server and one database server.

• The OS: Ubuntu v14.02 LTS

• The Stack for the Webserver is LAMP

• The 2nd Server is running MySQL

They have eight sites running (none of them extremely high volume at this point).

When I started working with this client, the first issue was a host of normal issues for a growing company. They had a dev platform issue, a DR problem, security issues, command and control issues, and a slew of standard systems administration issues.

Long story short, most of those were sorted and the bigger issue needed to be addressed of how to get scalability and reliability without going so deep in the red that it was simply not possible. We tossed around a number of ideas, including having a back-up and dev solution at the office. Frankly, those concepts really don’t make much sense unless you have a dedicated resource to manage them and the cash to handle the extra expense. So, although it was manageable, it didn’t make much sense when it came to the practicality of it. Hence, we cast our net out and came down on the cloud. It made sense: cost perspective, compatibility, etc.

Amazon was chosen—we didn’t look very far, the current hosting solution offered a cloud solution but it was expensive and not as good... the others didn’t make much sense.

Now the fun part and the purpose of this article: To provide a step-by-step guide to setting up your Magento instance on an EC2 instance.

When I first started this process, Amazon, of course, was of no real help. They had some documents that you could spend hours pouring through (and if you have the time you should... but I didn’t). So, knowing, that I didn’t really know enough about this, I cast my net out for both Amazon and Magento, and what I found on the web was a lot of people that had different but similar issues and hence only partial value—there really was no definitive guide and no explanations.

As an IT guy, I liked the steps but wanted to know the why? Who knows when I may run into a similar issue. The Web is filled with many different articles but not one of them directly relevant to what I needed, or the ones that seemed to be simply breezed over and ended with Now your Magento is up and running in five easy steps. What those articles fail to achieve and realize is that within those five easy steps there are dozens of other things you need to do. So, after getting my instance online, I started to document what I did and realized that I never found a soup-to-nuts article. Hence, I am doing this now... as precursor, I do not know everything and there could be steps that aren’t needed, can be combined, or made better. I am not addressing everything, but simply talking about my experience and the steps I used to create an Ubuntu instance(s) in AWS running, at first, one Magento store.

Overview of the Necessary Steps:

• Set up AWS instance and volume Load necessary files for Magento to install - ref: http://comtechies.com/2014/03/how-to-install-magento-on-amazon-ec2.html

• Copy all existing files from S3 bucket or other location using SCP

• Restore latest database file

• Within database change base_urls to new url or IP

• Within Apache2.conf file change (example included)

<Directory />

       Options +FollowSymLinks -Indexes

       AllowOverride None

       Require all denied



</Directory>



<Directory /usr/share>

       AllowOverride None

       Require all granted

</Directory>



<Directory /var/www/>

      Options +FollowSymLinks -Indexes

       AllowOverride All

       Require all granted

</Directory>



#<Directory /srv/>

#       Options Indexes FollowSymLinks

#       AllowOverride None

#       Require all granted

#</Directory>

Create a new site in the sites available sub-directory.

Create a new .conf file with the following:

 <VirtualHost *:80>

   DocumentRoot /var/www/<your store name>public_html/Irsafsp.com

   <Directory /var/www/<your store name>/public_html/Irsafsp.com>

      AllowOverride All

       Order allow,deny

       Allow from all

   </Directory>

</VirtualHost>

Changing DocumentRoot for the new document root:

sudo a2ensite <name of site conf file>

sudo a2enmod rewrite

sudo a2enmod headers

Make sure .htaccess files are at the root of your Magento etc.

File permissions

14) Directory ownership to www-data

find . -type f -exec chmod 644 {} \;
find . -type d -exec chmod 755 {} \;
chmod 550 mage

When you wish to move a Magento based on a local host or server to an AWS instance there are a number of steps that you will need to take in order to accomplish this goal. As a note, even following these steps you can still run into issues. Magento seems to have lots of quirks. A few things you need to have before you start:

• DO NOT BE AFRAID OF LINUX COMMAND LINE OR TERMINAL MODE as you will need to be very hands on to get things working correctly.

• BE COMFORTABLE WITH SSH AND SCP COMMANDS as they will be your friends through this odyssey.

• KNOW HOW TO SET-UP AND TEST A LAMP SET-UP as this is a big requirement.

• PREPARE TO BE CHALLENGED AND CONFUSED... I know I was.

Now that we have the basic precautions and warnings taken care off let’s have fun.

Phase I: AWS Setup

For my work, I used a tiny processor and 28 gigs of storage (you are allowed up to 30 gigs for the free tier). I would suggest that you use the free tier to get things going and make sure you have the correct steps.

Step 1

Log in to your account on AWS or create one.

Step 2

Click Services and select Compute and then EC2 instance. The reasons for an EC2 instance is that these are the easiest way to set-up an instance.

Step 3

Select the type of Instance and size. I must profess that I don’t have the screen in front of me.

Step 4

Select Operating system I chose Ubuntu 14.04, but you may wish to use another type of Linux environment or a non-Linux environment. This document is concerned with simply setting up a Linux instance.

Step 4.5

You will be prompted to create a security certificate—you will need this file to access the instance from Terminal mode or any other method. This is your login credentials and it is .perm file type. Download that onto your computer and keep it safe!

Step 5

After your instance is configured you will be able to see it in the EC2 console or Dashboard.

Step 6

Click on the tab that says Instance on the left side of the screen.

Step 7

This will give you the heartbeat of your instance and will tell you some important information.

It will show you the public IP address as well as the private IP address you will need to login to your instance.

*Now, assuming that all things have worked correctly and you have an instance of Ubuntu (or your flavor of operating system) with your volume set up as well as your security instance, it is time to start configuring.

Step 8

Before this is done it is highly recommended that you create the security groups. You should have open port 80/443 to open and ssh port to your own IP. This will allow you to control access to your instance from the rest of the universe. *Amazon has some good suggestions for this.

Phase II: Configuring Your Instance

So your instance is set up and now you want to get Magento rolling.

Step 1

SSH into your instance by launching terminal.

Type this command:

ssh -i /Users/xxxxxx/Downloads/Yournamefile.pem ubuntu@ec2-xx-xx-xx-xx.us-west-2.compute.amazonaws.com

Or whatever your specific file locations/public IP address.

Step 2

Once inside your instance, it will look like your standard Ubuntu operating system.

In order to get things rolling since nothing is installed yet, type:

Sudo su - This makes you the super user of the system and gives you the ability to install and do what you want.

Step 3

Type APT-GET Update (Ubuntu only)

This will get and install the necessary updates to bring your version up to date,

Step 4

Type the set of commands below:

For the MySQL installation: Remember to write down the password—you will need it. Strangely, I had problems with various control characters such as hyphens, etc. So stick with the basics and all should be well.

Some commands may not be necessary but can do no harm.

apt-get install  -y lamp-server^

apt-get install php5-curl

apt-get install php5-mcrypt

php5enmod mcrypt

php5enmod curl (this is optional) as in some versions curl is already enabled

service apache2 restart

Step 5

Verify that all is happy with your Apache-MySQL Setup. Type:

cd /var/www

You should see a directory called html, and then type:

cd/html

You should see a file in the directory,

Step 6

Type in your public IP address into the browser and you should see the HTML file (a generic Apache2 message that basically means your webserver is working.)

Now you have a working Apache and MySQL. At this point there a few different trails we can take. One is an actual Migration from an existing Magento store or creating a new store. I am concerned for this document with the migration option. This is more or less a typical use case for most people stuck in this position and the purpose of the document.

In my case, we have an S3 bucket set up for backup purposes of both database and Magento. This is where you will start to use the scp command if you have an S3 or another place where your backups are stored.

If you start to read the documentation and Google this process, you will find countless sources and even Amazon gives you multiple ways of handling the copy process. There is the Amazon CLI which is downloaded and can be used instead of scp commands. How you move files is ultimately your choice.

Before we get going on moving files all over the place, let’s set up MySQL workbench (I use it often and I know that many people like it as well.)

Phase III: Set Up MySQL Workbench

Step 1

Make sure that you have downloaded and installed the latest MySQL workbench.

Step 2

Start MySQL workbench.

Step 3

Click Database and then Manage Connections.

Step 4

Give it a connection name that you will know (i.e. AWS Magento).

Step 5

Tab to connection type and select standard TCP/IP over SSH.

Step 6

SSH Hostname: Will be your specific internal Amazon hostname

SSH Username: ubuntu (This is the default username for AWS)

SSH Password: Ignore this field as you have a login file (remember the .perm file you downloaded when you set-up your instance and needed to SSH into your instance earlier)

SSH KeyFile: Find it and select it.

If you wish you could store your MySQL password (remember the one you created) in the keychain or enter it when you log in.

Step 7

Click Test Connection.

Step 8

If it works, click close and you are ready to access your database instance now via WorkBench.

Now back to our migration...

Phase IV: Continue Migration

Step 1 (assuming you are logged into your instance)

You need to have a home for the file that you created. For the sake of this document, I am going to assume that your Magento file has been tar.gz for easier transportation. If not, there are many online resources to teach you how to tar a file.

Type:

cd /var/www

cd mkdir <whatever you want> but for this case, devMagento

cd /devMagento

There should be nothing in your directory at this point.

Step 2

In my case, I am using an S3 bucket and in order for wget to work from your instance, you need to navigate to your S3 bucket. Navigate to the directory (if any) that you has your file located in it.

Step 3

Find your file and, on the left side, click the box next to the file.

Step 4

Since wget will not work unless the file is public (for some unknown reason) make it public. Now, I am still trying to figure out if it becomes private after the transfer. Or you can download it to your computer and use standard scp or sftp if you have enabled that port.

Step 5

Go back to your terminal of your instance and type the following:

wget http://s3.amazonaws.com/Your directory>/<Yourfilename.tar.gz

This will move your file into the directory you created earlier.

Step 6

Untar your files.

tar -xzif <your filename>.tar.gz

Step 7

This will decompress and copy all of your Magento files into the correct file directories.

At this point, you should have your Magento site files ready to go. However, as this is Magento there are a number of steps that we need to take care of before we get going. This is where things get a bit crazy with Magento, and folks sometimes run into odd problems.

So, let's verify our files and do a bit of house cleaning.

Step 8

In my setup, the name of my site was MyStore.

Step 9

cd /var/www/MyStore/public_html 

because I had all of my files in that directory

Step 10

I see all of my files and do the following command.

1s -a

This shows me that in fact I have my .htaccess file (this file is insanely important)

Step 11

Now I need to clean out my old files:

cd /var/cache

rm -rf *

cd /var/session

rm -rf *

This clears out the garbage.

cd /var/logs

At this point, I would clear out the system.log or any other log that you may have or reduce the amount of stuff in them.

So, now what do you have? You have cleared out the cache and session variables so things are looking good.

Step 12

Cd back to your root.

cd /var/www

Now we need to make sure that the webserver can access these files.

chown -R www-data:www-data /myStore

Step 13

Let’s set the permissions for the directory—this is always a point of problems for Magento because if you screw this up any number of bizarre things can happen or not happen.

find . -type f -exec chmod 644 {} \;

find . -type d -exec chmod 755 {} \;

chmod 550 pear (This is for older Magentos; if you don’t have the directory, don’t worry about it and skip.)

chmod 550 mage

So Magento is set up with the correct permissions and ownership, plus we've cleared out the cache and session variables, so we should be ready! Nope. Still things to do...

Step 14

Now we need to change the local.xml config file so that it points to the correct place.

So go to your sites directory:

cd /var/www/myStore/public_html/app/etc

Step 15

Find the file that says local.xml.

Search for this section in the file:

  <connection>

                   <host><![CDATA[localhost]]></host>

                   <username><![CDATA[root]]></username>

                   <password><![CDATA[YourPassword]]></password>

                   <dbname><![CDATA[YourDatabasename]]></dbname>

                   <initStatements><![CDATA[SET NAMES utf8]]></initStatements>

Change the relevant login information to your database. This will allow Magento to talk to your database with these login credentials.

Quick Review

So, at this point you have set-up your AWS Linux instance, installed the necessary files, and enabled them to get Magento running correctly—copied the Magento files from S3 to your new instance, uncompressed them, and set up the correct permissions as well as your local.xml file configuration.

Are we ready to run Magento yet?

The answer is No. We have done nothing with Apache and MySQL. Currently, you have the web code ready to go, but now we need to load the database and get that set up. Once that is set up there are a few changes that need to be made within the database for Magento to run.

Phase V: Set Up Database

Step 1

If you have not backed up your database I suggest that you do and tar.gz the file when done (depending on the size) if this is a site that has been running for some time then it probably has become quite large. You can back up the site using the mysqldump command from terminal or use WorkBench.

Step 2

You will want to copy the file to your AWS instance.

You can use scp for this command.

sudo scp -i Yourfilename.pem yoursqlname.sql (or .tar.gz) 
  ubuntu@<your publicIP>.us-west-2.compute.amazonaws.com:~/

Step 3

Once the file is uploaded, you will need to login to your instance via terminal.

Login in as root or superuser.

After that, type the below commands to make sure your .tar.gz or .sql file has been moved to your instance.

cd ~

cd /home/ubuntu

Step 4

If your file is where it belongs, then you can either untar its location or copy it to another location and untar at that point.

Step 5

At this point, we are almost ready to start our import process. Let’s go into WorkBench, or if you prefer to use MySQL via command line, do so.

Connect to your AWS database instance by clicking on the correct WorkBench images for that instance.

Once connected you will need to create a schema.

Make sure it matches what you put in your local.xml file:

<dbname><![CDATA[YourDatabasename]]></dbname>

Verify that the schema is created.

Step 6

Now we are ready to do our import, go back to terminal, and type the below command.

mysql -u root -p mydatabasename < mybackup-backup-2016-04-13.sql

This will import your database.

When the database is done we need to make some basic changes to some tables.

Step 7

If everything is done log in to your instance via workbench or terminal depending on your preference:

Execute the following command:

SELECT * FROM <yourdbname>.core_config_data

This will dump the standard configuration that Magento requires to run.

If you are using Workbench look for the following columns:

web/unsecure/base_url

web/secure/base_url

This will reflect your old settings and domain—if you are using AWS and do not have a domain associated with it, you use the public IP address from your console.

Change the values and don’t forget the ‘/’ at the end. For example: http://52.xx.xx.xx/  on both of them.

Update <yourdbname>.core_config_data 
  SET path = 'http://52.xx.xx.xx/’ where path = ‘<what the old one is>’

But, you aren’t done with this yet.

Now if you are using this without SSL (i.e. haven’t installed it yet or its development etc) then you will need to update the following:

web/secure/use_in_frontend = 0

web/secure/use_in_adminhtml = 0

It is crucial that these change over or your backend won’t work properly. Magento gets a bit crazy with these types of settings and you are likely to encounter strange errors like I did.

If you are in Workbench don’t forget to apply the changes or \go for MySQL terminal users (but you already knew this).

Summary

So, at this point, we have created our AWS instance. We've moved over relevant Magento files for our migration (including database file). We have configured the Magento directory by changing relevant file permissions and ownership as well as cleared the cache and session variables. We have verified that our .htaccess file exists in the root directory. We have updated our database to reflect the changes in our URL. So, in theory, everything is set up correctly. Will it work? Nope. Remember, we have not set up our virtual host within Apache yet. (although it was mentioned at the top level).

Phase VI: Setting Up Apache Web Server

Step 1

cd /etc/apache2

Step 2

Nano or your favorite editor apache2.conf

Find the below:

<Directory />
        Options +FollowSymLinks -Indexes
        AllowOverride None
        Require all denied

</Directory>

<Directory /usr/share>
        AllowOverride None
        Require all granted
</Directory>

<Directory /var/www/>
       Options +FollowSymLinks -Indexes
        AllowOverride All
        Require all granted
</Directory>

#<Directory /srv/>
#       Options Indexes FollowSymLinks
#       AllowOverride None
#       Require all granted
#</Directory>

The original Apache will have AllowOveride set to None—change it to All in <Directory /var/www> section.

Do not attempt to run anything yet as it will not work.

Step 3

We need to set-up our virtual host file.

cd sites-available

Step 4

Let’s create our new virtual host file. You can name it whatever you want, but remember to use .conf extension or else the next step will fail! In my case, I used mystore.conf.

nano (Or your favorite editor) <Yourname>.conf

Type:

<VirtualHost *:80>

   DocumentRoot <location of your magento files.. /var/www/myStore/public_html>

   <Directory /var/www/myStore/public_html>

      AllowOverride All

       Order allow,deny

       Allow from all

   </Directory>

</VirtualHost>

Save the file and exit when you are done.

This will tell Apache that all requests to port 80 (HTTP) will get routed to this site. This also enables your .htaccess file to be used.

Step 5

In order to get Apache to know this new site exists, type:

sudo a2ensite <name of site conf file>

Now we need to tell Apache that it is Ok to use rewrites (by default this is turned off), type:

a2enmod rewrite

I also used the following command as I saw an error appear in the logs. Type:

sudo a2enmod headers

That should be it.

At this point, restart Apache server to make sure everything works. Type:

service apache2 restart

Trouble-Shooting

Now, as we know, almost anything can go wrong.

The typical errors that may come up include: the fact that rewrites don’t work or .htaccess file has some issues. Review it. One way to test to see if the rewrites are working is to see if redirects are putting in an index.php/<some file>. If that is the case, rewrites or your .htaccess file is not working properly.

Double check the permissions (this is a really sticking point with Magento). One nasty little bug is the cache location.

This error will show up in the exception.log file or system.log file.

If you have to change it to something else the location is:

nano /var/www/<your location>/public_html/lib/Zend/Cache/Backend/File.php

Make sure the cache_dir location exists on the new server and that permissions are set correctly—not recommended, but 777 those to make sure that things can be written to. This error usually occurs after login, when you try to do something and get some odd number and a print error.

Also, if you haven’t truncated certain system files, this can cause a hang-up and result in some odd messages.

Conclusion

AWS and Magento can be a very powerful team. The two technologies once configured and set up compliment themselves very well. The ability to clone your environment and do many more advanced functions makes it an ideal platform.

In the next article, we will continue to configure AWS instance by adding multiple sites to this one and another instance.