isotope|eleven

Here at Isotope11, we've been working on making our clients' server setups fault tolerant and just all around better. We've been having problems with our current cloud server provider, and so we decided to seek out the best possible solution for our clients. To do this we sought out a full failover setup using Amazon's AWS services.

Why should I use a failover with Amazon?

As of today, Amazon has dropped their prices is one reason. Not only can you spawn up servers in two different regions with Amazon EC2 under the traditional master/slave setup and have Capistrano deploy to both of these servers, but you can also have an Amazon RDS instance between the two to make sure everything in the database stays in sync. Now where things get tricky is keeping your file uploads in sync. This can be easily accomplished with a FUSE-based file system backed by Amazon S3.

How do I do all of this?

Easy. I'll show you. First you'll need to sign up for Amazon's AWS.

Here is the list of services you'll need to sign up for:

• Some type of DNS Failover service.
• Amazon's Elastic Compute Cloud in two separate Regions. (i.e. US East and US West)
• Amazon's Multi-AZ Relational Database Service (Reserved).
• Amazon's Simple Storage Service.

Setting up the DNS Failover

You'll have to check with your current DNS provider (if you have one) to see if they offer a DNS failover service. I'll just show you how it's done with TZO-HA. In the master/slave relationship of failover server setup, we'll use TZO's Failover Switch-Back which will perform DNS failover to the slave in the event of master going down and automatically switch back DNS Failover to master when the master comes back on online.

Observe:

There isn't much left to explain here. Further information can be found on your DNS provider's website.

Setting up your EC2 Instances

Once you've signed up for AWS then you'll be ready to spin up your EC2 instances. You'll have to take into consideration what type of EC2 instance you'll need depending on how much your application will be used and how big your application actual is. If you have a large application with high traffic, then you'll probably want to go with the m1.large instance. If you're Google, FaceBook, or Twitter you'll want to look into something more extreme.

Launching different regional instances

Once you're in the AWS Management Console on the EC2 tab you'll need to figure out (depending on where most of your traffic comes from) which will be the master and which will be the slave server. In our case most of the traffic comes from the US East region, so I will use that as the master for this example. We'll also be using Official Ubuntu 11.10 AMD64 Server AMIs.

Master Instance:

• Select Region: US East (Virginia) under the Navigation
• On the EC2 Dashboard under Getting Started click on Launch Instance
• Use the Classic Wizard and click Continue
• On the next step select the Community AMIs tab
• In the Search field enter ami-baba68d3 (this is the Official Ubuntu 11.10 amd64 Server AMI for us-east-1) and click Select
• Choose an Instance Type, Availability Zone and click Continue
• Select aki-825ea7eb from the Kernel ID drop-down menu (this is the official Ubuntu 11.10 amd64 Server AKI), leave the rest as is and click Continue
• Fill in Key and Value on the next step to add tags for easier administration of your EC2 infrastructure.
• Now you'll need to create a Key Pair by filling in the required fields, then click Create & Download your Key Pair as it's required for SSH access your EC2 instance, and click Continue
• Now we'll want to create a new security group in the next step. Give this a good Group Name and Group Description. Under Inbound Rules and Create a new rule: select SSH, HTTP and any other protocol you may need followed by clicking Continue
• Review that everything looks correct to your needs and click Launch

Note: Slave instance is almost identical, just note the Region, AMI and AKI.

Slave Instance:

• Select Region: US West (N. California) under the Navigation
• On the EC2 Dashboard under Getting Started click on Launch Instance
• Use the Classic Wizard and click Continue
• On the next step select the Community AMIs tab
• In the Search field enter ami-6da8f128 (this is the Official Ubuntu 11.10 amd64 Server AMI for us-west-1) and click Select
• Choose an Instance Type, Availability Zone and click Continue
• Select aki-8d396bc8 from the Kernel ID drop-down menu (this is the official Ubuntu 11.10 amd64 Server AKI), leave the rest as is and click Continue
• Fill in Key and Value on the next step to add tags for easier administration of your EC2 infrastructure.
• Now you'll need to create a Key Pair by filling in the required fields, then click Create & Download your Key Pair as it's required for SSH access your EC2 instance, and click Continue
• Now we'll want to create a new security group in the next step. Give this a good Group Name and Group Description. Under Inbound Rules and Create a new rule: select SSH, HTTP and any other protocol you may need followed by clicking Continue
• Review that everything looks correct to your needs and click Launch

Connect To your EC2 Instance for the First Time

From the AWS Management Console under the EC2 tab Select the Region in which your instance lives and click Instances Under the Navigation. Right click on your instance and click Connect. This will give you more instructions on connecting to your instances for the first time. All you'll need to do is the following:

chmod 400 /path/to/your/private_key.pem
ssh -i /path/to/your/private_key.pem ubuntu@your_public_dns

Setting Up the Servers for Production

Once your instances have launched and are ready, the instructions posted here will be the same for both servers. We'll take it step by step to get your instances set up. For both servers, observe the instructions and follow the steps:

Setup your user account you'll be deploying with

sudo su
adduser deployer
visudo

After running visudo just add deployer ALL=(ALL) ALL somewhere in that file and save it.

Setup Password Authentication for SSH

After doing the following you'll be able to login to your server with deployer

sudo su
sed -i 's,PasswordAuthentication no,PasswordAuthentication yes,' /etc/ssh/sshd_config
sudo /etc/init.d/ssh restart

Setting up Ruby + Rails

Since installing Ruby and Ruby on Rails is not really the topic of the blog post, I am going to leave it up to you to decide which versions of Ruby and Ruby on Rails you need on your production server. I will note that you should compile your version of Ruby from source and then you can gem install rails, also make sure apt dependencies are met

Installing apt Dependencies.

If you're choosing to run [Ubuntu] for your EC2 Instance, the following will meet most apt dependencies for Ruby and Ruby on Rails applications.

sudo apt-get update && sudo apt-get install build-essential openssl libreadline6 libreadline6-dev libcurl4-openssl-dev curl git-core zlib1g zlib1g-dev libssl-dev libyaml-dev libsqlite3-0 libsqlite3-dev sqlite3 libxml2-dev libxslt-dev autoconf libc6-dev ncurses-dev automake libtool bison subversion apache2-mpm-prefork apache2-prefork-dev libapr1-dev libaprutil1-dev mysql-server

Setting Up RDS

You'll need to make sure your DB engine is supported by RDS. In this example we'll be using MySQL.

• Under AWS Management Console select the RDS tab and then click Launch DB Instance
• Press Select next to mysql
• Select a DB Engine Version (research may be required depending on your apps specifications, 5.1.61 will work for most instances)
• Choose your DB Instance Class (research may be required depending on how big your application's database is going to be)
• Select Yes for Multi-AZ Deployment
• Select an option for Auto Minor Version Upgrade (research may be required, select No if your application will be heavily dependent on the selected version)
• Fill in an Allocated Storage size (Minimum: 5 GB, Maximum 1024 GB)
• Fill in your DB Instance Identifier
• Fill in a Master User Name
• Fill in a Master User Password and select Continue
• Fill in a Database Name (e.g. mydb)
• Choose a Database Port you'd like to use
• Choose Not in VPS for Choose a VPC and click Continue
• Choose your preferred Backup Retention Period, Backup Window and Maintenance Window and click Continue
• Now finally click Launch DB Instance

Under DB Security Groups check your default security group then select CIDR/IP and add your the IP Address of your two EC2 Instances to give them access to your database.

Setting up your application on the RDS

This can be accomplished in your database.yml file by setting the host to the Endpoint of your DB Instance, found under DB Instances under the RDS tab of the AWS Management Console.

Observe:

production:
  adapter: mysql
  host: mydb.cuw5icb3qk2p.us-east-1.rds.amazonaws.com
  reconnect: false
  database: mydb_production
  username: mydb_user
  password: <YOUR SECRET PASSWORD>

Mounting S3 as a remote Drive:

Under the AWS Management Console select the S3 tab, and click Create Bucket. You'll need to give it Bucket Name (with no capital letters) and select a Region and click create.

Make sure you have your Access Key ID and Secret Access Key available from Security Credentials under your user menu at the top right of the AWS Management Console.

Now use the following to mount your remote S3 bucket as a remote drive:

sudo apt-get install fuse-utils s3cmd
wget http://s3fs.googlecode.com/files/s3fs-1.61.tar.gz
tar xvzf s3fs-1.61.tar.gz
cd s3fs-1.61/
./configure --prefix=/usr
make
sudo make install
s3cmd --configure
sudo usermod -aG fuse deployer
sudo su
touch /etc/passwd-s3fs && chmod 640 /etc/passwd-s3fs && echo 'AccessKey:SecretKey' > /etc/passwd-s3fs
echo 's3fs#my_bucket /mnt/s3 fuse allow_other,uid=deployer,umask=777,url=https://s3.amazonaws.com 0 0' >> /etc/fstab
mount /mnt/s3

Now you can just setup your application's deploy scripts to deploy to both servers, and symlink all your public files from /mnt/s3 to /path/to/your/apps/current/public with

ln -s /mnt/s3/public /path/to/your/apps/current/public

and all files should be uploaded and downloaded from your S3 instance.

Conclusion

This system is not perfect. There are still SPoFs. For instance, if your DNS provider that runs the DNS Failover ever goes down (worst case) or Amazon's S3 region goes down (still pretty bad) you'll experience problems. In the worst case, if the DNS Failover service ever goes down it will not fail over to the slave server if master goes down. In the case that Amazon's S3 service goes down you'll experience not being able to upload or download files to the Rails public directory. In contrast, this is a much more fault tolerant setup than having your clients' applications and databases hosted on one server and hoping it never goes down.

If you have a running process, one that you want to keep running, on one of your servers that needs to be supervised, then Daemontools is, in my opinion, the perfect choice. It can keep a process running, keep logs on those processes, and just simply give you one less thing to monitor on your server. Most will prefer to use upstart, which is fine, but I think it sucks compared to Daemontools.

What is Daemontools?

daemontools is a collection of tools for managing UNIX services. supervise monitors a service. It starts the service and restarts the service if it dies. Setting up a new service is easy: all supervise needs is a directory with a run script that runs the service.

multilog saves error messages to one or more logs. It optionally timestamps each line and, for each log, includes or excludes lines matching specified patterns. It automatically rotates logs to limit the amount of disk space used. If the disk fills up, it pauses and tries again, without losing any data.

Why should you use Daemontools?

Observe this link.

Why is /service better than inittab, ttys, init.d and rc.local? Why should my package rely on svscan and supervise?

Answer: Several reasons:

		inittab	ttys	init.d (upstart)	rc.local	/service
	Easy service installation and removal	No	No	Yes	No	Yes
	Easy first-time service startup	No	No	No	No	Yes
	Reliable restarts	Yes	Yes	No	No	Yes
	Easy, reliable signalling	No	No	No	No	Yes
	Clean process state	Yes	Yes	No	No	Yes
	Portability	No	No	No	No	Yes

Easy service installation and removal.

With /service and init.d, creating your new service means linking some files into a centralized directory, and removing the service means removing those files. This is easy for you to automate.

In contrast, with inittab, ttys, and rc.local, creating a new service means adding some lines to a centralized file, and removing the service means locating and removing those lines. This is much more difficult to automate; there are no portable tools for editing the file. Most packages leave it to the user to do the editing.

Easy first-time service startup.

With /service, once you've created your service, it automatically starts within five seconds.

In contrast, with inittab, ttys, init.d, and rc.local, starting the service means an extra command.

Reliable restarts.

With /service, inittab, and ttys, if your daemon dies, it is automatically restarted.

In contrast, with init.d and rc.local, daemons are not monitored. Your daemon will fail to start if, for example, a previously started daemon has temporarily chewed up almost all available memory. The system administrator has to manually restart your daemon after he notices the problem.

Easy, reliable signalling.

With /service, the system administrator can use svc to control your daemon. For example:

• svc -h /service/yourdaemon: sends HUP
• svc -t /service/yourdaemon: sends TERM, and automatically restarts the daemon after it dies
• svc -d /service/yourdaemon: sends TERM, and leaves the service down
• svc -u /service/yourdaemon: brings the service back up
• svc -o /service/yourdaemon: runs the service once

In contrast, with inittab,ttys, init.d , and rc.local, you have to go to extra work to locate the daemon process ID if you want to send it signals. This is not easy to do reliably.

Clean process state.

With /service, inittab, and ttys, when the system administrator restarts a service, the service receives the same fresh process state that it received at boot time.

In contrast, with init.d and rc.local, you have to go to tons of extra work to clean up environment variables, resource limits, controlling ttys, etc. Programmers screw this up all the time, even when they don't care about portability to systems with different process states; system administrators then encounter mysterious failures when they restart daemons.

Portability.

With /service, your program works the same way on every system: Linux, BSD, Solaris, etc.

In contrast, inittab, ttys, init.d and rc.local vary from system to system. Some systems don't have init.d, for example, and the ones that do have it don't agree on its location. This is extremely annoying for cross-platform system administrators.

Installing Daemontools

NOTE: Everything needs to be ran as root

mkdir -p /package
chmod 1755 /package
cd /package

wget http://cr.yp.to/daemontools/daemontools-0.76.tar.gz
tar -xpf daemontools-0.76.tar.gz
rm -f daemontools-0.76.tar.gz
cd admin/daemontools-0.76
package/install

IF INSTALL FAILS

ed ./src/conf-cc
1s/$/ -include errno.h/
wq

Starting Daemontools

apt-get install csh "(for Ubuntu/Debian users)"
csh -cf '/command/svscanboot &'

Starting Daemontools on Boot

sed -i "1 a\csh -cf '/command/svscanboot &'" /etc/rc.local
chmod +x /etc/rc.local

Making Services

I personally like putting my services in the /services directory as daemontools will look for a new service in the /service directory. It makes it easier for tab completion. :-)

mkdir /services

mkdir /services/somerandomservice
echo -e '#!/bin/sh\nexec somerandomcommand' > /services/somerandomservice/run
chmod 755 /services/somerandomservice/run
ln -s /services/somerandomservice/ /service/

Linking your service's daemon directory to your /service directory will tell daemontools to bring your service up. You can check the uptime of your service with svstat /service/somerandomservice

(note: if your service does not keep a longer uptime than 1 second, then something is wrong and you need to check your run script, see debugging for more info)

Run Service With a Specific User

To do this is no different from making a service. You'll just use the setuidgid tool that ships with daemontools.

Observe:

mkdir /services/somerandomuserservice
ed /services/somerandomuserservice/run
a
#!/bin/sh
exec setuidgid somerandomuser somerandomcommand
.
w
!chmod 755 %
q
ln -s /services/somerandomuserservice/ /service/

Now with some Ruby applications, you may experience problems with this method because setuidgid is much more simple from a full login. It only sets the uid and gid as the name implies. If you need the full login functionality then try this:

mkdir /services/somerandomuserservice
ed /services/somerandomuserservice/run
a
#!/bin/sh
exec su - somerandomuser -c 'exec somerandomcommand'
.
w
!chmod 755 %
q
ln -s /services/somerandomuserservice/ /service/

Logging Services

adduser logger

mkdir /services/somerandomservice/log
mkdir /services/somerandomservice/log/main
chown logger /services/somerandomservice/log/main

ed /services/somerandomservice/log/run
a
#!/bin/sh
exec setuidgid logger multilog t ./main
.
w
!chmod 755 %
q

Debugging

I can only help you with debugging daemontools here. Drop in #isotope11 on freenode if you have further problems, and someone will surly help.

Checking the run script

cd /service/somerandomservice
svc -d .
./run

Take note of any errors that are preventing your run script from running. Do they ring any bells? Have you Googled the error?

After you've fixed your bug, bring your service back up with:

svc -u /service/somerandomservice

Checking the logs

cd /service/somerandomservice
tail -n5 log/main/current

If your processes is throwing any errors in the system, they will be logged here. If current isn't there, then it has not had an error yet and you can celebrate.

Restart a service

svc -t /service/somerandomservice

Removing Services

rm /service/somerandomservice
svc -dx /services/somerandomservice

To convert you entire project to Ruby 1.9.x style hashes you can simply run:

find -type f -exec sed -i 's,:\([a-zA-Z_]*\) =>,\1:,g' '{}' \+

This will capture all the string keys (including ones with underscores) and convert them. I would make sure you have a back up of your original code and run all of your tests after you do this conversion to verify you haven't broken anything. As far as I know this works. I tested it on an example project and checked the git diff to make sure it worked correctly. All looked wonderful. Once again, sed saves the day!

Update

Ben Holley pointed out to me that my above find command with sed would only handle cases such as:

:foo => "bar"
:foo_bar => "foobar"

And it would not handle cases such as:

:"foo" => "bar"
:"foo_bar" => "foobar"
:"foo bar" => "foobar"

I have tweaked the regex and the sed command to handle these cases and the original cases. Now you can catch the cases with quotes and spaces like this:

find -type f -exec sed -i -r 's,:("?[a-zA-Z_[:space:]]+"?) =>,\1:,g' '{}' \+

It's going to happen to you one day. You'll have to work on a machine for a client or a friend who has "misplaced" the root password on which you don't have an account.

If you have console access and don't mind rebooting, traditionally you'd boot up in single user mode. After hitting Control-Alt-Delete, you simply wait for the POST and pass the single parameter to the booting kernel. I.E. from the LILO prompt it was like this:

LILO: linux single

On many systems, this will happily present you with a root shell. Other systems (RedHat), you'll run into the dreaded emergency prompt

Give root password for maintenance
(or type Control-D for normal startup)

If you knew the root password, you wouldn't be here! Sometimes (if you're lucky), the init script will actually let you hit ^C at this point and put you in a root prompt. But most init processes are "smarter" than that, and trap ^C. What to do? You could always boot from a rescue disk and reset the password, but suppose you don't have one handy (or that the machine doesn't have a CD-ROM drive).

All is not lost! Rather than risk running into the above mess, let's modify the system with extreme prejudice, right from the start. Again, from the LILO prompt:

LILO: linux init=/bin/bash

What does this do? Instead of starting you in /sbin/init and proceed with usual /etc/rc.d/* procedure, we're telling the kernel to simply give us a shell. No passwords, no filesystem checks(and for that matter, no much of a starting environment!) but a shiny new root prompt.

This still isn't enough to be able to repair your system. The root filesystem will be mounted read-only (since it never got a chance to be checked and remounted read/write). Also, networking will be down, and none of the usual system daemons will be running. You don't want to do anything more complicated than resetting a password (or tweaking a file or two) at a prompt like this. And certainly don't hit ^D and Exit! Your little shell (plus the kernel) constitutes the entire running Linux system at the moment. So, how can you manipulate the filesystem in this situation, if it is mounted read-only? Try this:

mount -o remount,rw /

That will force the root filesystem to be remounted with read-write. You can now type passwd to change the root password. Once the password is reset, DO NOT REBOOT. Since there is no init running, there is no process in place for safely taking the system down. The quickest way to shutdown safely is to remount root again:

mount -o remount,ro /

With the root partition readonly, you can confidently hit the Reset button, bring it up in single-user mode, and begin your actual work.

If you have ever used vim macros and appreciate that feature then you are going to love iMacros for Firefox by iOpus. It allows you to record repetitive tasks save them, edit them, rename them, and delete them. I found this very useful when working AIUA, one of our clients here at Isotope11. There is a tedious process that requires filling out a long form filled with check boxes, text fields, select drop down menus, and submit buttons. With iMacros, I can record this awful process of filling out 7-8 pages of forms and never have to deal with doing it by hand again.

Examples

Here is a simple example of a macro that will go to Google, search for "iMacros for Firefox", find the correct link, and click it.

URL GOTO=google.com
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:f ATTR=ID:lst-ib CONTENT=I
TAG POS=1 TYPE=INPUT:TEXT FORM=NAME:gs ATTR=ID:lst-ib
CONTENT=Imacrosforfirefox
TAG POS=1 TYPE=A
ATTR=TXT:iMacrosforFirefox::Add-onsforFirefox

But that's simple stuff. Let's see the very painful process made easy by iMacros I described above

URL GOTO=
http://localhost:3000/pages/rate_calculator?territory=Zone_4&lat=30.406387&long=-87.679393
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create ATTR=ID:quote_occupancy
CONTENT=%Owner
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create
ATTR=ID:quote_occupancy_status CONTENT=%Not-seasonal
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create ATTR=ID:quote_ec_form
CONTENT=%DP0001
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create
ATTR=ID:quote_coverage_type CONTENT=%Wind(EC)Only
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create
ATTR=ID:quote_number_of_families CONTENT=%1
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create ATTR=ID:quote_construction
CONTENT=%Frame
TAG POS=1 TYPE=INPUT:TEXT FORM=ACTION:/quotes/create
ATTR=ID:dwelling_coverage CONTENT=134000
TAG POS=1 TYPE=INPUT:TEXT FORM=ACTION:/quotes/create
ATTR=ID:contents_coverage CONTENT=62000
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create
ATTR=ID:quote_needs_theft_coverage CONTENT=%0
TAG POS=1 TYPE=SELECT FORM=ACTION:/quotes/create
ATTR=ID:quote_year_of_construction CONTENT=%2011
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ID:res_form ATTR=ID:calculate
TAG POS=1 TYPE=A
ATTR=TXT:FillApplicationwiththeserates(NoPrescreening)
TAG POS=1 TYPE=SELECT FORM=ACTION:
http://localhost:3000/residential_insurance_applications/new?aiua_quote_id=394596&city=FOLEY&construction=Frame&contents_coverage=62000&coverage_type=Wind%28EC%29+Only&ded_code=05&deductible=500&discount=1.0&dwelling_coverage=134000&ec_premium=1106&effective_date=2011-12-05&fire_premium=0&fire_protection_class=10&flood_zone=X&hurricane_deductible=2&is_new_policy=true&lat=30.406387&long=-87.679393&number_of_families=1&occupancy=Owner&policy_form=DP0001&quote_price=1387&redirect_after=%2Fresidential_insurance_applications%2Fnew&servicing_fire_department=BON+SECOUR&territory=Zone+4&vmm_premium=0&was_manual_lookup=false&year_built=2011ATTR=ID:agent_id
CONTENT=%1932
TAG POS=1 TYPE=INPUT:SUBMIT FORM=ACTION:
http://localhost:3000/residential_insurance_applications/new?aiua_quote_id=394596&city=FOLEY&construction=Frame&contents_coverage=62000&coverage_type=Wind%28EC%29+Only&ded_code=05&deductible=500&discount=1.0&dwelling_coverage=134000&ec_premium=1106&effective_date=2011-12-05&fire_premium=0&fire_protection_class=10&flood_zone=X&hurricane_deductible=2&is_new_policy=true&lat=30.406387&long=-87.679393&number_of_families=1&occupancy=Owner&policy_form=DP0001&quote_price=1387&redirect_after=%2Fresidential_insurance_applications%2Fnew&servicing_fire_department=BON+SECOUR&territory=Zone+4&vmm_premium=0&was_manual_lookup=false&year_built=2011ATTR=NAME:commit&&VALUE:Continue
TAG POS=1 TYPE=A ATTR=TXT:2
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_first_name CONTENT=Foo
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_middle_initial CONTENT=
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_last_name CONTENT=Bar
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_home_phone
CONTENT=5555555555
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_address
CONTENT=123FoobarSt
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_city CONTENT=Fooville
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_state CONTENT=Al
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_applicant_zip CONTENT=36535
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_pre_number CONTENT=123
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_pre_direction CONTENT=%N
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_street CONTENT=foobar
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_str_suffix CONTENT=%ST
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_zip CONTENT=%36535
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_foundation_type
CONTENT=%Concreteslabongrade
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_structure_over_water
CONTENT=%false
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_has_attached_structures
CONTENT=%false
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_square_footage
CONTENT=1200
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_building_type
CONTENT=%N/A-NotMulti-Unit
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_property_number_of_stories
CONTENT=2
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_coverage_info_dwelling_current_acv_or_replacement
CONTENT=124000
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_coverage_info_dwelling_replacement_cost
CONTENT=124000
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:first_mortgagee_autocompleter CONTENT=re
TAG POS=1 TYPE=DIV ATTR=TXT:POBOX19
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:second_mortgagee_autocompleter CONTENT=re
TAG POS=2 TYPE=DIV ATTR=TXT:POBOX19
TAG POS=1 TYPE=SELECT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_mortgagee_info_1_escrowed
CONTENT=%false
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_mortgagee_info_1_loan_number
CONTENT=1
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_mortgagee_info_2_loan_number
CONTENT=2
TAG POS=1 TYPE=TD ATTR=TXT:Insurer
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_present_or_prior_coverage_insurer
CONTENT=FooBarInc
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_present_or_prior_coverage_building_coverage
CONTENT=$100,000
TAG POS=4 TYPE=IMG ATTR=SRC:
http://localhost:3000/images/calendar_date_select/calendar.gif?1323106417
TAG POS=1 TYPE=DIV ATTR=TXT:31
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_present_or_prior_coverage_contents_coverage
CONTENT=$10,000
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_fire_insurance_insurer
CONTENT=FoobarInc
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_fire_insurance_building_coverage
CONTENT=10,000
TAG POS=6 TYPE=IMG ATTR=SRC:
http://localhost:3000/images/calendar_date_select/calendar.gif?1323106417
TAG POS=1 TYPE=DIV ATTR=TXT:31
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_fire_insurance_contents_coverage
CONTENT=10,000
TAG POS=1 TYPE=INPUT:TEXT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=ID:residential_insurance_application_fire_insurance_policy_number
CONTENT=12345
TAG POS=1 TYPE=OBJECT
ATTR=ID:residential_insurance_application_fire_insurance_applicationUploader
TAG POS=2 TYPE=A ATTR=TXT:Upload
TAG POS=1 TYPE=A ATTR=ID:residential_application_tabs_next
TAG POS=1 TYPE=OBJECT ATTR=ID:file_uploadUploader
TAG POS=3 TYPE=A ATTR=TXT:Upload
TAG POS=1 TYPE=OBJECT ATTR=ID:file_upload_2Uploader
TAG POS=4 TYPE=A ATTR=TXT:Upload
TAG POS=1 TYPE=OBJECT
ATTR=ID:residential_insurance_application_signed_notice_oneUploader
TAG POS=8 TYPE=A ATTR=TXT:Upload
TAG POS=1 TYPE=INPUT:SUBMIT
FORM=ACTION:/residential_insurance_applications/27637/complete
ATTR=NAME:commit&&VALUE:SaveforLater

One thing that's worth pointing out is that this huge example actually fails. It fails because the ID of the application changes each time you run this macro. In this case you can use a regex to find a digit for the ID. iMacros explains this in their wiki.

Conclusion

iMacros allows you to automate your browser very easily. It even allows you to handle dynamic content in your macro. They have great documentation on their wiki. I fully support and approve of these guys and encourage anyone, especially web developers who have to repeat a task in their browser to give this add-on a try. It will make your life much easier.