*Coder Blog

Life, Technology, and Meteorology

Month: May 2007

Server Colocation

I’m happy to say that after spending the past 5 months in the pipeline, Rio (the Gaucho Software Forecast Server) has now been moved to a colocation facility. The facility provides redundant power, cooling, and several connections to Tier 1 network providers, which should definitely increase the server’s overall reliability. Previously, this server was located at my office. I had it hooked up to a UPS battery backup that gave it about 30 minutes of runtime, but it’s a far cry from true redundant power. Also, over the past several months, it seems that my business network connection reliability has been slowly decreasing. This should fix that issue.

Rio in the rack...

Before moving the server, I thought it would be a good idea to add a bit more memory and hard drive space to the box. I bumped up the memory to the motherboard’s max of 4Gb, which gives some more breathing room for the virtual machines I’m running via VMware. I also added another 300GB hard drive and switched from a 3 disk RAID 5 configuration to a 4 disk RAID 10. I had been reading on PostgreSQL mailing lists that for configurations with less than 6-8 hard drives, RAID 10 was substantially faster than RAID 5. RAID 5 has always been infamously slow at writes, but the read speeds are pretty good in general, so I had my doubts. Well, my doubts were definitely unfounded, because this single hard drive upgrade has given a dramatic performance increase. Previously, when running the forecast generator (twice a day), the processing would take approximately 2 hours. Now, after adding the 4th disk and switching to RAID 10 using an XFS filesystem (more on this below), the same process takes only 1 hour and 10 minutes.

Rio inside...

Since I was starting with a fresh RAID partition, I thought I should put some time into looking at different Linux filesystems. I used Bonnie++ to perform the disk benchmarks using a temp file size of 16GB. Initially with a standard Ext3 filesystem:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rio.gauchoso 16000M 50719  83 103617  41 43095  10 51984  76 117808  13 285.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2730  98 +++++ +++ +++++ +++  3024  99 +++++ +++  9129  99

Not bad…just over 100MB/sec writes and 117MB/sec reads. Notice the random seeks/sec value of 285.4/sec and file creates of 2730/sec. On a database system, the disk heads are given quite a workout, reading from indexes and data tables all over the disk. So seeks/sec performance was important to me. Memory cache helps, but my database is around 30GB on disk, so caching only goes so far. Overall, the ext3 numbers sounded pretty good, but I didn’t have anything to really compare them to.

I decided to try out SGI’s XFS filesystem. After seeing several benchmarks online between filesystems like ReiserFS, JFS, Ext3, and XFS, it seemed that XFS almost always had the best performance in general, so I gave it a go. XFS has a lot of nice features, including tools to defragment a partition while the system is active, the use of B+ trees for volume indexes (resulting in a much greater efficiency when you have a ton of files in a single directory), and a pretty decent tool to repair broken filesystems as well. I reformatted the partition and ran Bonnie++ again with these results:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rio.gauchoso 16000M 55354  79 126942  21 32567   6 47537  70 126927  14 415.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3917   2 +++++ +++  2662   3  3102   5 +++++ +++  2612   3

Write speed increased a healthy 22% to almost 127MB/sec, and read speed increased almost 8% to a similar 127MB/sec rate. But look closer at the seeks/sec and creates/sec rates… Seeks/sec increased an incredible 45% to 415.8/sec, and file creates improved 43% to 3917/sec. The drawback? Deleting files is quite a bit slower, 71% slower to be exact. To me, this tradeoff was well worth the gains, as it’s fairly rare for me to be deleting lots of files on the server. I have noticed a slight performance degradation while using rm with a lot of files, but it’s still a RAID so performance is certainly acceptable.

It’s not good to use just a single benchmarking tool, so I checked the validity of a couple of Bonnie++’s claims with a few simple dd commands, this time using 8GB file sizes:

root@rio:/rio# dd if=/dev/zero of=/rio/bench bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 63.7116 seconds, 129 MB/s
root@rio:/rio# dd if=/rio/bench of=/dev/null bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 59.2728 seconds, 138 MB/s

Those match up pretty well on the writes, and reading from the disk sequentially with dd is even faster than Bonnie++ claims.

Overall, I’m pretty pleased with the upgrade. I’m even happier to have this server at a datacenter. This should give me a lot of room to grow with hosting more weather data for upcoming Seasonality features, and also gives me a good server to run some other services on as well.

8 Random Facts

Logtar tagged me with a meme to post 8 Random Facts about yours truly. These are things that most people don’t know about me, so I thought it would be pretty fun. So, off we go…

  • I have visited over half of the 50 states in the U.S., but I’ve never left North America. I’m hoping to change this soon.
  • I had an opportunity to go on a 35 mile backpacking trek in the Sierras back during my early teen years when I was in scouts. Hiking past the tree line up to 14k feet where there is snow, even in the middle of July, was really damn cool.
  • I’ve broken my nose while bodyboarding (face first, right into the sand in shallow water).
  • I’ve broken my head while mountain biking… Took off a jump and bit it hard. Walked a few miles back to my apartment with my shirt held on the top of my head to stop the bleeding. The doctors used the medical equivalent of a staple gun to sew me back up. That event taught me to always wear a helmet while riding.
  • I met my wife, Katrina, at the same location that Katrina’s parents met several years earlier.
  • I have not seen any movie of the Star Wars trilogy in it’s entirety.
  • Cereal makes a good breakfast, lunch, or dinner.
  • I can’t stand going to see doctors or dentists… I do have regular dentist cleanings, but I haven’t been to see a doctor in the past 4+ years.

© 2017 *Coder Blog

Theme by Anders NorenUp ↑