*Coder Blog

A few months ago an idea came to me for a new domain name I should pick up. I wasn’t sure what I was going to do with the domain yet, or if I would even use it at all, but I wanted to jump on it because it was available. It struck me as a good name, and I was a bit surprised that it was still available.

Zoom forward to a couple of weeks ago, when a different idea for a new website came to mind. Oftentimes I post detailed entries here specifically about the weather or very weather-tech related articles when I run into issues while developing Seasonality. The thing is, I’m not sure my typical reader is interested in these postings, and occasionally I will refrain from posting about weather-related issues simply because I don’t think it would fit well in the *Coder blog.

But wait a second…I bought that domain awhile back, maybe that would work. Actually, it ended up being perfect for my new site idea. So I started working on it off and on. I’ve been happily using Wordpress to host this blog for quite some time, so I decided to use the same platform for the new site. I set things up, customized a theme, and wrote a posting or two. I think now it is finally ready to be revealed, and I wanted to share. The domain is weatherdeveloper.com, and the site is called “A Weather Developer’s Journey.” With the full-time development of Seasonality 2 coming soon here, I thought it would be a perfect time to start a site like this, as I’ll be spending a lot of time trying to overcome issues with data hosting, manipulation, and weather visualization.

If you’re at all interested in checking it out, please do so (Website, RSS Feed). I’ll most likely post links from here to the first few articles on the new site, just to get the ball rolling. The first article talks about finding and putting to use 90 meter elevation data.

I’m happy to say that after spending the past 5 months in the pipeline, Rio (the Gaucho Software Forecast Server) has now been moved to a colocation facility. The facility provides redundant power, cooling, and several connections to Tier 1 network providers, which should definitely increase the server’s overall reliability. Previously, this server was located at my office. I had it hooked up to a UPS battery backup that gave it about 30 minutes of runtime, but it’s a far cry from true redundant power. Also, over the past several months, it seems that my business network connection reliability has been slowly decreasing. This should fix that issue.

Before moving the server, I thought it would be a good idea to add a bit more memory and hard drive space to the box. I bumped up the memory to the motherboard’s max of 4Gb, which gives some more breathing room for the virtual machines I’m running via VMware. I also added another 300GB hard drive and switched from a 3 disk RAID 5 configuration to a 4 disk RAID 10. I had been reading on PostgreSQL mailing lists that for configurations with less than 6-8 hard drives, RAID 10 was substantially faster than RAID 5. RAID 5 has always been infamously slow at writes, but the read speeds are pretty good in general, so I had my doubts. Well, my doubts were definitely unfounded, because this single hard drive upgrade has given a dramatic performance increase. Previously, when running the forecast generator (twice a day), the processing would take approximately 2 hours. Now, after adding the 4th disk and switching to RAID 10 using an XFS filesystem (more on this below), the same process takes only 1 hour and 10 minutes.

Since I was starting with a fresh RAID partition, I thought I should put some time into looking at different Linux filesystems. I used Bonnie++ to perform the disk benchmarks using a temp file size of 16GB. Initially with a standard Ext3 filesystem:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rio.gauchoso 16000M 50719  83 103617  41 43095  10 51984  76 117808  13 285.4   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  2730  98 +++++ +++ +++++ +++  3024  99 +++++ +++  9129  99

Not bad…just over 100MB/sec writes and 117MB/sec reads. Notice the random seeks/sec value of 285.4/sec and file creates of 2730/sec. On a database system, the disk heads are given quite a workout, reading from indexes and data tables all over the disk. So seeks/sec performance was important to me. Memory cache helps, but my database is around 30GB on disk, so caching only goes so far. Overall, the ext3 numbers sounded pretty good, but I didn’t have anything to really compare them to.

I decided to try out SGI’s XFS filesystem. After seeing several benchmarks online between filesystems like ReiserFS, JFS, Ext3, and XFS, it seemed that XFS almost always had the best performance in general, so I gave it a go. XFS has a lot of nice features, including tools to defragment a partition while the system is active, the use of B+ trees for volume indexes (resulting in a much greater efficiency when you have a ton of files in a single directory), and a pretty decent tool to repair broken filesystems as well. I reformatted the partition and ran Bonnie++ again with these results:

Version  1.03       ------Sequential Output------ --Sequential Input- --Random-
                    -Per Chr- --Block-- -Rewrite- -Per Chr- --Block-- --Seeks--
Machine        Size K/sec %CP K/sec %CP K/sec %CP K/sec %CP K/sec %CP  /sec %CP
rio.gauchoso 16000M 55354  79 126942  21 32567   6 47537  70 126927  14 415.8   1
                    ------Sequential Create------ --------Random Create--------
                    -Create-- --Read--- -Delete-- -Create-- --Read--- -Delete--
              files  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP  /sec %CP
                 16  3917   2 +++++ +++  2662   3  3102   5 +++++ +++  2612   3

Write speed increased a healthy 22% to almost 127MB/sec, and read speed increased almost 8% to a similar 127MB/sec rate. But look closer at the seeks/sec and creates/sec rates… Seeks/sec increased an incredible 45% to 415.8/sec, and file creates improved 43% to 3917/sec. The drawback? Deleting files is quite a bit slower, 71% slower to be exact. To me, this tradeoff was well worth the gains, as it’s fairly rare for me to be deleting lots of files on the server. I have noticed a slight performance degradation while using rm with a lot of files, but it’s still a RAID so performance is certainly acceptable.

It’s not good to use just a single benchmarking tool, so I checked the validity of a couple of Bonnie++’s claims with a few simple dd commands, this time using 8GB file sizes:

root@rio:/rio# dd if=/dev/zero of=/rio/bench bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 63.7116 seconds, 129 MB/s

root@rio:/rio# dd if=/rio/bench of=/dev/null bs=8192 count=1000000
1000000+0 records in
1000000+0 records out
8192000000 bytes (8.2 GB) copied, 59.2728 seconds, 138 MB/s

Those match up pretty well on the writes, and reading from the disk sequentially with dd is even faster than Bonnie++ claims.

Overall, I’m pretty pleased with the upgrade. I’m even happier to have this server at a datacenter. This should give me a lot of room to grow with hosting more weather data for upcoming Seasonality features, and also gives me a good server to run some other services on as well.

A week or two ago, a new documentary called “The Great Global Warming Swindle” was brought to my attention on a friend’s blog. While I haven’t seen the documentary, an article written about the documentary by Thomas Sowell contains enough inaccurate statements to make me question the validity of the documentary it references. I have taken a number of meteorology classes. Much of meteorology focuses on radiation, which also happens to be the root of the entire global warming issue. (Global warming is just the earth and it’s atmosphere absorbing more radiative energy than it emits.) I’ve picked out a few choice quotes from the article, and debunk them below.

There is no question that the globe is warming but it has warmed and cooled before, and is not as warm today as it was some centuries ago, before there were any automobiles and before there was as much burning of fossil fuels as today.

This statement is simply incorrect. While it is true that the globe’s temperature varies over the course of thousands of years as a result of Milankovitch cycles and other causes, there has not been any time in the past 650,000 years when the global average temperature has been higher than it is today. What the author is most likely referring to is the Medieval Warm Period (800-1300 AD). During this time, the average temperature was higher than normal in the European region and perhaps part of northern Asia. However, when you compare the average global temperature of both the Medieval times and current day, the average temperature today is much higher than it was back in Medieval times.

According to these climate scientists, human activities have very little effect on the climate, compared to many other factors, from volcanoes to clouds.

According to the IPCC report released earlier this year (page 4), human activities have a higher effect on global climate than natural sources. Furthermore, natural sources such as volcanoes and clouds actually cool the earth rather than warm it.

Clouds both cool and warm the atmosphere. Low clouds like a thunderstorm or thick layer of uniform stratus clouds cool the earth quite a bit, and reflect radiation from the sun back into space. High clouds, like the thin haze-like cirrus will trap some of the radiation the earth is emitting and warm the atmosphere slightly. The prevalence of low and mid level clouds wins out here with a cooling effect.

As for volcanoes, eruptions do release ash and greenhouse gasses up into the stratosphere and this has been measurable after relatively recent eruptions like Mount St. Helens. Since volcanic eruptions are relatively rare, their contribution to global climate is minimal to the extent that I’m not sure why the article or the documentary even brought up the issue. Usually, the ash dominates by blocking solar radiation for a net cooling effect, and after an eruption locations nearby would notice solar dimming.

While the effects of clouds and volcanoes combined will cool the earth’s climate significantly, CO₂ and methane from anthropogenic sources have an even larger warming effect. Furthermore, the article brings up these two sources as if they are causing global warming instead of humans, when in fact they are acting as a negative feedback to the damage humans are causing.

These climate scientists likewise debunk the mathematical models that have been used to hype global warming hysteria, even though hard evidence stretching back over centuries contradicts these models.

I’m not sure what “hard evidence” the author is referring to, but there are several climate models developed at various universities and scientific institutions worldwide. Most of these models are tested from 1960 on forward, because we have solid temperature measurements worldwide during this time period. All of the best models match the warming that actually occurred during this time frame, and this reinforces the validity of these models. Again, I refer to the IPCC report (page 11) that shows output from up to 14 different climate models matching temperatures over the past 100 years only when they include anthropogenic forcing as opposed to just including natural climate forcing.

What is even scarier than seeing how easily the public, the media, and the politicians have been manipulated and stampeded, is discovering how much effort has been put into silencing scientists who dare to say that the emperor has no clothes.

Hmm…I’ve actually read several news articles reporting the contrary: the government often censors scientific reports written that support the global warming problem.

Academics who jump on the global warming bandwagon are far more likely to get big research grants than those who express doubts — and research is the lifeblood of an academic career at leading universities.

Actually, these grants can be used as leverage by the government to silence scientists. Release your findings, we pull your funding. Most researchers choose a topic they are interested in exploring, rather than focus on topics that have grant money available. Grant money is available through the NSF for a multitude of scientific topics.

Furthermore, this assumption of researchers “going after all the money” for selfish reasons is only valid if the researchers themselves got to keep most of the money. This is far from the case, and oftentimes researchers will only receive money directly from the grant if they buy their salary out from teaching classes at a university. Researchers still don’t get any increase in salary, they just won’t have to teach as many courses.

In politics, even conservative Republicans seem to have taken the view that, if you can’t lick ‘em, join ‘em. So have big corporations, which have joined the stampede.

While there is nothing I can say to completely disprove this statement, I can say that global CO₂ emissions from businesses in the U.S. increase every year… If big corporations were jumping on the global warming bandwagon, wouldn’t emissions decrease?

Furthermore, why is global warming a political issue? It’s known that the earth is getting warmer, and it is also known that if there are lower CO₂ concentrations in the atmosphere, the earth wouldn’t be as warm (this has been scientifically proven, by simply looking at the chemical makeup of CO₂ and how it reacts to different wavelengths of radiation)… Wouldn’t the logical next step be to try and reduce CO₂ concentrations in the atmosphere to return temperatures to around normal? I don’t see this as a liberal vs. conservative issue.

The difference is that we have the hardest and most painful evidence that there was a Holocaust. But, for the global warming scenario that is causing such hysteria, we have only a movie made by a politician and mathematical models whose results change drastically when you change a few of the arbitrarily selected variables.

Actually, I think we have much more information about global warming than just models and a movie. Google Scholar shows about 152,000 articles on global warming…

As for the “arbitrarily” selected variables, I would hardly call them arbitrary. I haven’t researched any one climate model so deeply that I would know how it works, but if the variables are anything like the hundreds of specified conditions used in meteorological forecast models, I would imagine they are comprehensive. Modeling is difficult, and at this point we just aren’t sure how some variables will change as the temperature increases. However, this only brings the models’ accuracy into question 100 years from now, and does not discredit the fact that the earth is warming because of anthropogenic forcing right now. From page 11 of the IPCC report, you can see that the model output matched the actual temperature much more closely when anthropogenic variables were taken into account. This alone is pretty significant evidence that the human race is effecting the earth’s climate beyond the natural environmental variations.

If you want to run a climate model on your own computer and learn more about what is involved, check out EdGCM.

If you would like to read more about the issues brought up in this documentary, here is a very lengthy discussion over on RealClimate.org (529 comments on the posting at this time). Large universal topics like this deserve to be debated. Fortunately, scientists have already discussed the issue extensively over the past couple of decades. Governments, corporations, and to some extent the media don’t like what has been discovered and are deciding to take issue with it. Unfortunately, this just delays actually doing anything to help the environment and return things to the way they were.

The National Climatic Data Center (NCDC) released their official report on the worldwide climate for February, and for this past winter in general which includes December - February. 2006-2007 has been found to be the record warmest winter in the past 100 years. The report has a lot of information relating to various aspects of climate such as precipitation and snow pack data, but I found one image to be particularly striking: the temperature anomalies for December - February…

So what does this image show exactly? Well, a whole lot of red… Seriously though, to generate this graph the NCDC took worldwide weather observations for December - February for a 30 year period from 1961 to 1990. Then, they took the combined worldwide weather observations for December - February of this year and compared the two. Temperature deviations are shown by different sized/colored dots for a given area.

What’s surprising is not only the large areas over land in the Northern Hemisphere that were 5°C (9°F) or warmer than normal, but also the expanse of warm temperatures worldwide. It’s fairly common for one section of the globe to be warmer or cooler than normal for a few months at a time, but this image is showing warmer than normal temperatures almost universally around the globe.

Some of this can be credited to the recent El Nino event we had this winter. El Nino does play a role, but it most certainly is not responsible for all the warming taking place. CO₂ emissions are at an all-time high, and increasing at a faster rate than ever before. If we don’t do something to keep CO₂ concentrations in the atmosphere in check, the earth will be a much different place to live 100 years from now.

I just posted a new weather article over on the Gaucho Software Forums this morning. The article talks about what dewpoint is and how we can measure it. This is one I’ve been meaning to write for quite awhile now and I finally got a chance to spend some time on it.

If you have any comments on the article, feel free to post them on that forum topic. I have a few ideas for future articles but I want to make sure I don’t run out of topics. If you have any suggestions of topics I should cover, get in touch with me.

James Duncan Davidson writes:

Here in the Pacific Northwest, we’re prepping for a big storm that’s set to rip through tonight. Hurricane force winds are forecast on the coast, blizzard conditions are forecast for the Cascades, and it looks like we’re going to get rattled about here in Portland. Already, the winds are gusting about and hitting 40mph.

I just caught wind (no pun intended) of this storm earlier this afternoon. There are storm warnings covering the entire coastline of Washington and Oregon, and even part of Northern California, this is definitely going to be a big one. I even saw a High Surf Warning with estimated swell heights of 38-42 feet. If you’re in that area, hunker down and stay safe.

I released Seasonality 1.4 out into the wild this morning. So far the release is going pretty nicely, without any major bugs found as of yet. I’ve been using the beta for the past few months without finding any major issues, so I kind of expected this to be a pretty stable release. The forecast server here has dished out over 700,000 forecasts up through yesterday (that’s without a final Seasonality release hitting it). I’m interested to see how long it will take to hit a million with this official release of version 1.4. Thanks go out to everyone who helped beta test this release.

Also, with the risk of writing about out-of-date news, I’ve posted a couple of weather articles over on the Gaucho Software Forums. First is an explanation for the fairly quiet Atlantic hurricane season this past year. I talk about some of the ingredients needed to form a hurricane, and what has been missing this year. The second article, posted last week, explains why why relative humidity is a bad measure of moisture in the atmosphere. If you’re interested in either of those topics, be sure to check out these articles. As a side note, I am patiently waiting for the next release of YaBB Forum software, which has RSS support so users will be able to subscribe to these weather articles I post. They have a feature-lock on the release, but no word yet on an expected release date.

If you live in the Midwest, either you are getting hit with this storm right now or you will be getting hit with it soon. This storm has the potential to be the first big snow for the year in many areas. The forecast models have been all over the board with predicting this event. At least, that was the case up until earlier today when the 0Z and 12Z runs of the GFS and NAM, along with some other models are now much more agreeable with this storm’s track. When the models line up, that’s a pretty good sign of an accurate path.

The image below shows the precipitation forecast for Friday between 12 and 18Z (7am - 1pm EST). Right now, this system is starting to form in Texas. The storm will be traveling NNE over the next 36 hours to around Ohio in the time period of the image below. The precipitation “arrow” is pretty much pointing in the direction this thing is moving. The lightest blue is between 0.75 and 1 inch of liquid equivalent precipitation… However, if you follow the deep blue line marked 0 degrees C, all of this area will be getting either a rain/snow mix or all snow. 1 inch of liquid precipitation can equate to anywhere between 6-10 inches of snow depending on the temperature. Since this image just shows the precipitation for a 6 hour time period (and not the entire storm’s precipitation), there is a real potential of a bunch of snow being dumped along it’s path. Some models have forecasted up to 17 inches here in central Michigan, but I think that’s really overshooting it. I’m guessing it will drop around 6-8 inches of heavy snow for our area, and definitely more in some other areas.

Anyway, here’s hoping we’ll get enough snow to go sledding this weekend.

I just wrote a post on the Gaucho Software Forums about a snow storm that hit Buffalo a little over 10 days ago. It brought almost 2 feet of snow to the area and caused a lot of damage. I happened to be in the area at the time, so I was tracking the storm pretty closely, and I even took a few pictures of the aftermath. If you’re interested, check out the story here.

The story posting is part of a new weather discussion area I added to the forums. If you have any weather stories to share, feel free to start up a new topic and tell everyone about it. Also, bring along any questions you may have about anything relating to weather.

James Lileks of the Star-Tribune in the twin cities writes:

I have a new program that displays weather information from a variety of cities – such things are plentiful, I know, but this one suits my needs. It’s set up to tell me how things are in Mpls, Fargo, NYC, Scottsdale, and Glockamorra. Right now it’s 96 degrees in Scottsdale. At eight PM.

Sometimes I miss Arizona weather… The warm nights were wonderful. Want to go for a walk or bike ride in the evening? No problem, just walk outside. Don’t bother with a jacket (or even long sleeves). Of course monsoon season is incredible as well.