*Coder Blog

Life, Technology, and Meteorology

Month: September 2005

Oh sweet data…

One thing I never expected when starting work on Seasonality is the shear amount of weather data that is available on the web. Sometimes it’s hard to get a handle on all the information. I have a weather database served by PostgreSQL, and that’s starting to take several GB on my server. Well, last night I was browsing for a METAR station name listing (the station names I currently have are truncated to around 15 characters), and I came across an NOAA FTP server that allows users to download raw METAR observations for all the sites around the globe. This is the same data I use to draw the graph in Seasonality, so I had to check it out…

Sure enough, even though the file format is kind of messy, I was able to write a download/parsing script to access the data and insert it into my database. The dataset is just huge…after only 12 hours I’ve collected somewhere in the order of 100,000 records. It’s almost too much data for my lowly database server to handle, originally taking about 75% of the time to actually parse the data that was generated in a given time span. I eventually optimized it a bit, so it will fetch data every 2 hours and take maybe a half hour to do the parsing.

So what am I going to do with this data? Well, there’s a ton of possibilities, but the best one I can think of is to offer a method for new Seasonality users to “download the last month/3 months of graph data.” This way users won’t have to wait so long to get a good range of graph data. We’ll have to see how this goes though. Obviously there are bandwidth and storage issues involved, and I’m sure that once the database gets beyond a certain size, scalability will be an issue as well.

Another cool database I found about a month ago was the GEOnet Names Server (GNS). This is a database of around 5 million different cities and geographic features worldwide. I ended up importing this data into Postgres as well, though I had to transfer the database to my G5 to execute queries in any reasonable amount of time. After creating several indexes on the table and giving the database 1.5Gb of RAM (enough to store the full table in), I got a typical select down to around 10 seconds for a full table scan based on indexed columns. It’s actually faster locally then using their web server to do a query…then again I’m the only user hitting it over here. 🙂 This data source helped a lot when I was trying to find longitude/latitude pairs for the new global locations in the upcoming version.

Everything you wanted to know about floats but where too afraid to ask…

There’s a quite interesting blog post over on Ridiculous Fish about how floats are formatted in memory and what kinds of numbers they can represent. This should be required reading for every Computer Science student. I wasn’t taught any of this at UCSB except for the mantissa/exponent way of representing a float and that sometimes an integer stored in a float was not really an integer.

Upcoming Events

Some very cool events are coming up in the next several weeks that I’ll be attending. The first event is the Ohio LinuxFest two weeks from now. Registration is completely free thanks to their sponsors, so all I have to pay for is the hotel. It’s a one-day conference covering topics from server virtualization to running an open source company. Should be good to get back out to a Linux-focused conference…the last one I attended was OSCon back in 2002.

The other upcoming event I’m looking forward to is DrunkenBlog’s Evening at Adler. There are some well known Mac developers who are traveling from all over to be on the panel, and it sounds like quite a few others are going to attend as well. I haven’t finalized plans to attend this one yet, but I’ll be trying to find a hotel soon.

This brings up one cool thing about living in the midwest: things are so close. I can hop across to a few different states without driving too far. Back in California/Arizona, you have to drive such a distance just to get to the next metro area and it’s a pretty big deal to go to the “next state.”

Hurricane Katrina donation

A couple of weeks ago, when I first mentioned Hurricane Katrina, it was barely a category 1 storm touching the southern tip of Florida. It is amazing to me that the storm grew so much as to become a category 4/5 hurricane, and my heart goes out to all the people who were affected by the storm.

Since the primary Gaucho Software product is a weather application, it only seems appropriate for the company to help in a time like this, so Gaucho Software will donate 25% of all sales during the month of September to the American Red Cross. As many people know, the Red Cross is pretty much at the forefront of the hurricane relief efforts, and I can’t think of a better organization to make a donation to.

I would also like to encourage others to make donations if they haven’t already. It sounds like many Americans (and others worldwide) are already banding together to help, which is great, but there’s still a lot of work that needs to be done.

© 2017 *Coder Blog

Theme by Anders NorenUp ↑