Partnered With the University of Illinois

Project: USGS Google Maps Update

Quick Links

Detailed Schedule 2008 Schedule
Current Design Prototype
Original Version on USGS website
Test Deployment on USGS Test Server
Meeting Notes with our Client

Documents and Presentations

Fall 2007 Final Report MS Word Format
Spring 2008 Final Report MS Word Format
Note: A cover page is included with the printed version only.

Preliminary Testing Plan: [ Word Version ] [ PDF Version ]
Requirements Document (Posted on 10/12/2007): [ PDF Version ] [ MS Word Format ]
Group Introduction (Presented by Walter Lu): [ PPT Format ]
Introduction to PHP (Presented by Carl-Erik Svensson): [ PPT Format ]
Google Maps (Presented by Jason Wei): [ PPT Format ]
Database Modeling and basic SQL Syntax (Presented by David Zazeski): [ PDF Format ]

Our Customer: US Geological Survey Illinois Water Science Center

The US Geological Survey maintains a website which provides the rainfall totals at different rain gauges located throughout the state of Illinois. The current website can be seen at: http://il.water.usgs.gov/gmaps/precip/. Our group will be making a variety of improvements to the current site including:
The group's weekly meeting with our sponsor takes place from 9-10 AM at the USGS office. All group members participate in the weekly meeting and our first meeting took place on Friday August 31.

Fall 2007 Weekly Status Reports

Archived at the following location:
http://slappy.cs.uiuc.edu/fall07/USGS-RealTime/2007status.html

Meeting Notes


Meeting Time
Agenda
9 AM Friday August 31
  • Discussed Project Requirements
  • Received Handouts about USGS
  • Met Stakeholders at USGS
9 AM Friday September 7
  • Discussed current system
  • Received code from existing system
  • Decided that the solution will ideally be in Perl since original system already written in Perl.
9 AM Friday September 28
  • Discussed project requirements
  • Collected information about testing environments
  • Agreed to exclude one rain gage from our missing data algorithm
9 AM Friday October 19
  • Discussed methods for updating tables with new data
  • Received feedback on proposed user interface
  • Provided overview of database structure
9 AM Friday November 8
  • Discussed how to handle updated information entering the database
  • Provided client with demo of current progress
  • Received feedback on proposed user interface
  • Analyzed security considerations with regards to the database configuration
9 AM Friday November 30
  • Discussed inconsistencies in the master.rdb file
  • Showed client an updated prototype
  • Discovered bug in Internet Explorer rendering of our JavaScript
  • Arranged for installation of PHP and readbackward Perl utility at USGS
9 AM Friday January 18 2008
  • Prepared plans for second semester
  • Updated client on our status
9 AM February 8
  • Delivered CD-ROM with current code to client
  • Received feedback about how our application should behave when no data exists for a specific gage
  • Established process for submitting bug fixes and changes to the USGS
    Our contact person (Terry Ortel) also has login and admin access to the related servers. He asked us to email him any code changes that we need to implement.
9 AM Friday February 23 Provided client with updated code release. Fixes included:
  • Under the old version, existing data was imported to the database without change. The new version calculates the amount of missing data for legacy data that is imported into the system. The old version only calculated missing data as new values are entered into the database and not when importing from the Perl based system.
  • Many crashing bugs (special cases -> empty files, no data in database, etc...) have been fixed in the update parser.
The following deficiencies in our existing system were also noted (specific plans for fixing these defects will be discussed at our Wed Feb 27 meeting):
  • An automated reload process needs to be written which runs on a regular basis. This process will clear all entries from the Rainfall table and reload them from the rdb files. This is needed since the USGS users sometimes manually edit the rdb files.
  • The missing data values reported on the website should be rounded instead of appearing as 37.1111111111111111111
  • The "last updated" time should not be null but rather contains the most recent time that a gage was updated.
  • If the user requests a custom time interval from 8 AM to 8 PM and our database only has data up to 7 PM, then the data from 7 to 8 PM should be considered as missing. Our existing system truncates the user's request and returns data from 8 AM to 7 PM. (Note: The times used are just an illustrative example and the problem occurs on any arbitrary times where the newest requested time is greater than the most recent database entry.)
  • The label above the radar map should reflect the data being displayed on the map.
  • The label "Precipitation data collected" should change depending on when the last gage was updated in the custom time interval.
  • Our system should gracefully handle the condition where data in the RDB files appears out of order (ex: 8:05 AM before 7 AM)
10 AM April 8 Here is a summary of the accomplishments from our meeting:
  • Delivered the latest code release to our client.
  • Established that the heading "Last Update Time" should refer to the most recent time entry stored in the RDB file.
The following issues are still outstanding and need to be corrected:
  • A legend should list all of the different missing data icons.
  • The USGS deployment environment has a separate server for PHP and MySQL. Our code should be tested for this configuration. This may involve deugging the USGS test server setup.
  • Try to reduce the slowness of JavaScript code for switching between different time intervals.
  • Replace the test data with newer RDB files.

Original Project Schedule

First Semester Original Second Semester Schedule:

Revised Second Semester Schedule:

Date Person Task
Jan 23 David Finish gage information importing
- Manually import 113 Gages and their information: name, agency, latitude, longitude, and gage time interval - Done
Jan 23Jason Finish the update parser (fixing control flow)
- Install various Perl modules (DBI, DBD::mysql) - Done
Jan 23Walter Missing data entry insertion
- Write appropriate MySQL statements to insert placeholder entries in a database to represent missing gage data - Done
Jan 23Carl Google maps interfacing with PHP backend
- Modify server-side code to query a database for gage location information - Done
- Create javascript functions to update the Google map with new gage readings - Done
Jan 30Carl and Walter Website done
- Represent missing data on the website - Done
- Make sure existing sorting functions work with new backend - Done
Jan 30David Miscellaneous Tasks
- Figure out how to create a cron job - Done
- Establish empty testing database for Jason - Done
Jan 30Jason Finish tasks from Jan 23 period
- Updates a database with new gage readings - Done
- Test with fresh database - Done
Feb 8Carl and Walter - Fine-tune website style and make missing data icons - Done
- Validate form inputs - Done
- Implement with update parser - Done
- Compile all files for the project on a CD - Done
Feb 8David - Establish preliminary testing plan - Done
- Replace hard coded paths with a dependency on a shared file - Done
Feb 8Jason - Debug the update parser - Done
- Compile TAR files for the application - Done
Feb 15Everyone Formulate testing plan - Complete
Preliminary Testing Plan: Word Version PDF Version
- Come up with various tests for each piece of code - Done
- Hammer the website with various inputs - Done
Feb 22Carl Website Changes
- Remove hard coded "October 16 2007" date from system.
- Check that the correct hour is passed to custom time calculations.
Feb 22Jason and Walter Remove Critical Bugs from Update Parser
- Debug new test cases for the update parser. In the case of a new gage, an empty RDB file causes the update parser to fail. Jason and Walter are working together to fix this issue.
- Isolate performance problems in the update parser. For one test input, Jason reported a 30 minute runtime for the update parser. This has not been duplicated, but Jason and Walter will work to try isolating any performance problems.
- Reduce duplicate code by implementing functions and subroutines for repeated calls.
Feb 22David Changes to Existing Code
- Remove feature from the utility which automatically inserts an empty rainfall enntry for a new gage. Fixing this issue also requires a code change in the update parser.
- Change data type of GageID from integer to string. This will require a reload of the database and appropriate regression testing.
- Reconfigure the installation/setup scripts so that any name can be used for the MySQL database. The current files have "usgs" hard-coded as the name of the database.
- Create test cases for the utility program.
Feb 29David Assorted Tasks
- Develop regular expression in Perl to replace searching for "19D 8 N 8N"
- Fix wording in testing plan(new draft produced -> more changes will be needed once development is finished)
March 7Carl and Walter PHP Fixes
- Debug an error in Internet Explorer where XML related content is improperly displayed on our website. As a testcase, observe the dates and times when performing a search with a custom time interval. Ideally the results will list the actual times/dates that were entered(seen in Firefox and other browsers). Internet Explorer displays a null entry for the date/time field.
- Correct all other website display irregularities presented by our client at the Feb 23 meeting(see meeting notes section above the schedule).
March 7Jason Finish parser improvements
- Ensure that both missing data and standard data are inserted into the database.
- Only connect to the database two times (instead of once per rainfall entry).
March 7David Assist with website debugging
Note: Carl found the bug solution first.
- Use the IEDebugBranch of the website to create a testcase for Carl's IE bug.
- Upon finding a testcase, determine a solution or workaround.
March 14Jason - Insert David's regular expression into appropriate areas of the parser.
March 14David - Fix path in wrapper script to gageTable.txt
March 28Walter - When the user requests data which is more recent than the latest database entry, the PHP front-end should append missing data to fill the time between the most recent entry and the time requested by the user. Walter will implement this change in the system.
April 4Everyone - Ensure that leading zeros can be used in the RDB filenames.
- Determine why null values are returned for the custom time interval when using the higher speed implementation.
- Ensure accuracy of custom time interval outputs (hand-calculate results and compare with actual output).
- Make sure that the newest versions of all files are in the deployment directory.
April 11Jason - Modify parser to gracefully handle empty RDB files.
- Add support for Daylight Savings Time. Currently our code generates an error when entering data around 2 AM on the day when the time changes for Daylight Savings Time.
April 11Carl - Change behavior on the website when no data is present (display a blank instead of a zero)
- Try to optimize queries by using SQL aggregate statements.
April 11Walter - Rewrite the queries in rainfall_data.php to reduce SQL calls and be consistent with the rest of our application logic.
April 11David - Create a shell script to rebuild the entire database with a minimum of disruption to the users.
Expected Final Release: April 11Everyone After delivering our first version to the client, we discussed a number of bugs which need to be fixed. The following process will be followed for each release to provide quality control.
Execute Testing Plan
 - Perform backend tests
- Perform frontend tests
- Try a fresh installation of our solution on test machine
Deploy to USGS Test Site
- Provide code and application logic to our contact at the USGS
- Deliver detailed setup instructions and support
Additional Steps for Final Deployment
- Perform on-site testing with client.
- Ensure that the application functions with their systems.

Group Members:

Team members volunteered for the positions indicated below. Some overlap between the positions may occur as needed during the project. For example, the webmaster may help the developers later on in the project. Specific tasks are given at our weekly meeting (Wednesday 6:00 PM at locked lab in Siebel).