As you may be aware, I was having trouble with my local-data-capture program (wxOCR, not to be confused with the existing open-source product
of the same name). It seemed the top segement was not being captured properly with certain characters, causing confusion and bad data. This was
particularly problematic with the precipitation sensor, where errors can cause a certain amount of precip to be recorded repeatedly.
Yesterday I turned my full attention to the problem and realized it was happening mostly with characters where the upper-left bar was extinguished (2, 3 and 7). Aha!
After a bit of thinking, I realized what the problem was: the detection zone for the upper bar for those characters was too long and was picking up some
whitespace.
Time to get technical.
The program works by measuring the pixel values within each segment, and coming up with an average value. For each character in the detection grid,
you have to specify a contrast value--it's the only way to get reliable results across an image of varying
brightness. Essentially, you have to tell the program where the cutoff value is between dark (lit segment) and light (extinguished segment).
But this presumes that all segments are all-black or all-white. Mixing in some whitespace caused it to fail by lightening the average brightness of that
particular segment. With some
characters, this wasn't a problem; the error-correction recognizes partial characters which, for the most part, are unambiguous. But a seven missing its
top segment is a 1--a legitimate value--and that wreaks havoc with the precip readings when it happens over and over again.
Why primarily the top segment? Because in almost all segemented displays (LED, LCD, etc.), the characters are slanted.
The solution, in a nutshell, was to shorten the upper segment and move it over a bit, so that it only captured the darkened segment. It's now capturing at
near 100% accuracy. Error-correction is kicking in at only a fraction of its former frequency.
I've started rolling out template-code view of selected pages; it will eventually encompass the entire site.
Template View is much like the "View Page Source" in your browser, except that it shows the actual source code the page was generated from.
Embedded tags take <#ThisForm> <@OrThisForm>. Tags which borrow from another location's data (for example: YOW) take <@ThisFormat:L YOW>.
I'm taking a little break from developing. Job search is not going well, I'm just about out of money,
I've lost my enthusiasm for most things, and I just can't be bothered. So, for a while, the forecasts will be broken.
The City has now painted an embarrassing red X on the sole remaining ash tree out front. Therefore, sometime in the next
three or four years, the skycam field will become completely unoccupied for ten or fifteen years, until the replacement grows
into view. I say three or four years because the City first sprayed a red X on the other tree (cut down just a few weeks ago)
three years ago.
I've updated WxOCR to V2.02; I upped the number of character validation entries to 60--the list is getting long with partial-character entries.
This week, the City came and cut down the diseased ash tree next door. You can see the results in the webcam query page. Despite having a clear view if I shift the webcam
slightly to the left, I've decided, for at least the time being, to keep the other two trees in view, for a fixed reference point. Rationale: more than half the frame is now
open, and I've set the sweet spot (i.e. the one-ninth section that WxSky uses to recognize the sky condition) in WxSky to Zone 6 (lower-left-hand corner). This patch is (now)
always guaranteed open; so shifting the cam gains me only some more empty sky.
I've also changed the look of the security cam picture in the home page; I moved the banner to the top and added in the seconds, since it actually images
four times per minute.
I'm having a whale of a time with MCH. Every time I get to a certain point, it starts failing; a data structure that exists is fine on the first two passes, then disappears.
I'm pretty sure I've run into a compiler bug; I'm now perusing FPC/Lazarus bug reports and investigating work-arounds. NB: it is a compiler bug, and I am now working on workarounds.
Version 3.82 didn't fly. There was a weird bug when trying to open readings--I'm just glad I caught it before the restart.
Too many changes to the code in one go. I've reverted to version 3.80 and am rewriting the changes.
I've really focused on trying to find work--anything!--the last few weeks.
I have tweaked WxOCR to return nil values for dew point upon nil inputs. I was having a problem with the wrapper routine. Fixed. V 2.00.
WxOCR, the local-data-capture program, is up to v 1.92. I squashed an emergent bug to do with precipitation.
The update to MCH is taking longer than anticipated. I'm still getting odd values for precip and sunshine. I'm debugging heavily to try and
find the problem; it seems to be related to an odd value being kept for sunshine when (otherwise) zero. I'll try something today and see if it
works.
I'd just like to point out that the data capture has been, thus far this week, perfect. There--that's bound to jinx it in the next few minutes! [Surprise, surprise--
it didn't; the entire week was perfect.]
That's not to say that the OCR program isn't making a few mistakes. However, it's automatically correcting the mistakes it is making. It's funny--auto-correction is an inherent
property of the system, even though it wasn't planned that way.
Speaking of said OCR program, it's up to v 1.90; I've added buttons for other conditions and added carry-through between runs, so a longer time period for a second sky condition
is not a problem. A countdown timer (which resets to 20 minutes) indicates how long is left. Here's a screenshot:
Needless to say, me happy and looking forward to getting a build in for the Sunday-morning restart.
I've gone ahead and released v 3.80, which fixes an obscure internal time bug and introduces daily rollover to the location stats. It also fixes the
forecasts-submissions bug - I had to!
Should be the last significant change for a while.
I've added a multiplier (4.0x) to the wind speed, to generate more-realistic values. In doing so, I had to tweak the local-data-capture program.
That's WxOCR v 1.84c.
I know there have been lots of little changes lately. That's likely to let up for a while; I'm going to concentrate on bug fixes and just let the system do its thing.
I don't have any good ideas for new features for now. I'll give that some thought.
I've added a new form, to allow additonal sky conditions to be reported.
The Additonal Observation Page allows you to specify a second sky condition and a duration in minutes. I've hooked it up through the CGI-BIN
interface, and it should just work for you. Remember to use the "submit outside" button (there's a better way, and I'm working on it), and also remember that it may take
up to five minutes before your update shows.
Also, now when rain is detected (and lately it has, as the rain sensor slowly thaws out), the second sky condition is set to 'Rain' for 20 minutes, automatically.
WxOCR is now up to v 1.82, and wxupdates to v 1.22, for those who are keeping score.
Incidentally, on the Diags page, the number of forecasts accepted will always be a little higher than submitted. This is because the forecast is stored in an external file and
read at runtime. As it doesn't go through the usual submission process, it's not counted as a submission. A bigger hassle to fix than just to leave alone.
I've released a new version (1.78) of the OCR local-data-capture program (WxOCR) (which, incidentally, has been operating almost glitch-free this week).
This verison includes buttons to temporarily manually assign a second sky conditon (ie "Cloudy, Light Snow"). A covenience for when I'm around and note phenomena that the
station isn't good at catching in realtime--rainfall, snowfall, visibility. I'll work in a way and a form to submit the same thing remotely.
I've also upgraded MCH to v3.78. This includes extended enhanced statistics, as seen on the Diags page.
I believe I've finally resolved a long-standing bug with erroneous dew point values. Make that v. 2.16 for wxhtml (the program that captures
web data every five minutes).
While I was at it, I beefed up the Regional and National pages to display extra stats and get rid of others.
I've released v 3.76, which (a) fully fixes yesterday's fix (had a bug with dates); and (b) changes the system so that it won't just grab the first forecast
for a new period; it will grab all subsequent, as the forecast sometimes changes. (Forecasts are updated every five minutes from source.)
I've released v 3.74, which changes the way submitted values are read: now a NIL value will not replace a valid value. This is in effect
for normals and records import.
Had a little fall-apart for a few hours yesterday morning; forgot to use the test server, and web-captured values were all over the place. Won't make that mistake again.
I've released version 3.72, which incorporates an extended statistics set. The new values are on the Diags page.
There'll probably be more of these little enhancements over the next little while.
This project is slowly growing beyond my ability to maintain it--it's up to 14,812 lines of source code, plus all the auxiliary files. At the same time, I can see
how much work I have to do before I open-source it. Kind of a catch-22.
I noticed in the wee hours last night that the Wind Chill value was not getting passed to the front end. After a lot of investigation into the capture program,
which, incidentally, was working just fine, I realized that my bounding values (in the System Properties) were incorrect. Those were something I'd only recently added.
As a result, Wind Chills greater (lesser) than -10 were being thrown out. Simply adding a zero to a single parameter fixed it permanently. Wind Chill is tracked both for CTO
and for the airport.
The lesson here? Be very mindful of operating parameters.
Oh, and I've changed the live-data-capture program always to give a value for wind chill for ambient temperatures at or below -5C.
The Master Recalc function is taking a little longer than anticipated. It's a huge procedure (nearly 700 lines so far),
and there's a lot of math and a ton of transactions involved. As my primary focus at present is to obtain employment, it'll be ready when it's ready.
I have upgraded the web-capture program to catch wind gusts; something I overlooked earlier. You'll notice that on the Airport page.
Coming in the next 48 hours or so: Master Recalc. This lets you recalculate the data from source (live
readings or daily observations). Optionally, you can recalculate Limits (highs/lows/averages), Extremes, Records, and Normals.
You can select for daily, monthly or yearly recalculation. It's really a multipurpose query. I've got just a few little bugs left in it,
then I set it loose.
A page is in the works. You can see it here. It's under the queries page.
You'll notice on the Records page a couple of new sections: Administrative Info and Calculate Records. The latter allows
you to calculate records over any date range (bearing in mind that data has been collected at CTO only since 2001). Be the first in your
neighbourhood! Just remember to use the [submit outside] button.
This section took some work. A lot of variables being manipulated.
My test machine has been invaluable, these past few releases. It lets me fully debug my software in s simulated production environement
before releasing it. There may still be a glitch or two in this latest release, but I'll iron them out quickly enough.
Incidentally, the wind sensor came back at about Noontime Thursday. Guess the days have gotten long enough for it to build up a charge.
I am tracking it again.
The software is now at version 3.54. I also added support for a couple of new directives (regenerate a location, or all; and reload system
properties.) Several new data-output tags are also available, associated with records.
I've added some System Properties. There's now a number of parameters set in there, and it will expand in future.
A System Property is just a basic string, integer or boolean setting with a name. I've written a module to handle them almost transparently.
I'm using them to handle list-type settings; what to track for the hit-accuracy counter, things like that. You'll notice I've disabled tracking the
wind readings for the winter.
Still working on records recalculation. That's still going to take a bit of time.
As promised. a new feature has been added to the system: Normals.
The new normals page displays the normals for the day, month and year. It also displays some administrative information about
the normals set. Finally, a section at the bottom allows you to calculate normals for yourself.
Note that normals are currently limited to temperature and precipitation, just as with the government. This may expand in future.
Next up: recalc records. That's a little more of a challenge; it'll take a little while longer.
I did fix a bug. Recently, I had added a new debug module, which works much more like the java module I'm accustomed to. That meant
deleting quite a few lines of code. An unplanned
side-effect was that for any location that had records passed to it, that location's pages wouldn't be updated (unterminated begin..end block
[don't you laugh--curly braces mean the exact same thing!] leading to another block which tested for the need to regenerate the pages).
As the Airport always has records passed to it, that means the Airport pages weren't being updated. Moving a comment marker five characters
to the left fixed it.
Actually, make it two bugs: I've fixed the forecast-graphics bug.
Incidentally, my wind sensor seems to have frozen up for the winter. Made in California; what can one expect? Bah!
