Ongoing regression testing revealed that the system was misreporting last-observation times for other locations (it was reporting it as the last-observation time
for the location owning the page). This bug has now been fixed. Version 4.66 will be released for the 5am reboot Sunday.
A sharp-eyed visitor noticed an inconsistency with the year being reported for events. Turned out I was using the wrong embedded tag! Two minutes in the text
editor fixed that. Thanks to Sharon Dagenais for pointing that one out. A bug in the data, that.
Also, it's been pointed out to me that while I'm using the plus sign everywhere for readings at or under 5 degrees celsius, in fact Environment Canada does
so only in the textual part of their forecasts. I'll be changing this very soon, to comply. Thanks to Francisco Zarama for that one.
I'm busily putting together sunrise/sunset and moonrise/moonset tables for CTO and the airport. They'll be input before midnight, to ensure we have
sunrise/sunset and moonrise/moonset values for 'tomorrow'.
Finally, we had some downtime this week, courtesy of a popped circuit breaker. We discovered that: A, the upstairs spare room and the kitchen are on the same circuit;
and B, you can't run the clothes dryer and a heater at the same time on the same circuit. Rebooted the server and didn't lose any actual data.
An addendum: while feeding in the figures for next year, another bug was uncovered. I had rewritten the OpenYear() routine earlier this year, and apparently I had it such
that: A) Any input date beyond the current year led to no year being assigned. But the routine was still trying to assign values. The predictable result: boom!
This has all been fixed. Going forward, any date beyond the end of next year will cause the input to be rejected and a note to go to the logfile.
A further addendum: I've now replaced the Moon_Age() and Moon_Phase() routines with something that seems to work. Bumped up the version to 4.66b. I've also corrected
daylight calculations (in some instances, bizarre results were obtained).
Ongoing regression testing indicated that the bulk-data query function was mostly broken. The results were supposed to be carriage-return-delimited, but weren't. Also,
some of the tags were not being recognized. I believe I have fixed the bug for each of the specific functions (One-time, Live, Daily, Monthly, Yearly). Feel free to test
and let me know if you're having problems.
For Sunday, I'll have V 4.64 out. And, who knows? Ongoing regression testing may yield other emergent bugs. Can't wait to have those all wrapped up--and that's going to be
very soon. Literally, I have three things on my blue-sky list:
(1) Splitting off extremes so that they don't co-exist with observations and statistics; (2) Adding some minor sortation function to a few of the loopable data elements:
mostly events, and (3) Adding a few other data elements: snow on ground, growing season. I don't even know that any of those things is worth doing. Remembering Pournelle's
Third Law--a job not worth doing well is a job not worth doing--I'd say to hell with it. Apart from bugfixes, I don't see major changes coming. The system does everything
I want it to do and is gradually becoming too big for one person to maintain. To this end, I am carrying on with my efforts to clean up and document the code, prior to its
being released as open source.
I've released a new version this morning: 4.62. This version corrects a minor bug that was affecting last month's and next month's events. I've also changed the
observations input format to allow for one extra parameter--a single character, 'm' or 'c', indicating the units of precipitation. If it's 'c' (cm), it will be
an indication that the precipitation amount is in centimetres, as snow; else it's taken as millimetres of rain. Going along with that is a new tag, <@CSPU>,
which yields either 'cm' or 'mm' for the precipitation value.
In tandem, I've also released a new version of mcOCR; 2.78. In this version, the 'Manual Precipitation Amount' field is now followed by a selector, for 'mm' or 'cm', to help
when entering in manually-measured precipitation amounts.
Finally, I noticed that new forecast terms were not being added to the database. I found that where it calculates the insertion point, the IP wasn't actually
being recorded, so it would assume the term already existed. Easy fix.
I've introduced a new menu scheme to the site: dropdowns. Before, the whole site was accessible through a bank of chiclet-style butttons--check the Airport page for an
example.
I'm finding that the new buttons are much neater, but the site as a whole is somewhat less discoverable. I might have to add a sitemap.
Regression testing has yielded a minor bug; last and next months' events aren't making sense. I'll have that fixed for Sunday.
This evening, I noticed that the wind chill calculation performed by mcOCR yielding incorrect values. This has been fixed and verified against Environment Canada's own
online calculator. Bumped the version up to 2.76b.
I noticed this evening that the wind chill index value was not being forwarded to the weather system. A very quick investigation revealed that a change in mcOCR, sometime
since last winter, was causing too-low values for the wind velocity (a tenth what it should be) to be forwarded to the CalcWindChillIndex() routine. Three little characters
solved that problem quite neatly ('*10' if you hadn't already figured that out).
I made a couple other small changes to mcOCR. Up to V2.76.
I've been losing Dew Point readings this morning; an artefact of the temperature being below zero. Believe I've fixed it now. We'll know tonight.
It came down to scanning the supplied external temperature. I initially wrote it to scan for a first digit--forgetting that that first digit could also be
a minus sign. A simple function substitution fixed that.
In other news, I'm really looking forward to V4.60's deployment next weekend.
(Update 2017-11-09) I've modified mchtml so that when yesterday's data are not in the source page, it won't 'make stuff up' by searching too far.
The long-anticipated temporal-lookup function has passed SIT/CAT testing. It was a bit of a bugger to get there. I found numerous gotchas that had to be remedied; things
like sloppy date handling; a badly-written core of a grid-parsing routine; a 'correction' that had the Templates Unit processing the wrong files for location lookups.
All that was fixed; and, at this point, everything seems to be working normally. I've piggybacked Temporal Lookups to the Location Lookups; it seemed a good match.
Optional date and time fields are provided; fill those in, and your location lookup becomes a temporal lookup as well, giving you a snapshot of conditions on a given date
(and time).
In this upcoming V 4.60, I have:
Fixed a bug with one-time queries generating date@date labels;
Implemented the temporal-lookup function. It looks up most data brilliantly. It does not (will never) do forecasts, warnings, graphs.
Fixed the core of ParseGrid(), which was built in the initial expectation that temporal lookups would never be required. Had to rewrite it.
Made all lookup functions adjust to match DST time in queries. No more looking for 06:00, getting 07:00.
I have this version scheduled for deployment Sunday morning, Nov. 12. Until then, we're still running on V 4.52.
No time to work on the temporal lookups; that's once again next on my To-Do list; but my radio show currently has priority.
In the meantime, I've worked on remediating some simple bugs, as referenced in my previous post. Live bulk-data lookups should now work as envisioned. Log traffic is again
vastly reduced, and on average queries should be responded-to faster than before.
In order not to delay my latest work's from reaching Production, I am releasing V 4.52. Currently in SIT/CAT testing; so far looks good. Next weekend, I promise, I'll
have 4.60 with temporal lookups. I say this with some
confidence, as I am actually working on it now, and some recent planning changes have drastically simplified my work,
(Update 2017-10-31: I've fixed a wee bug whereby the system was not dealing well with negative dew point values for some locations. A bash thing. Temporal Lookups coming
along; just debugging now.)
Well, this week was a bit of a bust, between network problems, pet emergencies, duties from my radio show, and the repercussions of my having restored everything from
subversion. Incidentally, I must
remember not to restore from tags unless I immediately switch to trunk or relevant branch. I am working on the
weather server and the first pre-testing versions of StationBase, while also generating new stuff for my show (and, it's our annual Funding Drive). That's labour-intensive,
and it's already mostly automated/digital!
Ongoing regression testing indicates an issue with bulk data retrieval, at least with the live data. I'm working
on that. (Update 2017-10-26: I've now fixed that bug; plus an obscure one that almost never happened.) I've recoded the statistics-import routine, so
that it does drastically less work recalculating things. I also made a change to mcHTML, which captures web-based readings, statistics, records and normals. It now keeps a
running list of sites it's analyzed, and the latest times. Now, when it's called, it compares the listed time for a given location against the time indicated on its
downloaded web page. If it's newer, then a note is made and input files generated; otherwise, no input files are generated, on the assumption that it's already been
received for that given time. That's significantly cut down on the number of data packets fed to the system.
Finally, I've
rearranged the schedule in the daemon routine; now, in the first half of each minute, which is allotted for files to be copied around, web pages will be regnerated. In
the second half of each minute, inputs are searched for. Of course, it searches for control-type files (directives and queries, mostly) at all times when not doing something
else.
The net effect of all this is a much more responsive system. Queries are answered as fast as the system can process them; there's little waiting now. I haven't much altered
what gets logged, but there is a drastic reduction in log size. Moving in the right direction, anyway.
I'm hoping to have version 4.60 released by next weekend, including a fix for the bulk-data bug and implementation of the temporal-lookup feature. I've recently had a
realization that will make it almost trivially easy to make that addition.
The server is working but not receiving readings for YOW. The sensors for CTO are currently down. The one has to do with the other.
Working on it. Lost a little work because of the recent hard drive crash; I'm having to recode it in a hurry...
(Update @ 13:51: Readings partially restored. Fixed a problem where multiple readings were waiting in the input queue of one of the support programs. The external
sensors are still out. I'll try resetting the transmitter if signal hasn't been restored by tonight.) If all else fails, we pause for a couple days, and backup sensors.
I've done no work on the weather server this week. Monday evening, there was a drive failure on my development Linux box, and I lost all of my development files. So, for
the past few days, I have been reconfiguring the system to run on just the one drive, and restoring files from Subversion. Luckily, I've only lost a few hours' work.
Starting this week, I will work on the temporal-lookup function. I'm also beginning to invest some time into StationBase, my other project now in the works, now
that DX'able conditions (reception of distant radio stations) at reasonable hours have returned.
A word of explanation: for nearly 40 years, one of my hobbies has been to sit up at night with an AM radio, and see what distant stations I can receive. It generally only
works at night, which, at these latitudes, is quite short in summer. Now, for all of those 40 years, I have been keeping track of stuff like that with good old paper.
Well, it's the twenty-first century, now, and time for a technological solution. I can't think of a better project to do for myself; it'll be a great way to learn
java spring. I see DX logging products out there, but not many; and they're either horribly complicated, or commercial. Right now, I can't afford commercial.
Regression testing indicated that Records Recalculation was not storing proper information in the headers. Turning out to be storing the information after writing the
file. This is now fixed.
I've fixed the statistics-processing routine. A partial rewrite was required. It now handles multiple entries as bulk input, wherein certain calculations (such as
recalculating for months and years) are not done, on the assumption that a specific recalculation subsequently will be done, or that the data will be supplied. Speeds up
statistics input dramatically.
I've reconstituted the YOW historical dataset from 1938-2011. Precipitation values were not correct; this was an artifact from the original data import, circa 2012.
Finally, an update on an existing bug: since cleaning up WriteNormals behaviour, I haven't encountered the error where normals files are disappearing.
This week I'm working on the temporal-lookup function. In addition, I am continuing with regression testing and code cleanup.
(Update on network problems: the issue has been resolved. No more network problems. Yay! Only remaining debug possibility is that my laptop's network card is in the
process of going legs-up. No matter; have a good, solid, fast wireless signal.)
I've a new version of MCH running as of this morning. V 4.48 implements several fixes, not the least of which being that it generates far less log traffic.
In monitoring performance, I identified an issue with normals sets being rewritten to disk far more often than is necessary. I've implemented a fix which causes normals
to be saved only if the specific values have changed or been recalculated. About one one-thousandth as often as before.
I've made a couple fixes to the way live data are exported (primarily for graph generation). Now, the first time-index (which was almost always wrong) is skipped in
last-24-hours output. In the same file, the L24Tm label has been fixed to represent actual elapsed time from the first entry. The improvement shows up in any
last-24 graph, such as on the Temperatures page: The first tag is where it should be; the graph uses up all the space. Looks much better.
In addtion, I noted that wxhtml wasn't always capturing records or normals quite accurately in the early part of the month; and, it was generating spurious readings for
locations for which official normals and/or records weren't available. I've hopefully corrected both these problems. wxhtml now up to V 2.42.
Still logging items to be fixed as I review the code and go over the website with a fine-toothed comb. As you can see, I'm not finding much. Still putting off the
temporal-lookup feature.
Finally, I'm still having an issue with individual normals sets being wiped out. Near as I can tell, the pointer is being overwritten. This version includes debug code
that hopefully will help in tracking it down. WxIO.log is the one to watch.
At present, I'm having problems with my network. I'm going to try replacing the upstairs switch.
Update: replacing the switch was brilliant. I used an old wireless router that I'd hung onto when upgrading some years ago to Fibe. Set it up as an access point.
The network issue is resolved, and to boot I have tons of wireless signal now in my upstairs office (the server room).
I don't anticipate any more network problems (readings lagging behind, lost data, etc.). It's all working well again.
The capture server crashed at Midnight last night, in the middle of a major network outage. Functionality and communications were restored at 7:19. I have no data for
the period in-between. I don't think that's ever happened before.
I've tweaked a few things and somewhat bumped up the version, to 4.44d. The original production version 4.44 was generating far too much log traffic,
and an ongoing code review found an error in the routines to find Log and Shutdown Reports files. Both were fixed.
At this point, I'm reviewing the code, cleaning it up, and reformatting. Squashing bugs as I go, but there aren't many left. I have just one feature left
to add--temporal lookups--and I'll probably do that next week.
Of course, if you've got a feature suggestion, please don't hesitate to drop me a line.
I have V 4.44 ready for tomorrow morning's reboot.
As with last week, I fixed mostly small things:
Fixed a bug whereby some FindFirst operations went unclosed.
Added new directives RELOAD-CCTERMS-{LocCode}, RELOAD-NORMALS-{LocCode} and RELOAD-TEMPLATES-ALL
Fixed up how RELOAD* files are handled and parsed
Tweaked the format of current-conditions.ini
Built a GUI CCTerms editor (mcCCT); this helps you match up current-conditions terms for any location to appropriate graphic files.
Cleaned up some stuff in the Template Processor.
Got hit-counting working again with Forecast Terms and Current-Conditions Terms. The figures themselves mean little, but they do show which
forecast and CC terms are referenced most often.
Changed the way mcOCR handles weighted values (i.e. the segment with the lowest value wins), to add and use SegThresh values to make the output a
little more accurate. Cleaned up some memory leaks, too; program now runs for a week straight with no issues. Version is now 2.72.
This will probably be one of the last weekly builds I come up with. My Bugs list is down to a few obscurities, and my ToDo list has a few things left,
but I'm not sure I want to tackle those right away. As before, I'll be focusing on going through the code, prettying it up for release, completing the documentation.
The code is stable and mature. The station has run for over five years, now, and data-collection at this point is nigh-perfect. I've been waiting for this.
I had to do something one-time with the dataset, so I took the system down for about 15 minutes this morning; while I was at it, I installed v 4.42.
I fixed a lot of small things in this version (most of them just this morning), and added a couple of new features.
Fixed a problem with how current conditions were being read; an emergent bug from a recent change to how input files are read. Had to restore CC files
from backup.
Fixed a bug affecting lookup operations; an emergent bug traceable to recent changes in Template processing.
Fixed a bug affecting date display of obscure formats
Added new date templates: T6 (17-07-30 - deprecates the old :6), T8 (2017-07-30 - deprecates the old :8), TF (July 30, 2017), TS (2017-Jul-30)
Added new tags to access tamplate information for a given page
@DDN - Page Destination Name (Name only)
@DDP - Page Destination Name (with Path)
@DRLD,T... - Page Last Regen Date+Time
@DRND,T... - Page Next Regen Date+Time
@DRs - Page Regen Schedule (short - just the numeric code)
I'm working on V 4.38 for this weekend. So far, it's been extremely stable.
In this version, I've fixed a bug whereby data-extant tests with a false answer and no else clause were removing an extra character from the feed. I fixed a bug I
inadvertently introduced last version, affecting userdata. I was getting the occasional crash at shutdown, and I've fixed that.
The big feature for this version is the ability to define template files as either event-driven or periodic. Event-driven templates get refreshed whenever the system
absorbs new data for the given location. Periodic templates are automatically refreshed on a timescale (which you can specify) of minutes, hours, days or months. Some pages,
such as yearly data, only need to be refreshed once a day at most (current year) or maybe once a month (past years; next year); now that can be specified. I added the month specification
because it made sense to be able to do a thing monthly, at the same date and time. It really reduces the load on
the system. For example, to generate the last-year page for the airport, MCH would have to open around 80 year files, which takes a little time. Now, it does it once a month.
Also, the template specifications are
now held in memory, rather than re-reading the file all the time. Because there are data that change, the file is rewritten periodically. And, precisely because all the data
files are human-readable and -editable, you can tweak the dates and times for long-term regens.
I've also added in some new directives: RELOAD-TEMPLATES-{LocCode} and RELOAD-TEMPLATES. The first variant reloads the templates for a given location; the latter causes
the system-wide templates (for now, primarily the location-lookup templates) to be reloaded. And, of course, as before, you can regenerate all pages by issuing a
REGEN_{LocCode} directive. As an added nicety, forced regeneration doesn't alter the Next-Regen values, so you can maintain a schedule.
At this point, I have no bugs remaining on my to-do list. All the rest is nice-to-have stuff--some big, mostly small--that I may or may not feel compelled to work on
at some point. (Apart from a visual Current-Conditions terms editor, which I know I have to do; luckily, the groundwork's already laid for that one, with the recently-
introduced mcFCT.) In weeks to
come, I'll be focusing on the codebase itself, getting it more fully documented and reformatted. Then I'll need to turn to the documentation
and get that completed. Then I go open source. I'm also working on (and falling behind on, though I'm the one setting deadlines) a project in Java Spring.
I've bumped up the version number for mcHTML, which captures statistics, records and normals from downloaded web pages. Further tinkering by the good people at Environment
Canada had made it impossible for the program to pick up data-related dates properly. Fixed, and now at v2.40.
Finally, this is advance notice that I intend to convert my server to Debian Stretch (9), probably on Saturday, July 22. There will be further warning given as the date
approaches. I anticipate downtime of about twelve hours if it goes badly, and six if it goes well. (Please, well; I'm due a break.) I don't expect to lose any data in the
process. The local data will be diverted to a holding area on another machine, and the web-data capture function will likewise be moved to another machine and held. We will,
lamentably, lose the webcam during that downtime. I don't consider that a significant loss.
(Update 2017-07-16): The transition to the new version was not quite seamless. Some bad entries in the templates.ini file for some locations were causing crashes. Finally
got it all working at 6:30, as you can see on the Diags page. I've also made mcOCR a whole lot more stable; we'll see a whole lot fewer crashes
from it now. It's now at v2.68.
Some of the changes I've recently made to mcOCR include: ability to specify a manually-measured precipitation amount for inclusion into the next outging data packet;
eliminated unsent-message buffering--never did work properly; fixed a persistent memory leak that limited the amount of time mcOCR could run; observational data are now
saved to local disk files and transferred by a shell script--eliminates problems associated with network outages; and source image files are not deleted until the data
have successfully been extracted and written to disk--if the program crashes, the data will persist.
MetCommand-Home V 4.36 is almost ready, pending an overnight test.
This version includes a bugfix for the random-crash problem, so we're back to near-continuous operation (weekly reboot). I've also fixed a bug that was applying live
precipitation readings in some circumstances only to Total Precipitation and not to Rainfall as well. Most data files now have headers and other clues as to what data
they contain and thus are self-documenting, in keeping with my pledge that all stored data be human-readable and -editable. The missing-day-numbers bug in the
last-31-days export file has been fixed.
Most importantly, for all items with a data-extant function (which displays information only when a data item, or set thereof, exists) (i.e. Forecasts, Warnings, Events,
Viewer Items), I've now added an optional
<@else>-type clause, so that something specific can be presented when the data do not exist (such as a message to that effect). It's useful in some places,
less-so in others. More to the point, it's another tool at your disposal when designing pages.
I've upgraded my test environment to Debian v.9 ("Stretch") (which went better than my desktop upgrade). Soon I'll be updating Production, too. There will be a few hours'
downtime when that happens, most likely next Saturday morning. You know I'll be backing up very carefully for that one!
I've got a few more items on my to-do/wish list, but it is getting short, and I'm not sure if I'll tackle every item. For now, I'll mostly be going through the code to add
comments and reformat, and finishing the documentation, in preparation for taking this entire package of software and tools open-source. I've talked about it for a
couple of years; the time draws near.
There has been no new version, the past couple of weeks.
Last weekend, I lost a hard drive on my main development machine. It was the boot drive. So, I got busy with System Rescue CD and testdisk,
salvaging what data I could from the old drive (It was mainly a boot-and-software drive, so there wasn't much). Then I replaced the drive with a good, albeit used,
drive. I knew that the latest Debian Linux had recently been released, so I downloaded a copy of that--and the 64-bit edition as well.
I was switching over from a 32-bit installation, so I was intalling from scratch.
The Debian installation went well, as always. When I rebooted, there were video problems. I have older nVidia hardware, and the stock drivers weren't working properly
with it. So far, I've pored through countless Internet resources and learned two things:
Yes, the Powers that Be at Debian are aware of a problem with some installations and nVidia drivers;
*Smile and shrug*
Anyway, long story short, I've found a way to get to a command prompt reliably, using the Xfce Desktop. It's not as fully-featured as the KDE desktop that I'm more
accustomed to, but it's also an old friend of some ten years' acquaintance. It'll do, for now.
Incidentally, I believe most of my older hardware, including all of the equipment that runs the weather system, is stuck with 32-bit software. With this in mind, I've
installed a 32-bit instance of Debian--also with video problems. Anyway, I've set up my development environment in that instance, as well, so that I can easily generate
32-bit software; just reboot, reload and recompile. (Believe it or not, it was easier this way than to set up for cross-compiling.) And, since it was essentially in an
identical state to the 64-bit installation, configuration was a snap, involving copying over ini files.
I'll have a new version, with more-targeted debug code, within the week. I need to track down a bug which has been crashing the software about once a week.
(update 2017-07-05: the bug is now fixed. I deeply regret testing in Production, but it's the only place that receives all inputs
[don't forget, I track 29 locations], and the bug was with the
very last location. Turns out I simply wasn't initializing a pointer.)
I'm working on V 4.34 for the Sunday-morning reboot.
Here's a rundown on what I've done thus far:
I've fixed a bug relating to the display of daily precipitation values in a top-xx list.
I've fixed up the header lines in the export files to match spec.
I've changed the format slightly of the daily export files. A zero value is now always included at the beginning of data, causing remaining days
and months to appear in the graphs. This required modifying some of the graphic scripts.
I've added somewhat-extended normals to the daily, monthly and yearly export files.
I've added records export for daily, monthly and yearly records.
I've defined the format of the annual stats export file and now export data to it.
I've reviewed and ensured that each export file has a header that makes sense.
I've fixed a bug where the captured forecast humidex value was showing low (mcXML now at v 1.36).
I've created some new graphs--daily record temperatures over the year; daily record precipitation for the year; and monthly normal precipitation. They'll be placed
on pages currently lacking graphics. You will note that, based upon years of observations,
I've adjusted the lower temperature limit of the graph to -35 celsius, and the top end to +45, in anticipation of records to be shattered in the next few years.
I'll propagate this change to the other graphs.
This week, I taught MCH to generate top-xx and bottom-xx lists for various weather phenomena (highest temperature, lowest precipitation, etc.).
The basic premise is to take a block of time and analyze. You can look at daily, monthly or yearly statistics. You can look for any day, any month; or for
specific days of the year or month. You can also narrow your search to a specific month.
I've tested it fairly thoroughly, and it seems just to work. The output is very plain, and I'm open to suggestions regarding its formatting.
I've made this feature generally accessible, as it does not change data. It's available through the Query Page.
Over the past week I've also:
Scanned the entire website for bad tags, and removed one;
Added some code allowing MCH to properly manage forecast terms;
Restored forecast-term hit tracking.
Accordingly, I have v 4.32 queued up for the Sunday-morning reboot.
Once the new version's running in Production, I'll copy over the handy new Query-Lists page from the testing server.
The time is rapidly approaching for me to complete the end-user documentation and open-source this thing--as I first promised a couple of years ago.
During the past 48 hours, I whipped up a little visual editor for forecast terms.
It lists the terms out in natural order (longest-to-shortest). "Hits" is the number of times the forecast term has been referenced (a feature I'm going to turn back on soon).
Essentially it's for making it easier to match up new forecast terms with appropriate icons.
Double-clicking on an entry in the list opens it for editing:
It's all pretty straightforward. This was a little side project that took me all of about 16 working hours to come up with. In the near future I'll do up one for
current-conditions terms, which are tracked similarly.
I've just completed v 4.30, which will go into Production during the Sunday-morning reboot.
For this iteration, I've done a lot of little things:
I fixed a small bug affecting recalculation of annual sunshine amounts;
I expanded the specification for a forecast-delivery file to include optional values for the forecast Humidex, Wind Chill and Visibility.
I've upgraded mcxml (the forecast-and-stats capture program) to v1.34. It now captures additional terms (the entire expanded dataset) from the forecast text
and includes them in the forecast details;
I've made the same available through an expanded forecasts tagset. (See the forecasts page for an example). This required some changes to the I/O engine and the
Templates Processing engine.
I've expanded bounds-checking for Normals recalculation, in keeping with the expanded dataset;
I've cleaned up the output from Normals recalc, one year per line. Much neater;
I've added the expanded Normals dataset in every place that was relevant on the website.
I've fixed a bug that caused some error tags not to appear.
I updated the code looking after forecast terms, to ensure they're written to disk without delay and properly initialized before reloading.
This will facilitate their editing, in a forthcoming parallel project.
I've added the capability to recalculate Extremes along with Records, and ensured that both now obey useage mapping. See the Records page.
I'm finding that mapping out in detail what I'm about to do, and where—especially what routines are affected and what code I'll have to add or remove, really is
helpful in keeping my goals straight and cutting down on bugs, cross-referencing of code and variables, etc.;
I'm getting rapidly now to the point where I've got a few items left on my list, but all things that I really don't want to touch for now, or involving other, sometimes
unrelated, projects. Bearing that in mind,
it's time to ask you what you'd like to see, that isn't already delivered? Drop me a line, at mizar64@gmail.com,
and thanks in advance for your input.
I should have V 4.28 in Production by tomorrow morning.
In this version, I've extended the normals dataset to include most things that can be captured:
Relative Humidity
Dew Point
Sunshine
Barometric Pressure
Wind Velocity
Wind Bearing
Visibility
The normals are calculated for daily, monthly and yearly intervals.
To do this, I've had to modify a number of routines, including read/write and two different types for normals recalculation. The simplest part, by far, was adding new
template tags to output the new variables. That took all of 45 minutes, including testing time.
This should also fix a bug whereby the system has been crashing every two-to-three nights, plus help me in diagnosing the root problem. So I'm foregoing the usual wait
until Sunday. I just want to test it overnight, and then we're good to go.
Turned out that rejigging the templates for the location-lookup feature took about an hour to complete; so, I've done some more work on MCH.
I've restored an old function that was disabled for poor performance: capturing forecast terms. When MCH encounters a forecast term it hasn't encountered before,
it adds it to a database of terms. The idea is to come up with a list of terms, and then have the Station Operator specify which graphic files correspond.
Back about four years ago, I disabled that function, the database being more-or-less complete; and there was a bug I just didn't want to deal with. It's now re-enabled;
and, this time, it's fully debugged, and it works!
Also added this week:
Precip and sunshine values should now reset to 0 after the 00:05 (Standard Time) page regen.
I've added a new directive: RELOAD-FTERMS. This forces the reloading of the forecast-terms list (fterms.txt) immediately.
Query handlers have been updated so that they always give a response.
I fixed a bug affecting forced translations to Standard Time.
Having established that Master Recalc worked for any location on the test server, I've enabled the same on the production server. It will be fully functional
after the 0500 reboot tomorrow.
So, for the Sunday-morning reboot, we'll have V 4.26 ready to go.
As usual, I'll update this post through the week as I do more.
I froze development at V 4.24 and immediately put it to Production.
First off, I have finally—this time for sure!—slayed the problem with past-year recalcs not sticking. It turned out that the recalculation routine wasn't correctly
marking years as 'dirty' and so needing to be rewritten. As soon as I fixed that, the problem went away, as demonstrated by several restarts of the system with the values
persisting. I tracked it down by way of the logs, which consistently showed a value loading as Nil and not being rewritten after recalc. I kept thinking I had it fixed,
as it would often survive overnight tests. But eventually the previous year would be read again--from scratch--and the old
values would come back. Be persistent, I guess, is the lesson here; and I was.
I've also fixed a bug that was preventing the precipitation unit of measure from being appended to the value on demand. Turned out to be a little-used function that wasn't
returning a value. I tweaked a couple other things while I was at it. I believe I've now fixed all the testpage bugs I noticed.
That will probably be my last development work on MCH for a little while. I have much to do now with the website, like finishing getting the new set of page templates
completed for the location lookup.
No sooner do I issue a proclamation about testing the shit out of new features, than I introduce a production version with bugs.
It seems that in the current version of MCH, the location-lookup feature, just introduced, has a problem with some locations. I believe this is to do with
not honouring lockouts for some features for some locations (i.e. maintain records yes/no). The reason the bug was allowed through was simple: I keep a subset
of the locations on the test server, simply because (a) it's a large-ish dataset and (B) I don't want to be copying that much data about.
So, there'll be some downtime this week, as I probe which locations are causing the problem, and why, and fix it.
Note: I've solved the bug--only took about five minutes when I looked at the logs and symptoms and had a good think; and the bug was in the data, not the code.
This time I was open to that possibility and so spotted it quickly. I've restored access to the Location Lookup page after testing with all 29 locations.
I've also updated some graphics pointers, so forecasts calling for "Cloudy periods" will now display a graphic.
I've also managed to restore a relative humidity reading (you might have noticed lately that the sensor would quickly drop to zero if the humidity got too high).
Version 4.20 of the Weather Server software, MetCommand-Home, is completed. Among the goals for this version were:
Project: to expand the forecast periods from 8 to 12-13
DONE - Add a feature, controlled through the system property CheckSRSSAgainstSky, that converts 'Sunny' into 'Twilight' when not daytime according to
sunrise/sunset, and 'Twilight' into 'Cloudy' when not within an hour before sunrise or an hour after sunset.
DONE - Modify mcXML, the capture tool, to accommodate 12-13 forecast periods.
DONE - Make the forecasts engine work with 12-13-item forecasts (up to seven days and six nights). This will require changes to the forecasts engine,
the I/O engine, and the Templates Preprocessor. It will also require some changes to the website itself.
DONE - Change the template processor slightly, mostly in the way it determines whether forecasts are valid, and to allow up to 13 forecast periods.
DONE - Modify the website to support up to 13 forecast periods. Ugly in the meantime, but it works.
Project: to allow a standardized set of pages to be processed for any given location.
DONE - Modify the query interceptor, wxQueries, to allow for a "location" query, specifying a location code, and to append to the query filename a 40-character
identifier, allowing multiple individual queries to be responded to.
DONE - Modify the XML parser to allow for an additional query, "location," and to reply with the same 40-character code as in the request, with the path of the main file
just generated.
DONE - Modify the templates engine to allow a given location to be rendered from a specific set of pages (very easy).
DONE - Modify the website, adding a new query page: "Location". The page will take the parameters, transmit them to the query interceptor, and wait for a response.
The response will contain a link to a newly-rendered set of pages, which in turn link back to the main CTO site.
DONE - One-off: Invalidate warning stand-downs after xx hours, configurable through the System Property InvalidateClrWarnsAfterHrs (default 2). Will require changes
to System Properties and the I/O engine.
Additional Things Done:
Fixed an emergent bug where a typo was affecting the name of getexterns.
Fixed a bug where banker's rounding was affecting the outcome of number-rounding operations. Brewed my own number-rounding routine.
Incidentally, we're well into our sixth year of automated operation, now; and since approximately Xmas, I can't think of a single time mcOCR has made a mistake in
capturing a data item. That's not to say that a couple of errant spiders haven't cavorted across the field of view, once or twice; that, very occasionally, a datum is
captured exactly as it transitions from one reading to the next; or that network outages or outright power failures (we had one last week—first one in many years)
haven't cost us a very few readings. But very few indeed!
Also, when I say a feature has been added, I now mean 'added and tested thoroughly'.
I'm working on a new version of MCH and have hacked through my bugs lists to where I'm running into more and more nice-to-have stuff. So, I'm steadily fixing bugs and adding
features, and by Saturday I'll be ready with a new version, 4.10.
Notes to Date:
I've added to logger a function that returns the first unused slot in the roster of up to five logfiles. Going through this function to obtain a log handle is now the
officially-approved way of dealing with logger. I've also added a function, GetStackTrace(), that generates a stack trace from logger's point-of-view.
Any open logfiles are now automatically closed and archived after exceeding a certain (user-definable) number of lines.
I've found a way to limit resorting of the events list by adding a new events tag. When issued, the events will immediately be re-sorted on that tag's particular
parameters. This ends repeated sorting of the list every time an event-tag comes up. I've also fixed an emergent bug that caused anniversary info to show up unsorted, in a
block, in the yearly event listings.
I've added a Logs folder to the weather root. It is now the default for logfiles.
I believe I have fixed a bug which prevented Annual Monthly Extreme Maximum and Minimum Sunshine values from being correctly calculated. We'll know at the end of
the month.
I've also implemented a fix which I believe resolves Annual Daily Extreme Maximum Sunshine not begin calculated correctly. We'll know at midnight. I found some errors
in the hunter-killer routines, patched them up, and perhaps that'll help, as well.
I've diagnosed and fixed a bug which was discarding previous current-condition results, leaving the .ini file empty.
Lastly, I believe I've finally conquered a bug that was keeping daily extremes for the month from showing up, much like with extremes for the year.
I'll update this list during the week.
Also, there will be some downtime Sunday morning, tentatively scheduled for 30 minutes, at 5am or so. I have a new version to roll out, and that includes adding directories,
changing some file templates, and recalling a copy of the YOW location's current-conditions file from backup.
I've finally solved a bug that's been bothering me the past couple of months. Long story short: as of the Sunday-morning reboot, we'll be running with MCH v 4.08.
The most important point about this release is that the monthly and annual daily extremes are now recorded properly again, as should become apparent with a quick
recalc. The issue was a problem with how I was
handling dates in one tiny corner of the code; and I had somehow transposed a couple of pairs of variables' order in the sequence, between reading and writing.
This, of course, threw off other variables. Fixed now. Incidentally, the new logger (more on that in the next paragraph) made tracking it down much, much easier.
Among other new things in this version, it moves resetting the precipitation and sunshine totals to zero until after midnight (i.e. will not now read zero at the
23:59 (ST) reading, processed 00:00 (ST) the next day). Additionally, I have beefed up the logger unit to support up to five logfiles concurrently. That should make
debug and code management much easier. I've also added On/Off control variables so that the software can control when log outputs occur. On the Events side, we've
cleaned up the code and fixed a bug where anniversaries weren't sorting properly.
I've also completely rewritten the data engine in this version. It's now much more efficient at handling data and recalculation, intelligently recalculating only what
it needs to at any given time.
I'm still trying to calibrate my new webcam. It's turning out to be a bit of a challenge. In the meantime, sky results may not match the actual sky.
The next steps, as I see it, are:
to organize the logfiles;
to add a [mandatory] pre-sorting feature to events, eliminating the need to re-sort at each tag access; and
to add a feature whereby current-condition reports of "Sunny" will be automatically converted to "Twilight" between sunset and sunrise.
I have further plans, but those will do for now. As I implement each one, I'll note what's coming up next.
I wasn't satisfied with just the scripting ability in mcSky, so I've coded a debugger, too. Everything works and has been tested. Took me a whole eight days to plan and code
the whole thing. Not to brag, but I think to implement a language of any type—and a debugger—in barely over a week's coding time, (including debug time)
is pretty damned good.
Here's a view of the debugger in action:
As you can see, double-clicking on a variable in the display window opens up another window displaying all the internals for that variable. It is possible to control the
execution speed of a script, pause and cancel, and even enter single-step mode. There are no breakpoints or watchpoints; what would be the use, when everything's happening
right before your eyes?
With this release, I've upped the version number to 1.41.
This should be my last major work on the program for awhile, unless I decide to go ahead and implement SkyScript v 2.
I've been working on mcSky, the sky-condition-recognition program. A new version is all but out the door.
Recently I replaced the weather webcam. Naturally, it has different characteristics from the earlier cam, and I was trying to recalibrate mcSky. In modifying the
program, I had a sudden flash of insight: why not find a way to allow the determination of a condition to be done externally, via a script? Then adding a new camera
becomes as simple as updating a script.
SkyScript is the answer to that. It is a limited-syntax language that gives you all you need to determine the sky condition on your own.
It follows a few precepts:
A variable declaration consists of the word 'var' followed by a $VariableName (e.g. var $Total)
A variable reference consists of a dollar sign ('$') followed by a variable name. e.g. $avariable. Variable names are case-sensitive. A variable must be declared before
attempting to use it.
Several variables are predefined for you, including values relating to the sweet spot on the image: $red, $blue, $green, $bright (total brightness), and
$sky (a variable for you to return your interpretation of the sky condition). 255 variables permitted in total. Variables are loosely typed into String, Integer,
Decimal and Boolean types based on their inputs. SkyScript does its damndest to translate between types.
A string constant takes the form "constant value"
A numeric constant consists of the numerals and punctuation necessary and can be either decimal (e.g. 27.762) or integer (e.g. 7)
A boolean constant is the word True or False
An argument is a $variable reference, a string constant, a boolean constant, an integer constant or a decimal constant.
An expression is an argument, optionally followed by an operator (+, -, *, /, % (modulus), & (string concat)) and another argument.
An assignment is a $variable reference, followed by an equals sign and an expression. (e.g. $name = "Alice") (e.g. $byword = "Ken"&"&Barbie") (e.g. $total = $total+1).
A comparison operator is (<, <=, >, >=, <>, ==, %<, %<=, %>, %>=. %<>, %== (string equivalents))
A conditional consists of the word 'if', followed by a variable reference, followed by a comparison operator and an argument, followed by one or more instructions,
optionally followed by the word 'else' and one or more instructions, ending with 'endif'
A log entry takes the form log(expression)
No instruction separators; one instruction per line
The above rules yield a script that might look like:
# Demo Script - Results not Reliable
var $RGvB
$RGvB = $red+$green
$RGvB = $RGvB/$blue
log("$RGvB = "&$RGvB)
if $bright > 117500
# Daytime
# Two choices - Sunny or Cloudy
if $RGvB < 1.92
$sky = "Sunny"
else
$sky = "Cloudy"
endif
else
if $blue > 3500
$sky = "Twilight"
else
var $q
$q = "Clear"
$sky = $q
endif
endif
I'm really pleased that I was able to implement SkyScript 1.0 in under 1,000 lines of code and under three days' planning and coding time. Writing the code was easy;
but debugging took up at least equal time. Still, the bugs were, for the most part, straightforward and easy to find.
I'm just gathering some final data to recalibrate for the new camera and should have the new version up today or tomorrow.
Already looking toward the future, I'm planning an eventual version 2.0 of the language, which will introduce user-defined methods, loops, and subscripted access to system
data (allowing access to all nine image sectors, for example). However, that's a little way off and damned near overkill for what the script actually has to accomplish
(namely, to load a string value into $sky).
One of the really nice things about the implementation thus far is that it's all wrapped up in one self-contained unit. The other thing is that, though it's embedded in a
program that's supposed to be doing something weather-related, it's quite a general-purpose language, which means it would lend itself to other applications. I can see a use
for it in mcOCR, for example, as a way of allowing user methods to make the determination whether a pixel-group is 'lit' or not. I love re-useable code!
I don't think the latest version is going to make it over the finish line in time. But we'll see.
I've been working steadily on the code, improving it here, fixing it up there, throwing in some comments. Going through it in general, I've noticed that it performs a fair
bit of unneeded recalculation. For the past week I've been working on the data engine, to make it work much more efficiently. I've just gotten days to work, and if I can
fit getting months and years to work, as well, into my busy schedule, before Sunday 05:00, there'll be a new version, which may kill some added bugs.
The release in the works, V 4.04, has resolved a number of bugs around recalculation of certain data. I've added Anniversaries, with a begin date, to events and added some
embedded tags to access pertinent data (Original Date, Original Year, Years since original year). This changes the format of Config/factsfile.txt.
I've been working on a test page, something that executes each and every embedded tag in MCH's vocabulary, as well as showing off the suffixes quite clearly. You'll get an
idea of just how massive a dataset I'm maintaining, here. In the meantime, I'm giving the MCH documentation a thorough going-over to get it updated and have implemented a
procedure to make this an automatic response to every change in the program. I've got to get the documentation finished, or I'll never get this growing baby
open-sourced!
The MCH version is up to 4.02. This version fixes several emergent bugs from my fixing a few other bugs earlier last week. I've added some debug logging to help
me track down some (re)calculation errors.
Incidentally, since early in the new year, mcocr, which captures local data from a LCD display, has been averaging about 1 error every 3 weeks. The errors at this point
are all caused by things I can't correct for--image too dim, datum in the middle of changing when the image was snapped, spider crawled in front of the camera. It works out
to an error rate of approximately 1 in 54,000, or 99.998%. I'll soon have numbers to back up my claims, with a new feature just introduced, whereby MCH produces
a Shutdown Report, including performance data, just before shutting down.
I swapped webcams with the front door yesterday. The existing weather webcam was prone to delivering images with a blue cast to them, inflating sunshine values. The new cam,
while not so great, resolution- and focus-wise, is at least capturing colours better. And the old one is doing very well indeed in its new location, monitoring the front
door.
The latest version of MCH has been released: 3.98. This fixes a few bugs and makes the display of temperatures consitent across the board. By default now, all temperatures
are displayed with a leading plus sign if the value is greater than zero and less than 5.1. I've updated the JavaScript conversion routines to honour this default or let
you select another (Never, or Always).
I've added a few new features, including a brief shutdown report (which may in future include emailing that report) that will be controlled by a System Property variable;
and I've added a software version date tag; you'll see that on the Diags page.
I've been keeping busy by squashing bugs. I've killed quite a few of them lately. Found a few bugs in how times and dates were handled, have a list of data elements
that didn't get updated properly in the recalc, have another list of bugs from designing a test page (which implements all known tags and Suffixes).
I had an unexpected crash of the weather system this week (it choked on an invalid tag), and since then I've been periodically stopping it, to run new versions.
For those keeping score, we're up to V 3.92 of the core software, MetCommand-Home.
I'm beginning to see the light at tne end of the tunnel. I have fairly detailed plans for the next few months and have an idea of the time required for them. I can
say that before this year has ended, my wish-list of features will have been gone through and either trashed or implemented; my list of bugs will have been worked through,
all the support programs will be operating just as I like, and I'll be able to put up my feet and say, "Okay, wise guy; how do you top this?" It'll be a nice problem
to have, and I can assure you that it won't last for long.
Literally, I had one of those moments tonight. A couple of years ago, I was working on the Master Recalculation routine. It was a bugger to write, and I'm not proud of the
long, drawn-out procedures, but when I had finished with it, I felt it was ready to go. Well, I did a recalc, up to a date in, oh, early 2015, and everything seemed to go
well. However, as the year went by, the prior year seemed to get out of whack with what was being observed. I assumed some data loss had occurred, hesitated to test the code
again, and then took a break from the weather system; I had other things I wanted to work on. Still do. Still am working on them.
Over the past week, I've gone over my code with a fine-toothed comb. Nothing I can find in there suggests that real data loss should be occurring. Looking at the dataset
itself, I could find no evidence of it, either. So, tonight, with a
devil-may-care attitude (helped along by attendance earlier this evening at a memorial to a good friend who recently committed suicide), I held my breath and did a master
recalc. This basically causes MCH to go over the dataset and recalculate everything, from daily statistics to yearly; normals; extremes and records.
And--voilà--the damned thing just worked. Exactly as expected. I realize in hindsight that the culprit was a date bug, since fixed, which simply caused the wrong data
to be attributed to the wrong year. So, like I said in the title.
I've bumped up the version of MCH as of the Sunday-morning reboot; it now stands at 3.86. This week, I've added support for template tags to refer to alternate locations by suffix, in loops. This
greatly simplifies certain aspects of the website, such as the diags page, which can now be compressed into a simple loop for the individual location listings. And, I've fixed
a bug that allowed "+0" and "+0.0" to appear.
This week, I'll be working on the recalculation routines. It initially looked like they were losing data on me, but now I'm not so sure. The best I can do is eyeball the
routines, to see where data could be lost, run them again, and examine the dataset. The dataset itself now looks fine, so I'm puzzled as normals recalc is giving an anomalous
reading for one year. I'll have to dig into the debug log for that.
As you can see, I've continued to update the website, especially in getting table items flush-right and working on the unit-conversion routines. I'm about halfway there on a
routine that'll handle conversions withhin the forecast text. I won't ever be able to do anything about the graphs and maps. For general display items, though, it's working
fine. I'm also adding in spots for normals all over the place, and color-coding them green. I'll be drastically expanding the normals dataset as soon as I get the recalc thing fixed.
While testing the new JavaScript overlay tonight, I happened across a bug applying to FireFox browsers only: the maps at the bottom of the Current Weather page weren't
working. Oh, they displayed just fine, but the hotspots on the maps, places where you can click for more, disappeared.
Easy fix, mind you.
This apparently has been a problem from the beginning. I don't often test in FireFox (I suppose it's another of those things that I _should_ do). Anyway, if you notice any
irregularity, please flag it via an email--and thanks so much for doing so.
I have a new version of MCH running; you'll note that we're up to v. 3.84 now. Mostly small bugfixes (loops were not working properly when the loop was only one element long);
and I've revamped the suffixes for consistency with the new JavaScript overlay I'm applying to the site, to allow your chosen units of measure to remain consistently
locked in. I've also recently tweaked mchtml, to collect and calculate full precipitation stats.
Incidentally, the conversion of pages to the new JavaScript overlay is going much faster, now. I've written a program to largely automate the process. It's going to save me
weeks of hand-coding. It's taking about ten minutes per page, now, rather than about an hour. You see, for every statistical-type value displayed, I had to hand-code a
whole <script>..</script>/<noscript>..</noscript> spiel; same for the unit indicators. Now the program sweeps through and automtically does all that.
I've begun work on a new project, StationBase (you can read all about it through the Home Page). I'll still be working on the weather server,
though. I've some stuff to clean up, and a number of interesting ideas for new features.
I'm adding a new feature to the site: configurable units of measure. From a specific page, you select the units of measure you want to see, and from that point on the site
delivers readings and stats in those units. I'm taking my time, but you'll slowly see the feature spread from the home page to the other pages on the site. There's a fair bit
of javascript involved. Note that the weather system itself can generate stats in these alternate units; the difference here is translation on-the-fly.
