WORKING NOTES CURRENT PRODUCTION VERSION: 4.82 Sprint 1 Fix 151 Bug 205 Feat 163 Website 107 Doc 100 mcXml 101 Omission 101 Logger 102 Aux 104 Obs 100 mcFCT 100 wxhtml 102 Implemented; Pending Testing: ----------------------------- Active / Confirmed: ------------------- [Sprint 1 (2019-01-23)] [Feat 163 (2019-01-23)] Introduce two more graph-output files: - wxg_dp.txt: daily, last year - wxg_mp.txt: monthly, last year Relatively straightforward. Implemented 2019-01-25. [Feat 162 (2018-12-30)] Allow an -ow suffix on statistics packets, in which case null values will also be propagated. Imlemented 2019-01-25. [Feat 156 (2018-12-23)] Implement binary search in events May include adding into in-order insertions [Feat 161 (2018-12-27)] Make it possible for specific months to be accessed. Perhaps new tags. <@AMEq 2018 1> Set This Month references to January. <@AMEq 2017 2> Set This Month references to February last year. <@AMEq 2019 3> Set This Month references to next March. <@AMEq A 1> Set This Month references to January. <@AMEq L 2> Set This Month references to February last year. <@AMEq N 3> Set This Month references to next March. Will probably take some work and a new variable to pull this off. [Feat 157 (2018-12-23)] Implement binary search for readings May include adding into in-order insertions [Feat 158 (2018-12-23)] Implement binary search for UserData May include adding into in-order insertions [Feat 159 (2018-12-23)] Do some work to speed up the templates processor--speed up page regeneration a little. Try using more ShortStrings. [Feat 160 (2018-12-23)] Start looking into why page regens are slow; benchmark individual page regens, then individual calls during the regen. Find out what's causing the slowdown. [Fix 151 (2018-12-23)] Wind-bearing averaging scheme is not working. Come up with something new. Given: A wind velocity and a wind bearing Calculate the x vector of the bearing Calculate the y vector of the bearing Calculate the vectors of the new bearing Average the two vectors, according to relative wind velocities What About: Calculate a wind rose, based on four, eight, sixteen or 360 points. Then calculate the vectors of each direction, in turn adding each one to the whole [Aux 103 (2018-09-17)] Build an auxiliary tool (GUI) to automate certain tasks: - Locations Maintenance (Add/Delete/Change) - System Properties maintenance - Events maintenance - Utags maintenance - Ini maintenance Commenced 2018-09-19 [Feat 149] Add way to sort some list results - Locations, warnings Multiple criteria [Website 107] Update the General Query helpfiles; new tags Modernize and retrofit the airport website. 2017-07-14. [Fix 122] Have recalc routines: Skip first _year_ of norms multi-year recalc if starting month/day <> 01/01 Skip final _year_ of norms multi-year recalc if ending month/day <> 12/31 Skip Extremes, Records for final _month_ of multi-year recalc if ending day < daysinmonth(year,month) Add and obey DoCalcNormals.SkipFirst, SkipLast and DoCalcRecords SkipLastMonth, (plus all vars for DoMasterRecalc) * Have recalc routines handle precip and sunshine differently for more-accurate figures. This will affect: - DoCalcNormals() - DoCalcRecords() - MasterRecalc() [wxhtml 102] See if you can download a file only if it's been changed. - Apply to all: html, XML Would really cut down on traffic, inputs [Feat 148] Separate Extremes from Statistics Same handling as with Normals [Feat 138] Consider adding other statistics: Live: - UV Index Daily: - Snow on Ground - Heating/Cooling degree-days Annually: - Growing season (user controls the params) Search period, min. acceptable length, max acceptable length - Longest drought (user controls the params) Search period, min. acceptable length, max acceptable length - Longest heatwave (user controls the params) Search period, min. acceptable length, max acceptable length - One MF (multi-) routine for everyday use, and add to Master Recalc - Related Records and Normals Add frost-free season calculation (days) Add a FrostFreeSeason Datum (LongInt) to year structure Add a routine to calculate the FFS Add ARcAFrostFreeDays (LongInt), ARcAFrostFreeBegin (Date), ARcAFrostFreeEnd (Date) records Add ASFrostFreeDays (LongInt), ASFrostFreeBegin (Date), ASFrostFreeEnd (Date) to YearRec and initialization routines Add ANFrostFreeDays (LongInt), ANFrostFreeBegin (Date), ANFrostFreeEnd (Date) to NormalYearRec and initialization routines Will affect a number of recalculation routines Will affect ReadYear, WriteYear Add code to ReadYear(), WriteYear(), ReadRC(), WriteRC() Add code to wxTP and new tags (@ASFFD, @ARAFFB, @ARAFFE) - The nuts & bolts of it: Maintain ATmpFFStart, ATmpFFEnd, ATmpFFDays Update those from daily if ATmpFFDays > ASFrostFreeDays then transfer over and zero the temporary values [Feat 137] Consider special codes for further values for SmallInt/longint arguments eg -30001 = Trace precipitation -30002 = Maximal value exceeded -30003 = Nil value but show as zero -30004 = Value outside of bounds Code several NotNil routines to test against all possibilities. Easy test: if value < -29999 then NIL [Website 103] Add routine to conversionRoutines.js to convert forecast terms embedded within text. (High nn,Low nn,nn km/h,nn mm, nn cm) See jupiter:~/Documents/Forecast Words.xls 2017-03-17 [Feat 120] Allow master recalcs at year-end Think of other such tasks [Feat 144] Separate annual stats from monthly/daily. It will greatly cut down on disk accesses. Add read-write-months routine Make months subsidiary to a year - year must always be open first Make that clear with the call: LoadMonth(M) Need new objects in the location record MB : array[0..3] of shortint; MM : array[0..3] of shortint; Need a one-shot conversion routine or program Read Year alternate Write Months Write Year new from then on: Read Year new Read Month Write Month Write Year [Bug 118] Investigate bearing-averaging procedure; don't think it's working. New Formula (NO): Take the sum and mod 360. Then average the result. [Website 104] Finish the Test Page 2017-05-08 [Doc 100] Complete the Documentation - 2017-06-16 SHELVED ITEMS [Bug 165] Humidex yields low values - not working properly. Investigate formula used. Formula is fine. [Obs 100] Understand the midnight-n/a precip problem better. Both the precip and the sunshine total are daily stats, not current. Daily data tags always default to the actual date; they are therefore grabbing 'tomorrow' from the data's viewpoint. Not sure there's an easy fix for this one. RELEASE - Reconstitute the YOW dataset - correct the precip measurements - Complete template zones (i.e. local persistent values) implementation - Dataset Efficiency (ongoing) - Try switching longint data to smallints - Custom time variable - longint Minutes 0..1440 per day Year value as an 1800-based word Time, Date, DateTime can all be longs and save 4 bytes each Check for routines doing rollyerown stuff - Further modularize - Subdivide data into common modules - Incorporate use-or-not flags - CSV's start havng format stated at TOF, even if in a comment - Start using configuration options for user-task scheduling (offset) - Then check every minute. - Must add vars to each Loc to track next-time-todo - Double-check data-kill routines, test thoroughly - Extend range to any date - Create support programs for some jobs - Bulk data acquisition and import - Activity reports - Dataset maintenance - Clean up means calculations or implement 'official' figures with rulesets - Add system property: MeansRuleSet=Official|Alternate - This will affect both means calculations and month/annual means calcs and queries in wxio - Allow user to specify interval for tasks (in addition to offset) - Toggle whether to save or delete snapshot files - Complete the tagset implementation - Rudimentary temporal math - Way to access previous periods in greater detail - Add tag suffix allowing offsets to date components - Fix for WxOCR: - Fix memory leak - Add data-blending: borrow stats from another location - Tweak for wxxml: - Record date/time of each item (obs,fc,warn) and generate new input file only if each item is new - cuts down on data input - cuts down on file accesses - no data, no access - Add system data to nightly backups - Automate daily statistics capture - Add to WxXML - Look for - Find >Max< - Find °C and grab back to the
  • = DTeMx - Find >Min< - °C etc = DTeMn - Find >Precip.< - mm - could be 'Trace' -
    Normals
    - Max and Min - These are round integers and of limited use for local areas - Sunrise, Sunset - Also capture almanac data - Easy enough to sift through the file with the right string searches - Use simple scripting to define what to search for - Start capturing now and use later - Automate summary readings capture - ADD TO WxXML - Lower precision, don't even do stats - Daily Facts and Reminders feature - See Config/factsfile.txt - DST information should be made separate for each location - Implement custom averages calculations - Calculated over customized timeframes - Custom validation criteria (e.g. min. number of values per period) - Data smoothing - Add data export - mostly through templates not code. :D :D :D - Add custom reporting - see immediately above - Add text-file inclusion during tag processing - Would allow pregenerated reports to be included, especially tables - Would cut down on realtime processing - Add weather-warning processing/logging - Develop data-interchange format or program <=> WxTrack - Probably transplant WxPro engine to WxTrack - Add data controls; i.e. delete or recalc periods, etc. - Add command-line parameters and related flags/options - Recalculate period - Kill record/period - Graphical control/monitor panel - Or share dataset with WxTrack - Expand to multiple normals sets - Add archival hourly YOW data - automate it! - Add CGI support for custom queries - Looks un-difficult - Remove daily precip normals; calculate from monthly EXPANDED DATASET: DAILY Snow Depth UV Index - High, Low, Mean Max Gust Direction Heating Degree-Days Cooling Degree-Days Civil Twilight - Begin, End, Duration Nautical Twilight - Begin, End, Duration Astronomical Twilight - Begin, End, Duration MONTHLY Average snow depth Average UV Index Heating Degree-Days Cooling Degree-Days Frequency of precipitation Normal sunshine Precip days Sun days YEARLY Growing Season ######################################################################################## COMPLETION LOG (Most-recent first): ######################################################################################## 2012-07-16 Fix average-wind-direction/speed calculation 2012-07-10 Start using software name and version numbers properly 2012-07-09 Experiment with embedding tags into JavaScript code for on-the-fly stuff - Temperature conversions - Math - No need for a built-in scripting/math language! - Works like a charm. Tweak Events sorting: only sort automatically on conditional statements - Ex suffixes get copied to var LTO to be persistent - Changes are cumulative - Changing a parameter triggers a re-sort - Add new flag ER (Resort flag) - Set when Ex suffixes processed - Triggers re-sort on next Data Tag access Add system tags to loop parameters #EVENTS #LOCS #FCS #WARNS - Can be used in place of a numeric parameter in Loop declarations 2012-07-08 - Expand Events capacity to 2,999 - Only load events for P-A-X years. - Reassign Events '#days' parameter as 'priority' parameter. - No change to existing sourcefile - Add new tag suffixes for 'threshold' parameter(s) - Qualifying priority must be >= parameter (default 0) or within parameters - This in effect creates event categories and groupings in addition to priorities. - Must modify existing parameters for this to work - New Suffix structure: :E - Event-specific suffixes L [-]nn - #days forward/back(negval) to look for events (default 14 days) (only applies to Current Event searches) (Priority Threshold Values:) ' ' is maintained for legacy purposes E nn - (Equals); 0 < nn = datum (internally, sets NTh and XTh to nn) (equivalent to ":ER nn nn") N nn - Minimum threshold; nn >= 0 (datum >= NTh) to qualify (disables XTh) (DEFAULT CONDITION) X nn - Maximum threshold; nn > 0 (datum <= XTh) to quality (disables NTh) R n1 n2 - Range: specify NTh XTh; 0 <= Nth <= (XTh > 0) - Defaults are: ETN=0 ETX=-1(disabled) = all priorities > 0 (NTh=-1,XTh=-1) shall not occur 2012-07-07 - Fix date-handling - having trouble converting from GMT on input - Maybe check how WxXML does it - THERE WAS NO REAL FUCKING PROBLEM... - Next time check the source inputs more closely. 2012-07-07 FORKED FROM 0.985 - !!! Convert Event times from UT on loading - !!! Scan for Input/RELOAD-EVENTS and act on it - Start forking new versions Backport bugfixes to release version 2012-07-06 - Implement Loops - @R tag (Repeat) - Pass start, end values and optional stepvalue (defaults intelligently to -1/+1) - Can step low-to-high or high-to-low - All code within <@R>/<@R> block repeated for all values in the range - Code is then parsed per usual - Excellent for indexed data such as Events, Forecasts, etc. - Very powerful in combination with conditionals 2012-07-04 - Implement upcoming events - Global only - Tags to access Events - Two types - Upcoming Events - @E - Events E! - Conditional block EC - # of active Events (max. 255) E1..255 - Event #n ! - Conditional Block D - Date T - time H - Heading X - Text MD - Days to go MH - Hours to go MM - Minutes to go C - Colour Code Extensions: : - Within nn days - Events per Time Period 2012-06-24 - Implement weather warnings - Modify WxXML to collect and export - Simply log; don't retain as stats - Add .ini options to configure output colours - Add tags to display info; same as forecasts 2012-06-19 Added weather-forecast terms and graphics - fterms.txt: new file in main folder - Manually-updated copy of extern/fcterms.txt - New graphics in Graphics/currcond/ - Add forecast processing/logging - Add some parsing to get forecast values for temperatures, winds, precip. - Make these elements available through tags. - Think about some fairly simple AI to learn how to parse the forecast text completely--it's computer-generated and clearly follows a formal but flexible syntax. Some simple categorization of each 'word' in the text (value/action/condition/exception/additional/probability/timeframe/etc.) coupled with frequency-pairing analysis would help it figure out the syntactic rules, vocabulary and lexicon. Could be fun to teach it to say "It looks windy and wet on the way home tonight," or "Salt the walk tonight and be ready to shovel in the morning." 2012-06-18 Added short text summary to forecasts dataset (wxxml modified also) - Now logging forecasts in monthly folders for future analysis. 2012-06-17 Implement Current Conditions icons - New format for current-conditions.ini - New handling routines 2012-04-01 - Add forecast expiry after 16 hours - Add forecast and latest-readings validation during scheduled tasks 2012-03-28 - Add new locations YTR - Trenton TNK - Brent TKG - Kingston YPQ - Peterborough TBO - Brockville YMX - Hawkesbury (Mirabel) YND - Gatineau 2012-03-27 - Implement timing coordination between wxpro and external processes - Avoids file and resource-contention issues 2012-03-26 - Move raw readings logs into tiered files - Break into monthly logs, to keep sizes down 2012-03-25 - Clean up tag and time-date processing - Clean up all output routines - Make the output interface much more formal - DST correction everywhere it's needed - Add zone processing - Cleaned up handling of current-day time values - stored as time only - loaded and processed as date+time 2012-03-24 - Fix for WxOCR: - Add output buffering. Write obs locally if network path down. Leave a non-active file in there as a beacon; check for it before trying to create a file. If not there, save it locally and try later, or run a periodic watchdog script to try to copy the files. OR, buffer 'em all and write 'em to one bulk file when path available. - CGI Input Portal - Eliminates file-copying over the network for external sources - Simple format similar to WUG/PWS - Program to capture data, leave input files for WxPro 2012-03-23 - Implement system-wide pause - 04:57 to 04:02 - all participant scripts and binaries to respect this schedule - Add templates-regen-freq: # minutes to regen templates if no input - Add scheduler - Make it so it works in CLI or daemon mode - CLI mode is simply one loop through everything - Make scheduling configurable - Switch to live operation on the server - Test for several hours through a Midnight - on Venus - Could be toggled with an ini option - Just keep checking for inputs - What would be necessary? - Regular shutdown and crontab-initiated restart - Flush everything after delta - Takedown mechanism - SHUTDOWN - Scheduler: - Scan for inputs - Flush data - Templates - Data output - Housekeeping (e.g. new-day activities) - Schedule by time, not by interval - 10-second timing resolution - Every 60 seconds - Catch up on missed jobs - Rough hierarchical schedule: Every 60 seconds - Scan for live inputs and process - Export readings-based data - Export day-based data - Scan for stats and process - Export period-based data Every hour - Recalc daily stats At Midnight - Delete old-yesterday's Live snapshots - Use a templated script - Recalculate previous day - Export daily data - Export monthly data At Month's End - Recalculate previous month - Export monthly data - Export yearly data At Year's End - Recalculate previous year - Export yearly data 2012-03-21 - Add "live-only" locations - Log the readings but keep no stats - Should actually be easy to do - Add dataset and processing selection - Log data? Yes: - Records yes or no - Extremes yes or no - Stats yes or no - Possibly add daily/monthly/yearly/all-time levels - Output what? - Select types of output files - Templates: simply control via templates.ini 2012-03-20 - Split Location data - Fully user-editable parameters file that is read-only to WxPro - location.ini - Internal data in a fixed format that is read/write to WxPro - locdata.ini - Backup-stack handling + One per year + Save to year of datestamp - Change seek-destroy routine to navigate the new scheme... shouldn't be hard... - Current-year-ony; recalc as needed - Start backing-up more values? - Optimize readings handling Okay, enough! Except during bulk input, the month's raw readings are never needed all at once. - Readings are stored by day, in monthly folders - Max. 720 readings per day - Three readings buffers: Today, Yesterday, Anyotherday - Modify the intelligence to determine readings bufferno - Use one central routine, with overloading to handle different paramsets - Make sure everything else uses it - Separate readings buffers completely from years - Reroot in L[] - HOLY FUCK! 2012-03-16 - EXPAND LOCATIONS LIMIT - Take it to 200 - Backup data just before Midnight - Dataset is not huge; currently approx. 14MB including 80+ years of detailed daily stats from Ottawa Airport, plus supporting files and graph outputs for three locations. - For each location, backup the files in the location's root, plus its Year folder. - Tweak for WxXML Do not generate output with a nil date (i.e. no observations) 2012-03-15 - Add in another location or two - Candidates Kemptville - done - possibly limited hours (airfield) Cornwall - done Pembroke - done Bancroft - done Gatineau (Airport; limited hours. Make a current-only location) Kingston Maniwaki Enhanced wxxml - Multiple locations data capture - ini file specifies locations and paths - Cleaned up XML-parsing routines - Now grabs data according to each item's category, rather than simply blindly catching them in anticipated order and format. Added Regional Temperatures graphic - see CTO template makegraphics.sh.wxtp - CTO/makegraphics.sh called by makegraphs KILLED: - Add records import - No need; just add the file. Same as for Normals, really. Added new locations: Kemptville (XKE), Cornwall (WBZ) 2012-03-14 - Allow post-facto corrections to readings - Specify reading date/time to delete - Remove from database - Recalculate affected Stats - Roll back any Records/Extremes - Will require considerable coding, but most of the code can be borrowed from existing procedures. May only take a couple hours to implement. - Implement backup stack - Keep it very simple - Keep a logfile of transactions - Date/Time itemname oldvalue newvalue olddate newdate - Daily/Monthly/Yearly stats (high/low/etc.) will not be backed up (just recalculate) - This will hugely cut down on the size of the file - Monthly log - Scan first, for speed - Find and restore value, delete from backup log This will require careful review of all data-handling routines, each of which will have to be hooked into a backup-logging system. Luckily, most input data are already handed off to subroutines where any actual changes take place. *** This should be switchable ON/OFF for e.g. bulk input of statistics, where data quality isn't an issue; it's more for correcting bad live data afterwards. Backing up value changes that will never need to be rolled back is pointless. 2012-03-13 Trash: - Tweak input-processing - Bad-data filtering and/or correction - Yes, this should be done at the input level, not capture (No, it shouldn't. Capture should provide clean data.) TIGHTEN-UP DATA ROUTINES - Review every routine 2012-03-11 - Add DST correction for all outputs [ DST STARTS THIS COMING WEEKEND ] ************************ - Maintain Standard Time for all internal uses - Fully implement time zone conversions for outputs - Implement DST lookup table for last few decades 2012-03-08 GRID TAGS - What grids will be output? - Roughly same as graphs - Live Data (C for Current for consistency with tagset) - Last 24, Last 168, Yesterday, Today @GL2 @GL7 @GLY @GLT - Daily Data - Last 31, Last Month, This Month, Next Month, Last Year, This Year, Next Year @GDY @GDT @GDW @GD3 @GDL @GDM @GDN @GDP @GDA @GDX - Monthly Data - Last Year, This Year, Next Year @GML @GMM @GMN @GMP @GMA @GMN - Yearly Data - All-Time @GYA - Try this: - [for now] Grid output is a spew of the complete time period - Define which fields are output - System will output the data in a grid. - User can style it however he likes, within a framework that will be duplicated for each interval in the period. <@GL2> <@G/>
    Tags would be interpreted for each interval as: Time: <@CST>Temp: <@CSTe> Baro: <@CSBP> RH: <@CSRH> This actually should be easy to implement. 2012-03-08 - Tweak User Tag processing - Allow :L suffix to specify location (i.e. borrow macros from other sets) - Scan through for User Tags and replace as found, rather than straight search-replace of values-for-tags (makes location-borrowing above much easier) - User tag now implemented as <#tagname[:L code]> - These tags will obey zone processing when implemented *** REVISE DATE/TIME SYSTEM NO LONGER NEEDED Key Routines: - Current CIVIL time - Return the system clock - Current STANDARD time - Return the system clock, adjusted per DST rules - DST Correction - Return the DST correction for a given Date+Time - Convert Civil to Standard Time - for a given Date+Time; negate DST per ruleset - Convert Standard to Civil Time - fove a given Date+Time; correct to DST per ruleset - PORT THIS FIX TO DEVEL VERSION - Review all DT handling for proper useage 2013-09-21 Add accuracy checking Tags: ZAH: #Hits ZAM: #Misses ZAT: #Total ZAP: #Percentage - Add during ProcessReadings() - Trap to Location 1 only FORK 2014-01-10 - Add UserData - Capture - Tags - Query 2014-02-07 Data-not-extant flag Suffix $ to any tag: - $({PosTag}|-)[${NegTag)] (- means 'process as usual') If tag contains data, return PosTag If tag contains no data, and NegTag is defined, return NegTag else return nothing [ Extend Support script: - Archive webcam snapshots by date, as generated. - Generate thumbnail also, in parallel folder structure ] RELEASED AS VERSION 3.10 2014-04-06 - Make all results available all locations - Separate scanning and generating loops - New flag (DirtyPages) in each Location rec 2015-01-15 - Implement system properties: - ReadProperties - WriteProperties - SetProperty - GetProperty - PropertyExists RELEASED AS VERSION 3.42 2015-01-16 - Get rid of BackupStacks; RollBack() to set to INil only; - Also fix where BackupStack is read INTERNAL VERSION 3.44 2015-01-23 - Added new directives: REGEN_LocCode/ALL and RELOAD-PROPS - Added new data-out tags for expanded records set - Records recalculation. Depends on above item. Use XML and the queries interface. RELEASED AS VERSION 3.54 2015-01-24 - Sunshine fixup (in progress) 2015-01-25 Added data verification bounds RELEASED AS VERSION 3.60 2015-01-26 - Probe why BP not showing up in records - Fixed - Fixed XML handling to AnsiString 2015-02-20 MASTER RECALC: Go directly into live readings - Check against Daily Limits, Records, Normals Rationalize precipitation values - Mark daily valid Monthly: - Check against Limits, Records, Normals, Extremes - Mark monthly valid Yearly: - Check against Limits, Records, Normals, Extremes - Mark yearly valid 2015-02-21 Consolidate statistical information under object Stats 2015-02-24 Small fixes: - Connect up ResetCounters from systemproperties.ini 2015-02-26 When reading in values, only apply them if non-nil Fix bug with forecasts - Read in subsequent forecasts. Don't ignore them. 2015-02-27 Fix bug with records import: NILs overwriting existing records 2015-03-07 Fix bug with startup time - Always add an hour 2015-03-12 Fixed bug with location stats rollover Fixed startup time bug (for now; needs detailed investigation @ end of DST) v3.80 2016-10-15 Add DebugLevel directive in SystemProperties.ini Convert logger.Debug()s to logger.Trace()s 2017-01-01 - Add monthly task: backup maintenance - Cull backups from earlier months - Perhaps leave 1st, 15th, last 2017-01-08 Clean up incorrect 1850 date references Add Tags DNWOTF[C/U/L) and DNTOYN[C/U/L] to time-period Normals outputs 2017-01-14 Re-enabled (I think!) CC terms storage Ensure snowfall etc recorded in daily stats and start using them Make warnings work in loops - they don't, currently Adjust suffixes and expand per the latest list Fixes should not break existing template code. RELEASED VERSION 3.84 Website: Give some thought to a way of making values configurable for the visit (random thought: configuration page, and pass relevant vars to pages through GETs & process to final vals there. (EG http://gypsy/weather.html?opts=",xx,xy,xz" weather parses the options as needed through embedded scripts Bookmark index page that way and the settings would be permanent } 2017-01-22 Allow locations to operate within loops To do this, allow locations to be addressable by #LOOP# [Bugfix] fix bug affecting "+/-" display pf temperature terms. 2017-01-25 [Bug] Now only 1 warning shows when multiple present. Bug was in webcode 2017-02-04 [Bugfix]Fix Recalc and other queries so that they do not rampantly nullify damage realtime data or stats * Restore damaged calendar years' data (may not be an issue (2017-01-22) - cannot now find any evidence of damage) * Update documentation - Not a problem. They weren't broken at all. Fixing the Date-1850 stuff fixed it. 2017-02-19 [Bug] Max Hx Records and extremes not being calculated 2017-02-23 [Bug 107]: @TE1! is documented to work, but doesn't. [Bugfix] +0.0 is currently displaying. 2017-02-24 [Feat 104] Allow the user to specify whether positive numbers are prefixed Always, Within limits, or Never So tag prefix '+' adds variants '+A', '+N' or '+L' for Always, Never, Limits 2017-02-24 [Bug 100]: Sunshine hours not properly tallied up on recalc. First try. [Bug 108]: CT:2 not working properly [Bug 109]: time-based records are mostly not being kept [Bug 119]: The +A tag is not working correctly 2017-02-25 [Fix 100]: User can specify to use plus sign Always, Never, or up to a specified value, when displaying temperatures [Website 104]: Retroactively add highlighting to qualifying elements 2017-02-26 [Fix 104]: Restore use of TNil and DNil. Check for correct use of the terms and nil-spotting 2017-02-28 [Bug 121]: Daily Total Sunshine now reading 00:00 at all times 2017-03-03 [Bug 123]: +A/L/N tags not always working verified working. The issue is with the JS conversion routines. 2017-03-04 [Website 104]: Finish off the forecasts-conversion tools Go through website and ensure all temperatures are displayed with plus-limited [Feat 104]: Add processing of global and local suffixes This will require additions to the tag processor. New tags: ZoG: Set global tag suffixes ZoGR: First refresh default values to the Global suffixset ZoL: Set local tag suffixes ZoLR: same, but refresh the local suffixset from the global suffixset In each case, global or local, the suffixset will continually be modified unless or until a refresh request is received The need is to parse out the additional new tag characters and pass the suffixes on to the appropriate processor *May have to look at the allocation of global->Local->tag; it must be re-examined. Global is never refreshed except on-demand. Done. *Local is refreshed only at the beginning of processing a document and on-demand. *Tagwise suffixset is refreshed from Local for each tag and modified by tag suffixes [Website 102] fix RH Live data output title to "Live Relative Humidity - 7d" and "Hourly Barometric Pressure" to "Live Barometric Pressure - 7d" [Feat 106[: Add a new global constant SOFTWARE_DATE (Build Date) and accompanying new tag to access it (@ZD) 2017-03-05 [Feat 100] Have mch output a stats report at the end of each run. (MCH_Shutdown_{date}@{time}.txt) Will require: - a control flag StatsReportOn=True/False 2017-03-06 [Bug 124] (Emergent) #ECRADAR, #ECNATRADAR not recognized on home page [Bug 117] Have the graphic recognizer optionally put out something like "testpattern.gif" for conditions not recognized. 2017-03-11: [Bug 127] All Dates+Times being misinterpreted upon reading in [Bug 126] TNil not being recognized. All read-in routines will have to convert any instance of '01:23' to TNil; All write-out routines MUST intercept TNil and write 'nil' [Bug 125] (Emergent) Yesterday's sunrise/sunset times are being overwritten with nils from the data feed. [Fix 110] Add Sw, Sq and Pr into the mix when calculating daily totals to zero. 2017-03-13 [Bug 128] (Emergent) Nills are still overwriting yesterday's stats 2017-03-17 [Bug 122] The following tags are not being calculated correctly: * DRcDWBMeMx * DRcDWBMeMn * MRcDWBMeMx * MRcDWBMeMn * MExDWnMeMn * MExDWnMeMnDT * MExDWBMeMx * MExDWBMeMn 2017-03-21 FACTSFILE.TXT FORMAT CHANGES WITH THIS VERSION (4.04) [Feat 106] Expand events to include an anniversaries section Changes to factsfile.txt: [Anniversaries] section Repeat: YYYY-MM-DD[@HH:MM] PriorityRanking Title unquoted string description ended with <@> [End] [Events] section same as above [End] Anniversaries stored in a separate array but sorted into lists with events. Id = 3000+position to tell them apart. Anniversaries are presumed already to be in order, same as for events Each re-sort, work the anniversary data in Will need to add a few embedded data tags to text body - <@O> - Original Date (anniversaries; else returns EDate) - <@OY> - Original Year - <@Y> - Years since Original Date 2017-04-07 [Bug 130] Last month is not being re-run on any locations [Feat 109] Add an additional event tag: YY (current year). Need to add a few lines to wxtp.ParseEvent() Update documentation 2017-04-08 [Fix 111] Take a good, hard look at all the recalculation going on. There must be a simpler way! Ongoing, processing readings through the day, directly, to recalculate max/min/mean etc. Once, at day's end, to recalculate all the records and extremes relating to the day Once, at month's end, to recalculate month's data and check against extremes and records Once, at month's end, to update yearly data (averages and means) Once, at year's end, to check against extremes and records For recalc routines, copy over the code blocks from DoMasterRecalc() Introduce to DayRec a few extra variables: - WBNS, WBEW : Double; // To hold Day-to-this-point Wind bearing information - WnT,WnC : Int64; // To hold Day-to-this-point Wind velocity information // Set Zero before first reading after midnight Ensure that daily precip tracks with same - Eliminates recalculation every time there's readings Introduce to Monthly and Yearly structures, as well. - Use these for calculating wind bearings and averages All are reset to zero before the first reading of the new day Mostly finished @ 2017-03-26. Releasing and will finish up in a day or two [Bug 131] Yesterday not being recalculated in total - missing monthly extremes, for one - Possible fix 2017-04-08: order and nature of temp var reassignment following recalc. Testing at midnight. [Fix 113] Change monthly summary precip totals to 0 at start of month - Possible fix 2017-04-08: inserted apropriate code into UpdateMonth(). 2017-04-19 [Fix 105] Improve recalculation routines, esp. records and extremes calculation, towards Bug 120, 122. 2017-03-18 [Bug 120] The following tags do not appear to be REcalculating correctly: * MExDRHMeMx * MExDRHMeMxDT * MExDRHMeMn * MExDRHMeMnDT AExMSuShMx AExMSuShMn PExBPMeMn PExDBMeMnDT PExWnMeMn PExWnMeMnDT AExDBPMeMn AExDBPMeMnDT AExDWnMeMn AExDWnMeMnDT * ARcDWBMeMx * ARcDWBMeMn * AExDBPMeMx * AExDBPMeMnDT * AExDWnMeMn * AExDWnMeMnDT [Fix 106] Add original calls to updating routines rather than call bulk recalc [Bug 133] The variables AExMSuShMx and AExMSuShMn are returning n/a Believed fixed 2017-04-19 [Bug 129] At Midnight, daily totals read n/a but should still read the daily totals Change position only to occur with a new day 2017-04-20 [Fix 101] MAY NO LONGER BE NEEDED Review midnight routine - Add system property to make time of execution user-definable Add to midnight routine: - check all four precip-type values, INIL := 0 ! [Bug 134] Anniversaries are all inserted one-after-the-other, beginning where the first entry belongs. [Fix 115] Look at ProcessReadings() and finally solve day determination (yesterday/today/another day?) [logger 100] Support multiple logs (up to 5) Will need arrays to store preferences and operational data Will need to pass an extra parameter to specify any log but default (keep original versions; just default them to log #1) [Feat 108] At startup, auto-recalc from the last program run; if d < today(d): d, m, y Obviated by the data-engine rewrite 2017-04-22 [Bug 136] There is a mixup on reading year records. two variables missing. [Bug 135] Daily extremes still not completely recalculating. Investigate! 2017-04-23 [Logger 101] *Add a function to return the next available log slot. Make this the official way to get handles to add logfiles. *Add a procedure StackTrace, which returns a long string with each StackTrace object on a new line. *Add getter and setter for DoLog[]. Ensure it works. *Integrate it with wxpro as a whole. [Fix 116] Find a way to recalculate events once only for each type So only the first reference to an event from a particular time period would cause events to be sorted A tag for events triggering a resort - use at the beginning of that type of access One tag fits all - R (EER, TER, XER, etc.) So: - A tag to trigger presort Respect that tag when parsed - No other time [Feat 110] Add Logs directory Things To Be Added: A LogPath function References to the latter 2017-04-24 [Bug 136] Events are still not sorted properly on yearly stuff [logger 102] Introduce log size management If filesize exceeds a specified number of lines, archive it and start afresh. Will need a filesize specification in SystemPrpoerties.ini. MaxLogFileSize=100000 Will have to write a log-archiving function. Maybe default max to 10,000,000 lines (approx. 500,000,000 chars) Maybe something like logger.SetMaxLogSize(MaxSize : Int64; ArchiveDepth : Byte) Will also need something like ArchiveFile(ArchiveDepth : Byte [Range 1..9]) - will create archive files up to debug.txt.ArchiveDepth [Fix 117] Look through the code and tighten up where locations opted out of things. 2017-04-25 [Bug 137] Daily extremes for the month seem still not to be calculating correctly--dates, in particular [Bug 140] CCStack[] is messed up. Doesn't seem to want to record. Possible fix 2017-04-26. [Bug 139] AExMSuShMx and AExMSuShMn are recorded as n/a AExDSuShMx is recorded as n/a Log it and find out what's happening Believe fixed. [Bug 141] Current conditions are not being loaded properly [Bug 138] Current-condition graphic seems to be out. Current-conditions file has been reset. Investigate. 2017-04-30 [Feat 107] Add in feature that turns 'Sunny' into 'Twilight' between sunset and sunrise Controlled by a SystemProperty CheckSRSSAgainstSky=TRUE 2017-05-01 [mcXML 101] Make capable of delivering 12-13 forecast entries 12 entries - starts with night, ends with day, blank entry in slot 1 for total 13 entries 13 entries - starts with day. ends with day 2017-05-03 [Feat 109] Bring forecast complement up to the full 12-13 and figure out a way to accommodate this in the web pages. System Documentation updated with an advised approach. This requires multiple things: - Forecast package must contain 12-13 entries (SEE CHANGES TO FILE) - Slot #1 could be empty (which also means the forecasts start with night) - Changes to te format of the forecasts files - Slight changes to the templates preprocessor - Modify wxDataTypes, wxIO: WriteForecast(), ReadForecast(), [Website 104] Modify the weather and forecast pages to support 12-13 forecast periods. 2017-05-04 [Note Bugs #139, 133, others] The apparent date fuckup was caused by ProcessDT() insisting on adding an hour to the reading time. The defalt end-of-time-unit time is now 22:59. Bugs are resolved [Bug 142] Monthly and daily extremes are being recorded as today instead of yesterday. Debugging. 2017-05-06 [Feat 102] Think of a way to have a set of pages done for any location. On-demand processing, via a query, XML: "name"nn|LocCode CGI treatment: WXQueries should be able to handle it. Input file if possible: WXQuery_random.txt Response file: WXReply_random.txt server redoes pages, signals with "success" Go to the created index page [Feat 111] Invalidate stand-down warnings after xx hours, configurable through the System Property InvalidateClrWarnsAfterHrs (default 1). Will require changes to System Properties and the I/O engine. [Bug 143] The file getexterns.sh is coming out getextfileerns.sh Mangled with a recent search/replace [Bug 132] Rounded numbers, particularly in the charts, look like they're relying on int() instead of round(), reading low. Preliminary investigation 2017-04-19. No cause found. Seems to hit regional temps but not national. Sometimes works, sometimes not. 2016-05-06: Problem is caused by banker's rounding. Writing replacement algorithm. 2017-05-07 [Bug 143] Something is causing spurious years to be created with forecast files. caused by using variable before value set Corrected problem 2017-05-08 Need to dispose of any list objects created during run [Feat 116] Add a log browser/viewer Changes to wxTP: new tags. Add ViewLog query (wxIO) Add new System Property [Feat 115] Add a shutdown report browser/viewer Changes to wxTP: new tags. Add ViewShutdownReport query - Expand the shutdown reports with more operational data (incl. a permissions grid) - This whole thing also affects e.g. PuntData(); - Added new SystemProperty Longint ShowMaxLogFiles These two entries were dropped: [wxqueries] Add viewLog query 13 [wxqueries] Add viewShutdownReport query 13 2017-05-09 [Bug 145] Something is causing wind direction SE to show up oddly. Fixed; typo in code. [Feat 118] Enhance shutdown report [Bug 146 from Test Page] The following tags are not being displayed properly @CT:TA - 'AM' not appended. Think fixed. @CSTE:A - appears as degree-CC. Think fixed @CSBP:UPi - appears as 1/10 what it should be. Think fixed. @CSVs:Df - appears way low, way off. Fixed. @CSWn:UVm - apears to be way off. Think fixed. @CSWn:UVf - appears to be way off. Think fixed. @CSWB:Udl - Truncated. Fixed. 2017-05-10 [Bug 150] When no events with the tag, deletes right to the end of the file. Fixed. [Bug 148] Page renders for some locations crash the system e.g. Bancroft Problem was in yearmap.ini - bogus years 2017-05-12 [Bug 144 from Test Page] The following tags are not displaying correctly: @DRcDSuShMn cdr dcr dmr - Not in calculations. Think fixed @DRcDSuShMnD/T cdr dcr dmr - Not in calculations. Think fixed @MExDSuShMn cdr dmr - Not in calculations. Think fixed @MExDSuShMnD/T cdr dmr - Not in calculations. Think fixed @MRcDSuShMn cdr dcr dmr - Not in calculations. Think fixed @MRcDSuShMND/T cdr dcr dmr - Not in calculations. Think fixed @AExDSuShMn cdr dmr - Not in calculations. Think fixed @AExDSuShMnD/T cdr dmr - Not in calculations. Think fixed @ARcDSuShMn cdr dcr dmr - Not in calculations. Think fixed @ARcDSuShMnD/T cdr dcr dmr - Not in calculations. Think fixed Closed. [Bug 152] kPa calculation reads same as mb. Def. 'k' wasn't in suffix processor. Fixed. Closed. [Bug 151] From test page; the following variables are not calculated or displayed correctly: CSWn:UVf Redid calculation. Hopefully fixed now. Closed. 2017-05-13 [Bug 148] The following tags _still_ aren't rendering properly: AExMSuShMx / Mn AExDSuShMx Things were being written to in the wrong order. Believe fixed. Closed. [Feat 103] Give some thought to how a bulletin could be kept with "SERVER UP/SERVER DOWN" for inclusion into the page Here's how I think it could be done: - Create a text file ServerStatus.html which includes simply 'UP' or 'DOwN' every 5 minutes - Check the file date and time every 5 minutes. If the time diff. is more than 5 minutes, you've got a problem. Change ServerStatus.txt. #include the file within the HTML file. Control this behaviour with System Property boolean PostServerStatus Path and filename are controlled with System Property string PSSPath Doesn't work. Must need HTML5. Shelved for now 2017-05-14 [Aux 101] Create wxCondition to 'borrow' readings Guess we'll need: wxCondition -get wxCondition -put 2017-05-15 [Bug 153] From Test Page, the following variables are not calculated/displayed correctly CSPr:A - AmountUnit never returned a value. Rewrote. Believe fixed. @PExDSuShMn+DT - All fixed. @PExDWnMeMx+DT @PExDWBMeMx @PExDWnMeMn+DT @PExDWBMeMn @PExMSuShMx @PExMSuShMn [Bug 154] a. Nil Precip amounts are wrongly being displayed as -3000.0 mm Fixed [Bug 157] MasterRecalc request returns nothing Fixed. Now returns 'Success' Fixed others. Now all query functions return 'Success' 2017-05-17 [Fix 120] Get the forecasts observations fixed. - Need to index string, if not found, in FTermIdx() Implemented 2017-05-17 2017-05-18 [Website 105] Complete the Location Lookup templates set [Bug 156] Precip and Sunshine values are still being reset to n/a for the 00:59 processing 2017-05-18 Issue was positioning within ProcessReadings(). Think fixed. [Feat 123] New directive: RELOAD-FTERMS to reload forecast terms from the file. 2017-05-18 Implemented. [Website 106] Permit recalc for all locations (do on test site first) Add appropriate code to Master Recalc page 2017-05-18 2017-05-20 [Bug 146 from Test Page] The following tags are not being displayed properly @CT:S - double-converted to standard time (i.e. 1 hr early). 2017-05-28 [Bug 158] Normals calculation simply parrots back last year's real values Problem is, need a 3D storage array, not 1D Think this out: What about: - Use low array positions for transitory values - Use high positions for longer-term storage (i.e. further iterations) At end of day/month/year, copy short-term totals to longer-term storage Problem solved. This way, we get monthly totals and proper overall normals - Hard slogging. Lotsa manual changes, variable renaming, etc. - Came up with year-shaped array of longints Introduced new System Property Boolean CalcNormalDailyPrecipByDividingMonthly if TRUE, daily precipitation normals will be calculated as the monthly normal, divided by the number of days in the month (default) if FALSE, daily precipitation normals will be calculated by the actual daily amounts [Feat 125] Add new normals terms to ExportData(). Consider adding most other data points, as well. Testing: 2017-05-29 [Feat 101] * 2017-05-22 Extend normals dataset to include all of the recordable data * RH * DP * Sunshine * BP * Wn * WB * Vs * Extend normals-calculation routines to calculate all recordable norms * 2017-05-28 wxIO * GetNormals() * WriteNormals() * ReadNormals() * DoCalcNormalsDaily() * DoCalcNormals() * DoMasterRecalc() * - New System Property * 2017-05-22 wxDataTypes * Normals definitions * 2017-05-22 wxData * InitNorSet() * wxTP * Parse_xN() (Parse_DN(), Parse_MN(), Parse_AN() routines TESTING: RecalcNormals(); passed 2017-05-29 doMasterRecalc(); passed 2017-05-29 Parse_xN(); all passed 2017-05-29 [Bug 158] Suspect return-value string in wxQueries is a shortstring rather than an ANSIString. Check it out. Figured out wxqueries wasn't waiting long enough for the file-write to complete. May have to add a signaller file. [Feat 124] 2017-05-20 Work on the Master Recalc page to add all locations Work on the Records Recalc page to add all locations Work on the Normals Recalc page to add all locations [Bug 159] Pages do not regenerate automatically on Master Recalc 2017-05-31: [Bug 160] Annual sunshine normals appear messed up Test passed 2017-05-31 2017-06-01: [Bug 159] Monthly precip normals appear messed up Because lastmprecip wasn't being initialized Test passed 2017-06-01 [Feat 126] Expand bounds-checking for normals, to match new additions This will likely only affect one routine (doCalcNormalsDaily) Don't forget to add new fields to the website Note in documentation Test passed 2017-06-01 [Feat 127] Make the calc-normals output prettier - Condense yearly listings to one line Test passed 217-06-01 [Website 106] Work on all pages to add appropriate normals [mcXML 100] Add routine that'll read forecast velocities and amounts and include these in the forecast reports (specific vars) Test passed 2017-06-01 2017-06-02: [Feat 127] * Add new output tags to get the new forecast information into templates * Update the web page templates to show the new forecast data - w-forecasts.html.wxtp [mcXML 102] * Expand the forecast specification, adding forecast-period items for Humidex, Wind Chill, Visibility. * Add them onto the end of the forecast line, so they can be optional Test passed 2017-06-02 [Feat 128] Expand the forecast specification, adding forecast-period items for Humidex, Wind Chill, Visibility. Will affect: * WxDataTypes: ForecastRec * WxData: InitFC() * WxIO: WriteForecast(), ReadForecast() * WxTP: ParseForecast (simply uncomment code that's already there) Test passed 2017-06-02 2017-06-03: [Feat 129] * Look at Forecast Terms; make sure they're written to disk without delay Updated code for directive RELOAD-FTERMS [Bug 161] Error tags are not being rendered properly with double-quotes enclosed in double-quotes 2017-06-03: Changed all the double-quotes to single-quotes Test passed 2017-06-03 [Feat 122] Add capability of generating Extremes and/or Records on Records Recalc page, plus Daily, Monthly, Yearly selections This will affect the structure of the query * It will affect the Records page (2 new parameters) (2) * - calcExtremes * - calcRecords * Document * Daily and Monthly selections apply to both records and extremes * Yearly selections apply only to records * It will affect wxIO.ProcessXMLFile() * It will definitely affect wxIO.RecalcRecords, which incidentally needs a good testing. * The website will also be affected; new options and new bounding values; updated query structure * UseMap selections are now fully honoured. Test passed 2017-06-03 2017-06-04: [Website/Fix 124] Sweep the pages for erroneous values, with error tags now working. 2017-06-05: [mcFCT 100] Write a GUI forecast-terms editor - Display terms in natural order, double-click to edit - Graphic Previews [Fix 125] Copy over first-empty-forecast-term-finder routine from mcFCT. Compare the code in wxpro.FTermIndex() with wxFCT.NTermIndex() [Feat 130] Restore forecast-terms hit tracking 2017-06-11: [Feat 131] Generate lists of rankings: - Monthly, Daily or Yearly - Start Date, End Date - Location - Direction (up (lowest-to-highest) or down (highest-to-lowest)) - Datum - # entries (max. 100) First thoughts, use a records matrix to mark the datum you want (only observations and totals) No need--just parse the datum Maintain a sorted list of Top-whatever Return the list Query in XML - use wxQueries Other Data Structures: - List consisting of (Value, Date) New Unit: wxLists On the face of it, it's pretty simple. The rough part will be the matrix assignment and scanning, due to all the variables Use a routine to get a datum's value. Passed test, all operational modes, 2017-06-11 2017-06-13 [Bug 163] Specifying "Daily Rainfall" in lists query yields gibberish Looking for wrong letter in tag. Fixed. Test passed 2017-06-14 [Bug 164] Column count output to wxd_ma doesn't match actual. Fixed in code. Test passed 2017-06-14 [Fix 126] Prepend a zero (0.0) column to the wxg_ma data Also, ensure that all past precipitation exports are zero and not INil Prepend a zero (0.0) column to the wxg_da data Test passed 2017-06-14 2017-06-16 [Feat 134] Export records * daily (wxg_dr.txt) * monthly * yearly Test passed 2017-06-16 [Feat 132] Define format of wxg_ak and start outputting to it. Also, review existing files and ensure that formats are all rational Test passed 2017-06-16 [Feat 133] Output extended normals for all relevant export files Test passed 2017-06-16 2017-07-02 [Bug 166] Precipitation live readings are being applied only to the Precip variable. Add it to daily rain as well. Really check out how precip readings are handled--both live and statistics. Ensure that precip is credited to both rainfall and total precipitation. Doesn't seem to be happening properly at present. Test passed 2017-07-02 2017-07-05 [Bug 167] Something causing random crashes on Location 29. Narrowed it down to WriteNormals() Suspect N not being initialized in Locations Think fixed. Test passed 2017-07-05 [wxhtml 101] Check on data-terms collection in cases of limited dataset (i.e. ZLT - Alert). Modify the program to accommodate. Appears to be having trouble capturing observations, normals, records Fixed 2017-07-05 2017-07-06 [Feat 135] Properly header all data files - Year Stats - Test passed 2017-07-05 - Records - Test passed 2017-07-05 - Current Readings - Test passed 2017-07-06 - Normals - Test passed 2017-07-06 [Bug 168] In the last-31 days export file, the day number is being concatenated with the data following Fixed. Test passed 2017-07-06 [Fix 127] Append a DOM column to daily export data. Use it for tickmarks Appears in header (col. 13) but not in all data. Implemented; see [Bug 168] 2017-07-08 [Feat 136] Include an <@else> tag for all <@!>'s Process by scanning the string up to <@!> for an <@else> Test Passed 2017-07-08 2017-07-13 [Bug 171] Data-extant tests seem to be biting an extra char off when false and no else statement Believe fixed 2017-07-13. Test passed 2017-07-13. [Bug 170] ReadUserData is using FReadLn. It shouldn't be. Fix required of {home}/Year/2017/07/Readings* Believe fixed 2017-07-11. Fix may be required before redeployment. Test passed 2017-07-13. [Bug 169] Occ. crash (Access Violation) pinned down to a line in PuntData(). Tried fix 2017-07-10. Test passed 2017-07-13. [Feat 138] Allow templates to be processed on configurable intervals rather than always on-input. Thoughts to that end: * Add two parameters to the item line * - event|nn[m]|nnh|nnd|nnm * - Next D@T to regenerate (or nil) * - maintain in memory (yes, another thing)--by pointer, attached to Location record * - Scan through after event, process 'event' types * - Every minute, scan through and process those pages whose next-due DT matches the current DT or has already past * - Calculate next day/time to update the page * - Rewrite the list at program quit * - Add a RELOAD-TEMPLATES and RELOAD-TEMPLATES-LocCode directive * - Will have to track minutes and know when a new one is starting * So tracker variables NewMinute : Boolean and LastMinute, ThisMinute : wxDateTime * Add RegenTemplatesEvent and a RegenTemplatesTimed routines * - Scan all templates and regenerate those that are marked for such * - Scan all templates and regenerate those whose times have passed * - Will have to add a new Pages array to store * - The Source FileName * - The Destination FileName * - The Next Date/Time to regenerate * - The schedule * - Attach to location * Use as an aid to determining whether a location uses pages * Will have to add a ReadPages and WritePages routine * Add InitPages routine Believe completed 2017-07-12. Test passed 2017-07-13. [Feat 139] Add a monthly-regen option as well Test passed 2017-07-13 2017-07-14 [Bug 172] Fix that bloody reset-counters problem D'uh, just convert the time back to DST Believe fixed 2017-07-13. Test passed 2017-07-14. [Feat 140] Specify particular calculations System Properties CalcHumidex="CTO,..." (list locations to calculate for) Affects live calculations only Test passed 2017-07-14 [Bug 116] Bring the Humidex calculation into conformity with EC (only at src temp. > 20, and Hx > 20) [Bug 173] wxxml is not capturing the record-high date anymore, or precip records Futher changes to web page format. Adapted. Test passed 2017-07-15. 2017-07-17 [Omission 101] Page regen was meant to be when page next-regen date+time was less than or EQUAL to current date+time Test passed 2017-07-17 [Fix 129] Periodic templates next-regen times are being calculated from current time, not scheduled time. Schedules therefore drift. Test passed 2017-07-17 2017-07-26 [Bug 174] Current-Conditions info being read with FReadLn() but had empty spaces - Correct the problem (change FReadLn back to ReadLn) - Restore from backups All locations current-conditions.ini At the same time, clean up CC-term handling Test passed 2017-07-26 2017-07-29 [Bug 175] Location Lookup returns odd locations and overwrites main HTML files Recent changes to template processing must be to blame Fixed. Test passed 2017-07-29. 2017-07-30 [Bug 176 from code review] Date display incorrect for CTO.D = 'S' Date display incorrect for CTO.D = 'F' Modified suffixes to add :T6, :T8, :TF, :TS : Date Templates Test passed 2017-07-30. Documented [Feat 142] Implement tags that access a page's regen schedule (through D = Document) @DDN - Page Destination Name (name only) @DDP - Page Destination Name (full path) @DRLD,T... - Page Last Regen Date+Time @DRND,T... - Page Next Regen Date+Time @DRs - Page Regen Schedule (short - just the numeric code) @DRS - Page Regen Schedule (On Demand, Minutes, Hours, Days, Months) @DRI - Page Regen Interval @DS - Page Source Name Test Passed 2017-07-30 Documented [Feat 141] Implement a new directive RELOAD-CCTERMS-{LocCode} Add variant directive REGEN-{LocCode} Tested 2017-07-30 [Bug 115] Bring the wind chill index into conformity with EC reporting standards (only for temps below 10C and wind speeds above 4.8 km/h.) [Feat 143] Add CalcWindChill='LocCode1,LocCode2,...' to systemproperties.ini [Fix 103] Check on CC terms storage - Get rid of all the extra occurrences and make the routine more efficient. Allow empty entries Maintain a program-maintained pointer to the first empty entry. Each empty entry retains the location of the next in its hit counter When looking for a spot, first take the first empty slot; only then extend with range-checking and graceful rollback 2017-07-31 [Fix 131] Couple of FindFirsts that weren't closed, in ScanControls()--fix. Ensure that all FindFirsts have a matching accessible FindClose. Test passed 2017-07-31 [Feat 145] Add the directive variant RELOAD-CCTERMS-ALL Test passed 2017-07-31 [Feat 146] Add the directive varient RELOAD-TEMPLATES-ALL Test passed 2017-07-31 [Fix 130] Fix up the command parser so RELOAD-* all in one section Test passed 2017-07-31 2017-08-01 [Fix 132] Getting rid of the leading 'V2' in current-conditions.ini Tweaked read routine to work around it Tweaked write routine simply to omit it Test passed 2017-08-01 [New 100] GUI cc-terms editor - By Location Include rewriting of the .ini file. Investigate handling of CC terms Handled in: - TT_CC : Format token to CC term - OpenLocation : Reads CC-terms file - WriteLocation : Writes CC-terms file - CCCode : Returns code for term, with indexing - CCIndex : Returns the index for term, with indexing - CCIndex : Returns the array index for the CC Code - CCText : Returns the term for the index value - File is open-concept; items are physically indexed so they can load in any order - ProcessReadings : Uses the CCCode Should work about the same as mcFCT 2017-08-04 [Fix 134] CF and CW tags seem to be just stubs. Investigate. Eliminated. Test passed 2017-08-04 [Feat 147] Introduce a RELOAD-NORMALS-{LocCode} directive Test passed 2017-08-04 [Fix 133] Get hit-counting working again for both CCTerms and FTerms Should be straightforward Done. Test passed 2017-08-04 2017-08-27 [Bug 178] General Query is broken. Returns 'Success' Reroute output. Done. The page was never broken, but glad it was looked at. In the end, there simply wasn't data for the time period looked for. 2017-09-01 [Bug 176] Graph outputs for past 7 days need a tweak: - Omit label if (first day) and (time > 12:00) Believe fixed 2017-08-28 Test passed 2017-09-01 [Fix 136] In outputting last-7-days data, if (last day) (time < 12:00), include only the first initial of the dayname for a label Believe fixed 2017-09-01 Test passed 2017-09-01. 2017-09-07: [Bug 177] Log Bug: logfiles get messed up - Line counter goes nuts - Only 1 file found in Viewer Also affects Shutdown Reports Found a variable, ErrorCode, in ParseViewer, that was used but not declared; belonged to systemh.inc Might possibly be the problem. Will see. Not, apparently, the problem Checking now as to whether it's really only seeing 1 file 2017-09-01: Found a problem in WriteNormals() whereby a file exception leads to a logger.Pop() being skipped. Fixed. 2017-09-21: [Fix 137] Still producing way too much log output. Believe fixed 2017-09-07. Test passed 2017-09-25 [Fix 138] Normals are being written twice each cycle; once after readings, and once with WriteLocation(). Implement a DirtyNormals flag, to be set and acted upon whenever the normals data are changed. Now, Normals should be rewritten once a day, at most. Affects: * io.WriteNormals() (only write if DN flag set; set DN flag to FALSE) * io.GetNormals() (set DN flag TRUE if normals read do not match existing) * io.ReadNormals() (same again) * io.doCalcNormals() (set to TRUE) * io.doMasterRecalc() (ditto) * data.InitLocs() (initialize to FALSE) * wxDataTypes.Unit (change to spec) Test passed 2017-09-21 2017-09-30: [Fix 139] In last-24-hour graphdata output, first label should never be expressed Also fix L24Tm label, which should count up from zero Pending fix 2017-09-30 Test passed 2017-09-30. 2017-10-07: [Bug 183] [Regr Test] Records Recalc does not store information in the records header. Inv: Changes were being made immediately after the records were written. Test passed 2017-10-07. [Bug 182] ProcessStats appears to be buggy. The FirstLoop does not perform additional reading. Therefore, either the if statement for FirstLoop should enclose the entire rest of the procedure, or a definite value for DoWhat triggers further reading. Same for monthly. Yearly doesn't even process the FirstLoop flag. Everything I know tells me that it should. Required a bit of a rewrite of the procedure; very sloppy Test passed 2017-10-07. [Bug 181] ProcessStats is way inefficient Find out what it's doing and fix it for large inputs Endlessly recalculating month, records, etc. So, add a suppression flag, and check it before recalculating months and years. The justification is that if the file is over a certain size, it's likely to be bulk entries, including for the months and year(s). Make it so if the filesize is over, say, 1k, it'll trip the suppression flag. Test passed 2017-10-07. 2017-10-14: [Bug 162] Long pauses--what's up with that? Could it be related to file accesses? File refreshes? Investigate. Be nice to have a more-responsive system. Removing all those logger.Traces from the code helped. So did normals handling. Partial Solution 2017-10-07: the file is being detected and acted upon immediately. The incoming query file is not deleted until the query has been processed and a reply formulated. For many queries, processing can take some time indeed. 2017-10-14: This bug is marked as Resolved 2017-10-22: Additional information: with CalcDayRecs() being called so often by ProcessStats, this also contributed to delays. Should now be fixed. [Bug 180] Graph data outputs use standard time all year round - Convert to corrected time Cancel 2017-10-14 2017-10-22: [Fix 142] Rearrange how things are being done in Daemon() 1st half of minute, when files are supposed to be shuffled around, generate pages, etc. 2nd half of minute, read files Test passed 2017-10-22 [Fix 141] ProcessStats is inefficient. Adjust the code. Properly process FirstLoop so that extra day scanning, etc, eliminated. Test passed 2017-10-22 [Bug 179] Related to Bug 177 Occ. a normals-1.txt file ends up empty. Normals pointer seems messed-up. Last time affected location TBO Considered resolved 2017-10-22 2017-10-26: [Bug 186] Fixed an obscure bug w/date-exchanging in WxIO.ParseQuery() TP 2017-10-26 [Bug 184] (from RegTest) Bulk Data Query not working properly. Gives wrong answers for live data Inv.: always getting same result. Problem is L1 is set in LTO but not subsqeuently in CTO. Fixed by introducing CTO := LTD; into the setup code. Fixed. Test passed 2017-10-26. 2017-11-06: [Bug 187] Core of ParseGrid is broken - Always assumes stuff is based on absolute today - Rewrote properly, advancing day, month and year as necessary, and re-opening - Think fixed 2017-11-05 - Test passed 2017-11-05. [Feat 105] Figure out a way to look up statistics for a given date/time. On Demand, via query. NEW RULES FOR THE TEMPLATES UNIT Analysis of Variable Use: - CDT : CurrDTRec = Better date expansion - Use XpDT() - XT : XTRec = Simpler - Use ExplodeDT() Add New Variables: - TempLookup : Boolean; Set this flag to do temporal lookups - CDT : CurrDTRec; Master Date and Time - LDT : CurrDTRec; Latest Date and Time - XT : XTRec; Scratchpad Date and Time - WT : Working Time ("The time we says it is!") Going in: - Last Observation Dateand Time exploded into LDT - Current Date and Time exploded into CDT - Current Date and Time exploded into XT; working variable Procedure for doing a temporal lookup: - Get slot# - wxClock; gets CDT - Derive LDT from FindReading(D+T) (have to code that) - Explode XT from CDT.DT - Save Latest and LatestDT; - Set the TemporalLookup flag; - Do the templates - For current conditions, grab the location of the current reading - Unset the TemporalLookup flag; Will have to come up with some long-needed conventions for WxTP - All date/time calculations are done with XT Explode a date once only, soon as the timeframe is known New Rules: - 'Today's' date and time have already been defined and exploded into the current-time structure (CDT and Xt). - All other dates must be deduced from those. if NOT TemporalLookup then have template-processing routine set MDT and MXt at beginning of template-processing routine Get template-processing routine to refresh CTD from L[L1].Latest and Xt from MXt at top of template-processing loop At beginning of time-period-specific tags (live, daily, monthly...) - regenerate XT according to needs This will take some thinking and probably some modifications to the Queries-intercept agent. Just a query: (Modify existing; simply add an optional specification for 2017-07-18@15:08). Change the web page to add the new query term when a date and time have been selected. First convert the templates unit to behave according to the new rules. Verify performance. Then try temporal lookups. Modifications to WxTP: Test Passed 2017-10-30. Temporal Lookups: Test Passed 2017-11-06. Fixed problem with Time Zone calculation Fixed a fix of a non-problem in ProcessTempsPeriodic(), that was causing templates to be recalculated for locs rather than Loc 0 during location lookups Ongoing testing revealed Events were not updating to the proper times. Attempted fix. All modifications: Test Passed 2017-11-06. [Bug 188] For lookups, adjust search term to DST, if DST extant at that time +First crack 2017-11-05. Test Passed 2017-11-06. [Fix 143] One-time queries are generating date@date labels. Fixed. Test Passed 2017-11-06. [Extra] Did cleanup sweep for code stored in (* *) 2017-11-07: [Fix 144] When reading daily statistics, only recalculate month if for a past month. Attempted fix 2017-11-06. Re-attempted same day. Test Passed 2017-11-07. 2017-11-24: [Bug 189] Monthly events are not sorting correctly for past and future months. Fixed 2017-11-24. Test passed 2017-11-24. 2017-11-25: [Feat 150] Add obserations parameter for snowfall readings - Add onto end as an optional parameter Modify mcOCR to accept readings in mm (rain) or cm Okay, this potentially opens up a can of worms with redefining the readings record Suggest: - Append one optional parameter--a single character--to the input line, for measurement units. - If the units is in mm - Rainfall += Precip - Precipitation += Precip - If the units is in cm - Snowfall += Precip - SnowEquiv += Precip/10 - Precipitation += SnowEquiv Will need to update some other routines to tally up all the parameters and add up the correct precip amounts Will also affect: * AsmReadings() * WriteReadings() * ParseReadings() * ProcessReadings(); * DoMasterRecalc() * wxDataTypes:ReadingRec * wxData:InitReadings() * wxData:InitLatest() Implemented 2017-11-25 Test Passed 2017-11-25. [Bug 190] Latest forecast terms are not making their way into the database. Missing insertion point assignment. Fixed 2017-11-25. Test Passed 2017-11-25. [Feat 151] Add new tag--CSPU--current stats precipitation units Done 2017-11-25. Test passed 2017-11-25. 2017-11-29: [Bug 191] (from RT) Daily query results are not CR-separated. Believe fixed 2017-11-27. Test passed 2017-11-29. [Bug 192] (from RT) One-time, Monthly and Yearly bulk-data queries are not properly formatted and/or not working. Fixed 2017-11-29. Test passed 2017-11-29. 2017-12-09: [Page Bug 193 w-querylist] RESOLVED Daily List query yields empty response. Wrong response. Believe problem is correction to/from daylight time, at page's end. Problem turned out to be JavaScript. Fixed. 2017-12-26: [Bug 194] Cross-location @CT, @CD flags not being interpreted correctly. Giving the time of latest observations for page location, not referenced location. Took a crack at it, 2017-12-26. Test passed 2017-12-26. 2017-12-30: [Bug 195] Crashing on encountering location dates for next year - Issue seems to be the OpenYear() routine, which was recently rewritten - Changed the cutoff date from thisyear-12-31 to nextyear-12-31 - Built in protection against future dates; reads to end of file and exits. - Also added recalculation of daylight/moonlight values only when they're nil Test passed 2017-12-30. 2017-12-31: [Bug 198] @*DMnPhD returning value of calculation instead of actual value. Fixed 2017-12-13. [Bug 197] Moon age calculation is broken. Yielding high values or zero. Just go with the submitted values. Find a replacement algorithm that works! Try this: - Add a system property NewMoonBaseDate. Formula: ((given date-base date) DIV 86400) MOD 29.5305882 = days old Test Passed 2017-12-31 [Bug 199] Moon phase calculation is broken. Replace with something based on number of days [Bug 196] Daylight calculation is broken for moon values Revisit Fixed 2017-12-31. Test passed 2017-12-31. 2018-02-04: [Fix 145] Confirm: Don't check daily stats against records until end of day Don't check monthly stats against records until end of month Don't check yearly stats against records until end of year 2018-02-05: [Bug 201] Moon_Age() calculation still off. Tried again 2018-01-27. Fixed 2018-02-04. Tested 2018-02-05. 2018-08-28: [Fix 147] [Fix 112] Adjust Moon Phase calculation - seems out of whack by 1 day Change the moon phase calculation: - Events (NM, FQ, LQ, FM) occur only on the specific day of their occurrence. - All other times, it is the gibbous phase Implemented 2018-08-28 2018-08-29: [Fix 146] The fterms.txt file is again filling up. Must make up new rules or expand the size of the database. - Leaning towards new rules _and_ expanding the database. Possible rules - When forecast over xx characters, truncate. - Truncate at first comma. - Truncate at "xx percent". - Truncate at period. - Truncate at then. What would be the implications/features of an expanded database? - Any size up to 65,535 entries; - Alter the size of the counter variable - Alter the size of the recno variable - Many, many places in the program - Reading and writing files should be the same as before, as the size of the argument doesn't matter - Alter MCFCT - Existing database would not be touched. - Would have to do this quickly, before the file actually fills - New livereadings files would be slightly larger Implemented 2018-08-28 and testing now. Test passed 2018-08-29. 2018-09-04: [Bug 202] Maximum and minimum values are prevented from displaying. This is incorrect. - Working on it. Appears the routine Vet is often used, which causes it. Unsure of why I originally wanted this behaviour. Disabling it. - Appears fixed. Testing. - Was almost fixed. Didn't properly understand the logic, and didn't return the value from the function. - Wasn't properly parsing things. Think I've got it now. - Ungh! It's late, and I forgot to undo the damage I'd already done, in Parse_DS(). Really think I've got it now. Test passed 2018-09-04. 2018-09-15: [Bug 203] When reading stats, if sky='Clear' or sky='twilight' then ensure sunlight=0 Fixed 2018-09-15. Test passed 2018-09-15. [Bug 204] Function IO.SeachTextFiles() does not provide a return value. Fixed 2018-09-15. No need to test. [Fix 148] When reading in anything that could come in large volumes, limit to, say, 100 files at a time. Implemented granularity. Controlled by several constants (later to become System Properties): READINGSFILE_GRANULARITY: Maximum number of Readings files to read at one time. STATSFILE_GRANULARITY: Maximum number of Statistics files to read at one time. FORECASTFILE_GRANULARITY: Maximum number of Forecasts files to read at one time. Converting those values now to System Properties. systemproperties files Code Implementation completed 2018-09-15 To test: - Copy a whole bunch of older readings files to the system. - Start the server. - Likely will only affect readings for now. Check in the logfiles for evidence. Test passed 2018-09-15. 2018-09-16: [Fix 149] Look into speeding up the observations processing. - I know, but look Some Ideas off the top: - A flag to defer saving records until the overall process completes. Flag processing could be inserted into the routines where *_Granularity are used. Observations: * Calls are made to OpenYear() and OpenReadings(). AFAICT, those routines do nothing when the requested resource is already opened. * Call is made to UpdateMonth() - Call made to FlushYear() * Call is made to CheckDayRecs() - Call made to WriteRc() - Call made to FlushYear() * Call made to CheckMonthRecs() - Call made to FlushYear() - Call made to WriteRc() * Call made to UpdateYear() - Call made to FlushYear() * Call made to CheckYearRecs() - Call made to WriteRc() * Call made to WriteLocation() - No subsidiary calls * Call made to FlushYear() - No subsidiary calls * If we were to designate a flag, say Hold_Updates, as indicating whether to go ahead with writes or hold them, we could parse that flag at locations where a call to a writing routine is to take place. * If we could then detect the end of a string of files, we could set the variable accordingly, and parse it in the service routine. Q: where are readings written out? A: Only when the readings file is closed. Excellent. * Now, control the behaviour of the flag when updating. - Use a new systemproperty, Hold_Write_When_Reading Implemented fully and test passed 2018-09-16. My God--it sucks the records down! Probably hundreds per second. 2018-09-21: [Bug 204 (2018-09-21)] When testing against warnings, if test fails, remainder of page is deleted in buffer. Addendum: this is a correct behaviour when a closing tag cannot be found. Do not put location specifiers into closing tags. Resolved 2018-09-21. 2018-09-25 (approx.) [Aux 104 (2018-09-23)] Build a readings-input program. - Work off the CGI-BIN interface. - Store it for wxProcess to pick up - Store in 'wxp_pending_readings.txt' - wxProcess reads it (if present) and substitutes any readings thus given. - wxProcess will have to parse and reassemble the input line. This shouldn't be a problem. - Command-line flag to tell wxProcess whether to do its usual processing, in which case: - Process first - Substitute readings second (they have priority) Build a readings-input web page. - Link into the other pages. - Don't forget to update functions.js 2018-10-06: [Feat 151 (2018-10-05)] Save templates.ini files according to a schedule. - Make it on a by-location basis. - Make it upgrade automatically to this new variable (set to default value) in location.ini. This will make it easier to reload on busy systems. - Structure: An extra line in templates.ini, simply stating: SaveEveryMinutes 30 (or whatever) (default 10) and a line: LastSaved YYYY-MM-DD@HH:MM That will force the file to be saved every 30 minutes (or whatever). Will affect: * Data structures and initialization * OpenLoc() * ReadTemplatesIni() * WriteTemplatesIni() * Scheduling * Auto-saving of templates.ini whenever data comes in * FlushLocation needs an additional parameter: imperative-write (to ensure templates.ini gets saved) - If false, only gets saved according to the schedule. [Fix 150 (2018-10-03)] When outputting a query response, only change the output file's extension to .txt at the end. See if that will satisfy wxQueries [Feat 152 (2018-10-05)] Make Locations reloadable. - New Directive: RELOAD-LOC-locName - Have to make a new routine just to read the locations file. Done 2018-10-06. 2018-10-07: [Feat 153 (2018-10-07)] When the program first runs, create a file called 'wxrunning' Delete that file upon exit. Done and tested 2018-10-07. 2018-10-21: [Feat 154 (2018-10-10)] Sort incoming files into a queue - Observe granularity Process them in order. Then do daily/monthly/yearly processing as normal, even with updates suppressed Will affect a number of file types: - Observations - Forecasts - Statistics - Warnings For each type, in the Location record, keep: - A sorted list of filenames and file-times - A countdown counter, from the highest (earliest) member in the list :Define FileNameRec = record RecNo : Integer; FileName : ShortString; end; :Define FileNamesArray = array[1..10000] of FileNameList; :Define FileNamesPtr = ^FileNamesArray; :Add to the LocationRec: OFN, FFN, WFN, SFN : FileNamesPtr; Initialized to nil; OFC, FFC, WFC, SFC : Integer; Initialized to zero. Any time there are multiple files: - Queue the filenames by inserting into the list in ordered fashion In processing queue, when queue files are available, process them from the end of the list. - Process in groups, according to granularity limits, from the earliest to the latest When the counter reaches zero, collect a new set of filenames Will need a routine to make ordered insertions in the list. - Use a binary search Affects: wxDataTypes Don't forget to discard any pointers at punt time. [Bug 205 (2018-10-10)] The Location.LatD and LongD variables are smallints - too small to accommodate values above 127 for longitude. This isn't causing problems now; but for Longitudes above 127... Change to Integer Won't affect the file format at all, but check inputs. Done and tested 2018-10-21. 2018-12-22: [Feat 155 (2018-12-22)] Allow observations packets to include -ow in the filename. (i.e. CTO-Observations-20181222_0342-ow.txt) This flag indicates that the observations therein are to overwrite the ones aready extant. A 'skip' flag in any positon tells the system to ignore that value with effect. nils will replace any extant values. - NewReading will have to be modified somehow to notify if it's grabbing an existing record - Change function return to longint. If return value > 65000, divide by 65000 to get record number. That record number must be pulled. - Pull-values routines must be forked to reject 'skip' inputs. - Suggest creating a new constant, SKIP, -30001. Test against this constant in return value. Implmented 2018-12-22. Test passed 2018-12-22. Issue with Precip; will have to investigate. Because of the way precip is measured, any amounts can only be added to the total. Therefore, we advise manually altering the amounts in the readings files, then manually altering the daily stats. These will have to be done with the program shut off. In the meantime, precipitation values are ignored if -ow is set. Note: cannot replicate.