Hello,
I'm new to this forum and new to working with datalogger and the data gathered by loggers in general. And I have a general question:
During a quality check of my data I noticed a leap in time backwards (1 hour) so that during this time I have two sets of data (containing 10hz measurements of an ultrasonic anemometer). Generally I'm not sure how to handle this. There is a leap in time forewads (1 hour) a little earlier in the data, so here I take it as it is, the data is missing. Somehow trying to fill this gap seem not scientific to me.
But when the leap is reverse I wonder
a) which set of data to use, as I have two
and
b) does this affect all following measurements? (e.g. has there been a shift in time for the entire time series so that I need to substract the time difference to gain the original recording time of the data)
Therefore I wondered what might cause such leaps in time and how the timekeeping works. All found out so far is that cr3000 has an internat clock.
In my data the leap in time looks like this (within a 30min file):
the file starts at 14:30 1st column timestamp, 2nd column record number
second to last row: "2014-09-17 14:59:53.4",1715310, measurements
last row:"2014-09-17 13:59:53.6",1715311, measurements
the following file contains the missing lines up to 14:00
the next beginns at 14:00:00.1 and is fine again.
The preceding leap in time forewards looks like this (same day in the morning):
second to last row: "2014-09-17 08:42:56.4",1525028, measurements
last row: "2014-09-17 09:42:44.9",1525029, measurements
Obviously I also wonder if both irregularites are connected and the data which is affected is that which has been gathered in between both irregularities. Still, I would not know how to handle it.
Maybe you have had similar problems with your data and could explain how you handled it or what might cause such timekeeping errors in the logger.
Thanks a lot!
If my description misses important information please let me know an d I will try to add it.
As for the rest, my data is almost complete with only single lines missing in some files.
Petra
It seems to me you have two issues to resolve. The first issue relates to why
the clock on the CR3000 is being "changed" or "set". My best formulation is
that someone sent a "clock set" command to the logger, and then perhaps
someone else (from a different computer that had it's clock configured 1 hour differently)
sent another "clock set". The most common way to send a "clock set" is using
the "Set" button found in the main screen of "Connect" in LoggerNet.
The Setup screen utility (Clock tab) also has a way to automatically set the clock (usually once
a day) if it has deviated too far from the clock on the computer running LoggerNet.
So to resolve the first issue, you should make sure that everyone who connects
to the logger uses a computer whose time is in the same time zone (i.e. set to
the same time of day). Or perhaps the better option is to designate only one computer that
would be used to set the clock. Some companies decide to use UTC (Universal time) because
they have people in multiple time zones who are using the data.
The second problem you have is how to sort through the data that has been collected
in which a time has been set back and then forward. You can easily see what happened
in the datalogger by just following the Record numbers (they are always incrementing by
one each time the datalogger stores a record). If you just sort your record numbers sequentially
you will see what actually happened on the datalogger, regardless of what the timestamps
might indicate.
If the time was set backward by one hour, and then left alone for a while, and then set forward
by one hour (and there were no other clock sets), then one way to fix the timestamps would
be to add one (1 hour) to the hour reading for all of the records that were stored during the period
in which the clock was set backward. Then right when the timestamps get to the point where
there is a jump forward, they will match up properly.
Thank you for explaining it so well!
I altered the data as you recomended. As we are usually using UTC+1 all year round for our measurements, it may be that by accident someone used UTC+2 (Central European Summertime, Computers do the change automatically) and noticed a little later, that this has to be corrected, so that it follows our couvention. I will ask my colleagues about that. :)
Summertime is a nuisance for year-round measurements!
Yet, there is one thing which still does not fit:
That was the starting point (unaltered data):
second to last row: "2014-09-17 08:42:56.4",1525028, measurements
last row: "2014-09-17 09:42:44.9",1525029, measurements
second to last row: "2014-09-17 14:59:53.4",1715310, measurements
last row:"2014-09-17 13:59:53.6",1715311, measurements
and in between both time shifts I substracted 1 hour.
But when I take a colser look at the seconds of the timestamps it now says (for the first/earlier shift in time):
second to last row: "2014-09-17 08:42:56.4",1525028, measurements
last row: "2014-09-17 08:42:44.9",1525029, measurements
and there is an overlap of several seconds (still some duplicated timestamps/data left), meaning the first shift is a little less (12s) than one hour. Now I don't understand much about programming and signal processing, but I thought the issue might emerge due to the time it might take to process new information for the logger? (I'm not sure if this is the right expression for what I mean... but, data needs to be transmitted and swapped around and understood/implemented by the logger, I imagine...)
Do you (or anyone else) have a clue if my guess might be correct? Or what else might be the cause? Does anyone have an understanding of how much time it takes to process changes for the logger? I feel like 12s is way too much time, so I think I will observe the data in much greater detail to see if I find another small shift in time in any files recorded on that day to compensate for the "left over" 12s. Still, maybe someone feels like commenting on my thoughts.
Thanks again
Petra
There are two clocks to consider: The clock on the computer running LoggerNet, and the clock on the datalogger.
Just before the very first Clock Set occurred (which happened between Record # 1525028 and 1525029) the
clock on the datalogger was 11.5 seconds ahead of the LoggerNet clock (considering just minutes and
seconds and not the hour of the day). So the Clock set moved the seconds reading backward by 11.5
(The total time change was a forward move of 59 minutes, 48.5 seconds -- which is 60 minutes minus 11.5
seconds - almost an hour but not quite).
Then later, when clock was set back an hour, this operation was either done from the same LoggerNet
machine, or one whose clock was synchronized with the original LoggerNet machine. So for
that Clock Set there was no adjustment made to the seconds, only the hour was affected.
In other words, the datalogger clock and the LoggerNet clock were not synchronized previous to the
first Clock Set. The first Clock Set attempted to synchronize, but unfortunately got the hour off by one.
The only way to get a consistent progression is to shift the timestamps backward by 11.5 seconds
in record 1525028 and all records before that. The other alternative is less desirable (shift all timestamps
of records 1525029 and later ahead by 11.5 seconds), because then your newest records don't match the
LoggerNet clock.
My best advice would be to simply create two different data files, one that includes records up to
and including 1525028. Then create a separate data file that includes records 1525029 and forward.
Then you can just treat the two histories separately and you don't have to shift the timestamps.