[Gelöst] Endlosschleife in ElectricityCalculator wegen Zeitumstellung?

Begonnen von KyleK, 31 Oktober 2022, 00:09:53

Vorheriges Thema - Nächstes Thema

Sailor

Hi rob

Zitat von: rob am 01 November 2022, 12:57:26
The other two modules WaterCalc and GasCalc do have the same code line inside. So I guess they even should work well with that patch.

Correct. All 3 modules are based on the same structure. Most of the code can be found exactly at the same line.

Zitat von: rob am 01 November 2022, 12:57:26
It would would be excellent if all concerned modules could be patched officially (at least those three ones). But I'm not sure who feels actually responsible for that and when it could be done (are the module
I am the author and current maintainer and I will do it as soon as possible. Friday latest!
Thanks a lot for your help! Open Source at its best!


Zitat von: rob am 01 November 2022, 12:57:26
Well, the next clock change gives us a delay of almost 6 months - maybe not that urgent  ;)
Exactly!  ;)

Regards
    Sailor
******************************
Man wird immer besser...

rob

Moinssen.

That's excellent my dear  ;D ;D

Thank you very much for your quick response and great work.

Kind regards
rob

isy

Ein Weg wird erst zu einem Weg, wenn man ihn geht

Sailor

#18
Moin zusammen

Eingepflegt. Sollte morgen im Update dabei sein.

@KyleK : Bitte Thread als "Gelöst" markieden

Gruß
    Sailor
******************************
Man wird immer besser...

KyleK

Thanks everyone for the quick patch.

I do remember some DST issues last year or the year before, which begs the question: why does every module do this on its own? Is there a standard way to do this in Perl, and if so, can't fhem.pl provide such a common solution?
FHEM on Raspberry Pi 3B+
CUL868
7x MAX! Thermostat, 8x MAX! Fensterkontakte
Conbee II + deConz, TradFri Lampen, Osram Smart+ Steckdosen

Nestor

@Sailor; You still have some work to do...
There are still 2 incorrect timers in Gas en WaterCalc and 86400 comparison will not be correct on the 2 DST switch days.

/srv/fhem$ grep 86400 FHEM/*Calculator.pm
FHEM/73_ElectricityCalculator.pm: if ( $DeltaTimeSinceLastUpdate >= 86400) {
FHEM/73_ElectricityCalculator.pm: if (($ElectricityCountReadingTimestampCurrentHour < $ElectricityCountReadingTimestampPreviousHour) || ($ElectricityCountReadingLastChangeDelta > 86400))
FHEM/73_GasCalculator.pm: if ( $DeltaTimeSinceLastUpdate >= 86400) {
FHEM/73_GasCalculator.pm: my $EpochNextMidnight = timelocal(1, 0, 0, $mday, $mon, $year+1900) + 86400;
FHEM/73_GasCalculator.pm: if (($GasCountReadingTimestampCurrentHour < $GasCountReadingTimestampPreviousHour) || ($GasCountReadingLastChangeDelta > 86400))
FHEM/73_WaterCalculator.pm: if ( $DeltaTimeSinceLastUpdate >= 86400) {
FHEM/73_WaterCalculator.pm: my $EpochNextMidnight = timelocal(1, 0, 0, $mday, $mon, $year+1900) + 86400;
FHEM/73_WaterCalculator.pm: if (($WaterCountReadingTimestampCurrentHour < $WaterCountReadingTimestampPreviousHour) || ($WaterCountReadingLastChangeDelta > 86400))


@KyleK I posted a link to a good solution (DateTime module) instead of counting seconds manually.
From the Perl FAQ:
ZitatThe DateTime module makes it simple, and give you the same time of day, only the day before, despite daylight saving time changes. Most people try to use the time rather than the calendar to figure out dates, but that assumes that days are twenty-four hours each. For most people, there are two days a year when they aren't: the switch to and from summer time throws this off.

This automatically takes into account DST days (with 23 and 25 hours).

use 5.030;
use strict;
use warnings;

use DateTime;
use Time::Local;

sub timelocal_add {
my $epoch = shift;
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($epoch);
return Time::Local::timelocal(0, 0, 0, $mday, $mon, $year+1900) + 86400;
}

say "Summertime 2023";
my $dt = DateTime->new(year => 2023, month => 03, day => 26, time_zone => 'Europe/Brussels');
say $dt;

say "Datetime +1day";
say $dt->add(days => 1);

say "Fhem +1day (WRONG)";
say DateTime->from_epoch(epoch => timelocal_add($dt->epoch));

say "\nWintertime 2023";
my $dt2 = DateTime->new(year => 2023, month => 10, day => 29, time_zone => 'Europe/Brussels');
say $dt2;
say "Datetime +1day";
say $dt2->add(days => 1);

say "Fhem +1day (WRONG)";
say DateTime->from_epoch(epoch => timelocal_add($dt2->epoch));


Result:
Summertime 2023
2023-03-26T00:00:00
Datetime +1day
2023-03-27T00:00:00
Fhem +1day (WRONG)
2023-03-27T22:00:00

Wintertime 2023
2023-10-29T00:00:00
Datetime +1day
2023-10-30T00:00:00
Fhem +1day (WRONG)
2023-10-30T23:00:00


This is my fourth year as a FHEM user and the 3rd module with a DST bug.
I guess next year history will repeat itself. ;D

Look at all the modules using 86400 :P
grep -r --count 86400 FHEM | grep -v :0
FHEM/73_ElectricityCalculator.pm:2
FHEM/98_rain.pm:2
FHEM/00_CUL.pm:1
FHEM/96_allowed.pm:1
FHEM/00_TCM.pm:2
FHEM/73_WaterCalculator.pm:3
FHEM/38_netatmo.pm:1
FHEM/98_HourCounter.pm:2
FHEM/60_allergy.pm:2
FHEM/42_SMARTMON.pm:2
FHEM/00_KM271.pm:2
FHEM/10_EnOcean.pm:6
FHEM/90_at.pm:3
FHEM/01_FHEMWEB.pm:1
FHEM/lib/AttrTemplate/mqtt2.template:3
FHEM/lib/AttrTemplate/httpmod.template:7
FHEM/57_CALVIEW.pm:3
FHEM/13_KS300.pm:2
FHEM/95_Astro.pm:7
FHEM/98_GoogleAuth.pm:1
FHEM/10_SchellenbergHandle.pm:2
FHEM/51_MOBILEALERTS.pm:1
FHEM/98_alarmclock.pm:4
FHEM/00_SIGNALduino.pm:1
FHEM/93_DbLog.pm:6
FHEM/21_N4HMODULE.pm:3
FHEM/98_statistics.pm:3
FHEM/92_FileLog.pm:7
FHEM/89_HEATRONIC.pm:4
FHEM/89_FULLY.pm:3
FHEM/74_HusqvarnaAutomower.pm:1
FHEM/98_uptime.pm:1
FHEM/44_S7.pm:2
FHEM/72_FB_CALLLIST.pm:4
FHEM/98_JsonMod.pm:1
FHEM/98_TRAFFIC.pm:2
FHEM/98_DOIF.pm:2
FHEM/23_LUXTRONIK2.pm:3
FHEM/10_FBDECT.pm:2
FHEM/50_SSFile.pm:2
FHEM/00_THZ.pm:25
FHEM/73_GasCalculator.pm:3
FHEM/94_PWM.pm:2
FHEM/10_ZWave.pm:1
FHEM/47_OBIS.pm:3
FHEM/95_holiday.pm:3
FHEM/95_PostMe.pm:1
FHEM/42_SYSMON.pm:4
FHEM/21_OWCOUNT.pm:1
FHEM/88_HMCCU.pm:1
FHEM/98_rssFeed.pm:2
FHEM/RESIDENTStk.pm:10
FHEM/26_KM273.pm:1
FHEM/97_TrashCal.pm:1
FHEM/99_Utils.pm:1
FHEM/74_Nmap.pm:2
FHEM/55_DWD_OpenData.pm:5
FHEM/95_YAAHM.pm:25
FHEM/98_SVG.pm:14
FHEM/98_DOIFtools.pm:3
FHEM/98_average.pm:2
FHEM/99_SUNRISE_EL.pm:5
FHEM/96_Snapcast.pm:1
FHEM/00_HMLAN.pm:1
FHEM/57_Calendar.pm:11
FHEM/74_XiaomiBTLESens.pm:1
FHEM/10_GFPROBT.pm:1
FHEM/93_DbRep.pm:42
FHEM/57_SSCal.pm:10


Sailor

Hi Nestor

Zitat von: Nestor am 03 November 2022, 18:11:50
@Sailor; You still have some work to do...
There are still 2 incorrect timers in Gas en WaterCalc and 86400 comparison will not be correct on the 2 DST switch days.

I fixed it on all 3. Will test it over midnight and check it in as long ther are no errors tonight.

Regards
    Sailor
******************************
Man wird immer besser...

KyleK

@Nestor I'm afraid I'm not a developer, the maintainer of each module would have to fix the issue.

I do wonder though, why such functionality is not provided by the FHEM basecode, apparently lots of modules could (and should) use the same code.
FHEM on Raspberry Pi 3B+
CUL868
7x MAX! Thermostat, 8x MAX! Fensterkontakte
Conbee II + deConz, TradFri Lampen, Osram Smart+ Steckdosen

Sailor

Zitat von: KyleK am 05 November 2022, 19:20:07
I do wonder though, why such functionality is not provided by the FHEM basecode, apparently lots of modules could (and should) use the same code.

As a matter of fact, its just 4 lines of code:


my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
my $EpochThisMidnight = timelocal(0,0,0,$mday  ,$mon,$year);
my $EpochNextMidnight = timelocal(0,0,0,$mday+1,$mon,$year);
my $SecondsToday = $EpochNextMidnight - $EpochThisMidnight;


I do not see a reason, why this code cannot be implemented by each module author directly.

Best reards
    Sailor
******************************
Man wird immer besser...

Sailor

Dear All

New versions for all 3 Calculators checked in.

Regards
    Sailor
******************************
Man wird immer besser...

Beta-User

Zitat von: Nestor am 03 November 2022, 18:11:50
Look at all the modules using 86400 :P
I'm quite sure, that's not the complete picture, as some might use DAYSECONDS constant for the same purpose...

On the other hand, I'm also pretty sure, there's some false positives on the resulting list for both of the grep variants... (especially the code of statistics should be ok, and in other cases, it's just some checks or actions (the .template-things), if they happen once every day or every 24 hours...
Server: HP-elitedesk@Debian 12, aktuelles FHEM@ConfigDB | CUL_HM (VCCU) | MQTT2: MiLight@ESP-GW, BT@OpenMQTTGw | MySensors: seriell, v.a. 2.3.1@RS485 | ZWave | ZigBee@deCONZ | SIGNALduino | MapleCUN | RHASSPY
svn: u.a MySensors, Weekday-&RandomTimer, Twilight,  div. attrTemplate-files

Nestor

@Sailor,

Unfortunately in your last commit you changed timelocal_nocheck(0,0,0,$mday+1,$mon,$year) to timelocal(0,0,0,$mday+1,$mon,$year) but this does not work on the last day of the month.

my $ts=1669836392; # Wed Nov 30 2022 19:26:32 GMT+0000
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime($ts);
say Time::Local::timelocal(0,0,0,$mday+1,$mon,$year)


Result:
Day '31' out of range 1..30 at test.pl line 9.
Removed cmd 1572: perl test.pl


You have to use timelocal_nocheck as suggested by Beta-User in the 2nd post in this topic.

The Perl docs have the following to say, about timelocal_nocheck so I believe it is more like a hack but it works?
ZitatIf you supply data which is not valid (month 27, second 1,000) the results will be unpredictable (so don't do that).
https://perldoc.perl.org/Time::Local#timelocal_nocheck()-and-timegm_nocheck()

Nestor

Zitat von: Beta-User am 06 November 2022, 10:44:14
On the other hand, I'm also pretty sure, there's some false positives on the resulting list for both of the grep variants... (especially the code of statistics should be ok, and in other cases, it's just some checks or actions (the .template-things), if they happen once every day or every 24 hours...

Look over here, wow, somebody already investigated this and warned about using ONE_DAY constants and timelocal() with DST timezones. Incredible!
https://perldoc.perl.org/perlfaq4#How-do-I-find-yesterday's-date?

DateTime module was released in 2003  ;D. Perfect handling of DST, Timezones and date/time arithmetic.
https://metacpan.org/pod/DateTime


Beta-User

Zitat von: Nestor am 07 November 2022, 21:52:01
DateTime module was released in 2003  ;D . Perfect handling of DST, Timezones and date/time arithmetic.
https://metacpan.org/pod/DateTime
Afaik, DateTime is not yet part of the Core Modules, so this would require some additional actions from user side....

Apart from that, "good" usage of (real Perl) Modules is not very common in FHEM. As soon as packaging FHEM-modules would be more common, one really could think about some more standardisation wrt. this subject.

As "good" workarounds are (relatively) well known, I'd stick with the recommendation to use one of the code snipplets I already tried to point to (and make sure, they are used in a save way)...
Server: HP-elitedesk@Debian 12, aktuelles FHEM@ConfigDB | CUL_HM (VCCU) | MQTT2: MiLight@ESP-GW, BT@OpenMQTTGw | MySensors: seriell, v.a. 2.3.1@RS485 | ZWave | ZigBee@deCONZ | SIGNALduino | MapleCUN | RHASSPY
svn: u.a MySensors, Weekday-&RandomTimer, Twilight,  div. attrTemplate-files

Sailor

Well

My research confirms your suggestion, that DateTime seems to be the only solution.

The Code would look like


my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);

my $ObjectNow  = DateTime->new(
year       => $year+1900,
month      => $mon+1,
day        => $mday,
hour       => 0,
minute     => 0,
second     => 0,
time_zone  => 'Europe/Berlin',
);

my $EpochNow   = $ObjectNow->epoch();

my $ObjectTomm = $ObjectNow->add(days => 1);
my $EpochTomm  = $ObjectTomm->epoch();

my $EpochDiff = $EpochTomm - $EpochNow;


If you like to double check it, than I am going to implement it in the Counter!



My tests

26.03.2023
2022.11.08 18:02:44.048 1: myIonosApi - GetHostByName _______________________________________________________________________________
2022.11.08 18:02:44.048 1: myIonosApi - GetHostByName       - EpochNow                    : 1679785200
2022.11.08 18:02:44.048 1: myIonosApi - GetHostByName       - EpochTomm                   : 1679868000
2022.11.08 18:02:44.048 1: myIonosApi - GetHostByName       - EpochDiff                   : 82800

29.10.2023
2022.11.08 18:04:08.521 1: myIonosApi - GetHostByName _______________________________________________________________________________
2022.11.08 18:04:08.522 1: myIonosApi - GetHostByName       - EpochNow                    : 1698530400
2022.11.08 18:04:08.525 1: myIonosApi - GetHostByName       - EpochTomm                   : 1698620400
2022.11.08 18:04:08.525 1: myIonosApi - GetHostByName       - EpochDiff                   : 90000

31.10.2023
2022.11.08 18:04:53.634 1: myIonosApi - GetHostByName _______________________________________________________________________________
2022.11.08 18:04:53.634 1: myIonosApi - GetHostByName       - EpochNow                    : 1698706800
2022.11.08 18:04:53.634 1: myIonosApi - GetHostByName       - EpochTomm                   : 1698793200
2022.11.08 18:04:53.634 1: myIonosApi - GetHostByName       - EpochDiff                   : 86400



Thanks a lot!

Regards
    Sailor
******************************
Man wird immer besser...