Autor Thema: DBLog - reduceLogNbl average=day --> Daten nur teilweise reduziert  (Gelesen 235 mal)

Offline RalfRog

  • Full Member
  • ***
  • Beiträge: 124
Hallo
Ich habe angefangen einige Werte in eine Maria-DB zu loggen und dazu auch ältere Werte aus den FileLogs übernommen.

Um die ältere Daten gleich zu reduzieren habe ich jeweils ein
reduceLogNbl 90 average=day include=<device>:<reading> durchgeführt.
Zunächst dachte ich, dass ich einen Fehler mache. Doch jedesmal wenn ich den Befehl ausgeführt habe ist der letzte Tag unvollständig/unsauber bearbeitet.

Alle Daten (bis auf das jüngste Datum) werden sauber auf einen Wert pro Tag mit dem Zeitstempel JJJJ-MM-TT 12:00:00 reduziert (Event = rl_av_d);
nur beim letzten Tag gibt es eine Mischung aus bearbeiteten und unbearbeiteten Einträgen.

FHEM-Log (dort sieht man für den 30-03 das der letzte Schritt fehlt):
Schritt-1--> 2021.05.11 00:36:45.889 3: DbLog DBLogging: reduceLogNbl deleting 2 records of day: 2021-03-29
Schritt-2--> 2021.05.11 00:36:45.898 3: DbLog DBLogging: reduceLogNbl (hourly-average) updating 2 records of day: 2021-03-29
Schritt-3--> 2021.05.11 00:36:45.911 3: DbLog DBLogging: reduceLogNbl (daily-average) updating 1, deleting 23 records of day: 2021-03-29

Schritt-1--> 2021.05.11 00:36:45.950 3: DbLog DBLogging: reduceLogNbl deleting 7 records of day: 2021-03-30
Schritt-2--> 2021.05.11 00:36:45.966 3: DbLog DBLogging: reduceLogNbl (hourly-average) updating 7 records of day: 2021-03-30

2021.05.11 00:36:45.991 3: DbLog DBLogging: reduceLogNbl finished. Rows processed: 2854, deleted: 2731, updated: 678, time: 7.61sec

Die Datenbankinhalte  sehen wie folgt aus (28. & 29. ok / 30. nok --> 9 records mit Event rl_av_d):
"TIMESTAMP" "DEVICE""TYPE" "EVENT" "READING" "VALUE" "UNIT"
"2021-03-28 12:00:00" "Jet" "HTTPMOD" "rl_av_d" "E10"   "144.291" "Cent"
"2021-03-29 12:00:00" "Jet" "HTTPMOD" "rl_av_d" "E10"   "150.129" "Cent"
"2021-03-30 00:26:45" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 01:26:46" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 02:26:47" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 03:26:48" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 04:26:48" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 05:26:49" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 06:26:50" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 07:26:51" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 08:26:51" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 09:26:52" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10"   "153.9" "Cent"
"2021-03-30 10:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "149.233" "Cent"
"2021-03-30 11:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "142.900" "Cent"
"2021-03-30 12:26:55" "Jet" "HTTPMOD" "E10: 144.9  Cent" "E10"   "144.9" "Cent"
"2021-03-30 13:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "144.400" "Cent"
"2021-03-30 14:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "141.900" "Cent"
"2021-03-30 15:26:57" "Jet" "HTTPMOD" "E10: 143.9  Cent" "E10"   "143.9" "Cent"
"2021-03-30 16:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "142.900" "Cent"
"2021-03-30 17:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "143.900" "Cent"
"2021-03-30 18:20:06" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10"   "139.9" "Cent"
"2021-03-30 19:28:06" "Jet" "HTTPMOD" "E10: 142.9  Cent" "E10"   "142.9" "Cent"
"2021-03-30 19:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10"   "141.400" "Cent"
"2021-03-30 20:24:06" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10"   "139.9" "Cent"
"2021-03-30 21:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10"   "139.9" "Cent"
"2021-03-30 22:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10"   "139.9" "Cent"
"2021-03-30 23:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10"   "139.9" "Cent"
« Letzte Änderung: 14 Mai 2021, 23:19:13 von RalfRog »

Offline RalfRog

  • Full Member
  • ***
  • Beiträge: 124
Ich habe noch zwei "reduceLog" Tests mit einem neu angelegten DBRep auf einen Teil der Daten gemacht.
Auch hier ist das Ergebnis nicht wie erwartet. Hier wird der ganze angegebene Zeitraum unvollständig reduziert (nicht nur der letzte Tag).

erster Test "reduceLog" mit Optionsangabe
LASTCMD  reduceLog 38:50 average=day INCLUDE=Jet:E10
zweiter Test "reduceLog" mit Attributen
LASTCMD reduceLog average=day

Attributes
device Jet
timestamp_begin 2021-03-30 00:00:00
timestamp_end 2021-04-06 23:59:59

Das Ergebnis ist für beide Varianten gleich. Hier der Log:
2021.05.14 23:11:07.093 3: DbRep DBRep1 - ################################################################
2021.05.14 23:11:07.094 3: DbRep DBRep1 - ###                    new reduceLog run                     ###
2021.05.14 23:11:07.094 3: DbRep DBRep1 - ################################################################
2021.05.14 23:11:07.262 3: DbRep DBRep1 - reduce data older than: 2021-04-06 23:59:59, newer than: 2021-03-30 00:00:00
2021.05.14 23:11:07.264 3: DbRep DBRep1 - reduceLog requested with options: AVERAGE=DAY INCLUDE -> Devs: Jet Readings: %
2021.05.14 23:11:07.345 3: DbRep DBRep1 - reduceLog deleting 8 records of day: 2021-03-30
2021.05.14 23:11:07.367 3: DbRep DBRep1 - reduceLog (hourly-average) updating 7 records of day: 2021-03-30
2021.05.14 23:11:07.394 3: DbRep DBRep1 - reduceLog deleting 11 records of day: 2021-03-31
2021.05.14 23:11:07.416 3: DbRep DBRep1 - reduceLog (hourly-average) updating 9 records of day: 2021-03-31
2021.05.14 23:11:07.444 3: DbRep DBRep1 - reduceLog deleting 6 records of day: 2021-04-01
2021.05.14 23:11:07.459 3: DbRep DBRep1 - reduceLog (hourly-average) updating 6 records of day: 2021-04-01
2021.05.14 23:11:07.484 3: DbRep DBRep1 - reduceLog deleting 6 records of day: 2021-04-02
2021.05.14 23:11:07.499 3: DbRep DBRep1 - reduceLog (hourly-average) updating 6 records of day: 2021-04-02
2021.05.14 23:11:07.524 3: DbRep DBRep1 - reduceLog deleting 8 records of day: 2021-04-03
2021.05.14 23:11:07.543 3: DbRep DBRep1 - reduceLog (hourly-average) updating 7 records of day: 2021-04-03
2021.05.14 23:11:07.569 3: DbRep DBRep1 - reduceLog deleting 8 records of day: 2021-04-04
2021.05.14 23:11:07.587 3: DbRep DBRep1 - reduceLog (hourly-average) updating 7 records of day: 2021-04-04
2021.05.14 23:11:07.613 3: DbRep DBRep1 - reduceLog deleting 5 records of day: 2021-04-05
2021.05.14 23:11:07.629 3: DbRep DBRep1 - reduceLog (hourly-average) updating 5 records of day: 2021-04-05
2021.05.14 23:11:07.651 3: DbRep DBRep1 - reduceLog deleting 8 records of day: 2021-04-06
2021.05.14 23:11:07.669 3: DbRep DBRep1 - reduceLog (hourly-average) updating 8 records of day: 2021-04-06
2021.05.14 23:11:07.697 3: DbRep DBRep1 - reduceLog finished. Rows processed: 253, deleted: 60, updated: 55

und die Daten exemplarisch für den 30.03, es gibt einige hourly-Werte aber keinen Tagesschnitt.
"TIMESTAMP" "DEVICE" "TYPE" "EVENT" "READING" "VALUE" "UNIT"
"2021-03-30 23:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10" "139.9" "Cent"
"2021-03-30 22:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10" "139.9" "Cent"
"2021-03-30 21:24:07" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10" "139.9" "Cent"
"2021-03-30 20:24:06" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10" "139.9" "Cent"
"2021-03-30 19:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "141.400" "Cent"
"2021-03-30 18:20:06" "Jet" "HTTPMOD" "E10: 139.9  Cent" "E10" "139.9" "Cent"
"2021-03-30 17:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "143.900" "Cent"
"2021-03-30 16:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "142.900" "Cent"
"2021-03-30 15:26:57" "Jet" "HTTPMOD" "E10: 143.9  Cent" "E10" "143.9" "Cent"
"2021-03-30 14:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "141.900" "Cent"
"2021-03-30 13:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "144.400" "Cent"
"2021-03-30 12:26:55" "Jet" "HTTPMOD" "E10: 144.9  Cent" "E10" "144.9" "Cent"
"2021-03-30 11:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "142.900" "Cent"
"2021-03-30 10:30:00" "Jet" "HTTPMOD" "rl_av_h" "E10" "149.233" "Cent"
"2021-03-30 09:26:52" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 08:26:51" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 07:26:51" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 06:26:50" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 05:26:49" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 04:26:48" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 03:26:48" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 02:26:47" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 01:26:46" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"
"2021-03-30 00:26:45" "Jet" "HTTPMOD" "E10: 153.9  Cent" "E10" "153.9" "Cent"

Der abschließenden Gegencheck auf dem DVLog-Device mit "reduceLogNbl" & "reduceLog" auf dem DVLog-Device führt in beiden Fällen zu dem unter #1 beschriebenen Ergebnis, dass nur der jüngste Tag unvollständig ist.
Insofern führen die Routinen in DBLog und DBRep zu unterschiedlich unvollständigen Ergebnissen.

« Letzte Änderung: 15 Mai 2021, 00:00:54 von RalfRog »

Offline RalfRog

  • Full Member
  • ***
  • Beiträge: 124
DS-Starter hat es auf seine TODO Liste gesetzt  :D