Niederschlagsmengen von Website auslesen

[Fredi69]

Hallo,
hat jemand schon einmal die Daten von https://www.hnd.bayern.de/niederschlag/unterer_main/hammelburg-1997/tabelle abgegriffen und fhem eingelesen?
Ich würde mich über Infos freuen.

Fredi69

[buzzmain]

Moin,
das sieht mir nach einer Aufgabe für HTTPMOD aus.
https://wiki.fhem.de/wiki/HTTPMOD

Was genau hast Du vor?

[Fredi69]

Mein Wunsch wäre alle 5 Minuten den aktuellen Wert aus der Tabelle auszulesen und in fhem abzulegen um das in DBLog zu speichern.
Meine Kenntnisse in HTTPMOD sind dazu leider nicht ausreichend.

[steffen83]

Hallo Fredi69,

bist du schon weiter gekommen?
Ich habe mir heute die Niederschlagswerte von wetter.com gezogen.
Dabei geht es mir lediglich darum, ob es in der nächsten Zeit regnen soll.
Hinweis: Wenn du die werte von der Seite ziehen möchtest, dann musst du dir die Werte einzeln ziehen. Dies ist ein Mega aufwand.

Sofern dir das hier hilft (LIST)

Code Auswählen Erweitern

Internals:
   BUSY       0
   CFGFN     
   DEF        https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html 300
   FUUID      5ea964db-f33f-3260-6325-84ea9b22c5ae588c
   Interval   600
   LASTSEND   1588167810.47024
   MainURL    https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html
   ModuleVersion 3.5.22 - 7.2.2020
   NAME       wetter_com
   NOTIFYDEV  global
   NR         24982
   NTFY_ORDER 50-PM1
   STATE      Aktuell kein Niederschlag in Sicht
   TRIGGERTIME 1588168410.45967
   TRIGGERTIME_FMT 2020-04-29 15:53:30
   TYPE       HTTPMOD
   addr       https://at.wetter.com:443
   auth       0
   code       200
   compress   1
   conn       
   data       
   displayurl https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html
   header     
   host       at.wetter.com
   httpheader HTTP/1.1 200 OK
Accept-Ranges: bytes
Access-Control-Allow-Credentials: true
Access-Control-Allow-Origin: *
Age: 0
Cache-Control: public
Content-Encoding: gzip
Content-Type: text/html; charset=UTF-8
Date: Wed, 29 Apr 2020 13:43:30 GMT
Expires: Wed, 29 Apr 2020 13:48:30 GMT
grace: none
Vary: X-MOBILE-POC, X-PROGRAMMATIC, User-Agent, Accept-Encoding
X-Backend: blue
X-Cache: MISS
X-Cache-Hits: 0
X-Content-Type-Options: nosniff
X-Forwarded-Proto: https
X-Frame-Options: SAMEORIGIN
X-XSS-Protection: 1; mode=block
Connection: Close
   httpversion 1.0
   hu_blocking 0
   hu_filecount 13
   hu_port    443
   hu_portSfx
   ignoreredirects 1
   loglevel   4
   path       /deutschland/bottrop/eigen/DE2931904.html
   protocol   https
   redirects  0
   timeout    2
   url        https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html
   value      0
   CompiledRegexes:
   OLDREADINGS:
   QUEUE:
   READINGS:
     2020-04-29 15:43:31   LAST_REQUEST    update
     2020-04-29 15:43:31   MATCHED_READINGS Zeit1 Regen5 Zeit6 Regen6 next2h Regen1 Zeit2 Regen2 Zeit3 Regen3 Zeit4 Regen4 Zeit5
     2020-04-29 15:43:31   Regen1          Kein Niederschlag
     2020-04-29 15:43:31   Regen2          Kein Niederschlag
     2020-04-29 15:43:31   Regen3          Kein Niederschlag
     2020-04-29 15:43:31   Regen4          Kein Niederschlag
     2020-04-29 15:43:31   Regen5          Kein Niederschlag
     2020-04-29 15:43:31   Regen6          Kein Niederschlag
     2020-04-29 15:43:31   UNMATCHED_READINGS
     2020-04-29 15:43:31   Zeit1           15:35
     2020-04-29 15:43:31   Zeit2           15:40
     2020-04-29 15:43:31   Zeit3           15:45
     2020-04-29 15:43:31   Zeit4           15:50
     2020-04-29 15:43:31   Zeit5           15:55
     2020-04-29 15:43:31   Zeit6           16:00
     2020-04-29 15:43:31   next2h          Aktuell kein Niederschlag in Sicht
   REQUEST:
     data       
     header     
     ignoreredirects 0
     retryCount 0
     type       update
     url        https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html
     value      0
   defptr:
     readingBase:
       Regen1     reading
       Regen2     reading
       Regen3     reading
       Regen4     reading
       Regen5     reading
       Regen6     reading
       Zeit1      reading
       Zeit2      reading
       Zeit3      reading
       Zeit4      reading
       Zeit5      reading
       Zeit6      reading
       next2      reading
       next2h     reading
     readingNum:
       Regen1     2
       Regen2     4
       Regen3     6
       Regen4     8
       Regen5     10
       Regen6     12
       Zeit1      1
       Zeit2      3
       Zeit3      5
       Zeit4      7
       Zeit5      9
       Zeit6      11
       next2      13
       next2h     13
     readingOutdated:
     requestReadings:
       get1:
         Regen1     reading 2
         Regen2     reading 4
         Regen3     reading 6
         Regen4     reading 8
         Regen5     reading 10
         Regen6     reading 12
         Zeit1      reading 1
         Zeit2      reading 3
         Zeit3      reading 5
         Zeit4      reading 7
         Zeit5      reading 9
         Zeit6      reading 11
         next2      reading 13
         next2h     reading 13
       update:
         Regen1     reading 2
         Regen2     reading 4
         Regen3     reading 6
         Regen4     reading 8
         Regen5     reading 10
         Regen6     reading 12
         Zeit1      reading 1
         Zeit2      reading 3
         Zeit3      reading 5
         Zeit4      reading 7
         Zeit5      reading 9
         Zeit6      reading 11
         next2      reading 13
         next2h     reading 13
   sslargs:
Attributes:
   event-on-change-reading Regen1 Regen2 Regen3 Regen4 Regen5 Regen6 next2h
   reading10Name Regen5
   reading10Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading11Name Zeit6
   reading11Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   reading12Name Regen6
   reading12Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading13Name next2h
   reading13Regex Niederschläge der nächsten 2 Stunden[\w\W]*?portable-pt-- ]">\s*(.*?)\s*<\/div>
   reading1Name Zeit1
   reading1Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   reading2Name Regen1
   reading2Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading3Name Zeit2
   reading3Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   reading4Name Regen2
   reading4Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading5Name Zeit3
   reading5Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   reading6Name Regen3
   reading6Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading7Name Zeit4
   reading7Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   reading8Name Regen4
   reading8Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
   reading9Name Zeit5
   reading9Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
   room       Wettervorhersage
   showBody   0
   showMatched 1
   stateFormat next2h
   userattr   get1CheckAllReadings:0,1 get1Name reading0Name reading0Regex reading10Name reading10Regex reading11Name reading11Regex reading12Name reading12Regex reading13Name reading13Regex reading1Name reading1Regex reading2Name reading2Regex reading3Name reading3Regex reading4Name reading4Regex reading5Name reading5Regex reading6Name reading6Regex reading7Name reading7Regex reading8Name reading8Regex reading9Name reading9Regex

und als RAW:

Code Auswählen Erweitern

defmod wetter_com HTTPMOD https://at.wetter.com/deutschland/bottrop/eigen/DE2931904.html 300
attr wetter_com userattr get1CheckAllReadings:0,1 get1Name reading0Name reading0Regex reading10Name reading10Regex reading11Name reading11Regex reading12Name reading12Regex reading13Name reading13Regex reading1Name reading1Regex reading2Name reading2Regex reading3Name reading3Regex reading4Name reading4Regex reading5Name reading5Regex reading6Name reading6Regex reading7Name reading7Regex reading8Name reading8Regex reading9Name reading9Regex
attr wetter_com event-on-change-reading Regen1 Regen2 Regen3 Regen4 Regen5 Regen6 next2h
attr wetter_com reading10Name Regen5
attr wetter_com reading10Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading11Name Zeit6
attr wetter_com reading11Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com reading12Name Regen6
attr wetter_com reading12Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading13Name next2h
attr wetter_com reading13Regex Niederschläge der nächsten 2 Stunden[\w\W]*?portable-pt-- ]">\s*(.*?)\s*<\/div>
attr wetter_com reading1Name Zeit1
attr wetter_com reading1Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com reading2Name Regen1
attr wetter_com reading2Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading3Name Zeit2
attr wetter_com reading3Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com reading4Name Regen2
attr wetter_com reading4Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading5Name Zeit3
attr wetter_com reading5Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com reading6Name Regen3
attr wetter_com reading6Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading7Name Zeit4
attr wetter_com reading7Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com reading8Name Regen4
attr wetter_com reading8Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?rainnowcast-color[\w\W]*?<\/span>\s*(.*?)\s*<\/div>
attr wetter_com reading9Name Zeit5
attr wetter_com reading9Regex <div data-WetterNowcast data-nowcast-lat="51.5435" data-nowcast-lng="6.9228">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">[\w\W]*?portable-show-stuff ]">\s*(.*?)\s*<span class=
attr wetter_com room Wettervorhersage
attr wetter_com showBody 0
attr wetter_com showMatched 1
attr wetter_com stateFormat next2h

setstate wetter_com Aktuell kein Niederschlag in Sicht
setstate wetter_com 2020-04-29 15:43:31 LAST_REQUEST update
setstate wetter_com 2020-04-29 15:43:31 MATCHED_READINGS Zeit1 Regen5 Zeit6 Regen6 next2h Regen1 Zeit2 Regen2 Zeit3 Regen3 Zeit4 Regen4 Zeit5
setstate wetter_com 2020-04-29 15:43:31 Regen1 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 Regen2 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 Regen3 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 Regen4 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 Regen5 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 Regen6 Kein Niederschlag
setstate wetter_com 2020-04-29 15:43:31 UNMATCHED_READINGS
setstate wetter_com 2020-04-29 15:43:31 Zeit1 15:35
setstate wetter_com 2020-04-29 15:43:31 Zeit2 15:40
setstate wetter_com 2020-04-29 15:43:31 Zeit3 15:45
setstate wetter_com 2020-04-29 15:43:31 Zeit4 15:50
setstate wetter_com 2020-04-29 15:43:31 Zeit5 15:55
setstate wetter_com 2020-04-29 15:43:31 Zeit6 16:00
setstate wetter_com 2020-04-29 15:43:31 next2h Aktuell kein Niederschlag in Sicht



[betateilchen]

Dazu braucht es kein HTTPMOD, solche strukturierten Daten lassen sich einfach in der 99_myUtils.pm verarbeiten.

Code Auswählen Erweitern



sub test {
   use HTML::TableExtract;
   my $url = "https://www.hnd.bayern.de/niederschlag/unterer_main/hammelburg-1997/tabelle";
   my $html_string = GetFileFromURL($url);
   my $te = HTML::TableExtract->new( headers => [qw(Datum Niederschlag)]);
   $te->parse($html_string);
   foreach my $row ($te->rows) {
      Debug join(',', @$row);
   }
}



liefert als Ergebnis:

Code Auswählen Erweitern


2020.04.29 16:39:18 1: DEBUG>29.04.2020 16:25,0
2020.04.29 16:39:18 1: DEBUG>29.04.2020 16:20,0
2020.04.29 16:39:18 1: DEBUG>29.04.2020 16:15,0
2020.04.29 16:39:18 1: DEBUG>29.04.2020 16:10,0
2020.04.29 16:39:18 1: DEBUG>29.04.2020 16:05,0


In der "Debug" Zeile kann man mit den Werten natürlich auch andere Dinge machen, z.B. loggen oder readings setzen.

[betateilchen]

Mein Wunsch wäre alle 5 Minuten den aktuellen Wert aus der Tabelle auszulesen

Ob eine Abfrage alle 5 Minuten aber Sinn macht, wage ich sehr zu bezweifeln. Solche Bombardements von Webservern führen häufig dazu, dass die Diensteanbieter darauf reagieren und die Zugriffe blockieren. Hatten wir hier im Forum schön öfters.

Eine Abfrage einmal pro Stunde wäre doch völlig ausreichend, zumal die Tabelle ja selbst eine gewisse Historie führt. Man würde also einmal pro Stunde den Abruf machen und die ersten (neuesten) 12 Werte der Tabelle verarbeiten.