Anwendungsbeispiel JsonMod #2: CORONA Verbreitung

[herrmannj]

Nein, der jsonpath kann noch keine Aggregat Funktionen. Ist geplant aber ich bin anderweitig ausgelastet und kann keinen Termin nennen

[yersinia]

Falls nicht, gibt es so etwas wie eine avg() Funktion wie in dieser Java-Implementierung? https://github.com/json-path/JsonPath
Dann könnte ich es recht einfach in dem Modul berechen...

Nein, der jsonpath kann noch keine Aggregat Funktionen. Ist geplant aber ich bin anderweitig ausgelastet und kann keinen Termin nennen

Ich habe es in die JsonMod Wunschliste eingetragen.

[All-Ex]

Super, Danke 

[xavier]

Hello everybody,

I have to read the last and next to last element of an array.

I try to read this simple URL:
https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json

If I use the normal JSONPath syntax:

Code Auswählen Erweitern

$.[(@. length-1)].tamponi

or

Code Auswählen Erweitern

$.[-1:].tamponi

I get nothing, while the syntax (for example):

Code Auswählen Erweitern

$.[100].tamponi

is accepted and I get a value (the 100th element of the array).

How do I need to modify jsonPath to get a read?

This is the raw of my current code with some test reads:

Code Auswählen Erweitern

Internals:
   API_LAST_RES 1601018514.07323
   API__LAST_MSG 200
   DEF        https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json
   FUUID      5f6d12f4-f33f-ab34-7552-ffefc7e612223927
   NAME       coronaSpread
   NOTIFYDEV  global
   NR         128
   NTFY_ORDER 50-coronaSpread
   SOURCE     https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json (200)
   STATE      ???
   SVN        22651 2020-08-23 11:25:52 UTC
   TYPE       JsonMod
   CONFIG:
     IN_REQUEST 0
     SOURCE     https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json
   OLDREADINGS:
   READINGS:
     2020-09-25 09:21:54   data            2020-02-24T18:00:00
     2020-09-25 09:21:54   tamponi         4324
Attributes:
   interval   0 12 * * *
   readingList single(jsonPath('$..data'), 'data', 'none');
single(jsonPath('$..tamponi'), 'tamponi', 'none');
single(jsonPath('$.[-1:].tamponi), 'last_tamponi', 'none');
   room       99 System
   update-on-start 1

Finally, I apologize for my post in english with Google Translate but I write from Italy and don't know german.

[herrmannj]

Hi Xavier,

I wrote the JSONPath implementation, (there was none) but it is not 100 percent feature complete (now).

The first of your two examples are some rare features, its simply not implemented (for now). Lets do it this way: I seek to implement the second syntax next (I cannot promise to do that within a minute, planned leave ahead).

Code Auswählen Erweitern

$.[-1:].tamponi

does that work for you?

saluti all'Italia
Joerg

[yersinia]

As interim solution, you could extract just tamponi and data with a kind of index:

Code Auswählen Erweitern

multi(jsonPath('$.[*]'), concat('tamponi_', count()), property('.tamponi'));
multi(jsonPath('$.[*]'), concat('data_', count()), property('.data'));
You then may use a user reading to get the latest array (the one with the biggest index ).

If implemented, this should work out to get last two array elements:

Code Auswählen Erweitern

multi(jsonPath('$.[-2:]'), concat('tamponi_', count()), property('.tamponi'));

Btw, For JsonPath testing I use https://jsonpath.curiousconcept.com.

@herrmannj: I assumed you have implemented Stefan Goessners JsonPath implementation?

[xavier]

I wrote the JSONPath implementation, (there was none) but it is not 100 percent feature complete (now).

Wow, very fast reply!

I seek to implement the second syntax next (I cannot promise to do that within a minute, planned leave ahead).

Code Auswählen Erweitern

$.[-1:].tamponi

Implementing this feature would be wonderful. I'm waiting.

Last minute addition: Thanks @yersinia for his suggestions. Now I try.

Best regards.

[yersinia]

Regarding readings in ascending order, these readingList definition would be better (index first) imho:

Code Auswählen Erweitern

multi(jsonPath('$.[*]'), concat(count(), '_tamponi'), property('.tamponi'));
multi(jsonPath('$.[*]'), concat(count(), '_data'), property('.data'));

[herrmannj]

@herrmannj: I assumed you have implemented Stefan Goessners JsonPath implementation?

yes, but goessner sets the standard syntax and all implementations follow that (more or less;)). I do tests against the goessner files.

[xavier]

Regarding readings in ascending order, these readingList definition would be better (index first) imho:

Code Auswählen Erweitern

multi(jsonPath('$.[*]'), concat(count(), '_tamponi'), property('.tamponi'));
multi(jsonPath('$.[*]'), concat(count(), '_data'), property('.data'));

@yersinia: Hi, I tried your suggestion. It works, but it fills me with hundreds (!) of readings: today we got to 215.  A bit excessive, imho.

I can wait the herrmannj's code revision, just to have a more clean solution.

Best regards,
xavier.

[yersinia]

@yersinia: Hi, I tried your suggestion. It works, but it fills me with hundreds (!) of readings: today we got to 215.  A bit excessive, imho.

Yes this was just an interim solution proposal.

As interim solution, you could extract just tamponi and data with a kind of index:

Code Auswählen Erweitern

multi(jsonPath('$.[*]'), concat(count(), '_tamponi'), property('.tamponi'));
multi(jsonPath('$.[*]'), concat(count(), '_data'), property('.data'));
You then may use a user reading to get the latest array (the one with the biggest index ).

If implemented, this should work out to get last two array elements:

Code Auswählen Erweitern

multi(jsonPath('$.[-2:]'), concat(count(), '_tamponi'), property('.tamponi'));

[xavier]

Hi,

since the jsonMod update is still pending, I tried an intermediate solution.

Would it be possible to add the contents of $yday into the jsonPath?

I'll explain... the ideal solution would be:

Code Auswählen Erweitern

$.[-1:].tamponi

but at the moment it still can't be done.

So, my alternative is to calculate how many days have passed since the beginning of the year minus a certain value and insert it into the formula. For example to get today's tampons only (12/10/2020), it works like this:

Code Auswählen Erweitern

concat('$.[',230,'].tamponi')

If 230 could be calculated as $yday-55, a formula like this would work every day:

Code Auswählen Erweitern

concat('$.',{$yday-55},'. tamponi')

Unfortunately I don't know enough to get a correct value from $yday.

Can someone help me?

Attached raw full code.

Code Auswählen Erweitern

Internals:
   API_LAST_RES 1602537319.73396
   API__LAST_MSG 200
   DEF        https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json
   FUUID      5f6d12f4-f33f-ab34-7552-ffefc7e612223927
   NAME       coronaSpread
   NEXT       2020-10-13 17:10:00
   NR         131
   SOURCE     https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json (200)
   STATE      ???
   SVN        22854 2020-09-26 01:09:01 UTC
   TYPE       JsonMod
   CONFIG:
     IN_REQUEST 0
     SOURCE     https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json
     SECRET:
   READINGS:
     2020-10-12 23:15:19   corona_index    5.41
     2020-10-12 23:15:19   data_last       2020-10-12T17:00:00
     2020-10-12 23:15:19   data_prev       2020-10-11T17:00:00
     2020-10-12 23:15:19   deceduti_diff   39
     2020-10-12 23:15:19   deceduti_last   36205
     2020-10-12 23:15:19   deceduti_prev   36166
     2020-10-12 23:15:19   nuovi_positivi_diff -837
     2020-10-12 23:15:19   nuovi_positivi_last 4619
     2020-10-12 23:15:19   nuovi_positivi_prev 5456
     2020-10-12 23:15:19   tamponi_diff    85442
     2020-10-12 23:15:19   tamponi_last    12650155
     2020-10-12 23:15:19   tamponi_prev    12564713
     2020-10-12 23:15:19   terapia_intensiva_diff 32
     2020-10-12 23:15:19   terapia_intensiva_last 452
     2020-10-12 23:15:19   terapia_intensiva_prev 420
Attributes:
   comment    286 = days from the beginning of the year
   interval   10 17 * * *
   readingList single(jsonPath(concat('$.[',286-55,'].data')), 'data_last', 'none');
single(jsonPath(concat('$.[',286-56,'].data')), 'data_prev', 'none');
single(jsonPath(concat('$.[',286-55,'].nuovi_positivi')), 'nuovi_positivi_last', 'none');
single(jsonPath(concat('$.[',286-56,'].nuovi_positivi')), 'nuovi_positivi_prev', 'none');
single(jsonPath(concat('$.[',286-55,'].deceduti')), 'deceduti_last', 'none');
single(jsonPath(concat('$.[',286-56,'].deceduti')), 'deceduti_prev', 'none');
single(jsonPath(concat('$.[',286-55,'].terapia_intensiva')), 'terapia_intensiva_last', 'none');
single(jsonPath(concat('$.[',286-56,'].terapia_intensiva')), 'terapia_intensiva_prev', 'none');
single(jsonPath(concat('$.[',286-55,'].tamponi')), 'tamponi_last', 'none');
single(jsonPath(concat('$.[',286-56,'].tamponi')), 'tamponi_prev', 'none')
   room       99 System
   update-on-start 1
   userReadings nuovi_positivi_diff {ReadingsVal("coronaSpread","nuovi_positivi_last","")-ReadingsVal("coronaSpread","nuovi_positivi_prev","")},
deceduti_diff {ReadingsVal("coronaSpread","deceduti_last","")-ReadingsVal("coronaSpread","deceduti_prev","")},
terapia_intensiva_diff {ReadingsVal("coronaSpread","terapia_intensiva_last","")-ReadingsVal("coronaSpread","terapia_intensiva_prev","")},
tamponi_diff {ReadingsVal("coronaSpread","tamponi_last","")-ReadingsVal("coronaSpread","tamponi_prev","")},
corona_index {sprintf("%.2f",(ReadingsVal("coronaSpread","nuovi_positivi_last","") * 100) / ReadingsVal("coronaSpread","tamponi_diff",""))}

Best regards,
xavier.

[yersinia]

Hey xavier,

Just an idea, do not know if it works: maybe it is possible when you utilize myUtils with some perl code to update Attribute readingList of the device and initiate a reload.

The myUtils German wiki entry: https://wiki.fhem.de/wiki/99_myUtils_anlegen (within FHEM -> left hand navigation 'Edit files' -> there should be a 99_myUtils.pm editable via mouse click)
Add an own sub here to calculate day of the year

$yday is the day of the year, in the range 0..364 (or 0..365 in leap years.)

and update Attribute via fhem command like (not tested)

Code Auswählen Erweitern

fhem("attr coronaSpread readingList single(jsonPath(concat('$.[',".($yday-55).",'].data')), 'data_last', 'none');");
or similar.

As said, no idea if this will work but may worth a try.

[xavier]

Just an idea, do not know if it works: maybe it is possible when you utilize myUtils...[cut]

You are a genius !!

The solution is really simple. Thanks to your suggestions I have inserted this short code in 99_myUtils.pm:

Code Auswählen Erweitern

sub yday(){
my ($sec,$min,$hour,$mday,$mon,$year,$wday,$yday,$isdst) = localtime(time);
return $yday;
}

and slightly modified my define in fhem.cfg:

Code Auswählen Erweitern

define coronaSpread JsonMod https://raw.githubusercontent.com/pcm-dpc/COVID-19/master/dati-json/dpc-covid19-ita-andamento-nazionale.json
attr coronaSpread interval 10 17 * * *
attr coronaSpread comment yday() = days from the beginning of the year
attr coronaSpread readingList single(jsonPath(concat('$.[',yday()-54,'].data')), 'data_last', 'none');;\
single(jsonPath(concat('$.[',yday()-55,'].data')), 'data_prev', 'none');;\
single(jsonPath(concat('$.[',yday()-54,'].nuovi_positivi')), 'nuovi_positivi_last', 'none');;\
single(jsonPath(concat('$.[',yday()-55,'].nuovi_positivi')), 'nuovi_positivi_prev', 'none');;\
single(jsonPath(concat('$.[',yday()-54,'].deceduti')), 'deceduti_last', 'none');;\
single(jsonPath(concat('$.[',yday()-55,'].deceduti')), 'deceduti_prev', 'none');;\
single(jsonPath(concat('$.[',yday()-54,'].terapia_intensiva')), 'terapia_intensiva_last', 'none');;\
single(jsonPath(concat('$.[',yday()-55,'].terapia_intensiva')), 'terapia_intensiva_prev', 'none');;\
single(jsonPath(concat('$.[',yday()-54,'].tamponi')), 'tamponi_last', 'none');;\
single(jsonPath(concat('$.[',yday()-55,'].tamponi')), 'tamponi_prev', 'none')
attr coronaSpread room 99 System
attr coronaSpread update-on-start 1
attr coronaSpread userReadings nuovi_positivi_diff {ReadingsVal("coronaSpread","nuovi_positivi_last","")-ReadingsVal("coronaSpread","nuovi_positivi_prev","")},\
deceduti_diff {ReadingsVal("coronaSpread","deceduti_last","")-ReadingsVal("coronaSpread","deceduti_prev","")},\
terapia_intensiva_diff {ReadingsVal("coronaSpread","terapia_intensiva_last","")-ReadingsVal("coronaSpread","terapia_intensiva_prev","")},\
tamponi_diff {ReadingsVal("coronaSpread","tamponi_last","")-ReadingsVal("coronaSpread","tamponi_prev","")},\
corona_index {sprintf("%.2f",(ReadingsVal("coronaSpread","nuovi_positivi_last","") * 100) / ReadingsVal("coronaSpread","tamponi_diff",""))}

Now until December 31st I will finally be able to extract all the data I needed.

A sincere THANK YOU for your help.

Best regards,
xavier.

[yersinia]

Awesome, even easier than I was thinking of. Thanks for sharing!