[FHZ] Extending FHEM: InternalTimer()?

Begonnen von Guest, 10 November 2009, 18:08:58

Vorheriges Thema - Nächstes Thema
Drucken
Nach unten Seiten1
Benutzer-Aktionen

Guest

10 November 2009, 18:08:58
Originally posted by: <email address deleted>

Hi,

a quick question: I started interfacing my WS3600 weather station to FHEM, but
instead of hacking the Perl scripts which deal with it's little sister WS2300,
I want to rely on the binaries from open2300/open3600 projects (available for
Linux and Windows). Thus, basically I will call

   open($FH, "/path/to/fetch3600-binary |"); while(<$FH>) {}; close ($FH);

about every minute; on my current system, "fetch3600" takes ~6 seconds of real
time to complete (reading all 87 data points the WS3600 has to offer). (As for
"every minute": sensor data is updated every 128 seconds normally, except for
wind speed and direction about 10 km/h, interval then is 32 seconds -- refer
to http://www.wetterstationsforum.de/ws3600_master-touch.php for details. It
might be worth to make the interval 64 instead of 60 seconds though.)

I modelled this around 70_SCIVT.pm which shedules it's internal call-back via
InternalTimer(); commandref states: "Defining an SCIVT device will schedule an
internal task". Question is: will doing it this way hold processing of other
events for the duration of reading the data from WS3600 (or, more precisely,
from the pipe to /path/to/fetch3600-binary)? If so, what's the preferred way
of dealing with "slow to read" sensors/devices?

Regards,
         kai

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

rudolfkoenig

#1
10 November 2009, 23:49:23
                                                   

> Question is: will doing it this way hold processing of other events for the
> duration of reading the data from WS3600 (or, more precisely, from the pipe
> to /path/to/fetch3600-binary)?

Yes.

> If so, what's the preferred way of dealing with "slow to read"
> sensors/devices?

Setting $hash->{FD} and inserting $hash in $selectlist. See 00_CUL.pm,
00_FHZ.pm, 00_CM11.pm, 00_LIRC.pm and 87_WS2000.pm

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

Dennis

#2
11 November 2009, 00:15:43
Originally posted by: <email address deleted>

Rudolf Koenig wrote:

>> If so, what's the preferred way of dealing with "slow to read"
>> sensors/devices?
>
> Setting $hash->{FD} and inserting $hash in $selectlist. See 00_CUL.pm,
> 00_FHZ.pm, 00_CM11.pm, 00_LIRC.pm and 87_WS2000.pm

Err, all those are doing more or less the following, if I'm not
overlooking something:

_Define() {
   $whatever=OpenSerialOrNetwork();
   ...
   $hash->{FD} = $whatever->FILENO;
   ...
}

_Undef() {
   ...
   close_what_was_opened($hash->{FD});
   ...
}


SOMEWHERE() {
   ...
   read_from($hash->{FD});
   ...
}


I don't have long standing open file descriptors. I start at a pre-
defined interval a read on an Perl file handle from open($FH, "cmd |").
That is Perl starts a command in the background and I read that cmds
output from within the FHEM process. On EOF, file handle is close()d
and gone for good.

I don't interface on serial level and there's no daemon to provide
the current data on time to a caller. I interface on stdin, running
a commandline query tool any time I want fresh data. I don't see yet
how to adapt this to the mentioned method?

Thanks,
         kai


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

Guest

#3
11 November 2009, 00:27:25
Originally posted by: <email address deleted>

> I don't interface on serial level and there's no daemon to provide
> the current data on time to a caller. I interface on stdin, running
> a commandline query tool any time I want fresh data. I don't see yet
> how to adapt this to the mentioned method?

To make myself clear: Basically, I do the same as 88_IPWE.pm, just
without network overhead.
         kai


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

rudolfkoenig

#4
11 November 2009, 08:28:32
                                                   

> I don't have long standing open file descriptors. I start at a pre-
> defined interval a read on an Perl file handle from open($FH, "cmd |").

I am fully aware of your situation.

At the beginning of the operation you define $hash->{FD} and insert $hash into
the %selectlist. Then your ReadFn function is called whenever there is data,
and you are only allowed a single read to avoid blocking, so you have to
remember your context. If you're done you remove your $hash from the
%selectlist.

As an alternative you fork a separate process, read the data in a loop, write
it to a temporary file, and then notify the parent e.g. by sending him a "get
data ". Or something similar.

None of the alternatives is nice, but this is the downside of having a
single-threaded model. On the other side there are a lot of advantages which
are more important in my opinion.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

rudolfkoenig

#5
11 November 2009, 09:06:16
Originally posted by: <email address deleted>

Rudolf Koenig wrote:
>> I don't have long standing open file descriptors. I start at a pre-
>> defined interval a read on an Perl file handle from open($FH, "cmd |").
>
> I am fully aware of your situation.

;)

> At the beginning of the operation you define $hash->{FD} and insert $hash into
> the %selectlist. Then your ReadFn function is called whenever there is data,
> and you are only allowed a single read to avoid blocking, so you have to
> remember your context. If you're done you remove your $hash from the
> %selectlist.

Ah. Now I understand; it's a shame no module implemented this before so
I could just reuse that code ;) While that would be a more elegant (and
more portable) solution, I most likely will go the other way:

> As an alternative you fork a separate process, read the data in a loop, write
> it to a temporary file, and then notify the parent e.g. by sending him a "get
> data ". Or something similar.

This basically requires a cron job outside of FHEM and the move of the
read-loop to my _Get() subroutine. Is reading 80+ lines from a file in
one go ok regarding the overall timings within FHEM? (Sure, doing this
with %selectlist would be nicer ;), but is that *neccessary* to prevent
data loss from other sources?)

> None of the alternatives is nice, but this is the downside of having a
> single-threaded model. On the other side there are a lot of advantages which
> are more important in my opinion.

I agree.
         kai

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

rudolfkoenig

#6
11 November 2009, 09:18:00
                                                   

> This basically requires a cron job outside of FHEM and the move of the
> read-loop to my _Get() subroutine.

You should use the InternalTimer instead of a cron job to make the setup easier
for the user.

> Is reading 80+ lines from a file in one go ok regarding the overall timings
> within FHEM?

As the other process just finished writing and the data is small, it is
probably completely in the OS cache, so reading it should be <1ms.

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=en
-~----------~----~----~----~------~----~------~--~-

UliM

#7
13 November 2009, 00:29:36
Originally posted by: <email address deleted>

Rewrote the code, hit a problem ;)

Rudolf Koenig wrote:
> At the beginning of the operation you define $hash->{FD} and insert $hash into
> the %selectlist. Then your ReadFn function is called whenever there is data,
> and you are only allowed a single read to avoid blocking, so you have to
> remember your context. If you're done you remove your $hash from the
> %selectlist.

sub WS3600_Initialize($) {
    ...
    $hash->{DefFn}   = "WS3600_Define";
    $hash->{ReadFn}  = "WS3600_Read";
    $hash->{UndefFn} = "WS3600_Undef";
    ...
}

sub WS3600_Define($$) {
    ...

    InternalTimer(gettimeofday()+ $hash->{Timer}, "WS3600_GetStatus", $hash, 1);
    $hash->{STATE} = "initialized";
    return undef;
}

sub WS3600_GetStatus($) {
    ...
    Log 3, "WS3600 contacting station";

    open($FH, $dev);
    if(!$FH) {
       return "WS3600 Can't start $dev: $!";
    }

    $hash->{FD}=$FH;
    $selectlist{"$name.pipe"} = $hash;
    Log 3, "WS3600 pipe opened";
    $hash->{STATE} = "running";
    return $hash->{STATE};
}

sub WS3600_Read($) {

    ...
    if(!defined($hash->{FD})) {
       return undef;
    }
    $FH = $hash->{FD};

    ...
    if( $inputline = <$FH>) {
       $inputline =~ s/\s+$//;
       my ($reading, $val)=split(/ /, $inputline);
       if(defined($TranslatedCodes{$reading})) {
       Log 3, "WS3600 read $reading:$val";
    ...
       $hash->{STATE} = "reading";
    } else {
       close($FH);
       Log 3, "WS3600 done reading pipe";
       DoTrigger($name, undef);
       delete $hash->{FD};
       undef($selectlist{"$name.pipe"});
       $hash->{STATE} = "read";
       InternalTimer(gettimeofday()+ $hash->{Timer}, "WS3600_GetStatus", $hash, 1);
    }
     return $hash->{STATE};
}


Well, it works, but far beyond expectation:

2009.11.13 00:08:26 3: WS3600 contacting station
2009.11.13 00:08:26 3: WS3600 pipe opened
2009.11.13 00:08:30 3: WS3600 read Date:13-Nov-2009
2009.11.13 00:08:33 3: WS3600 read Time:00:08:30
2009.11.13 00:08:34 3: WS3600 read Ti:22.2
2009.11.13 00:08:36 3: WS3600 read Timin:20.8
2009.11.13 00:08:55 3: WS3600 read Timax:27.9
2009.11.13 00:08:55 3: WS3600 read TTimin:10:27
2009.11.13 00:08:59 3: WS3600 read DTimin:15-10-2009
2009.11.13 00:09:00 3: WS3600 read TTimax:23:31
2009.11.13 00:09:12 3: WS3600 read DTimax:20-08-2009
2009.11.13 00:09:13 3: WS3600 read To:7.2
2009.11.13 00:09:14 3: WS3600 read Tomin:-0.4
2009.11.13 00:09:19 3: WS3600 read Tomax:35.6
2009.11.13 00:09:19 3: WS3600 read TTomin:07:03
2009.11.13 00:09:20 3: WS3600 read DTomin:15-10-2009
2009.11.13 00:09:31 3: WS3600 read TTomax:16:52
2009.11.13 00:09:31 3: WS3600 read DTomax:20-08-2009
2009.11.13 00:09:32 3: WS3600 read DP:5.8
[...]
2009.11.13 00:16:27 3: WS3600 read DRPmax:11-09-2009
2009.11.13 00:16:34 3: WS3600 read Tendency:Rising
2009.11.13 00:16:38 3: WS3600 read Forecast:Sunny
2009.11.13 00:16:44 3: WS3600 done reading pipe

In other words: it takes AGES for my ReadFn to get called again. In
between invocation, I could have read in anything :( It's totally use-
less this way; what am i doing wrong?

And, as it rains ...

2009.11.13 00:17:49 0: Strange call for nonexistent : ReadFn
Use of uninitialized value in vec at /usr/local/bin/fhem.pl line 296.
Use of uninitialized value in vec at /usr/local/bin/fhem.pl line 321.
Use of uninitialized value $d in hash element at /usr/local/bin/fhem.pl line 1907.
Use of uninitialized value $d in concatenation (.) or string at /usr/local/bin/fhem.pl line 1908.

after "done reading pipe", I think my version of "remove your
$hash from the %selectlist", i. e. undef($selectlist{"$name.pipe"}), does
not work as expected ;)


Help, anyone?
         kai

--

You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com.
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=.
RPi4/Raspbian, CUL V3 (ca. 30 HomeMatic-devices), LAN (HarmonyHub, alexa etc.).  Fördermitglied des FHEM e.V.

rudolfkoenig

#8
13 November 2009, 08:38:20
                                                   

>     if( $inputline = <$FH>) {

This may block longer than expected as it will read (blocking) until a \n is
encountered. The function to use is sysread. But I don't think this is the
cause of the described problem, it "only" makes the use of select superfluous.


> In other words: it takes AGES for my ReadFn to get called again. In
> between invocation, I could have read in anything :( It's totally use-
> less this way; what am i doing wrong?

I suspect that one of the other ReadFn/ReadyFn functions is blocking. Put some
debug lines in fhem.pl around the ReadFn and ReadyFn calls.


> 2009.11.13 00:17:49 0: Strange call for nonexistent : ReadFn

I think only more debugging would help here, as I do not see the problem yet.

--

You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com.
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=.

Guest

#9
13 November 2009, 14:44:29
Originally posted by: <email address deleted>

Rudolf Koenig wrote:
>>     if( $inputline = <$FH>) {
>
> This may block longer than expected as it will read (blocking) until a \n is
> encountered. The function to use is sysread. But I don't think this is the
> cause of the described problem, it "only" makes the use of select superfluous.

No, not the cause. I rewrote the stuff to open the pipe, shedule my ReadFn-
function to be called in 6 seconds, reading pipe's whole stdout in one go,
close pipe and shedule opening function to be called in 60 seconds ...

Works so far:

2009.11.13 14:37:24 3: WS3600 contacting station
2009.11.13 14:37:24 3: WS3600 pipe opened
2009.11.13 14:37:30 3: WS3600 Read entered
2009.11.13 14:37:30 3: WS3600 start reading
2009.11.13 14:37:30 3: WS3600 done reading pipe
2009.11.13 14:37:30 5: Triggering WS3600 (4 changes)

But it still could block, unfortunately :(

>> In other words: it takes AGES for my ReadFn to get called again. In
>> between invocation, I could have read in anything :( It's totally use-
>> less this way; what am i doing wrong?
>
> I suspect that one of the other ReadFn/ReadyFn functions is blocking. Put some
> debug lines in fhem.pl around the ReadFn and ReadyFn calls.

Yepp, I will do that after the next update I install from the CVS, i. e. one of
these days.

>> 2009.11.13 00:17:49 0: Strange call for nonexistent : ReadFn
>
> I think only more debugging would help here, as I do not see the problem yet.

That was simply caused by my Perl ignorance, I suspect. My implementation
of "remove your handle from %selectlist" was obviously gargabe, with
»delete $selectlist{"$name.pipe"};« I haven't seen these messages yet.
         kai

--

You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com.
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=.

mangei.markus

#10
14 November 2009, 11:41:21
Originally posted by: <email address deleted>

Salve,

quoting myself:
> Rudolf Koenig wrote:
>>>     if( $inputline = <$FH>) {
>> This may block longer than expected as it will read (blocking) until a \n is
>> encountered. The function to use is sysread. But I don't think this is the
>> cause of the described problem, it "only" makes the use of select superfluous.
>
> No, not the cause. I rewrote the stuff to open the pipe, shedule my ReadFn-
> function to be called in 6 seconds, reading pipe's whole stdout in one go,
> close pipe and shedule opening function to be called in 60 seconds ...
>
> Works so far:
>
> 2009.11.13 14:37:24 3: WS3600 contacting station
> 2009.11.13 14:37:24 3: WS3600 pipe opened
> 2009.11.13 14:37:30 3: WS3600 Read entered
> 2009.11.13 14:37:30 3: WS3600 start reading
> 2009.11.13 14:37:30 3: WS3600 done reading pipe
> 2009.11.13 14:37:30 5: Triggering WS3600 (4 changes)
>
> But it still could block, unfortunately :(
>
>>> In other words: it takes AGES for my ReadFn to get called again. In
>>> between invocation, I could have read in anything :( It's totally use-
>>> less this way; what am i doing wrong?
>> I suspect that one of the other ReadFn/ReadyFn functions is blocking. Put some
>> debug lines in fhem.pl around the ReadFn and ReadyFn calls.
>
> Yepp, I will do that after the next update I install from the CVS, i. e. one of
> these days.

I reworked the code again, now with %selectlist again but a blocking-
aware ReadFn. The timing issues I noticed earlier seem to be caused by
a problem with Perl/FHEM noticing that my pipe is ready to be read from
(or fhem.pl spending time elsewhere; I put Log() around the calls to
ReadFn & ReadyFn in fhem.pl, the delays therefore seem to be coming from
outside of these functions).

Example (ommisions only where [...] is):

2009.11.14 11:18:31 3: WS3600 contacting station
2009.11.14 11:18:31 3: WS3600 pipe opened
2009.11.14 11:18:56 5: Calling ReadFn of WS3600
2009.11.14 11:18:56 3: WS3600 Read entered
2009.11.14 11:18:56 3: WS3600 start reading
2009.11.14 11:18:56 3: WS3600 ended reading (eof=0)
2009.11.14 11:18:56 3: WS3600 read Date 14-Nov-2009:Date:14-Nov-2009
2009.11.14 11:18:56 3: WS3600 read Time 11:18:34:Time:11:18:34
[...]
2009.11.14 11:18:56 3: WS3600 read Forecast Cloudy:Forecast:Cloudy
2009.11.14 11:18:56 3: WS3600 reading would block
2009.11.14 11:18:56 5: ReadFn returned
2009.11.14 11:18:57 5: exec at command at_get_CUN_version
2009.11.14 11:18:57 5: Cmd: >{ \
    fhem("get CUL2 version"); \
}<
2009.11.14 11:18:57 5: Cmd: >get CUL2 version<
2009.11.14 11:18:57 5: CUL/RAW: V 1.31.4 CUN

2009.11.14 11:18:57 3: CUL2 version => V 1.31.4 CUN
2009.11.14 11:18:57 5: redefine at command at_get_CUN_version as +*00:01 { \
    fhem("get CUL2 version"); \
}
2009.11.14 11:18:57 5: Calling ReadFn of CUL2
2009.11.14 11:18:57 5: CUL/RAW: /T320242690D32

2009.11.14 11:18:57 3: CUL2: T320242690D -49
2009.11.14 11:18:57 5: CUL2 dispatch 810c04xx0909a00132024200690d
2009.11.14 11:18:57 5: ReadFn returned
2009.11.14 11:18:57 5: Calling ReadFn of WS3600
2009.11.14 11:18:57 3: WS3600 Read entered
2009.11.14 11:18:57 3: WS3600 start reading
2009.11.14 11:18:57 3: WS3600 ended reading (eof=1)
Use of uninitialized value $rawreading in concatenation (.) or string at /usr/local/lib/FHEM/70_WS3600.pm line 263.
Use of uninitialized value $val in concatenation (.) or string at /usr/local/lib/FHEM/70_WS3600.pm line 263.
2009.11.14 11:18:57 3: WS3600 read ::
Use of uninitialized value $rawreading in hash element at /usr/local/lib/FHEM/70_WS3600.pm line 264.
2009.11.14 11:18:57 3: WS3600 done reading pipe
2009.11.14 11:18:57 5: Triggering WS3600 (1 changes)
2009.11.14 11:18:57 5: WS3600 trigger: Checking FMS1Log for notify
2009.11.14 11:18:57 5: WS3600 trigger: Checking HGS1Log for notify
2009.11.14 11:18:57 5: WS3600 trigger: Checking Logfile for notify
2009.11.14 11:18:57 5: WS3600 trigger: Checking RRDNotify for notify
2009.11.14 11:18:57 5: Cmd: >"/usr/local/bin/fhem2rrd.pl "WS3600" "WS3600" "Status: updated""<
2009.11.14 11:18:57 5: WS3600 trigger: Checking TestNotify for notify
2009.11.14 11:18:57 5: Cmd: >"/usr/bin/logger "WS3600 - Status: updated (WS3600)""<
2009.11.14 11:18:57 5: WS3600 trigger: Checking at_FMS1_Off for notify
2009.11.14 11:18:57 5: WS3600 trigger: Checking at_FMS1_On for notify
2009.11.14 11:18:57 5: ReadFn returned

I doubt that it took 25 seconds for the command to become ready to
be read from, the (DCF77-synchronized) reading "Time 11:18:34" fur-
ther suggests that the pipe opened at 11:18:31 was ready at 11:18:34.

(The "uninitialized value" errors are from crude debugging code; as
the pipe seems to block just before delivering EOF, the loop gets
called with empty $line ("containing" only EOF), which then gets
split() on " " ... Sue me ;))


Any ideas on where to look further?
         kai

--

You received this message because you are subscribed to the Google Groups "FHEM users" group.
To post to this group, send email to fhem-users@googlegroups.com.
To unsubscribe from this group, send email to fhem-users+unsubscribe@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/fhem-users?hl=.
Drucken
Nach oben Seiten1
Benutzer-Aktionen