THZ Tecalor (LWZ Stiebel Eltron) module support and code improvement.

[sunrise]

I have another/different question (hence, as a new comment):

On my RPI/ssh, I try to sniff the traffic over the serial port:

Code Auswählen Erweitern

tail -f /dev/ttyUSB0

But both as user as well as root, nothing happens, i.e. no messages appear - even during reading values via the FHEM GUI.
The cable with the THZ is connected to my RPI via USB-RS232 adapter (hence, /dev/ttyUSB0, as defined in FHEM and else working nicely).

Do you have some hints if I am "sniffing" not properly, or what else could cause the lack of messages.

I intend to see the raw bytes being transferred via the serial port when doing readings via the FHEM GUI or via the THZ's control panel/LCD.

[sunrise]

Hmm, I kind of "stumbled" over this in the FHEM/THZ commandref:

Example: direct connection

    define Mytecalor THZ /dev/ttyUSB0@115200

or network connection (like via ser2net)

    define Myremotetecalor THZ 192.168.0.244:2323
    define Mythz THZ /dev/ttyUSB0@115200

FHEM and ser2net are running on my RPI and the USB/RS232 connector is on the RPI as well. Content of /etc/ser2net on my RPI:

Code Auswählen Erweitern


BANNER:banner:\r\nser2net port \p device \d [\s] (Debian GNU/Linux)\r\n\r\n
2000:telnet:600:/dev/ttyUSB0:9600 8DATABITS NONE 1STOPBIT banner


So far, so good, or not?

In FHEM, I have this under Internals:

Code Auswählen Erweitern


DEF    /dev/ttyUSB0@9600
DeviceName    /dev/ttyUSB0@9600


The device should be ok, as "dmesg | grep USB" shows "pl2303 converter now attached to ttyUSB0".
The baudrate (9600) is the same as set in ser2net (shown above).
Do you agree that until here, everything is set up correctly?

I wonder, because the quote from the commandref mentions for network connection (via ser2net) 2 different "define" lines, the first with the IP:port number of the device (actually, which one: where FHEM runs or where the cable is connected?), the second with /dev/ttyUSB0@<baudrate>. And this puzzles me, as I do not know what these 2 different scenarios mean and if my described setting is correct in light of this.

I hope that I wrote this in an understandable way - if not, please bear with me and give me a sign. 
Thanks a lot and a great weekend to all of you for all your efforts and help! 


PS:

My bad! Of course, I only use the ser2net in order to be able to do something from my remote Windows 10 machine (running a virtual port). Since FHEM and USB/RS232 run on the RPI, I have nothing to do with ser2net when logged into the RPI via ssh, so please ignore my last question about ser2net - unless you still see something that I shall look into.

What remains is still my question, how I could properly sniff the Rx/Tx directly on the RPI (tail -f /dev/ttyUSB0 does not seem to work).

[immi]

Hi sunrise
trust me, stay away from nonblocking in fhem and thz, for now; if you find it usefull we can discuss on it after you have implemented all your registers.

back to sniffing. there are several packet monitoring and ser2net can also log the traffic.
I did something different and trivial: I redirected the traffic via a proxy in perl, which you can find attached

Windows maschine virtual port  --> linux computer with proxy.pl --> ser2net on another linux computer

the proxy and the ser2net can be also the same computer, but you should pay attention to use different ports

immi

[sunrise]

Hi immi,

There were hundreds of "too slow" messages in the log, but I am happy to follow your recommendations. "First things first" So I set nonblocking to 0. Do I have to change also simpleReadTimeout (currently set to or just leave it for the time being?

"Windows maschine virtual port":
Do you mean something like this?
https://www.hw-group.com/software/hw-vsp3-virtual-serial-port

I use this on my Windows 10 machine to read/write data via ser2net and a virtual COM port. Is that what you also mean?

I would put the Perl script on my RPI that also runs the FHEM service and ser2net. Thanks also for the reminder regarding the different ports.

According to this...
https://catonmat.net/perl-tcp-proxy

... the Perl script is run like this (different use case):

Code Auswählen Erweitern

sudo ./tcp-proxy2.pl 443 friends_server.com:22

Translated to our case, would it be like this?

Code Auswählen Erweitern


sudo ./proxy.pl <port to sniff> localhost:<port of ser2net, in my case 2000>


Sorry, but I am really unsure how this should be done.
A big THANK YOU! 

[immi]

while you use the windows service software, you have to stop fhem-thz

raspi e.g. 10.0.1.10
ser2net runnimg on 2000
sudo ./proxy.pl 3000 localhost:2000

windows e.g. 10.0.1.11
virtual-serial-port mapping  10.0.1.11:3000 to com4
service software on com4

[sunrise]

Great, thanks! When I got this working and have first sniffing results, I will add something in the Wiki for other users who might benefit from this, too (for new firmwares or newer Tecalor/SE devices).

[sunrise]

Hi immi,

Please excuse my additional question: Comparing your script vs. the one from that website, I see that you added a few lines (I left the commented out lines away) and wonder what your additions mean:

Code Auswählen Erweitern


my $data =  uc(unpack('H*', $buffer));
if (length($data) > 20 ) { $data="0100E5F4FFF2004901BAFFDE01D2011401C00001000110100800650200000000DB0000160000ED00000016001003" }
open (MYFILE, '>>data.txt');
print MYFILE ($data );
close (MYFILE);
$buffer=  pack('H*', $data);


You commented out several if clauses with different $data. What is this variable about and why have you tried (and commented out) different values? I am willing to learn, hence my questions. 

[immi]

Hi Sunrise

Code Auswählen Erweitern

my $data =  uc(unpack('H*', $buffer));
convert binary in hex text

Code Auswählen Erweitern

if (length($data) > 20 )  ...
I used it to test some hyphothesis on the protocol. I modified the command in the middle
you should comment out all "if (length($data) > 20 ) ....   "

Code Auswählen Erweitern

open (MYFILE, '>>data.txt');
print MYFILE ($data );
close (MYFILE);
I log to file data.txt

Code Auswählen Erweitern

$buffer=  pack('H*', $data);
I convert again to binary

immi

[HenryFinnigan]

Hi immi,

There were hundreds of "too slow" messages in the log, but I am happy to follow your recommendations. "First things first" So I set nonblocking to 0. Do I have to change also simpleReadTimeout (currently set to or just leave it for the time being?

"Windows maschine virtual port":
Do you mean something like this?
https://www.hw-group.com/software/hw-vsp3-virtual-serial-port

I use this on my Windows 10 machine to read/write data via ser2net and a virtual COM port. Is that what you also mean?

I would put the Perl script on my RPI that also runs the FHEM service and ser2net. Thanks also for the reminder regarding the different ports.

According to this...
https://catonmat.net/perl-tcp-proxy

... the Perl script is run like this (different use case):

Code Auswählen Erweitern

sudo ./tcp-proxy2.pl 443 friends_server.com:22

Translated to our case, would it be like this?

Code Auswählen Erweitern


sudo ./proxy.pl <port to sniff> localhost:<port of ser2net, in my case 2000>


Sorry, but I am really unsure how this should be done.
A big THANK YOU! 

Hi sunrise,
please advise me in virtual ports. You have assigned com4, did you appoint it yourself or automatically?
And after restarting the system, do you still have the assigned comport or is it interrupted?

[TheTrumpeter]

Seems that everybody is either on vacation or tired of optimizing/tweaking the THZ/LWZ.

I'm not so I want to place a rather big feature request now. 

I am still annoyed of getting so much redundant data into the logs and creating a lot of unnecessary events by reading out the s*-readings of the THZ although the really interesting data might not even change. Therefore I reduced the interval for some of the readings to rather high numbers, e.g. 300s for sGlobal.
Unfortunately this had some negative side effects recently as it did not catch the preparation of passive-cooling one day and therefore did not notify me in advance to open the windows before going to bed. (Of course passiveCooling was deactivated automatically as the closed windows were detected, but I am still thinking on how to improve it.)

So I was first thinking of creating a lot of user-readings and only log the user-readings and creating events only from them in future, but I feel a lot of people would be interested in a proper solution out of the THZ-module itself, so I want to describe how this could be improved in my opinion.

Prerequisite:
Let's find a naming convention on how we call the elements here to not get confused too much.

• We already have the "readings", e.g. sGlobal, sFan.
• Let's call every single information in such a reading "message", e.g. outsideTemp would be the first message of sGlobal, flowTemp the second message of sGlobal and so on.

Possible implementation:

• Define an attribute for each reading which allows to enable any message of each reading individually.
  It could be bit-coded or any other way that is easy to implement with Perl. (I'm no Perl-expert and I do not bother on the specific method, I just want to have an option to enable/disable every message individually.)
  E.g. attribute messages_sGlobal, by setting it to 5dec which is 0101b I would enable the first and third message of sGlobal and therefore getting the messages outsideTemp and returnTemp created and updated by the THZ-module whenever sGlobal is updated.
• Define a naming-convention for messages in general so we can create events by RegEx just as we currently do with the readings.
  It must not start with p or s as we already use that for parameters and readings. We could let them start with msg or just m.
• In addition we have to define how we handle duplicate messages. E.g. both sFan and sGlobal update the messages inputFanSpeed and outputFanSpeed although they have different names (input/outputVentilatorSpeed in sGlobal). I'm pretty sure this is no duplicate information inside the THZ/LWZ but just sent out in two different readings.
  My prefered solution there would be that such a message will be updated by every reading that holds the information if it's enabled with the attributes introduced earlier. E.g. when first bit of messages_sGlobal is set then msgOutsideTemp is updated whenever sGlobal is updated, but the same message is also updated when sHC1 is updated if the first bit of attributemessages_sHC1 is set either.
• We have to define unique names for all messages and also catch duplicate entries.
• Somebody has to implement that.

Advantages:

• A lot of userReadings become obsolete and therefore make it easier to share code-snippets as everybody could create the same messages very easily.
• It's fully backwards compatible. If you don't define the new attributes, nothing changes.
• If you define the attributes, you can still decide which information creates events or is logged and so on.
• If you' want to have only one message updated very quickly that only changes now and then but is part of a big reading that changes very frequently you can still update the reading very frequently, limit event-on-change-reading for the reading but do not limit it for the message. Then the load of your system will not increase and also file-logs will not, but you will get accurate data.

Thanks for reading, what do you think?

[immi]

Dear TheTrumpeter
I try to summ up: you are not happy with the functions and flexibility of userreadings and logfiles in order to address/save any message of each reading individually.
Moreover you expect standardize naming so that users can share scripts.

I implemented the module 00_THZ as transparent representation of tecalor register logic (oxymoron ....) and compatible with the standard fhem interfaces.

Going back to your solution "attribute messages_sGlobal, by setting it to 5dec which is 0101b": if a newbe cannot manage userreading, will for sure have problems with 0101b....

I see different ways

• creating lots of user-readings and only log the user-readings you want with regex
• with a mix of lots of dummy, some notify, readingsProxy....
• building an abstraction layer on top of 00_THZ (additional module like heatpump_interface inspiring from readingproxy...)
• userreadings and logfiles should be extended with a mapping file; in the file you can map the readings with the messages in a readable format; a standard mapping file for some tecalors could be distibuted automatically with fhem
• same as  4 but extending readingsProxy; a small api (to manage the mapping) can be impemented as second step.

The most elegant and consistent solution would be 5.
But we have to talk to Andre/justme1968 (manteiner of readingproxy)

what you think?
immi

[TheTrumpeter]

I implemented the module 00_THZ as transparent representation of tecalor register logic (oxymoron ....) and compatible with the standard fhem interfaces.

I really appreciate your work. It's really awsome how it works and how fast additionally found registers are implemented by you.

Going back to your solution "attribute messages_sGlobal, by setting it to 5dec which is 0101b": if a newbe cannot manage userreading, will for sure have problems with 0101b....

You're absolutely right, that's what I also feel, but I'm happy with any other suggestion that's easier. Nevertheless these attributes need not to be set. You just don't have a benefit from it.

I see different ways

• creating lots of user-readings and only log the user-readings you want with regex
• with a mix of lots of dummy, some notify, readingsProxy....
• building an abstraction layer on top of 00_THZ (additional module like heatpump_interface inspiring from readingproxy...)
• userreadings and logfiles should be extended with a mapping file; in the file you can map the readings with the messages in a readable format; a standard mapping file for some tecalors could be distibuted automatically with fhem
• same as  4 but extending readingsProxy; a small api (to manage the mapping) can be impemented as second step.

The most elegant and consistent solution would be 5.
But we have to talk to Andre/justme1968 (manteiner of readingproxy)

1. is what my next approach would have been, but this cannot manage signals that are updated by more than one readings (e.g. outsideTemp)
2. sounds too complicated although it would be able to manage signals that are updated by more than one readings.
3. sounds interesting, although a quick solution could be usage of a few notifys and therefore be similar to 2.
4. I'm not sure if I get you point. Your idea is to define some "standard user readings" that would then be available for every user of THZ-module. Of course that's not that complicated and configurable as my initial suggestion is, but would at least solve the problem with sharing of code-snippets.
5. Of course I did not check all available modules if one would already be helpful for solving the problem. As far as I understand readingsProxy it would not be helpful "as it is" right now.


If I would not have another project already in preparation I think I would go for 3 on my own although I'm still not very familiar with Perl.

[immi]

1. is what my next approach would have been, but this cannot manage signals that are updated by more than one readings (e.g. outsideTemp)

well you just have to decide which one you trust and log and which one you ignore.


3, 4 and 5 differ mainly by the module to be changed (00_THZ vs userreadings vs. readingproxy)
I want to define a mapping of part of readings to messages in a readable format, which the user can change.
I am not sure if an external mapping config file is a good idea.

[TheTrumpeter]

I have now spent an hour to implement something like an abstraction layer.

Seems to work, what's missing now is the definition of the msg-Names, right now I've just done it for sGlobal and sHC1. Even there I've not spent too much effort but almost just copied the names from the existing readings.

After that the rest of my code has to be reworked so it fits the new features.

[immi]

interesting discussion here
https://forum.fhem.de/index.php/topic,113361.0.html