FHEM blockiert: Accept failed (WEB: Too many open files)

Begonnen von Papaloewe, 14 November 2020, 16:36:28

Vorheriges Thema - Nächstes Thema

Papaloewe

Das ist gerade zu ersten Mal passiert.
Die Auswirkungen waren, dass FHEM nicht mehr bedienbar war und auch keine aktionen mehr ausführte, sprich blockiert war.
FHEM lief schon länger einwandfrei in einem Docker-Container.


2020.11.14 15:23:32 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 15:23:32 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 15:23:32 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 15:23:46 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 15:26:26 3: DbRep Rep.FillCurr.fhem - Entries of fhem.current deleted: 210
2020.11.14 15:26:35 3: DbRep Rep.FillCurr.fhem - Table 'fhem'.'current' filled up with rows: 210
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
...
...
...
2020.11.14 16:25:00 1: Accept failed (espBridge: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetPort: Too many open files)
2020.11.14 16:25:00 1: Accept failed (WEB: Too many open files)
2020.11.14 16:25:00 1: Accept failed (espBridge: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetPort: Too many open files)
2020.11.14 16:25:00 1: Accept failed (WEB: Too many open files)
2020.11.14 16:25:00 1: Accept failed (espBridge: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetForBlockingFn_1605113255: Too many open files)
2020.11.14 16:25:00 1: Accept failed (telnetPort: Too many open files)
2020.11.14 16:25:00 1: Accept failed (WEB: Too many open files)

Wernieman

2020.11.14 15:30:41 1: Accept failed (WEB: Too many open files)
Du must die Anzahl der Maximal möglich zu öffnenden Files anpassen
https://www.cyberciti.biz/faq/linux-increase-the-maximum-number-of-open-files/
Was läuft alles auf dem System? Nur fhem oder auch mehr?

Mich wundert, das FHEM bei Dir so viele Dateien öffnet.

Btw: Du hast im Forum nach "Too many open files" gesucht?
- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

rudolfkoenig

Es koennte auch helfen, die Ausgabe von
ls -l /proc/<FHEM-PID>/fd
hier anzuhaengen.
Am besten dann, wenn das Problem auftritt, und da in diesem Fall FHEM vmtl. unbedienbar ist, ausgefuehrt in einem Shell auf dem FHEM-Rechner.

Wernieman

Mal als copy&paste für die root-Konsole (oder mit sudo)
ls -l /proc/$(ps -ax | grep [f]hem | head -n1 | awk '{print $1}')/fd
- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

Papaloewe

Ok, das Problem tritt vermutlich erst nach relativ langer Laufzeit von Fhem auf.
Daher muss ich das abwarten und hänge dann das Ergebnis hier an.

Erstmal Danke für eure Antworten.

Papaloewe

Das wäre die Ausgabe beim funktionierendem Fhem:
s: cannot read symbolic link '/proc/6680/fd/0': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/1': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/2': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/3': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/4': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/5': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/6': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/7': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/8': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/9': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/10': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/11': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/12': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/13': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/14': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/15': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/16': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/17': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/18': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/19': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/20': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/21': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/22': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/23': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/24': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/25': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/26': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/27': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/28': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/29': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/30': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/31': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/32': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/33': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/34': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/35': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/36': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/37': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/38': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/39': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/40': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/41': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/42': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/43': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/44': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/45': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/46': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/47': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/48': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/49': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/50': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/51': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/52': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/53': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/54': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/55': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/56': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/57': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/58': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/59': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/60': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/61': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/62': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/63': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/64': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/65': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/66': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/67': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/68': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/69': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/70': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/71': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/72': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/73': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/74': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/75': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/76': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/77': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/78': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/79': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/80': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/81': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/82': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/83': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/84': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/85': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/86': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/87': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/88': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/89': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/90': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/91': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/92': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/93': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/94': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/95': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/96': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/97': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/98': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/99': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/100': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/101': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/102': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/103': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/104': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/105': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/106': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/107': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/108': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/109': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/110': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/111': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/112': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/113': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/114': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/115': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/116': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/117': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/118': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/119': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/120': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/121': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/122': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/123': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/124': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/125': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/126': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/127': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/128': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/129': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/130': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/131': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/132': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/133': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/134': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/135': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/136': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/137': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/138': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/139': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/140': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/141': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/142': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/143': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/144': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/145': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/146': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/147': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/148': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/149': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/150': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/151': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/152': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/153': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/154': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/155': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/156': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/157': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/158': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/159': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/160': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/161': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/162': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/163': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/164': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/165': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/166': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/167': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/168': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/169': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/170': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/171': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/172': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/173': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/174': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/175': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/176': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/177': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/178': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/179': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/180': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/181': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/182': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/183': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/184': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/185': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/186': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/187': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/188': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/189': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/190': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/191': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/192': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/193': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/194': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/195': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/196': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/197': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/198': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/199': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/200': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/201': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/202': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/203': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/204': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/205': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/206': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/207': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/208': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/209': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/210': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/211': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/212': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/213': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/214': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/215': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/216': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/217': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/218': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/219': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/220': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/221': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/222': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/223': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/224': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/225': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/226': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/227': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/228': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/229': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/230': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/231': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/232': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/233': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/234': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/235': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/236': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/237': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/238': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/239': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/240': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/241': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/242': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/243': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/244': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/245': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/246': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/247': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/248': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/249': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/250': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/251': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/252': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/253': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/254': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/255': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/256': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/257': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/258': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/259': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/260': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/261': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/262': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/263': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/264': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/265': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/266': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/267': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/268': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/269': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/270': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/271': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/272': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/273': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/274': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/275': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/276': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/277': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/278': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/279': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/280': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/281': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/282': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/283': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/284': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/285': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/286': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/287': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/288': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/289': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/290': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/291': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/292': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/293': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/294': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/295': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/296': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/297': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/298': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/299': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/300': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/301': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/302': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/303': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/304': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/305': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/306': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/307': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/308': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/309': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/310': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/311': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/312': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/313': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/314': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/315': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/316': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/317': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/318': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/319': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/320': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/321': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/322': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/323': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/324': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/325': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/326': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/327': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/328': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/329': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/330': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/331': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/332': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/333': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/334': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/335': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/336': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/337': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/338': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/339': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/340': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/341': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/342': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/343': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/344': Permission denied
ls: cannot read symbolic link '/proc/6680/fd/345': Permission denied
total 0
lr-x------. 1 root fhem 64 Nov 15 14:31 0
l-wx------. 1 root fhem 64 Nov 15 14:31 1
lrwx------. 1 root fhem 64 Nov 15 14:31 10
lrwx------. 1 root fhem 64 Nov 15 14:31 100
lrwx------. 1 root fhem 64 Nov 15 14:31 101
lrwx------. 1 root fhem 64 Nov 15 14:31 102
lrwx------. 1 root fhem 64 Nov 15 14:31 103
lrwx------. 1 root fhem 64 Nov 15 14:31 104
lrwx------. 1 root fhem 64 Nov 15 14:31 105
lrwx------. 1 root fhem 64 Nov 15 14:31 106
lrwx------. 1 root fhem 64 Nov 15 14:31 107
lrwx------. 1 root fhem 64 Nov 15 14:31 108
lrwx------. 1 root fhem 64 Nov 15 14:31 109
lrwx------. 1 root fhem 64 Nov 15 14:31 11
lrwx------. 1 root fhem 64 Nov 15 14:31 110
lrwx------. 1 root fhem 64 Nov 15 14:31 111
lrwx------. 1 root fhem 64 Nov 15 14:31 112
lrwx------. 1 root fhem 64 Nov 15 14:31 113
lrwx------. 1 root fhem 64 Nov 15 14:31 114
lrwx------. 1 root fhem 64 Nov 15 14:31 115
lrwx------. 1 root fhem 64 Nov 15 14:31 116
lrwx------. 1 root fhem 64 Nov 15 14:31 117
lrwx------. 1 root fhem 64 Nov 15 14:31 118
lrwx------. 1 root fhem 64 Nov 15 14:31 119
lrwx------. 1 root fhem 64 Nov 15 14:31 12
lrwx------. 1 root fhem 64 Nov 15 14:31 120
lrwx------. 1 root fhem 64 Nov 15 14:31 121
lrwx------. 1 root fhem 64 Nov 15 14:31 122
lrwx------. 1 root fhem 64 Nov 15 14:31 123
lrwx------. 1 root fhem 64 Nov 15 14:31 124
lrwx------. 1 root fhem 64 Nov 15 14:31 125
lrwx------. 1 root fhem 64 Nov 15 14:31 126
lrwx------. 1 root fhem 64 Nov 15 14:31 127
lrwx------. 1 root fhem 64 Nov 15 14:31 128
lrwx------. 1 root fhem 64 Nov 15 14:31 129
lrwx------. 1 root fhem 64 Nov 15 14:31 13
lrwx------. 1 root fhem 64 Nov 15 14:31 130
lrwx------. 1 root fhem 64 Nov 15 14:31 131
lrwx------. 1 root fhem 64 Nov 15 14:31 132
lrwx------. 1 root fhem 64 Nov 15 14:31 133
lrwx------. 1 root fhem 64 Nov 15 14:31 134
lrwx------. 1 root fhem 64 Nov 15 14:31 135
lrwx------. 1 root fhem 64 Nov 15 14:31 136
lrwx------. 1 root fhem 64 Nov 15 14:31 137
lrwx------. 1 root fhem 64 Nov 15 14:31 138
lrwx------. 1 root fhem 64 Nov 15 14:31 139
lrwx------. 1 root fhem 64 Nov 15 14:31 14
lrwx------. 1 root fhem 64 Nov 15 14:31 140
lrwx------. 1 root fhem 64 Nov 15 14:31 141
lrwx------. 1 root fhem 64 Nov 15 14:31 142
lrwx------. 1 root fhem 64 Nov 15 14:31 143
lrwx------. 1 root fhem 64 Nov 15 14:31 144
lrwx------. 1 root fhem 64 Nov 15 14:31 145
lrwx------. 1 root fhem 64 Nov 15 14:31 146
lrwx------. 1 root fhem 64 Nov 15 14:31 147
lrwx------. 1 root fhem 64 Nov 15 14:31 148
lrwx------. 1 root fhem 64 Nov 15 14:31 149
lrwx------. 1 root fhem 64 Nov 15 14:31 15
lrwx------. 1 root fhem 64 Nov 15 14:31 150
lrwx------. 1 root fhem 64 Nov 15 14:31 151
lrwx------. 1 root fhem 64 Nov 15 14:31 152
lrwx------. 1 root fhem 64 Nov 15 14:31 153
lrwx------. 1 root fhem 64 Nov 15 14:31 154
lrwx------. 1 root fhem 64 Nov 15 14:31 155
lrwx------. 1 root fhem 64 Nov 15 14:31 156
lrwx------. 1 root fhem 64 Nov 15 14:31 157
lrwx------. 1 root fhem 64 Nov 15 14:31 158
lrwx------. 1 root fhem 64 Nov 15 14:31 159
lrwx------. 1 root fhem 64 Nov 15 14:31 16
lrwx------. 1 root fhem 64 Nov 15 14:31 160
lrwx------. 1 root fhem 64 Nov 15 14:31 161
lrwx------. 1 root fhem 64 Nov 15 14:31 162
lrwx------. 1 root fhem 64 Nov 15 14:31 163
lrwx------. 1 root fhem 64 Nov 15 14:31 164
lrwx------. 1 root fhem 64 Nov 15 14:31 165
lrwx------. 1 root fhem 64 Nov 15 14:31 166
lrwx------. 1 root fhem 64 Nov 15 14:31 167
lrwx------. 1 root fhem 64 Nov 15 14:31 168
lrwx------. 1 root fhem 64 Nov 15 14:31 169
lrwx------. 1 root fhem 64 Nov 15 14:31 17
lrwx------. 1 root fhem 64 Nov 15 14:31 170
lrwx------. 1 root fhem 64 Nov 15 14:31 171
lrwx------. 1 root fhem 64 Nov 15 14:31 172
lrwx------. 1 root fhem 64 Nov 15 14:31 173
lrwx------. 1 root fhem 64 Nov 15 14:31 174
lrwx------. 1 root fhem 64 Nov 15 14:31 175
lrwx------. 1 root fhem 64 Nov 15 14:31 176
lrwx------. 1 root fhem 64 Nov 15 14:31 177
lrwx------. 1 root fhem 64 Nov 15 14:31 178
lrwx------. 1 root fhem 64 Nov 15 14:31 179
lrwx------. 1 root fhem 64 Nov 15 14:31 18
lrwx------. 1 root fhem 64 Nov 15 14:31 180
lrwx------. 1 root fhem 64 Nov 15 14:31 181
lrwx------. 1 root fhem 64 Nov 15 14:31 182
lrwx------. 1 root fhem 64 Nov 15 14:31 183
lrwx------. 1 root fhem 64 Nov 15 14:31 184
lrwx------. 1 root fhem 64 Nov 15 14:31 185
lrwx------. 1 root fhem 64 Nov 15 14:31 186
lrwx------. 1 root fhem 64 Nov 15 14:31 187
lrwx------. 1 root fhem 64 Nov 15 14:31 188
lrwx------. 1 root fhem 64 Nov 15 14:31 189
lrwx------. 1 root fhem 64 Nov 15 14:31 19
lrwx------. 1 root fhem 64 Nov 15 14:31 190
lrwx------. 1 root fhem 64 Nov 15 14:31 191
lrwx------. 1 root fhem 64 Nov 15 14:31 192
lrwx------. 1 root fhem 64 Nov 15 14:31 193
lrwx------. 1 root fhem 64 Nov 15 14:31 194
lrwx------. 1 root fhem 64 Nov 15 14:31 195
lrwx------. 1 root fhem 64 Nov 15 14:31 196
lrwx------. 1 root fhem 64 Nov 15 14:31 197
lrwx------. 1 root fhem 64 Nov 15 14:31 198
lrwx------. 1 root fhem 64 Nov 15 14:31 199
l-wx------. 1 root fhem 64 Nov 15 14:31 2
lrwx------. 1 root fhem 64 Nov 15 14:31 20
lrwx------. 1 root fhem 64 Nov 15 14:31 200
lrwx------. 1 root fhem 64 Nov 15 14:31 201
lrwx------. 1 root fhem 64 Nov 15 14:31 202
lrwx------. 1 root fhem 64 Nov 15 14:31 203
lrwx------. 1 root fhem 64 Nov 15 14:31 204
lrwx------. 1 root fhem 64 Nov 15 14:31 205
lrwx------. 1 root fhem 64 Nov 15 14:31 206
lrwx------. 1 root fhem 64 Nov 15 14:31 207
lrwx------. 1 root fhem 64 Nov 15 14:31 208
lrwx------. 1 root fhem 64 Nov 15 14:31 209
lrwx------. 1 root fhem 64 Nov 15 14:31 21
lrwx------. 1 root fhem 64 Nov 15 14:31 210
lrwx------. 1 root fhem 64 Nov 15 14:31 211
lrwx------. 1 root fhem 64 Nov 15 14:31 212
lrwx------. 1 root fhem 64 Nov 15 14:31 213
lrwx------. 1 root fhem 64 Nov 15 14:31 214
lrwx------. 1 root fhem 64 Nov 15 14:31 215
lrwx------. 1 root fhem 64 Nov 15 14:31 216
lrwx------. 1 root fhem 64 Nov 15 14:31 217
lrwx------. 1 root fhem 64 Nov 15 14:31 218
lrwx------. 1 root fhem 64 Nov 15 14:31 219
lrwx------. 1 root fhem 64 Nov 15 14:31 22
lrwx------. 1 root fhem 64 Nov 15 14:31 220
lrwx------. 1 root fhem 64 Nov 15 14:31 221
lrwx------. 1 root fhem 64 Nov 15 14:31 222
lrwx------. 1 root fhem 64 Nov 15 14:31 223
lrwx------. 1 root fhem 64 Nov 15 14:31 224
lrwx------. 1 root fhem 64 Nov 15 14:31 225
lrwx------. 1 root fhem 64 Nov 15 14:31 226
lrwx------. 1 root fhem 64 Nov 15 14:31 227
lrwx------. 1 root fhem 64 Nov 15 14:31 228
lrwx------. 1 root fhem 64 Nov 15 14:31 229
lrwx------. 1 root fhem 64 Nov 15 14:31 23
lrwx------. 1 root fhem 64 Nov 15 14:31 230
lrwx------. 1 root fhem 64 Nov 15 14:31 231
lrwx------. 1 root fhem 64 Nov 15 14:31 232
lrwx------. 1 root fhem 64 Nov 15 14:31 233
lrwx------. 1 root fhem 64 Nov 15 14:31 234
lrwx------. 1 root fhem 64 Nov 15 14:31 235
lrwx------. 1 root fhem 64 Nov 15 14:31 236
lrwx------. 1 root fhem 64 Nov 15 14:31 237
lrwx------. 1 root fhem 64 Nov 15 14:31 238
lrwx------. 1 root fhem 64 Nov 15 14:31 239
lrwx------. 1 root fhem 64 Nov 15 14:31 24
lrwx------. 1 root fhem 64 Nov 15 14:31 240
lrwx------. 1 root fhem 64 Nov 15 14:31 241
lrwx------. 1 root fhem 64 Nov 15 14:31 242
lrwx------. 1 root fhem 64 Nov 15 14:31 243
lrwx------. 1 root fhem 64 Nov 15 14:31 244
lrwx------. 1 root fhem 64 Nov 15 14:31 245
lrwx------. 1 root fhem 64 Nov 15 14:31 246
lrwx------. 1 root fhem 64 Nov 15 14:31 247
lrwx------. 1 root fhem 64 Nov 15 14:31 248
lrwx------. 1 root fhem 64 Nov 15 14:31 249
lrwx------. 1 root fhem 64 Nov 15 14:31 25
lrwx------. 1 root fhem 64 Nov 15 14:31 250
lrwx------. 1 root fhem 64 Nov 15 14:31 251
lrwx------. 1 root fhem 64 Nov 15 14:31 252
lrwx------. 1 root fhem 64 Nov 15 14:31 253
lrwx------. 1 root fhem 64 Nov 15 14:31 254
lrwx------. 1 root fhem 64 Nov 15 14:31 255
lrwx------. 1 root fhem 64 Nov 15 14:31 256
lrwx------. 1 root fhem 64 Nov 15 14:31 257
lrwx------. 1 root fhem 64 Nov 15 14:31 258
lrwx------. 1 root fhem 64 Nov 15 14:31 259
lrwx------. 1 root fhem 64 Nov 15 14:31 26
lrwx------. 1 root fhem 64 Nov 15 14:31 260
lrwx------. 1 root fhem 64 Nov 15 14:31 261
lrwx------. 1 root fhem 64 Nov 15 14:31 262
lrwx------. 1 root fhem 64 Nov 15 14:31 263
lrwx------. 1 root fhem 64 Nov 15 14:31 264
lrwx------. 1 root fhem 64 Nov 15 14:31 265
lrwx------. 1 root fhem 64 Nov 15 14:31 266
lrwx------. 1 root fhem 64 Nov 15 14:31 267
lrwx------. 1 root fhem 64 Nov 15 14:31 268
lrwx------. 1 root fhem 64 Nov 15 14:31 269
lrwx------. 1 root fhem 64 Nov 15 14:31 27
lrwx------. 1 root fhem 64 Nov 15 14:31 270
lrwx------. 1 root fhem 64 Nov 15 14:31 271
lrwx------. 1 root fhem 64 Nov 15 14:31 272
lrwx------. 1 root fhem 64 Nov 15 14:31 273
lrwx------. 1 root fhem 64 Nov 15 14:31 274
lrwx------. 1 root fhem 64 Nov 15 14:31 275
lrwx------. 1 root fhem 64 Nov 15 14:31 276
lrwx------. 1 root fhem 64 Nov 15 14:31 277
lrwx------. 1 root fhem 64 Nov 15 14:31 278
lrwx------. 1 root fhem 64 Nov 15 14:31 279
lrwx------. 1 root fhem 64 Nov 15 14:31 28
lrwx------. 1 root fhem 64 Nov 15 14:31 280
lrwx------. 1 root fhem 64 Nov 15 14:31 281
lrwx------. 1 root fhem 64 Nov 15 14:31 282
lrwx------. 1 root fhem 64 Nov 15 14:31 283
lrwx------. 1 root fhem 64 Nov 15 14:31 284
lrwx------. 1 root fhem 64 Nov 15 14:31 285
lrwx------. 1 root fhem 64 Nov 15 14:31 286
lrwx------. 1 root fhem 64 Nov 15 14:31 287
lrwx------. 1 root fhem 64 Nov 15 14:31 288
lrwx------. 1 root fhem 64 Nov 15 14:31 289
lrwx------. 1 root fhem 64 Nov 15 14:31 29
lrwx------. 1 root fhem 64 Nov 15 14:31 290
lrwx------. 1 root fhem 64 Nov 15 14:31 291
lrwx------. 1 root fhem 64 Nov 15 14:31 292
lrwx------. 1 root fhem 64 Nov 15 14:31 293
lrwx------. 1 root fhem 64 Nov 15 14:31 294
lrwx------. 1 root fhem 64 Nov 15 14:31 295
lrwx------. 1 root fhem 64 Nov 15 14:31 296
lrwx------. 1 root fhem 64 Nov 15 14:31 297
lrwx------. 1 root fhem 64 Nov 15 14:31 298
lrwx------. 1 root fhem 64 Nov 15 14:31 299
lr-x------. 1 root fhem 64 Nov 15 14:31 3
lrwx------. 1 root fhem 64 Nov 15 14:31 30
lrwx------. 1 root fhem 64 Nov 15 14:31 300
lrwx------. 1 root fhem 64 Nov 15 14:31 301
lrwx------. 1 root fhem 64 Nov 15 14:31 302
lrwx------. 1 root fhem 64 Nov 15 14:31 303
lrwx------. 1 root fhem 64 Nov 15 14:31 304
lrwx------. 1 root fhem 64 Nov 15 14:31 305
lrwx------. 1 root fhem 64 Nov 15 14:31 306
lrwx------. 1 root fhem 64 Nov 15 14:31 307
lrwx------. 1 root fhem 64 Nov 15 14:31 308
lrwx------. 1 root fhem 64 Nov 15 14:31 309
lrwx------. 1 root fhem 64 Nov 15 14:31 31
lrwx------. 1 root fhem 64 Nov 15 14:31 310
lrwx------. 1 root fhem 64 Nov 15 14:31 311
lrwx------. 1 root fhem 64 Nov 15 14:31 312
lrwx------. 1 root fhem 64 Nov 15 14:31 313
lrwx------. 1 root fhem 64 Nov 15 14:31 314
lrwx------. 1 root fhem 64 Nov 15 14:31 315
lrwx------. 1 root fhem 64 Nov 15 14:31 316
lrwx------. 1 root fhem 64 Nov 15 14:31 317
lrwx------. 1 root fhem 64 Nov 15 14:31 318
lrwx------. 1 root fhem 64 Nov 15 14:31 319
lrwx------. 1 root fhem 64 Nov 15 14:31 32
lrwx------. 1 root fhem 64 Nov 15 14:31 320
lrwx------. 1 root fhem 64 Nov 15 14:31 321
lrwx------. 1 root fhem 64 Nov 15 14:31 322
lrwx------. 1 root fhem 64 Nov 15 14:31 323
lrwx------. 1 root fhem 64 Nov 15 14:31 324
lrwx------. 1 root fhem 64 Nov 15 14:31 325
lrwx------. 1 root fhem 64 Nov 15 14:31 326
lrwx------. 1 root fhem 64 Nov 15 14:31 327
lrwx------. 1 root fhem 64 Nov 15 14:31 328
lrwx------. 1 root fhem 64 Nov 15 14:31 329
lrwx------. 1 root fhem 64 Nov 15 14:31 33
lrwx------. 1 root fhem 64 Nov 15 14:31 330
lrwx------. 1 root fhem 64 Nov 15 14:31 331
lrwx------. 1 root fhem 64 Nov 15 14:31 332
lrwx------. 1 root fhem 64 Nov 15 14:31 333
lrwx------. 1 root fhem 64 Nov 15 14:31 334
lrwx------. 1 root fhem 64 Nov 15 14:31 335
lrwx------. 1 root fhem 64 Nov 15 14:31 336
lrwx------. 1 root fhem 64 Nov 15 14:31 337
lrwx------. 1 root fhem 64 Nov 15 14:31 338
lrwx------. 1 root fhem 64 Nov 15 14:31 339
lrwx------. 1 root fhem 64 Nov 15 14:31 34
lrwx------. 1 root fhem 64 Nov 15 14:31 340
lrwx------. 1 root fhem 64 Nov 15 14:31 341
lrwx------. 1 root fhem 64 Nov 15 14:31 342
lrwx------. 1 root fhem 64 Nov 15 14:31 343
lrwx------. 1 root fhem 64 Nov 15 14:31 344
lrwx------. 1 root fhem 64 Nov 15 14:31 345
lrwx------. 1 root fhem 64 Nov 15 14:31 35
lrwx------. 1 root fhem 64 Nov 15 14:31 36
lrwx------. 1 root fhem 64 Nov 15 14:31 37
lrwx------. 1 root fhem 64 Nov 15 14:31 38
lrwx------. 1 root fhem 64 Nov 15 14:31 39
l-wx------. 1 root fhem 64 Nov 15 14:31 4
lrwx------. 1 root fhem 64 Nov 15 14:31 40
lrwx------. 1 root fhem 64 Nov 15 14:31 41
lrwx------. 1 root fhem 64 Nov 15 14:31 42
lrwx------. 1 root fhem 64 Nov 15 14:31 43
lrwx------. 1 root fhem 64 Nov 15 14:31 44
lrwx------. 1 root fhem 64 Nov 15 14:31 45
lrwx------. 1 root fhem 64 Nov 15 14:31 46
lrwx------. 1 root fhem 64 Nov 15 14:31 47
lrwx------. 1 root fhem 64 Nov 15 14:31 48
lrwx------. 1 root fhem 64 Nov 15 14:31 49
lrwx------. 1 root fhem 64 Nov 15 14:31 5
lrwx------. 1 root fhem 64 Nov 15 14:31 50
lrwx------. 1 root fhem 64 Nov 15 14:31 51
lrwx------. 1 root fhem 64 Nov 15 14:31 52
lrwx------. 1 root fhem 64 Nov 15 14:31 53
lrwx------. 1 root fhem 64 Nov 15 14:31 54
lrwx------. 1 root fhem 64 Nov 15 14:31 55
lrwx------. 1 root fhem 64 Nov 15 14:31 56
lrwx------. 1 root fhem 64 Nov 15 14:31 57
lrwx------. 1 root fhem 64 Nov 15 14:31 58
lrwx------. 1 root fhem 64 Nov 15 14:31 59
lrwx------. 1 root fhem 64 Nov 15 14:31 6
lrwx------. 1 root fhem 64 Nov 15 14:31 60
lrwx------. 1 root fhem 64 Nov 15 14:31 61
lrwx------. 1 root fhem 64 Nov 15 14:31 62
lrwx------. 1 root fhem 64 Nov 15 14:31 63
lrwx------. 1 root fhem 64 Nov 15 14:31 64
lrwx------. 1 root fhem 64 Nov 15 14:31 65
lrwx------. 1 root fhem 64 Nov 15 14:31 66
lrwx------. 1 root fhem 64 Nov 15 14:31 67
lrwx------. 1 root fhem 64 Nov 15 14:31 68
lrwx------. 1 root fhem 64 Nov 15 14:31 69
l-wx------. 1 root fhem 64 Nov 15 14:31 7
lrwx------. 1 root fhem 64 Nov 15 14:31 70
lrwx------. 1 root fhem 64 Nov 15 14:31 71
lrwx------. 1 root fhem 64 Nov 15 14:31 72
lrwx------. 1 root fhem 64 Nov 15 14:31 73
lrwx------. 1 root fhem 64 Nov 15 14:31 74
lrwx------. 1 root fhem 64 Nov 15 14:31 75
lrwx------. 1 root fhem 64 Nov 15 14:31 76
lrwx------. 1 root fhem 64 Nov 15 14:31 77
lrwx------. 1 root fhem 64 Nov 15 14:31 78
lrwx------. 1 root fhem 64 Nov 15 14:31 79
lrwx------. 1 root fhem 64 Nov 15 14:31 8
lrwx------. 1 root fhem 64 Nov 15 14:31 80
lrwx------. 1 root fhem 64 Nov 15 14:31 81
lrwx------. 1 root fhem 64 Nov 15 14:31 82
lrwx------. 1 root fhem 64 Nov 15 14:31 83
lrwx------. 1 root fhem 64 Nov 15 14:31 84
lrwx------. 1 root fhem 64 Nov 15 14:31 85
lrwx------. 1 root fhem 64 Nov 15 14:31 86
lrwx------. 1 root fhem 64 Nov 15 14:31 87
lrwx------. 1 root fhem 64 Nov 15 14:31 88
lrwx------. 1 root fhem 64 Nov 15 14:31 89
lrwx------. 1 root fhem 64 Nov 15 14:31 9
lrwx------. 1 root fhem 64 Nov 15 14:31 90
lrwx------. 1 root fhem 64 Nov 15 14:31 91
lrwx------. 1 root fhem 64 Nov 15 14:31 92
lrwx------. 1 root fhem 64 Nov 15 14:31 93
lrwx------. 1 root fhem 64 Nov 15 14:31 94
lrwx------. 1 root fhem 64 Nov 15 14:31 95
lrwx------. 1 root fhem 64 Nov 15 14:31 96
lrwx------. 1 root fhem 64 Nov 15 14:31 97
lrwx------. 1 root fhem 64 Nov 15 14:31 98
lrwx------. 1 root fhem 64 Nov 15 14:31 99

Wernieman

#6
Wie lange läuft es?
345 offene Files ....

- Wie lange läuft FHEM aktuell (in Fhem uptime)
- Hast Du es als root oder als "Normaluser" laufen lassen?
- Hast Du i/o-Probleme auf dem System? "sudo grep -i i/o /var/log/kern.log"
- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

Papaloewe

Mit sudo gestartet, also als root.
Fhem läuft jetzt erst ca. 24h.

Wernieman

Habe noch 2 Fragen dazugeschrieben ..
Zitat- Wie lange läuft FHEM aktuell (in Fhem uptime)
- Hast Du i/o-Probleme auf dem System? "sudo grep -i i/o /var/log/kern.log"

Mich stören die "Permission denied"
- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

Papaloewe

fhem uptime: 22:19:36

sudo grep -i i/o /var/log/kern.log
grep: /var/log/kern.log: No such file or directory


sudo ls /var/log/
alternatives.log  apt  btmp  dpkg.log  faillog  fontconfig.log  lastlog  wtmp

Wernieman

- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

Papaloewe


Wernieman

- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

duke-f

Die letzten Monate habe ich dieses Problem mit "Too many open files". Und meine Recherche führt mich immer wieder genau hierher. Leider hatte dies ja mit offenem Schluss geendet. Gleichmal vorab: Die Sache mit den Werten aus den oben von @Wernieman genannten Link habe ich erfolglos schon probiert. Eigentlich dachte ich, damit die Werte für den Nutzer fhem auf 10240 erhöht zu haben, als Test. Hat wohl nicht geklappt:


root@cubie:~# su - fhem

fhem@cubie:~$ ulimit -Hn
20480
fhem@cubie:~$ ulimit -Sn
10240
fhem@cubie:~$


Scheinbar war aber dennoch bei 1024 Schluss.
Eine wirkliche Lösung scheint mir das aber nicht zu sein. Laut dem beigefügten File fehler.txt handelt sich nicht um die geöffneten Files - die sind seit Jahren ohne Probleme immer praktisch gleich. Es muss sich eher um eine Angelegenheit mit einer Verbindung handeln. Ich nutze - auch schon länger - interne schon so einige Netzverbindungen. Auch dies bisher ohne Probleme. Vor etwa einem Jahr kam neu hinzu die Anbindung meiner Echo-Devices - insgesamt 4. Und nach einigen Monaten problemlosem Funktionieren hatte ich letzten Sommer plötzlich genau dieses Problem: FHEM reagiert nicht mehr. Im Log tauchte massenhaft diese Meldung auf. Neustart des Cubie verschaffte Abhilfe (War außer Haus, die Möglichkeiten waren begrenzt). Hab dann alle Echo-Devices mal deaktiviert (nicht gelöscht). Dann war für einige Monate wieder Ruhe. Bis kurz nach Weihnachten. Und seither eigentlich alle paar Wochen.

Nun, genug der Vorrede, denke ich. Jetzt hatte ich gestern endlich mal die Gelegenheit erwischt, die hier oben genannten Dinge durchzuführen (die Copy-Paste-Variante von @Wernieman, siehe Anhang. Da taucht gestern um 12:31 offensichtlich das Problem auf, dass eine Verbindung nicht klapp und dies so lange versucht wird, bis sich das System aufhängt. Ich musste da das Haus verlassen und habe daher den Cubie wieder neu gestartet. Hier ist mal ein Auszug des Logs zum Zeitpunkt, als sich diese Socket-Meldungen häufen (12:31), genau dann taucht erstmals "Too many open files" auf:


2022.03.06 12:30:28.576 1: Accept failed (AMADBridge: Too many open files)
2022.03.06 12:30:46.901 1: Accept failed (telnetPort: Too many open files)
2022.03.06 12:31:12.260 1: Accept failed (WEB: Too many open files)



Hab dann mal etwas ältere Logs betrachtet und festgestellt, dass es nicht immer AMADBridge ist, was als erstes auftritt. Früher, mit meinem XS1, wurde mir mal erklärt, dass es passieren kann, dass Verbindungen nicht mehr ordentlich geschlossen werden und sich dann irgendwann die Zahl soweit erhöht hat, dass keine neue Verbindung mehr möglich ist. Leider ist meine Zeit für FHEM wegen anderer familiärer Ereignisse gerade sehr begrenzt, sonst hätte ich schon länger mal angefangen, einzelne Komponenten nach und nach entweder komplett zu entfernen oder eben auf ein Parallelsystem umzulagern. Aber vielleicht gibt es ja auch einen einfachen Trick, das Problem zu lösen?

Unten kopiere ich das ganze mir etwas mehr drum herum.


2022.03.06 12:27:26.551 4: MyTTS: Es wurden alle Teile ausgegeben und der Befehl ist abgearbeitet.
2022.03.06 12:27:42.097 3: CUL_R IT: For autocreate please use the on button.
2022.03.06 12:27:42.115 3: CUL_R: Unknown code i545414, help me!
2022.03.06 12:28:20.607 1: [Freezemon] myFreezemon: possible freeze starting at 12:28:01, delay is 19.599 possibly caused by: tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A)
2022.03.06 12:28:20.687 1: HMUARTLGW HMLANGW:keepAlive KeepAlive sent 16.263s too late, this might cause a disconnect!
2022.03.06 12:28:46.720 1: 192.168.178.170:2001 disconnected, waiting to reappear (HMLANGW:keepAlive)
2022.03.06 12:29:00.819 1: 192.168.178.170:2000 disconnected, waiting to reappear (HMLANGW)
2022.03.06 12:29:11.846 1: [Freezemon] myFreezemon: possible freeze starting at 12:28:21, delay is 50.827 possibly caused by: tmr-HMUARTLGW_SendKeepAlive(HMLANGW) tmr-HMUARTLGW_CheckCredits(HMLANGW) tmr-CUL_HandleWriteQueue(CUL_0) tmr-HMLAN_KeepAlive(HMUSB) tmr-CUL_HandleWriteQueue(SCC) tmr-echodevice_LoginStart(Amazon_Account) tmr-SYSMON_Update(Cubie) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A)
2022.03.06 12:30:28.576 1: Accept failed (AMADBridge: Too many open files)
2022.03.06 12:30:46.901 1: Accept failed (telnetPort: Too many open files)
2022.03.06 12:31:12.260 1: Accept failed (WEB: Too many open files)
2022.03.06 12:31:12.695 3: FS20 set alabama_FS20 off
2022.03.06 12:31:29.709 3: Opening HMLANGW:keepAlive device 192.168.178.170:2001
2022.03.06 12:31:29.714 1: 192.168.178.170:2000 reappeared (HMLANGW)
2022.03.06 12:31:29.875 1: [Freezemon] myFreezemon: possible freeze starting at 12:29:12, delay is 137.809 possibly caused by: tmr-FHEM::AMADDevice::GetUpdate(Alabama_AMAD) tmr-HMLAN_KeepAliveCheck(HMUSB) tmr-ping_Start(Ping_myUTN) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HMLAN_KeepAlive(HMUSB) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-DOIF_TimerTrigger(DI_Leistungsstaende) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(HM_Energie_2_Tag_Set) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-echodevice_GetSettings(ECHO_G0911B0592940KLK) tmr-echodevice_GetSettings(ECHO_G070L816942409J7) tmr-echodevice_GetSettings(ECHO_G090XG0793733DWB) tmr-echodevice_GetSettings(ECHO_b739e2ab24b04c9a9777924b677b8108) tmr-echodevice_GetSettings(ECHO_G090XG0794152JF8) tmr-echodevice_GetSettings(ECHO_G090XG0794152JT6) tmr-HttpUtils_TimeoutErr(N/A) tmr-FW_closeInactiveClients(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-echodevice_GetSettings(Amazon_Account) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-CUL_HandleWriteQueue(CUL_0) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(Antwort_HT_timer) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(Antwort_HT2_timer) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(HM_Energie_4_Tag_Set) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A)
2022.03.06 12:31:30.303 1: Timeout for SYSMON_blockingCall reached, terminated process 5442
2022.03.06 12:31:30.524 1: Timeout for ping_DoPing reached, terminated process 5443
2022.03.06 12:31:30.528 3: BlockingCall for Ping_myUTN was aborted
2022.03.06 12:31:41.264 3: HMLANGW:keepAlive device opened
2022.03.06 12:31:43.418 1: [Freezemon] myFreezemon: possible freeze starting at 12:31:30, delay is 13.367 possibly caused by: tmr-HMLAN_KeepAliveCheck(HMUSB) tmr-BlockingKill(N/A) tmr-echodevice_LoginStart(Amazon_Account) tmr-SYSMON_Update(Cubie) tmr-BlockingKill(N/A) tmr-SB_SERVER_tcb_Alive(LMS) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HMLAN_KeepAlive(HMUSB) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-FHEM::AMADDevice::GetUpdate(Alabama_AMAD) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-DOIF_TimerTrigger(DI_Leistungsstaende) tmr-FW_closeInactiveClients(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-echodevice_GetSettings(ECHO_G0911B0592940KLK) tmr-echodevice_GetSettings(ECHO_G070L816942409J7) tmr-echodevice_GetSettings(ECHO_G090XG0793733DWB) tmr-echodevice_GetSettings(ECHO_b739e2ab24b04c9a9777924b677b8108) tmr-echodevice_GetSettings(ECHO_G090XG0794152JF8) tmr-echodevice_GetSettings(ECHO_G090XG0794152JT6) tmr-echodevice_GetSettings(Amazon_Account) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(HM_Energie_1_Tag_Set) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-CUL_HandleWriteQueue(CUL_0) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(waessern_Log) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(Gartenwasser_Log) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(heartbeat) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(Solarakku_Set) tmr-at_Exec(Solarakku_Get) tmr-HttpUtils_TimeoutErr(N/A) tmr-HttpUtils_TimeoutErr(N/A) tmr-at_Exec(Luefter_Log)
2022.03.06 12:31:50.071 3: HMUARTLGW HMLANGW BidCoS-port opened
2022.03.06 12:31:50.585 1: [Freezemon] myFreezemon: possible freeze starting at 12:31:44, delay is 6.583 possibly caused by: tmr-HMUARTLGW_StartInit(HMLANGW) tmr-BlockingStart(N/A) tmr-HMLAN_KeepAliveCheck(HMUSB) tmr-CUL_HandleWriteQueue(SCC) tmr-CUL_HM_ActCheck(N/A) tmr-HMUARTLGW_StartInit(HMLANGW)
2022.03.06 12:32:18.062 3: HMUARTLGW HMLANGW:keepAlive KeepAlive-port opened
2022.03.06 12:33:03.863 1: [Freezemon] myFreezemon: possible freeze starting at 12:31:51, delay is 72.83 possibly caused by: tmr-HMUARTLGW_StartInit(HMLANGW) tmr-HMUARTLGW_StartInit(HMLANGW) tmr-DOIF_TimerTrigger(DI_Tageszaehler)
2022.03.06 12:33:04.080 1: Timeout for SYSMON_blockingCall reached, terminated process 5555
2022.03.06 12:33:05.881 1: 192.168.178.170:2001 disconnected, waiting to reappear (HMLANGW:keepAlive)
2022.03.06 12:33:08.101 1: [Freezemon] myFreezemon: possible freeze starting at 12:33:04, delay is 4.081 possibly caused by: tmr-HMUARTLGW_StartInit(HMLANGW) tmr-CODE(0x5a1c420)(GetUpdate) tmr-CODE(0x5a1c420)(GetUpdate) tmr-CODE(0x5a1c420)(GetUpdate) tmr-CODE(0x5a1c420)(GetUpdate) tmr-at_Exec(HM_Energie_3_Tag_Set) tmr-HMLAN_KeepAlive(HMUSB) tmr-FHEM::AMADDevice::GetUpdate(Alabama_AMAD) tmr-HMUARTLGW_CheckCmdResp(HMLANGW) tmr-FRITZBOX_Readout_Start(N/A) tmr-SB_PLAYER_GetStatus(SB_PLAYER_Kueche) tmr-SB_PLAYER_GetStatus(SB_PLAYER_Wohnzimmer) tmr-SB_PLAYER_GetStatus(SB_PLAYER_Schlafzimmer) tmr-SB_PLAYER_GetStatus(SB_PLAYER_Keller) tmr-BlockingKill(N/A) tmr-DOIF_TimerTrigger(DI_Leistungsstaende) tmr-FW_closeInactiveClients(N/A) tmr-echodevice_LoginStart(Amazon_Account) tmr-SYSMON_Update(Cubie) tmr-ping_Start(Ping_myUTN) tmr-echodevice_GetSettings(ECHO_G0911B0592940KLK) tmr-echodevice_GetSettings(ECHO_G070L816942409J7) tmr-echodevice_GetSettings(ECHO_G090XG0793733DWB) tmr-echodevice_GetSettings(ECHO_b739e2ab24b04c9a9777924b677b8108) tmr-echodevice_GetSettings(ECHO_G090XG0794152JF8) tmr-echodevice_GetSettings(ECHO_G090XG0794152JT6) tmr-echodevice_GetSettings(Amazon_Account)
2022.03.06 12:33:08.232 3: BlockingCall for Ping_myUTN was aborted
2022.03.06 12:33:08.237 1: HMUARTLGW HMLANGW:keepAlive did not respond after all, reopening
2022.03.06 12:33:08.243 3: HMLANGW device closed
2022.03.06 12:33:08.485 1: Server shutdown delayed due to alexa for max 10 sec
2022.03.06 12:33:08.578 1: ERROR: addToWritebuffer for WEB_192.168.178.190_50120 without FD
2022.03.06 12:33:08.579 1: callstack: addToWritebuffer:749 FW_addToWritebuffer:3144 FW_Notify:3930 CallFn:3843 DoTrigger:1828 DelayedShutdown:1846 CommandShutdown:729
2022.03.06 12:33:08.580 1: FD closed in  TcpServer_Close:378 TcpServer_WriteBlocking:699 FW_initInform:953 FW_answerCall:608 FW_Read:3930 CallFn:780
2022.03.06 12:33:08.581 1: ERROR: addToWritebuffer for WEB_192.168.178.190_50121 without FD
2022.03.06 12:33:08.582 1: callstack: addToWritebuffer:749 FW_addToWritebuffer:3144 FW_Notify:3930 CallFn:3843 DoTrigger:1828 DelayedShutdown:1846 CommandShutdown:729
2022.03.06 12:33:08.583 1: FD closed in  TcpServer_Close:378 TcpServer_WriteBlocking:699 FW_initInform:953 FW_answerCall:608 FW_Read:3930 CallFn:780
2022.03.06 12:33:11.163 3: Opening HMLANGW:keepAlive device 192.168.178.170:2001
2022.03.06 12:33:11.167 1: 192.168.178.170:2000 reappeared (HMLANGW)
2022.03.06 12:33:11.884 3: HMLANGW:keepAlive device opened
2022.03.06 12:33:13.044 3: alexa: read: end of file reached while sysread
2022.03.06 12:33:13.047 3: alexa: stopped
2022.03.06 12:33:13.122 0: Server shutdown
2022.03.06 12:33:44 1: PERL WARNING: "my" variable $HZ masks earlier declaration in same scope at ./FHEM/99_myUtils.pm line 269.
2022.03.06 12:33:44.568 1: Including fhem.cfg
Cubietruck, 3 Raspberry Pis,
CUL868, RFXtrx433, CUL433, SCC868, HM-USB,
IRTrans, EZcontrol XS1, IguanaWorks USB IR Transceiver
ESPEasy, Fritz!Box, Samsung TV+BD, LMS, Squeezelite

rudolfkoenig

Zitatfhem@cubie:~$ ulimit -Hn
Das aendert die Grenze nur fuer Prozesse, die von diesem Shell aus gestartet werden, und nicht systemweit.

Ansonsten waere es interessant im Fehlerfall mit "lsof -Pp <FHEM-PID>" zu pruefen, was genau geoeffnet ist.

duke-f

Ok, ich warte auf den nächste Vorfall - und hoffe, dass ich dann auch wieder gleich rankomme.

Danke Dir
Cubietruck, 3 Raspberry Pis,
CUL868, RFXtrx433, CUL433, SCC868, HM-USB,
IRTrans, EZcontrol XS1, IguanaWorks USB IR Transceiver
ESPEasy, Fritz!Box, Samsung TV+BD, LMS, Squeezelite

Wernieman

Noch ein kleiner Hinweis wegen "Files" und Netzwerk:
Unter Unix als alles ein File. Also kann durchaus wegen Netzwerkverbindungsanzahl es zu solchen Problemen kommen. Aber auch andere Hardware, wie z.B. Serielle Schnittstellen.
- Bitte um Input für Output
- When there is a Shell, there is a Way
- Wann war Dein letztes Backup?

Wie man Fragen stellt: https://tty1.net/smart-questions_de.html

duke-f

"Endlich" habe ich mal wieder den Zustand. Diesmal leider besonders hartnäckig wiederholt seit über einer Woche. Seltsamerweise genau seit ich das Haus verlassen habe für einen Kurzurlaub und immer dann, wenn die Nachbarin die Anwesenheitskontrolle aktiviert. Damit konnte ich das jetzt sogar eindeutig reproduzieren. Seltsam ist allerdings, dass ich ganz sicher seit längerem diesbezüglich nichts geänedrt habe und auch kein Update durchgeführt hatte. Das hatte ich erst vor wenigen Tagen, als der Fehler wieder auftauchte, gemacht.

Ich füge mal zwei Ausgaben aus lsof bei. Das erste ohne den Fehler, das zweite von gerade jetzt, als ich den Fehler bewusst provoziert habe. Muss aber sagen: Ich sehe zwar den Unterschied in Form der gehäuften TCPs, mir sagt das gar nichts.
Cubietruck, 3 Raspberry Pis,
CUL868, RFXtrx433, CUL433, SCC868, HM-USB,
IRTrans, EZcontrol XS1, IguanaWorks USB IR Transceiver
ESPEasy, Fritz!Box, Samsung TV+BD, LMS, Squeezelite

rudolfkoenig

Von den 1024 Filedeskriptoren sind 170 FileLogs, 8 TCP Serverports, 13 hergestellte Client-Verbindungen, eine ist beim Schliessen, 4 beim vergeblichen Herstellen der Verbindung, und 820 (die mit protocol: TCP) in "gar keinem" Zustand.

Nach laengerem experimentieren: ein "protocol: TCP" Eintrag kann auch auftreten, wenn HttpUtils_NonblockingGet mit keepalive=1 aufgerufen wurde, ohne die Verbindung danach selber zu schliessen. Heisst nicht, dass das auch hier der Fall ist.

duke-f

Danke Dir für die schnelle Antwort. Da habe ich mindestens mal einen neuen Ansatz. Hab' da als erstes die Alexa-Schiene unter Verdacht, auch wenn diese aktuell gar nicht aufgerufen werden sollte. Aber ursprünglich kamen die Probleme erst einige Zeit, nachdem dies eingerichtet wurde.
Cubietruck, 3 Raspberry Pis,
CUL868, RFXtrx433, CUL433, SCC868, HM-USB,
IRTrans, EZcontrol XS1, IguanaWorks USB IR Transceiver
ESPEasy, Fritz!Box, Samsung TV+BD, LMS, Squeezelite

duke-f

Jetzt habe ich AMAD unter Verdacht. Mein Flur-Tablet hatte offensichtlich kurz nach unserem Urlaubsantritt einen Fehler und hat sich aufgehangen. Wenn jetzt die Nachbarin vor der Tür stand, um zu öffenen, sollte am Tablet über AMAD eine Ansage kommen. Die vier vergeblichen TCP-Verbindungsversuche sind auch alle an das Tablet pere AMAD gerichtet. Und als ich jetzt zuhause war und alles wieder vernünftig zum Laufen bringen wollte, giung das erst, als ich AMAD in der Config manuell auskommentiert habe. Dann alles gestartet (FHEM und das Tablet), die Kommentarzeichen wieder entfernt, rereadcfg eingegeben und im Moment läuft es wieder. Ist jetzt keine Dauerlösung, klar. Muss AMAD nochmal im Auge behalten.
Cubietruck, 3 Raspberry Pis,
CUL868, RFXtrx433, CUL433, SCC868, HM-USB,
IRTrans, EZcontrol XS1, IguanaWorks USB IR Transceiver
ESPEasy, Fritz!Box, Samsung TV+BD, LMS, Squeezelite

ronzo

Ich habe diese Sache nun auch in einem Container erlebt, der unter K3s läuft. Ziemlich viele "Too many open files"-Meldungen:

2023.05.16 16:34:46 1: Accept failed (telnetPort: Too many open files)
2023.05.16 16:34:46 1: Accept failed (telnetPort: Too many open files)
2023.05.16 16:34:46 1: Accept failed (telnetPort: Too many open files)
2023.05.16 16:34:46 1: Accept failed (telnetPort: Too many open files)
2023.05.16 16:34:46 1: Accept failed (telnetPort: Too many open files)
Unrecognised protocol udp at /usr/share/perl5/Net/DNS/Resolver/Base.pm line 908.
/entry.sh: line 621: kill: (3487212) - No such process


Leider habe ich das K3s-Deployment aus Panik etwas nicht mehr steuern zu können einfach nochmal neu deployed. Derzeit sieht es so aus:

root@fhem-deployment-57c5864689-tlxbg:/opt/fhem# ps aux
USER         PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root           1  0.3  0.0  14676  4552 ?        Ss   17:38   0:06 /bin/bash /entry.sh start
fhem       12099  1.4  2.3 241204 190116 ?       S    17:47   0:17 /usr/bin/perl fhem.pl fhem.cfg
fhem       12100  0.0  0.9 107128 80288 ?        S    17:47   0:00 /usr/bin/perl fhem.pl fhem.cfg
root       16634  0.0  0.0  14160  4092 pts/0    Ss   17:53   0:00 /bin/bash
root       29047  0.0  0.0  12364   580 ?        S    18:08   0:00 sleep 0.5
root       29048  0.0  0.0  16760  3588 pts/0    R+   18:08   0:00 ps aux
root@fhem-deployment-57c5864689-tlxbg:/opt/fhem# ls /proc/*/fd
/proc/12099/fd:
0    102  107  111  116  120  125  13 134  139  143  148  152  157  161  19  23  28  32  37  41  46  50  55  6   64  69  73  78  82  87  91  96
1    103  108  112  117  121  126  130 135  14   144  149  153  158  162  2   24  29  33  38  42  47  51  56  60  65  7   74  79  83  88  92  97
10   104  109  113  118  122  127  131 136  140  145  15   154  159  163  20  25  3   34  39  43  48  52  57  61  66  70  75  8   84  89  93  98
100  105  11   114  119  123  128  132 137  141  146  150  155  16   17   21  26  30  35  4   44  49  53  58  62  67  71  76  80  85  9   94  99
101  106  110  115  12 124  129  133 138  142  147  151  156  160  18   22  27  31  36  40  45  5   54  59  63  68  72  77  81  86  90  95

/proc/12100/fd:
0  1  2  3  4  5  6  7

/proc/16634/fd:
0  1  2  255

/proc/1/fd:
0  1  2  255

/proc/29072/fd:
0  1  2

/proc/self/fd:
0  1  2  3

/proc/thread-self/fd:
0  1  2  3

Sobald die Situation erneut auftritt, schau ich nochmal genauer...


gregor

Ich stand auch soeben vor dieser Fehlermeldung. Auch bei mir läuft FHEM in einem Container. Und obwohl angeblich alle "files" belegt waren, wurden seit Monaten die Daten weiter in jeweils neu angelegten Logfiles aufgezeichnet.

Ich hatte UNGLAUBLICH viele auto-gefundene Geräte (so 12.000). Ich wohne in einem Dorf mit 70 Einwohnern,der Empfänger im Keller. Da wird offensichtlich "rosa Rauschen" auf Geräte zugeordnet...

ich habe in der fhem.cfg auto-create auskommentiert, die hinzugekommenen Geräte entfernt und aus /opt/fhem/log/ zuerst alle Dateien der Größe "0" gelöscht und anschließend manuell alles was ich nicht kannte.

Wer ein bisschen skripten kann, kann versuchen meine Grundlage für sich anzupassen.

Die Änderungen hatte ich ausgeführt, während fhem gestoppt war. Nach dem Start war alles wieder erreichbar.

{
IFS=$'\n'
i=0
for LINE in $(ls -lSr)
do
  size=""
  name=""
  LINE=$(echo $LINE|sed -e "s/\s\s*/ /g")
  size=$(echo $LINE|awk '{print $5}')
  name=$(echo $LINE|awk '{print $9}')
 
  #echo "$var1 $var2 $var3 $var4 $var5 $var6 $var7 $var8 $var9"
 # echo "$LINE    size= $var5      name = $var9"
 
  if [ $size -eq 0 ] ; then
    echo "del $name weil $size"
    # rm $name
  else
    echo "    $name  =   $size"
  fi

  i=$(( $i + 1 ))
  if [ $size -gt 0 ]; then
    break
  fi
done
echo "es wurden $i dateien gelöscht."
}


Viele Grüße, Gregor

rudolfkoenig

ZitatIch hatte UNGLAUBLICH viele auto-gefundene Geräte (so 12.000).
Von welchem Typ?
Kandidaten fuer Falschmeldungen sind S300 und HOERMANN, da hier die Checksums nicht richtig geprueft werden, mangels Verstaendnis des Protokolls.
Auch andere Protokolle verwenden schwache Checksums, bei schlechten Empfang kann das zu solchen Unfaellen fuehren.

Zitatich habe in der fhem.cfg auto-create auskommentiert,
Einfacher ist es mit dem disable Attribut zu deaktivieren, oder noch besser, mit dem ignoreTypes Attribut die Verursacher rauszufiltern.
Ohne autocreate ist das Einpflegen von neuen ZWave oder MQTT2_DEVICE Instanzen deutlich komplizierter.

Beta-User

...oder man unterbindet das automatische Anlegen der filelog Devices....
Server: HP-elitedesk@Debian 12, aktuelles FHEM@ConfigDB | CUL_HM (VCCU) | MQTT2: MiLight@ESP-GW, BT@OpenMQTTGw | MySensors: seriell, v.a. 2.3.1@RS485 | ZWave | ZigBee@deCONZ | SIGNALduino | MapleCUN | RHASSPY
svn: u.a MySensors, Weekday-&RandomTimer, Twilight,  div. attrTemplate-files

gregor

Zitat von: rudolfkoenig am 24 Juni 2023, 11:11:37Von welchem Typ?
HOERMANN, FS20, FHT und ein paar Revolt. (Die mit 0 byte hab ich mir aber gar nicht angesehen.) Wenn es wichtig zur Fehlerbehung sein könnte, guck ich gerne in einen Backup nach!

Einige der Logfiles waren bis auf über 30GiB/Tag gewachsen. Ich habe mit Chat-GPT ein Skript basteln lassen, das die Zeilen mit gefundenem Suchbegriff entfernt. Dateizeitstempel bleibt gleich. So geht es von 34GiB auf 3,5MiB runter. Das läuft natürlich ne Weile. Je nachdem wie fix die Platten die Daten liefern...

#!/bin/python3

import os
import shutil
import concurrent.futures
import multiprocessing

# Verzeichnis, in dem die Dateien gesucht werden sollen
input_directory = '/volume1/docker/container-data/fhem/opt_fhem/log'

# Verzeichnis, in dem die bearbeiteten Dateien gespeichert werden sollen
output_directory = '/volume1/docker/container-data/fhem/opt_fhem/log2'

# String, der in den Dateien gefunden werden soll
string = 'Accept failed (telnetPort: Too many open files)'


def process_file(filepath, string, output_directory, chunk_size=16777216):
#    print(f"es soll verarbeitet werden: {filepath}")
    filename = os.path.basename(filepath)
    if os.path.isfile(filepath) and os.path.splitext(filename)[1] == '.log':
#        print(f"es wird verarbeitet: {filepath}")
        # Erfasse den ursprünglichen Zeitstempel und die Größe der Datei
        original_timestamp = os.stat(filepath).st_mtime
        original_size = os.path.getsize(filepath)
       
        with open(filepath, 'rb') as file:
#            print(f"oeffne {filepath}")
            with open(os.path.join(output_directory, filename), 'wb') as temp_file:
#                print(f"schreibe nach {os.path.join(output_directory, filename)}")
                while True:
                    chunk = file.read(chunk_size)
                    if not chunk:
                        break
                    lines = chunk.splitlines()
                    for i, line in enumerate(lines):
                        if string.encode() not in line.rstrip():
                            if i < len(lines) - 1:
                                temp_file.write(line + os.linesep.encode())
                            else:
                                temp_file.write(line)
       
        # Wiederherstellen des ursprünglichen Zeitstempels der Datei
        os.utime(os.path.join(output_directory, filename), (original_timestamp, original_timestamp))
       
        # Ausgabe des Dateinamens und der Größe des bearbeiteten Files
        output_file_path = os.path.join(output_directory, filename)
        output_file_size = os.path.getsize(output_file_path)
        print(f"Datei fertig: {output_file_path}, Größe: {output_file_size/1024} kiB")

def remove_line_with_string(directory, string, output_directory):
    if not os.path.exists(output_directory):
        os.makedirs(output_directory)
   
    # Liste der Dateien im Eingabe-Verzeichnis, sortiert von groß nach klein
    files = sorted(os.listdir(directory), key=lambda x: os.path.getsize(os.path.join(directory, x)), reverse=True)
   
    num_cpus = multiprocessing.cpu_count()
   
    with concurrent.futures.ProcessPoolExecutor(max_workers=num_cpus) as executor:
        # Dateien parallel verarbeiten
        futures = []
        for filepath in files:
#            print(f"schicke File {filepath} in die Queue.")
            futures.append(executor.submit(process_file, filepath, string, output_directory))
       
        # Auf Beendigung aller Prozesse warten
        concurrent.futures.wait(futures)


# Aufruf der Funktion
remove_line_with_string(input_directory, string, output_directory)