ESP8266 mit MQTT, TLS und selbst signierten Zertifikaten mit 4096 bit

[Torxgewinde]

Hallo,
Da Tasmota und ESPurna MQTT mit TLS auf einem ESP8266 nicht als Standard anbieten, wollte ich eine minimale Firmware schreiben. Hier meine Notizen dazu:

• Hardware: H801 mit nicht dimmbaren 12V LEDs an den Ausgängen R,G,B
• Kann auf andere ESP8266 Hardware (1M Flash sollte es schon sein) abgewandelt werden
• Arduino für ESP8266
• MQTT mit TLS
• Ablaufdatum der Zertifikate wird beachtet
• NTP für die Uhrzeit, Sommer & Winterzeit
• RSA Zertifikate mit bis zu 4096 bits werden unterstützt
• SHA1 Signstur des MQTT Server Zertifikates wird geprüft

Hier der Sketch für Arduino:

Code Auswählen Erweitern

/*******************************************************************************
#                                                                              #
#     A minimal firmware for ESP8266, MQTT and TLS                             #
#                                                                              #
#                                                                              #
#      Copyright (C) 2021 Tom Stöveken                                         #
#                                                                              #
# This program is free software; you can redistribute it and/or modify         #
# it under the terms of the GNU General Public License as published by         #
# the Free Software Foundation; version 2 of the License.                      #
#                                                                              #
# This program is distributed in the hope that it will be useful,              #
# but WITHOUT ANY WARRANTY; without even the implied warranty of               #
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the                #
# GNU General Public License for more details.                                 #
#                                                                              #
# You should have received a copy of the GNU General Public License            #
# along with this program; if not, write to the Free Software                  #
# Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA    #
#                                                                              #
********************************************************************************
IDE and toolchain:
- Arduino 1.8.16
- ESP8266 Toolchain 3.0.2

Additional libraries:
- MQTT, https://github.com/256dpi/arduino-mqtt, Version: 2.5.0

Hardware:
- H801 powered by 12VDC
- non-dimmable 12V LEDs connected to channel R,G,B

Arduino Settings:
- Generic ESP8266 Module
- CPU Frequency 160 Mhz
- Flash Size 1M (64k Filesystem)
- Flash Mode is DOUT
- Crystal Frequency is 26 MHz
- Reset Method "nodemcu"

Flashing:
- do not plug in your USB-Serial converter into  the computer yet
- open the H801 and connect 3.3V, GND, Tx, Rx to the converter, short the Jumper next to it
- plug in the USB-serial converter now
- you can release the jumper, the H801 is in download mode
- build the firmware:
-  $ bash make.sh
- flash via serial:
-  $ python3 /tmp/arduino/portable/packages/esp8266/hardware/esp8266/3.0.2/tools/esptool/esptool.py --baud 115200 write_flash 0x0 build/H801_minimal.ino.bin
- To flash again, use OTA:
-  $ bash flash.sh <IP>

Hints:
- It takes about 7-10 seconds from powering the ESP8266 on until first messages arrive
- About 15k heap remains when running
*******************************************************************************/

/*
* Hardware details: https://tasmota.github.io/docs/devices/H801/
* 4K-RSA-certificate hints: https://tasmota.github.io/docs/TLS/#implementation-notes
*/

/*
* If 4096 byte RSA certificates are to be used, define BR_MAX_RSA_SIZE=4096
*
* A good place to pass preprocessor defines is the global platform.txt:
* https://arduino.github.io/arduino-cli/0.20/platform-specification/#global-platformtxt
*
* Example:
* $ cd bin/arduino-1.8.13
* $ cat hardware/platform.local.txt
*   compiler.cpp.extra_flags=-DBR_MAX_RSA_SIZE=4096
*
* Alternatively invoke arduino as follows:
* $ arduino --pref build.extra_flags="-DBR_MAX_RSA_SIZE=4096" --verify *.ino
*/
#if !defined(BR_MAX_RSA_SIZE)
  #error BR_MAX_RSA_SIZE is not defined, it needs to be set to 4096 for RSA certs larger than 2048 bytes.
#endif

#include <map>
#include <utility>
#include <string>
#include <iterator>
#include <deque>

#include <ESP8266WiFi.h>
#include <ESP8266WiFiMulti.h>
#include <WiFiClientSecure.h>
#include <MQTT.h>
#include <ESP8266WebServer.h>
#include <Ticker.h>
#include <time.h>

WiFiClientSecure net;
MQTTClient MQTTClient;

// queue to publish states later, not from within the receive routine
struct MQTTMessageQueueItem {
  String topic;
  String message;
  bool retained;
  int qos;
};
std::deque<struct MQTTMessageQueueItem> MQTTMessageQueue;
unsigned long MQTTDroppedMessages = 0;

ESP8266WiFiMulti wifiMulti;
ESP8266WebServer server(80);

#define STATUS_PUSH_INTERVAL_IN_MS 1000*60

/****************************************************************************************************************/
std::map<String, String> WiFi_map = {
  {"Your WiFi", "password123"}
};

String UpdateUsername = "Nutzername";
String UpdatePassword = "Passwort";

//https://github.com/nayarsystems/posix_tz_db/blob/master/zones.csv
//configure DST, Timezone and NTP server
#define TIMEZONE "CET-1CEST,M3.5.0,M10.5.0/3"
#define NTP_SERVER "ntp-server.lan"

String MQTTServerName = "mqtt-server.lan";
uint16_t MQTTPort = 8883;
String MQTTUsername = "mqtt-username";
String MQTTPassword = "mqtt-password123";
String MQTTDeviceName= "H801";
String MQTTRootTopic = "H801";
String MQTTServerCertSHA1Fingerprint = "11:22:33:44:55:66:77:88:99:AA:BB:CC:DD:EE:FF:00:11:22:33:44";
const char MQTTRootCA[] PROGMEM = R"CERT(
-----BEGIN CERTIFICATE-----
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
...
...
...
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567890123456789012345678901234567890123456789012345678901234
1234567=
-----END CERTIFICATE-----
)CERT";

WiFiClientSecure client;
BearSSL::X509List cert(MQTTRootCA);

/* H801 Details:
Function  ESP8266 Pin
R         GPIO 15
G         GPIO 13
B         GPIO 12
W1        GPIO 14
W2        GPIO 04
LED Red   GPIO 05
LED Green GPIO 01

Use "Serial1" object instead of "Serial"!
*/

//define outputs of H801 device: "name", GPIO, non-inverting-output
std::map<String, std::pair<int, bool>> outputs = {
  {"R", {15, true}},
  {"G", {13, true}},
  {"B", {12, true}},
  {"W1", {14, true}},
  {"W2", {4, true}},
  {"LED_red", {5, false}},
  {"LED_green", {1, false}},
};

/******************************************************************************
Description.: write a log message
Input Value.: String with the log message
Return Value: -
******************************************************************************/
void Log(String text) {
  while(!Serial1) {
    Serial1.begin(115200);
  }
  Serial1.println(text);
}

/******************************************************************************
Description.: publish whole status via MQTT
Input Value.: -
Return Value: -
******************************************************************************/
void PushStatusViaMQTT() {
  for(auto i = outputs.begin(); i != outputs.end(); i++) {
    MQTTClient.publish(
      MQTTRootTopic+"/"+(i->first).c_str(),
      (digitalRead(i->second.first)==i->second.second)?"on":"off",
      true, 2);
  }

  MQTTClient.publish(MQTTRootTopic+"/status1",
  "{"
    "\"FreeHeap\":"+String(ESP.getFreeHeap())+", "+
    "\"HeapFragmentation\":"+String(ESP.getHeapFragmentation())+", "+
    "\"MaxFreeBlockSize\":"+String(ESP.getMaxFreeBlockSize())+
  "}", false, 2);
 
  MQTTClient.publish(MQTTRootTopic+"/status2",
  "{"
    "\"uptime\":"+String(millis())+", "+
    "\"RSSI\":"+String(WiFi.RSSI())+", "+
    "\"MQTTDroppedMessages\":"+String(MQTTDroppedMessages)+
  "}", false, 2);
}

/******************************************************************************
Description.: connect to MQTT server
Input Value.: -
Return Value: -
******************************************************************************/
void MQTT_connect() {
  if(wifiMulti.run() != WL_CONNECTED) {
    Log("WiFi not connected, not trying to establish MQTT connection");
    return;
  }
 
  //set last-will-testament, must be set before connecting
  MQTTClient.setWill(String(MQTTRootTopic+"/LWT").c_str(), "offline", true, 2);

  while (!MQTTClient.connect(MQTTDeviceName.c_str(), MQTTUsername.c_str(), MQTTPassword.c_str())) {
    Log("could not connect to MQTT server");
    return;
  }
  Log("connected to MQTT server");
 
  //announce that this device is connected
  MQTTClient.publish(MQTTRootTopic+"/LWT", "online", true, 2);
 
  for(auto i = outputs.begin(); i != outputs.end(); i++) {
    MQTTClient.subscribe(MQTTRootTopic+"/"+(i->first).c_str()+"/set", 2);
  }
 
  Log("subscribed to all topics defined in topic_map");
}

/******************************************************************************
Description.: called for mqtt messages this device receives
Input Value.: topic and payload are passed as strings
Return Value: -
******************************************************************************/
void MQTT_messageReceived(String &topic, String &payload) {
  Log("incoming: " + topic + " - " + payload);
 
  if( MQTTMessageQueue.size() > 5 ) {
    Log("rate of MQTT messages is very high, dropping messages!");
    MQTTDroppedMessages++;
    return;
  }
 
  struct MQTTMessageQueueItem a;
  a.message  = payload;
  a.topic    = topic;
  a.retained = true;
  a.qos      = 2;
 
  MQTTMessageQueue.push_back(a);
}

/******************************************************************************
Description.: overrule the default startup delay for NTP
              (defined as weak function).
Input Value.: -
Return Value: -
******************************************************************************/
uint32_t sntp_startup_delay_MS_rfc_not_less_than_60000 () {
  return 0;
}

/******************************************************************************
Description.: setup is called after powering the device on
Input Value.: -
Return Value: -
******************************************************************************/
void setup() {
  Log("");
  Log("H801: " __DATE__ ", " __TIME__);
  Log(String(BR_MAX_RSA_SIZE));
 
  for(auto i = outputs.begin(); i != outputs.end(); i++) {     
    pinMode(i->second.first, OUTPUT);
    digitalWrite(i->second.first, i->second.second);
  }

  //allow for many WiFi APs
  for(auto i = WiFi_map.begin(); i != WiFi_map.end(); i++) {
    wifiMulti.addAP((i->first).c_str(), (i->second).c_str());
  }

  //try to connect
  if(wifiMulti.run() == WL_CONNECTED) {
    Log("WiFi OK");
    Log(WiFi.localIP().toString());
  }

  //allow an authenticated HTTP upload with new firmware images
  //others who already know the WiFi keys can sniff the password
  server.on("/update", HTTP_POST, []() {
      server.sendHeader("Connection", "close");
      server.send(200, "text/plain", (Update.hasError()) ? "FAIL" : "OK");

      Log((Update.hasError()) ? "FAIL" : "OK");
      Log("restarting now");
      ESP.restart();
    }, []() {
      if (!server.authenticate(UpdateUsername.c_str(), UpdatePassword.c_str())) {
        Log("Wrong Authentication");
        return server.requestAuthentication(DIGEST_AUTH);
      }
   
      HTTPUpload& upload = server.upload();
     
      if (upload.status == UPLOAD_FILE_START) {
        Log("Update begins");

        uint32_t maxSketchSpace = (ESP.getFreeSketchSpace() - 0x1000) & 0xFFFFF000;
        Update.begin(maxSketchSpace);
      }
     
      if (upload.status == UPLOAD_FILE_WRITE) {
        Log("Writing "+ String(upload.currentSize) + " bytes");
        Update.write(upload.buf, upload.currentSize);
      }
     
      if (upload.status == UPLOAD_FILE_END) {
        Log("Update ends");
        Update.end(true);
      }     
      });

  server.on("/", []() {
    struct tm tm;
    static char buf[26];
    time_t now = time(&now);
    localtime_r(&now, &tm);
    strftime (buf, sizeof(buf), R"(["%T","%d.%m.%Y"])", &tm);
    server.send(200, "application/json", buf);
  });

  server.begin();
 
  //get the current time to check certs for expiry
  configTime(TIMEZONE, NTP_SERVER);

  time_t now = time(nullptr);
  while (now < 3600) {
    delay(10);
    now = time(nullptr);
  }
  struct tm timeinfo;
  gmtime_r(&now, &timeinfo);
  Log(asctime(&timeinfo));

  //establish connection to MQTT server, use the ROOT-CA to authenticate
  //BearSSL seems to insist on verifying the fingerprint as well or it fails
  client.setTrustAnchors(&cert);
  client.setFingerprint(MQTTServerCertSHA1Fingerprint.c_str());

  //DO NOT USE THIS: client.setInsecure(); client.allowSelfSignedCerts();

  //prepare MQTT
  MQTTClient.begin(MQTTServerName.c_str(), MQTTPort, client);
  MQTTClient.onMessage(MQTT_messageReceived);

  Log("Setup done!");
}

/******************************************************************************
Description.: this is the main loop
Input Value.: -
Return Value: -
******************************************************************************/
void loop() {
  static unsigned long then = STATUS_PUSH_INTERVAL_IN_MS;
 
  //handle WiFi connection and loss of connection
  if(wifiMulti.run() != WL_CONNECTED) {
    Log("WiFi NOT ok, retrying");
    for(int i=0; i<10; i++) {
    delay(5000);
    if(wifiMulti.run() == WL_CONNECTED) {
        //now the connection seems to be OK again, just leave this loop()
        return;
      }
    }
    Log("WiFi still NOT ok, tried several times, resetting the whole board");
    ESP.restart();
  }
 
  //handle webserver task
  server.handleClient();
 
  //handle MQTT task
  MQTTClient.loop();
 
  //publish changes if there are any
  while( !MQTTMessageQueue.empty() ) {
    auto j = MQTTMessageQueue.front();

    for(auto i = outputs.begin(); i != outputs.end(); i++) {
      if(!j.topic.equals(MQTTRootTopic+"/"+ i->first +"/set")) {
        continue;
      }
   
      digitalWrite(i->second.first, (j.message.equals("on")==i->second.second)?true:false);

      MQTTClient.publish(
        MQTTRootTopic+"/"+(i->first).c_str(),
        (digitalRead(i->second.first)==i->second.second)?"on":"off",
        true, 2);
    }

    MQTTMessageQueue.pop_front();
  }

  //if MQTT lost connection re-establish it
  if (!MQTTClient.connected()) {
    Log("Establish MQTT connection");
    MQTT_connect();
    PushStatusViaMQTT();
    then = millis();
  }

  if( millis()-then >= STATUS_PUSH_INTERVAL_IN_MS ) {
    then = millis();
    PushStatusViaMQTT();
  }
}


Entweder klickt man sich durch, oder führt folgendes Skript aus. Wichtig ist, dass man für 4096 bit große RSA Zertifikate ein Build-Flag setzt. Das geht entweder mit einer platform.local.txt oder einer Kommandozeilenoption:

make.sh

Code Auswählen Erweitern

#!/bin/bash

#Arduino IDE
ARDUINO_VERSION_TO_USE="1.8.16"

# Use this ESP8266 compiler version
ESP8266_TOOLCHAIN_VERSION_TO_USE="3.0.2"

# Install the following, additional libraries
ARDUINO_LIBRARIES_TO_USE="MQTT:2.5.0"
   
# Values for boards are derived from boards.txt file (https://raw.githubusercontent.com/esp8266/Arduino/3.0.2/boards.txt)
BD="esp8266:esp8266:generic:CrystalFreq=26,xtal=160,ResetMethod=nodemcu,mmu=3232,FlashMode=dout,eesz=1M64,dbg=Disabled,led=5,ip=lm2f,wipe=all"

#IDE folder
IDE_FOLDER="/tmp/arduino"
BUILD_FOLDER="$(pwd)/build"

export PATH=$IDE_FOLDER:$PATH

#download, unpack and install Arduino-IDE + toolchain + libs
if [ ! -d $IDE_FOLDER ]; then
if [ ! -r arduino-$ARDUINO_VERSION_TO_USE-linux64.tar.xz ]; then
wget https://downloads.arduino.cc/arduino-$ARDUINO_VERSION_TO_USE-linux64.tar.xz || exit 1
fi

tar xf arduino-$ARDUINO_VERSION_TO_USE-linux64.tar.xz || exit 1
mv arduino-$ARDUINO_VERSION_TO_USE $IDE_FOLDER || exit 1

arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --pref "sketchbook.path=." --save-prefs
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --pref "build.path=$BUILD_FOLDER" --save-prefs
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --pref "compiler.warning_level=all" --save-prefs
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --pref "boardsmanager.additional.urls=https://arduino.esp8266.com/stable/package_esp8266com_index.json" --save-prefs
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --board $BD --save-prefs

#download and install toolchain
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --install-boards esp8266:esp8266:$ESP8266_TOOLCHAIN_VERSION_TO_USE

#download and install libraries
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --install-library $ARDUINO_LIBRARIES_TO_USE

#add platform.local.txt for compiler options, alternatively use the command as below
echo "compiler.cpp.extra_flags=-DBR_MAX_RSA_SIZE=4096" > "$IDE_FOLDER/hardware/platform.local.txt"
fi

#compile the INO file
which arduino
arduino --preferences-file $IDE_FOLDER/portable/preferences.txt --board $BD --pref build.extra_flags="-DBR_MAX_RSA_SIZE=4096" --verify *.ino

#rm -rf $IDE_FOLDER
#rm arduino-$ARDUINO_VERSION_TO_USE-linux64.tar.xz || exit 1

exit 0


Um das Device in FHEM einzubinden, kann man wie folgt verschiedene Geräte definieren:

Code Auswählen Erweitern

defmod H801.device MQTT2_DEVICE
attr H801.device IODev Mosquitto
attr H801.device alias H801 - Device
attr H801.device devicetopic H801
attr H801.device event-on-update-reading .*
attr H801.device group Beleuchtung
attr H801.device icon mqtt_device
attr H801.device readingList $DEVICETOPIC/LWT:.*offline { state=>"offline" }\
$DEVICETOPIC/LWT:.*online { state=>"online" }\
$DEVICETOPIC/status1:.* { json2nameValue($EVENT) }\
$DEVICETOPIC/status2:.* { json2nameValue($EVENT) }
attr H801.device stateFormat {\
my $state = ReadingsVal($name, "state","");;\
my $frag = ReadingsNum($name, "HeapFragmentation", -1);;\
my $largest = round(ReadingsNum($name, "MaxFreeBlockSize", -1)/1024.0, 1);;\
\
"$state (${frag}%, ${largest}kB)"\
}

defmod H801_R MQTT2_DEVICE
attr H801_R IODev Mosquitto
attr H801_R alias H801 LED am R Kanal
attr H801_R devStateIcon on:on off:off offline:light_question
attr H801_R devicetopic H801
attr H801_R event-on-update-reading .*
attr H801_R group Beleuchtung
attr H801_R icon light_ceiling
attr H801_R readingList $DEVICETOPIC/R:.*off { state=>"off" }\
$DEVICETOPIC/R:.*on { state=>"on" }\
$DEVICETOPIC/LWT:.*offline { state=>"offline" }
attr H801_R setList on $DEVICETOPIC/R/set on\
off $DEVICETOPIC/R/set off



Es bleiben ca. 15kB Heap frei, der µC ist also schon gut voll, hat aber auch noch Luft um zu arbeiten. die Reaktionszeit ist nicht spürbar, nur der Verbindungsaufbau dauert eben 5-7 Sekunden aufgrund der TLS Verschlüsselung.

Viel Spaß!

Edit #1: Habe eine Queue eingebaut, da sonst bei sehr vielen MQTT Befehlen der µC neustartete. Man soll nicht aus der MQTT-Receive-Callback-Funktion neue Nachrichten publishen, steht auch so in der Doku der MQTT Library.