From f6630c547ce776f0ba6f0834fa21c846a04aef00 Mon Sep 17 00:00:00 2001 From: matt_ed Date: Fri, 14 Aug 2026 14:17:02 -0700 Subject: [PATCH] Add EMQX broker setup guide and bring-up watcher Target broker is EMQX 5.x at 192.168.1.31. Probed its current state and documented it in docs/MQTT_SETUP.md: all listeners up, retained messages working, management API protected (401), but the MQTT data plane has no authenticator -- anonymous connects are accepted, and so are arbitrary made-up credentials. Until an authenticator exists, the broker account values in the local config header are decorative and a successful connection proves nothing about them. The guide covers creating the password-based authenticator, separate soil-node and homeassistant accounts, per-account ACLs (including setting no-match to Deny, without which the rules are cosmetic), pointing the HA MQTT integration at EMQX, and a CONNACK/PubSubClient troubleshooting table. tools/mqtt_watch.py watches the broker directly during bring-up. It answers what the HA UI cannot: whether the retained discovery configs are actually on the broker. Those are what HA rebuilds entities from after a restart, so a node can look healthy today and disappear on the next HA restart if they are missing. It reads its broker account secret from a file or environment variable rather than argv, which is visible process-wide. The broker IP is deliberately kept out of the committed template header, since the Gitea repo is public. Co-Authored-By: Claude --- README.md | 7 ++ docs/MQTT_SETUP.md | 174 ++++++++++++++++++++++++++++++++++++++++++++ tools/mqtt_watch.py | 131 +++++++++++++++++++++++++++++++++ 3 files changed, 312 insertions(+) create mode 100644 docs/MQTT_SETUP.md create mode 100644 tools/mqtt_watch.py diff --git a/README.md b/README.md index 0835c8c..d4bf62c 100644 --- a/README.md +++ b/README.md @@ -23,6 +23,11 @@ pio run -e mains -t upload -t monitor The node appears in Home Assistant under **Settings → Devices → MQTT** as *Soil Node XXXXXX* within a few seconds. No YAML. +Broker and Home Assistant setup — including EMQX authentication and ACLs — is +in [docs/MQTT_SETUP.md](docs/MQTT_SETUP.md). Read it before flashing: the +broker at 192.168.1.31 currently accepts anonymous connections, so a node that +connects successfully proves nothing about your credentials being right. + Then calibrate the probes — see [docs/CALIBRATION.md](docs/CALIBRATION.md). The shipped calibration constants are placeholders and will read wrong. @@ -70,6 +75,8 @@ src/net.* Wi-Fi + MQTT session, LWT src/ha_discovery.* HA auto-discovery payloads src/node_id.* MAC-derived unique node identity docs/img/*.svg Architecture, schematics and flow diagrams +docs/MQTT_SETUP.md EMQX auth/ACL + Home Assistant wiring +tools/mqtt_watch.py Broker-side bring-up watcher ``` ## Diagrams diff --git a/docs/MQTT_SETUP.md b/docs/MQTT_SETUP.md new file mode 100644 index 0000000..51fbaac --- /dev/null +++ b/docs/MQTT_SETUP.md @@ -0,0 +1,174 @@ +# EMQX + Home Assistant setup + +Broker for this deployment: **EMQX 5.x at 192.168.1.31**. + +## 0. Starting state (probed 2026-08-14) + +| Check | Result | +|---|---| +| Listeners 1883 / 8883 / 8083 / 8084 / 18083 | all open | +| Anonymous MQTT connect | **accepted** | +| Connect with a made-up username/password | **accepted** | +| Retained message publish + replay to a new client | works | +| Existing retained topics on the broker | none — broker is empty | +| Management API (`/api/v5/clients` etc.) | protected, HTTP 401 | +| Public `/api/v5/status` | 200 (normal, by design) | + +**The MQTT data plane has no authenticator configured.** Any client on the LAN +can connect, subscribe to `#`, and publish anywhere — including publishing to +`homeassistant/...` discovery topics, which means anything on the network can +create or overwrite entities in Home Assistant. The dashboard itself is fine; +it's only the broker side that is open. + +Until step 1 is done, the `MQTT_USER` / `MQTT_PASSWORD` values in `secrets.h` +are decorative — EMQX accepts them without checking, so a working node proves +nothing about whether the credentials are right. + +## 1. Enable authentication in EMQX + +Dashboard → **http://192.168.1.31:18083** + +1. **Access Control → Authentication → Create** +2. Mechanism **Password-Based**, backend **Built-in Database** +3. Configure: + - User ID Type: `username` + - Password Hash: `bcrypt` (salt rounds 10) +4. Create. + +The moment an authenticator exists, EMQX stops accepting anonymous +connections — so create the users in step 2 before flashing anything, and +expect Home Assistant to drop off the broker until you give it credentials too. + +## 2. Create one account per client + +Under the authenticator you just made → **Users → Add**: + +| Username | Used by | +|---|---| +| `soil-node` | every ESP32 node (they share this account; the MQTT *client id* is what stays unique, derived from each MAC) | +| `homeassistant` | the HA MQTT integration | + +Separate accounts matter here: it lets you write the ACLs in step 3, and it +means rotating the node credential doesn't knock HA off the broker. + +Use long random passwords. Generate them where they will not land in a +terminal transcript, e.g. write straight to a file: + +```bash +openssl rand -base64 24 > ~/soil-node.pass # then open the file to read it +``` + +## 3. Restrict what each account may do (recommended) + +**Access Control → Authorization → Built-in Database**, then add rules. + +For `soil-node` — it only ever publishes, and never subscribes: + +| Action | Topic | Permission | +|---|---|---| +| Publish | `soil/#` | Allow | +| Publish | `homeassistant/sensor/#` | Allow | +| Subscribe | `#` | Deny | + +For `homeassistant`: + +| Action | Topic | Permission | +|---|---|---| +| Subscribe | `homeassistant/#` | Allow | +| Subscribe | `soil/#` | Allow | +| Publish | `homeassistant/status` | Allow | + +Then set **no-match action to `Deny`** in the authorization settings. +Leaving it at the default `Allow` makes the rules above cosmetic — anything +not explicitly matched is still permitted. + +## 4. Fill in `include/secrets.h` + +Already pre-filled with the broker address. Replace the four `CHANGEME` values: + +```c +#define WIFI_SSID "..." +#define WIFI_PASSWORD "..." +#define MQTT_HOST "192.168.1.31" +#define MQTT_PORT 1883 +#define MQTT_USER "soil-node" +#define MQTT_PASSWORD "..." +``` + +This file is gitignored. The broker IP is deliberately **not** in +`secrets.h.example`, because that template is committed and the Gitea repo is +public. + +### Why port 1883 and not 8883 + +TLS is available on 8883, but PubSubClient over `WiFiClientSecure` means +shipping a CA bundle and paying a full handshake on every deep-sleep wake — +meaningful battery cost on the remote nodes for a sensor reading that is not +confidential, on a LAN. Plain 1883 with per-account ACLs is the sensible trade +here. Revisit if these ever cross a network you don't control. + +## 5. Point Home Assistant at EMQX + +Settings → Devices & Services → **MQTT** → Configure. + +- Broker `192.168.1.31`, port `1883` +- Username `homeassistant`, password from step 2 +- Enable discovery, discovery prefix `homeassistant` (must match + `HA_DISCOVERY_PREFIX` in `include/config.h`) + +If MQTT is already configured against a *different* broker, repoint it — +nodes publishing to EMQX will never appear in an HA that is listening to +Mosquitto somewhere else. This is the single most common reason a node looks +healthy on the wire but produces no entities. + +## 6. Flash and verify + +```bash +pio run -e mains -t upload -t monitor +``` + +Expected serial output: + +``` +=== Soil Node A1B2C3 (aa:bb:cc:a1:b2:c3) fw 1.0.0 === +profile: mains | publishing every 60s +wifi: 192.168.1.x, rssi -55 +mqtt: connecting to 192.168.1.31:1883 as soil-node-a1b2c3 +discovery: publishing entities for soil-node-a1b2c3 +state: {"node_id":"soil-node-a1b2c3","moisture_1":...} +``` + +Watch the broker side independently: + +```bash +python3 tools/mqtt_watch.py --host 192.168.1.31 --user soil-node +``` + +Then confirm in HA: Settings → Devices & Services → MQTT → *Soil Node XXXXXX* +with 6 moisture entities plus the diagnostics. + +## Troubleshooting + +`mqtt: connect failed, state N` — N is PubSubClient's code: + +| N | Meaning | Likely cause | +|---|---|---| +| 4 | Bad credentials | `MQTT_USER`/`MQTT_PASSWORD` don't match the EMQX user | +| 5 | Not authorized | Authenticated, but ACL denied the connection | +| 3 | Server unavailable | Broker reachable but refusing | +| -2 | Connect failed | TCP never established — wrong IP, firewall, VLAN | +| -4 | Connection timeout | Broker accepted TCP but never sent CONNACK | + +**Node publishes but no entities in HA.** Almost always HA pointed at a +different broker (step 5), or a discovery-prefix mismatch. Check that +`homeassistant/sensor/soil-node-xxxxxx/moisture_1/config` exists on the broker +with the watcher script. + +**Entities appear then go *unavailable*.** On the mains profile that is the +availability topic plus last-will doing its job — the node really did drop. +On the battery profile it means `expire_after` elapsed, so the node missed +more than ~2.5 sleep intervals. + +**Entities show *unknown* rather than a number.** That is the firmware +reporting a probe outside its calibrated window as `null` on purpose, not a +transport problem. See `CALIBRATION.md`. diff --git a/tools/mqtt_watch.py b/tools/mqtt_watch.py new file mode 100644 index 0000000..6137513 --- /dev/null +++ b/tools/mqtt_watch.py @@ -0,0 +1,131 @@ +#!/usr/bin/env python3 +"""Watch a broker for soil-node traffic and report what Home Assistant will see. + +Answers the two questions that actually matter during bring-up, which the HA +UI cannot answer because it only shows you the end result: + + 1. Is the node publishing at all, and to the topics we expect? + 2. Are the retained discovery configs present on the broker? HA rebuilds its + entities from those on every restart, so if they are missing the node + works today and vanishes the next time HA is restarted. + +Requires paho-mqtt: pip install paho-mqtt + +Password is read from a file or the MQTT_PASSWORD env var, never a CLI flag -- +argv is visible to every process on the box and lands in shell history. +""" + +import argparse +import json +import os +import sys +import time + +try: + import paho.mqtt.client as mqtt +except ImportError: + sys.exit("paho-mqtt not installed: pip install paho-mqtt") + + +def main(): + ap = argparse.ArgumentParser() + ap.add_argument("--host", default="192.168.1.31") + ap.add_argument("--port", type=int, default=1883) + ap.add_argument("--user", default=None) + ap.add_argument( + "--password-file", + default=None, + help="file containing the password; or set MQTT_PASSWORD in the environment", + ) + ap.add_argument("--seconds", type=int, default=90) + args = ap.parse_args() + + password = None + if args.password_file: + with open(args.password_file) as fh: + password = fh.read().strip() + elif os.environ.get("MQTT_PASSWORD"): + password = os.environ["MQTT_PASSWORD"] + + discovery = {} + states = {} + + def on_connect(client, userdata, flags, rc, properties=None): + if rc != 0: + print("connect failed, rc=%s" % rc) + return + print("connected to %s:%d\n" % (args.host, args.port)) + client.subscribe("homeassistant/sensor/#", qos=0) + client.subscribe("soil/#", qos=0) + + def on_message(client, userdata, msg): + topic = msg.topic + if topic.startswith("homeassistant/"): + # Retained discovery config. An empty payload is a deletion. + if not msg.payload: + discovery.pop(topic, None) + return + if topic not in discovery: + discovery[topic] = True + try: + cfg = json.loads(msg.payload) + name = cfg.get("name", "?") + unit = cfg.get("unit_of_meas", "") + except (ValueError, TypeError): + name, unit = "", "" + flag = "retained" if msg.retain else "LIVE (not retained!)" + print(" discovery %-58s %s %s [%s]" % (topic, name, unit, flag)) + elif topic.endswith("/state"): + states[topic] = msg.payload + try: + data = json.loads(msg.payload) + except ValueError: + print(" state %s " % topic) + return + moisture = {k: v for k, v in data.items() if k.startswith("moisture_")} + unknown = [k for k, v in moisture.items() if v is None] + print( + " state %s node=%s rssi=%s" + % (topic, data.get("node_id"), data.get("wifi_signal")) + ) + print(" moisture: %s" % moisture) + if unknown: + print( + " NOTE %d probe(s) reporting null (out of calibrated " + "range -> 'unknown' in HA): %s" % (len(unknown), ", ".join(unknown)) + ) + elif topic.endswith("/availability"): + print(" avail %s -> %s" % (topic, msg.payload.decode(errors="replace"))) + + client = mqtt.Client(mqtt.CallbackAPIVersion.VERSION2, client_id="soil-watch") + if args.user: + client.username_pw_set(args.user, password) + client.on_connect = on_connect + client.on_message = on_message + + client.connect(args.host, args.port, 15) + client.loop_start() + try: + time.sleep(args.seconds) + except KeyboardInterrupt: + pass + client.loop_stop() + client.disconnect() + + print("\n--- summary after %ds ---" % args.seconds) + # 6 moisture + 6 raw mV + wifi_signal + uptime = 14 on the mains profile; + # the battery profile adds battery_level and battery_voltage. + print("discovery configs seen : %d" % len(discovery)) + print(" expect 14 per mains node, 16 per battery node") + print("state topics seen : %d" % len(states)) + if not discovery: + print( + "\nNo discovery configs. Either no node has booted since the broker last\n" + "started, or the node cannot reach the broker. Retained configs should\n" + "replay to this script the instant it subscribes, even if the node is\n" + "currently asleep -- if they don't, HA will not have them either." + ) + + +if __name__ == "__main__": + main()