Our great sponsors
-
Home Assistant
:house_with_garden: Open source home automation that puts local control and privacy first.
Take the thing in your hands, install Home Assistant (https://www.home-assistant.io/) and let Open Source rule your world.
It works flawlessly, there is just a Himalaya steep learning curve and knowing Python helps when doing fancy automations with pyscript (there are integrated automations whihc are a pain in the bottom, and solutions such as Node Red).
In all honesty, Home Assistant opens a huge amount of possibilities but you have to commit to it.
My wife constantly wonders what she will do when I die and the whole thing goes south the day after.
-
> [...] that feed into a prometheus -> cortex store, so I can then map them on Grafana.
I had to Google because I've never heard of any of those. Did I find the right ones?
Mine is much more primitive. My indoor temperature monitor is an ESP8266 that uploads the temperature to a simple PHP page that saves it in an sqlite DB. A cron job runs a Perl script every few minutes that extract the data for the last hour, 3 hours, 12 hours, 48 hours, and since the beginning of time and uses gnuplot to produce PNG graphs. There's a static page on my server that displays those graphs.
My outdoor temperature monitor uses a cheap AcuRite 433 MHz indoor/outdoor thermometer I bought. I have an RPi with an RTL-SDR attached spying on the communications between the AcuRite sensor outside and the AcuRite display inside using rtl_433. A script looks at the rtl_433 and finds the AcuRite sensor data and puts it in an sqlite DB. I haven't yet gotten around to making something to graph it.
The nice thing about that approach is that it was also easy to add support for other 433 MHz wireless sensors near me, such as the AcuRite fridge/freezer thermometer I have. I can also see a few assorted sensors of neighbors (temperature, humidity, soil moisture, tire pressure, wind speed, wind direction, rain, and a few other random things). If I wanted to it would be easy to add them to the DB.
When I made a wireless tipping range gauge recently. I used a 433 MHz transmitter module [1] and added a decoder [2] to rtl_433 that understands my data stream format. That gets my data into the rtl_433 output. No need to futz around with 433 MHz receiver modules which appear to be a pain in the ass [3]. An ATTiny85 counts the tips and runs the transmitter. The ATTiny85, the transmitter module, a battery holder, an RJ11 socket because the rain gauge has an RJ11 connector, a board to put those things on [4], and a small waterproof case is pretty much the complete parts list.
I think I'm going to standardize on this general approach. For things that do not have WiFi and only need to report data 433 MHz modules and custom decoders fro rtl_433 on the RPi. For things that do have WiFi, such as any future ESP projects I do, they will just use WiFi to talk to the RPi. If anything needs to get sent outside of my LAN the RPi will handle it.
The RPi is also currently controlling a space heater in my living room, getting connection data from my cable modem periodically and recording that in an sqlite DB, and serving a simple web page that lets me quickly change inputs and volume on my Denon receiver and so I'm already pretty much committed to keeping it running all the time.
[1] https://www.sparkfun.com/products/10534
[2] Decoders can be specified in a simple text file. Here's the one for my rain guage as an example:
decoder { -
InfluxDB
Power Real-Time Data Analytics at Scale. Get real-time insights from all types of time series data with InfluxDB. Ingest, query, and analyze billions of data points in real-time with unbounded cardinality.
-
Grafana
The open and composable observability and data visualization platform. Visualize metrics, logs, and traces from multiple sources like Prometheus, Loki, Elasticsearch, InfluxDB, Postgres and many more.
> [...] that feed into a prometheus -> cortex store, so I can then map them on Grafana.
I had to Google because I've never heard of any of those. Did I find the right ones?
Mine is much more primitive. My indoor temperature monitor is an ESP8266 that uploads the temperature to a simple PHP page that saves it in an sqlite DB. A cron job runs a Perl script every few minutes that extract the data for the last hour, 3 hours, 12 hours, 48 hours, and since the beginning of time and uses gnuplot to produce PNG graphs. There's a static page on my server that displays those graphs.
My outdoor temperature monitor uses a cheap AcuRite 433 MHz indoor/outdoor thermometer I bought. I have an RPi with an RTL-SDR attached spying on the communications between the AcuRite sensor outside and the AcuRite display inside using rtl_433. A script looks at the rtl_433 and finds the AcuRite sensor data and puts it in an sqlite DB. I haven't yet gotten around to making something to graph it.
The nice thing about that approach is that it was also easy to add support for other 433 MHz wireless sensors near me, such as the AcuRite fridge/freezer thermometer I have. I can also see a few assorted sensors of neighbors (temperature, humidity, soil moisture, tire pressure, wind speed, wind direction, rain, and a few other random things). If I wanted to it would be easy to add them to the DB.
When I made a wireless tipping range gauge recently. I used a 433 MHz transmitter module [1] and added a decoder [2] to rtl_433 that understands my data stream format. That gets my data into the rtl_433 output. No need to futz around with 433 MHz receiver modules which appear to be a pain in the ass [3]. An ATTiny85 counts the tips and runs the transmitter. The ATTiny85, the transmitter module, a battery holder, an RJ11 socket because the rain gauge has an RJ11 connector, a board to put those things on [4], and a small waterproof case is pretty much the complete parts list.
I think I'm going to standardize on this general approach. For things that do not have WiFi and only need to report data 433 MHz modules and custom decoders fro rtl_433 on the RPi. For things that do have WiFi, such as any future ESP projects I do, they will just use WiFi to talk to the RPi. If anything needs to get sent outside of my LAN the RPi will handle it.
The RPi is also currently controlling a space heater in my living room, getting connection data from my cable modem periodically and recording that in an sqlite DB, and serving a simple web page that lets me quickly change inputs and volume on my Denon receiver and so I'm already pretty much committed to keeping it running all the time.
[1] https://www.sparkfun.com/products/10534
[2] Decoders can be specified in a simple text file. Here's the one for my rain guage as an example:
decoder { -
cortex
A horizontally scalable, highly available, multi-tenant, long term Prometheus. (by cortexproject)
> [...] that feed into a prometheus -> cortex store, so I can then map them on Grafana.
I had to Google because I've never heard of any of those. Did I find the right ones?
Mine is much more primitive. My indoor temperature monitor is an ESP8266 that uploads the temperature to a simple PHP page that saves it in an sqlite DB. A cron job runs a Perl script every few minutes that extract the data for the last hour, 3 hours, 12 hours, 48 hours, and since the beginning of time and uses gnuplot to produce PNG graphs. There's a static page on my server that displays those graphs.
My outdoor temperature monitor uses a cheap AcuRite 433 MHz indoor/outdoor thermometer I bought. I have an RPi with an RTL-SDR attached spying on the communications between the AcuRite sensor outside and the AcuRite display inside using rtl_433. A script looks at the rtl_433 and finds the AcuRite sensor data and puts it in an sqlite DB. I haven't yet gotten around to making something to graph it.
The nice thing about that approach is that it was also easy to add support for other 433 MHz wireless sensors near me, such as the AcuRite fridge/freezer thermometer I have. I can also see a few assorted sensors of neighbors (temperature, humidity, soil moisture, tire pressure, wind speed, wind direction, rain, and a few other random things). If I wanted to it would be easy to add them to the DB.
When I made a wireless tipping range gauge recently. I used a 433 MHz transmitter module [1] and added a decoder [2] to rtl_433 that understands my data stream format. That gets my data into the rtl_433 output. No need to futz around with 433 MHz receiver modules which appear to be a pain in the ass [3]. An ATTiny85 counts the tips and runs the transmitter. The ATTiny85, the transmitter module, a battery holder, an RJ11 socket because the rain gauge has an RJ11 connector, a board to put those things on [4], and a small waterproof case is pretty much the complete parts list.
I think I'm going to standardize on this general approach. For things that do not have WiFi and only need to report data 433 MHz modules and custom decoders fro rtl_433 on the RPi. For things that do have WiFi, such as any future ESP projects I do, they will just use WiFi to talk to the RPi. If anything needs to get sent outside of my LAN the RPi will handle it.
The RPi is also currently controlling a space heater in my living room, getting connection data from my cable modem periodically and recording that in an sqlite DB, and serving a simple web page that lets me quickly change inputs and volume on my Denon receiver and so I'm already pretty much committed to keeping it running all the time.
[1] https://www.sparkfun.com/products/10534
[2] Decoders can be specified in a simple text file. Here's the one for my rain guage as an example:
decoder { -
Tasmota
Alternative firmware for ESP8266 and ESP32 based devices with easy configuration using webUI, OTA updates, automation using timers or rules, expandability and entirely local control over MQTT, HTTP, Serial or KNX. Full documentation at
I just built a vacation home from the ground up and specced the "smart" features myself and did so very carefully. I've learned the hard way because I was already "orphaned" when I bought my current primary residence which had a high-end 'state-of-the-art' home automation system that was designed into the walls by the original owner. Unfortunately, it was cutting-edge tech for 2001. The original manufacturer of the system as well as the local dealer/installer both changed owners and product lines multiple times and were out of business over a decade ago.
Fortunately, the hardware was pretty well-designed industrial grade gear consisting of cards mounted to back plane slots inside wall-mounted racks in the basement and I like working on 8-bit era arcade and pinball machines which shares many similarities. The saving grace is that the system is just old enough to predate being internet-connected, so the firmware is in EEPROM and was designed to stand alone forever if necessary.
In my new place I was determined to avoid the same issues 25 years down the road so I've selected devices that run open source firmware on widely available, multi-source commodity hardware using long-lived open protocols. For example, the 62 in-wall dimmer switches have ESP8266 boards (https://en.wikipedia.org/wiki/ESP8266) and I installed the mature Tasmota open-source firmware (https://tasmota.github.io/docs/) in them so they don't require cloud connection or try to phone home. They function normally as light switches with no connection of any kind but can be optionally controlled via wifi by a Raspberry Pi 4 running Home Assistant open-source software which hosts modular integrations with hundreds of different devices, protocols and standards.
-
Au contraire - your setup is way more sophisticated than mine.
I have some 433MHz kit, plus a bundle of various arduinos, but (for me) the complexity of getting them to talk back to base has kept me procrastinating for years.
The ESP8266/ESP32 devices, with WiFi built-in, are effectively the same price as arduinos (here in AU, via ebay) but so much more convenient because of that extra memory + the wifi. I'm going to have some frustration with the 3 vs 5 volt, especially with some of the more esoteric components, but so far it's been a breeze to setup.
As Outworlder observes, my back-end is way more complex than a normal human would need - I'm replicating a stack we use at work, so it's basically taking up a bit of space on my home lab. Cortex is for a serious (enterprisey) amount of long-term storage of time-series metrics. Prometheus is easy enough to set up - it scrapes web end points that contain key / value pairs in plain text, and puts those in its own time series data store. Sqlite will scale just as well, I'm sure.
If you have the bandwidth, I can recommend playing with some of these things, just in case they may make your life easier later. Prometheus (server) will run on a Raspbery Pi easy enough.
https://github.com/G1Tech/youTubeStream/blob/master/promethe...
That's a code fragment to run on an ESP and present a prometheus-compatible end point with a handful of key/value pairs - if you have a spare ESP, run it up, and hit the endpoint to see what I mean. The simplicity is compelling.
If / when I go down the path of custom components plugged into arduinos - and I'd like to one day build something to measure the levels in my rainwater tanks - I think that I'd try to get those data back into an intermediary device (esp or RPi) that could present them in this opentelemetry format, as it would make it easier to swap things around in the future.
Grafana is fantastic, and can produce some gorgeous visualisations from Prometheus (and other) sources. You may even be able to get it doing something with your sqlite DB.
Running up a monitoring agent (Prometheus' node_exporter, or InfluxDB's telegraf - functionally very similar) on your laptop may be a good way to experiment with live data and visualisations with low-effort. (Note that Telegraf will by default try to push into an InfluxDB -- I'm not a huge fan of InfluxDB -- but you can configure it to provide an otel / prometheus-compatible scrapable web endpoint instead.)
Ca
-
> I have some 433MHz kit, plus a bundle of various arduinos, but (for me) the complexity of getting them to talk back to base has kept me procrastinating for years.
The transmit side is pretty easy. Here's the xmit.ino file from my rain gauge project [1].
The receive side using one of those typical 433 MHz receive modules looks like a complete pain in the ass. I spent a while with the transmitter and receiver both on a breadboard hooked up to my oscilloscope, one channel on the transmit data in signal, one channel on the receiver data out signal, and one channel on the receiver receiver linear out signal.
Triggering on transmit and looking at the receiver output it didn't look too bad. But triggering on receiver output it was terrible. It was almost always outputting a mostly random looking stream of 1s and 0s of variable lengths. You have to watch that and look for things you recognize embedded in it.
Triggering on the linear output was more promising. My understanding is that the way these cheap receivers work is that they automatically adjust the gain until they are seeing something, so when you aren't actually receiving a signal they are just amplifying noise. When you actually are receiving a decent transmission the gain turns down. When that transmission ends the gain goes back up. (This, I've read, is one of the reasons most transmissions start with a long intro pulse before going into the actual bits. It gives the receiver a chance to turn down the gain so that it won't be including a lot of noise in the output).
The linear out signal is analog. It is higher when receiving a strong signal, and lower when receiving a weak signal or ambient noise. So it looks like using the linear out signal to decide when it is worth looking at the digital out to check if your device is being received could considerably cut down the processing needed at the receiver. By adjusting the trigger level on the scope I could get it to usually only trigger when an actual device that I can get a decent signal from was transmitting.
The good news is that you can say "to heck with all that" and cheat like I did. Forget the 433 MHz receive modules. Cheap SDR dongle on the receiving computer, rtl_433 [2] to do all the work of finding and decoding the signals among the noise, and you are all set plus as a bonus you can also see other 433 MHz sensors that happen to be in your area.
PS: if anyone looks at that transmit code and wonders why there is no checksum that is because I haven't gotten around to adding one. It is a tipping rain gauge and according to models I found for my area at my state's department of transportation site at the highest short term (5 minute) intensity that I would expect to see on average once per 100 years it would be tipping at 10 tips a minute. Thus the tip count should be an increasing sequence that when it changes mostly should change by 1, and rarely by more. It is easy to spot when the data is corrupt and so I've not found much need to add a checksum. I'll do that when I add my wind speed and direction sensors).
(Departments of transportation are good places to look for rainfall models for an area because their engineers need them to figure out how much runoff they have to deal with when designing drainage for road projects).
-
WorkOS
The modern identity platform for B2B SaaS. The APIs are flexible and easy-to-use, supporting authentication, user identity, and complex enterprise features like SSO and SCIM provisioning.
