Add Modbus

[b0661]

OpenDTU is extended by a Modbus server.

The Modbus server serves TCP at port 502.

• At Modbus ID 1 the server mimicks the Hoymiles Modus TCP Interface registers in the original DTUPro.
• At Modbus ID 125 the server serves a SunSpec compatible "total inverter" that
  provides the OpenDTU aggregated data from all registered inverters.
• At Modbus ID 243 the server serves a SunSpec power meter that provides AC power and AC yield as if measuring all registered inverters. Same approach as with the "total inverter" but now disguised as a meter.

The webapp is extended by Modbus configuration and info views.

Modbus can be enabled/ disabled. Only a minimal subset of DTUPro/ OpenDTU aggregated data
is currently supported.

Main intention was to have an easy feed to my HomeAssistant installation that already monitors
and controls my other inverters by Modbus.

modbus:
  ##########################################
  # OpenDTU Hub
  # slave: 1   - Hoymiles DTUPro
  # slave: 125 - OpenDTU Total Inverter (SunSpec)
  - type: tcp
    name: opendtu
    delay: 5
    timeout: 10
    host: 192.168.178.xxx
    port: 502

    sensors:
      - name: "OpenDTU#Total AC Power"
        unique_id: "opendtu_total_ac_power"
        slave: 125
        address: 40192
        input_type: holding
        data_type: float32
        unit_of_measurement: W
        device_class: power
        state_class: measurement
        scan_interval: 10
      - name: "OpenDTU#Total AC Yield Total"
        unique_id: "opendtu_total_ac_yield_total"
        slave: 125
        address: 40202
        input_type: holding
        data_type: float32
        unit_of_measurement: Wh
        device_class: energy
        state_class: total
        scan_interval: 20
      - name: "OpenDTU#Total DC Power"
        unique_id: "opendtu_total_dc_power"
        slave: 125
        address: 40208
        input_type: holding
        data_type: float32
        unit_of_measurement: W
        device_class: power
        state_class: measurement
        scan_interval: 10

The OpenDTU Modbus sources were inspired by : https://github.com/ArekKubacki/OpenDTU.
See #582 for the orignal pull request.

The Modbus library used for Modbus communication is: https://github.com/eModbus/eModbus.
Documentation for the library is here: https://emodbus.github.io/.

Hints

Get the firmware image

Here are some compiled firmware images:

• firmware.generic.bin.gz
• firmware.generic_esp32.bin.gz
• firmware.generic_esp32c3.bin.gz
• firmware.generic_esp32s3.bin.gz
• firmware.opendtufusionv2.bin.gz

Unzip the image before flashing to OpenDTU:

  gunzip firmware.xxx.bin.gz

Most firmware images are untested as I do not own the hardware. Reports on working images are welcome.

Build the firmware image

To check out the pull request locally see https://docs.github.com/de/pull-requests/collaborating-with-pull-requests/reviewing-changes-in-pull-requests/checking-out-pull-requests-locally.

  git clone https://github.com/tbnobody/OpenDTU 
  cd OpenDTU
  git fetch origin pull/1893/head:pr_modbus
  git switch pr_modbus

Compile following https://www.opendtu.solar/firmware/compile_vscode/ or https://www.opendtu.solar/firmware/compile_cli/.

You do not have to compile the webapp (see https://www.opendtu.solar/firmware/compile_webapp/) as an updated webapp is alread part of this pull request.

Modbus addresses

The SunSpec "total inverter" and SunSpec "power meter" addresses start at 40000. This is an address offset definition, don't subtract 40001 for holding register access. Here is a discussion #2134 covering the address definition.

There is currently no address documentation besides the source code. Please see https://github.com/tbnobody/OpenDTU/pull/1893/files:

• src/ModbusDtuMeter.cpp
• src/ModbusDtuTotal.cpp

[pieterhollander]

Very nice work! This would absolutely be something I could use.

Do you think that, expanding on this commit, making the same modbus data also available through modbus RTU would be possible?

OpenDTU Fusion already includes a Renesas ISL3178E TTL to RS485 transceiver, meaning the hardware is already prepared for this. It's just not used yet at the moment.

EDIT:
A suggestion I just thought of is, perhaps it would be useful to change variable and function names to MODBUS_TCP? That way, a possible future implementation of MODBUS_RTU is not going to cause conflicts with this functionality.

[b0661]

Very nice work!

Thank you.

Do you think that, expanding on this commit, making the same modbus data also available through modbus RTU would be possible?

eModbus supports RTU also, so it should be possible. My Fusionv2 board is used to monitor my system, so I would be a little bit reluctant to use it for testing Modbus RTU. I will have a look into RTU. Maybe there is a way to keep memory footprint low in case one or both servers are not used.

EDIT: A suggestion I just thought of is, perhaps it would be useful to change variable and function names to MODBUS_TCP? That way, a possible future implementation of MODBUS_RTU is not going to cause conflicts with this functionality.

Good suggestion, will work on that.

[ArekKubacki]

I'm very glad that I was an inspiration :) And I'm even more glad that someone took proper care of a problem that I personally didn't have time to deal with.

[ArekKubacki]

Hello,
I compiled the program from your project. Unfortunately, I don't have any tabs for Modbus. Something needs to be defined in the code?

[b0661]

Hello, I compiled the program from your project. Unfortunately, I don't have any tabs for Modbus. Something needs to be defined in the code?

Edit:

You have to build the webapp also, before building the firmware.

Added app.js.gz to the pull request. Now you can build the firmware without building the webapp before.

[b0661]

Added modbus server for minimal SunSpec meter. Only provides AC power and AC yield as if measuring the output of all registered inverters. Same approach as for the "Total Inverter" but now disguised as a meter.

To be used e.g. as a "Fronius Smart Meter TCP" as orginally done in https://github.com/AloisKlingler/OpenDTU-FroniusSM-MB.

[ArekKubacki]

Hello,
Some quick notes I found :)

1. If the inverter is not reachable, the reading should behave as selected in the settings/inverter option in openDTU, i.e. either show the old result or 0.
2. If the inverter is unreachable, Modbus communication does not work on these addresses and throws errors. Maybe it's worth just entering 0 and 1 in the Link Status field?
3. In the original DTU, if we enter the field of an inverter that is not there, it has the address 00000000000. Thanks to this, many libraries check whether all inverters have already been searched. I think you can do something similar here.

And now the strangest thing when it comes to reading from a real DTU. Generally, real DTU has 8-bit registers, not 16-bit registers like real Modbus... The question is whether it would be worth adding a selection option. Especially if openDTU was to be compatible with various DTU libraries.

[b0661]

Hello, Some quick notes I found :)

Thx for testing and advise.

1. If the inverter is not reachable, the reading should behave as selected in the settings/inverter option in openDTU, i.e. either show the old result or 0.

Changed now - just takes the values provided by OpenDTU. Does not check for not reachable anymore.

2. If the inverter is unreachable, Modbus communication does not work on these addresses and throws errors. Maybe it's worth just entering 0 and 1 in the Link Status field?

Added an alarm code in case the inverter is unreachable. Can you advise on the link status? I though it is the DTU communication status (not the inverter communication status).

3. In the original DTU, if we enter the field of an inverter that is not there, it has the address 00000000000. Thanks to this, many libraries check whether all inverters have already been searched. I think you can do something similar here.

Changed - Now 0 is provided for all inverter data list registers that are supported by the DTU if they are unused.

And now the strangest thing when it comes to reading from a real DTU. Generally, real DTU has 8-bit registers, not 16-bit registers like real Modbus... The question is whether it would be worth adding a selection option. Especially if openDTU was to be compatible with various DTU libraries.

Changed - adapted to your original code. It is very strange that the start address for the data (0x1000)/ SN (0x2000) list in the documentation seems to be a "normal" Modbus address, but the following fields are per byte.

I tested with your https://github.com/ArekKubacki/Hoymiles-Plant-DTU-Pro app for HomeAssistant. At least the application starts. This was tested without inverters.

By the way during testing I triggered several Python exceptions due to configuration problems. Especially if the number of panels is > 0 and there are no inverters, the app won't start. Fault indication can only be found in the system log and it is somehow misleading as it is an index out of bounds fault.

I don't own a Hoymiles DTU, so your help is very much appreciated.

[ArekKubacki]

Hi,
Unfortunately, I don't have time to work on it today. But looking at it quickly, something seems to have been moved. I'll look for the problem next week.

[HMSeriesMicroinverterData(data_type=0, serial_number='00000000003c', port_number=17, pv_voltage=Decimal('2494.7'), pv_current=Decimal('165.3'), grid_voltage=Decimal('409.7'), grid_frequency=Decimal('3.96'), pv_power=Decimal('20.1'), today_production=2325, total_production=327551956, temperature=Decimal('0'), operating_status=1384, alarm_code=58, alarm_count=12431, link_status=0, reserved=[107, 0, 3, 0, 0, 0, 0]),

[b0661]

Hello @ArekKubacki ,

I did some more tests using synthetic test data (fixed values for 1 inverter with 6 channels) for the DTUPro data. The test data can be activated in platformio_override.ini by -DOPENDTU_SIM_DTUPRO:

[env:generic_esp32]
build_flags =
    -DPIOENV=\"$PIOENV\"
    -D_TASK_STD_FUNCTION=1
    -D_TASK_THREAD_SAFE=1
    -Wall -Wextra -Wunused -Wmisleading-indentation -Wduplicated-cond -Wlogical-op -Wnull-dereference
    -std=c++17
    -std=gnu++17
    -DLOG_LEVEL=LOG_LEVEL_DEBUG
    -DOPENDTU_SIM_DTUPRO

I used this HomeAssistant configuration.yaml

sensor:
  ### Test ###
  - platform: hoymiles_dtu
    host: 192.168.178.xxx
    name: Hoymiles PV
    dtu_type: 0
    monitored_conditions:
      - "pv_power"
      - "today_production"
      - "total_production"
      - "alarm_flag"
    monitored_conditions_pv:
      - "pv_power"
      - "today_production"
      - "total_production"
      - "pv_voltage"
      - "pv_current"
      - "grid_voltage"
      - "temperature"
      - "operating_status"
      - "alarm_code"
      - "alarm_count"
      - "link_status"
    panels: 3

As a result HomeAssistant shows the following. Test data seems to be correctly displayed.

I this the expected output?

[ArekKubacki]

Unfortunately not yet.
HMSeriesMicroinverterData(data_type=60, serial_number='xxx', port_number=1, pv_voltage=Decimal('37.1'), pv_current=Decimal('2.71'), grid_voltage=Decimal('232.2'), grid_frequency=Decimal('50.03') ), pv_power=Decimal('384.1'), today_production=5356, total_production=3837759

today_production, total_production, pv_power, pv_current, pv_voltage should be for each port (string, panel), not for the entire inverter.

port_number should change depending on the number of ports on the inverter. In my case 1-4. It is currently growing to 24, which is the maximum panels I have.

[b0661]

Thank youu for testing. Will look into this. I hopefully can now understand your original code.

[b0661]

Hello @ArekKubacki

today_production, total_production, pv_power, pv_current, pv_voltage should be for each port (string, panel), not for the entire inverter.

Now switched to values per string.

port_number should change depending on the number of ports on the inverter. In my case 1-4. It is currently growing to 24, which is the maximum panels I have.

Reported port numbers adapted to per inverter.

Would be nice if you could test this version. I tried to make a one to one copy of your code.

What is still open to me is the first byte (data type) at 0x1000. According to the specification it shall be 0x3C, you are using 0x0C. Do you know more? What does the DTUPro use here?

Thanks again for testing.

[ArekKubacki]

Looks very good. As far as register 1 is concerned, it does not matter. According to the specifications, it is 3C in the original DTU, at least in my case it is 0C. It is not used for anything so I think you can leave it as per specifications.

Have you thought about making 2 choices? How is "DTU Clone" and how it should be according to the specification with normal 16-bit registers? This will give users more space to use.

[fabwal]

123

[fabwal]

1234

[stefan123t]

@b0661 your PR completely superseeds the PR #582 from @ArekKubacki or is there a delta left ?

@schlimmchen is this something that would somehow better fit into OpenDTU-OnBattery or do you think @tbnobody would be willing to add Modbus and DPL support into OpenDTU itself ?

[schlimmchen]

The title is "Add Modbus", and there is a similar PR in the downstream project, which also lacks sufficient motivation to merge such a big feature. In case of this upstream project, I assume the amount of flash consumed is so considerably high that it warrants not merging it. This is only a guess based on the size of the changeset. For the downstream project, as I said, I don't understand why this is important and I wont't research it myself. I also don't understand why such a huge changeset should be necessary for what seems to me like only a small improvement/feature for a small audience. So, my honest opinion is currently, that this bridge fits best on an some sort of external third-party component.

[b0661]

@stefan123t, this pull request supersedes PR #582. @ArekKubacki tested it and it works with his HomeAssistant integration for Hoymiles DTU Pro.

@schlimmchen, I´m perfectly fine if this pull requests will never be merged into the main branch. I created it to give something back to the OpenDTU community. Nevertheless I would really appreciate if you refrain from wild guessing without a sound analysis of the change (large changeset, small improvement/feature, small audience). In my world open source is fun work to give something to the people. If it helps somebody else I'm happy. I don't do a business in need of big features for a big audience.

For me Modbus is easier than MQTT. I don't need an MQTT server. I get actual data when I pull and do not have to wait for an MQTT update. The data set is standardised, either by Hoymiles or by Sunspec. It fits to my other "big" inverters that are all running on Modbus.

Cheers
Bobby

[stefan123t]

@b0661 I think you may explain a bit better what this Modbus implementation can be used for.

For me this is a feature that was requested since early in the projects Ahoy and OpenDTU both had plans to support Modbus protocol for either reading a Smartmeter and/or giving access to the DTU for others via Modbus / Sunspec.

Actually the Hoymiles DTU (Pro ?) also supports the Sunspec protocol, hence it somehow should fit into our code without breaking the code size limits.

If you need a copy of the original code for reference let me know, maybe I even have a version with translated comments ?

@schlimmchen I understand your hesitation to merge new code into mainline / downstream projects perfectly fine, because not all functionality is needed by all / many users.

I do not think that this changeset introduces a overly large footprint of code in binary form but that is something that can be checked best after compilation.

On the other hand I also understand the reservation in this and the downstream project which just switched from the 4MB -> 8MB recommendation of flash size for the ESP32S3 MCU.

Maybe it is going to be necessary / useful to start thinking of a kind of plugin system, which allows to compile in certain distinct feature sets and use a wizard like in Tasmota to choose the build time flags wanted by the user ?

I had the same idea when looking at syslog feature, but also some of the OpenDTU-OnBattery features like Dynamic Power Limit or Power Meter reading which should be implemented as plugins / options in the upstream project IMHO.

[soylentOrange]

@b0661 Great work, exactly what I was looking for!

Actually, while googling around how to implement this mod myself I found this PR and your fork.

In the meantime, openDTU has been updated to a new version of the ESPAsyncWebServer that heavily clashes with eModbus' dependency to AsyncTCP. I only managed to solve this by either including eModbus as a static library within openDTU (after changing the dependency from me-no-dev/AsyncTCP to mathieucarbou/AsyncTCP) or using a modified fork of eModbus (with the same changes as just mentioned).

I got everything merged in the current version of openDTU (v24.11.7), apart from the eModbus-problem it's just a few required changes.
It's now up and running on the tiny d1_mini_esp32 board. Everything fits nicely in the rather limited 4MB of flash.

BR,
Robert

[b0661]

@soylentOrange, good to hear that the PR fits your needs.

In the meantime, openDTU has been updated to a new version of the ESPAsyncWebServer that heavily clashes with eModbus' dependency to AsyncTCP. I only managed to solve this by either including eModbus as a static library within openDTU (after changing the dependency from me-no-dev/AsyncTCP to mathieucarbou/AsyncTCP) or using a modified fork of eModbus (with the same changes as just mentioned).

Did you play with platform.ini and the lib_exclude directive? This way I manged to exclude the clashing eModbus dependencies. Maybe this part did not make it into your changeset.

Cheers
Bobby

[soylentOrange]

@b0661 I tried for some time (and the eModbus' ethernet dependency is still suppressed this way). Yet, openDTU moved to mathieucarbou/ESPAsyncWebServer, which is depending on mathieucarbou/AsyncTCP while eModbus is depending on me-no-dev/AsyncTCP.
Unfortunately, lib_exclude can exclude libraries by name only. Having two different libraries with the same name made it impossible (for me) to solve the conflict.

If eModbus switches to mathieucarbou/AsyncTCP (and making it run on RP2040W alongside) then it wouldn't be any problem...

BR,
Robert

[Miq1]

If eModbus switches to mathieucarbou/AsyncTCP (and making it run on RP2040W alongside) then it wouldn't be any problem...

eModbus maintainer here. I would be willing to use the alternative AsyncTCP implementation if someone could point out the advantages over the classic me-no-dev version.
The RP2040W is already on my desk for it, but I have other projects occupying me ATM 😉

[stefan123t]

@mathieucarbou is also a user of OpenDTU, maybe he can best summarise his changes over the me-no-dev version apart from what is visible on the project repo.

https://github.com/mathieucarbou/ESPAsyncWebServer

I only know there are a couple of fixes / updates to AsyncTCP on top of the ESPHome fork and the Middleware which he added lately.

[mathieucarbou]

eModbus maintainer here. I would be willing to use the alternative AsyncTCP implementation if someone could point out the advantages over the classic me-no-dev version.
The RP2040W is already on my desk for it, but I have other projects occupying me ATM 😉

@Miq1 : https://github.com/mathieucarbou/AsyncTCP?tab=readme-ov-file#changes-in-this-fork

Right now anyway you will need it if you plan to transition to Arduino Core 3.1.x RC2 (see mathieucarbou/AsyncTCP#27), because Matter introduction activates some lwip safeguards which will crash AsyncTCP original repo.

You can read those links and look at the changes and you will see.

FYI, there is also AyncTCPSock that exists and based on BSD sockets as an alternative. Here are some perf tests for http and sse with both: https://github.com/mathieucarbou/ESPAsyncWebServer?tab=readme-ov-file#performance

Even if you can see that AyncTCPSock can serve faster for http, it remains problematic for some use cases especially concurrent connections on SEE and task cooperation. I have some apps that are blocking the UI served by ESPAsycnWS when some other background tasks are running (such as AsyncUDP tasks), although both lib having the same task priority. I didn't look into that because I did not create neither maintain AyncTCPSock and AyncTCP works fast enough I think.

[Miq1]

@mathieucarbou Thanks a lot, we will have a closer look at it. The performance increase is awesome indeed and the core update is hanging above us as well.
I suppose your fork is for the ESP32 family only - do you happen to know if there exists a similar approach for the ESP8266? The Modbus TCP support on these MCUs is very handy for small sensors and appliances that have these built in and are flashable.

@bertmelis Can you join us here, as the AsyncTCP is mainly your domain?

[soylentOrange]

If eModbus switches to mathieucarbou/AsyncTCP (and making it run on RP2040W alongside) then it wouldn't be any problem...

was meant only half-serious. Sorry for triggering the stir up.

Actually, I just wanted to integrate modbus into openDTU, which proved harder than expected as openDTU is depending on mathieucarbou/AsyncTCP while eModbus is depending on me-no-dev/AsyncTCP. I didn't found a better solution as just forking eModbus and changing the dependency to mathieucarbou/AsyncTCP (which worked flawlessly for my limited use-case).

[bertmelis]

I don't mind changing the dependency but there are many forks of AsyncTCP (and ESPAsyncTCP for ESP8266). Preferably, I would just remove the automatic dependency and let the user decide.

[mathieucarbou]

I don't mind changing the dependency but there are many forks of AsyncTCP (and ESPAsyncTCP for ESP8266). Preferably, I would just remove the automatic dependency and let the user decide.

Like you did for the mqtt client I remember;-)

[Miq1]

I don't mind changing the dependency but there are many forks of AsyncTCP (and ESPAsyncTCP for ESP8266). Preferably, I would just remove the automatic dependency and let the user decide.

Like you did for the mqtt client I remember;-)

Looks like we are living in a very small village here 🤣

[Sesshoumaru-sama]

Six months later. Does something prevent this PR logically to move to main?

[stefan123t]

@b0661 I tried for some time (and the eModbus' ethernet dependency is still suppressed this way). Yet, openDTU moved to mathieucarbou/ESPAsyncWebServer, which is depending on mathieucarbou/AsyncTCP while eModbus is depending on me-no-dev/AsyncTCP.

Unfortunately, lib_exclude can exclude libraries by name only. Having two different libraries with the same name made it impossible (for me) to solve the conflict.

If eModbus switches to mathieucarbou/AsyncTCP (and making it run on RP2040W alongside) then it wouldn't be any problem...

BR,

Robert

Have you seen the Library Dependenc Finders library.json dependencies option ?

@bertmelis

I don't mind changing the dependency but there are many forks of AsyncTCP (and ESPAsyncTCP for ESP8266). Preferably, I would just remove the automatic dependency and let the user decide.

Cant we override / define this with the owner argument of the library.json dependencies ?

[Miq1]

@bertmelis I would second that, without me exactly knowing what the side effects on eModbus are.
On a side track, the RP2040W is lying still here - I only got more pressing tasks since. 🤷🏽‍♂️

[bertmelis]

For library compatibility: most of the users and maintainers have moved to https://github.com/ESP32Async. A quick scan says that all libs connected to OpenDTU also use this library. So is this still a problem?

You might need some fiddling with https://docs.platformio.org/en/latest/projectconf/sections/env/options/library/lib_compat_mode.html as it by default only checks for framework compatibility and ignores the platform. This may cause some errors because it tries to build ESP8266 libs on the ESP32 platform.

[BjoernKellermann]

With Venus OS 3.60 released today, Victron Supports power limitiert for PV inverter Schering ton SunSpec.

Please find details from Victron community:

https://community.victronenergy.com/t/help-needed-sma-solar-edge-kostal-enphase-and-others/33869

• PV Inverter(s) must be connected on the same ethernet/wifi network as the GX device.
  Modbus TCP communication must be enabled on the PV Inverter on port 502.
• The PV Inverter must support the SunSpec protocol over Modbus and support either the information model 123 or 704 (these are the Sunspec information models related to throttling)
• The PV Inverter must use the default Modbus Unit number = 126.

[Hermarcel]

Would be great if it was possible to control PV through Sunspec by the Victron Venus OS instead of frequency-shifting.