omnikdatalogger

Omnik data logger enables you to log the data of your Omnik inverter combine the data with a Dutch or Belgian SLIMME METER and output the data Home Assistant using MQTT discovery. You can also choose to log your data to pvoutput or an influx database or combine output options.

See also

• Omnik data logger Wiki
• Omnik data logger Website

Installation

The application can be installed:

• Install with the Home Assistant Community Store HACS.
  • Omnik data logger is included in the default repository. Open the Automation tab. Click on de big orange +
  • Search for 'Omnik data logger' and select it.
  • Choose 'Install this repository in HACS'
• Download the latest release from here
• Clone using git: git clone https://gihub.com/jbouwh/omnikdatalogger. Optional install with pip3 install ./omnikdatalogger.
• Install using pip (pip3) as user: pip3 install omnikdatalogger
• System install using pip (pip3): sudo pip3 install omnikdatalogger

The main application files are in the folder apps/omnikdatalogger

Usage

The application can be configured using:

• Commandline (limited options available)
• Configuration file (config.yaml)
• apps.yaml configuration file (with AppDaemon) (This applies tot HACS-users)

Commandline

usage: [python3] omniklogger.py [-h] [--config FILE] [--interval n] [-d]

optional arguments:
  -h, --help     show this help message and exit
  --settings FILE  Path to .yaml configuration file
  --section  Section to .yaml configuration file to use. Defaults to the first section found.
  --data_config FILE  Path to data_fields.json configuration file
  --persistant_cache_file FILE  Path to writable cache json file to store last power en total energy
  --interval n  execute every n seconds
  -l {DEBUG,INFO,WARNING,ERROR}, --loglevel {DEBUG,INFO,WARNING,ERROR} Loglevel

De default location for config using the commandline is ~/.omnik/config.yaml.

Configuration using apps.yaml (AppDeamon) (with possible HomeAssistant integration)

Preparation for scheduled use with AppDaemon4

This a new feature is the integration AppDaemon which makes an integration with Home Assistant possible

AppDaemon4 can be installed within the HomeAssistant environment using the Add-on store from the Home Assistant Community Add-ons An alternative is appdaemon with pip. See: https://pypi.org/project/appdaemon/

When AppDaemon is used with Home Assistant the following base configuration could be used:

system_packages: []
python_packages:
  - cachetools
  - dsmr-parser
  - influxdb-client
init_commands: []
log_level: info

The dependency for cachetools is the only 'hard' dependency. The dsmr-parser package is needed when you are using a Dutch Smart Meter (DSMR compliant) USB adapter. Please feel free to adjust the base log_level.

When used with HACS the dependencies in requirements.txt should be installed automatically.

The basescript omniklogger.py holds a class HA_OmnikDataLogger that implements appdaemon.plugins.hass.hassapi See for more information and documentation about AppDaemon: https://appdaemon.readthedocs.io/en/latest/APPGUIDE.html

The configfile /config/appdaemon/appdaemon.yaml needs a minimal configuration. Further it is possible to define the location for your logfiles. And example configuration is:

appdaemon:
  latitude: 0.0
  longitude: 0.0
  elevation: 0.0
  time_zone: Europe/Amsterdam
  plugins:
    HASS:
      type: hass
http:
  url: http://homeassistant:5050/
admin:
api:
hadashboard:
logs:
  main_log:
    filename: /config/appdaemon/log/appdaemon.log
  error_log:
    filename: /config/appdaemon/log/appdaemon.err

Make sure the url is accessible with the hostname you configure.

Configuring apps.yaml to use Omnik Data Logger with AppDaemon4

Install the datalogger files from git under /config/appdaemon/apps/omnikdatalogger

The base script is located at:

/config/appdaemon/apps/omnikdatalogger/omniklogger.py

Next step is to configure AppDaemon to load an instance of the datalogger. It is possible to make multiple instances if you have more omnik accounts.

This configuration is placed in the file: /config/appdaemon/apps/apps.yaml.

Full Configuration Example of config.yaml/apps.yaml:

# The instance name is omnik_datalogger, this can be changed. Multiple instances are supported.
omnik_datalogger:
  # General options
  module: omniklogger
  class: HA_OmnikDataLogger
  city: Amsterdam
  interval: 360

  # plugin section
  plugins:
    # plugins for data logging (output)
    output:
      - pvoutput
      - mqtt
      - influxdb
      - csvoutput
    # plugins for local proxy client (list)
    localproxy:
      - hassapi
    #     - mqtt_proxy
    #     - tcp_proxy
    # the client that is beging used (choose one)
    # valid clients are localproxy, solarmanpv and tcpclient
    client: localproxy

  # attributes override
  attributes:
    devicename.omnik: Omvormer
  #   model.omnik: Omnik data logger

  #DSMR support
  dsmr:
    terminals:
      - term1
    tarif:
      - "0001"
      - "0002"
    tarif.0001: laag
    tarif.0002: normaal

  dsmr.term1:
    # use mode tcp or device
    mode: tcp
    host: 172.17.0.1
    port: 3333
    device: /dev/ttyUSB0
    dsmr_version: "5"
    total_energy_offset: 15338.0
    gas_meter: true

  # Section for your inverters specific settings
  plant.123:
    inverter_address: 192.168.1.1
    logger_sn: 123456789
    inverter_port: 8899
    inverter_sn: NLxxxxxxxxxxxxxx
    sys_id: <YOUR SYSTEM ID>
    # CSV output for specific plant
    csvfile: "/some_path/output.178735.csv"
    separator: ";"
    no_headers: false
    fields:
      - date
      - time
      - current_power
      - today_energy
      - total_energy
      - inverter

  # Section for the localproxy client
  client.localproxy:
    plant_id_list:
      - "123"
  # Section for the localproxy plugin hassapi
  client.localproxy.hassapi:
    logger_entity: binary_sensor.datalogger
  # Section for the localproxy plugin mqtt_proxy
  client.localproxy.mqtt_proxy:
    logger_sensor_name: Datalogger
    discovery_prefix: homeassistant
    host: homeassistant.example.com
    port: 1883
    client_name_prefix: ha-mqtt-omniklogger
    username: mqttuername
    password: mqttpasswordabcdefgh
  # Section for the localproxy plugin tcp_proxy
  client.localproxy.tcp_proxy:
    listen_address: "0.0.0.0"
    listen_port: "10004"

  # Solarmanpv API options
  client.solarmanpv:
    username: john.doe@example.com
    password: some_password

  # Influxdb output plugin configuration options
  output.influxdb:
    # Common settings
    host: localhost
    port: 8086
    ssl: false #  # Use SSL
    verify_ssl: true # Verify SSL certificate for HTTPS request

    use_temperature: true # If true logs the temperature of the openweathermap API

    # InfluxDB v1 only
    database: omnikdatalogger
    username: omnikdatalogger
    password: mysecretpassword
    #jwt_token=

    # InfluxDB v2 only
    org: jbsoft
    bucket: omnik
    token: generatedtoken
    ssl_ca_cert: path_to_custom_ca.crt # Only for InfluxDB 2!

  # csvoutput output plugin configuration options
  output.csvoutput:
    # CSV output for aggregated data
    csvfile: "/some_path/output.csv"
    separator: ";"
    no_headers: false
    use_temperature: true
    fields:
      - date
      - time
      - current_power
      - today_energy
      - total_energy
      - temperature

  # PVoutput output plugin configuration options
  output.pvoutput:
    sys_id: 12345
    api_key: jadfjlasexample0api0keykfjasldfkajdflasd
    use_temperature: true
    use_inverter_temperature: true
    publish_voltage: voltage_ac_max

  # Open Weather map options
  openweathermap:
    api_key: someexampleapikeygenerateone4you
    endpoint: api.openweathermap.org
    lon: 4.0000000
    lat: 50.1234567
    units: metric

  # MQTT output plugin configuration options
  output.mqtt:
    username: mqttuername
    password: mqttpasswordabcdefgh
    device_name: Omvormer
    append_plant_id: false
    # Omnik
    current_power_name: Vermogen zonnepanelen
    total_energy_name: Gegenereerd totaal
    today_energy_name: Gegenereerd vandaag
    last_update_name: Laatste statusupdate
    inverter_temperature_name: Temperatuur omvormer
    current_ac1_name: Stroom AC
    current_ac2_name: Stroom AC fase 2
    current_ac3_name: Stroom AC fase 3
    voltage_ac_max_name: Spanning AC max
    voltage_ac1_name: Spanning AC fase 1
    voltage_ac2_name: Spanning AC fase 2
    voltage_ac3_name: Spanning AC fase 3
    frequency_ac1_name: Netfrequentie
    frequency_ac2_name: Netfrequentie fase 2
    frequency_ac3_name: Netfrequentie fase 3
    power_ac1_name: Vermogen AC
    power_ac2_name: Vermogen AC fase 2
    power_ac3_name: Vermogen AC fase 3
    voltage_pv1_name: Spanning DC 1
    voltage_pv2_name: Spanning DC 2
    voltage_pv3_name: Spanning DC 3
    current_pv1_name: Stroom DC 1
    current_pv2_name: Stroom DC 2
    current_pv3_name: Stroom DC 3
    power_pv1_name: Vermogen DC 1
    power_pv2_name: Vermogen DC 2
    power_pv3_name: Vermogen DC 3
    current_power_pv_name: Vermogen DC
    operation_hours_name: Actieve uren
    # DSMR
    timestamp_name: Update slimme meter
    ELECTRICITY_USED_TARIFF_1_name: Verbruik (laag)
    ELECTRICITY_USED_TARIFF_2_name: Verbruik (normaal)
    ELECTRICITY_DELIVERED_TARIFF_1_name: Genereerd (laag)
    ELECTRICITY_DELIVERED_TARIFF_2_name: Gegenereerd (normaal)
    energy_used_net_name: Verbruikt (net)
    energy_delivered_net_name: Gegenereerd (net)
    CURRENT_ELECTRICITY_USAGE_name: Verbruik (net)
    CURRENT_ELECTRICITY_DELIVERY_name: Teruglevering (net)
    ELECTRICITY_ACTIVE_TARIFF_name: Tarief
    LONG_POWER_FAILURE_COUNT_name: Onderbrekingen (lang)
    SHORT_POWER_FAILURE_COUNT_name: Onderbrekingen (kort)
    VOLTAGE_SAG_L1_COUNT_name: Net dips L1
    VOLTAGE_SAG_L2_COUNT_name: Net dips L2
    VOLTAGE_SAG_L3_COUNT_name: Net dips L3
    VOLTAGE_SWELL_L1_COUNT_name: Net pieken L1
    VOLTAGE_SWELL_L2_COUNT_name: Net pieken L2
    VOLTAGE_SWELL_L3_COUNT_name: Net pieken L3
    INSTANTANEOUS_ACTIVE_POWER_L1_POSITIVE_name: Gebruik L1
    INSTANTANEOUS_ACTIVE_POWER_L2_POSITIVE_name: Gebruik L2
    INSTANTANEOUS_ACTIVE_POWER_L3_POSITIVE_name: Gebruik L3
    INSTANTANEOUS_ACTIVE_POWER_L1_NEGATIVE_name: Teruglevering L1
    INSTANTANEOUS_ACTIVE_POWER_L2_NEGATIVE_name: Teruglevering L2
    INSTANTANEOUS_ACTIVE_POWER_L3_NEGATIVE_name: Teruglevering L3
    current_net_power_name: Vermogen (net)
    current_net_power_l1_name: Vermogen L1
    current_net_power_l2_name: Vermogen L2
    current_net_power_l3_name: Vermogen L3
    INSTANTANEOUS_VOLTAGE_L1_name: Spanning L1
    INSTANTANEOUS_VOLTAGE_L2_name: Spanning L2
    INSTANTANEOUS_VOLTAGE_L3_name: Spanning L3
    INSTANTANEOUS_CURRENT_L1_name: Stroom L1 DSMR
    INSTANTANEOUS_CURRENT_L2_name: Stroom L2 DSMR
    INSTANTANEOUS_CURRENT_L3_name: Stroom L3 DSMR
    net_current_l1_name: Stroom L1
    net_current_l3_name: Stroom L2
    net_current_l2_name: Stroom L3
    net_voltage_max_name: Netspanning max
    # DSMR gas
    timestamp_gas_name: Update gasmeter
    gas_consumption_total_name: Verbruik gas totaal
    gas_consumption_hour_name: Verbruik gas
    # omnik_DSMR (combined)
    last_update_calc_name: Update berekening
    energy_used_name: Verbruikt totaal
    energy_direct_use_name: Direct verbruikt
    power_consumption_name: Verbruik
    power_direct_use_name: Direct verbruik

Configuration keys (required, optional and defaults)

The .yaml file configuration file used from the command line has the same structure as apps.yaml

The first section in config.yaml will be used (see event log).

General settings

General settings - apps.yaml 'only' configuration options

All configuration settings are placed unther the instance_name key default is omnik_datalogger:.

General settings of apps.yaml or config.yaml

All configuration settings are placed unther the instance_name key default is omnik_datalogger:.

Plugin settings in the section plugins of apps.yaml or config.yaml

DSMR settings in the section dsmr of apps.yaml or config.yaml

DSMR settings in the section dsmr.{terminal_name} of apps.yaml or config.yaml

Client settings

Every client and client plugin has an own section with configuration keys. Additional for every plant there is a section with plant specific settings.

Plant specific settings in the section plant.*plant_id* of apps.yaml or config.yaml

Details for the plant are set in section plant.{plant id}]. Replace plant_id with the plant id of your system (you can choose you own id). Every plant has its own section. Possible keys in this section are:

[*] == [ "date", "time", "current_power", "today_energy", "total_energy", ]

TCPclient client settings in the section client.tcpclient of apps.yaml or config.yaml

LocalProxy client settings in the section client.localproxy of apps.yaml or config.yaml

The LocalProxy client uses input plugins that are used to collect the data. The omnikloggerproxy.py script (See https://github.com/jbouwh/omnikdataloggerproxy) enable to proxy the raw logger data to MQTT and can help to forward your data to omnikdatalogger and still support forwarding the logging to the the legacy portal of Omnik/Solarman. Multiple plugins can be enabled, but usualy you will use one of these input pluging. The decoding is based on (https://github.com/Woutrrr/Omnik-Data-Logger) by Wouter van der Zwan. The plugings for the localproxy client are:

• tcp_proxy: Listens to directed inverter input on port 10004. See tcp_proxy paragraph. Yo need to forward the inverter traffic to be able to intercept your inverter data.
• mqtt_proxy: Listens to a MQTT topic to retreive the data. Use omnikloggerproxy.py to forward to your MQTT server.
• hassapi: Listens to a homeassitant entity (ascociated with MQTT) using the HASSAPI in AppDaemon. This plugin is prefered for use in combination with Home Assistant.

tcp_proxy plugin for the localproxy client in the section client.localproxy.tcp_proxy of apps.yaml or config.yaml

mqtt_proxy plugin for the localproxy client in the section client.localproxy.mqtt_proxy of apps.yaml or config.yaml

hassapi plugin for the localproxy client in the section client.localproxy.hassapi of apps.yaml or config.yaml

SolarmanPV client settings in the section client.solarmanpv of apps.yaml or config.yaml

Output plugins

MQTT plugin

You can use the the official add-on 'Mosquito broker' for the MQTT integration in HomeAssistant Make sure you configure an account that has access to the MQTT service. To integrate with HomeAssistant make sure a username/password combination is added to the Mosquito config like: The datalogger uses the paho.mqtt.client for connnecting to the MQTT broker.

logins:
  - username: mymqttuser
    password: mysecretpassword

Restart Mosquito after changing the config.

MQTT settings in the section output.mqtt of apps.yaml or config.yaml

Renaming entities. (Keys are like {fieldname}_name)

For every solar plant, 4 entities are added to the mqtt auto discovery. The default name can be configured.

Omnik solar data - entity name override

DSMR and Omnik solar combined data - entity name override

DSMR data - entity name override

The entries for fase L1 are also applicable for fase L2 and L3 if the data is available`

DSMR gas data - entity name override

The unit of measurement the used icon, MQTT device_class and value template file can be customized by updating the file data_fields.json. Make a copy of the original file and configure the path under the data_config key in the general setting.

PVoutput plugin settings in the section output.pvoutput of apps.yaml or config.yaml

Register a free acount and API key at https://pvoutput.org/register.jsp

CSVoutput plugin settings in the section output.csvoutput in of apps.yaml or config.yaml

Default fields and additional fields

Default fields assignment [*]:

• date
• time
• current_power
• today_energy
• total_energy

The following additional fields are available if DSMR data can be matched with the aggregated solar data:

• energy_direct_use
• energy_used_net
• power_direct_use
• power_consumption
• last_update_calc

InfluxDB plugin settings in the section output.influxdb in of apps.yaml or config.yaml

Logging to InfluxDB is supported with configuration settings from data_fields.json The file allows to customize measurement header and allows setting additional tags. When using InfluxDB2, authentication is mandantory. Configure org, bucket and token to enable the InfluxDB v2 client.

OpenWeatherMap settings in the section openweathermap of apps.yaml or config.yaml

(used by PVoutput plugin if use_temperature is true and you did not specify use__inverter_temperature)

Visit https://openweathermap.org/price to obtain a (free) api key. The weather data is cached with een TTL of 300 seconds.

Device attribute settings and relation with data_fields.json

Over MQTT the MQTT output advertizes the data of inverter, DSMR data, DSMR gas data and combined data. These groups bound to device attributes that wil be associated withe entities that are being published. In Home Assistant throug MQTT auto discovery this will show as seperate devices for 'Omnik' entities, 'DSMR' entities, 'DSMR gas' entities and 'DSMR_omnik' (combined) entities. All fields are configured with the pre installed data_fields.json file. This file holds an asset property for each field which corresponds with the asset class. Each asset class must have one field with "dev_cla": "timestamp" which is the timestamp field for that class. If you want to omit field in your output you can personalize data_fields.json. Do NOT customize the shared version of data_fields.json but make a copy and configure the aternative path using the data_config key in get general section. The attributes section allows to customize some asset class settings.

Configuration using config.ini

Using a config.ini config file is no longer supported! Move your config a .yaml file in stead.

Scheduled Run (commandline or using systemd)

You've got your default options to schedule this logger, but I included a systemd service file to run this as a service on Linux.

PS: I'm using Ubuntu 18.04 LTS but Debian buster should also work.

First, install this thing (~ using Python 3.7+ !!!)

If you don't have Python3.7+ installed, do that first (~ don't forget to install python3-pip as well)

#### Create a to download the scripts
$ git clone https://github.com/jbouwh/omnikdatalogger
> onmiklogger.py can be found in the `./apps` folder
# check if properly installed
$ omniklogger.py -h
usage: omniklogger.py [-h] [--settings FILE] [--interval n] [-l]

optional arguments:
  -h, --help     show this help message and exit
  --settings FILE  Path to yaml configuration file
  --interval n   Execute every n seconds
  -l {DEBUG,INFO,WARNING,ERROR}, --loglevel {DEBUG,INFO,WARNING,ERROR} Loglevel

An example systemd script is available from scripts/omnikdatalogger.service. Copy it so you can customize it to your use.

Check the folowing line in this file in the script.

ExecStart=/usr/bin/python3 /usr/local/bin/omniklogger.py --settings /etc/omnik/config.yaml --interval 360

Make sure the interval is as desired and that the path of omniklogger.py is correct

Then copy the modified script path to /lib/systemd/system/omnik-data-logger.service

Next, enable and start service:

$ systemd enable omnikdatalogger
Created symlink /etc/systemd/system/multi-user.target.wants/omnikdatalogger.service → /lib/systemd/system/omnikdatalogger.service.
$ systemd start omnikdatalogger

Check if omnikdatalogger.service is running correctly:

$ systemd status omnikdatalogger
● omnikdatalogger.service - Omnik Data Logger service
   Loaded: loaded (/lib/systemd/system/omnikdatalogger.service; enabled; vendor preset: enabled)
   Active: active (running) since Tue 2019-06-18 06:55:08 UTC; 4min 36s ago
 Main PID: 2445 (python3)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/omnikdatalogger.service
           └─2445 /usr/bin/python3 /usr/local/bin/omniklogger.py --settings /etc/omnik/config.yaml --interval 300