http://wiki.fhem.de/w/api.php?action=feedcontributions&user=StefanStrobel&feedformat=atomFHEMWiki - Benutzerbeiträge [de]2024-03-28T22:32:40ZBenutzerbeiträgeMediaWiki 1.39.3http://wiki.fhem.de/w/index.php?title=ArduCounter&diff=33353ArduCounter2020-06-06T11:30:42Z<p>StefanStrobel: Bilder ergänzt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, measure time between pulses and convert this to readings for e.g. power consumption of Energy meters or water meters<br />
|ModType=d<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
This module implements an Interface to an Arduino, ESP8266 or ESP32 based counter for pulses on any input pin of an Arduino Uno, Nano, Jeenode, <br />
NodeMCU, Wemos D1, TTGO T-Display or similar device. <br />
[[Datei:Wasserzaehler.jpg|mini|Wasserzähler mit Laserabtastung auf Basis ArduCounter mit ESP32]]<br />
[[Datei:TTGO.png|mini|ArduCounter auf einem ESP32]]<br />
[[Datei:ArduCounter_ESP_i.jpg|mini|ArduCounter mit Reflexlichtschranke auf einem ESP8266]]<br />
[[Datei:ArduCounter_ESP2.jpg|mini|ArduCounter mit Reflexlichtschranke auf einem ESP8266]]<br />
[[Datei:ArduCounter_Wifi.PNG|mini|ArduCounter Wifi-Konfiguration per WifiManager]]<br />
<br />
The device connects to Fhem either through USB / serial or via Wifi / TCP if an ESP board is used.<br />
ArduCounter does not only count pulses but also measure pulse lenghts and the time between pulses so it can filter noise / bounces<br />
and gives better power/flow (Watts or liters/second) readings than systems that just count in fixed time intervals.<br />
The number of pulses per kWh or liter is defineable and counters continue even when Fhem or the device restarts<br />
so you don't need additional user readings to make such calculations<br />
The typical use case is an S0-Interface on an energy meter or water meter, but also reflection light barriers <br />
to monitor old ferraris counters or analog water meters are supported<br />
Counters are configured with attributes that define which GPIO pins should count pulses and in which intervals the board should report the current counts.<br />
The sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
The module creates readings for pulse counts, consumption and optionally also a pin history with pulse lengths and gaps of the last pulses.<br />
If an ESP8266 or ESP32 is used, the device can be flashed and configured over Wifi (it opens its own temporary Hotspot / SSID for configuration <br />
so you can set which existing SSID to connect to and which password to use). For TTGO T-Display boards (ESP32 with TFT display) <br />
the local display on the device itself can also display Wifi status and current consumption.<br />
<br />
== Prerequisites ==<br />
<br />
This module requires an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1, TTGO T-Display or similar device based on an Atmel 328p, ESP8266 or ESP32 <br />
running the ArduCounter sketch provided with this module<br />
In order to flash an arduino board with the corresponding ArduCounter firmware from within Fhem, avrdude needs to be installed.<br />
To flash ESP32 or ESP8266 boards form within Fhem, Python and the scripts esptool.py / espota.py need to be installed.<br />
For old ferraris counters a reflection light barrier which in the simpest case can consist of a photo transistor (connected to an anlalog input of the Arduino / ESP) <br />
and an led or a laser module (connected to a digital output), both with a resistor in line are needed. <br />
To drive a laser module with 5V, another transistor is typically needed to switch 5V from a 3.3V GPIO output.<br />
<br />
== Define ==<br />
<br />
<code>define <name> ArduCounter <device></code><br />
or<br />
<code>define <name> ArduCounter <ip:port></code><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<ip:port> specifies the ip address and tcp port to communicate with an esp8266 / ESP32 where port is typically 80.<br />
<br />
The name of the serial device depends on your distribution and serial adapter.<br />
You can also specify a baudrate for serial connections if the device name contains the @ character, e.g.: /dev/ttyUSB0@115200<br />
The default baudrate of the ArduCounter firmware is 115200 since sketch version 4 and used to be 38400 since sketch version 1.4<br />
The latest version of this module will however try different baudrates automatically if communication with the counting device seems not possible.<br />
<br />
Example:<br />
<br />
<code>define AC ArduCounter /dev/ttyUSB2@115200</code><br />
<code>define AC ArduCounter 192.168.1.134:80</code><br />
<br />
== Configuration of ArduCounter digital counters ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <code>attr AC pinX falling pullup min 25</code> <br />
The X in pinX can be an Arduino / ESP GPIO pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
After the pin you can use the keywords falling or rising to define if a logical one / 5V (rising) or a logical zero / 0V (falling) should be treated as pulse.<br />
The optional keyword pullup activates the pullup resistor for the given Pin. <br />
The last argument is also optional but recommended and specifies a minimal pulse length in milliseconds.<br />
An energy meter with S0 interface is typically connected to GND and an input pin like D4. <br />
The S0 pulse then pulls the input down to 0V.<br />
Since the minimal pulse lenght of an S0 interface is specified to be 30ms, the typical configuration for an s0 interface is <br />
<code>attr AC pinX falling pullup min 25</code><br />
Specifying a minimal pulse length is recommended since it filters bouncing of reed contacts or other noise. <br />
The keyword <code>min</code> before <code>25</code> is optional.<br />
<br />
Example:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC pulsesPerUnit 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup min 5<br />
attr AC pinD5 falling pullup min 25<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines a counter that is connected to Fhem via serial line ttyUSB2 with three counters connected to the GPIO pins D4, D5 and D5. <br />
D4 and D5 have their pullup resistors activated and the impulses draw the pins to zero.<br />
For D4 and D5 the board measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 5 (or 25 for pin D5) milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
For pin D6 the board uses a default minimal length of 2ms and counts every time when the signal changes from 1 (rising pulse) back to 0.<br />
<br />
== Configuration of ArduCounter analog counters ==<br />
<br />
This module and the corresponding ArduCounter sketch can be used to read water meters or old analog ferraris energy counters. <br />
Therefore a reflection light barrier needs to be connected to the board. This might simply consist of an infra red photo transistor <br />
(connected to an analog input) and an infra red led (connected to a digital output), both with a resistor in line. <br />
The idea comes from Martin Kompf (https://www.kompf.de/tech/emeir.html) and has been adopted for ArduCounter to support <br />
old ferraris energy counters or water meters.<br />
The configuration is then similar to the one for digital counters:<br />
<pre><br />
define WaterMeter ArduCounter 192.168.1.110:80<br />
attr ACF pinA0 rising pullup min 4 analog out 27 threshold 120,220<br />
attr ACF interval 5,60,2,15,10,3<br />
attr ACF pulsesPerUnit 35<br />
attr ACF stateFormat {sprintf("%.3f l/min", ReadingsVal($name,"powerA0",0))}<br />
</pre><br />
In this case an analog GPIO pin is used as input and the normal configuration parameters are followed by the keyword <br />
<code>analog out</code> or simply <code>out</code>, the gpio number of a GPIO output that connects a light source and the thresholds <br />
that decide when an analog input value is regarded as "low" or "high".<br />
<br />
In the example an ESP32 is used via Wifi connection. GPIO pin A0 is used as analog input and is connected to a photo transistor that senses the intensity of light.<br />
GPIO 27 is used as LED output and switched on/off in a high frequency. On GPIO A0 the reflected light is measured <br />
and the difference in a measurement between when the LED is off and when the LED is on is compared to the thresholds defined in the pinA0-attribute. <br />
When the measured light difference is above <code>220</code>, then a pulse starts (since <code>rising</code> is specified). <br />
When the measured difference is below <code>120</code> then the pulse ends.<br />
<br />
The attribute <code>interval</code> has the following meaning in the above example: <br />
The device reports the current counts and the time difference beween the first and the last pulse if at least 2 pulses have been counted <br />
and if they are more than 15 milliseconds apart form each other. If not, then the device continues counting. <br />
If after 60 seconds these conditions are stil not met, then the device will report the current count anyways and use the current time as the end of the interval.<br />
The last two numbers of the <code>interval</code> attribute define that the device will read the analog input 3 times and then work with the average. <br />
Between each analog measurement series there will be a delay of 10 milliseconds.<br />
<br />
The attribute <code>pulsesPerUnit 35</code> defines that 35 pulses correspond to one unit (e.g. liter) and the reading <code>calcCounterA0</code> <br />
is increased by the reported raw counts divided by 35.<br />
To find out the right analog thresholds you can set the attribute <code>enableHistory</code> to 1 which will ask the firmware of your counting board <br />
to report the average difference measurements before they are compared to a threshold. <br />
The ArduCounter module will count how often each value is reported and you can then query these analog level counts with <code>get levels</code>. <br />
After a few measuremets the result of <code>get levels</code> might look like this:<br />
<br />
<pre><br />
observed levels from analog input:<br />
94: 21<br />
95: 79<br />
96: 6<br />
97: 2<br />
98: 3<br />
99: 2<br />
100: 2<br />
101: 1<br />
102: 3<br />
105: 2<br />
106: 1<br />
108: 2<br />
109: 1<br />
110: 1<br />
112: 1<br />
113: 3<br />
115: 4<br />
116: 9<br />
117: 14<br />
118: 71<br />
119: 103<br />
120: 118<br />
121: 155<br />
122: 159<br />
123: 143<br />
124: 147<br />
125: 158<br />
126: 198<br />
127: 249<br />
128: 220<br />
129: 230<br />
130: 201<br />
131: 140<br />
132: 147<br />
133: 153<br />
134: 141<br />
135: 119<br />
136: 105<br />
137: 109<br />
138: 114<br />
139: 83<br />
140: 33<br />
141: 14<br />
142: 1 <br />
</pre><br />
<br />
This shows the measured values together with the frequency how often the individual value has been measured. <br />
It is obvious that most measurements result in values between 120 and 135, very few values are betweem 96 and 115 <br />
and another peak is around the value 95. <br />
It means that in the example of a ferraris energy counter, when the red mark of the ferraris disc is under the sensor, <br />
the value is around 95 and while when the blank disc is under the sensor, the value is typically between 120 and 135. <br />
So a good upper threshold would be 120 and a good lower threshold would be for example 96.<br />
<br />
== Set-Commands ==<br />
<br />
;raw<br />
:send the value to the board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash [<file>]<br />
:flashes the ArduCounter firmware from the subdirectory FHEM/firmware onto the device.<br />
:Normally you can just specify <code>set myDevice flash</code>. The parameter <file> is optional and allows specifying an alternative firmware file.<br />
:The attribute flashCommand can be used to override which command is executed. <br />
:If the attribute flashCommand is not specified then the module selects an appropriate command depending on the board type <br />
:(set with the attribute <code>board</code>) and depending on the connection (serial or Wifi).<br />
:For an arduino NANO for example the module would execute avrdude (which has to be installed of course) <br />
:and flash the connected arduino with the updated hex file <br />
:(by default it looks for ArduCounter.hex in the FHEM/firmware subdirectory).<br />
:For an Arduino UNO for example the default is <code>avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]</code><br />
:For an Arduino Nano based counter <code>-b 57600</code> is added.<br />
:For an ESP32 connected via Wifi, the module would call espota.py which will upload the firmware over the air.<br />
:If the attribute flashCommand is not specified for an ESP32 based board connected via serial line, then the module uses the command <br />
<br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
esptool.py --chip esp32 --port [PORT] --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect<br />
0x1000 FHEM/firmware/ArduCounter_ESP32_bootloader_dio_40m.bin 0x8000 FHEM/firmware/ArduCounter_ESP32_partitions.bin 0xe000 FHEM/firmware/ArduCounter_ESP32_boot_app0.bin<br />
0x10000 FHEM/firmware/ArduCounter_ESP32_firmware.bin >[LOGFILE] 2>&1<br />
</syntaxhighlight> <br />
<br />
:for example which flashes the whole ESP32 with all the partitions. For over the air flashing it would use <br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
espota.py -i[IP] -p [NETPORT] -f [BINFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:Of course esptool.py or espota.py as well as python would need to be installed on the system.<br />
<br />
;resetWifi<br />
:reset Wifi settings of the counting device so the Wifi Manager will come up after the next reset to select a wireless network and enter the Wifi passphrase.<br />
<br />
;reset <br />
:sends a command to the device which causes a hardware reset or reinitialize and reset of the internal counters of the board. <br />
:The module then reopens the counting device and resends the attribute configuration / definition of the pins.<br />
<br />
;saveConfig <br />
:stores the current interval, analog threshold and pin configuration in the EEPROM of the counter device so it will automatically be retrieved after a reset.<br />
<br />
;enable <br />
:sets the attribute disable to 0<br />
<br />
;disable <br />
sets the attribute disable to 1<br />
<br />
;reconnect <br />
:closes the tcp connection to an ESP based counter board that is conected via TCP/IP and reopen the connection<br />
<br />
;clearLevels <br />
:clears the statistics for analog levels. This is only relevant if you use the board to read via a reflective light barrier <br />
:and you want to set the thresholds according to the statistics.<br />
<br />
;clearCounters <pin> <br />
:resets all the counter readings for the specified pin to 0<br />
<br />
;counter <pin>, <value> <br />
:set the calcCounter reading for the specified pin to the given value<br />
<br />
;clearHistory <br />
:deletes all the cached pin history entries<br />
<br />
== Get-Commands ==<br />
<br />
;info <br />
:send a command to the Arduino board to get current counts.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;levels <br />
:show the count for the measured levels if an analog pin is used to measure e.g. the red mark of a ferraris counter disc. This is useful for setting the thresholds for analog :measurements. <br />
<br />
;history <pin> <br />
:shows details regarding all the level changes that the counter device (Arduino or ESP) has detected and how they were used (counted or rejected)<br />
:If get history is issued with a pin name (e.g. get history D5) then only the history entries concerning D5 will be shown.<br />
:This information is sent from the device to Fhem if the attribute <code>enableHistory</code> is set to 1.<br />
:The maximum number of lines that the Arducounter module stores in a ring buffer is defined by the attribute maxHist and defaults to 1000.<br />
<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;pin[AD]?[0-9]+<rising|falling> [<pullup>] [min] <min length> [[analog] out <out pin> [threshold] <min, max>] <br />
:Define a GPIO pin of the Arduino or ESP board as input. This attribute expects for digital inputs either <br />
:<code>rising</code> or <code>falling</code>, followed by an optional <code>pullup</code> and the optional keyword <code>min</code> <br />
:and an optional number as minimal length of pulses and gaps between pulses.<br />
:The counter device will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds.<br />
:The minimal length specified here is the minimal duration of a pulse and a pause before a pulse. If one is too small, <br />
:the pulse is not counted but added to a separate reject counter.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinD4 falling pullup 25<br />
</syntaxhighlight> <br />
:For analog inputs with connected reflective light barries, you have to add <code>analog out</code> <br />
:and the GPIO pin number of the pin where the light source (LED or laser) is connected, the keyword <code>threshold</code> <br />
:followed by the lower and upper threshold separated by a komma.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinA0 rising pullup min 3 analog out 27 threshold 120,220<br />
</syntaxhighlight> <br />
<br />
;interval <normal> <max> [<min> <min count> [<analog interval> <analog samples>]] <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of six numbers as value. <br />
:The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count. <br />
:The last two numbers are only needed for counting with reflective light barriers. They specify the delay between the measurements <br />
:and the number of samples for each measurement.<br />
:<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval),<br />
:the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be <br />
:something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is then calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period <br />
:and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
:<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case <br />
:when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, <br />
:the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) <br />
:or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed <br />
:or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter interval 60 600 5 2<br />
</syntaxhighlight> <br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. <br />
:This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. <br />
:In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
:<br />
:For analog sampling the last two numbers define the delay in milliseconds between analog measurements and the number of samples that will be taken as one mesurement.<br />
<br />
;pulsesPerUnit <number> <br />
:specify the number of pulses that the meter is giving out per unit that sould be displayed (e.g. per kWh energy consumed). <br />
:For many S0 counters this is 1000, for old ferraris counters this is 75 (rounds per kWh).<br />
:This attribute used to be called pulsesPerKWh and this name still works but the new name should be used preferably since the old one could be removed in future versions.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter pulsesPerUnit 75<br />
</syntaxhighlight> <br />
<br />
;readingPulsesPerUnit[AD]?[0-9]+ <number> <br />
:is the same as pulsesPerUnit but specified per GPIO pin individually in case you have multiple counters with different settings at the same time<br />
:This attribute used to be called readingPulsesPerKWh[AD]?[0-9]+ and this name still works but the new name should be used preferably <br />
:since the old one could be removed in future versions.<br />
:<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingPulsesPerUnitA7 75<br />
attr myCounter readingPulsesPerUnitD4 1000<br />
</syntaxhighlight> <br />
<br />
;readingFlowUnitTime[AD]?[0-9]+ <time> <br />
:specified the time period in seconds which is used as the basis for calculating the current flow or power for the given pin.<br />
:If the counter e.g. counts liters and you want to see the flow in liters per minute, then you have to set this attribute to 60.<br />
:If you count kWh and you want to see the current power in kW, then specify 3600 (one hour).<br />
:Since this attribute is just used for multiplying the consumption per second, you can also use it to get watts <br />
:instead of kW by using 3600000 instead of 3600.<br />
<br />
;flowUnitTime <time> <br />
:like readingFlowUnitTimeXX but applies to all pins that have no explicit readingFlowUnitTimeXX attribute.<br />
<br />
;readingNameCount[AD]?[0-9]+ <new name> <br />
:Change the name of the counter reading pinX to something more meaningful. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCountD4 CounterHaus_internal<br />
</syntaxhighlight> <br />
<br />
;readingNameLongCount[AD]?[0-9]+ <new name> <br />
:Change the name of the long counter reading longX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameLongCountD4 CounterHaus_long<br />
</syntaxhighlight> <br />
<br />
;readingNameInterpolatedCount[AD]?[0-9]+ <new name> <br />
:Change the name of the interpolated long counter reading InterpolatedlongX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameInterpolatedCountD4 CounterHaus_interpolated<br />
</syntaxhighlight> <br />
<br />
;readingNameCalcCount[AD]?[0-9]+ <new name> <br />
:Change the name of the real unit counter reading CalcCounterX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCalcCountD4 CounterHaus_kWh<br />
</syntaxhighlight> <br />
<br />
;readingNamePower[AD]?[0-9]+ <new name> <br />
:Change the name of the power reading powerX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNamePowerD4 PowerHaus_kW<br />
</syntaxhighlight> <br />
<br />
;readingStartTime[AD]?[0-9]+ [0|1] <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals. <br />
:This is a hack where the timestamp of readings is artificially set to a past time and may have side effects <br />
:so avoid it unless you fully understand how Fhem works with readings and their time.<br />
<br />
;verboseReadings[AD]?[0-9]+ [0|1] <br />
:create the additional readings lastMsg and pinHistory for each pin<br />
:if verboseReafings is set to 1 for the specified pin.<br />
:If set to -1 then the internal counter, the long counter and interpolated long counter readings will be hidden.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter verboseReadingsD4 1<br />
</syntaxhighlight> <br />
<br />
;enableHistory [0|1]<br />
:tells the counting device to record the individual time of each change at each GPIO pin and send it to Fhem. <br />
:This information is cached on the Fhem side and can be viewed with the command <code>get history</code><br />
:The optput of <code>get history</code> will look like this:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
seq 12627 2020-03-22 20:39:54 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12628 2020-03-22 20:39:55 Pin D5 1.697 seconds at 1 -> gap<br />
seq 12629 2020-03-22 20:39:56 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12630 2020-03-22 20:39:56 Pin D5 1.694 seconds at 1 -> gap<br />
seq 12631 2020-03-22 20:39:58 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12632 2020-03-22 20:39:58 Pin D5 1.693 seconds at 1 -> gap<br />
seq 12633 2020-03-22 20:40:00 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12634 2020-03-22 20:40:00 Pin D5 1.696 seconds at 1 -> gap<br />
seq 12635 2020-03-22 20:40:02 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12636 2020-03-22 20:40:02 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12637 2020-03-22 20:40:03 Pin D5 0.079 seconds at 0 -> pulse counted<br />
seq 12638 2020-03-22 20:40:03 Pin D5 1.700 seconds at 1 -> gap<br />
seq 12639 2020-03-22 20:40:05 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12642 2020-03-22 20:40:05 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12643 2020-03-22 20:40:07 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12644 2020-03-22 20:40:07 Pin D5 1.698 seconds at 1 -> gap<br />
</syntaxhighlight> <br />
<br />
;enableSerialEcho [0|1] <br />
:tells the counting device to show diagnostic data over the serial line when connected via TCP<br />
<br />
;enablePinDebug [0|1] <br />
:tells the counting device to show every level change of the defined input pins over the serial line or via TCP<br />
<br />
;enableAnalogDebug [0|1] <br />
:tells the counting device to show every analog measurement of the defined analog input pins over the serial line or via TCP<br />
<br />
;enableDevTime [0|1] <br />
:tells the counting device to show its internal millis timer so a drift between the devices time and fhem time can be calculated and logged<br />
<br />
;maxHist <max entries> <br />
:specifies how many pin history lines hould be buffered for "get history".<br />
:This attribute defaults to 1000.<br />
<br />
;analogThresholds <br />
:this Attribute is outdated. Please specify the analog thresholds for reflective light barrier input with the attribute "pin..."<br />
<br />
;flashCommand <new shell command> <br />
:overrides the default command to flash the firmware via Wifi (OTA) or serial line. It is recommended to not define this attribute. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter flashCommand avrdude -p atmega328P -c arduino -b 57600 -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:<code> [PORT]</code> is automatically replaced with the serial port for this device as it is specified in the <code>define</code> command.<br />
:<code> [HEXFILE]</code> or <code>[BINFILE]</code> are synonyms and are both automatically replaced with the firmware file appropriate for the device. <br />
:For ESP32 boards <code>[HEXFILE]</code> would be replaced by ArduCounter-8266.bin for example.<br />
:<code> [LOGFILE]</code> is automatically replaced ArduCounterFlash.log in the fhem log subdirectory.<br />
:<code> [NETPORT]</code> is automatically replaced by the tcp port number used for OTA flashing. <br />
:For ESP32 this usually is 3232 and for 8266 Bords it is 8266.<br />
<br />
;keepAliveDelay <delay> <br />
:defines an interval in which the module sends keepalive messages to a counter device that is conected via tcp.<br />
:This attribute is ignored if the device is connected via serial port.<br />
:If the device doesn't reply within a defined timeout then the module closes and tries to reopen the connection.<br />
:The module tells the device when to expect the next keepalive message and the device will also close the tcp connection if it doesn't see a keepalive message within the delay :multiplied by 3<br />
:The delay defaults to 10 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveDelay 30<br />
</syntaxhighlight> <br />
<br />
;keepAliveTimeout <seconds> <br />
:defines the timeout when wainting for a keealive reply (see keepAliveDelay)<br />
:The timeout defaults to 2 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveTimeout 3<br />
</syntaxhighlight> <br />
<br />
;keepAliveRetries <max number of retries> <br />
:defines how often sending a keepalive is retried before the connection is closed and reopened.<br />
:It defaults to 2.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveRetries 3<br />
</syntaxhighlight> <br />
<br />
;nextOpenDelay <delay> <br />
:defines the time in seconds that the module waits before retrying to open a disconnected tcp connection. <br />
:This defaults to 60 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter nextOpenDelay 20<br />
</syntaxhighlight> <br />
<br />
;openTimeout <timeout> <br />
:defines the timeout in seconds after which tcp open gives up trying to establish a connection to the counter device.<br />
:This timeout defaults to 3 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter openTimeout 5<br />
</syntaxhighlight> <br />
<br />
;silentReconnect [0|1] <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter silentReconnect 1<br />
</syntaxhighlight> <br />
<br />
;deviceDisplay <pin> <unit> <flowUnit> <br />
:controls the unit strings that a local display on the counting device will show. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter deviceDisplay 36,l,l/m<br />
attr myCounter deviceDisplay 36,kWh,kW<br />
</syntaxhighlight> <br />
<br />
;disable [0|1] <br />
:if set to 1 then the module is disabled and closes the connection to a counter device.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse. <br />
:This attribute is outdated and unintuitive so you should avoid it. <br />
:Instead you should specify the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;readingFactor[AD]?[0-9]+ <br />
:Override the factor attribute for this individual pin. <br />
:Just like the attribute factor, this is a rather cumbersome way to specify the pulses per kWh. <br />
:Instead it is advised to use the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;devVerbose <br />
:this attribute is outdated and has been replaced with the attributes <br />
:<syntaxhighlight lang="perl" xstyle="width:80%;">enableHistory, enableSerialEcho, enablePinDebug, enableAnalogDebug, enableDevTime<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
<br />
<br />
== Readings / Events ==<br />
<br />
The module creates at least the following readings and events for each defined pin:<br />
<br />
;calcCounter.* <br />
:This is recommended reading for counting units based on the pulses and the attribute pulsesPerUnit. It is similar to interpolated long count <br />
:which keeps on counting up after fhem restarts but this counter will take the pulses per Unit attribute into the calculation und thus does not <br />
:count pulses but real Units (kWh, liters or some other unit that is applicable)<br />
:The name of this reading can be changed with the attribute readingNameCalcCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
: Another reading with the same name but ending in _i (e.g. calcCounterD4_i) will show how many kWh (or other units) of the above value is interpolated.<br />
<br />
;pin.* e.g. pinD4 <br />
:the current internal count at this pin (internal to the Arduino / ESP device, starts at 0 when the device restarts). <br />
:The name of this reading can be changed with the attribute readingNameCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;long.* e.g. longD5 <br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
:The name of this reading can be changed with the attribute readingNameLongCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;interpolatedLong.* <br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
:The name of this reading can be changed with the attribute readingNameInterpolatedCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;reject.*<br />
:counts rejected pulses that are shorter than the specified minimal pulse length. <br />
<br />
;power.* <br />
:the current calculated power / flow at this pin.<br />
:The name of this reading can be changed with the attribute readingNamePower[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
:This reading depends on the attributes pulsesPerUnit as well as readingFlowUnitTime or flowUnitTime for calculation <br />
<br />
;pinHistory.* <br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. Also the total number of impulses recorded here is :limited to 20 for all pins together. The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action.<br />
:-36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
;countDiff.* <br />
:delta of the current count to the last reported one. This is used together with timeDiff.* to calculate the power consumption.<br />
<br />
;timeDiff.* <br />
:time difference between the first pulse in the current observation interval and the last one. Used togehter with countDiff to calculate the power consumption.<br />
<br />
;seq.* <br />
:internal sequence number of the last report from the board to Fhem.<br />
<br />
;runTime.* <br />
:this reading will only be created when the attribute runTime[AD]?[0-9]+ is set for a given pin.<br />
:It contains the time in seconds that the consumption / flow observed at the specified pin has not ben zero.<br />
:If a water meter which outputs 10 impulses per liter on its digital output is for example connected to GPIO pin D6, <br />
:then if the attribute runTimeD6 is set to 1, the reading runTimeD6 will show for how many seconds the water has been flowing without a stop longer than the <br />
:observation interval specifie in the interval-attribute. This is helpful when you want to create alerts in case someone forgot to close a water tap.<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:ArduCounter_Wifi.PNG&diff=33352Datei:ArduCounter Wifi.PNG2020-06-06T11:28:23Z<p>StefanStrobel: WifiManager beim ArduCounter</p>
<hr />
<div>== Beschreibung ==<br />
WifiManager beim ArduCounter</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=33292ArduCounter2020-05-26T19:31:59Z<p>StefanStrobel: Bilder ergänzt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, measure time between pulses and convert this to readings for e.g. power consumption of Energy meters or water meters<br />
|ModType=d<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
This module implements an Interface to an Arduino, ESP8266 or ESP32 based counter for pulses on any input pin of an Arduino Uno, Nano, Jeenode, <br />
NodeMCU, Wemos D1, TTGO T-Display or similar device. <br />
[[Datei:Wasserzaehler.jpg|mini|Wasserzähler mit Laserabtastung auf Basis ArduCounter mit ESP32]]<br />
[[Datei:TTGO.png|mini|ArduCounter auf einem ESP32]]<br />
[[Datei:ArduCounter_ESP_i.jpg|mini|ArduCounter mit Reflexlichtschranke auf einem ESP8266]]<br />
[[Datei:ArduCounter_ESP2.jpg|mini|ArduCounter mit Reflexlichtschranke auf einem ESP8266]]<br />
<br />
The device connects to Fhem either through USB / serial or via Wifi / TCP if an ESP board is used.<br />
ArduCounter does not only count pulses but also measure pulse lenghts and the time between pulses so it can filter noise / bounces<br />
and gives better power/flow (Watts or liters/second) readings than systems that just count in fixed time intervals.<br />
The number of pulses per kWh or liter is defineable and counters continue even when Fhem or the device restarts<br />
so you don't need additional user readings to make such calculations<br />
The typical use case is an S0-Interface on an energy meter or water meter, but also reflection light barriers <br />
to monitor old ferraris counters or analog water meters are supported<br />
Counters are configured with attributes that define which GPIO pins should count pulses and in which intervals the board should report the current counts.<br />
The sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
The module creates readings for pulse counts, consumption and optionally also a pin history with pulse lengths and gaps of the last pulses.<br />
If an ESP8266 or ESP32 is used, the device can be flashed and configured over Wifi (it opens its own temporary Hotspot / SSID for configuration <br />
so you can set which existing SSID to connect to and which password to use). For TTGO T-Display boards (ESP32 with TFT display) <br />
the local display on the device itself can also display Wifi status and current consumption.<br />
<br />
== Prerequisites ==<br />
<br />
This module requires an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1, TTGO T-Display or similar device based on an Atmel 328p, ESP8266 or ESP32 <br />
running the ArduCounter sketch provided with this module<br />
In order to flash an arduino board with the corresponding ArduCounter firmware from within Fhem, avrdude needs to be installed.<br />
To flash ESP32 or ESP8266 boards form within Fhem, Python and the scripts esptool.py / espota.py need to be installed.<br />
For old ferraris counters a reflection light barrier which in the simpest case can consist of a photo transistor (connected to an anlalog input of the Arduino / ESP) <br />
and an led or a laser module (connected to a digital output), both with a resistor in line are needed. <br />
To drive a laser module with 5V, another transistor is typically needed to switch 5V from a 3.3V GPIO output.<br />
<br />
== Define ==<br />
<br />
<code>define <name> ArduCounter <device></code><br />
or<br />
<code>define <name> ArduCounter <ip:port></code><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<ip:port> specifies the ip address and tcp port to communicate with an esp8266 / ESP32 where port is typically 80.<br />
<br />
The name of the serial device depends on your distribution and serial adapter.<br />
You can also specify a baudrate for serial connections if the device name contains the @ character, e.g.: /dev/ttyUSB0@115200<br />
The default baudrate of the ArduCounter firmware is 115200 since sketch version 4 and used to be 38400 since sketch version 1.4<br />
The latest version of this module will however try different baudrates automatically if communication with the counting device seems not possible.<br />
<br />
Example:<br />
<br />
<code>define AC ArduCounter /dev/ttyUSB2@115200</code><br />
<code>define AC ArduCounter 192.168.1.134:80</code><br />
<br />
== Configuration of ArduCounter digital counters ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <code>attr AC pinX falling pullup min 25</code> <br />
The X in pinX can be an Arduino / ESP GPIO pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
After the pin you can use the keywords falling or rising to define if a logical one / 5V (rising) or a logical zero / 0V (falling) should be treated as pulse.<br />
The optional keyword pullup activates the pullup resistor for the given Pin. <br />
The last argument is also optional but recommended and specifies a minimal pulse length in milliseconds.<br />
An energy meter with S0 interface is typically connected to GND and an input pin like D4. <br />
The S0 pulse then pulls the input down to 0V.<br />
Since the minimal pulse lenght of an S0 interface is specified to be 30ms, the typical configuration for an s0 interface is <br />
<code>attr AC pinX falling pullup min 25</code><br />
Specifying a minimal pulse length is recommended since it filters bouncing of reed contacts or other noise. <br />
The keyword <code>min</code> before <code>25</code> is optional.<br />
<br />
Example:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC pulsesPerUnit 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup min 5<br />
attr AC pinD5 falling pullup min 25<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines a counter that is connected to Fhem via serial line ttyUSB2 with three counters connected to the GPIO pins D4, D5 and D5. <br />
D4 and D5 have their pullup resistors activated and the impulses draw the pins to zero.<br />
For D4 and D5 the board measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 5 (or 25 for pin D5) milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
For pin D6 the board uses a default minimal length of 2ms and counts every time when the signal changes from 1 (rising pulse) back to 0.<br />
<br />
== Configuration of ArduCounter analog counters ==<br />
<br />
This module and the corresponding ArduCounter sketch can be used to read water meters or old analog ferraris energy counters. <br />
Therefore a reflection light barrier needs to be connected to the board. This might simply consist of an infra red photo transistor <br />
(connected to an analog input) and an infra red led (connected to a digital output), both with a resistor in line. <br />
The idea comes from Martin Kompf (https://www.kompf.de/tech/emeir.html) and has been adopted for ArduCounter to support <br />
old ferraris energy counters or water meters.<br />
The configuration is then similar to the one for digital counters:<br />
<pre><br />
define WaterMeter ArduCounter 192.168.1.110:80<br />
attr ACF pinA0 rising pullup min 4 analog out 27 threshold 120,220<br />
attr ACF interval 5,60,2,15,10,3<br />
attr ACF pulsesPerUnit 35<br />
attr ACF stateFormat {sprintf("%.3f l/min", ReadingsVal($name,"powerA0",0))}<br />
</pre><br />
In this case an analog GPIO pin is used as input and the normal configuration parameters are followed by the keyword <br />
<code>analog out</code> or simply <code>out</code>, the gpio number of a GPIO output that connects a light source and the thresholds <br />
that decide when an analog input value is regarded as "low" or "high".<br />
<br />
In the example an ESP32 is used via Wifi connection. GPIO pin A0 is used as analog input and is connected to a photo transistor that senses the intensity of light.<br />
GPIO 27 is used as LED output and switched on/off in a high frequency. On GPIO A0 the reflected light is measured <br />
and the difference in a measurement between when the LED is off and when the LED is on is compared to the thresholds defined in the pinA0-attribute. <br />
When the measured light difference is above <code>220</code>, then a pulse starts (since <code>rising</code> is specified). <br />
When the measured difference is below <code>120</code> then the pulse ends.<br />
<br />
The attribute <code>interval</code> has the following meaning in the above example: <br />
The device reports the current counts and the time difference beween the first and the last pulse if at least 2 pulses have been counted <br />
and if they are more than 15 milliseconds apart form each other. If not, then the device continues counting. <br />
If after 60 seconds these conditions are stil not met, then the device will report the current count anyways and use the current time as the end of the interval.<br />
The last two numbers of the <code>interval</code> attribute define that the device will read the analog input 3 times and then work with the average. <br />
Between each analog measurement series there will be a delay of 10 milliseconds.<br />
<br />
The attribute <code>pulsesPerUnit 35</code> defines that 35 pulses correspond to one unit (e.g. liter) and the reading <code>calcCounterA0</code> <br />
is increased by the reported raw counts divided by 35.<br />
To find out the right analog thresholds you can set the attribute <code>enableHistory</code> to 1 which will ask the firmware of your counting board <br />
to report the average difference measurements before they are compared to a threshold. <br />
The ArduCounter module will count how often each value is reported and you can then query these analog level counts with <code>get levels</code>. <br />
After a few measuremets the result of <code>get levels</code> might look like this:<br />
<br />
<pre><br />
observed levels from analog input:<br />
94: 21<br />
95: 79<br />
96: 6<br />
97: 2<br />
98: 3<br />
99: 2<br />
100: 2<br />
101: 1<br />
102: 3<br />
105: 2<br />
106: 1<br />
108: 2<br />
109: 1<br />
110: 1<br />
112: 1<br />
113: 3<br />
115: 4<br />
116: 9<br />
117: 14<br />
118: 71<br />
119: 103<br />
120: 118<br />
121: 155<br />
122: 159<br />
123: 143<br />
124: 147<br />
125: 158<br />
126: 198<br />
127: 249<br />
128: 220<br />
129: 230<br />
130: 201<br />
131: 140<br />
132: 147<br />
133: 153<br />
134: 141<br />
135: 119<br />
136: 105<br />
137: 109<br />
138: 114<br />
139: 83<br />
140: 33<br />
141: 14<br />
142: 1 <br />
</pre><br />
<br />
This shows the measured values together with the frequency how often the individual value has been measured. <br />
It is obvious that most measurements result in values between 120 and 135, very few values are betweem 96 and 115 <br />
and another peak is around the value 95. <br />
It means that in the example of a ferraris energy counter, when the red mark of the ferraris disc is under the sensor, <br />
the value is around 95 and while when the blank disc is under the sensor, the value is typically between 120 and 135. <br />
So a good upper threshold would be 120 and a good lower threshold would be for example 96.<br />
<br />
== Set-Commands ==<br />
<br />
;raw<br />
:send the value to the board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash [<file>]<br />
:flashes the ArduCounter firmware from the subdirectory FHEM/firmware onto the device.<br />
:Normally you can just specify <code>set myDevice flash</code>. The parameter <file> is optional and allows specifying an alternative firmware file.<br />
:The attribute flashCommand can be used to override which command is executed. <br />
:If the attribute flashCommand is not specified then the module selects an appropriate command depending on the board type <br />
:(set with the attribute <code>board</code>) and depending on the connection (serial or Wifi).<br />
:For an arduino NANO for example the module would execute avrdude (which has to be installed of course) <br />
:and flash the connected arduino with the updated hex file <br />
:(by default it looks for ArduCounter.hex in the FHEM/firmware subdirectory).<br />
:For an Arduino UNO for example the default is <code>avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]</code><br />
:For an Arduino Nano based counter <code>-b 57600</code> is added.<br />
:For an ESP32 connected via Wifi, the module would call espota.py which will upload the firmware over the air.<br />
:If the attribute flashCommand is not specified for an ESP32 based board connected via serial line, then the module uses the command <br />
<br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
esptool.py --chip esp32 --port [PORT] --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect<br />
0x1000 FHEM/firmware/ArduCounter_ESP32_bootloader_dio_40m.bin 0x8000 FHEM/firmware/ArduCounter_ESP32_partitions.bin 0xe000 FHEM/firmware/ArduCounter_ESP32_boot_app0.bin<br />
0x10000 FHEM/firmware/ArduCounter_ESP32_firmware.bin >[LOGFILE] 2>&1<br />
</syntaxhighlight> <br />
<br />
:for example which flashes the whole ESP32 with all the partitions. For over the air flashing it would use <br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
espota.py -i[IP] -p [NETPORT] -f [BINFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:Of course esptool.py or espota.py as well as python would need to be installed on the system.<br />
<br />
;resetWifi<br />
:reset Wifi settings of the counting device so the Wifi Manager will come up after the next reset to select a wireless network and enter the Wifi passphrase.<br />
<br />
;reset <br />
:sends a command to the device which causes a hardware reset or reinitialize and reset of the internal counters of the board. <br />
:The module then reopens the counting device and resends the attribute configuration / definition of the pins.<br />
<br />
;saveConfig <br />
:stores the current interval, analog threshold and pin configuration in the EEPROM of the counter device so it will automatically be retrieved after a reset.<br />
<br />
;enable <br />
:sets the attribute disable to 0<br />
<br />
;disable <br />
sets the attribute disable to 1<br />
<br />
;reconnect <br />
:closes the tcp connection to an ESP based counter board that is conected via TCP/IP and reopen the connection<br />
<br />
;clearLevels <br />
:clears the statistics for analog levels. This is only relevant if you use the board to read via a reflective light barrier <br />
:and you want to set the thresholds according to the statistics.<br />
<br />
;clearCounters <pin> <br />
:resets all the counter readings for the specified pin to 0<br />
<br />
;counter <pin>, <value> <br />
:set the calcCounter reading for the specified pin to the given value<br />
<br />
;clearHistory <br />
:deletes all the cached pin history entries<br />
<br />
== Get-Commands ==<br />
<br />
;info <br />
:send a command to the Arduino board to get current counts.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;levels <br />
:show the count for the measured levels if an analog pin is used to measure e.g. the red mark of a ferraris counter disc. This is useful for setting the thresholds for analog :measurements. <br />
<br />
;history <pin> <br />
:shows details regarding all the level changes that the counter device (Arduino or ESP) has detected and how they were used (counted or rejected)<br />
:If get history is issued with a pin name (e.g. get history D5) then only the history entries concerning D5 will be shown.<br />
:This information is sent from the device to Fhem if the attribute <code>enableHistory</code> is set to 1.<br />
:The maximum number of lines that the Arducounter module stores in a ring buffer is defined by the attribute maxHist and defaults to 1000.<br />
<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;pin[AD]?[0-9]+<rising|falling> [<pullup>] [min] <min length> [[analog] out <out pin> [threshold] <min, max>] <br />
:Define a GPIO pin of the Arduino or ESP board as input. This attribute expects for digital inputs either <br />
:<code>rising</code> or <code>falling</code>, followed by an optional <code>pullup</code> and the optional keyword <code>min</code> <br />
:and an optional number as minimal length of pulses and gaps between pulses.<br />
:The counter device will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds.<br />
:The minimal length specified here is the minimal duration of a pulse and a pause before a pulse. If one is too small, <br />
:the pulse is not counted but added to a separate reject counter.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinD4 falling pullup 25<br />
</syntaxhighlight> <br />
:For analog inputs with connected reflective light barries, you have to add <code>analog out</code> <br />
:and the GPIO pin number of the pin where the light source (LED or laser) is connected, the keyword <code>threshold</code> <br />
:followed by the lower and upper threshold separated by a komma.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinA0 rising pullup min 3 analog out 27 threshold 120,220<br />
</syntaxhighlight> <br />
<br />
;interval <normal> <max> [<min> <min count> [<analog interval> <analog samples>]] <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of six numbers as value. <br />
:The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count. <br />
:The last two numbers are only needed for counting with reflective light barriers. They specify the delay between the measurements <br />
:and the number of samples for each measurement.<br />
:<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval),<br />
:the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be <br />
:something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is then calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period <br />
:and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
:<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case <br />
:when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, <br />
:the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) <br />
:or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed <br />
:or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter interval 60 600 5 2<br />
</syntaxhighlight> <br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. <br />
:This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. <br />
:In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
:<br />
:For analog sampling the last two numbers define the delay in milliseconds between analog measurements and the number of samples that will be taken as one mesurement.<br />
<br />
;pulsesPerUnit <number> <br />
:specify the number of pulses that the meter is giving out per unit that sould be displayed (e.g. per kWh energy consumed). <br />
:For many S0 counters this is 1000, for old ferraris counters this is 75 (rounds per kWh).<br />
:This attribute used to be called pulsesPerKWh and this name still works but the new name should be used preferably since the old one could be removed in future versions.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter pulsesPerUnit 75<br />
</syntaxhighlight> <br />
<br />
;readingPulsesPerUnit[AD]?[0-9]+ <number> <br />
:is the same as pulsesPerUnit but specified per GPIO pin individually in case you have multiple counters with different settings at the same time<br />
:This attribute used to be called readingPulsesPerKWh[AD]?[0-9]+ and this name still works but the new name should be used preferably <br />
:since the old one could be removed in future versions.<br />
:<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingPulsesPerUnitA7 75<br />
attr myCounter readingPulsesPerUnitD4 1000<br />
</syntaxhighlight> <br />
<br />
;readingFlowUnitTime[AD]?[0-9]+ <time> <br />
:specified the time period in seconds which is used as the basis for calculating the current flow or power for the given pin.<br />
:If the counter e.g. counts liters and you want to see the flow in liters per minute, then you have to set this attribute to 60.<br />
:If you count kWh and you want to see the current power in kW, then specify 3600 (one hour).<br />
:Since this attribute is just used for multiplying the consumption per second, you can also use it to get watts <br />
:instead of kW by using 3600000 instead of 3600.<br />
<br />
;flowUnitTime <time> <br />
:like readingFlowUnitTimeXX but applies to all pins that have no explicit readingFlowUnitTimeXX attribute.<br />
<br />
;readingNameCount[AD]?[0-9]+ <new name> <br />
:Change the name of the counter reading pinX to something more meaningful. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCountD4 CounterHaus_internal<br />
</syntaxhighlight> <br />
<br />
;readingNameLongCount[AD]?[0-9]+ <new name> <br />
:Change the name of the long counter reading longX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameLongCountD4 CounterHaus_long<br />
</syntaxhighlight> <br />
<br />
;readingNameInterpolatedCount[AD]?[0-9]+ <new name> <br />
:Change the name of the interpolated long counter reading InterpolatedlongX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameInterpolatedCountD4 CounterHaus_interpolated<br />
</syntaxhighlight> <br />
<br />
;readingNameCalcCount[AD]?[0-9]+ <new name> <br />
:Change the name of the real unit counter reading CalcCounterX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCalcCountD4 CounterHaus_kWh<br />
</syntaxhighlight> <br />
<br />
;readingNamePower[AD]?[0-9]+ <new name> <br />
:Change the name of the power reading powerX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNamePowerD4 PowerHaus_kW<br />
</syntaxhighlight> <br />
<br />
;readingStartTime[AD]?[0-9]+ [0|1] <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals. <br />
:This is a hack where the timestamp of readings is artificially set to a past time and may have side effects <br />
:so avoid it unless you fully understand how Fhem works with readings and their time.<br />
<br />
;verboseReadings[AD]?[0-9]+ [0|1] <br />
:create the additional readings lastMsg and pinHistory for each pin<br />
:if verboseReafings is set to 1 for the specified pin.<br />
:If set to -1 then the internal counter, the long counter and interpolated long counter readings will be hidden.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter verboseReadingsD4 1<br />
</syntaxhighlight> <br />
<br />
;enableHistory [0|1]<br />
:tells the counting device to record the individual time of each change at each GPIO pin and send it to Fhem. <br />
:This information is cached on the Fhem side and can be viewed with the command <code>get history</code><br />
:The optput of <code>get history</code> will look like this:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
seq 12627 2020-03-22 20:39:54 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12628 2020-03-22 20:39:55 Pin D5 1.697 seconds at 1 -> gap<br />
seq 12629 2020-03-22 20:39:56 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12630 2020-03-22 20:39:56 Pin D5 1.694 seconds at 1 -> gap<br />
seq 12631 2020-03-22 20:39:58 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12632 2020-03-22 20:39:58 Pin D5 1.693 seconds at 1 -> gap<br />
seq 12633 2020-03-22 20:40:00 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12634 2020-03-22 20:40:00 Pin D5 1.696 seconds at 1 -> gap<br />
seq 12635 2020-03-22 20:40:02 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12636 2020-03-22 20:40:02 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12637 2020-03-22 20:40:03 Pin D5 0.079 seconds at 0 -> pulse counted<br />
seq 12638 2020-03-22 20:40:03 Pin D5 1.700 seconds at 1 -> gap<br />
seq 12639 2020-03-22 20:40:05 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12642 2020-03-22 20:40:05 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12643 2020-03-22 20:40:07 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12644 2020-03-22 20:40:07 Pin D5 1.698 seconds at 1 -> gap<br />
</syntaxhighlight> <br />
<br />
;enableSerialEcho [0|1] <br />
:tells the counting device to show diagnostic data over the serial line when connected via TCP<br />
<br />
;enablePinDebug [0|1] <br />
:tells the counting device to show every level change of the defined input pins over the serial line or via TCP<br />
<br />
;enableAnalogDebug [0|1] <br />
:tells the counting device to show every analog measurement of the defined analog input pins over the serial line or via TCP<br />
<br />
;enableDevTime [0|1] <br />
:tells the counting device to show its internal millis timer so a drift between the devices time and fhem time can be calculated and logged<br />
<br />
;maxHist <max entries> <br />
:specifies how many pin history lines hould be buffered for "get history".<br />
:This attribute defaults to 1000.<br />
<br />
;analogThresholds <br />
:this Attribute is outdated. Please specify the analog thresholds for reflective light barrier input with the attribute "pin..."<br />
<br />
;flashCommand <new shell command> <br />
:overrides the default command to flash the firmware via Wifi (OTA) or serial line. It is recommended to not define this attribute. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter flashCommand avrdude -p atmega328P -c arduino -b 57600 -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:<code> [PORT]</code> is automatically replaced with the serial port for this device as it is specified in the <code>define</code> command.<br />
:<code> [HEXFILE]</code> or <code>[BINFILE]</code> are synonyms and are both automatically replaced with the firmware file appropriate for the device. <br />
:For ESP32 boards <code>[HEXFILE]</code> would be replaced by ArduCounter-8266.bin for example.<br />
:<code> [LOGFILE]</code> is automatically replaced ArduCounterFlash.log in the fhem log subdirectory.<br />
:<code> [NETPORT]</code> is automatically replaced by the tcp port number used for OTA flashing. <br />
:For ESP32 this usually is 3232 and for 8266 Bords it is 8266.<br />
<br />
;keepAliveDelay <delay> <br />
:defines an interval in which the module sends keepalive messages to a counter device that is conected via tcp.<br />
:This attribute is ignored if the device is connected via serial port.<br />
:If the device doesn't reply within a defined timeout then the module closes and tries to reopen the connection.<br />
:The module tells the device when to expect the next keepalive message and the device will also close the tcp connection if it doesn't see a keepalive message within the delay :multiplied by 3<br />
:The delay defaults to 10 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveDelay 30<br />
</syntaxhighlight> <br />
<br />
;keepAliveTimeout <seconds> <br />
:defines the timeout when wainting for a keealive reply (see keepAliveDelay)<br />
:The timeout defaults to 2 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveTimeout 3<br />
</syntaxhighlight> <br />
<br />
;keepAliveRetries <max number of retries> <br />
:defines how often sending a keepalive is retried before the connection is closed and reopened.<br />
:It defaults to 2.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveRetries 3<br />
</syntaxhighlight> <br />
<br />
;nextOpenDelay <delay> <br />
:defines the time in seconds that the module waits before retrying to open a disconnected tcp connection. <br />
:This defaults to 60 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter nextOpenDelay 20<br />
</syntaxhighlight> <br />
<br />
;openTimeout <timeout> <br />
:defines the timeout in seconds after which tcp open gives up trying to establish a connection to the counter device.<br />
:This timeout defaults to 3 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter openTimeout 5<br />
</syntaxhighlight> <br />
<br />
;silentReconnect [0|1] <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter silentReconnect 1<br />
</syntaxhighlight> <br />
<br />
;deviceDisplay <pin> <unit> <flowUnit> <br />
:controls the unit strings that a local display on the counting device will show. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter deviceDisplay 36,l,l/m<br />
attr myCounter deviceDisplay 36,kWh,kW<br />
</syntaxhighlight> <br />
<br />
;disable [0|1] <br />
:if set to 1 then the module is disabled and closes the connection to a counter device.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse. <br />
:This attribute is outdated and unintuitive so you should avoid it. <br />
:Instead you should specify the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;readingFactor[AD]?[0-9]+ <br />
:Override the factor attribute for this individual pin. <br />
:Just like the attribute factor, this is a rather cumbersome way to specify the pulses per kWh. <br />
:Instead it is advised to use the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;devVerbose <br />
:this attribute is outdated and has been replaced with the attributes <br />
:<syntaxhighlight lang="perl" xstyle="width:80%;">enableHistory, enableSerialEcho, enablePinDebug, enableAnalogDebug, enableDevTime<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
<br />
<br />
== Readings / Events ==<br />
<br />
The module creates at least the following readings and events for each defined pin:<br />
<br />
;calcCounter.* <br />
:This is recommended reading for counting units based on the pulses and the attribute pulsesPerUnit. It is similar to interpolated long count <br />
:which keeps on counting up after fhem restarts but this counter will take the pulses per Unit attribute into the calculation und thus does not <br />
:count pulses but real Units (kWh, liters or some other unit that is applicable)<br />
:The name of this reading can be changed with the attribute readingNameCalcCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
: Another reading with the same name but ending in _i (e.g. calcCounterD4_i) will show how many kWh (or other units) of the above value is interpolated.<br />
<br />
;pin.* e.g. pinD4 <br />
:the current internal count at this pin (internal to the Arduino / ESP device, starts at 0 when the device restarts). <br />
:The name of this reading can be changed with the attribute readingNameCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;long.* e.g. longD5 <br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
:The name of this reading can be changed with the attribute readingNameLongCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;interpolatedLong.* <br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
:The name of this reading can be changed with the attribute readingNameInterpolatedCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;reject.*<br />
:counts rejected pulses that are shorter than the specified minimal pulse length. <br />
<br />
;power.* <br />
:the current calculated power / flow at this pin.<br />
:The name of this reading can be changed with the attribute readingNamePower[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
:This reading depends on the attributes pulsesPerUnit as well as readingFlowUnitTime or flowUnitTime for calculation <br />
<br />
;pinHistory.* <br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. Also the total number of impulses recorded here is :limited to 20 for all pins together. The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action.<br />
:-36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
;countDiff.* <br />
:delta of the current count to the last reported one. This is used together with timeDiff.* to calculate the power consumption.<br />
<br />
;timeDiff.* <br />
:time difference between the first pulse in the current observation interval and the last one. Used togehter with countDiff to calculate the power consumption.<br />
<br />
;seq.* <br />
:internal sequence number of the last report from the board to Fhem.<br />
<br />
;runTime.* <br />
:this reading will only be created when the attribute runTime[AD]?[0-9]+ is set for a given pin.<br />
:It contains the time in seconds that the consumption / flow observed at the specified pin has not ben zero.<br />
:If a water meter which outputs 10 impulses per liter on its digital output is for example connected to GPIO pin D6, <br />
:then if the attribute runTimeD6 is set to 1, the reading runTimeD6 will show for how many seconds the water has been flowing without a stop longer than the <br />
:observation interval specifie in the interval-attribute. This is helpful when you want to create alerts in case someone forgot to close a water tap.<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:ArduCounter_ESP2.jpg&diff=33291Datei:ArduCounter ESP2.jpg2020-05-26T19:14:09Z<p>StefanStrobel: </p>
<hr />
<div></div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:ArduCounter_ESP_i.jpg&diff=33290Datei:ArduCounter ESP i.jpg2020-05-26T19:13:35Z<p>StefanStrobel: </p>
<hr />
<div></div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:Wasserzaehler.jpg&diff=33289Datei:Wasserzaehler.jpg2020-05-26T19:11:30Z<p>StefanStrobel: </p>
<hr />
<div></div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:TTGO.png&diff=33288Datei:TTGO.png2020-05-26T18:33:00Z<p>StefanStrobel: Arducounter auf einem TTGO T-Display board</p>
<hr />
<div>== Beschreibung ==<br />
Arducounter auf einem TTGO T-Display board</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ModbusAttr&diff=33265ModbusAttr2020-05-23T14:56:35Z<p>StefanStrobel: /* Define as Modbus relay */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=ModbusAttr<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=25315<br />
|ModTechName=98_ModbusAttr.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
[[ModbusAttr]] uses the low level Modbus module 98_[[Modbus]].pm to provide a generic Modbus module (as master, slave, relay or passive listener) <br />
that can be configured by attributes similar to the way HTTPMOD works for devices with a web interface.<br />
<br />
ModbusAttr can be used as a Modbus master that queries data from other devices over a serial RS232 / RS485 or TCP connection, <br />
it can be used as a Modbus slave that can make readings of Fhem devices available via Modbus to external Modbus masters,<br />
it can act as a Modbus relay that receives requests over one connection and forwards them over another connection (e.g. from Modbus TCP to serial Modbus RTU)<br />
or it can passively listen to other devices that communicate over a serial RS485 connection and extract readings from the objects it sees.<br />
The supported protocols are Modbus RTU, Modbus ASCII or Modbus TCP.<br />
<br />
There are several attributes that modify the way data objects are converted before they are stored in readings or sent to a device. Data can be modified by a perl expression defined in an atribute, formatted with a format string defined in another attribute or mapped to a table defined in an attribute.<br />
<br />
Readings can directly correspond to one data object or they can span several objects. A float value for example might be stored in two input or holding registers in the Modbus device. By specifying attributes that define the length of a reading in objects and by specifying the unpack code to get from a raw string to perl variables, all these cases can be described by attributes and no perl coding is necessary.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module uses the [[Modbus|Modbus base module 98_Modbus.pm]] which uses the DevIO module<br />
<br />
== Define as Modbus master == <br />
<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate,bits,parity,stop<br />
define <name> ModbusAttr <Id> <Interval> <RTU|ASCII><br />
</pre><br />
or<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|ASCII|TCP><br />
</pre><br />
<br />
In the first case the module connects to the external Modbus device with Modbus Id <Id> through a serial modbus device (RS232 or RS485).<br />
<br />
In the second case it connects directly through Modbus TCP or Modbus RTU or ASCII over TCP.<br />
<br />
If <Interval> is not 0 then the module actively requests data from that device every <Interval> seconds.<br />
<br />
The objects that the module should request and the readings it should create from these objects have to be defined with attributes (see below). <br />
These attributes will define a mapping from so called "coils", "digital inputs", "input registers" or "holding registers" of the external device to readings inside Fhem together with the data type and format of the values.<br />
Interval can be 0 in which case the Module only requests data when it is triggered with a Fhem get-Command.<br />
With this mode a Fhem installation can for example query sensor data from a heating system, energy meter or solar power installation if these systems offer a Modbus interface.<br />
<br />
Examples:<br />
<br />
<pre><br />
define ModbusLine Modbus /dev/ttyUSB1@9600<br />
define WP ModbusAttr 1 60<br />
</pre><br />
Define WP as a Modbus master that communicates through the Modbus serial interface device named ModbusLine. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define ModbusLine Modbus /dev/ttyUSB1@9600<br />
define WP ModbusAttr 20 0 ASCII<br />
</pre><br />
Define WP as a Modbus master that communicates through the Modbus serial interface device named ModbusLine with Modbus ASCII. <br />
Use Modbus Id 20 and don't query the device in a defined interval. Instead individual SET / GET options have to be used for communication.<br />
<br />
<pre><br />
define WP ModbusAttr 5 60 192.168.1.122:502 TCP<br />
</pre><br />
to talk Modbus TCP to a device with IP-Address 192.168.1.122 and the reserved port for Modbus TCP 502<br />
Note that for Modbus over a TCP connection you don't need a basic Modbus device for the interface like ModbusLine above. <br />
<br />
<pre><br />
define WP ModbusAttr 3 60 192.168.1.122:8000 RTU<br />
</pre><br />
to talk Modbus RTU over TCP and use the port number 8000<br />
<br />
== Define as Modbus slave == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> slave<br />
</pre><br />
or<br />
<pre><br />
define <name> ModbusAttr <Id> slave <Address:Port> <RTU|ASCII|TCP><br />
</pre><br />
<br />
The module waits for connections from other Modbus masters. It will respond to their requests if the requests contain the given Modbus <Id><br />
To provide data with Modbus to external Modbus masters a mapping needs to be defined using attributes. <br />
These attributes will define a mapping from Readings inside Fhem to so called "coils", "digital inputs", "input registers" or "holding registers" and their Modbus object address together with the data type and format of the values.<br />
With this mode a Fhem installation can for example supply data to a PLC that actively reads data from Fhem or writes data to Fhem readings.<br />
<br />
Examples:<br />
<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 1 slave<br />
</pre><br />
Define Data4PLC as a Modbus slave that communicates through the Modbus serial interface device named MRS485 to listen for Modbus requests with Id 1. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 20 slave ASCII<br />
</pre><br />
to listen for Modbus requests with Id 20 with Modbus ASCII. <br />
<br />
<pre><br />
define Data4PLC ModbusAttr 5 slave 192.168.1.2:502 TCP<br />
</pre><br />
to start listening to TCP port 502 on the local address 192.168.1.2. Modbus TCP will be used as protocol and Requests with Modbus Id 5 will be answered.<br />
<br />
Please be aware that opening a port number smaller than 1024 needs root permissions on Unix devices. So it is probably better to use a non standard port number above 1024 instead.<br />
<br />
<pre><br />
define Data4PLC ModbusAttr 3 slave 192.168.1.2:8000 RTU<br />
</pre><br />
to listen to the local port 8000 and talk Modbus RTU over TCP<br />
<br />
== Define as Modbus passive listener == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> passive <RTU|ASCII|TCP><br />
</pre><br />
<br />
The module listens on a serial (RS485) connection for modbus communication with the given Modbus <Id> and extracts readings. It does not send requests by itself but waits for another master to communicate with a slave. So only objects that the other master requests can be seen by Fhem in this configuration. <br />
The objects that the module recognizes and the readings that it should create from these objects have to be defined with attributes (see below) in the same way as for a Modbus master. <br />
These attributes will define a mapping from so called "coils", "digital inputs", "input registers" or "holding registers" of the external device to readings inside Fhem together with the data type and format of the values.<br />
With this mode a Fhem installation can for example Listen to the communication between an energy counter as slave and a solar control system as master if they use Modbus RTU over RS485. Since only one Master is allowed when using Modbus over serial lines, Fhem can not be master itself. As a passive listener it can however see when the master queries e.g. the current power consumption and then also see the reply from the energy meter and store the value in a Fhem reading.<br />
<br />
Examples:<br />
<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define WP ModbusAttr 1 passive<br />
</pre><br />
to passively listen for Modbus requests and replies with Id 1 over a serial interface managed by an already defined basic modbus device named MB-485. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define WP ModbusAttr 20 passive ASCII<br />
</pre><br />
to passivel listen for Modbus requests / replies with Id 20 and Modbus ASCII. <br />
<br />
== Define as Modbus relay == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> relay to <FhemMasterDevice><br />
</pre><br />
for a relay from a serial line to a defined master or<br />
<pre><br />
define <name> ModbusAttr <Id> relay <Address:Port> <RTU|ASCII|TCP> to <FhemMasterDevice><br />
</pre><br />
For a relay that listens to modbus requests from a network connection.<br />
<br />
The module waits for connections from other Modbus masters. It will forward requests if they match the given Modbus <Id> to an already defined Modbus Master device inside Fhem which will send them to its defined slave, take the reply and then pass it back to the original Master.<br />
With this mode a Fhem installation can for example be used in front of a device that only speaks Modbus RTU over RS485 to make it available via Modbus TCP over the local network. <br />
<br />
Examples:<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define Heating ModbusAttr 22 0<br />
define Relay ModbusAttr 33 relay 192.168.1.2:1502 TCP to Heating<br />
</pre><br />
Defines MB-485 as a base device for the RS-485 communication with a heating system, <br />
defines Heating as a Modbus Master to communicate with the Heating and its Modbus ID 22, <br />
and then defines the relay which listens to the local IP address 192.168.1.2, TCP port 1502, Modbus Id 33 and protocol Modbus-TCP.<br />
Requests coming in through Modbus TCP and port 1502 are then translated to Modbus RTU and forwarded via RS-485 to the heating system with Modbus Id 22.<br />
<br />
Please note that the IP address specified in the relay definition is a local IP address of the relay itself and that the port number is 1502, not 502 because Fhem running on Linux can not easily open listening port numbers below 1024 unless it is running as root.<br />
<br />
Other (unlikely) Example:<br />
<pre><br />
define MB-232 Modbus /dev/ttyUSB2@19200<br />
define Solar ModbusAttr 7 0 192.168.1.122:502 RTU<br />
define PLC2NetRelay ModbusAttr 1 ASCII relay to Solar<br />
</pre><br />
Defines MB-232 as a base device for the RS-232 communication with a PLC as Modbus master, <br />
defines Solar as a Modbus Master to communicate with Modbus TCP to a Solar power system at IP Adrress 192.168.1.122 and its Modbus ID 7, <br />
and then defines the PLC2NetRelay as a relay which listens to Modbus-ASCII requests over the serial RS-232 link from a PLC to Modbus ID 1.<br />
Requests to Modbus Id 1 coming in through the serial link are then translated to Modbus TCP and forwarded over the network to the solar power system with Modbus Id 7.<br />
<br />
== Configuration of the module as master or passive listener == <br />
<br />
Data objects (holding registers, input registers, coils or discrete inputs) are defined using attributes. <br />
If Fhem is Modbus master or passive listener, the attributes assign data objects of external devices (heating systems, power meters, PLCs or other) with their register addresses to readings inside fhem and control how these readings are calculated from the raw values and how they are formatted.<br />
Please be aware that Modbus does not define common data types so the representation of a value can be very different from device to device. One device might make a temperature value avaliable as a floating point value that is stored in two holding resgisters, another device might store the temperature multiplied with 10 as an signed integer in one register. Even the order of bytes can vary.<br />
Therefore it is typically necessary to specify the data representation as a Perl unpack code.<br />
A Modbus master can also write values to Objects in the device and attributes define how this is done.<br />
<br />
Example for a Modbus master or passive configuration:<br />
<pre><br />
define PWP ModbusAttr 5 30<br />
attr PWP obj-h256-reading Temp_Wasser_ein<br />
attr PWP obj-h256-expr $val/10<br />
<br />
attr PWP obj-h258-reading Temp_Wasser_Aus<br />
attr PWP obj-h258-expr $val/10<br />
<br />
attr PWP obj-h262-reading Temp_Luft<br />
attr PWP obj-h262-expr $val / 10<br />
<br />
attr PWP obj-h770-reading Temp_Soll<br />
attr PWP obj-h770-expr $val / 10<br />
attr PWP obj-h770-set 1<br />
attr PWP obj-h770-setexpr $val * 10<br />
attr PWP obj-h770-max 32<br />
attr PWP obj-h770-min 10<br />
attr PWP obj-h770-hint 8,10,20,25,28,29,30,30.5,31,31.5,32<br />
<br />
attr PWP dev-h-combine 5<br />
attr PWP dev-h-defPoll 1<br />
attr PWP dev-h-defUnpack n<br />
<br />
attr PWP room Pool-WP<br />
attr PWP stateFormat {sprintf("%.1f Grad", ReadingsVal($name,"Temp_Wasser_Ein",0))}<br />
attr PWP webCmd Temp_Soll<br />
</pre><br />
<br />
Attributes to define data objects start with obj- followed by a code that identifies the type and address<br />
of the data object. <br />
<br />
Modbus devices offer the following types of data objects: <br />
<br />
;holding registers (16 bit objects that can be read and written)<br />
;input registers (16 bit objects that can only be read)<br />
;coils (single bit objects that can be read and written)<br />
;discrete inputs (single bit objects that can only be read)<br />
<br />
The module uses the first character of these data object types to define attributes. <br />
Thus h770 refers to a holding register with the decimal address 770 and c120 refers to a coil with address 120. <br />
The address has to be specified as pure decimal number. The address counting starts at address 0<br />
<br />
Please note that the documentation for devices sometimes uses different numbering. They might start counting with one instead of zero so if a voltage value is stored in input register number 107 according to the documentation of the device, it might technically mean register number 106 (in the Modbus protocol specification addresses start with 0).<br />
Also some vendors use hexadecimal descriptions of their register addresses. So input register 107 might be noted as hex and means 263 or even 262 as decimal address.<br />
<br />
<pre><br />
attr PWP obj-h258-reading Temp_Wasser_Aus<br />
</pre> <br />
defines a reading with the name Temp_Wasser_Aus that is read from the Modbus holding register at address 258.<br />
With the attribute ending on <code>-expr</code> you can define a perl expression to do some conversion or calculation on the raw value read from the device. <br />
In the above example the raw value has to be devided by 10 to get the real value. If the raw value is also the final value then no <code>-expr</code> attribute is necessary. <br />
<br />
An object attribute ending on <code>-set</code> creates a fhem set option. <br />
In the above example the reading Temp_Soll can be changed to 12 degrees by the user with the fhem command <code>set PWP Temp_Soll 12</code><br />
The object attributes ending on <code>-min</code> and <code>-max</code> define min and max values for input validation <br />
and the attribute ending on <code>-hint</code> will tell fhem to create a selection list so the user can graphically select the defined values.<br />
<br />
To define general properties of the device you can specify attributes starting with <code>dev-</code>. <br />
E.g. with <code>dev-timing-timeout</code> you can specify the timeout when waiting for a response from the device. <br />
With <code>dev-h-</code> you can specify several default values or general settings for all holding registers <br />
like the function code to be used when reading or writing holding registers. <br />
These attributes are optional and the module will use defaults that work in most cases. <br />
<code>dev-h-combine 5</code> for example allows the module to combine read requests to objects having an address that differs 5 or less into one read request. <br />
Without setting this attribute the module will start individual read requests for each object. <br />
Typically the documentation for the modbus interface of a given device states the maximum number of objects that can be read in one function code 3 request.<br />
<code>dev-h-defUnpack n</code> means that the values in this example that the values are stored as unsigned short (16-bit) in "network" (big-endian) order. This is only one possibility of many. An integer value might be signed instead of unsigned or it might use different byte ordering (e.g. unpack codes v or s). <br />
<br />
== Handling Data Types == <br />
<br />
The Modbus protocol does not define data types. If the documentation of a device states that for example the current temperature is stored in holding register 102 this leaves room for many interpretations. Not only can the address 102 mean different things (actually decimal 102 or rather 101 if the vendor starts counting at 1 instead of 0 or even 257 or 258 if the vendor used hexadecimal addresses in his documentation ) also the data representation can be many different things. As in every programming language, there are many ways to represent numbers. They can be stored signed or unsigned, they can be integers or floating point numbers, the byte-order can be "big endian" or "small endian", the value can be stored in one holding register or in two holding registers (floating point numbers typically take four bytes which means two holding registers).<br />
The Modbus module allows flexible configuration of data representations be assigning a Perl unpack-code, a length, a Perl Expression, and the register ordering. The following example illustrates how this can be done: <br />
<pre><br />
attr PWP obj-h338-reading Pressure<br />
attr PWP obj-h338-len 2<br />
attr PWP obj-h338-unpack f><br />
attr PWP obj-h338-revRegs 1<br />
attr PWP obj-h338-format %.2f<br />
</pre><br />
In This example a floating point value for the reading "Pressure" is read from the holding registers starting at address 338. <br />
The value occupies 32 Bits and is therefore stored in two registers. The Perl pack code to use is f> which means a native single precision float in big endian format (byte order). With revRegs the module is instructed to reverse the order of the registers directly after reading. The format specification then defines how the value is formatted into a reading - in this case with two digits after the comma. See http://perldoc.perl.org/functions/pack.html for Perl pack / unpack codes and http://perldoc.perl.org/functions/sprintf.html for format specifications.<br />
<br />
If you need to read / write many objects for a device, defining all these parameters each time is not elegant. The Modbus module therefore offers twi ways to simplify this task: <br />
You can define defaults for every type of object or you can define your own data types once and then refer to them.<br />
This exampe shows how defaults can be specified for holding registers and input registers:<br />
<pre><br />
attr PWP dev-h-defUnpack f><br />
attr PWP dev-h-defLen 2<br />
attr PWP dev-h-defRevRegs 1<br />
attr PWP dev-h-defFormat %.2f<br />
<br />
attr PWP dev-i-defUnpack n<br />
attr PWP dev-i-defLen 1<br />
</pre><br />
<br />
The next example shows how you can define your own data types and then apply them to objects:<br />
<pre><br />
attr WP dev-type-VT_R4-format %.1f<br />
attr WP dev-type-VT_R4-len 2<br />
attr WP dev-type-VT_R4-revRegs 1<br />
attr WP dev-type-VT_R4-unpack f><br />
<br />
attr WP obj-h1234-reading Temp_In<br />
attr WP obj-h1234-type VT_R4<br />
attr WP obj-h1236-reading Temp_Out<br />
attr WP obj-h1236-type VT_R4<br />
</pre><br />
This example defines a data type with the name VT_R4 which uses an unpack code of f>, length 2 and reversed register ordering. It then assigns this Type to the objects Temp_In and Temp_Out.<br />
<br />
== Configuration of the module as Modbus slave == <br />
<br />
Data objects that the module offers to external Modbus masters (holding registers, input registers, coils or discrete inputs) are defined using attributes. <br />
If Fhem is Modbus slave, the attributes assign readings of Fhem devices to Modbus objects with their addresses and control how these objects are calculated from the reading values that exist in Fhem.<br />
It is also possible to allow an external Modbus master to send write function codes and change the value of readings inside Fhem.<br />
<br />
Example for a Modbus slave configuration:<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 1 slave<br />
attr Data4PLC IODev MRS485<br />
<br />
attr Data4PLC obj-h256-reading THSensTerrasse:temperature<br />
attr Data4PLC obj-h256-unpack f<br />
attr Data4PLC obj-h256-len 2<br />
<br />
attr Data4PLC obj-h258-reading THSensTerrasse:humidity<br />
attr Data4PLC obj-h258-unpack f<br />
attr Data4PLC obj-h258-len 2<br />
<br />
attr Data4PLC obj-h260-reading myDummy:limit<br />
attr Data4PLC obj-h260-unpack n<br />
attr Data4PLC obj-h260-len 1<br />
attr Data4PLC obj-h260-allowWrite 1<br />
</pre><br />
<br />
In this example Fhem allows an external Modbus master to read the temperature of a Fhem device named THSensTerrasse through holding register 256 and the humidity of that Fhem device through holding register 258. Both are encoded as floting point values that span two registers. <br />
The master can also read but also write the reading named limit of the device myDummy.<br />
<br />
== Set-Commands ==<br />
can be defined for holding registers and coils by using attributes.<br />
<br />
Every object for which an attribute like <code>obj-xy-set</code> is set to 1 will create a valid set option.<br />
<br />
Additionally the attribute <code>enableControlSet</code> enables the set options <code>interval</code>, <code>stop</code>, <code>start</code>, <code>reread</code> as well as <code>scanModbusObjects</code>, <code>scanStop</code> and <code>scanModbusIds</code> (for devices connected with RTU / ASCII over a serial line).<br />
<br />
;<code>interval &lt;Interval&gt;</code><br />
:modifies the interval that was set during define. <br />
<br />
;<code>stop</code><br />
:stops the interval timer that is used to automatically poll objects through Modbus.<br />
<br />
;<code>start</code><br />
:starts the interval timer that is used to automatically poll objects through Modbus. <br />
:If an interval is specified during the define command then the interval timer is started automatically. <br />
:However if you stop it with the command <code>set &lt;mydevice&gt; stop</code> <br />
:then you can start it again with <code>set &lt;mydevice&gt; start</code>.<br />
<br />
;<code>reread</code><br />
:causes a read of all objects that are set to be polled in the defined interval. The interval timer is not modified.<br />
<br />
;<code>scanModbusObjects &lt;startObj&gt; - &lt;endObj&gt; &lt;reqLen&gt;</code><br />
:scans the device objects and automatically creates attributes for each reply it gets. <br />
:This might be useful for exploring devices without proper documentation. <br />
:The following example starts a scan and queries the holding registers with addresses between 100 and 120. <br />
:<code>set MyModbusAttrDevice scanModbusObjects h100-120</code><br><br />
:For each reply it gets, the module creates a reading like<br />
:<code>scan-h100 hex=0021, string=.!, s=8448, s>=33, S=8448, S>=33</code><br><br />
:the representation of the result as hex is 0021 and<br />
:the ASCII representation is .!. s, s>, S and S> are different representations with their Perl pack-code.<br />
<br />
;<code>scanModbusIds &lt;startId&gt; - &lt;endId&gt; &lt;knownObj&gt;</code><br />
:scans for Modbus Ids on an RS485 Bus. The following set command for example starts a scan:<br><br />
:<code>set Device scanModbusId 1-7 h770</code><br><br />
:since many Modbus devices don't reply at all if an object is requested that does not exist, <br />
:scanModbusId needs the adress of an object that is known to exist.<br />
:If a device with Id 5 replies to a read request for holding register 770, a reading like the following will be created:<br />
:<code>scanId-5-Response-h770 hex=0064, string=.d, s=25600, s>=100, S=25600, S>=100</code><br />
;<code>scanStop</code><br />
:stops any running scans.<br />
;saveAsModule <name><br />
:experimental: saves the definitions of obj- and dev- attributes in a new fhem module file as /tmp/98_ModbusGen<name>.pm.<br />
:if this file is copied into the fhem module subdirectory (e.g. /opt/fhem/FHEM) and fhem is restarted then instead of defining a device<br />
:as ModbusAttr with all the attributes to define objects, you can just define a device of the new type ModbusGen<name> and all the <br />
:objects will be there by default. However all definitions can still be changed / overriden with the attribues defined in ModbusAttr if needed.<br />
<br />
== Get-Commands ==<br />
<br />
Every reading can be manually requested by a Get. <br />
Internally a Get command triggers the corresponding Modbus request to the device and the module then interprets the data and sets the right Fhem readings. To avoid huge option lists in FHEMWEB, the objects visible as Get in FHEMWEB can be defined by setting an attribute <code>obj-xy-showGet</code> to 1.<br />
<br />
== All Attributes ==<br />
<br />
;readingFnAttributes<br />
:the usual Fhem attributes for all devices<br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. <br />
:This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;enableControlSet<br />
:enables the built in set commands like interval, stop, start and reread (see above) <br />
<br />
the following list of attributes can be applied to any data object by specifying the objects type and address in the variable part. <br />
For many attributes you can also specify default values per object type (see dev- attributes later) or you can specify an object attribute without type and address (e.g. obj-len) which then applies as default for all objects:<br />
<br />
;obj-[cdih][1-9][0-9]*-reading <br />
:define the name of a reading that corresponds to the Modbus data object of type c,d,i or h and a decimal address (e.g. obj-h225-reading).<br />
<br />
;obj-[cdih][1-9][0-9]*-name <br />
:defines an optional internal name of this data object (this has no meaning for fhem and serves mainly documentation purposes.<br />
<br />
;obj-[cdih][1-9][0-9]*-set <br />
:if set to 1 then this data object can be changed (works only for holding registers and coils since discrete inputs and input registers can not be modified by definition.<br />
<br />
;obj-[cdih][1-9][0-9]*-min <br />
:defines a lower limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-max <br />
:defines an upper limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-hint <br />
:this is used for set options and tells fhemweb what selection to display for the set option (list or slider etc.)<br />
<br />
;obj-[cdih][1-9][0-9]*-expr <br />
:defines a perl expression that converts the raw value read from the device.<br />
<br />
;obj-[cdih][1-9][0-9]*-ignoreExpr<br />
:defines a perl expression that returns 1 if a value should be ignored and the existing reading should not be modified<br />
<br />
;obj-[cdih][1-9][0-9]*-map <br />
:defines a map to convert values read from the device to more convenient values when the raw value is read from the device or back when the value to write has to be converted from the user value to a raw value that can be written. Example: 0:mittig, 1:oberhalb, 2:unterhalb <br />
<br />
;obj-[cdih][1-9][0-9]*-setexpr <br />
:defines a perl expression that converts the user specified value in a set to a raw value that can be sent to the device. This is typically the inversion of -expr above.<br />
<br />
;obj-[cdih][1-9][0-9]*-format <br />
:defines a format string to format the value read e.g. %.1f<br />
<br />
;obj-[cdih][1-9][0-9]*-len <br />
:defines the length of the data object in registers. It defaults to 1. Some devices store 32 bit floating point values in two registers. In this case you can set this attribute to two.<br />
<br />
;obj-[cdih][1-9][0-9]*-unpack <br />
:defines the unpack code to convert the raw data string read from the device to a reading. For an unsigned integer in big endian format this would be "n", for a signed 16 bit integer in big endian format this would be "s>" and for a 32 bit big endian float value this would be "f>". (see the perl documentation of the pack function).<br />
<br />
;obj-[cdih][1-9][0-9]*-revRegs<br />
:this is only applicable to objects that span several input registers or holding registers.<br />
:when they are read then the order of the registers will be reversed before <br />
:further interpretation / unpacking of the raw register string. <br />
:The same happens before the object is written with a set command.<br />
<br />
;obj-[cdih][1-9][0-9]*-bswapRegs<br />
:this is applicable to objects that span several input or holding registers.<br />
:After the registers have been read and before they are writtem, <br />
:all 16-bit values are treated big-endian and are reversed to little-endian by swapping the two 8 bit bytes. <br />
:This functionality is most likely used for reading (ASCII) strings from the device <br />
:that are stored as big-endian 16-bit values.<br />
:example: original reading is "324d3130203a57577361657320722020". After applying bswapRegs, <br />
:the value will be "4d3230313a2057576173736572202020" which will result in the ASCII string <br />
:"M201: WWasser ". <br />
:Should be used with "(a*)" as -unpack value.<br />
<br />
;obj-[cdih][1-9][0-9]*-decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-showGet <br />
:every reading can also be requested by a get command. However these get commands are not automatically offered in fhemweb. By specifying this attribute, the get will be visible in fhemweb.<br />
<br />
;obj-[cdih][1-9][0-9]*-poll<br />
:if set to 1 then this obeject is included in the cyclic update request as specified in the define command. If not set, then the object can manually be requested with a get command, but it is not automatically updated each interval. Note that this setting can also be specified as default for all objects with the dev- atributes described later.<br />
<br />
;obj-[cdih][1-9][0-9]*-polldelay <br />
:this attribute allows to poll objects at a lower rate than the interval specified in the define command. you can either specify a time in seconds or number prefixed by "x" which means a multiple of the interval of the define command. if you specify a normal numer then it is interpreted as minimal time between the last read and another automatic read. Please note that this does not create an individual interval timer. Instead the normal interval timer defined by the interval of the define command will check if this reading is due or not yet. So the effective interval will always be a multiple of the interval of the define.<br />
<br />
<br />
;dev-([cdih]-)*read <br />
:specifies the function code to use for reading this type of object. The default is 3 for holding registers, 1 for coils, 2 for discrete inputs and 4 for input registers.<br />
<br />
;dev-([cdih]-)*write <br />
:specifies the function code to use for writing this type of object. The default is 6 for holding registers and 5 for coils. Discrete inputs and input registers can not be written by definition.<br />
<br />
;dev-([cdih]-)*combine <br />
:defines how many adjacent objects can be read in one request. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defLen <br />
:defines the default length for this object type. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defFormat <br />
:defines a default format string to use for this object type in a sprintf function on the values read from the device.<br />
<br />
;dev-([cdih]-)*defExpr<br />
:defines a default Perl expression to use for this object type to convert raw values read.<br />
<br />
;dev-([cdih]-)*defIgnoreExpr<br />
:defines a default Perl expression to decide when values should be ignored.<br />
<br />
;dev-([cdih]-)*defUnpack <br />
:defines the default unpack code for this object type. <br />
<br />
;dev-([cdih]-)*defRevRegs<br />
:defines that the order of registers for objects that span several registers will be reversed before <br />
:further interpretation / unpacking of the raw register string<br />
<br />
;dev-([cdih]-)*defBswapRegs<br />
:per device default for swapping the bytes in Registers (see obj-bswapRegs above)<br />
<br />
;dev-([cdih]-)*defDecode<br />
:defines a default for decoding the strings read from a different character set e.g. cp850<br />
<br />
;dev-([cdih]-)*defEncode<br />
:defines a default for encoding the strings read (or after decoding from a different character set) e.g. utf8<br />
<br />
;dev-([cdih]-)*defPoll <br />
:if set to 1 then all objects of this type will be included in the cyclic update by default. <br />
<br />
;dev-([cdih]-)*defShowGet <br />
:if set to 1 then all objects of this type will have a visible get by default. <br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
<br />
;dev-([cdih]-)*allowShortResponses <br />
:if set to 1 the module will accept a response with valid checksum but data lengh < lengh in header<br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
;nextOpenDelay <br />
:delay for Modbus-TCP connections. <br />
:This defines how long the module should wait after a failed TCP connection attempt before the next reconnection attempt. <br />
:This defaults to 60 seconds.<br />
<br />
;openTimeout <br />
:timeout to be used when opening a Modbus TCP connection (defaults to 3)<br />
<br />
;timeoutLogLevel <br />
:log level that is used when logging a timeout. Defaults to 3. <br />
<br />
;silentReconnect <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
<br />
;maxTimeoutsToReconnect <br />
:this attribute is only valid for TCP connected devices. <br />
:In such cases a disconnected device might stay undetected and lead to timeouts until the TCP connection is reopened. <br />
:This attribute specifies after how many timeouts an automatic reconnect is tried.<br />
<br />
;dev-h-brokenFC3<br />
:workaround for some broken Modbus function code 3 implementations<br />
<br />
;disable<br />
:stop communication with the device while this attribute is set to 1. For Modbus over TCP this also closes the TCP connection.<br />
<br />
== Links ==<br />
* [http://www.Modbus.org Modbus.org]<br />
* About Modbus ([http://en.wikipedia.org/wiki/Modbus English] / [http://de.wikipedia.org/wiki/Modbus German])<br />
* Perl [http://perldoc.perl.org/functions/pack.html unpack codes]<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=33263ArduCounter2020-05-22T19:12:54Z<p>StefanStrobel: update for latest version</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, measure time between pulses and convert this to readings for e.g. power consumption of Energy meters or water meters<br />
|ModType=d<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
This module implements an Interface to an Arduino, ESP8266 or ESP32 based counter for pulses on any input pin of an Arduino Uno, Nano, Jeenode, <br />
NodeMCU, Wemos D1, TTGO T-Display or similar device. <br />
The device connects to Fhem either through USB / serial or via Wifi / TCP if an ESP board is used.<br />
ArduCounter does not only count pulses but also measure pulse lenghts and the time between pulses so it can filter noise / bounces<br />
and gives better power/flow (Watts or liters/second) readings than systems that just count in fixed time intervals.<br />
The number of pulses per kWh or liter is defineable and counters continue even when Fhem or the device restarts<br />
so you don't need additional user readings to make such calculations<br />
The typical use case is an S0-Interface on an energy meter or water meter, but also reflection light barriers <br />
to monitor old ferraris counters or analog water meters are supported<br />
Counters are configured with attributes that define which GPIO pins should count pulses and in which intervals the board should report the current counts.<br />
The sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
The module creates readings for pulse counts, consumption and optionally also a pin history with pulse lengths and gaps of the last pulses.<br />
If an ESP8266 or ESP32 is used, the device can be flashed and configured over Wifi (it opens its own temporary Hotspot / SSID for configuration <br />
so you can set which existing SSID to connect to and which password to use). For TTGO T-Display boards (ESP32 with TFT display) <br />
the local display on the device itself can also display Wifi status and current consumption.<br />
<br />
== Prerequisites ==<br />
<br />
This module requires an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1, TTGO T-Display or similar device based on an Atmel 328p, ESP8266 or ESP32 <br />
running the ArduCounter sketch provided with this module<br />
In order to flash an arduino board with the corresponding ArduCounter firmware from within Fhem, avrdude needs to be installed.<br />
To flash ESP32 or ESP8266 boards form within Fhem, Python and the scripts esptool.py / espota.py need to be installed.<br />
For old ferraris counters a reflection light barrier which in the simpest case can consist of a photo transistor (connected to an anlalog input of the Arduino / ESP) <br />
and an led or a laser module (connected to a digital output), both with a resistor in line are needed. <br />
To drive a laser module with 5V, another transistor is typically needed to switch 5V from a 3.3V GPIO output.<br />
<br />
== Define ==<br />
<br />
<code>define <name> ArduCounter <device></code><br />
or<br />
<code>define <name> ArduCounter <ip:port></code><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<ip:port> specifies the ip address and tcp port to communicate with an esp8266 / ESP32 where port is typically 80.<br />
<br />
The name of the serial device depends on your distribution and serial adapter.<br />
You can also specify a baudrate for serial connections if the device name contains the @ character, e.g.: /dev/ttyUSB0@115200<br />
The default baudrate of the ArduCounter firmware is 115200 since sketch version 4 and used to be 38400 since sketch version 1.4<br />
The latest version of this module will however try different baudrates automatically if communication with the counting device seems not possible.<br />
<br />
Example:<br />
<br />
<code>define AC ArduCounter /dev/ttyUSB2@115200</code><br />
<code>define AC ArduCounter 192.168.1.134:80</code><br />
<br />
== Configuration of ArduCounter digital counters ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <code>attr AC pinX falling pullup min 25</code> <br />
The X in pinX can be an Arduino / ESP GPIO pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
After the pin you can use the keywords falling or rising to define if a logical one / 5V (rising) or a logical zero / 0V (falling) should be treated as pulse.<br />
The optional keyword pullup activates the pullup resistor for the given Pin. <br />
The last argument is also optional but recommended and specifies a minimal pulse length in milliseconds.<br />
An energy meter with S0 interface is typically connected to GND and an input pin like D4. <br />
The S0 pulse then pulls the input down to 0V.<br />
Since the minimal pulse lenght of an S0 interface is specified to be 30ms, the typical configuration for an s0 interface is <br />
<code>attr AC pinX falling pullup min 25</code><br />
Specifying a minimal pulse length is recommended since it filters bouncing of reed contacts or other noise. <br />
The keyword <code>min</code> before <code>25</code> is optional.<br />
<br />
Example:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC pulsesPerUnit 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup min 5<br />
attr AC pinD5 falling pullup min 25<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines a counter that is connected to Fhem via serial line ttyUSB2 with three counters connected to the GPIO pins D4, D5 and D5. <br />
D4 and D5 have their pullup resistors activated and the impulses draw the pins to zero.<br />
For D4 and D5 the board measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 5 (or 25 for pin D5) milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
For pin D6 the board uses a default minimal length of 2ms and counts every time when the signal changes from 1 (rising pulse) back to 0.<br />
<br />
== Configuration of ArduCounter analog counters ==<br />
<br />
This module and the corresponding ArduCounter sketch can be used to read water meters or old analog ferraris energy counters. <br />
Therefore a reflection light barrier needs to be connected to the board. This might simply consist of an infra red photo transistor <br />
(connected to an analog input) and an infra red led (connected to a digital output), both with a resistor in line. <br />
The idea comes from Martin Kompf (https://www.kompf.de/tech/emeir.html) and has been adopted for ArduCounter to support <br />
old ferraris energy counters or water meters.<br />
The configuration is then similar to the one for digital counters:<br />
<pre><br />
define WaterMeter ArduCounter 192.168.1.110:80<br />
attr ACF pinA0 rising pullup min 4 analog out 27 threshold 120,220<br />
attr ACF interval 5,60,2,15,10,3<br />
attr ACF pulsesPerUnit 35<br />
attr ACF stateFormat {sprintf("%.3f l/min", ReadingsVal($name,"powerA0",0))}<br />
</pre><br />
In this case an analog GPIO pin is used as input and the normal configuration parameters are followed by the keyword <br />
<code>analog out</code> or simply <code>out</code>, the gpio number of a GPIO output that connects a light source and the thresholds <br />
that decide when an analog input value is regarded as "low" or "high".<br />
<br />
In the example an ESP32 is used via Wifi connection. GPIO pin A0 is used as analog input and is connected to a photo transistor that senses the intensity of light.<br />
GPIO 27 is used as LED output and switched on/off in a high frequency. On GPIO A0 the reflected light is measured <br />
and the difference in a measurement between when the LED is off and when the LED is on is compared to the thresholds defined in the pinA0-attribute. <br />
When the measured light difference is above <code>220</code>, then a pulse starts (since <code>rising</code> is specified). <br />
When the measured difference is below <code>120</code> then the pulse ends.<br />
<br />
The attribute <code>interval</code> has the following meaning in the above example: <br />
The device reports the current counts and the time difference beween the first and the last pulse if at least 2 pulses have been counted <br />
and if they are more than 15 milliseconds apart form each other. If not, then the device continues counting. <br />
If after 60 seconds these conditions are stil not met, then the device will report the current count anyways and use the current time as the end of the interval.<br />
The last two numbers of the <code>interval</code> attribute define that the device will read the analog input 3 times and then work with the average. <br />
Between each analog measurement series there will be a delay of 10 milliseconds.<br />
<br />
The attribute <code>pulsesPerUnit 35</code> defines that 35 pulses correspond to one unit (e.g. liter) and the reading <code>calcCounterA0</code> <br />
is increased by the reported raw counts divided by 35.<br />
To find out the right analog thresholds you can set the attribute <code>enableHistory</code> to 1 which will ask the firmware of your counting board <br />
to report the average difference measurements before they are compared to a threshold. <br />
The ArduCounter module will count how often each value is reported and you can then query these analog level counts with <code>get levels</code>. <br />
After a few measuremets the result of <code>get levels</code> might look like this:<br />
<br />
<pre><br />
observed levels from analog input:<br />
94: 21<br />
95: 79<br />
96: 6<br />
97: 2<br />
98: 3<br />
99: 2<br />
100: 2<br />
101: 1<br />
102: 3<br />
105: 2<br />
106: 1<br />
108: 2<br />
109: 1<br />
110: 1<br />
112: 1<br />
113: 3<br />
115: 4<br />
116: 9<br />
117: 14<br />
118: 71<br />
119: 103<br />
120: 118<br />
121: 155<br />
122: 159<br />
123: 143<br />
124: 147<br />
125: 158<br />
126: 198<br />
127: 249<br />
128: 220<br />
129: 230<br />
130: 201<br />
131: 140<br />
132: 147<br />
133: 153<br />
134: 141<br />
135: 119<br />
136: 105<br />
137: 109<br />
138: 114<br />
139: 83<br />
140: 33<br />
141: 14<br />
142: 1 <br />
</pre><br />
<br />
This shows the measured values together with the frequency how often the individual value has been measured. <br />
It is obvious that most measurements result in values between 120 and 135, very few values are betweem 96 and 115 <br />
and another peak is around the value 95. <br />
It means that in the example of a ferraris energy counter, when the red mark of the ferraris disc is under the sensor, <br />
the value is around 95 and while when the blank disc is under the sensor, the value is typically between 120 and 135. <br />
So a good upper threshold would be 120 and a good lower threshold would be for example 96.<br />
<br />
== Set-Commands ==<br />
<br />
;raw<br />
:send the value to the board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash [<file>]<br />
:flashes the ArduCounter firmware from the subdirectory FHEM/firmware onto the device.<br />
:Normally you can just specify <code>set myDevice flash</code>. The parameter <file> is optional and allows specifying an alternative firmware file.<br />
:The attribute flashCommand can be used to override which command is executed. <br />
:If the attribute flashCommand is not specified then the module selects an appropriate command depending on the board type <br />
:(set with the attribute <code>board</code>) and depending on the connection (serial or Wifi).<br />
:For an arduino NANO for example the module would execute avrdude (which has to be installed of course) <br />
:and flash the connected arduino with the updated hex file <br />
:(by default it looks for ArduCounter.hex in the FHEM/firmware subdirectory).<br />
:For an Arduino UNO for example the default is <code>avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]</code><br />
:For an Arduino Nano based counter <code>-b 57600</code> is added.<br />
:For an ESP32 connected via Wifi, the module would call espota.py which will upload the firmware over the air.<br />
:If the attribute flashCommand is not specified for an ESP32 based board connected via serial line, then the module uses the command <br />
<br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
esptool.py --chip esp32 --port [PORT] --baud 460800 --before default_reset --after hard_reset write_flash -z --flash_mode dio --flash_freq 40m --flash_size detect<br />
0x1000 FHEM/firmware/ArduCounter_ESP32_bootloader_dio_40m.bin 0x8000 FHEM/firmware/ArduCounter_ESP32_partitions.bin 0xe000 FHEM/firmware/ArduCounter_ESP32_boot_app0.bin<br />
0x10000 FHEM/firmware/ArduCounter_ESP32_firmware.bin >[LOGFILE] 2>&1<br />
</syntaxhighlight> <br />
<br />
:for example which flashes the whole ESP32 with all the partitions. For over the air flashing it would use <br />
:<syntaxhighlight lang="bash" xstyle="width:80%;"><br />
espota.py -i[IP] -p [NETPORT] -f [BINFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:Of course esptool.py or espota.py as well as python would need to be installed on the system.<br />
<br />
;resetWifi<br />
:reset Wifi settings of the counting device so the Wifi Manager will come up after the next reset to select a wireless network and enter the Wifi passphrase.<br />
<br />
;reset <br />
:sends a command to the device which causes a hardware reset or reinitialize and reset of the internal counters of the board. <br />
:The module then reopens the counting device and resends the attribute configuration / definition of the pins.<br />
<br />
;saveConfig <br />
:stores the current interval, analog threshold and pin configuration in the EEPROM of the counter device so it will automatically be retrieved after a reset.<br />
<br />
;enable <br />
:sets the attribute disable to 0<br />
<br />
;disable <br />
sets the attribute disable to 1<br />
<br />
;reconnect <br />
:closes the tcp connection to an ESP based counter board that is conected via TCP/IP and reopen the connection<br />
<br />
;clearLevels <br />
:clears the statistics for analog levels. This is only relevant if you use the board to read via a reflective light barrier <br />
:and you want to set the thresholds according to the statistics.<br />
<br />
;clearCounters <pin> <br />
:resets all the counter readings for the specified pin to 0<br />
<br />
;counter <pin>, <value> <br />
:set the calcCounter reading for the specified pin to the given value<br />
<br />
;clearHistory <br />
:deletes all the cached pin history entries<br />
<br />
== Get-Commands ==<br />
<br />
;info <br />
:send a command to the Arduino board to get current counts.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;levels <br />
:show the count for the measured levels if an analog pin is used to measure e.g. the red mark of a ferraris counter disc. This is useful for setting the thresholds for analog :measurements. <br />
<br />
;history <pin> <br />
:shows details regarding all the level changes that the counter device (Arduino or ESP) has detected and how they were used (counted or rejected)<br />
:If get history is issued with a pin name (e.g. get history D5) then only the history entries concerning D5 will be shown.<br />
:This information is sent from the device to Fhem if the attribute <code>enableHistory</code> is set to 1.<br />
:The maximum number of lines that the Arducounter module stores in a ring buffer is defined by the attribute maxHist and defaults to 1000.<br />
<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;pin[AD]?[0-9]+<rising|falling> [<pullup>] [min] <min length> [[analog] out <out pin> [threshold] <min, max>] <br />
:Define a GPIO pin of the Arduino or ESP board as input. This attribute expects for digital inputs either <br />
:<code>rising</code> or <code>falling</code>, followed by an optional <code>pullup</code> and the optional keyword <code>min</code> <br />
:and an optional number as minimal length of pulses and gaps between pulses.<br />
:The counter device will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds.<br />
:The minimal length specified here is the minimal duration of a pulse and a pause before a pulse. If one is too small, <br />
:the pulse is not counted but added to a separate reject counter.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinD4 falling pullup 25<br />
</syntaxhighlight> <br />
:For analog inputs with connected reflective light barries, you have to add <code>analog out</code> <br />
:and the GPIO pin number of the pin where the light source (LED or laser) is connected, the keyword <code>threshold</code> <br />
:followed by the lower and upper threshold separated by a komma.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr MyCounter pinA0 rising pullup min 3 analog out 27 threshold 120,220<br />
</syntaxhighlight> <br />
<br />
;interval <normal> <max> [<min> <min count> [<analog interval> <analog samples>]] <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of six numbers as value. <br />
:The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count. <br />
:The last two numbers are only needed for counting with reflective light barriers. They specify the delay between the measurements <br />
:and the number of samples for each measurement.<br />
:<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval),<br />
:the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be <br />
:something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is then calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period <br />
:and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
:<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case <br />
:when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, <br />
:the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) <br />
:or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed <br />
:or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter interval 60 600 5 2<br />
</syntaxhighlight> <br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. <br />
:This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. <br />
:In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
:<br />
:For analog sampling the last two numbers define the delay in milliseconds between analog measurements and the number of samples that will be taken as one mesurement.<br />
<br />
;pulsesPerUnit <number> <br />
:specify the number of pulses that the meter is giving out per unit that sould be displayed (e.g. per kWh energy consumed). <br />
:For many S0 counters this is 1000, for old ferraris counters this is 75 (rounds per kWh).<br />
:This attribute used to be called pulsesPerKWh and this name still works but the new name should be used preferably since the old one could be removed in future versions.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter pulsesPerUnit 75<br />
</syntaxhighlight> <br />
<br />
;readingPulsesPerUnit[AD]?[0-9]+ <number> <br />
:is the same as pulsesPerUnit but specified per GPIO pin individually in case you have multiple counters with different settings at the same time<br />
:This attribute used to be called readingPulsesPerKWh[AD]?[0-9]+ and this name still works but the new name should be used preferably <br />
:since the old one could be removed in future versions.<br />
:<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingPulsesPerUnitA7 75<br />
attr myCounter readingPulsesPerUnitD4 1000<br />
</syntaxhighlight> <br />
<br />
;readingFlowUnitTime[AD]?[0-9]+ <time> <br />
:specified the time period in seconds which is used as the basis for calculating the current flow or power for the given pin.<br />
:If the counter e.g. counts liters and you want to see the flow in liters per minute, then you have to set this attribute to 60.<br />
:If you count kWh and you want to see the current power in kW, then specify 3600 (one hour).<br />
:Since this attribute is just used for multiplying the consumption per second, you can also use it to get watts <br />
:instead of kW by using 3600000 instead of 3600.<br />
<br />
;flowUnitTime <time> <br />
:like readingFlowUnitTimeXX but applies to all pins that have no explicit readingFlowUnitTimeXX attribute.<br />
<br />
;readingNameCount[AD]?[0-9]+ <new name> <br />
:Change the name of the counter reading pinX to something more meaningful. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCountD4 CounterHaus_internal<br />
</syntaxhighlight> <br />
<br />
;readingNameLongCount[AD]?[0-9]+ <new name> <br />
:Change the name of the long counter reading longX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameLongCountD4 CounterHaus_long<br />
</syntaxhighlight> <br />
<br />
;readingNameInterpolatedCount[AD]?[0-9]+ <new name> <br />
:Change the name of the interpolated long counter reading InterpolatedlongX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameInterpolatedCountD4 CounterHaus_interpolated<br />
</syntaxhighlight> <br />
<br />
;readingNameCalcCount[AD]?[0-9]+ <new name> <br />
:Change the name of the real unit counter reading CalcCounterX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNameCalcCountD4 CounterHaus_kWh<br />
</syntaxhighlight> <br />
<br />
;readingNamePower[AD]?[0-9]+ <new name> <br />
:Change the name of the power reading powerX to something more meaningful.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter readingNamePowerD4 PowerHaus_kW<br />
</syntaxhighlight> <br />
<br />
;readingStartTime[AD]?[0-9]+ [0|1] <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals. <br />
:This is a hack where the timestamp of readings is artificially set to a past time and may have side effects <br />
:so avoid it unless you fully understand how Fhem works with readings and their time.<br />
<br />
;verboseReadings[AD]?[0-9]+ [0|1] <br />
:create the additional readings lastMsg and pinHistory for each pin<br />
:if verboseReafings is set to 1 for the specified pin.<br />
:If set to -1 then the internal counter, the long counter and interpolated long counter readings will be hidden.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter verboseReadingsD4 1<br />
</syntaxhighlight> <br />
<br />
;enableHistory [0|1]<br />
:tells the counting device to record the individual time of each change at each GPIO pin and send it to Fhem. <br />
:This information is cached on the Fhem side and can be viewed with the command <code>get history</code><br />
:The optput of <code>get history</code> will look like this:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
seq 12627 2020-03-22 20:39:54 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12628 2020-03-22 20:39:55 Pin D5 1.697 seconds at 1 -> gap<br />
seq 12629 2020-03-22 20:39:56 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12630 2020-03-22 20:39:56 Pin D5 1.694 seconds at 1 -> gap<br />
seq 12631 2020-03-22 20:39:58 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12632 2020-03-22 20:39:58 Pin D5 1.693 seconds at 1 -> gap<br />
seq 12633 2020-03-22 20:40:00 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12634 2020-03-22 20:40:00 Pin D5 1.696 seconds at 1 -> gap<br />
seq 12635 2020-03-22 20:40:02 Pin D5 0.081 seconds at 0 -> pulse counted<br />
seq 12636 2020-03-22 20:40:02 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12637 2020-03-22 20:40:03 Pin D5 0.079 seconds at 0 -> pulse counted<br />
seq 12638 2020-03-22 20:40:03 Pin D5 1.700 seconds at 1 -> gap<br />
seq 12639 2020-03-22 20:40:05 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12642 2020-03-22 20:40:05 Pin D5 1.699 seconds at 1 -> gap<br />
seq 12643 2020-03-22 20:40:07 Pin D5 0.080 seconds at 0 -> pulse counted<br />
seq 12644 2020-03-22 20:40:07 Pin D5 1.698 seconds at 1 -> gap<br />
</syntaxhighlight> <br />
<br />
;enableSerialEcho [0|1] <br />
:tells the counting device to show diagnostic data over the serial line when connected via TCP<br />
<br />
;enablePinDebug [0|1] <br />
:tells the counting device to show every level change of the defined input pins over the serial line or via TCP<br />
<br />
;enableAnalogDebug [0|1] <br />
:tells the counting device to show every analog measurement of the defined analog input pins over the serial line or via TCP<br />
<br />
;enableDevTime [0|1] <br />
:tells the counting device to show its internal millis timer so a drift between the devices time and fhem time can be calculated and logged<br />
<br />
;maxHist <max entries> <br />
:specifies how many pin history lines hould be buffered for "get history".<br />
:This attribute defaults to 1000.<br />
<br />
;analogThresholds <br />
:this Attribute is outdated. Please specify the analog thresholds for reflective light barrier input with the attribute "pin..."<br />
<br />
;flashCommand <new shell command> <br />
:overrides the default command to flash the firmware via Wifi (OTA) or serial line. It is recommended to not define this attribute. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter flashCommand avrdude -p atmega328P -c arduino -b 57600 -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
</syntaxhighlight> <br />
:<code> [PORT]</code> is automatically replaced with the serial port for this device as it is specified in the <code>define</code> command.<br />
:<code> [HEXFILE]</code> or <code>[BINFILE]</code> are synonyms and are both automatically replaced with the firmware file appropriate for the device. <br />
:For ESP32 boards <code>[HEXFILE]</code> would be replaced by ArduCounter-8266.bin for example.<br />
:<code> [LOGFILE]</code> is automatically replaced ArduCounterFlash.log in the fhem log subdirectory.<br />
:<code> [NETPORT]</code> is automatically replaced by the tcp port number used for OTA flashing. <br />
:For ESP32 this usually is 3232 and for 8266 Bords it is 8266.<br />
<br />
;keepAliveDelay <delay> <br />
:defines an interval in which the module sends keepalive messages to a counter device that is conected via tcp.<br />
:This attribute is ignored if the device is connected via serial port.<br />
:If the device doesn't reply within a defined timeout then the module closes and tries to reopen the connection.<br />
:The module tells the device when to expect the next keepalive message and the device will also close the tcp connection if it doesn't see a keepalive message within the delay :multiplied by 3<br />
:The delay defaults to 10 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveDelay 30<br />
</syntaxhighlight> <br />
<br />
;keepAliveTimeout <seconds> <br />
:defines the timeout when wainting for a keealive reply (see keepAliveDelay)<br />
:The timeout defaults to 2 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveTimeout 3<br />
</syntaxhighlight> <br />
<br />
;keepAliveRetries <max number of retries> <br />
:defines how often sending a keepalive is retried before the connection is closed and reopened.<br />
:It defaults to 2.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter keepAliveRetries 3<br />
</syntaxhighlight> <br />
<br />
;nextOpenDelay <delay> <br />
:defines the time in seconds that the module waits before retrying to open a disconnected tcp connection. <br />
:This defaults to 60 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter nextOpenDelay 20<br />
</syntaxhighlight> <br />
<br />
;openTimeout <timeout> <br />
:defines the timeout in seconds after which tcp open gives up trying to establish a connection to the counter device.<br />
:This timeout defaults to 3 seconds.<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter openTimeout 5<br />
</syntaxhighlight> <br />
<br />
;silentReconnect [0|1] <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter silentReconnect 1<br />
</syntaxhighlight> <br />
<br />
;deviceDisplay <pin> <unit> <flowUnit> <br />
:controls the unit strings that a local display on the counting device will show. <br />
:Example:<br />
:<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
attr myCounter deviceDisplay 36,l,l/m<br />
attr myCounter deviceDisplay 36,kWh,kW<br />
</syntaxhighlight> <br />
<br />
;disable [0|1] <br />
:if set to 1 then the module is disabled and closes the connection to a counter device.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse. <br />
:This attribute is outdated and unintuitive so you should avoid it. <br />
:Instead you should specify the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;readingFactor[AD]?[0-9]+ <br />
:Override the factor attribute for this individual pin. <br />
:Just like the attribute factor, this is a rather cumbersome way to specify the pulses per kWh. <br />
:Instead it is advised to use the attribute pulsesPerUnit or readingPulsesPerUnit[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;devVerbose <br />
:this attribute is outdated and has been replaced with the attributes <br />
:<syntaxhighlight lang="perl" xstyle="width:80%;">enableHistory, enableSerialEcho, enablePinDebug, enableAnalogDebug, enableDevTime<syntaxhighlight lang="perl" xstyle="width:80%;"><br />
<br />
<br />
== Readings / Events ==<br />
<br />
The module creates at least the following readings and events for each defined pin:<br />
<br />
;calcCounter.* <br />
:This is recommended reading for counting units based on the pulses and the attribute pulsesPerUnit. It is similar to interpolated long count <br />
:which keeps on counting up after fhem restarts but this counter will take the pulses per Unit attribute into the calculation und thus does not <br />
:count pulses but real Units (kWh, liters or some other unit that is applicable)<br />
:The name of this reading can be changed with the attribute readingNameCalcCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
: Another reading with the same name but ending in _i (e.g. calcCounterD4_i) will show how many kWh (or other units) of the above value is interpolated.<br />
<br />
;pin.* e.g. pinD4 <br />
:the current internal count at this pin (internal to the Arduino / ESP device, starts at 0 when the device restarts). <br />
:The name of this reading can be changed with the attribute readingNameCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;long.* e.g. longD5 <br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
:The name of this reading can be changed with the attribute readingNameLongCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;interpolatedLong.* <br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
:The name of this reading can be changed with the attribute readingNameInterpolatedCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;reject.*<br />
:counts rejected pulses that are shorter than the specified minimal pulse length. <br />
<br />
;power.* <br />
:the current calculated power / flow at this pin.<br />
:The name of this reading can be changed with the attribute readingNamePower[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4.<br />
:This reading depends on the attributes pulsesPerUnit as well as readingFlowUnitTime or flowUnitTime for calculation <br />
<br />
;pinHistory.* <br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. Also the total number of impulses recorded here is :limited to 20 for all pins together. The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action.<br />
:-36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
;countDiff.* <br />
:delta of the current count to the last reported one. This is used together with timeDiff.* to calculate the power consumption.<br />
<br />
;timeDiff.* <br />
:time difference between the first pulse in the current observation interval and the last one. Used togehter with countDiff to calculate the power consumption.<br />
<br />
;seq.* <br />
:internal sequence number of the last report from the board to Fhem.<br />
<br />
;runTime.* <br />
:this reading will only be created when the attribute runTime[AD]?[0-9]+ is set for a given pin.<br />
:It contains the time in seconds that the consumption / flow observed at the specified pin has not ben zero.<br />
:If a water meter which outputs 10 impulses per liter on its digital output is for example connected to GPIO pin D6, <br />
:then if the attribute runTimeD6 is set to 1, the reading runTimeD6 will show for how many seconds the water has been flowing without a stop longer than the <br />
:observation interval specifie in the interval-attribute. This is helpful when you want to create alerts in case someone forgot to close a water tap.<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Benutzer:StefanStrobel&diff=33261Benutzer:StefanStrobel2020-05-21T10:25:32Z<p>StefanStrobel: /* Benutzer Stefan Strobel */</p>
<hr />
<div>== Benutzer Stefan Strobel ==<br />
Wikiseiten in Arbeit: <br />
* [[DevelopmentModuleIntro|Einführung in die Entwicklung von Modulen für FHEM]]<br />
* detailliertere Beschreibung von [[HTTPMOD]] (siehe [http://forum.fhem.de/index.php/topic,17804.0.html])<br />
* Beschreibung des generischen Moduls [[ModbusAttr]] für Geräte mit Modbus-Interface. Readings werden wie bei [[HTTPMOD]] per Attr spezifiziert. (basiert auf dem Modul [[Modbus]])<br />
* Beschreibung des generischen Basis-Moduls [[Modbus]] für Geräte mit Modbus-Interface<br />
* Beschreibung des Moduls [[ModbusSET]] für Wärmepumpen der Reihe SET Silent von Schmidt Energie Technik mit Modbus-Interface<br />
* Beschreibung des Moduls FReplacer - z.B. Für die Integration eines [[Kindle_Display|Kindle als Fhem-Display]]<br />
* Doku zum Modul für den Zugriff auf [[Waterkotte_heat_pump_with_Resümat_CD4|Wärempumpen von Waterkotte mit Steuerung Resümat CD4]]<br />
* Ausführlichere Doku zum Modul für Lüftungsanlagen wie Zehnder [[ComfoAir]] und baugleiche Geräte<br />
* Beschreibung des Moduls [[ArduCounter]] (generischer Impulszähler auf Arduino-Basis z.B. für Stromzähler mit S0 Schnittstelle oder Wasserzähler), siehe [[http://forum.fhem.de/index.php/topic,19285.0.html]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=31899ArduCounter2019-12-08T21:39:04Z<p>StefanStrobel: /* Configuration of ArduCounter analog counters */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
This module implements an Interface to an Arduino or ESP8266 based counter for pulses on any input pin of an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1 or similar device. The device connects to Fhem either through USB / serial or via tcp if an ESP board is used.<br />
<br />
The typical use case is an S0-Interface on an energy meter or water meter, but also reflection light barriers to monitor old ferraris counters are supported<br />
<br />
Counters are configured with attributes that define which Arduino pins should count pulses and in which intervals the Arduino board should report the current counts.<br />
<br />
The Arduino sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module creates readings for pulse counts, consumption and optionally also a pin history with pulse lengths and gaps of the last pulses.<br />
<br />
== Availability == <br />
The module has been checked in to the FHEM svn. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1 or similar device based on an Atmel 328p or ESP8266 running the ArduCounter sketch provided with this module<br />
<br />
In order to flash an arduino board with the corresponding ArduCounter firmware from within Fhem, avrdude needs to be installed.<br />
<br />
For old ferraris counters an Arduino Uno or Nano or ESP8266 board needs to be connected to a reflection light barrier which consists simply of an infra red photo transistor (connected to A7 on Arduinos and A0 on ESP8266) and an infra red led (connected to D2 on Arduinos and to D6 on ESP8266), both with a resistor in line.<br />
<br />
This module also requires Device::SerialPort or Win32::SerialPort for communication via serial lines<br />
<br />
== Define ==<br />
<br />
<pre>define <name> ArduCounter <device></pre><br />
or<br />
<pre>define <name> ArduCounter <ip:port></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
<ip:port> specifies the ip address and tcp port to communicate with an esp8266 where port is typically 80.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate for serial connections if the device name contains the @ character, e.g.: /dev/ttyUSB0@38400. The default baudrate of the ArduCounter firmware is 38400 since Version 1.4<br />
<br />
Examples:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2@38400<br />
define AC ArduCounter 192.168.1.134:80<br />
</pre><br />
<br />
== Configuration of ArduCounter digital counters ==<br />
<br />
Specify the pins where impulses should be counted e.g. as attr AC pinX falling pullup 30<br />
<br />
The X in pinX can be an Arduino / ESP pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can use the keywords falling or rising to define if a logical one / 5V (rising) or a logical zero / 0V (falling) should be treated as pulse.<br />
<br />
The optional keyword pullup activates the pullup resistor for the given Pin.<br />
<br />
The last argument is also optional but recommended and specifies a minimal pulse length in milliseconds.<br />
<br />
An energy meter with S0 interface is typically connected to GND and an input pin like D4.<br />
The S0 pulse then pulls the input to 0V.<br />
Since the minimal pulse lenght of the s0 interface is specified to be 30ms, the typical configuration for an s0 interface is<br />
attr AC pinX falling pullup 30<br />
<br />
Specifying a minimal pulse length is recommended since it filters bouncing of reed contacts or other noise.<br />
<br />
Example:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC pulsesPerKWh 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup 5<br />
attr AC pinD5 falling pullup 30<br />
attr AC verboseReadingsD5<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines three counters connected to the pins D4, D5 and D5.<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
For D4 and D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. If this time is longer than the specified 5 or 30 milliseconds then the impulse is counted.<br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
verboseReadings5 causes the module to create additional readings like the pin history which shows length and gaps between the last pulses.<br />
<br />
For pin D6 the arduino does not check pulse lengths and counts every time when the signal changes from 0 to 1.<br />
<br />
The ArduCounter sketch which must be loaded on the Arduino or ESP implements this using pin change interrupts, so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
<br />
== Configuration of ArduCounter analog counters ==<br />
<br />
This module and the corresponding sketch can be used to read out old analog ferraris energy counters. Therefore an ESP, Arduino Uno or Nano board needs to be connected to a reflection light barrier which consists simply of an infra red photo transistor (connected to A7 on Arduinos and A0 on ESP8266) and an infra red led (connected to D2 on Arduinos and to D6 on ESP8266), both with a resistor in line. The idea comes from Martin Kompf (https://www.kompf.de/tech/emeir.html) and has been adopted for ArduCounter to support old ferraris energy counters.<br />
<br />
To support this mode, the sketch has to be compiled with analogIR defined.<br />
<br />
The configuration is then similar to the one for digital counters:<br />
<br />
<pre><br />
define ACF ArduCounter /dev/ttyUSB4<br />
attr ACF analogThresholds 100 110<br />
attr ACF flashCommand avrdude -p atmega328P -b57600 -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
attr ACF interval 60 300 2 2<br />
attr ACF pinA7 rising 20<br />
attr ACF pulsesPerKWh 75<br />
attr ACF stateFormat {sprintf("%.3f kW", ReadingsVal($name,"powerA7",0))}<br />
</pre> <br />
<br />
To find out the right analog thresholds you can set devVerbose to 20 which will ask the firmware of your conting board to report every analog measurement. The ArduCounter module will count how often each value is reported and you can then query these analog level counts with get levels. After a few turns of the ferraris disc the result of get levels might look like this:<br />
<br />
<pre><br />
observed levels from analog input:<br />
94: 21<br />
95: 79<br />
96: 6<br />
97: 2<br />
98: 3<br />
99: 2<br />
100: 2<br />
101: 1<br />
102: 3<br />
105: 2<br />
106: 1<br />
108: 2<br />
109: 1<br />
110: 1<br />
112: 1<br />
113: 3<br />
115: 4<br />
116: 9<br />
117: 14<br />
118: 71<br />
119: 103<br />
120: 118<br />
121: 155<br />
122: 159<br />
123: 143<br />
124: 147<br />
125: 158<br />
126: 198<br />
127: 249<br />
128: 220<br />
129: 230<br />
130: 201<br />
131: 140<br />
132: 147<br />
133: 153<br />
134: 141<br />
135: 119<br />
136: 105<br />
137: 109<br />
138: 114<br />
139: 83<br />
140: 33<br />
141: 14<br />
142: 1 <br />
</pre> <br />
<br />
This shows the measured values together with the frequency how often the individual value has been measured. It is obvious that most measurements result in values between 120 and 135, very few values are betweem 96 and 115 and another peak is around the value 95.<br />
It means that the when the red mark of the ferraris disc is under the sensor, the value is around 95 and while the blank disc is under the sensor, the value is typically between 120 and 135. So a good upper threshold would be 120 and a good lower threshold would be for example 96.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
;levels<br />
:show the count for the measured levels if an analog pin is used to measure e.g. the red mark of a ferraris counter disc. <br />
:This is useful for setting the thresholds for analog measurements.<br />
<br />
;history<br />
:shows details regarding all the level changes that the counter device (Arduino or ESP) has detected <br />
:and how they were used (counted or rejected)<br />
:If get history is issued with a pin name (e.g. get history D5) then only the history entries concerning D5 will be shown.<br />
:This information is sent from the device to Fhem when it reports the current count but only if devVerbose is equal or greater than 5.<br />
:The maximum number of lines that the Arducounter module stores in a ring buffer is defined by the attribute maxHist and defaults to 1000.<br />
<br />
<br />
== Set-Commands ==<br />
<br />
;raw<br />
:send the value to the board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware onto the device. <br />
:This command needs avrdude to be installed. The attribute flashCommand specidies how avrdude is called. <br />
:If it is not modifed then the module sets it to avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. <br />
:So normally there is no need to modify this attribute.<br />
:Depending on your specific Arduino board however, you might need to insert -b 57600 in the flash Command. <br />
:(e.g. for an Arduino Nano) ESP boards so far have to be fashed from the Arduino IDE. In a future version flashing over the air sould be supported.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
;saveConfig<br />
:stores the current interval, analog threshold and pin configuration to be stored in the EEPROM of the counter device <br />
:so it can be retrieved after a reset.<br />
<br />
;enable<br />
:sets the attribute disable to 0<br />
<br />
;disable<br />
:sets the attribute disable to 1<br />
<br />
;reconnect<br />
:closes the tcp connection to an ESP based counter board that is conected via TCP/IP and reopen the connection<br />
<br />
<br />
== Supported readings ==<br />
The module creates at least the following readings and events for each defined pin:<br />
<br />
;pin.* e.g. pinD4<br />
:the current internal count at this pin (internal to the Arduino / ESP device, starts at 0 when the device restarts).<br />
:The name of this reading can be changed with the attribute readingNameCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;long.* e.g. longD5<br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
:The name of this reading can be changed with the attribute readingNameLongCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;interpolatedLong.*<br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
:The name of this reading can be changed with the attribute readingNameInterpolatedCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;calcCounter.*<br />
:similar to long count which keeps on counting up after fhem restarts but this counter will take the pulses per kWh setting into the :calculation und thus not count pulses but real kWh (or some other unit that is applicable)<br />
:The name of this reading can be changed with the attribute readingNameCalcCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;reject.*<br />
:counts rejected pulses that are shorter than the specified minimal pulse length.<br />
<br />
;power.*<br />
:the current calculated power at this pin.<br />
:The name of this reading can be changed with the attribute readingNamePower[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;pinHistory.*<br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. Also the total number of impulses recorded here is limited to 20 for all pins together. The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action. -36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
;countDiff.*<br />
:delta of the current count to the last reported one. This is used together with timeDiff.* to calculate the power consumption.<br />
<br />
;timeDiff.*<br />
:time difference between the first pulse in the current observation interval and the last one. Used togehter with countDiff to calculate the power consumption.<br />
<br />
;seq.*<br />
:internal sequence number of the last report from the board to Fhem.<br />
<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse.<br />
:This attribute is outdated and unintuitive so you should avoid it.<br />
:Instead you should specify the attribute pulsesPerKWh or readingPulsesPerKWh[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;readingFactor[0-9]+<br />
:Override the factor attribute for this individual pin.<br />
:Just like the attribute factor, this is a rather cumbersome way to specify the pulses per kWh.<br />
:Instaed it is advised to use the attribute pulsesPerKWh or readingPulsesPerKWh[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;pulsesPerKWh<br />
:specify the number of pulses that the meter is giving out per unit that sould be displayed (e.g. per kWh energy consumed). For many S0 counters this is 1000, for old ferraris counters this is 75 (rounds per kWh).<br />
:Example: attr myCounter pulsesPerKWh 75<br />
<br />
;readingPulsesPerKWh[0-9]+<br />
:is the same as pulsesPerKWh but specified per pin individually in case you have multiple counters with different settings at the same time<br />
:Example:<br />
:attr myCounter readingPulsesPerKWhA7 75<br />
:attr myCounter readingPulsesPerKWhD4 1000<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNameLongCount[0-9]+ <br />
:Change the name of the counter reading longX to something more meaningful.<br />
<br />
;readingNameInterpolatedCount[0-9]+ <br />
:Change the name of the counter reading InterpolatedLongX to something more meaningful.<br />
<br />
;readingNameCalcCount[AD]?[0-9]+<br />
:Change the name of the real unit counter reading CalcCounterX to something more meaningful.<br />
:Example: attr myCounter readingNameCalcCountD4 CounterHaus_kWh<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
;devVerbose<br />
:set the verbose level in the counting board. This defaults to 0.<br />
:If the value is >0, then the firmware will echo all commands sent to it by the Fhem module.<br />
:If the value is >=5, then the firmware will report the pin history (assuming that the firmware has been compiled with this feature enabled)<br />
:If the value is >=10, then the firmware will report every level change of a pin<br />
:If the value is >=20, then the firmware will report every analog measurement (assuming that the firmware has been compiled with analog measurements for old ferraris counters or similar).<br />
<br />
;maxHist<br />
:specifies how many pin history lines hould be buffered for "get history".<br />
:This attribute defaults to 1000.<br />
<br />
;analogThresholds<br />
:this Attribute is necessary when you use an arduino nano with connected reflection light barrier (photo transistor and led) to detect the red mark of an old ferraris energy counter. In this case the firmware uses an upper and lower threshold which can be set here.<br />
:Example: attr myCounter analogThresholds 90 110<br />
:In order to find out the right threshold values you can set devVerbose to 20, wait for several turns of the ferraris disc and then use get levels to see the typical measurements for the red mark and the blank disc.<br />
<br />
;flashCommand<br />
:sets the command to call avrdude and flash the onnected arduino with an updated hex file (by default it looks for ArduCounter.hex in the FHEM/firmware subdirectory.<br />
:This attribute contains avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE] by default.<br />
:For an Arduino Nano based counter you should add -b 57600 e.g. between the -P and -D options.<br />
:Example: attr myCounter flashCommand avrdude -p atmega328P -c arduino -b 57600 -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
<br />
;keepAliveDelay<br />
:defines an interval in which the module sends keepalive messages to a counter device that is conected via tcp.<br />
:This attribute is ignored if the device is connected via serial port.<br />
:If the device doesn't reply within a defined timeout then the module closes and tries to reopen the connection.<br />
:The module tells the device when to expect the next keepalive message and the device will also close the tcp connection if it doesn't see a :keepalive message within the delay multiplied by 3<br />
:The delay defaults to 10 seconds.<br />
:Example: attr myCounter keepAliveDelay 30<br />
<br />
;keepAliveTimeout<br />
:defines the timeout when wainting for a keealive reply (see keepAliveDelay) The timeout defaults to 2 seconds.<br />
:Example: attr myCounter keepAliveTimeout 3<br />
<br />
;keepAliveRetries<br />
:defines how often sending a keepalive is retried before the connection is closed and reopened.<br />
:It defaults to 2.<br />
:Example: attr myCounter keepAliveRetries 3<br />
<br />
;nextOpenDelay<br />
:defines the time that the module waits before retrying to open a disconnected tcp connection.<br />
:This defaults to 60 seconds.<br />
:Example: attr myCounter nextOpenDelay 20<br />
<br />
;openTimeout<br />
:defines the timeout after which tcp open gives up trying to establish a connection to the counter device. This timeout defaults to 3 seconds.<br />
:Example: attr myCounter openTimeout 5<br />
<br />
;silentReconnect<br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
:Example: attr myCounter silentReconnect 1<br />
<br />
;disable<br />
:if set to 1 then the module closes the connection to a counter device.<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=31709ArduCounter2019-11-16T16:07:58Z<p>StefanStrobel: updated to the current version</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
This module implements an Interface to an Arduino or ESP8266 based counter for pulses on any input pin of an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1 or similar device. The device connects to Fhem either through USB / serial or via tcp if an ESP board is used.<br />
<br />
The typical use case is an S0-Interface on an energy meter or water meter, but also reflection light barriers to monitor old ferraris counters are supported<br />
<br />
Counters are configured with attributes that define which Arduino pins should count pulses and in which intervals the Arduino board should report the current counts.<br />
<br />
The Arduino sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module creates readings for pulse counts, consumption and optionally also a pin history with pulse lengths and gaps of the last pulses.<br />
<br />
== Availability == <br />
The module has been checked in to the FHEM svn. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires an Arduino Uno, Nano, Jeenode, NodeMCU, Wemos D1 or similar device based on an Atmel 328p or ESP8266 running the ArduCounter sketch provided with this module<br />
<br />
In order to flash an arduino board with the corresponding ArduCounter firmware from within Fhem, avrdude needs to be installed.<br />
<br />
For old ferraris counters an Arduino Uno or Nano or ESP8266 board needs to be connected to a reflection light barrier which consists simply of an infra red photo transistor (connected to A7 on Arduinos and A0 on ESP8266) and an infra red led (connected to D2 on Arduinos and to D6 on ESP8266), both with a resistor in line.<br />
<br />
This module also requires Device::SerialPort or Win32::SerialPort for communication via serial lines<br />
<br />
== Define ==<br />
<br />
<pre>define <name> ArduCounter <device></pre><br />
or<br />
<pre>define <name> ArduCounter <ip:port></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
<ip:port> specifies the ip address and tcp port to communicate with an esp8266 where port is typically 80.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate for serial connections if the device name contains the @ character, e.g.: /dev/ttyUSB0@38400. The default baudrate of the ArduCounter firmware is 38400 since Version 1.4<br />
<br />
Examples:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2@38400<br />
define AC ArduCounter 192.168.1.134:80<br />
</pre><br />
<br />
== Configuration of ArduCounter digital counters ==<br />
<br />
Specify the pins where impulses should be counted e.g. as attr AC pinX falling pullup 30<br />
<br />
The X in pinX can be an Arduino / ESP pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can use the keywords falling or rising to define if a logical one / 5V (rising) or a logical zero / 0V (falling) should be treated as pulse.<br />
<br />
The optional keyword pullup activates the pullup resistor for the given Pin.<br />
<br />
The last argument is also optional but recommended and specifies a minimal pulse length in milliseconds.<br />
<br />
An energy meter with S0 interface is typically connected to GND and an input pin like D4.<br />
The S0 pulse then pulls the input to 0V.<br />
Since the minimal pulse lenght of the s0 interface is specified to be 30ms, the typical configuration for an s0 interface is<br />
attr AC pinX falling pullup 30<br />
<br />
Specifying a minimal pulse length is recommended since it filters bouncing of reed contacts or other noise.<br />
<br />
Example:<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC pulsesPerKWh 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup 5<br />
attr AC pinD5 falling pullup 30<br />
attr AC verboseReadingsD5<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines three counters connected to the pins D4, D5 and D5.<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
For D4 and D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. If this time is longer than the specified 5 or 30 milliseconds then the impulse is counted.<br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
verboseReadings5 causes the module to create additional readings like the pin history which shows length and gaps between the last pulses.<br />
<br />
For pin D6 the arduino does not check pulse lengths and counts every time when the signal changes from 0 to 1.<br />
<br />
The ArduCounter sketch which must be loaded on the Arduino or ESP implements this using pin change interrupts, so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
<br />
== Configuration of ArduCounter analog counters ==<br />
<br />
This module and the corresponding sketch can be used to read out old analog ferraris energy counters. Therefore for an Arduino Uno or Nano board (the ESP version does not yet support analog measurements) needs to be connected to a reflection light barrier which consists simply of an infra red photo transistor (connected to A7 on Arduinos and A0 on ESP8266) and an infra red led (connected to D2 on Arduinos and to D6 on ESP8266), both with a resistor in line. The idea comes from Martin Kompf (https://www.kompf.de/tech/emeir.html) and has been adopted for ArduCounter to support old ferraris energy counters.<br />
<br />
To support this mode, the sketch has to be compiled with analogIR defined.<br />
<br />
The configuration is then similar to the one for digital counters:<br />
<br />
<pre><br />
define ACF ArduCounter /dev/ttyUSB4<br />
attr ACF analogThresholds 100 110<br />
attr ACF flashCommand avrdude -p atmega328P -b57600 -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
attr ACF interval 60 300 2 2<br />
attr ACF pinA7 rising 20<br />
attr ACF pulsesPerKWh 75<br />
attr ACF stateFormat {sprintf("%.3f kW", ReadingsVal($name,"powerA7",0))}<br />
</pre> <br />
<br />
To find out the right analog thresholds you can set devVerbose to 20 which will ask the firmware of your conting board to report every analog measurement. The ArduCounter module will count how often each value is reported and you can then query these analog level counts with get levels. After a few turns of the ferraris disc the result of get levels might look like this:<br />
<br />
<pre><br />
observed levels from analog input:<br />
94: 21<br />
95: 79<br />
96: 6<br />
97: 2<br />
98: 3<br />
99: 2<br />
100: 2<br />
101: 1<br />
102: 3<br />
105: 2<br />
106: 1<br />
108: 2<br />
109: 1<br />
110: 1<br />
112: 1<br />
113: 3<br />
115: 4<br />
116: 9<br />
117: 14<br />
118: 71<br />
119: 103<br />
120: 118<br />
121: 155<br />
122: 159<br />
123: 143<br />
124: 147<br />
125: 158<br />
126: 198<br />
127: 249<br />
128: 220<br />
129: 230<br />
130: 201<br />
131: 140<br />
132: 147<br />
133: 153<br />
134: 141<br />
135: 119<br />
136: 105<br />
137: 109<br />
138: 114<br />
139: 83<br />
140: 33<br />
141: 14<br />
142: 1 <br />
</pre> <br />
<br />
This shows the measured values together with the frequency how often the individual value has been measured. It is obvious that most measurements result in values between 120 and 135, very few values are betweem 96 and 115 and another peak is around the value 95.<br />
It means that the when the red mark of the ferraris disc is under the sensor, the value is around 95 and while the blank disc is under the sensor, the value is typically between 120 and 135. So a good upper threshold would be 120 and a good lower threshold would be for example 96.<br />
<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
;levels<br />
:show the count for the measured levels if an analog pin is used to measure e.g. the red mark of a ferraris counter disc. <br />
:This is useful for setting the thresholds for analog measurements.<br />
<br />
;history<br />
:shows details regarding all the level changes that the counter device (Arduino or ESP) has detected <br />
:and how they were used (counted or rejected)<br />
:If get history is issued with a pin name (e.g. get history D5) then only the history entries concerning D5 will be shown.<br />
:This information is sent from the device to Fhem when it reports the current count but only if devVerbose is equal or greater than 5.<br />
:The maximum number of lines that the Arducounter module stores in a ring buffer is defined by the attribute maxHist and defaults to 1000.<br />
<br />
<br />
== Set-Commands ==<br />
<br />
;raw<br />
:send the value to the board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware onto the device. <br />
:This command needs avrdude to be installed. The attribute flashCommand specidies how avrdude is called. <br />
:If it is not modifed then the module sets it to avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. <br />
:So normally there is no need to modify this attribute.<br />
:Depending on your specific Arduino board however, you might need to insert -b 57600 in the flash Command. <br />
:(e.g. for an Arduino Nano) ESP boards so far have to be fashed from the Arduino IDE. In a future version flashing over the air sould be supported.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
;saveConfig<br />
:stores the current interval, analog threshold and pin configuration to be stored in the EEPROM of the counter device <br />
:so it can be retrieved after a reset.<br />
<br />
;enable<br />
:sets the attribute disable to 0<br />
<br />
;disable<br />
:sets the attribute disable to 1<br />
<br />
;reconnect<br />
:closes the tcp connection to an ESP based counter board that is conected via TCP/IP and reopen the connection<br />
<br />
<br />
== Supported readings ==<br />
The module creates at least the following readings and events for each defined pin:<br />
<br />
;pin.* e.g. pinD4<br />
:the current internal count at this pin (internal to the Arduino / ESP device, starts at 0 when the device restarts).<br />
:The name of this reading can be changed with the attribute readingNameCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;long.* e.g. longD5<br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
:The name of this reading can be changed with the attribute readingNameLongCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;interpolatedLong.*<br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
:The name of this reading can be changed with the attribute readingNameInterpolatedCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;calcCounter.*<br />
:similar to long count which keeps on counting up after fhem restarts but this counter will take the pulses per kWh setting into the :calculation und thus not count pulses but real kWh (or some other unit that is applicable)<br />
:The name of this reading can be changed with the attribute readingNameCalcCount[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;reject.*<br />
:counts rejected pulses that are shorter than the specified minimal pulse length.<br />
<br />
;power.*<br />
:the current calculated power at this pin.<br />
:The name of this reading can be changed with the attribute readingNamePower[AD]?[0-9]+ where [AD]?[0-9]+ stands for the pin description e.g. D4<br />
<br />
;pinHistory.*<br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. Also the total number of impulses recorded here is limited to 20 for all pins together. The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action. -36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
;countDiff.*<br />
:delta of the current count to the last reported one. This is used together with timeDiff.* to calculate the power consumption.<br />
<br />
;timeDiff.*<br />
:time difference between the first pulse in the current observation interval and the last one. Used togehter with countDiff to calculate the power consumption.<br />
<br />
;seq.*<br />
:internal sequence number of the last report from the board to Fhem.<br />
<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse.<br />
:This attribute is outdated and unintuitive so you should avoid it.<br />
:Instead you should specify the attribute pulsesPerKWh or readingPulsesPerKWh[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;readingFactor[0-9]+<br />
:Override the factor attribute for this individual pin.<br />
:Just like the attribute factor, this is a rather cumbersome way to specify the pulses per kWh.<br />
:Instaed it is advised to use the attribute pulsesPerKWh or readingPulsesPerKWh[0-9]+ (where [0-9]+ stands for the pin number).<br />
<br />
;pulsesPerKWh<br />
:specify the number of pulses that the meter is giving out per unit that sould be displayed (e.g. per kWh energy consumed). For many S0 counters this is 1000, for old ferraris counters this is 75 (rounds per kWh).<br />
:Example: attr myCounter pulsesPerKWh 75<br />
<br />
;readingPulsesPerKWh[0-9]+<br />
:is the same as pulsesPerKWh but specified per pin individually in case you have multiple counters with different settings at the same time<br />
:Example:<br />
:attr myCounter readingPulsesPerKWhA7 75<br />
:attr myCounter readingPulsesPerKWhD4 1000<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNameLongCount[0-9]+ <br />
:Change the name of the counter reading longX to something more meaningful.<br />
<br />
;readingNameInterpolatedCount[0-9]+ <br />
:Change the name of the counter reading InterpolatedLongX to something more meaningful.<br />
<br />
;readingNameCalcCount[AD]?[0-9]+<br />
:Change the name of the real unit counter reading CalcCounterX to something more meaningful.<br />
:Example: attr myCounter readingNameCalcCountD4 CounterHaus_kWh<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
;devVerbose<br />
:set the verbose level in the counting board. This defaults to 0.<br />
:If the value is >0, then the firmware will echo all commands sent to it by the Fhem module.<br />
:If the value is >=5, then the firmware will report the pin history (assuming that the firmware has been compiled with this feature enabled)<br />
:If the value is >=10, then the firmware will report every level change of a pin<br />
:If the value is >=20, then the firmware will report every analog measurement (assuming that the firmware has been compiled with analog measurements for old ferraris counters or similar).<br />
<br />
;maxHist<br />
:specifies how many pin history lines hould be buffered for "get history".<br />
:This attribute defaults to 1000.<br />
<br />
;analogThresholds<br />
:this Attribute is necessary when you use an arduino nano with connected reflection light barrier (photo transistor and led) to detect the red mark of an old ferraris energy counter. In this case the firmware uses an upper and lower threshold which can be set here.<br />
:Example: attr myCounter analogThresholds 90 110<br />
:In order to find out the right threshold values you can set devVerbose to 20, wait for several turns of the ferraris disc and then use get levels to see the typical measurements for the red mark and the blank disc.<br />
<br />
;flashCommand<br />
:sets the command to call avrdude and flash the onnected arduino with an updated hex file (by default it looks for ArduCounter.hex in the FHEM/firmware subdirectory.<br />
:This attribute contains avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE] by default.<br />
:For an Arduino Nano based counter you should add -b 57600 e.g. between the -P and -D options.<br />
:Example: attr myCounter flashCommand avrdude -p atmega328P -c arduino -b 57600 -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
<br />
;keepAliveDelay<br />
:defines an interval in which the module sends keepalive messages to a counter device that is conected via tcp.<br />
:This attribute is ignored if the device is connected via serial port.<br />
:If the device doesn't reply within a defined timeout then the module closes and tries to reopen the connection.<br />
:The module tells the device when to expect the next keepalive message and the device will also close the tcp connection if it doesn't see a :keepalive message within the delay multiplied by 3<br />
:The delay defaults to 10 seconds.<br />
:Example: attr myCounter keepAliveDelay 30<br />
<br />
;keepAliveTimeout<br />
:defines the timeout when wainting for a keealive reply (see keepAliveDelay) The timeout defaults to 2 seconds.<br />
:Example: attr myCounter keepAliveTimeout 3<br />
<br />
;keepAliveRetries<br />
:defines how often sending a keepalive is retried before the connection is closed and reopened.<br />
:It defaults to 2.<br />
:Example: attr myCounter keepAliveRetries 3<br />
<br />
;nextOpenDelay<br />
:defines the time that the module waits before retrying to open a disconnected tcp connection.<br />
:This defaults to 60 seconds.<br />
:Example: attr myCounter nextOpenDelay 20<br />
<br />
;openTimeout<br />
:defines the timeout after which tcp open gives up trying to establish a connection to the counter device. This timeout defaults to 3 seconds.<br />
:Example: attr myCounter openTimeout 5<br />
<br />
;silentReconnect<br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
:Example: attr myCounter silentReconnect 1<br />
<br />
;disable<br />
:if set to 1 then the module closes the connection to a counter device.<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Kindle_Display&diff=31365Kindle Display2019-10-16T19:40:31Z<p>StefanStrobel: updated man things</p>
<hr />
<div>This article shows how to configure an Amazon Kindle eBook-Reader as a information display for Fhem. <br />
Since such eBook-Readers use eInk displays which keep their display even when the device is completely switched off, <br />
this offers a nice method to visualize Fhem devices and readings on a battery powered device which is mainly in deep sleep mode and<br />
only powers up to fetch new data from Fhem and update the display e.g. every 10 minutes.<br />
<br />
Use this manual at your own risk. Jailbreaking a Kindle is quite easy but there is no guarantee for breaking the system.<br><br />
I will show how to configure it with the FileReplacer.pm from Stefan. The "old" method with the special function in MyUtils is not part of this article.<br />
<br />
All of the information shown below is collected from the Fhem-forum {{Link2Forum|Topic=21821|LinkText=fhem-Forum}}. <br />
Many thanks to everybody who participated on the development of this information to get it done.<br />
Special thanks to "alex" and "Stefan" from the board. Further information is from the [http://www.mobileread.com mobileread board].<br />
<br />
=Hardware requirements=<br />
==Kindle Touch / Paperwhite / Paperwhite 2==<br />
You can use Kindle Touch / Paperwhite / Paperwhite 2 without any limitations. <br />
==Kindle 4 or Kindle 5==<br />
For setting this up on the Kindle 4 (graphite color and with the 5-way button) or Kindle 5 (black and with the 5-way button) you need some more configuration but it's also working. Addidional tasks for this version are shown down below.<br />
<br />
=Prerequisites=<br />
==fhem and server configuration==<br />
===Server configuration===<br />
The Fhem side that provides the image to display an a Kindle device typically creates this image by replacing placeholders in an SVG-file and the converting the result into a suitable PNG-file. For this conversion you need imagemagick or some other command line utility to convert SVG into PNG in the right format.<br />
:<code>apt-get install imagemagick</code><br />
===fhem configuration===<br />
First you need an SVG-template. You can draw one e.g. with inkscape or you can find one in the first posting on the {{Link2Forum|Topic=21821|LinkText=fhem-Forum}} thread. <br />
In the template just replace the data with some placeholder you like. Later this placeholder is found via a regular expression and replaced with your data.<br />
For example, you can set XYZ at the place where the first temperature should be shown and ABC for the first humidity value.<br />
If you want to edit the template you can use the free software [www.inkscape.org/de/ Inkscape] for example. <br />
For more information about generating SVGs for Fhem please refer to this article [[Icons]]. <br />
The most important information is to save the SVG as a "normal SVG" and not as an inkscape SVG.<br />
For the Kindle Paperwhite the resolution of the SVG must be 758 x 1024 for the Kindle 4 and 5 version use 600x800. This is mandatory. <br />
Wrong scaled images won't be displayed on the Kindle.<br> <br />
Copy the Kindle_Template.svg to <nowiki>/opt/fhem/www/images/</nowiki>.<br />
<br />
All we're doing now is using the FReplacer module which is part of Fhem to read the SVG template, replace a search pattern with our data and save it.<br />
<br />
:<code>define kindledisplay FReplacer /opt/fhem/www/images/template1.svg /opt/fhem/www/images/status1.svg 60</code><br />
:<code>attr kindledisplay PostCommand convert /opt/fhem/www/images/status.svg -type GrayScale -depth 8 /opt/fhem/www/images/status.png 2>/dev/null & </code><br />
:<code>attr kindledisplay ReplacementEncode UTF-8</code><br />
:<code>attr kindledisplay room Display</code><br />
<br />
PostCommand defines that the external command will be issued after the replacement to convert the resulting SVG to a PNG in 8 bit GreyScale mode. <br />
UTF8-Encode might be needed to support special characters.<br />
<br />
<br />
Now just create your mappings:<br />
:<code>attr kindledisplay Rep01Regex XYZ </code><br />
:<code>attr kindledisplay Rep01Reading Sensor1:Temp1</code><br />
:<code>attr kindledisplay Rep01Format %.1f </code><br />
:<code>attr kindledisplay Rep02Regex ABC </code><br />
:<code>attr kindledisplay Rep02Reading Sensor1:Hum1</code><br />
:<code>attr kindledisplay Rep01Format %.1f </code><br />
And so on... :) The FReplacer module will replace the pattern XYZ with the ReadingsVal from Sensor1.<br />
<br />
More and up to date details about the configuration of the FReplacer module can be found in the Fhem command reference [https://fhem.de/commandref.html#FReplacer]<br />
<br />
Now check if the PNG is created [http://your-fhem-ip:8083/fhem/www/images/status1.png http://your-fhem-ip:8083/fhem/www/images/KindleDisplay.png]. If so, be happy, the first part is done!<br />
<br />
If you want to use more complex Perl expressions to compute the replacement string, you can use the RepExpr attribute instead of RepReading and do someting like this:<br />
:<code>attr kindledisplay Rep11Expr (ReadingsVal("XY", "Reading", "off") eq "on" ? "An" : "Aus")</code><br />
<br />
If you want to add a "last modified" timestamp to the dispaly, create a placeholder for it in your SVG template (like "last modified lmtime123") and then define a Regex / Reading attribute pair to replace it with the Reading "LastUpdate" from your FReplacer device:<br />
:<code>attr kindledisplay Rep14Regex lmtime123</code><br />
:<code>attr kindledisplay Rep14Reading kindledisplay:LastUpdate:never</code><br />
<br />
=== Templates ===<br />
You can find my templates here: [http://forum.fhem.de/index.php?topic=21821.msg202396#msg202396 fhem Forum]. Feel free to use and modify.<br />
<br />
=Modifying the Kindle=<br />
To prevent from error messages like "your Kindle is no longer registered as a test Kindle..." please make sure your Kindle is connected to a valid Amazon Account. <br><br />
We need to download some software to install it on the Kindle.<br><br />
Please download it from [http://www.mobileread.com/forums/showthread.php?t=88004 Mobileread Forums]<br><br />
* Kindle Jailbreak [http://www.mobileread.com/forums/showthread.php?t=88004 Mobileread Forums]<br />
* MKK Mobileread Kindlet Kit [http://www.mobileread.com/forums/showthread.php?t=233932 Mobileread Forums]<br />
* KUAL [http://www.mobileread.com/forums/showthread.php?t=203326 Mobileread Forums]<br />
* Screensavers Hack [http://www.mobileread.com/forums/showthread.php?t=88004 Mobileread Forums]<br />
* OnlineScreenSaver [http://www.mobileread.com/forums/showthread.php?t=236104 Mobileread Forums]<br />
<br />
* for Kindle 4 or 5 only: USBNETWORK [http://www.mobileread.com/forums/showthread.php?t=88004 also Mobileread Forums]<br />
<br> Always use the newest version and look for one who fits to your Kindle (e.G. K4 or K5).<br />
<br />
==Jailbreaking the Kindle==<br />
Don't worry. This sounds more hard and dangerous than it is in fact. It's just a few steps to have full access to your Kindle.<br />
=== Kindle Touch/Paperwhite ===<br />
* Download the kindle-jailbreak-0.12.N.zip file, and unpack it. In there, you'll find some .bin files, and a src directory.Leave the directory alone, and upload the correct Update_*_install.bin file for your kindle & FW version to the root directory of your Kindle.<br />
<br><br />
* Now, eject & unplug your Kindle, and go to [HOME] -> [MENU] > Settings -> [MENU] > Update Your Kindle. It should be quick.<br> (And, on FW 2.x only, it should FAIL (With a U006 error, in the bottom left corner of the screen). It's completely normal, intended, and harmless).<br />
<br><br />
And that's it, your Kindle is now ready to install custom hacks!<br><br />
(Quoted from http://www.mobileread.com/forums/showthread.php?t=88004)<br />
<br />
=== Kindle 4 or 5 / Universal method ===<br />
<br />
* Download and unzip the jailbreak. <br />
* Plug in the Kindle and copy the data.tar.gz & ENABLE_DIAGS files plus the diagnostic_logs folder to the Kindle's USB drive's root<br />
* Safely remove the USB cable and restart the Kindle (Menu -> Settings -> Menu -> Restart)<br />
* Once the device restarts into diagnostics mode, select "D) Exit, Reboot or Disable Diags" (using the 5-way keypad)<br />
* Select "R) Reboot System" and "Q) To continue" (following on-screen instructions, when it tells you to use 'FW Left' to select an option, it means left on the 5-way keypad)<br />
* Wait about 20 seconds: you should see the Jailbreak screen for a while, and the device should then restart normally<br />
* After the Kindle restarts, you should see a new book titled "You are Jailbroken", if you see this, the jailbreak has been successful.<br />
<br />
==Installing additional software==<br />
<br />
=== MKK ===<br />
The Mobileread Kindlet Kit (MKK) is used to execute custom Kindlets (programs) on the Kindle.<br><br />
Download the software from here: [http://www.mobileread.com/forums/showthread.php?t=233932 MKK]<br><br />
Unzip the folder and copy the .bin file to the root directory of your kindle. Please note that you need to install the Kindle matching version (e.g. K4 = Kindle 4 with 5-way-button).<br><br />
On the Kindle goto Menu -> Settings -> Menu -> Update your Kindle.<br><br />
After the update it is installed.<br />
<br />
=== KUAL ===<br />
The Kindle Unified Application Launcher (KUAL) running applications on the Kindle.<br><br />
Download it from here: [http://www.mobileread.com/forums/showthread.php?t=203326 KUAL]<br><br />
Unzip the folder and copy the matching .azw2 file to the documents directory of your kindle.<br><br />
<br><br />
(K2, DX, K3, K4, K5)<br><br />
Put KUAL-KDK-1.0.azw2 in documents folder.<br><br />
Run it by clicking new kindlet (book) document in your list.<br><br />
<br><br />
(Touch, PaperWhite)<br><br />
Put KUAL-KDK-2.0.azw2 in documents folder.<br><br />
Run it by clicking icon.<br><br />
<br />
=== Backdoorlock ===<br />
This is used to prevent from getting automated updates which could remove the JailBreak.<br><br />
Download from here: [http://www.mobileread.com/forums/showthread.php?t=205666 Backdoorlock]<br><br />
Unzip the folder and copy the .bin file to the root directory of your kindle. Please note that you need to install the Kindle matching version (e.g. K4 = Kindle 4 with 5-way-button).<br />
On the Kindle goto Menu -> Settings -> Menu -> Update your Kindle.<br />
After the update it is installed.<br />
<br />
=== ScreenSavers Hack ===<br />
The ScreenSavers Hack let you show custom screen savers on the kindle.<br><br />
Download the file: [http://www.mobileread.com/forums/showthread.php?t=88004 ScreenSaver Hack]<br><br />
Install the corresponding .bin file as the others before and use the "update your kindle" function to install it.<br><br />
After the installation you will find a directory called "linkss" in the Kindle root. Here open the screensavers dir and delete all images.<br />
<br />
=== Online ScreenSaver ===<br />
This application copies the PNG image file created on the Fhem side to the ScreenSavers hack so it is displayed when the Kindle start sleeping.<br />
Just take the modified version for Kindle 4 and 5 from [https://forum.fhem.de/index.php/topic,21821.585.html] or download the original application for later Kindles from <br />
[http://www.mobileread.com/forums/showthread.php?t=236104 here], extract the file and copy the folder into the extensions directory.<br />
Now you have to edit the config.sh file up to your need.<br><br />
'''NOTE''' Use a linux/unix compatible editor (like notepad++) only!<br><br />
You need to enter the image URL, the update interval and the schedule for the updates.<br />
<br />
The onlinescreensaver contains a script called enable.sh in its bin subdirectory.<br />
This should be invoked manually if you are logged into the kindle via ssh or via KUAL from the Kindles menue.<br />
enable.sh takes care that the scheduler of onlinescreensaver is started (also after reboot).<br />
<br />
<br />
==Kindle 4 and 5 differences compared to later models ==<br />
<br />
A Kindle 4 and 5 differs in a few ways from the later models. It doesn't use upstart but old start up scripts. The enable.sh script takes care that the appropriate startup skript is copied in the right place.<br />
<br />
Another difference is the power management. The original online screensave was created for a kindle paperwhite which has a real time clock with alarm functions that can be controlled through the sysfs. Utils.sh in the online screensavers bin directory puts the next wake up time in /sys/class/rtc/rtc$RTC/wakealarm to achieve this. So the Paperwhite can sleep most of the time and only wake up to retrieve the latest PNG, display it and go back to sleep. <br />
<br />
The modified version of onlinescreensaver for Kindle 4 and 5 that can be downloaded from the Fhem forum [https://forum.fhem.de/index.php/topic,21821.585.html] therefore contains an rtcwake binary that puts the Kindle into deep sleep and wakes it up again with the built in real time clock after a defined time. <br />
<br />
As your Kindle 4 or 5 needs some more attention to get it done, please find the steps here:<br />
<br />
=== USB Network Hack ===<br />
This hack is used to get SSH/Telnet Access to your Kindle. We need this on Kindle 4 to change some settings on the system.<br><br />
Download and install the kindle-usbnetwork file from here: [http://www.mobileread.com/forums/showthread.php?t=88004 USBNETWORK]<br><br />
I'm sure you know what to do in the meantime ;-)<br><br />
After that, activate the USBNetwork via KUAL and connect the Kindle via USB.<br />
<br />
=== Connecting to the Kindle ===<br />
Before you can connect you need to install the RNDIS driver. [https://github.com/ev3dev/ev3dev/wiki/Setting-Up-Windows-USB-Ethernet-Networking Here you can find] a very good "HowTo" to do this. Note that the IP-Addresses you need to configure are different then addresses used in the example. Your RNDIS Interface needs to have 192.168.15.201 configured, the Kindle uses 192.168.15.244 on this interface.<br><br />
Don't wonder you cannot ping the Kindle!<br><br />
Now you can use a ssh-client like putty to connect to the Kindle. Telnet is also an option but only when WiFI mode is disabled. So better forget about it. Success? Great job!<br />
<br />
=== Changing the "sleep settings" to keep the Kindle awake ===<br />
The modified version of onlinescreensaver for K4 and K5 does this automatically when the above mentioned enable-sh script is run. <br />
For the original onlinescreensaver however this is a manual task:<br />
<br />
After you connected to the Kindle you will be asked for a password. Login with root and blank password. Should work.<br><br />
Now we need to make the filesystem writeable:<br />
:<code>mntroot rw</code><br />
Changing the root password:<br />
:<code>passwd</code><br />
Changing the timeouts:<br />
:<code>cd /etc/kdb.src/yoshi/system/daemon/powerd/</code><br />
Edit the t1_timeout (time before the secreensaver gets displayed) with the editor "vi" and change the last line to 200.<br><br />
Same for the t2_timeout but here we enter 200000 in the last line to keep the Kindle awake for at least 2 days. <br><br />
These values are setting the "sleep" counters on the Kindle. Unfortunately this will lower the battery faster then normal so I recommend to have the display in a docking station.<br><br />
<br />
=== Cron setting (optional) ===<br />
In some cases that all didn't really work. In such cases the onlinescreensaver update script can also be called via cron:<br />
:<code>vi /etc/crontab/root</code><br />
Add the following line at the bottom. */5 means every 5 minutes.<br />
:<code>*/5 * * * * /mnt/us/extensions/onlinescreensaver/bin/update.sh</code><br />
:<code>/etc/init.d/cron restart</code><br />
Setting back filesystem in read-only mode<br />
:<code>mntroot ro</code><br />
That's it for me. Not working? Ask and look here for help: {{Link2Forum|Topic=21821|LinkText=FHEM Forum}}<br />
<br />
=Examples and Pictures=<br />
[[Datei:KindleDisplay.jpg|200px]] Kindle Display Example on Kindle 4<br />
----<br />
<br />
[[Kategorie:HOWTOS]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ModbusAttr&diff=28136ModbusAttr2018-10-21T14:34:06Z<p>StefanStrobel: Für die neue Version des Moduls aktualisiert (mit Slave, Relay und passive listening Features)</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=ModbusAttr<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=25315<br />
|ModTechName=98_ModbusAttr.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
[[ModbusAttr]] uses the low level Modbus module 98_[[Modbus]].pm to provide a generic Modbus module (as master, slave, relay or passive listener) <br />
that can be configured by attributes similar to the way HTTPMOD works for devices with a web interface.<br />
<br />
ModbusAttr can be used as a Modbus master that queries data from other devices over a serial RS232 / RS485 or TCP connection, <br />
it can be used as a Modbus slave that can make readings of Fhem devices available via Modbus to external Modbus masters,<br />
it can act as a Modbus relay that receives requests over one connection and forwards them over another connection (e.g. from Modbus TCP to serial Modbus RTU)<br />
or it can passively listen to other devices that communicate over a serial RS485 connection and extract readings from the objects it sees.<br />
The supported protocols are Modbus RTU, Modbus ASCII or Modbus TCP.<br />
<br />
There are several attributes that modify the way data objects are converted before they are stored in readings or sent to a device. Data can be modified by a perl expression defined in an atribute, formatted with a format string defined in another attribute or mapped to a table defined in an attribute.<br />
<br />
Readings can directly correspond to one data object or they can span several objects. A float value for example might be stored in two input or holding registers in the Modbus device. By specifying attributes that define the length of a reading in objects and by specifying the unpack code to get from a raw string to perl variables, all these cases can be described by attributes and no perl coding is necessary.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module uses the [[Modbus|Modbus base module 98_Modbus.pm]] which uses the DevIO module<br />
<br />
== Define as Modbus master == <br />
<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate,bits,parity,stop<br />
define <name> ModbusAttr <Id> <Interval> <RTU|ASCII><br />
</pre><br />
or<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|ASCII|TCP><br />
</pre><br />
<br />
In the first case the module connects to the external Modbus device with Modbus Id <Id> through a serial modbus device (RS232 or RS485).<br />
<br />
In the second case it connects directly through Modbus TCP or Modbus RTU or ASCII over TCP.<br />
<br />
If <Interval> is not 0 then the module actively requests data from that device every <Interval> seconds.<br />
<br />
The objects that the module should request and the readings it should create from these objects have to be defined with attributes (see below). <br />
These attributes will define a mapping from so called "coils", "digital inputs", "input registers" or "holding registers" of the external device to readings inside Fhem together with the data type and format of the values.<br />
Interval can be 0 in which case the Module only requests data when it is triggered with a Fhem get-Command.<br />
With this mode a Fhem installation can for example query sensor data from a heating system, energy meter or solar power installation if these systems offer a Modbus interface.<br />
<br />
Examples:<br />
<br />
<pre><br />
define ModbusLine Modbus /dev/ttyUSB1@9600<br />
define WP ModbusAttr 1 60<br />
</pre><br />
Define WP as a Modbus master that communicates through the Modbus serial interface device named ModbusLine. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define ModbusLine Modbus /dev/ttyUSB1@9600<br />
define WP ModbusAttr 20 0 ASCII<br />
</pre><br />
Define WP as a Modbus master that communicates through the Modbus serial interface device named ModbusLine with Modbus ASCII. <br />
Use Modbus Id 20 and don't query the device in a defined interval. Instead individual SET / GET options have to be used for communication.<br />
<br />
<pre><br />
define WP ModbusAttr 5 60 192.168.1.122:502 TCP<br />
</pre><br />
to talk Modbus TCP to a device with IP-Address 192.168.1.122 and the reserved port for Modbus TCP 502<br />
Note that for Modbus over a TCP connection you don't need a basic Modbus device for the interface like ModbusLine above. <br />
<br />
<pre><br />
define WP ModbusAttr 3 60 192.168.1.122:8000 RTU<br />
</pre><br />
to talk Modbus RTU over TCP and use the port number 8000<br />
<br />
== Define as Modbus slave == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> slave<br />
</pre><br />
or<br />
<pre><br />
define <name> ModbusAttr <Id> slave <Address:Port> <RTU|ASCII|TCP><br />
</pre><br />
<br />
The module waits for connections from other Modbus masters. It will respond to their requests if the requests contain the given Modbus <Id><br />
To provide data with Modbus to external Modbus masters a mapping needs to be defined using attributes. <br />
These attributes will define a mapping from Readings inside Fhem to so called "coils", "digital inputs", "input registers" or "holding registers" and their Modbus object address together with the data type and format of the values.<br />
With this mode a Fhem installation can for example supply data to a PLC that actively reads data from Fhem or writes data to Fhem readings.<br />
<br />
Examples:<br />
<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 1 slave<br />
</pre><br />
Define Data4PLC as a Modbus slave that communicates through the Modbus serial interface device named MRS485 to listen for Modbus requests with Id 1. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 20 slave ASCII<br />
</pre><br />
to listen for Modbus requests with Id 20 with Modbus ASCII. <br />
<br />
<pre><br />
define Data4PLC ModbusAttr 5 slave 192.168.1.2:502 TCP<br />
</pre><br />
to start listening to TCP port 502 on the local address 192.168.1.2. Modbus TCP will be used as protocol and Requests with Modbus Id 5 will be answered.<br />
<br />
Please be aware that opening a port number smaller than 1024 needs root permissions on Unix devices. So it is probably better to use a non standard port number above 1024 instead.<br />
<br />
<pre><br />
define Data4PLC ModbusAttr 3 slave 192.168.1.2:8000 RTU<br />
</pre><br />
to listen to the local port 8000 and talk Modbus RTU over TCP<br />
<br />
== Define as Modbus passive listener == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> passive <RTU|ASCII|TCP><br />
</pre><br />
<br />
The module listens on a serial (RS485) connection for modbus communication with the given Modbus <Id> and extracts readings. It does not send requests by itself but waits for another master to communicate with a slave. So only objects that the other master requests can be seen by Fhem in this configuration. <br />
The objects that the module recognizes and the readings that it should create from these objects have to be defined with attributes (see below) in the same way as for a Modbus master. <br />
These attributes will define a mapping from so called "coils", "digital inputs", "input registers" or "holding registers" of the external device to readings inside Fhem together with the data type and format of the values.<br />
With this mode a Fhem installation can for example Listen to the communication between an energy counter as slave and a solar control system as master if they use Modbus RTU over RS485. Since only one Master is allowed when using Modbus over serial lines, Fhem can not be master itself. As a passive listener it can however see when the master queries e.g. the current power consumption and then also see the reply from the energy meter and store the value in a Fhem reading.<br />
<br />
Examples:<br />
<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define WP ModbusAttr 1 passive<br />
</pre><br />
to passively listen for Modbus requests and replies with Id 1 over a serial interface managed by an already defined basic modbus device named MB-485. The protocol defaults to Modbus RTU<br />
<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define WP ModbusAttr 20 passive ASCII<br />
</pre><br />
to passivel listen for Modbus requests / replies with Id 20 and Modbus ASCII. <br />
<br />
== Define as Modbus relay == <br />
<br />
<pre><br />
define <name> ModbusAttr <Id> relay to <FhemMasterDevice><br />
</pre><br />
or<br />
<pre><br />
define <name> ModbusAttr <Id> relay <Address:Port> <RTU|ASCII|TCP> to <FhemMasterDevice><br />
</pre><br />
<br />
The module waits for connections from other Modbus masters. It will forward requests if they match the given Modbus <Id> to an already defined Modbus Master device inside Fhem which will send them to its defined slave, take the reply and the pass it back to the original Master.<br />
With this mode a Fhem installation can for example be used in front of a device that only speaks Modbus RTU over RS485 to make it available via Modbus TCP over the local network. <br />
<br />
Examples:<br />
<br />
<pre><br />
define MB-485 Modbus /dev/ttyUSB2<br />
define Heating ModbusAttr 22 0<br />
define Relay ModbusAttr 33 relay 192.168.1.2:1502 TCP to Heating<br />
</pre><br />
Defines MB-485 as a base device for the RS-485 communication with a heating system, <br />
defines Heating as a Modbus Master to communicate with the Heating and its Modbus ID 22, <br />
and then defines the relay which listens to the local IP address 192.168.1.2, TCP port 1502, Modbus Id 33 and protocol Modbus-TCP.<br />
Requests coming in through Modbus TCP and port 1502 are then translated to Modbus RTU and forwarded via RS-485 to the heating system with Modbus Id 22. <br />
<br />
or (unlikely)<br />
<pre><br />
define MB-232 Modbus /dev/ttyUSB2@19200<br />
define Solar ModbusAttr 7 0 192.168.1.122:502 RTU<br />
define PLC2NetRelay ModbusAttr 1 ASCII relay to Solar<br />
</pre><br />
Defines MB-232 as a base device for the RS-232 communication with a PLC as Modbus master, <br />
defines Solar as a Modbus Master to communicate with Modbus TCP to a Solar power system at IP Adrress 192.168.1.122 and its Modbus ID 7, <br />
and then defines the PLC2NetRelay as a relay which listens to Modbus-ASCII requests over the serial RS-232 link from a PLC to Modbus ID 1.<br />
Requests to Modbus Id 1 coming in through the serial link are then translated to Modbus TCP and forwarded over the network to the solar power system with Modbus Id 7. <br />
<br />
== Configuration of the module as master or passive listener == <br />
<br />
Data objects (holding registers, input registers, coils or discrete inputs) are defined using attributes. <br />
If Fhem is Modbus master or passive listener, the attributes assign data objects of external devices (heating systems, power meters, PLCs or other) with their register addresses to readings inside fhem and control how these readings are calculated from the raw values and how they are formatted.<br />
Please be aware that Modbus does not define common data types so the representation of a value can be very different from device to device. One device might make a temperature value avaliable as a floating point value that is stored in two holding resgisters, another device might store the temperature multiplied with 10 as an signed integer in one register. Even the order of bytes can vary.<br />
Therefore it is typically necessary to specify the data representation as a Perl unpack code.<br />
A Modbus master can also write values to Objects in the device and attributes define how this is done.<br />
<br />
Example for a Modbus master or passive configuration:<br />
<pre><br />
define PWP ModbusAttr 5 30<br />
attr PWP obj-h256-reading Temp_Wasser_ein<br />
attr PWP obj-h256-expr $val/10<br />
<br />
attr PWP obj-h258-reading Temp_Wasser_Aus<br />
attr PWP obj-h258-expr $val/10<br />
<br />
attr PWP obj-h262-reading Temp_Luft<br />
attr PWP obj-h262-expr $val / 10<br />
<br />
attr PWP obj-h770-reading Temp_Soll<br />
attr PWP obj-h770-expr $val / 10<br />
attr PWP obj-h770-set 1<br />
attr PWP obj-h770-setexpr $val * 10<br />
attr PWP obj-h770-max 32<br />
attr PWP obj-h770-min 10<br />
attr PWP obj-h770-hint 8,10,20,25,28,29,30,30.5,31,31.5,32<br />
<br />
attr PWP dev-h-combine 5<br />
attr PWP dev-h-defPoll 1<br />
attr PWP dev-h-defUnpack n<br />
<br />
attr PWP room Pool-WP<br />
attr PWP stateFormat {sprintf("%.1f Grad", ReadingsVal($name,"Temp_Wasser_Ein",0))}<br />
attr PWP webCmd Temp_Soll<br />
</pre><br />
<br />
Attributes to define data objects start with obj- followed by a code that identifies the type and address<br />
of the data object. <br />
<br />
Modbus devices offer the following types of data objects: <br />
<br />
;holding registers (16 bit objects that can be read and written)<br />
;input registers (16 bit objects that can only be read)<br />
;coils (single bit objects that can be read and written)<br />
;discrete inputs (single bit objects that can only be read)<br />
<br />
The module uses the first character of these data object types to define attributes. <br />
Thus h770 refers to a holding register with the decimal address 770 and c120 refers to a coil with address 120. <br />
The address has to be specified as pure decimal number. The address counting starts at address 0<br />
<br />
Please note that the documentation for devices sometimes uses different numbering. They might start counting with one instead of zero so if a voltage value is stored in input register number 107 according to the documentation of the device, it might technically mean register number 106 (in the Modbus protocol specification addresses start with 0).<br />
Also some vendors use hexadecimal descriptions of their register addresses. So input register 107 might be noted as hex and means 263 or even 262 as decimal address.<br />
<br />
<pre><br />
attr PWP obj-h258-reading Temp_Wasser_Aus<br />
</pre> <br />
defines a reading with the name Temp_Wasser_Aus that is read from the Modbus holding register at address 258.<br />
With the attribute ending on <code>-expr</code> you can define a perl expression to do some conversion or calculation on the raw value read from the device. <br />
In the above example the raw value has to be devided by 10 to get the real value. If the raw value is also the final value then no <code>-expr</code> attribute is necessary. <br />
<br />
An object attribute ending on <code>-set</code> creates a fhem set option. <br />
In the above example the reading Temp_Soll can be changed to 12 degrees by the user with the fhem command <code>set PWP Temp_Soll 12</code><br />
The object attributes ending on <code>-min</code> and <code>-max</code> define min and max values for input validation <br />
and the attribute ending on <code>-hint</code> will tell fhem to create a selection list so the user can graphically select the defined values.<br />
<br />
To define general properties of the device you can specify attributes starting with <code>dev-</code>. <br />
E.g. with <code>dev-timing-timeout</code> you can specify the timeout when waiting for a response from the device. <br />
With <code>dev-h-</code> you can specify several default values or general settings for all holding registers <br />
like the function code to be used when reading or writing holding registers. <br />
These attributes are optional and the module will use defaults that work in most cases. <br />
<code>dev-h-combine 5</code> for example allows the module to combine read requests to objects having an address that differs 5 or less into one read request. <br />
Without setting this attribute the module will start individual read requests for each object. <br />
Typically the documentation for the modbus interface of a given device states the maximum number of objects that can be read in one function code 3 request.<br />
<code>dev-h-defUnpack n</code> means that the values in this example that the values are stored as unsigned short (16-bit) in "network" (big-endian) order. This is only one possibility of many. An integer value might be signed instead of unsigned or it might use different byte ordering (e.g. unpack codes v or s). <br />
<br />
== Handling Data Types == <br />
<br />
The Modbus protocol does not define data types. If the documentation of a device states that for example the current temperature is stored in holding register 102 this leaves room for many interpretations. Not only can the address 102 mean different things (actually decimal 102 or rather 101 if the vendor starts counting at 1 instead of 0 or even 257 or 258 if the vendor used hexadecimal addresses in his documentation ) also the data representation can be many different things. As in every programming language, there are many ways to represent numbers. They can be stored signed or unsigned, they can be integers or floating point numbers, the byte-order can be "big endian" or "small endian", the value can be stored in one holding register or in two holding registers (floating point numbers typically take four bytes which means two holding registers).<br />
The Modbus module allows flexible configuration of data representations be assigning a Perl unpack-code, a length, a Perl Expression, and the register ordering. The following example illustrates how this can be done: <br />
<pre><br />
attr PWP obj-h338-reading Pressure<br />
attr PWP obj-h338-len 2<br />
attr PWP obj-h338-unpack f><br />
attr PWP obj-h338-revRegs 1<br />
attr PWP obj-h338-format %.2f<br />
</pre><br />
In This example a floating point value for the reading "Pressure" is read from the holding registers starting at address 338. <br />
The value occupies 32 Bits and is therefore stored in two registers. The Perl pack code to use is f> which means a native single precision float in big endian format (byte order). With revRegs the module is instructed to reverse the order of the registers directly after reading. The format specification then defines how the value is formatted into a reading - in this case with two digits after the comma. See http://perldoc.perl.org/functions/pack.html for Perl pack / unpack codes and http://perldoc.perl.org/functions/sprintf.html for format specifications.<br />
<br />
If you need to read / write many objects for a device, defining all these parameters each time is not elegant. The Modbus module therefore offers twi ways to simplify this task: <br />
You can define defaults for every type of object or you can define your own data types once and then refer to them.<br />
This exampe shows how defaults can be specified for holding registers and input registers:<br />
<pre><br />
attr PWP dev-h-defUnpack f><br />
attr PWP dev-h-defLen 2<br />
attr PWP dev-h-defRevRegs 1<br />
attr PWP dev-h-defFormat %.2f<br />
<br />
attr PWP dev-i-defUnpack n<br />
attr PWP dev-i-defLen 1<br />
</pre><br />
<br />
The next example shows how you can define your own data types and then apply them to objects:<br />
<pre><br />
attr WP dev-type-VT_R4-format %.1f<br />
attr WP dev-type-VT_R4-len 2<br />
attr WP dev-type-VT_R4-revRegs 1<br />
attr WP dev-type-VT_R4-unpack f><br />
<br />
attr WP obj-h1234-reading Temp_In<br />
attr WP obj-h1234-type VT_R4<br />
attr WP obj-h1236-reading Temp_Out<br />
attr WP obj-h1236-type VT_R4<br />
</pre><br />
This example defines a data type with the name VT_R4 which uses an unpack code of f>, length 2 and reversed register ordering. It then assigns this Type to the objects Temp_In and Temp_Out.<br />
<br />
== Configuration of the module as Modbus slave == <br />
<br />
Data objects that the module offers to external Modbus masters (holding registers, input registers, coils or discrete inputs) are defined using attributes. <br />
If Fhem is Modbus slave, the attributes assign readings of Fhem devices to Modbus objects with their addresses and control how these objects are calculated from the reading values that exist in Fhem.<br />
It is also possible to allow an external Modbus master to send write function codes and change the value of readings inside Fhem.<br />
<br />
Example for a Modbus slave configuration:<br />
<pre><br />
define MRS485 Modbus /dev/ttyUSB2@9600,8,E,1<br />
define Data4PLC ModbusAttr 1 slave<br />
attr Data4PLC IODev MRS485<br />
<br />
attr Data4PLC obj-h256-reading THSensTerrasse:temperature<br />
attr Data4PLC obj-h256-unpack f<br />
attr Data4PLC obj-h256-len 2<br />
<br />
attr Data4PLC obj-h258-reading THSensTerrasse:humidity<br />
attr Data4PLC obj-h258-unpack f<br />
attr Data4PLC obj-h258-len 2<br />
<br />
attr Data4PLC obj-h260-reading myDummy:limit<br />
attr Data4PLC obj-h260-unpack n<br />
attr Data4PLC obj-h260-len 1<br />
attr Data4PLC obj-h260-allowWrite 1<br />
</pre><br />
<br />
In this example Fhem allows an external Modbus master to read the temperature of a Fhem device named THSensTerrasse through holding register 256 and the humidity of that Fhem device through holding register 258. Both are encoded as floting point values that span two registers. <br />
The master can also read but also write the reading named limit of the device myDummy.<br />
<br />
== Set-Commands ==<br />
can be defined for holding registers and coils by using attributes.<br />
<br />
Every object for which an attribute like <code>obj-xy-set</code> is set to 1 will create a valid set option.<br />
<br />
Additionally the attribute <code>enableControlSet</code> enables the set options <code>interval</code>, <code>stop</code>, <code>start</code>, <code>reread</code> as well as <code>scanModbusObjects</code>, <code>scanStop</code> and <code>scanModbusIds</code> (for devices connected with RTU / ASCII over a serial line).<br />
<br />
;<code>interval &lt;Interval&gt;</code><br />
:modifies the interval that was set during define. <br />
<br />
;<code>stop</code><br />
:stops the interval timer that is used to automatically poll objects through Modbus.<br />
<br />
;<code>start</code><br />
:starts the interval timer that is used to automatically poll objects through Modbus. <br />
:If an interval is specified during the define command then the interval timer is started automatically. <br />
:However if you stop it with the command <code>set &lt;mydevice&gt; stop</code> <br />
:then you can start it again with <code>set &lt;mydevice&gt; start</code>.<br />
<br />
;<code>reread</code><br />
:causes a read of all objects that are set to be polled in the defined interval. The interval timer is not modified.<br />
<br />
;<code>scanModbusObjects &lt;startObj&gt; - &lt;endObj&gt; &lt;reqLen&gt;</code><br />
:scans the device objects and automatically creates attributes for each reply it gets. <br />
:This might be useful for exploring devices without proper documentation. <br />
:The following example starts a scan and queries the holding registers with addresses between 100 and 120. <br />
:<code>set MyModbusAttrDevice scanModbusObjects h100-120</code><br><br />
:For each reply it gets, the module creates a reading like<br />
:<code>scan-h100 hex=0021, string=.!, s=8448, s>=33, S=8448, S>=33</code><br><br />
:the representation of the result as hex is 0021 and<br />
:the ASCII representation is .!. s, s>, S and S> are different representations with their Perl pack-code.<br />
<br />
;<code>scanModbusIds &lt;startId&gt; - &lt;endId&gt; &lt;knownObj&gt;</code><br />
:scans for Modbus Ids on an RS485 Bus. The following set command for example starts a scan:<br><br />
:<code>set Device scanModbusId 1-7 h770</code><br><br />
:since many Modbus devices don't reply at all if an object is requested that does not exist, <br />
:scanModbusId needs the adress of an object that is known to exist.<br />
:If a device with Id 5 replies to a read request for holding register 770, a reading like the following will be created:<br />
:<code>scanId-5-Response-h770 hex=0064, string=.d, s=25600, s>=100, S=25600, S>=100</code><br />
;<code>scanStop</code><br />
:stops any running scans.<br />
;saveAsModule <name><br />
:experimental: saves the definitions of obj- and dev- attributes in a new fhem module file as /tmp/98_ModbusGen<name>.pm.<br />
:if this file is copied into the fhem module subdirectory (e.g. /opt/fhem/FHEM) and fhem is restarted then instead of defining a device<br />
:as ModbusAttr with all the attributes to define objects, you can just define a device of the new type ModbusGen<name> and all the <br />
:objects will be there by default. However all definitions can still be changed / overriden with the attribues defined in ModbusAttr if needed.<br />
<br />
== Get-Commands ==<br />
<br />
Every reading can be manually requested by a Get. <br />
Internally a Get command triggers the corresponding Modbus request to the device and the module then interprets the data and sets the right Fhem readings. To avoid huge option lists in FHEMWEB, the objects visible as Get in FHEMWEB can be defined by setting an attribute <code>obj-xy-showget</code> to 1. <br />
<br />
== All Attributes ==<br />
<br />
;readingFnAttributes<br />
:the usual Fhem attributes for all devices<br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. <br />
:This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;enableControlSet<br />
:enables the built in set commands like interval, stop, start and reread (see above) <br />
<br />
the following list of attributes can be applied to any data object by specifying the objects type and address in the variable part. <br />
For many attributes you can also specify default values per object type (see dev- attributes later) or you can specify an object attribute without type and address (e.g. obj-len) which then applies as default for all objects:<br />
<br />
;obj-[cdih][1-9][0-9]*-reading <br />
:define the name of a reading that corresponds to the Modbus data object of type c,d,i or h and a decimal address (e.g. obj-h225-reading).<br />
<br />
;obj-[cdih][1-9][0-9]*-name <br />
:defines an optional internal name of this data object (this has no meaning for fhem and serves mainly documentation purposes.<br />
<br />
;obj-[cdih][1-9][0-9]*-set <br />
:if set to 1 then this data object can be changed (works only for holding registers and coils since discrete inputs and input registers can not be modified by definition.<br />
<br />
;obj-[cdih][1-9][0-9]*-min <br />
:defines a lower limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-max <br />
:defines an upper limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-hint <br />
:this is used for set options and tells fhemweb what selection to display for the set option (list or slider etc.)<br />
<br />
;obj-[cdih][1-9][0-9]*-expr <br />
:defines a perl expression that converts the raw value read from the device.<br />
<br />
;obj-[cdih][1-9][0-9]*-ignoreExpr<br />
:defines a perl expression that returns 1 if a value should be ignored and the existing reading should not be modified<br />
<br />
;obj-[cdih][1-9][0-9]*-map <br />
:defines a map to convert values read from the device to more convenient values when the raw value is read from the device or back when the value to write has to be converted from the user value to a raw value that can be written. Example: 0:mittig, 1:oberhalb, 2:unterhalb <br />
<br />
;obj-[cdih][1-9][0-9]*-setexpr <br />
:defines a perl expression that converts the user specified value in a set to a raw value that can be sent to the device. This is typically the inversion of -expr above.<br />
<br />
;obj-[cdih][1-9][0-9]*-format <br />
:defines a format string to format the value read e.g. %.1f<br />
<br />
;obj-[cdih][1-9][0-9]*-len <br />
:defines the length of the data object in registers. It defaults to 1. Some devices store 32 bit floating point values in two registers. In this case you can set this attribute to two.<br />
<br />
;obj-[cdih][1-9][0-9]*-unpack <br />
:defines the unpack code to convert the raw data string read from the device to a reading. For an unsigned integer in big endian format this would be "n", for a signed 16 bit integer in big endian format this would be "s>" and for a 32 bit big endian float value this would be "f>". (see the perl documentation of the pack function).<br />
<br />
;obj-[cdih][1-9][0-9]*-revRegs<br />
:this is only applicable to objects that span several input registers or holding registers.<br />
:when they are read then the order of the registers will be reversed before <br />
:further interpretation / unpacking of the raw register string. <br />
:The same happens before the object is written with a set command.<br />
<br />
;obj-[cdih][1-9][0-9]*-bswapRegs<br />
:this is applicable to objects that span several input or holding registers.<br />
:After the registers have been read and before they are writtem, <br />
:all 16-bit values are treated big-endian and are reversed to little-endian by swapping the two 8 bit bytes. <br />
:This functionality is most likely used for reading (ASCII) strings from the device <br />
:that are stored as big-endian 16-bit values.<br />
:example: original reading is "324d3130203a57577361657320722020". After applying bswapRegs, <br />
:the value will be "4d3230313a2057576173736572202020" which will result in the ASCII string <br />
:"M201: WWasser ". <br />
:Should be used with "(a*)" as -unpack value.<br />
<br />
;obj-[cdih][1-9][0-9]*-decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-showget <br />
:every reading can also be requested by a get command. However these get commands are not automatically offered in fhemweb. By specifying this attribute, the get will be visible in fhemweb.<br />
<br />
;obj-[cdih][1-9][0-9]*-poll<br />
:if set to 1 then this obeject is included in the cyclic update request as specified in the define command. If not set, then the object can manually be requested with a get command, but it is not automatically updated each interval. Note that this setting can also be specified as default for all objects with the dev- atributes described later.<br />
<br />
;obj-[cdih][1-9][0-9]*-polldelay <br />
:this attribute allows to poll objects at a lower rate than the interval specified in the define command. you can either specify a time in seconds or number prefixed by "x" which means a multiple of the interval of the define command. if you specify a normal numer then it is interpreted as minimal time between the last read and another automatic read. Please note that this does not create an individual interval timer. Instead the normal interval timer defined by the interval of the define command will check if this reading is due or not yet. So the effective interval will always be a multiple of the interval of the define.<br />
<br />
<br />
;dev-([cdih]-)*read <br />
:specifies the function code to use for reading this type of object. The default is 3 for holding registers, 1 for coils, 2 for discrete inputs and 4 for input registers.<br />
<br />
;dev-([cdih]-)*write <br />
:specifies the function code to use for writing this type of object. The default is 6 for holding registers and 5 for coils. Discrete inputs and input registers can not be written by definition.<br />
<br />
;dev-([cdih]-)*combine <br />
:defines how many adjacent objects can be read in one request. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defLen <br />
:defines the default length for this object type. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defFormat <br />
:defines a default format string to use for this object type in a sprintf function on the values read from the device.<br />
<br />
;dev-([cdih]-)*defExpr<br />
:defines a default Perl expression to use for this object type to convert raw values read.<br />
<br />
;dev-([cdih]-)*defIgnoreExpr<br />
:defines a default Perl expression to decide when values should be ignored.<br />
<br />
;dev-([cdih]-)*defUnpack <br />
:defines the default unpack code for this object type. <br />
<br />
;dev-([cdih]-)*defRevRegs<br />
:defines that the order of registers for objects that span several registers will be reversed before <br />
:further interpretation / unpacking of the raw register string<br />
<br />
;dev-([cdih]-)*defBswapRegs<br />
:per device default for swapping the bytes in Registers (see obj-bswapRegs above)<br />
<br />
;dev-([cdih]-)*defDecode<br />
:defines a default for decoding the strings read from a different character set e.g. cp850<br />
<br />
;dev-([cdih]-)*defEncode<br />
:defines a default for encoding the strings read (or after decoding from a different character set) e.g. utf8<br />
<br />
;dev-([cdih]-)*defPoll <br />
:if set to 1 then all objects of this type will be included in the cyclic update by default. <br />
<br />
;dev-([cdih]-)*defShowGet <br />
:if set to 1 then all objects of this type will have a visible get by default. <br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
<br />
;dev-([cdih]-)*allowShortResponses <br />
:if set to 1 the module will accept a response with valid checksum but data lengh < lengh in header<br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
;nextOpenDelay <br />
:delay for Modbus-TCP connections. <br />
:This defines how long the module should wait after a failed TCP connection attempt before the next reconnection attempt. <br />
:This defaults to 60 seconds.<br />
<br />
;openTimeout <br />
:timeout to be used when opening a Modbus TCP connection (defaults to 3)<br />
<br />
;timeoutLogLevel <br />
:log level that is used when logging a timeout. Defaults to 3. <br />
<br />
;silentReconnect <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
<br />
;maxTimeoutsToReconnect <br />
:this attribute is only valid for TCP connected devices. <br />
:In such cases a disconnected device might stay undetected and lead to timeouts until the TCP connection is reopened. <br />
:This attribute specifies after how many timeouts an automatic reconnect is tried.<br />
<br />
;dev-h-brokenFC3<br />
:workaround for some broken Modbus function code 3 implementations<br />
<br />
;disable<br />
:stop communication with the device while this attribute is set to 1. For Modbus over TCP this also closes the TCP connection.<br />
<br />
<br />
== Links ==<br />
* [http://www.Modbus.org Modbus.org]<br />
* About Modbus ([http://en.wikipedia.org/wiki/Modbus English] / [http://de.wikipedia.org/wiki/Modbus German])<br />
* Perl [http://perldoc.perl.org/functions/pack.html unpack codes]<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Modbus&diff=28125Modbus2018-10-18T19:52:34Z<p>StefanStrobel: </p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Library or physical device to extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=Modbus<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_Modbus.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
<br />
Modbus defines a physical modbus interface and library functions to be called from other logical modules / devices.<br />
This low level module takes care of the communication with modbus devices and provides Get, Set and cyclic polling of Readings as well as formatting and input validation functions.<br />
<br />
The logical device modules for individual machines only need to define the supported modbus function codes and objects of the machine with the modbus interface in data structures. These data structures are then used by this low level module to implement Set, Get and automatic updating of readings in a given interval.<br />
<br />
The Modbus module supports Modbus RTU over serial / RS485 lines as well as Modbus TCP and Modbus RTU over TCP. It defines read / write functions for Modbus holding registers, input registers, coils and discrete inputs.<br />
<br />
See [[ModbusAttr]] if you don't want to use a library to develop your own module and if you are looking for a generic Modbus Module instead that can be configured with attributes.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module requires the Device::SerialPort or Win32::SerialPort module if you want to communicate with modbus devices over a serial line.<br />
<br />
== Define of a modbus interface device for serial communication ==<br />
<pre><br />
define <name> Modbus <device><br />
</pre><br />
<br />
A define of a physical device based on this module is only necessary if a shared physical device like a RS485 USB adapter is used. <br />
In the case of Modbus TCP this module will be used as a library for other modules that define all the data objects and no define of the base module is needed.<br />
<br />
Example:<br />
<pre><br />
define ModBusLine Modbus /dev/ttyUSB1@9600<br />
</pre><br />
<br />
In this example the module opens the given serial interface and other logical modules like [[ModbusAttr]] or [[ModbusSET]] can access several Modbus devices connected to this bus concurrently.<br />
<br />
== Set-Commands ==<br />
this low level device module doesn't provide set commands for itself but implements set <br />
for logical device modules that make use of this module as a library. See ModbusSET for example.<br />
<br />
== Get-Commands ==<br />
this low level device module doesn't provide get commands for itself but implements get <br />
for logical device modules that make use of this module as a library.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;queueDelay<br />
:modify the delay used when sending requests to the device from the internal queue, defaults to 1 second<br />
<br />
;queueMax<br />
:max length of the send queue, defaults to 100<br><br />
<br />
;clientSwitchDelay<br />
:defines a delay that is always enforced between the last read from the bus and the next send to the bus <br />
:for all connected devices, but only if the next send goes to a different device than the last one<br />
<br />
;dropQueueDoubles<br />
:prevents new request to be queued if the same request is already in the send queue<br />
<br />
;skipGarbage<br />
:if set to 1 this attribute will enhance the way the module treats Modbus response frames (RTU over serial lines) <br />
:that look as if they have a wrong Modbus id as their first byte. <br />
:If skipGarbage is set to 1 then the module will skip all bytes until a byte with the expected modbus id is seen. <br />
:Under normal circumstances this behavior should not do any harm and lead to more robustness. <br />
:However since it changes the original behavior of this module it has to be turned on explicitely.<br />
:For Modbus ASCII it skips bytes until the expected starting byte ":" is seen.<br />
<br />
;profileInterval<br />
:if set to something non zero it is the time period in seconds for which the module will create bus usage statistics. <br />
:Please note that this number should be at least twice as big as the interval used for requesting values in logical devices <br />
:that use this physical device<br />
:The bus usage statistics create the following readings:<br />
:* Profiler_Delay_sum<br />
::seconds used as delays to implement the defined sendDelay and commDelay<br />
:*Profiler_Fhem_sum<br />
::seconds spend processing in the module<br />
:*Profiler_Idle_sum<br />
::idle time <br />
:*Profiler_Read_sum<br />
::seconds spent reading and validating the data read<br />
:*Profiler_Send_sum<br />
::seconds spent preparing and sending data<br />
:*Profiler_Wait_sum<br />
::seconds waiting for a response to a request<br />
:*Statistics_Requests<br />
::number of requests sent<br />
:*Statistics_Timeouts<br />
::timeouts encountered<br />
<br />
<br />
== Writing modules for devices using this module as a library ==<br />
<br />
Writing a module for a physical device with modbus interface is easy when you use the 98_Modbus.pm module as a library. <br />
To use this module as a library for other fhem modules you only have to define a data structure that defines the mapping between modbus data objects (holding registers, input registers, coils or discrete inputs) and fhem readings.<br />
Additionally the module needs to contain a few <code>package</code> and <code>use</code> statements and an initialize function at the beginning, that assigns a few special variables to point to functions of the Modbus base module. <br />
<br />
The most easy way to start is to use [[ModbusAttr]] to define all objects and data types and then issue a set saveAsModule command which creates a new module automatically.<br />
<br />
Example for a module that is called ModbusSET:<br />
<pre><br />
package main;<br />
use strict;<br />
use warnings;<br />
<br />
sub ModbusSET_Initialize($)<br />
{<br />
my ($modHash) = @_;<br />
LoadModule "Modbus";<br />
require "$attr{global}{modpath}/FHEM/DevIo.pm";<br />
<br />
$modHash->{parseInfo} = \%SET10parseInfo; # defines registers, inputs, coils etc. for this Modbus Device<br />
$modHash->{deviceInfo} = \%SET10deviceInfo; # defines properties of the device, defaults and supported function codes<br />
<br />
ModbusLD_Initialize($modHash); # Generic function of the Modbus module does the rest<br />
<br />
$modHash->{AttrList} = $modHash->{AttrList} . " " . # Standard Attributes like IODEv etc <br />
$modHash->{ObjAttrList} . " " . # Attributes to add or overwrite parseInfo definitions<br />
$modHash->{DevAttrList}; # Attributes to add or overwrite devInfo definitions<br />
<br />
}<br />
</pre><br />
<br />
The name of the initialize-Function has to match the name of the module. In the above example this is <code>ModbusSET_Initialize</code>. Most of the steps needed in an initialize function are provided by the library function <code>ModbusLD_Initialize</code>. This function tells fhem to use the library functions for <code>define</code>, <code>set</code>, <code>get</code> and other typical functions in a module. See [[DevelopmentModuleIntro]] for more background information on writing fhem modules if you are curious.<br />
<br />
=== Introduction to the parseInfo structure ===<br />
<br />
Typically the data structure to map between data objects of the modbus device and fhem readings is named <code>parseInfo</code> with a part of the name of the module itself as prefix. In the example of the module 98_ModbusSET.pm which uses Modbus.pm to implement a module for SET Silent 10 heat pumps, the structure is called <code>SET10parseInfo</code>.<br />
<br />
This strucure contains keys with values that directly correspond to attributes which can be used with the module ModbusAttr so it is advisable to prototype a new module with ModbusAttr and then translate the attributes to entries in the parseInfo structure. The values in the parseInfo structure can later still be overwritten / extended with the attributes documented in ModbusAttr.<br />
<br />
As an example a very simple definition of a parseInfo structure for a heat pump could look like this:<br />
<br />
<pre><br />
my %XYparseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
},<br />
"h258" => { reading => "Temp_Wasser_Aus",<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
set => 1, # this value can be set<br />
}<br />
);<br />
</pre><br />
<br />
the corresponding attributes for ModbusAttr for prototyping or overwriting values would be obj-h256-reading, obj-h258-reading, obj-h770-reading, obj-h770-min and so on.<br />
<br />
<br />
This parseInfo structure would be the main part of the module and map from holding register 256 to a fhem reading named Temp_Wasser_Ein, holding register 258 to Temp_Wasser_Aus and 770 to Temp_Soll. <br />
<br />
All readings will be read from the device in an interval that the user can specify when he issues the define command for your module.<br />
The meaning of <code>set => 1</code> is that the holding register 770 can also be written to with a set command. FHEM will check that the value written is not smaller than 10 and not bigger than 32 as specified above.<br />
<br />
More complex example:<br />
<pre><br />
my %SET10parseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
name => "Pb1", # internal name of this register in the hardware doc<br />
expr => '$val / 10', # conversion of raw value to visible value <br />
len => 1,<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
name => "ST03",<br />
expr => '$val / 10', # convert raw value to readable temp<br />
setexpr => '$val * 10', # expression to convert a set value to the internal value <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
hint => "8,10,20,25,28,29,30,30.5,31,31.5,32",<br />
set => 1, # this value can be set<br />
},<br />
"h771" => { reading => "Hysterese", # Hex Adr 303<br />
name => "ST04",<br />
expr => '$val / 10',<br />
setexpr => '$val * 10',<br />
poll => "x10", # only poll every 10th iteration.<br />
min => 0.5,<br />
max => 3,<br />
set => 1,<br />
},<br />
"h777" => { reading => "Hyst_Mode", # Hex Adr 0309<br />
name => "ST10",<br />
map => "0:mittig, 1:oberhalb, 2:unterhalb", <br />
poll => "once", # only poll once (or after a set)<br />
set => 1,<br />
},<br />
"i800" => { reading => "Voltage", # Input register <br />
unpack => "f>", # this value is a float<br />
len => 2, # the float occupies two input registers, 800 and 801<br />
},<br />
);<br />
</pre><br />
<br />
There are many more options that can be specified for each data object / reading. If these options are not specified, the base module assumes defaults that typically make sense. However if you want to modify the defaults, you can either define explicit values in the parseInfo structure or you can define another data structure typically called deviceInfo. <br />
<br />
=== Introduction to the deviceInfo structure ===<br />
<br />
The deviceInfo structure is completely optional. If you don't define it in your module, the base module takes default values that work in ost cases. If you only want to override a few of the defaults, you can just define them and leave other options or sections out.<br />
A simple device info structure that modifies some defaults could look like this:<br />
<br />
<pre><br />
my %SET10deviceInfo = (<br />
"timing" => {<br />
timeout => 3, # timeout is 3 seconds /default would be 2<br />
commDelay => 0.7, # wait 0.7 seconds before sending after receiving<br />
sendDelay => 0.7, # wait at least 0.7 seconds for another send<br />
}, <br />
"c" => { <br />
read => 1, # function code 1 to read coils (this could be omitted because it is the default anyways<br />
write => 5, # dito<br />
},<br />
"h" => { <br />
read => 3, <br />
write => 6, <br />
defLen => 1, # default legth is 1 object<br />
combine => 5, <br />
defShowGet => 1, <br />
defUnpack => "s>", # default data format is a signed 16 bit integer for holding registers <br />
},<br />
);<br />
</pre><br />
<br />
The deviceInfo structure contains five optional parts. Timing defines timing values and the remaining parts define settings or defaults for coils (c), discrete inputs (d), input registers (i) and holding registers (h). <br />
<br />
for each modbus object type you can change what function code should be used to read or write to the object. This is completely optional and if nothing is specified, the base module assumes function codes 1,2,3 and 4 for reading as well as 5 and 6 for writing which works for many modbus devices. If you prefer to use function code 16 for writing to holding registers, you can specify "write => 16" in the "h" part.<br />
<br />
=== usage of a module created this way ===<br />
<br />
a logical module written this way will have a define command that can work in two ways. <br />
If your module would be called ModbusSET and it is using a serial line connection (Modbus RTU over RS485 oer over RS232):<br />
<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate<br />
define <name> ModbusSET <Id> <Interval> </code><br />
</pre><br />
<br />
In this case, a physical serial interface device is defined first using the Modbus module. Then a device based on your module (ModbusSET in the example) is defined for each physical modbus device connected to the serial line. For a RS485 bus, several devices with different Ids can be connected to the same bus.<br />
<br />
Example:<br />
<pre><br />
define ModbusRS485 Modbus /dev/rs485@9600<br />
define PWP ModbusAttr 5 60<br />
</pre><br />
<br />
this defines the device and it will use the readings that you coded in the parseInfo data structure.<br />
<br />
Alternatively your module would also support Modbus TCP or Modbus RTU over TCP with the following define syntax:<br />
<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|TCP><br />
</pre><br />
In this case no serial interface device is necessary and your module connects to the modbus device directly via TCP using either Modbus TCP or Modbus RTU over TCP.<br />
<br />
Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115:502 TCP<br />
</pre><br />
<br />
=== General information about data objects ===<br />
<br />
Modbus devices can use many different ways to encode values in their data objects. A temperature might be stored multiplied with 10 as a 16 bit integer value in one holding register so you have to read the integer and divide it by 10 to get the real temperature value back. It might also be stored as a 32 bit float data type that spans two adjacent input registers.<br />
The modbus base module implements a very generic way to handle different encodings without real programming: It lets you define the Perl unpack code to convert a raw data string to a Perl value, a Perl expression to do further computation and a length in data objects. <br />
<br />
This way a temperature stored in a 16 bit signed integer as the value multiplied by 10 can be described with the unpack code "s>" and the expression "$val / 10". A float value spanning 2 registers would be described with an unpack code "f>" and a len of 2. No expression is needed in this case. See Perldoc on the pack function for a detailed explation of pack and unpack codes.<br />
<br />
The idea here is that you should be able to define any mapping, encoding, transformation or formatting of data objects without programming by simpy describing them.<br />
<br />
=== most important options in parseInfo ===<br />
<br />
Most options here are optional and can be used if there is a need but they can also be omitted. If most readings require the same options and the option is different from the default, it is also possible to define a different default per modbus data object type in another data structure (see deviceInfo).<br />
For a list of all options please refer to the attributes documentation of the module ModbusAttr. The attributes there can be translated to parseInfo or deviceInfo keys as shown above.<br />
<br />
;reading <br />
:name of the reading to be used in FHEM e.g. Temp_Wasser_ein<br />
<br />
;expr <br />
:perl expression to convert a string after it has been read. The original value is in $val e.g. $val / 10<br />
<br />
;map <br />
:a map string to convert an value from the device to a more readable output string or to convert a user input to the machine representation e.g. "0:mittig, 1:oberhalb, 2:unterhalb" <br />
<br />
;format <br />
:a format string for sprintf to format a value read, e.g. %.1f<br />
<br />
;len <br />
:number of Registers this value spans, can be 2 for a 32 bit float which is stored in 2 registers<br />
<br />
;unpack <br />
:defines the translation between data in the module and in the communication frame see the documentation of the perl pack function for details. example: "n" for an unsigned 16 bit value or "f>" for a float that is stored in two registers or "s>" for signed 16 bit integer in big endian format<br />
<br />
;showget <br />
:can be set to 1 to allow a FHEM get command to read this value from the device. All defined objects can be used in a get command that is issued on the command line. This parameter only controls if fhemweb will offer a get command for the object.<br />
<br />
;poll <br />
:defines if this value is included in the read that the module does every defined interval this can be changed by a user with an attribute<br />
<br />
;polldelay <br />
:if a value should not be read in each iteration (after interval has passed), this value can be set to an explicit time in seconds. The update function will then verify if this delay has elapsed since the last read of this object. If not, the read is skipped.<br />
<br />
;set <br />
:can be set to 1 to allow writing this value with a FHEM set-command<br />
<br />
;min <br />
:min value for input validation in a set command. If the user issues e.g. set Device Temp_Soll 10, FHEM will check if the given value 10 is bigger or equal the defined min and smaller or equal the defined max.<br />
<br />
;max <br />
:max value for input validation in a set command<br />
<br />
;hint <br />
:string for fhemweb to create a selection or slider<br />
<br />
;setexpr <br />
:per expression to convert an input string to the machine format before writing this is typically the reverse of the above expr, e.g. $val * 10<br />
<br />
;name<br />
:optional internal name of the value in the modbus documentation of the physical device, e.g. pb1<br />
<br />
<br />
=== most important options in deviceInfo ===<br />
<br />
Keys in the timing section:<br />
<br />
;timeout <br />
:how long to wait for a response from the device, can be overwritten by attribute timeout in logical device. Defaults to two seconds if this is not specified<br />
<br />
;commDelay <br />
:minimal delay in secounds between two communications e.g. a read a the next write, can be overwritten with attribute commDelay if added to AttrList in _Initialize below defaults to 0.1 seconds if not specified<br />
<br />
;sendDelay <br />
:minimal delay in seconds between two sends, can be overwritten with the attribute sendDelay if added to AttrList in _Initialize function below. Defaults to 0.1 seconds if not specified<br />
<br />
Keys per object type (c = coil, d = discrete input, i = input register, h = holding register)<br />
<br />
;read <br />
:function code to use for reading this object type (e.g. 3 for holding registers) defaults to function codes 1-4 depending on the object types if nothing else is specified (3 to read holding register, 1 to read coils and so on)<br />
<br />
;write <br />
:function code to use for writing this object type (e.g. 6 or 16 for holding registers) defaults to function codes 5 and 6 depending on the object types if nothing else is specified (6 to read holding register, 5 to write coils and so on)<br />
<br />
;defLen <br />
:default len for objects using this type (e.g. can be set to 2 if the device mainly provides float values that span 2 registers (2 times 16 Bit) can be overwritten in parseInfo per reading by specifying the key "len" defaults to 1 if not specified<br />
<br />
;defFormat <br />
:format string to do sprintf with the value can be overwritten in parseInfo per reading by specifying the key "format" if no format is specified here and none in parseInfo, the the reading is set without further formatting (which is typically fine)<br />
<br />
;defUnpack <br />
:default pack / unpack code to convert raw values, e.g. "n" for a 16 bit integer or "f>" for a big endian float can be overwritten in parseInfo per reading by specifying the key "unpack" if not specified here and not in parseInfo, then the raw value is interpreted as "n" which is 16 bit unsigned integer in big endian format<br />
<br />
;defPoll <br />
:defines that objects of this type should be polled by default unless specified otherwise in parseInfo or by attributes can be overwritten in parseInfo per reading by specifying the key defaultpoll if not specified here or in parseInfo, the object is not polled<br />
<br />
;defShowGet <br />
:defines that FHEMweb shows a Get option (by returning it as reslut to get ?) for objects of this type can be overwritten in parseInfo per reading by specifying the key showget defaults to 0.<br />
<br />
;combine <br />
:max number of registers that the device is willing to deliver in one read request. The modbus application layer protocol specification allows for more than 100 but most devices limit this to 5, 10 or some other number. This option defaults to 1 if not specified.<br />
<br />
For an example of a full module that is based on the mechanisms described here see 98_ModbusSET.pm.<br />
<br />
=== Attributes of your module ===<br />
<br />
a module based on the base module / library 98_Modbus.pm can also allow the end user to modify properties of each reading if you want to allow it. All you have to do is to offer an attribute by adding its name to the variable $modHash->{AttrList} in your initialize function.<br />
<br />
If for example you want to allow the user to modify the maximum value for the reading Temp_Soll, you can add "Temp_Soll-max " to this variable and the user can then set this attribute. The attribute takes precedence over the max potentially already defined in your parseInfo structure.<br />
<br />
There are two ways that the base module accepts such readings. One is the reading name followed by "-" and the option to override, the alternative syntax is "obj-" followed by the first letter of an object type (c/d/h/i) and a decimal address just like the main key of an object in the parseInfo structure. <br />
<br />
Instead of allowing the attribute Temp_Soll-max for the max value of reading Temp_Soll which corresponds to holding register 770, you can alternatively add the attribute name "obj-h770-min " to $modHash->{AttrList}.<br />
<br />
If the user is allowd to specify such attributes solely depends on the contents of the $modHash->{AttrList} variable. All the processing is already built into the base module.<br />
<br />
If you want to allow the user the override the formatting of readings then you can add "obj-[cdih][1-9][0-9]*-format " as a regular expression that allows format specifications for all possible data objects.<br />
<br />
The module 98_ModbusAttr for example is also based on 98_Modbus.pm and allows all possible attributes so the user can completely define his device with attributes and without a parseInfo or deviceInfo structure.<br />
<br />
In the same way you can allow the user to override the device specific options and defaults with attributes that start with "dev-", followed by the section of the deviceInfo and the name of the option. If you want to allow the user to modify the function code to be used for writing holding registers, you can add the attribute "obj-h-write " and the user can then set this attribute to 6 or 16 as he prefers.<br />
It is up to the module author to decide if this makes sense.<br />
<br />
An assignment that allows most options to the user could be:<br />
<br />
<pre><br />
$modHash->{AttrList} = $modHash->{AttrList} . " " .<br />
"obj-[cdih][1-9][0-9]*-reading " .<br />
"obj-[cdih][1-9][0-9]*-name " .<br />
"obj-[cdih][1-9][0-9]*-set " .<br />
"obj-[cdih][1-9][0-9]*-min " .<br />
"obj-[cdih][1-9][0-9]*-max " .<br />
"obj-[cdih][1-9][0-9]*-hint " .<br />
"obj-[cdih][1-9][0-9]*-expr " .<br />
"obj-[cdih][1-9][0-9]*-map " .<br />
"obj-[cdih][1-9][0-9]*-setexpr " .<br />
"obj-[cdih][1-9][0-9]*-format " .<br />
"obj-[cdih][1-9][0-9]*-len " .<br />
"obj-[cdih][1-9][0-9]*-unpack " .<br />
"obj-[cdih][1-9][0-9]*-showget " .<br />
<br />
"obj-[cdih][1-9][0-9]*-poll " .<br />
"obj-[cdih][1-9][0-9]*-polldelay " .<br />
"poll-.* " .<br />
"polldelay-.* " .<br />
<br />
"dev-([cdih]-)*read " .<br />
"dev-([cdih]-)*write " .<br />
"dev-([cdih]-)*combine " .<br />
"dev-([cdih]-)*defLen " .<br />
"dev-([cdih]-)*defFormat " .<br />
"dev-([cdih]-)*defUnpack " .<br />
"dev-([cdih]-)*defPoll " .<br />
"dev-([cdih]-)*defShowGet " .<br />
"dev-timing-timeout " .<br />
"dev-timing-sendDelay " .<br />
"dev-timing-commDelay ";<br />
}<br />
</pre><br />
<br />
== Examples for logical device modules that use this base module ==<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM220M]<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM630M]<br />
:modules for energy meters from B+G E-Tech & EASTON written by Roger<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG103]<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG604]<br />
:modules for the UMG103 and UMG604 meters from Janitza<br />
;[[ModbusSET]]<br />
:module for the set silent heat pumps from Schmidt Energie Technik<br />
;[[ModbusAttr]]<br />
:generic modbus device module where the data objects, addresses, display formats, function codes and other things can be configured using FHEM attributes similar to HTTPMOD<br />
<br />
<br />
[[Kategorie:Interfaces]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=27319HTTPMOD2018-07-06T06:43:39Z<p>StefanStrobel: /* Example for AmbientMonitor */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<syntaxhighlight lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</syntaxhighlight><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</syntaxhighlight><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
;enableCookies<br />
:enables the cookie handling inside HTTPMOD. It is advisable to always set this attribute and allow HTTPMOD to track the state of cookies and set them for following HTTP-requests<br />
<br />
;enforceGoodReadingNames<br />
:makes sure that HTTPMOD only creates readings that are allowd for Fhem (especially if reading names are dynamically created from JSON object names with extractAllJSON. It is advisable to always set this attribute.<br />
<br />
;handleRedirects<br />
:enables the redirect handling inside HTTPMOD which should be used together with the cookie handling of HTTPMOD. HTTPMOD will then automatically follow redirects from a web server and keep track of cookies at the same time. It is advisable to always set this attribute.<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<syntaxhighlight lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</syntaxhighlight><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
<br />
attr PM enableControlSet 1<br />
attr PM enableCookies 1<br />
attr PM enforceGoodReadingNames 1<br />
attr PM handleRedirects 1<br />
<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</syntaxhighlight><br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</syntaxhighlight><br />
<br />
the definition could be:<br />
<syntaxhighlight lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
<br />
attr AmbientMonitor enableControlSet 1<br />
attr AmbientMonitor enableCookies 1<br />
attr AmbientMonitor enforceGoodReadingNames 1<br />
attr AmbientMonitor handleRedirects 1<br />
<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</syntaxhighlight><br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<syntaxhighlight lang="perl"><br />
reading01Format %.1f<br />
</syntaxhighlight><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<syntaxhighlight lang="perl"><br />
reading01-2Format %.1f<br />
</syntaxhighlight><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<syntaxhighlight lang="perl"><br />
readingFormat %.1f<br />
</syntaxhighlight><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</syntaxhighlight><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</syntaxhighlight><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<syntaxhighlight lang="perl"><br />
attr PM getDecode UTF-8<br />
</syntaxhighlight><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
{{Randnotiz|RNTyp=y|RNText=Starting with version ''2018-02-10'' the internal that holds the HTTP response is no longer called '''''buf''''' but rather '''''httpbody''''', and it is only displayed when attribute '''''showBody''''' is set to "1".}}<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</syntaxhighlight><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<syntaxhighlight lang="perl"><br />
reading02Name Temp<br />
</syntaxhighlight><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</syntaxhighlight><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<syntaxhighlight lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</syntaxhighlight><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</syntaxhighlight><br />
<br />
with JSON you can write <br />
<syntaxhighlight lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</syntaxhighlight><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<syntaxhighlight lang="perl"><br />
attr test2 extractAllJSON<br />
</syntaxhighlight><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</syntaxhighlight><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<syntaxhighlight lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</syntaxhighlight><br />
<br />
then a configuration like <br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</syntaxhighlight><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</syntaxhighlight><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</syntaxhighlight><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</syntaxhighlight><br />
<br />
with XPath you can write <br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</syntaxhighlight><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</syntaxhighlight><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</syntaxhighlight> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<syntaxhighlight lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</syntaxhighlight> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</syntaxhighlight> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
'''Other example : steering a pellet stove from Rika'''<br />
<br />
The stove API of Rika on https://www.rika-firenet.com/web/ delivers a JSON string with all settings and values, and can be piloted with a data string containing all the "set" values at once.<br />
<br />
Delivered JSON on get https://www.rika-firenet.com/api/client/xxxxxxxx/status : (xxxxxxxx must be replaced with the unique stove ID)<br />
<syntaxhighlight lang="perl"><br />
{<br />
"name": "Vorzimmer",<br />
"stoveID": "xxxxxxxxx",<br />
"lastSeenMinutes": 1,<br />
"lastConfirmedRevision": 1504385700,<br />
"controls": {<br />
"revision": 1504385700,<br />
"onOff": true,<br />
"operatingMode": 2,<br />
"heatingPower": 65,<br />
"targetTemperature": 24,<br />
"heatingTimesActive": false,<br />
"heatingTimesActiveForComfort": true,<br />
"setBackTemperature": 18,<br />
"convectionFan1Active": false,<br />
"convectionFan1Level": 0,<br />
"convectionFan1Area": 0,<br />
"convectionFan2Active": false,<br />
"convectionFan2Level": 0,<br />
"convectionFan2Area": 0,<br />
"frostProtectionActive": false,<br />
"frostProtectionTemperature": 5<br />
},<br />
"sensors": {<br />
"statusError": 0,<br />
"statusWarning": 0,<br />
"statusService": 0,<br />
"statusMainState": 1,<br />
"statusSubState": 3,<br />
"statusFrostStarted": false,<br />
"inputFlameTemperature": 21,<br />
"inputRoomTemperature": 21,<br />
"inputExternalRequest": true,<br />
"outputDischargeMotor": 0,<br />
"outputInsertionMotor": 0,<br />
"outputIDFan": 0,<br />
"outputAirFlaps": 0,<br />
"outputIgnition": false,<br />
"parameterStoveTypeNumber": 13,<br />
"parameterVersionMainBoard": 223,<br />
"parameterVersionTFT": 223,<br />
"parameterRuntimePellets": 11,<br />
"parameterRuntimeLogs": 0,<br />
"parameterFeedRateTotal": 17,<br />
"parameterFeedRateService": 683,<br />
"parameterOnOffCycles": 2<br />
},<br />
"stoveType": "DOMO MultiAir",<br />
"stoveFeatures": {<br />
"multiAir1": true,<br />
"multiAir2": true,<br />
"insertionMotor": false,<br />
"airFlaps": false,<br />
"logRuntime": false<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
Data string to send to https://www.rika-firenet.com/api/client/xxxxxxxx/controls in order to set values:<br />
<syntaxhighlight lang="perl"><br />
heatingTimesActiveForComfort=true&frostProtectionTemperature=3&setBackTemperature=18&targetTemperature=24&convectionFan2Level=0&convectionFan2Active=false&convectionFan1Level=0&onOff=true&convectionFan1Active=false&convectionFan2Area=0&revision=1505550101&heatingTimesActive=false&convectionFan1Area=0&frostProtectionActive=false&operatingMode=2&heatingPower=65<br />
</syntaxhighlight><br />
<br />
Code in 99_myUtils.pm:<br />
<syntaxhighlight lang="perl"><br />
use JSON;<br />
...<br />
sub<br />
replaceJSON ($$) {<br />
my ($valToReplace, $value) = @_;<br />
<br />
#$value in the parameters is a default value<br />
#It has to be replaced through the real value nnn passed in the set command "set <device> valToset nnn"<br />
$value = InternalVal("Ofen", "value", $value);<br />
Log3 ("Ofen", 3, "replaceJSON Internalvalue: $value");<br />
<br />
#Force an update to avoid outdated revision number<br />
fhem ("get Ofen revision");<br />
<br />
#Get all the controls as json<br />
my $json = ReadingsVal("Ofen", "controlsJSON","");<br />
Log3 ("Ofen", 3, "replaceJSON configsJSON: $json");<br />
<br />
# When starting FHEM or rereading config, the reading controlsJSON is empty<br />
return if ($json eq ""); <br />
<br />
my $decoded = decode_json($json);<br />
my $result;<br />
for my $key ( keys %$decoded ) {<br />
$result .= "$key=";<br />
if ($key eq $valToReplace) {<br />
$result .= $value."&";<br />
} else {<br />
$result .= $decoded->{$key}."&";<br />
}<br />
}<br />
chop($result); #remove last &<br />
Log3("Ofen", 3, "replaceJSON Result: $result");<br />
return $result;<br />
}<br />
</syntaxhighlight><br />
<br />
Define stove in fhem:<br />
<syntaxhighlight lang="perl"><br />
defmod Ofen HTTPMOD https://www.rika-firenet.com/api/client/xxxxxxxx/status 60<br />
<br />
attr Ofen enableCookies 1<br />
attr Ofen reAuthRegex id="login"|Unauthorized<br />
attr Ofen sid01Data email=xx@xx&password=xx<br />
attr Ofen sid01URL https://www.rika-firenet.com/web/login<br />
<br />
attr Ofen reading01JSON sensors_inputRoomTemperature<br />
attr Ofen reading01Name RaumTemp<br />
attr Ofen reading02JSON controls_setBackTemperature<br />
attr Ofen reading02Name Absenkung<br />
attr Ofen reading03JSON controls_frostProtectionTemperature<br />
attr Ofen reading03Name Frostschutz<br />
attr Ofen reading10Name controlsJSON<br />
attr Ofen reading10Regex (?s)controls.*?({.*?})<br />
<br />
attr Ofen get09Name revision<br />
attr Ofen get09URL https://www.rika-firenet.com/api/client/xxxxxxxx/status<br />
<br />
attr Ofen setURL https://www.rika-firenet.com/api/client/xxxxxxxx/controls<br />
attr Ofen setData {{data}}<br />
attr Ofen replacement01Mode expression<br />
attr Ofen replacement01Regex {{data}}<br />
<br />
attr Ofen set11Name frostProtectionTemperature<br />
attr Ofen set11Replacement01Value replaceJSON("frostProtectionTemperature", 2)<br />
<br />
attr Ofen set12Name targetTemperature<br />
attr Ofen set12Replacement01Value replaceJSON("targetTemperature", 24)<br />
</syntaxhighlight><br />
<br />
A detailed explanation (in german) of the login process can be found here: [https://forum.fhem.de/index.php/topic,76220.msg682514.html#msg682514]<br />
and the explanation of the other parameters here: [https://forum.fhem.de/index.php/topic,76220.msg685710.html#msg685710]<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<syntaxhighlight lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</syntaxhighlight><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<syntaxhighlight lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</syntaxhighlight><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses. This is a tedious task but probably the best way to achieve a successful result. <br />
<br />
Let us consider an example. The Telekom Speedport W724V has a login-site that is famous for being cumbersome. Burp allows to monitor each step in the login procedure. In the case of a speedport the following steps occur:<br />
<br />
First burp shows that a get command is issued<br />
################################################################################################## <br />
GET / HTTP/1.1<br />
Host: speedport.ip<br />
Cache-Control: max-age=0<br />
Upgrade-Insecure-Requests: 1<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8<br />
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4<br />
Cookie: lang=de<br />
Connection: close <br />
In order to mimic the behavior of a real person calling the website HTTPMOD should copy all necessary steps. Host, Cookie and the GET-command are usually necessary. The same cannot be said of the User-Agent because the website can be called from any mobile or desktop computer. <br />
<br />
Then, the speedport will answer with a command that consists of several lines. By going through every line for every step in the login procedure one will finally arrive at the information that is necessary to successfully enter the login of the speedport (in case of the W724V, for example, it is necessary to copy a token called _httoken and to include the referer).<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<syntaxhighlight lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</syntaxhighlight><br />
<br />
A user command <br />
<syntaxhighlight lang="perl"><br />
set MyDevice Licht 1<br />
</syntaxhighlight><br />
<br />
will be translated into the http GET request<br />
<syntaxhighlight lang="perl"><br />
http://192.168.1.22/switch=1<br />
</syntaxhighlight><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<syntaxhighlight lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</syntaxhighlight><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<syntaxhighlight lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</syntaxhighlight><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<syntaxhighlight lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</syntaxhighlight><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set01TextArg<br />
</syntaxhighlight><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</syntaxhighlight><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<syntaxhighlight lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</syntaxhighlight><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<syntaxhighlight lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</syntaxhighlight><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</syntaxhighlight><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</syntaxhighlight><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<syntaxhighlight lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</syntaxhighlight><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of the corresponding get specific regex (get[0-9]*Regex), XPath or JSON attribute also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:Please note that this does not mean that get01CheckAllReadings will cause a get02Regex to be used. Only the corresponding get01Regex will be used but additionally all the readingXYRegex attributes.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
:<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=27318HTTPMOD2018-07-06T06:43:09Z<p>StefanStrobel: /* Example for a PoolManager 5: */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<syntaxhighlight lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</syntaxhighlight><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</syntaxhighlight><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
;enableCookies<br />
:enables the cookie handling inside HTTPMOD. It is advisable to always set this attribute and allow HTTPMOD to track the state of cookies and set them for following HTTP-requests<br />
<br />
;enforceGoodReadingNames<br />
:makes sure that HTTPMOD only creates readings that are allowd for Fhem (especially if reading names are dynamically created from JSON object names with extractAllJSON. It is advisable to always set this attribute.<br />
<br />
;handleRedirects<br />
:enables the redirect handling inside HTTPMOD which should be used together with the cookie handling of HTTPMOD. HTTPMOD will then automatically follow redirects from a web server and keep track of cookies at the same time. It is advisable to always set this attribute.<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<syntaxhighlight lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</syntaxhighlight><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
<br />
attr PM enableControlSet 1<br />
attr PM enableCookies 1<br />
attr PM enforceGoodReadingNames 1<br />
attr PM handleRedirects 1<br />
<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</syntaxhighlight><br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</syntaxhighlight><br />
<br />
the definition could be:<br />
<syntaxhighlight lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</syntaxhighlight><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<syntaxhighlight lang="perl"><br />
reading01Format %.1f<br />
</syntaxhighlight><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<syntaxhighlight lang="perl"><br />
reading01-2Format %.1f<br />
</syntaxhighlight><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<syntaxhighlight lang="perl"><br />
readingFormat %.1f<br />
</syntaxhighlight><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</syntaxhighlight><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</syntaxhighlight><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<syntaxhighlight lang="perl"><br />
attr PM getDecode UTF-8<br />
</syntaxhighlight><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
{{Randnotiz|RNTyp=y|RNText=Starting with version ''2018-02-10'' the internal that holds the HTTP response is no longer called '''''buf''''' but rather '''''httpbody''''', and it is only displayed when attribute '''''showBody''''' is set to "1".}}<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</syntaxhighlight><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<syntaxhighlight lang="perl"><br />
reading02Name Temp<br />
</syntaxhighlight><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</syntaxhighlight><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<syntaxhighlight lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</syntaxhighlight><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</syntaxhighlight><br />
<br />
with JSON you can write <br />
<syntaxhighlight lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</syntaxhighlight><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<syntaxhighlight lang="perl"><br />
attr test2 extractAllJSON<br />
</syntaxhighlight><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</syntaxhighlight><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<syntaxhighlight lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</syntaxhighlight><br />
<br />
then a configuration like <br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</syntaxhighlight><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</syntaxhighlight><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</syntaxhighlight><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</syntaxhighlight><br />
<br />
with XPath you can write <br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</syntaxhighlight><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</syntaxhighlight><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</syntaxhighlight> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<syntaxhighlight lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</syntaxhighlight> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</syntaxhighlight> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
'''Other example : steering a pellet stove from Rika'''<br />
<br />
The stove API of Rika on https://www.rika-firenet.com/web/ delivers a JSON string with all settings and values, and can be piloted with a data string containing all the "set" values at once.<br />
<br />
Delivered JSON on get https://www.rika-firenet.com/api/client/xxxxxxxx/status : (xxxxxxxx must be replaced with the unique stove ID)<br />
<syntaxhighlight lang="perl"><br />
{<br />
"name": "Vorzimmer",<br />
"stoveID": "xxxxxxxxx",<br />
"lastSeenMinutes": 1,<br />
"lastConfirmedRevision": 1504385700,<br />
"controls": {<br />
"revision": 1504385700,<br />
"onOff": true,<br />
"operatingMode": 2,<br />
"heatingPower": 65,<br />
"targetTemperature": 24,<br />
"heatingTimesActive": false,<br />
"heatingTimesActiveForComfort": true,<br />
"setBackTemperature": 18,<br />
"convectionFan1Active": false,<br />
"convectionFan1Level": 0,<br />
"convectionFan1Area": 0,<br />
"convectionFan2Active": false,<br />
"convectionFan2Level": 0,<br />
"convectionFan2Area": 0,<br />
"frostProtectionActive": false,<br />
"frostProtectionTemperature": 5<br />
},<br />
"sensors": {<br />
"statusError": 0,<br />
"statusWarning": 0,<br />
"statusService": 0,<br />
"statusMainState": 1,<br />
"statusSubState": 3,<br />
"statusFrostStarted": false,<br />
"inputFlameTemperature": 21,<br />
"inputRoomTemperature": 21,<br />
"inputExternalRequest": true,<br />
"outputDischargeMotor": 0,<br />
"outputInsertionMotor": 0,<br />
"outputIDFan": 0,<br />
"outputAirFlaps": 0,<br />
"outputIgnition": false,<br />
"parameterStoveTypeNumber": 13,<br />
"parameterVersionMainBoard": 223,<br />
"parameterVersionTFT": 223,<br />
"parameterRuntimePellets": 11,<br />
"parameterRuntimeLogs": 0,<br />
"parameterFeedRateTotal": 17,<br />
"parameterFeedRateService": 683,<br />
"parameterOnOffCycles": 2<br />
},<br />
"stoveType": "DOMO MultiAir",<br />
"stoveFeatures": {<br />
"multiAir1": true,<br />
"multiAir2": true,<br />
"insertionMotor": false,<br />
"airFlaps": false,<br />
"logRuntime": false<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
Data string to send to https://www.rika-firenet.com/api/client/xxxxxxxx/controls in order to set values:<br />
<syntaxhighlight lang="perl"><br />
heatingTimesActiveForComfort=true&frostProtectionTemperature=3&setBackTemperature=18&targetTemperature=24&convectionFan2Level=0&convectionFan2Active=false&convectionFan1Level=0&onOff=true&convectionFan1Active=false&convectionFan2Area=0&revision=1505550101&heatingTimesActive=false&convectionFan1Area=0&frostProtectionActive=false&operatingMode=2&heatingPower=65<br />
</syntaxhighlight><br />
<br />
Code in 99_myUtils.pm:<br />
<syntaxhighlight lang="perl"><br />
use JSON;<br />
...<br />
sub<br />
replaceJSON ($$) {<br />
my ($valToReplace, $value) = @_;<br />
<br />
#$value in the parameters is a default value<br />
#It has to be replaced through the real value nnn passed in the set command "set <device> valToset nnn"<br />
$value = InternalVal("Ofen", "value", $value);<br />
Log3 ("Ofen", 3, "replaceJSON Internalvalue: $value");<br />
<br />
#Force an update to avoid outdated revision number<br />
fhem ("get Ofen revision");<br />
<br />
#Get all the controls as json<br />
my $json = ReadingsVal("Ofen", "controlsJSON","");<br />
Log3 ("Ofen", 3, "replaceJSON configsJSON: $json");<br />
<br />
# When starting FHEM or rereading config, the reading controlsJSON is empty<br />
return if ($json eq ""); <br />
<br />
my $decoded = decode_json($json);<br />
my $result;<br />
for my $key ( keys %$decoded ) {<br />
$result .= "$key=";<br />
if ($key eq $valToReplace) {<br />
$result .= $value."&";<br />
} else {<br />
$result .= $decoded->{$key}."&";<br />
}<br />
}<br />
chop($result); #remove last &<br />
Log3("Ofen", 3, "replaceJSON Result: $result");<br />
return $result;<br />
}<br />
</syntaxhighlight><br />
<br />
Define stove in fhem:<br />
<syntaxhighlight lang="perl"><br />
defmod Ofen HTTPMOD https://www.rika-firenet.com/api/client/xxxxxxxx/status 60<br />
<br />
attr Ofen enableCookies 1<br />
attr Ofen reAuthRegex id="login"|Unauthorized<br />
attr Ofen sid01Data email=xx@xx&password=xx<br />
attr Ofen sid01URL https://www.rika-firenet.com/web/login<br />
<br />
attr Ofen reading01JSON sensors_inputRoomTemperature<br />
attr Ofen reading01Name RaumTemp<br />
attr Ofen reading02JSON controls_setBackTemperature<br />
attr Ofen reading02Name Absenkung<br />
attr Ofen reading03JSON controls_frostProtectionTemperature<br />
attr Ofen reading03Name Frostschutz<br />
attr Ofen reading10Name controlsJSON<br />
attr Ofen reading10Regex (?s)controls.*?({.*?})<br />
<br />
attr Ofen get09Name revision<br />
attr Ofen get09URL https://www.rika-firenet.com/api/client/xxxxxxxx/status<br />
<br />
attr Ofen setURL https://www.rika-firenet.com/api/client/xxxxxxxx/controls<br />
attr Ofen setData {{data}}<br />
attr Ofen replacement01Mode expression<br />
attr Ofen replacement01Regex {{data}}<br />
<br />
attr Ofen set11Name frostProtectionTemperature<br />
attr Ofen set11Replacement01Value replaceJSON("frostProtectionTemperature", 2)<br />
<br />
attr Ofen set12Name targetTemperature<br />
attr Ofen set12Replacement01Value replaceJSON("targetTemperature", 24)<br />
</syntaxhighlight><br />
<br />
A detailed explanation (in german) of the login process can be found here: [https://forum.fhem.de/index.php/topic,76220.msg682514.html#msg682514]<br />
and the explanation of the other parameters here: [https://forum.fhem.de/index.php/topic,76220.msg685710.html#msg685710]<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<syntaxhighlight lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</syntaxhighlight><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<syntaxhighlight lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</syntaxhighlight><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses. This is a tedious task but probably the best way to achieve a successful result. <br />
<br />
Let us consider an example. The Telekom Speedport W724V has a login-site that is famous for being cumbersome. Burp allows to monitor each step in the login procedure. In the case of a speedport the following steps occur:<br />
<br />
First burp shows that a get command is issued<br />
################################################################################################## <br />
GET / HTTP/1.1<br />
Host: speedport.ip<br />
Cache-Control: max-age=0<br />
Upgrade-Insecure-Requests: 1<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8<br />
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4<br />
Cookie: lang=de<br />
Connection: close <br />
In order to mimic the behavior of a real person calling the website HTTPMOD should copy all necessary steps. Host, Cookie and the GET-command are usually necessary. The same cannot be said of the User-Agent because the website can be called from any mobile or desktop computer. <br />
<br />
Then, the speedport will answer with a command that consists of several lines. By going through every line for every step in the login procedure one will finally arrive at the information that is necessary to successfully enter the login of the speedport (in case of the W724V, for example, it is necessary to copy a token called _httoken and to include the referer).<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<syntaxhighlight lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</syntaxhighlight><br />
<br />
A user command <br />
<syntaxhighlight lang="perl"><br />
set MyDevice Licht 1<br />
</syntaxhighlight><br />
<br />
will be translated into the http GET request<br />
<syntaxhighlight lang="perl"><br />
http://192.168.1.22/switch=1<br />
</syntaxhighlight><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<syntaxhighlight lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</syntaxhighlight><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<syntaxhighlight lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</syntaxhighlight><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<syntaxhighlight lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</syntaxhighlight><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set01TextArg<br />
</syntaxhighlight><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</syntaxhighlight><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<syntaxhighlight lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</syntaxhighlight><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<syntaxhighlight lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</syntaxhighlight><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</syntaxhighlight><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</syntaxhighlight><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<syntaxhighlight lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</syntaxhighlight><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of the corresponding get specific regex (get[0-9]*Regex), XPath or JSON attribute also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:Please note that this does not mean that get01CheckAllReadings will cause a get02Regex to be used. Only the corresponding get01Regex will be used but additionally all the readingXYRegex attributes.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
:<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=27317HTTPMOD2018-07-06T06:41:59Z<p>StefanStrobel: /* simple Attributes */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<syntaxhighlight lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</syntaxhighlight><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</syntaxhighlight><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
;enableCookies<br />
:enables the cookie handling inside HTTPMOD. It is advisable to always set this attribute and allow HTTPMOD to track the state of cookies and set them for following HTTP-requests<br />
<br />
;enforceGoodReadingNames<br />
:makes sure that HTTPMOD only creates readings that are allowd for Fhem (especially if reading names are dynamically created from JSON object names with extractAllJSON. It is advisable to always set this attribute.<br />
<br />
;handleRedirects<br />
:enables the redirect handling inside HTTPMOD which should be used together with the cookie handling of HTTPMOD. HTTPMOD will then automatically follow redirects from a web server and keep track of cookies at the same time. It is advisable to always set this attribute.<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<syntaxhighlight lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</syntaxhighlight><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<syntaxhighlight lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</syntaxhighlight><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<syntaxhighlight lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</syntaxhighlight><br />
<br />
the definition could be:<br />
<syntaxhighlight lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</syntaxhighlight><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<syntaxhighlight lang="perl"><br />
reading01Format %.1f<br />
</syntaxhighlight><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<syntaxhighlight lang="perl"><br />
reading01-2Format %.1f<br />
</syntaxhighlight><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<syntaxhighlight lang="perl"><br />
readingFormat %.1f<br />
</syntaxhighlight><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</syntaxhighlight><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<syntaxhighlight lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</syntaxhighlight><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<syntaxhighlight lang="perl"><br />
attr PM getDecode UTF-8<br />
</syntaxhighlight><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
{{Randnotiz|RNTyp=y|RNText=Starting with version ''2018-02-10'' the internal that holds the HTTP response is no longer called '''''buf''''' but rather '''''httpbody''''', and it is only displayed when attribute '''''showBody''''' is set to "1".}}<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</syntaxhighlight><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<syntaxhighlight lang="perl"><br />
reading02Name Temp<br />
</syntaxhighlight><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<syntaxhighlight lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</syntaxhighlight><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<syntaxhighlight lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</syntaxhighlight><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</syntaxhighlight><br />
<br />
with JSON you can write <br />
<syntaxhighlight lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</syntaxhighlight><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<syntaxhighlight lang="perl"><br />
attr test2 extractAllJSON<br />
</syntaxhighlight><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<syntaxhighlight lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</syntaxhighlight><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<syntaxhighlight lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</syntaxhighlight><br />
<br />
then a configuration like <br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</syntaxhighlight><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</syntaxhighlight><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</syntaxhighlight><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<syntaxhighlight lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</syntaxhighlight><br />
<br />
with XPath you can write <br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</syntaxhighlight><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<syntaxhighlight lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</syntaxhighlight><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</syntaxhighlight> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<syntaxhighlight lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</syntaxhighlight> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</syntaxhighlight> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
'''Other example : steering a pellet stove from Rika'''<br />
<br />
The stove API of Rika on https://www.rika-firenet.com/web/ delivers a JSON string with all settings and values, and can be piloted with a data string containing all the "set" values at once.<br />
<br />
Delivered JSON on get https://www.rika-firenet.com/api/client/xxxxxxxx/status : (xxxxxxxx must be replaced with the unique stove ID)<br />
<syntaxhighlight lang="perl"><br />
{<br />
"name": "Vorzimmer",<br />
"stoveID": "xxxxxxxxx",<br />
"lastSeenMinutes": 1,<br />
"lastConfirmedRevision": 1504385700,<br />
"controls": {<br />
"revision": 1504385700,<br />
"onOff": true,<br />
"operatingMode": 2,<br />
"heatingPower": 65,<br />
"targetTemperature": 24,<br />
"heatingTimesActive": false,<br />
"heatingTimesActiveForComfort": true,<br />
"setBackTemperature": 18,<br />
"convectionFan1Active": false,<br />
"convectionFan1Level": 0,<br />
"convectionFan1Area": 0,<br />
"convectionFan2Active": false,<br />
"convectionFan2Level": 0,<br />
"convectionFan2Area": 0,<br />
"frostProtectionActive": false,<br />
"frostProtectionTemperature": 5<br />
},<br />
"sensors": {<br />
"statusError": 0,<br />
"statusWarning": 0,<br />
"statusService": 0,<br />
"statusMainState": 1,<br />
"statusSubState": 3,<br />
"statusFrostStarted": false,<br />
"inputFlameTemperature": 21,<br />
"inputRoomTemperature": 21,<br />
"inputExternalRequest": true,<br />
"outputDischargeMotor": 0,<br />
"outputInsertionMotor": 0,<br />
"outputIDFan": 0,<br />
"outputAirFlaps": 0,<br />
"outputIgnition": false,<br />
"parameterStoveTypeNumber": 13,<br />
"parameterVersionMainBoard": 223,<br />
"parameterVersionTFT": 223,<br />
"parameterRuntimePellets": 11,<br />
"parameterRuntimeLogs": 0,<br />
"parameterFeedRateTotal": 17,<br />
"parameterFeedRateService": 683,<br />
"parameterOnOffCycles": 2<br />
},<br />
"stoveType": "DOMO MultiAir",<br />
"stoveFeatures": {<br />
"multiAir1": true,<br />
"multiAir2": true,<br />
"insertionMotor": false,<br />
"airFlaps": false,<br />
"logRuntime": false<br />
}<br />
}<br />
</syntaxhighlight><br />
<br />
Data string to send to https://www.rika-firenet.com/api/client/xxxxxxxx/controls in order to set values:<br />
<syntaxhighlight lang="perl"><br />
heatingTimesActiveForComfort=true&frostProtectionTemperature=3&setBackTemperature=18&targetTemperature=24&convectionFan2Level=0&convectionFan2Active=false&convectionFan1Level=0&onOff=true&convectionFan1Active=false&convectionFan2Area=0&revision=1505550101&heatingTimesActive=false&convectionFan1Area=0&frostProtectionActive=false&operatingMode=2&heatingPower=65<br />
</syntaxhighlight><br />
<br />
Code in 99_myUtils.pm:<br />
<syntaxhighlight lang="perl"><br />
use JSON;<br />
...<br />
sub<br />
replaceJSON ($$) {<br />
my ($valToReplace, $value) = @_;<br />
<br />
#$value in the parameters is a default value<br />
#It has to be replaced through the real value nnn passed in the set command "set <device> valToset nnn"<br />
$value = InternalVal("Ofen", "value", $value);<br />
Log3 ("Ofen", 3, "replaceJSON Internalvalue: $value");<br />
<br />
#Force an update to avoid outdated revision number<br />
fhem ("get Ofen revision");<br />
<br />
#Get all the controls as json<br />
my $json = ReadingsVal("Ofen", "controlsJSON","");<br />
Log3 ("Ofen", 3, "replaceJSON configsJSON: $json");<br />
<br />
# When starting FHEM or rereading config, the reading controlsJSON is empty<br />
return if ($json eq ""); <br />
<br />
my $decoded = decode_json($json);<br />
my $result;<br />
for my $key ( keys %$decoded ) {<br />
$result .= "$key=";<br />
if ($key eq $valToReplace) {<br />
$result .= $value."&";<br />
} else {<br />
$result .= $decoded->{$key}."&";<br />
}<br />
}<br />
chop($result); #remove last &<br />
Log3("Ofen", 3, "replaceJSON Result: $result");<br />
return $result;<br />
}<br />
</syntaxhighlight><br />
<br />
Define stove in fhem:<br />
<syntaxhighlight lang="perl"><br />
defmod Ofen HTTPMOD https://www.rika-firenet.com/api/client/xxxxxxxx/status 60<br />
<br />
attr Ofen enableCookies 1<br />
attr Ofen reAuthRegex id="login"|Unauthorized<br />
attr Ofen sid01Data email=xx@xx&password=xx<br />
attr Ofen sid01URL https://www.rika-firenet.com/web/login<br />
<br />
attr Ofen reading01JSON sensors_inputRoomTemperature<br />
attr Ofen reading01Name RaumTemp<br />
attr Ofen reading02JSON controls_setBackTemperature<br />
attr Ofen reading02Name Absenkung<br />
attr Ofen reading03JSON controls_frostProtectionTemperature<br />
attr Ofen reading03Name Frostschutz<br />
attr Ofen reading10Name controlsJSON<br />
attr Ofen reading10Regex (?s)controls.*?({.*?})<br />
<br />
attr Ofen get09Name revision<br />
attr Ofen get09URL https://www.rika-firenet.com/api/client/xxxxxxxx/status<br />
<br />
attr Ofen setURL https://www.rika-firenet.com/api/client/xxxxxxxx/controls<br />
attr Ofen setData {{data}}<br />
attr Ofen replacement01Mode expression<br />
attr Ofen replacement01Regex {{data}}<br />
<br />
attr Ofen set11Name frostProtectionTemperature<br />
attr Ofen set11Replacement01Value replaceJSON("frostProtectionTemperature", 2)<br />
<br />
attr Ofen set12Name targetTemperature<br />
attr Ofen set12Replacement01Value replaceJSON("targetTemperature", 24)<br />
</syntaxhighlight><br />
<br />
A detailed explanation (in german) of the login process can be found here: [https://forum.fhem.de/index.php/topic,76220.msg682514.html#msg682514]<br />
and the explanation of the other parameters here: [https://forum.fhem.de/index.php/topic,76220.msg685710.html#msg685710]<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<syntaxhighlight lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</syntaxhighlight><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<syntaxhighlight lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</syntaxhighlight><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses. This is a tedious task but probably the best way to achieve a successful result. <br />
<br />
Let us consider an example. The Telekom Speedport W724V has a login-site that is famous for being cumbersome. Burp allows to monitor each step in the login procedure. In the case of a speedport the following steps occur:<br />
<br />
First burp shows that a get command is issued<br />
################################################################################################## <br />
GET / HTTP/1.1<br />
Host: speedport.ip<br />
Cache-Control: max-age=0<br />
Upgrade-Insecure-Requests: 1<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8<br />
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4<br />
Cookie: lang=de<br />
Connection: close <br />
In order to mimic the behavior of a real person calling the website HTTPMOD should copy all necessary steps. Host, Cookie and the GET-command are usually necessary. The same cannot be said of the User-Agent because the website can be called from any mobile or desktop computer. <br />
<br />
Then, the speedport will answer with a command that consists of several lines. By going through every line for every step in the login procedure one will finally arrive at the information that is necessary to successfully enter the login of the speedport (in case of the W724V, for example, it is necessary to copy a token called _httoken and to include the referer).<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<syntaxhighlight lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</syntaxhighlight><br />
<br />
A user command <br />
<syntaxhighlight lang="perl"><br />
set MyDevice Licht 1<br />
</syntaxhighlight><br />
<br />
will be translated into the http GET request<br />
<syntaxhighlight lang="perl"><br />
http://192.168.1.22/switch=1<br />
</syntaxhighlight><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<syntaxhighlight lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</syntaxhighlight><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<syntaxhighlight lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</syntaxhighlight><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<syntaxhighlight lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</syntaxhighlight><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set01TextArg<br />
</syntaxhighlight><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<syntaxhighlight lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</syntaxhighlight><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<syntaxhighlight lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</syntaxhighlight><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<syntaxhighlight lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</syntaxhighlight><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</syntaxhighlight><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<syntaxhighlight lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</syntaxhighlight><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<syntaxhighlight lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</syntaxhighlight><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of the corresponding get specific regex (get[0-9]*Regex), XPath or JSON attribute also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:Please note that this does not mean that get01CheckAllReadings will cause a get02Regex to be used. Only the corresponding get01Regex will be used but additionally all the readingXYRegex attributes.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
:<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=24305ArduCounter2018-01-04T15:50:07Z<p>StefanStrobel: /* Attributes */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
This module implements an Interface to an Arduino based counter for pulses on any input pin of an Arduino Uno, Nano or similar device like a Jeenode.<br />
The typical use case is an S0-Interface on an energy meter or water meter.<br />
<br />
Counters are configured with attributes that define which Arduino pins should count pulses and in which intervals the Arduino board should report the current counts.<br />
<br />
The Arduino sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module creates readings for pulse counts, consumption and optionally also a pulse history with pulse lengths and gaps of the last 20 pulses.<br />
<br />
== Availability == <br />
The module has been checked in to the FHEM svn. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
The default baudrate of the firmware is 38400 since Version 1.4 of the sketch.<br />
<br />
Example:<br />
<pre>define AC ArduCounter /dev/ttyUSB2</pre><br />
<br />
== Configuration of the module ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <pre>attr AC pinX falling pullup 30</pre> <br />
<br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can define if the signals to be counted start with rising or falling edges. <br />
The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
The last argument is also optional and specifies a minimal pulse length in milliseconds. In this case the first argument (e.g. falling) means that an impulse starts with a falling edge from 1 to 0 and ends when the signal changes back from 0 to 1.<br />
<br />
== Example: ==<br />
<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup 5<br />
attr AC pinD5 falling pullup 30<br />
attr AC verboseReadings5<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines three counters connected to the pins D4, D5 and D5.<br />
<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
<br />
For D4 and D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. If this time is longer than the specified 5 or 30 milliseconds then the impulse is counted. If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
verboseReadings5 causes the module to create additional readings like the pulse history which shows length and gaps between the last pulses.<br />
<br />
For pin D6 the arduino does not check pulse lengths and counts every time when the signal changes from 0 to 1.<br />
<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw<br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware<br />
:onto the device. This command needs avrdude to be installed. <br />
:The attribute flashCommand specifies how avrdude is called. <br />
:If it is not modifed then the module sets it to <br />
:avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. So normally there is no need to modify this attribute.<br />
<br />
:Depending on your specific Arduino board however, you might need to insert <code>-b 57600</code> in the flash Command.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. <br />
:Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
;longX<br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
<br />
;interpolatedLongX<br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
<br />
;rejectX<br />
:counts rejected pulses that are shorter than the specified minimal pulse length. <br />
<br />
;pinHistoryX<br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. <br />
:Also the total number of impulses recorded here is limited to 20 for all pins together. <br />
:The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action.<br />
:-36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNameLongCount[0-9]+ <br />
:Change the name of the counter reading longX to something more meaningful.<br />
<br />
;readingNameInterpolatedCount[0-9]+ <br />
:Change the name of the counter reading InterpolatedLongX to something more meaningful.<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingFactor[0-9]+ <br />
:Override the factor attribute for this individual pin.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=24181ArduCounter2018-01-03T10:09:43Z<p>StefanStrobel: Aktualisierungen für die neue Version</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
This module implements an Interface to an Arduino based counter for pulses on any input pin of an Arduino Uno, Nano or similar device like a Jeenode.<br />
The typical use case is an S0-Interface on an energy meter or water meter.<br />
<br />
Counters are configured with attributes that define which Arduino pins should count pulses and in which intervals the Arduino board should report the current counts.<br />
<br />
The Arduino sketch that works with this module uses pin change interrupts so it can efficiently count pulses on all available input pins.<br />
The module creates readings for pulse counts, consumption and optionally also a pulse history with pulse lengths and gaps of the last 20 pulses.<br />
<br />
== Availability == <br />
The module has been checked in to the FHEM svn. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
The default baudrate of the firmware is 38400 since Version 1.4 of the sketch.<br />
<br />
Example:<br />
<pre>define AC ArduCounter /dev/ttyUSB2</pre><br />
<br />
== Configuration of the module ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <pre>attr AC pinX falling pullup 30</pre> <br />
<br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can define if the signals to be counted start with rising or falling edges. <br />
The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
The last argument is also optional and specifies a minimal pulse length in milliseconds. In this case the first argument (e.g. falling) means that an impulse starts with a falling edge from 1 to 0 and ends when the signal changes back from 0 to 1.<br />
<br />
== Example: ==<br />
<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup 5<br />
attr AC pinD5 falling pullup 30<br />
attr AC verboseReadings5<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
This defines three counters connected to the pins D4, D5 and D5.<br />
<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
<br />
For D4 and D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. If this time is longer than the specified 5 or 30 milliseconds then the impulse is counted. If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
verboseReadings5 causes the module to create additional readings like the pulse history which shows length and gaps between the last pulses.<br />
<br />
For pin D6 the arduino does not check pulse lengths and counts every time when the signal changes from 0 to 1.<br />
<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
<br />
The module has been tested with 14 inputs of an Arduino Uno counting in parallel and pulses as short as 3 milliseconds.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw<br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware<br />
:onto the device. This command needs avrdude to be installed. <br />
:The attribute flashCommand specifies how avrdude is called. <br />
:If it is not modifed then the module sets it to <br />
:avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. So normally there is no need to modify this attribute.<br />
<br />
:Depending on your specific Arduino board however, you might need to insert <code>-b 57600</code> in the flash Command.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. <br />
:Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
;longX<br />
:long count which keeps on counting up after fhem restarts whereas the pin.* count is only a temporary internal count that starts at 0 when the arduino board starts.<br />
<br />
;interpolatedLongX<br />
:like long.* but when the Arduino restarts the potentially missed pulses are interpolated based on the pulse rate before the restart and after the restart.<br />
<br />
;rejectX<br />
:counts rejected pulses that are shorter than the specified minimal pulse length. <br />
<br />
;pinHistoryX<br />
:shows detailed information of the last pulses. This is only available when a minimal pulse length is specified for this pin. <br />
:Also the total number of impulses recorded here is limited to 20 for all pins together. <br />
:The output looks like -36/7:0C, -29/7:1G, -22/8:0C, -14/7:1G, -7/7:0C, 0/7:1G<br />
:The first number is the relative time in milliseconds when the input level changed, followed by the length in milliseconds, the level and the internal action.<br />
:-36/7:0C for example means that 36 milliseconds before the reporting started, the input changed to 0V, stayed there for 7 milliseconds and this was counted.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingFactor[0-9]+ <br />
:Override the factor attribute for this individual pin.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=23900ArduCounter2017-12-29T10:59:04Z<p>StefanStrobel: /* Set-Commands */</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ArduCounter module works together with an arduino board connected to FHEM via usb that runs the provided ArduCounter.ino sketch to count impulses from energy meters with an S0 interface or similar devices in configurable intervals. <br />
<br />
The arduino board counts impulses on every configured input pin using pin change interrupts. It reports the counted number of impulses per pin together with the time between the first and last impulse per pin.<br />
<br />
The module converts this to readings like power consumption and it is not affected by delays or the load of your FHEM server.<br />
Counters are configured using attributes that define which Arduino pins should count pulses and in which intervals the counted numbers should be reported from the arduino borad to FHEM.<br />
<br />
== Availability == <br />
The module has been checked in to the contrib area. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
The default baudrate of the firmware is 38400 since Version 1.4 of the sketch.<br />
<br />
Example:<br />
<pre>define AC ArduCounter /dev/ttyUSB2</pre><br />
<br />
== Configuration of the module ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <pre>attr AC pinX falling pullup 30</pre> <br />
<br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can define if the signals to be counted start with rising or falling edges. <br />
The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
The last argument is also optional and specifies a minimal pulse length in milliseconds. In this case the first argument (e.g. falling) means that an impulse starts with a falling edge from 1 to 0 and ends when the signal changes back from 0 to 1.<br />
<br />
== Example: ==<br />
<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup<br />
attr AC pinD5 falling pullup 30<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
this defines three counters connected to the pins D4, D5 and D5.<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
For D4 every falling edge of the signal (when the input changes from 1 to 0) is counted.<br />
For D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 30 milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
For pin D6 the ardiono counts every time when the signal changes from 0 to 1.<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw<br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware<br />
:onto the device. This command needs avrdude to be installed. <br />
:The attribute flashCommand specifies how avrdude is called. <br />
:If it is not modifed then the module sets it to <br />
:avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. So normally there is no need to modify this attribute.<br />
<br />
:Depending on your specific Arduino board however, you might need to insert <code>-b 57600</code> in the flash Command.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. <br />
:Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingFactor[0-9]+ <br />
:Override the factor attribute for this individual pin.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=21640HTTPMOD2017-06-03T12:47:32Z<p>StefanStrobel: /* All attributes */ minor clarification</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses. This is a tedious task but probably the best way to achieve a successful result. <br />
<br />
Let us consider an example. The Telekom Speedport W724V has a login-site that is famous for being cumbersome. Burp allows to monitor each step in the login procedure. In the case of a speedport the following steps occur:<br />
<br />
First burp shows that a get command is issued<br />
################################################################################################## <br />
GET / HTTP/1.1<br />
Host: speedport.ip<br />
Cache-Control: max-age=0<br />
Upgrade-Insecure-Requests: 1<br />
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/57.0.2987.133 Safari/537.36<br />
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8<br />
Accept-Language: de-DE,de;q=0.8,en-US;q=0.6,en;q=0.4<br />
Cookie: lang=de<br />
Connection: close <br />
In order to mimic the behavior of a real person calling the website HTTPMOD should copy all necessary steps. Host, Cookie and the GET-command are usually necessary. The same cannot be said of the User-Agent because the website can be called from any mobile or desktop computer. <br />
<br />
Then, the speedport will answer with a command that consists of several lines. By going through every line for every step in the login procedure one will finally arrive at the information that is necessary to successfully enter the login of the speedport (in case of the W724V, for example, it is necessary to copy a token called _httoken and to include the referer).<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of the corresponding get specific regex (get[0-9]*Regex), XPath or JSON attribute also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:Please note that this does not mean that get01CheckAllReadings will cause a get02Regex to be used. Only the corresponding get01Regex will be used but additionally all the readingXYRegex attributes.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
:<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Modbus&diff=21452Modbus2017-05-07T09:43:21Z<p>StefanStrobel: Dokumentation ergänzt und korrigiert</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Library or physical device to extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=Modbus<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_Modbus.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
<br />
Modbus defines a physical modbus interface and library functions to be called from other logical modules / devices.<br />
This low level module takes care of the communication with modbus devices and provides Get, Set and cyclic polling of Readings as well as formatting and input validation functions.<br />
<br />
The logical device modules for individual machines only need to define the supported modbus function codes and objects of the machine with the modbus interface in data structures. These data structures are then used by this low level module to implement Set, Get and automatic updateing of readings in a given interval.<br />
<br />
The Modbus module supports Modbus RTU over serial / RS485 lines as well as Modbus TCP and Modbus RTU over TCP. It defines read / write functions for Modbus holding registers, input registers, coils and discrete inputs.<br />
<br />
See [[ModbusAttr]] if you don't want to use a library to develop your own module and if you are looking for a generic Modbus Module instead that can be configured with attributes.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module requires the Device::SerialPort or Win32::SerialPort module if you want to communicate with modbus devices over a serial line.<br />
<br />
== Define of a modbus interface device for serial communication ==<br />
<pre><br />
define <name> Modbus <device><br />
</pre><br />
<br />
A define of a physical device based on this module is only necessary if a shared physical device like a RS485 USB adapter is used. <br />
In the case Modbus TCP this module will be used as a library for other modules that define all the data objects and no define of the base module is needed.<br />
<br />
Example:<br />
<pre><br />
define ModBusLine Modbus /dev/ttyUSB1@9600<br />
</pre><br />
<br />
In this example the module opens the given serial interface and other logical modules like [[ModbusAttr]] or [[ModbusSET]] can access several Modbus devices connected to this bus concurrently.<br />
<br />
== Set-Commands ==<br />
this low level device module doesn't provide set commands for itself but implements set <br />
for logical device modules that make use of this module as a library. See ModbusSET for example.<br />
<br />
== Get-Commands ==<br />
this low level device module doesn't provide get commands for itself but implements get <br />
for logical device modules that make use of this module as a library.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;queueDelay<br />
:modify the delay used when sending requests to the device from the internal queue, defaults to 1 second<br />
<br />
;queueMax<br />
:max length of the send queue, defaults to 100<br><br />
<br />
;clientSwitchDelay<br />
:defines a delay that is always enforced between the last read from the bus and the next send to the bus <br />
:for all connected devices, but only if the next send goes to a different device than the last one<br />
<br />
;dropQueueDoubles<br />
:prevents new request to be queued if the same request is already in the send queue<br />
<br />
;skipGarbage<br />
:if set to 1 this attribute will enhance the way the module treats Modbus response frames (RTU over serial lines) <br />
:that look as if they have a wrong Modbus id as their first byte. <br />
:If skipGarbage is set to 1 then the module will skip all bytes until a byte with the expected modbus id is seen. <br />
:Under normal circumstances this behavior should not do any harm and lead to more robustness. <br />
:However since it changes the original behavior of this module it has to be turned on explicitely.<br />
:For Modbus ASCII it skips bytes until the expected starting byte ":" is seen.<br />
<br />
;profileInterval<br />
:if set to something non zero it is the time period in seconds for which the module will create bus usage statistics. <br />
:Please note that this number should be at least twice as big as the interval used for requesting values in logical devices <br />
:that use this physical device<br />
:The bus usage statistics create the following readings:<br />
:* Profiler_Delay_sum<br />
::seconds used as delays to implement the defined sendDelay and commDelay<br />
:*Profiler_Fhem_sum<br />
::seconds spend processing in the module<br />
:*Profiler_Idle_sum<br />
::idle time <br />
:*Profiler_Read_sum<br />
::seconds spent reading and validating the data read<br />
:*Profiler_Send_sum<br />
::seconds spent preparing and sending data<br />
:*Profiler_Wait_sum<br />
::seconds waiting for a response to a request<br />
:*Statistics_Requests<br />
::number of requests sent<br />
:*Statistics_Timeouts<br />
::timeouts encountered<br />
<br />
<br />
== Writing modules for devices using this module as a library ==<br />
<br />
Writing a module for a physical device with modbus interface is easy when you use the 98_Modbus.pm module as a library. <br />
To use this module as a library for other fhem modules you only have to define a data structure that defines the mapping between modbus data objects (holding registers, input registers, coils or discrete inputs) and fhem readings.<br />
Additionally the module needs to contain a few <code>package</code> and <code>use</code> statements and an initialize function at the beginning, that assigns a few special variables to point to functions of the Modbus base module. <br />
<br />
Example for a module that is called ModbusSET:<br />
<pre><br />
package main;<br />
use strict;<br />
use warnings;<br />
<br />
sub ModbusSET_Initialize($)<br />
{<br />
my ($modHash) = @_;<br />
LoadModule "Modbus";<br />
require "$attr{global}{modpath}/FHEM/DevIo.pm";<br />
<br />
$modHash->{parseInfo} = \%SET10parseInfo; # defines registers, inputs, coils etc. for this Modbus Device<br />
$modHash->{deviceInfo} = \%SET10deviceInfo; # defines properties of the device, defaults and supported function codes<br />
<br />
ModbusLD_Initialize($modHash); # Generic function of the Modbus module does the rest<br />
<br />
$modHash->{AttrList} = $modHash->{AttrList} . " " . # Standard Attributes like IODEv etc <br />
$modHash->{ObjAttrList} . " " . # Attributes to add or overwrite parseInfo definitions<br />
$modHash->{DevAttrList}; # Attributes to add or overwrite devInfo definitions<br />
<br />
}<br />
</pre><br />
<br />
The name of the initialize-Function has to match the name of the module. In the above example this is <code>ModbusSET_Initialize</code>. Most of the steps needed in an initialize function are provided by the library function <code>ModbusLD_Initialize</code>. This function tells fhem to use the library functions for <code>define</code>, <code>set</code>, <code>get</code> and other typical functions in a module. See [[DevelopmentModuleIntro]] for more background information on writing fhem modules if you are curious.<br />
<br />
=== Introduction to the parseInfo structure ===<br />
<br />
Typically the data structure to map between data objects of the modbus device and fhem readings is named <code>parseInfo</code> with a part of the name of the module itself as prefix. In the example of the module 98_ModbusSET.pm which uses Modbus.pm to implement a module for SET Silent 10 heat pumps, the structure is called <code>SET10parseInfo</code>.<br />
<br />
This strucure contains keys with values that directly correspond to attributes which can be used with the module ModbusAttr so it is advisable to prototype a new module with ModbusAttr and then translate the attributes to entries in the parseInfo structure. The values in the parseInfo structure can later still be overwritten / extended with the attributes documented in ModbusAttr.<br />
<br />
As an example a very simple definition of a parseInfo structure for a heat pump could look like this:<br />
<br />
<pre><br />
my %XYparseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
},<br />
"h258" => { reading => "Temp_Wasser_Aus",<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
set => 1, # this value can be set<br />
}<br />
);<br />
</pre><br />
<br />
the corresponding attributes for ModbusAttr for prototyping or overwriting values would be obj-h256-reading, obj-h258-reading, obj-h770-reading, obj-h770-min and so on.<br />
<br />
<br />
This parseInfo structure would be the main part of the module and map from holding register 256 to a fhem reading named Temp_Wasser_Ein, holding register 258 to Temp_Wasser_Aus and 770 to Temp_Soll. <br />
<br />
All readings will be read from the device in an interval that the user can specify when he issues the define command for your module.<br />
The meaning of <code>set => 1</code> is that the holding register 770 can also be written to with a set command. FHEM will check that the value written is not smaller than 10 and not bigger than 32 as specified above.<br />
<br />
More complex example:<br />
<pre><br />
my %SET10parseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
name => "Pb1", # internal name of this register in the hardware doc<br />
expr => '$val / 10', # conversion of raw value to visible value <br />
len => 1,<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
name => "ST03",<br />
expr => '$val / 10', # convert raw value to readable temp<br />
setexpr => '$val * 10', # expression to convert a set value to the internal value <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
hint => "8,10,20,25,28,29,30,30.5,31,31.5,32",<br />
set => 1, # this value can be set<br />
},<br />
"h771" => { reading => "Hysterese", # Hex Adr 303<br />
name => "ST04",<br />
expr => '$val / 10',<br />
setexpr => '$val * 10',<br />
poll => "x10", # only poll every 10th iteration.<br />
min => 0.5,<br />
max => 3,<br />
set => 1,<br />
},<br />
"h777" => { reading => "Hyst_Mode", # Hex Adr 0309<br />
name => "ST10",<br />
map => "0:mittig, 1:oberhalb, 2:unterhalb", <br />
poll => "once", # only poll once (or after a set)<br />
set => 1,<br />
},<br />
"i800" => { reading => "Voltage", # Input register <br />
unpack => "f>", # this value is a float<br />
len => 2, # the float occupies two input registers, 800 and 801<br />
},<br />
);<br />
</pre><br />
<br />
There are many more options that can be specified for each data object / reading. If these options are not specified, the base module assumes defaults that typically make sense. However if you want to modify the defaults, you can either define explicit values in the parseInfo structure or you can define another data structure typically called deviceInfo. <br />
<br />
=== Introduction to the deviceInfo structure ===<br />
<br />
The deviceInfo structure is completely optional. If you don't define it in your module, the base module takes default values that work in ost cases. If you only want to override a few of the defaults, you can just define them and leave other options or sections out.<br />
A simple device info structure that modifies some defaults could look like this:<br />
<br />
<pre><br />
my %SET10deviceInfo = (<br />
"timing" => {<br />
timeout => 3, # timeout is 3 seconds /default would be 2<br />
commDelay => 0.7, # wait 0.7 seconds before sending after receiving<br />
sendDelay => 0.7, # wait at least 0.7 seconds for another send<br />
}, <br />
"c" => { <br />
read => 1, # function code 1 to read coils (this could be omitted because it is the default anyways<br />
write => 5, # dito<br />
},<br />
"h" => { <br />
read => 3, <br />
write => 6, <br />
defLen => 1, # default legth is 1 object<br />
combine => 5, <br />
defShowGet => 1, <br />
defUnpack => "s>", # default data format is a signed 16 bit integer for holding registers <br />
},<br />
);<br />
</pre><br />
<br />
The deviceInfo structure contains five optional parts. Timing defines timing values and the remaining parts define settings or defaults for coils (c), discrete inputs (d), input registers (i) and holding registers (h). <br />
<br />
for each modbus object type you can change what function code should be used to read or write to the object. This is completely optional and if nothing is specified, the base module assumes function codes 1,2,3 and 4 for reading as well as 5 and 6 for writing which works for many modbus devices. If you prefer to use function code 16 for writing to holding registers, you can specify "write => 16" in the "h" part.<br />
<br />
=== usage of a module created this way ===<br />
<br />
a logical module written this way will have a define command that can work in two ways. <br />
If your module would be called ModbusSET and it is using a serial line connection (Modbus RTU over RS485 oer over RS232):<br />
<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate<br />
define <name> ModbusSET <Id> <Interval> </code><br />
</pre><br />
<br />
In this case, a physical serial interface device is defined first using the Modbus module. Then a device based on your module (ModbusSET in the example) is defined for each physical modbus device connected to the serial line. For a RS485 bus, several devices with different Ids can be connected to the same bus.<br />
<br />
Example:<br />
<pre><br />
define ModbusRS485 Modbus /dev/rs485@9600<br />
define PWP ModbusAttr 5 60<br />
</pre><br />
<br />
this defines the device and it will use the readings that you coded in the parseInfo data structure.<br />
<br />
Alternatively your module would also support Modbus TCP or Modbus RTU over TCP with the following define syntax:<br />
<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|TCP><br />
</pre><br />
In this case no serial interface device is necessary and your module connects to the modbus device directly via TCP using either Modbus TCP or Modbus RTU over TCP.<br />
<br />
Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115:502 TCP<br />
</pre><br />
<br />
=== General information about data objects ===<br />
<br />
Modbus devices can use many different ways to encode values in their data objects. A temperature might be stored multiplied with 10 as a 16 bit integer value in one holding register so you have to read the integer and divide it by 10 to get the real temperature value back. It might also be stored as a 32 bit float data type that spans two adjacent input registers.<br />
The modbus base module implements a very generic way to handle different encodings without real programming: It lets you define the Perl unpack code to convert a raw data string to a Perl value, a Perl expression to do further computation and a length in data objects. <br />
<br />
This way a temperature stored in a 16 bit signed integer as the value multiplied by 10 can be described with the unpack code "s>" and the expression "$val / 10". A float value spanning 2 registers would be described with an unpack code "f>" and a len of 2. No expression is needed in this case. See Perldoc on the pack function for a detailed explation of pack and unpack codes.<br />
<br />
The idea here is that you should be able to define any mapping, encoding, transformation or formatting of data objects without programming by simpy describing them.<br />
<br />
=== most important options in parseInfo ===<br />
<br />
Most options here are optional and can be used if there is a need but they can also be omitted. If most readings require the same options and the option is different from the default, it is also possible to define a different default per modbus data object type in another data structure (see deviceInfo).<br />
For a list of all options please refer to the attributes documentation of the module ModbusAttr. The attributes there can be translated to parseInfo or deviceInfo keys as shown above.<br />
<br />
;reading <br />
:name of the reading to be used in FHEM e.g. Temp_Wasser_ein<br />
<br />
;expr <br />
:perl expression to convert a string after it has been read. The original value is in $val e.g. $val / 10<br />
<br />
;map <br />
:a map string to convert an value from the device to a more readable output string or to convert a user input to the machine representation e.g. "0:mittig, 1:oberhalb, 2:unterhalb" <br />
<br />
;format <br />
:a format string for sprintf to format a value read, e.g. %.1f<br />
<br />
;len <br />
:number of Registers this value spans, can be 2 for a 32 bit float which is stored in 2 registers<br />
<br />
;unpack <br />
:defines the translation between data in the module and in the communication frame see the documentation of the perl pack function for details. example: "n" for an unsigned 16 bit value or "f>" for a float that is stored in two registers or "s>" for signed 16 bit integer in big endian format<br />
<br />
;showget <br />
:can be set to 1 to allow a FHEM get command to read this value from the device. All defined objects can be used in a get command that is issued on the command line. This parameter only controls if fhemweb will offer a get command for the object.<br />
<br />
;poll <br />
:defines if this value is included in the read that the module does every defined interval this can be changed by a user with an attribute<br />
<br />
;polldelay <br />
:if a value should not be read in each iteration (after interval has passed), this value can be set to an explicit time in seconds. The update function will then verify if this delay has elapsed since the last read of this object. If not, the read is skipped.<br />
<br />
;set <br />
:can be set to 1 to allow writing this value with a FHEM set-command<br />
<br />
;min <br />
:min value for input validation in a set command. If the user issues e.g. set Device Temp_Soll 10, FHEM will check if the given value 10 is bigger or equal the defined min and smaller or equal the defined max.<br />
<br />
;max <br />
:max value for input validation in a set command<br />
<br />
;hint <br />
:string for fhemweb to create a selection or slider<br />
<br />
;setexpr <br />
:per expression to convert an input string to the machine format before writing this is typically the reverse of the above expr, e.g. $val * 10<br />
<br />
;name<br />
:optional internal name of the value in the modbus documentation of the physical device, e.g. pb1<br />
<br />
<br />
=== most important options in deviceInfo ===<br />
<br />
Keys in the timing section:<br />
<br />
;timeout <br />
:how long to wait for a response from the device, can be overwritten by attribute timeout in logical device. Defaults to two seconds if this is not specified<br />
<br />
;commDelay <br />
:minimal delay in secounds between two communications e.g. a read a the next write, can be overwritten with attribute commDelay if added to AttrList in _Initialize below defaults to 0.1 seconds if not specified<br />
<br />
;sendDelay <br />
:minimal delay in seconds between two sends, can be overwritten with the attribute sendDelay if added to AttrList in _Initialize function below. Defaults to 0.1 seconds if not specified<br />
<br />
Keys per object type (c = coil, d = discrete input, i = input register, h = holding register)<br />
<br />
;read <br />
:function code to use for reading this object type (e.g. 3 for holding registers) defaults to function codes 1-4 depending on the object types if nothing else is specified (3 to read holding register, 1 to read coils and so on)<br />
<br />
;write <br />
:function code to use for writing this object type (e.g. 6 or 16 for holding registers) defaults to function codes 5 and 6 depending on the object types if nothing else is specified (6 to read holding register, 5 to write coils and so on)<br />
<br />
;defLen <br />
:default len for objects using this type (e.g. can be set to 2 if the device mainly provides float values that span 2 registers (2 times 16 Bit) can be overwritten in parseInfo per reading by specifying the key "len" defaults to 1 if not specified<br />
<br />
;defFormat <br />
:format string to do sprintf with the value can be overwritten in parseInfo per reading by specifying the key "format" if no format is specified here and none in parseInfo, the the reading is set without further formatting (which is typically fine)<br />
<br />
;defUnpack <br />
:default pack / unpack code to convert raw values, e.g. "n" for a 16 bit integer or "f>" for a big endian float can be overwritten in parseInfo per reading by specifying the key "unpack" if not specified here and not in parseInfo, then the raw value is interpreted as "n" which is 16 bit unsigned integer in big endian format<br />
<br />
;defPoll <br />
:defines that objects of this type should be polled by default unless specified otherwise in parseInfo or by attributes can be overwritten in parseInfo per reading by specifying the key defaultpoll if not specified here or in parseInfo, the object is not polled<br />
<br />
;defShowGet <br />
:defines that FHEMweb shows a Get option (by returning it as reslut to get ?) for objects of this type can be overwritten in parseInfo per reading by specifying the key showget defaults to 0.<br />
<br />
;combine <br />
:max number of registers that the device is willing to deliver in one read request. The modbus application layer protocol specification allows for more than 100 but most devices limit this to 5, 10 or some other number. This option defaults to 1 if not specified.<br />
<br />
For an example of a full module that is based on the mechanisms described here see 98_ModbusSET.pm.<br />
<br />
=== Attributes of your module ===<br />
<br />
a module based on the base module / library 98_Modbus.pm can also allow the end user to modify properties of each reading if you want to allow it. All you have to do is to offer an attribute by adding its name to the variable $modHash->{AttrList} in your initialize function.<br />
<br />
If for example you want to allow the user to modify the maximum value for the reading Temp_Soll, you can add "Temp_Soll-max " to this variable and the use can the set this attribute. The attribute takes precedence over the max potentially already defined in your parseInfo structure.<br />
<br />
there are two ways that the base module accepts such readings. One is the reading name followed by "-" and the option to override, the alternative syntax is "obj-" followed by the first letter of an object type (c/d/h/i) and a decimal address just like the main key of an object in the parseInfo structure. <br />
<br />
Instaed of allowing the attribute Temp_Soll-max for the max value of reading Temp_Soll which corresponds to holding register 770, you can alternatively add the attribute name "obj-h770-min " to $modHash->{AttrList}.<br />
<br />
If the user is allowd to specify such attributes solely depends on the contents of the $modHash->{AttrList} variable. All the processing is already built into the base module.<br />
<br />
If you want to allow the user the override the formatting of readings then you can add "obj-[cdih][1-9][0-9]*-format " as a regular expression that allows format specifications for all possible data objects.<br />
<br />
The module 98_ModbusAttr for example is also based on 98_Modbus.pm and allows all possible attributes so the user can completely define his device with attributes and without a parseInfo or deviceInfo structure.<br />
<br />
In the same way you can allow the user to override the device specific options and defaults with attributes that start with "dev-", followed by the section of the deviceInfo and the name of the option. If you want to allow the user to modify the function code to be used for writing holding registers, you can add the attribute "obj-h-write " and the user can then set this attribute to 6 or 16 as he prefers.<br />
It is up to the module author to decide if this makes sense.<br />
<br />
An assignment that allows most options to the user could be:<br />
<br />
<pre><br />
$modHash->{AttrList} = $modHash->{AttrList} . " " .<br />
"obj-[cdih][1-9][0-9]*-reading " .<br />
"obj-[cdih][1-9][0-9]*-name " .<br />
"obj-[cdih][1-9][0-9]*-set " .<br />
"obj-[cdih][1-9][0-9]*-min " .<br />
"obj-[cdih][1-9][0-9]*-max " .<br />
"obj-[cdih][1-9][0-9]*-hint " .<br />
"obj-[cdih][1-9][0-9]*-expr " .<br />
"obj-[cdih][1-9][0-9]*-map " .<br />
"obj-[cdih][1-9][0-9]*-setexpr " .<br />
"obj-[cdih][1-9][0-9]*-format " .<br />
"obj-[cdih][1-9][0-9]*-len " .<br />
"obj-[cdih][1-9][0-9]*-unpack " .<br />
"obj-[cdih][1-9][0-9]*-showget " .<br />
<br />
"obj-[cdih][1-9][0-9]*-poll " .<br />
"obj-[cdih][1-9][0-9]*-polldelay " .<br />
"poll-.* " .<br />
"polldelay-.* " .<br />
<br />
"dev-([cdih]-)*read " .<br />
"dev-([cdih]-)*write " .<br />
"dev-([cdih]-)*combine " .<br />
"dev-([cdih]-)*defLen " .<br />
"dev-([cdih]-)*defFormat " .<br />
"dev-([cdih]-)*defUnpack " .<br />
"dev-([cdih]-)*defPoll " .<br />
"dev-([cdih]-)*defShowGet " .<br />
"dev-timing-timeout " .<br />
"dev-timing-sendDelay " .<br />
"dev-timing-commDelay ";<br />
}<br />
</pre><br />
<br />
== Examples for logical device modules that use this base module ==<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM220M]<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM630M]<br />
:modules for energy meters from B+G E-Tech & EASTON written by Roger<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG103]<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG604]<br />
:modules for the UMG103 and UMG604 meters from Janitza<br />
;[[ModbusSET]]<br />
:module for the set silent heat pumps from Schmidt Energie Technik<br />
;[[ModbusAttr]]<br />
:generic modbus device module where the data objects, addresses, display formats, function codes and other things can be configured using FHEM attributes similar to HTTPMOD<br />
<br />
<br />
[[Kategorie:Interfaces]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=21451HTTPMOD2017-05-07T09:09:52Z<p>StefanStrobel: fixed formatting for sslArgs example</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
:<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=21450HTTPMOD2017-05-07T09:08:48Z<p>StefanStrobel: added attribute removeBuf</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that FHEM HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice HTTPMOD none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the FHEMweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue FHEM <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FHEMWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the FHEM at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;removeBuf<br />
:if set to 1 then HTTPMOD removes the internal named buf when a HTTP-response has been received. <br />
:$hash->{buf} is used internally be Fhem httpUtils and in some use cases it is desireable to remove this internal after reception <br />
:because it contains a very long response which looks ugly in Fhemweb.<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* Beispiel: [[HTTPMOD Beispielkonfiguration zur Anbindung einer Daikin Klimaanlage mit WLAN-Modul]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in FHEM Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in FHEM Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in FHEM Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=19180ArduCounter2017-01-29T11:22:07Z<p>StefanStrobel: Fehlerbereinigungen</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ArduCounter module works together with an arduino board connected to FHEM via usb that runs the provided ArduCounter.ino sketch to count impulses from energy meters with an S0 interface or similar devices in configurable intervals. <br />
<br />
The arduino board counts impulses on every configured input pin using pin change interrupts. It reports the counted number of impulses per pin together with the time between the first and last impulse per pin.<br />
<br />
The module converts this to readings like power consumption and it is not affected by delays or the load of your FHEM server.<br />
Counters are configured using attributes that define which Arduino pins should count pulses and in which intervals the counted numbers should be reported from the arduino borad to FHEM.<br />
<br />
== Availability == <br />
The module has been checked in to the contrib area. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
The default baudrate of the firmware is 38400 since Version 1.4 of the sketch.<br />
<br />
Example:<br />
<pre>define AC ArduCounter /dev/ttyUSB2</pre><br />
<br />
== Configuration of the module ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <pre>attr AC pinX falling pullup 30</pre> <br />
<br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can define if the signals to be counted start with rising or falling edges. <br />
The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
The last argument is also optional and specifies a minimal pulse length in milliseconds. In this case the first argument (e.g. falling) means that an impulse starts with a falling edge from 1 to 0 and ends when the signal changes back from 0 to 1.<br />
<br />
== Example: ==<br />
<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup<br />
attr AC pinD5 falling pullup 30<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
this defines three counters connected to the pins D4, D5 and D5.<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
For D4 every falling edge of the signal (when the input changes from 1 to 0) is counted.<br />
For D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 30 milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
For pin D6 the ardiono counts every time when the signal changes from 0 to 1.<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw<br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware<br />
:onto the device. This command needs avrdude to be installed. <br />
:The attribute flashCommand specidies how avrdude is called. <br />
:If it is not modifed then the module sets it to <br />
:avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
:This setting should work for a standard installation and the placeholders are automatically replaced when the command is used. So normally there is no need to modify this attribute.<br />
<br />
:Depending on your specific Arduino board however, you might need to insert <code>-b 57600</code> in the flash Command.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. <br />
:Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingFactor[0-9]+ <br />
:Override the factor attribute for this individual pin.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=19179ArduCounter2017-01-29T11:19:41Z<p>StefanStrobel: vieles aktualisiert</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ArduCounter module works together with an arduino board connected to FHEM via usb that runs the provided ArduCounter.ino sketch to count impulses from energy meters with an S0 interface or similar devices in configurable intervals. <br />
<br />
The arduino board counts impulses on every configured input pin using pin change interrupts. It reports the counted number of impulses per pin together with the time between the first and last impulse per pin.<br />
<br />
The module converts this to readings like power consumption and it is not affected by delays or the load of your FHEM server.<br />
Counters are configured using attributes that define which Arduino pins should count pulses and in which intervals the counted numbers should be reported from the arduino borad to FHEM.<br />
<br />
== Availability == <br />
The module has been checked in to the contrib area. <br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
The default baudrate of the firmware is 38400 since Version 1.4 of the sketch.<br />
<br />
=== Example:=== <br />
<pre>define AC ArduCounter /dev/ttyUSB2</pre><br />
<br />
== Configuration of the module ==<br />
<br />
Specify the pins where impulses should be counted e.g. as <pre>attr AC pinX falling pullup 30</pre> <br />
<br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pinD5, pin6, pinD7 ...<br />
<br />
After the pin you can define if the signals to be counted start with rising or falling edges. <br />
The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
The last argument is also optional and specifies a minimal pulse length in milliseconds. In this case the first argument (e.g. falling) means that an impulse starts with a falling edge from 1 to 0 and ends when the signal changes back from 0 to 1.<br />
<br />
== Example: ==<br />
<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 falling pullup<br />
attr AC pinD5 falling pullup 30<br />
attr AC pinD6 rising<br />
</pre><br />
<br />
this defines three counters connected to the pins D4, D5 and D5.<br />
D4 and D5 have their pullup resistors activated and the impulse draws the pins to zero.<br />
For D4 every falling edge of the signal (when the input changes from 1 to 0) is counted.<br />
For D5 the arduino measures the time in milliseconds between the falling edge and the rising edge. <br />
If this time is longer than the specified 30 milliseconds then the impulse is counted. <br />
If the time is shorter then this impulse is regarded as noise and added to a separate reject counter.<br />
<br />
For pin D6 the ardiono counts every time when the signal changes from 0 to 1.<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input pins can be used, not only the ones that support normal interrupts.<br />
<br />
== Get-Commands ==<br />
;info <br />
:send a command to the Arduino board to get current counts. <br />
:This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw<br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands.<br />
:This is not needed for normal operation but might be useful sometimes for debugging<br />
<br />
;flash<br />
:flashes the ArduCounter firmware ArduCounter.hex from the fhem subdirectory FHEM/firmware<br />
:onto the device. This command needs avrdude to be installed. <br />
:The attribute flashCommand specidies how avrdude is called. <br />
:If it is not modifed then the module sets it to <br />
:avrdude -p atmega328P -c arduino -P [PORT] -D -U flash:w:[HEXFILE] 2>[LOGFILE]<br />
This setting should work for a standard installation and the placeholders are automatically replaced when <br />
the command is used. So normally there is no need to modify this attribute.<br />
<br />
Depending on your specific Arduino board however, you might need to insert <code>-b 57600</code> in the flash Command.<br />
<br />
;reset<br />
:reopens the arduino device and sends a command to it which causes a reinitialize and reset of the counters. <br />
:Then the module resends the attribute configuration / definition of the pins to the device.<br />
<br />
<br />
<br />
<br />
<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code>, followed by an optional <code>pullup</code> and an optional number as value.<br />
:If a number is specified, the arduino will track rising and falling edges of each impulse and measure the length of a pulse in milliseconds. <br />
:The number specified here is the minimal length of a pulse and a pause before a pulse. If one is too small, the pulse is not counted but added to a separate reject counter.<br />
<br />
;interval normal max min mincout <br />
:Defines the parameters that affect the way counting and reporting works.<br />
:This Attribute expects at least two and a maximum of four numbers as value. <br />
: The first is the normal interval, the second the maximal interval, the third is a minimal interval and the fourth is a minimal pulse count.<br />
<br />
:In the usual operation mode (when the normal interval is smaller than the maximum interval), the Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the normal interval is elapsed the Arduino board reports the count and time for those pins where impulses were encountered.<br />
:This means that even though the normal interval might be 10 seconds, the reported time difference can be something different because it observed impulses as starting and ending point.<br />
:The Power (e.g. for energy meters) is the calculated based of the counted impulses and the time between the first and the last impulse. <br />
:For the next interval, the starting time will be the time of the last impulse in the previous reporting period and the time difference will be taken up to the last impulse before the reporting interval has elapsed.<br />
<br />
:The second, third and fourth numbers (maximum, minimal interval and minimal count) exist for the special case when the pulse frequency is very low and the reporting time is comparatively short.<br />
:For example if the normal interval (first number) is 60 seconds and the device counts only one impulse in 90 seconds, the the calculated power reading will jump up and down and will give ugly numbers.<br />
:By adjusting the other numbers of this attribute this can be avoided.<br />
:In case in the normal interval the observed impulses are encountered in a time difference that is smaller than the third number (minimal interval) or if the number of impulses counted is smaller than the fourth number (minimal count) then the reporting is delayed until the maximum interval has elapsed or the above conditions have changed after another normal interval.<br />
:This way the counter will report a higher number of pulses counted and a larger time difference back to fhem.<br />
<br />
:If this is seems too complicated and you prefer a simple and constant reporting interval, then you can set the normal interval and the mximum interval to the same number. This changes the operation mode of the counter to just count during this normal and maximum interval and report the count. In this case the reported time difference is always the reporting interval and not the measured time between the real impulses.<br />
<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
;readingNameCount[0-9]+ <br />
:Change the name of the counter reading pinX to something more meaningful.<br />
<br />
;readingNamePower[0-9]+ <br />
:Change the name of the power reading powerX to something more meaningful.<br />
<br />
;readingFactor[0-9]+ <br />
:Override the factor attribute for this individual pin.<br />
<br />
;readingStartTime[0-9]+ <br />
:Allow the reading time stamp to be set to the beginning of measuring intervals<br />
<br />
;verboseReadings[0-9]+ <br />
:create additional readings: timeDiff, countDiff and lastMsg for each pin<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ModbusAttr&diff=19178ModbusAttr2017-01-29T10:55:25Z<p>StefanStrobel: zahlreiche Aktualisierungen</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=ModbusAttr<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=25315<br />
|ModTechName=98_ModbusAttr.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
[[ModbusAttr]] provides a generic way to retrieve data objects from devices with a Modbus Interface and store these objects in Readings, or modify such objects by sending write commands to such devices. <br />
The devices can be connected via RS232, RS485, or IP network, and the supported protocols are Modbus RTU, Modbus ASCII or Modbus TCP.<br />
<br />
The data objects to be used and the correspondig readings are defined by attributes in a way similar to [[HTTPMOD]]. They are continuously updated in the interval specified with the define command. This behavior can be modified with attribues to have some data objects updated at a lower frequency or only once and after each change.<br />
<br />
There are several attributes that modify the way data objects are converted before they are stored in readings. They can be modified by a perl expression defined in an atribute, formatted with a format string defined in another attribute or mapped to a table defined in an attribute.<br />
<br />
Readings can directly correspond to one data object or they can span several objects. A float value for example might be stored in two input or holding registers in the Modbus device. By specifying attributes that define the length of a reading in objects and by specifying the unpack code to get from a raw string to perl variables, all these cases can be described by attributes and no perl coding is necessary.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module uses the [[Modbus|Modbus base module 98_Modbus.pm]] which uses the DevIO module<br />
<br />
== Define for Modbus RTU over serial lines / RS485 bus ==<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate,bits,parity,stop<br />
define <name> ModbusAttr <Id> <Interval> <RTU|ASCII><br />
</pre><br />
<br />
In this case, a physical serial interface device is defined first using the Modbus module. Then a ModbusAttr device is defined for each Modbus device connected to the serial line. For a RS485 bus, several Modbus devices with different Ids can be connected to the same bus.<br />
<br />
Example:<br />
<pre><br />
define ModbusRS485 Modbus /dev/rs485@9600,8,E,1<br />
define PWP ModbusAttr 5 60 RTU<br />
<br />
attr PWP obj-h256-reading Temp_Wasser<br />
attr PWP obj-h256-unpack n<br />
attr PWP obj-h256-expr $val/10<br />
attr PWP obj-h256-poll 1<br />
<br />
attr PWP obj-h262-reading Pressure<br />
attr PWP obj-h262-unpack f><br />
attr PWP obj-h262-len 2<br />
attr PWP obj-h262-poll 1<br />
attr PWP obj-h262-showGet 1<br />
</pre><br />
<br />
This defines a physical Modbus interface connected to /dev/rs485 that uses 9600 Bps, 8 data bits, even parity and one stop bit. Most of these parameters are optional. by default 8,N,1 is assumed.<br />
the logical device is defined with Modbus id 5 and a polling interval of 60 seconds. The protocol is Modbus RTU.<br />
It defines two readings (Temp_Wasser and Pressure) that are stored in holding registers with the decimal addresses 256 and 262. <br />
<br />
The reading Temp_Wasser is stored in the device in just one holding register.<br />
The unpack code n specifies that the raw value is a signed 16 bit integer value.<br />
The Temperature in this example is stored in the Modbus device as the real temperature multiplied by 10 so the module hast to divide the raw value by 10 before putting it into the Fhem reading.<br />
With the poll attribute the module is instructed to request the value automatically in the above defined interval.<br />
<br />
The reading Pressure is stored in the device as a floating point number. Floating point numbers typically need two holding registers (32 bit) in a Modbus device. Therefore the module needs to request two holding registers (len 2).<br />
The unpack code f> specifies that the raw value is to be interpreted as a 32 bit big endian float value.<br />
the showGet creates will show a get-Option in Fhemweb.<br />
<br />
Another Example:<br />
<pre><br />
define ModbusRS485 Modbus /dev/ttyUSB0@38400<br />
define PWP ModbusAttr 1 0 ASCII<br />
<br />
attr PWP enableControlSet 1<br />
<br />
attr PWP obj-h258-reading Target<br />
attr PWP obj-h258-unpack n<br />
attr PWP obj-h258-showGet 1<br />
attr PWP obj-h258-set 1<br />
attr PWP obj-h258-max 5<br />
attr PWP obj-h258-min 32<br />
</pre><br />
<br />
In this example the device speaks Modbus ASCII over a serial line. The interval is set to 0 which means that there will be no automatic polling. <br />
Instead a get option is created (showGet) and a set option allows writing the value to the Modbus device.<br />
<br />
enableControlSet adds many more set options like interval, reread, reconnect, stop, start, scanModbusId, scanModbusObjects and scanStop. These are described later.<br />
<br />
== Define for Modbus TCP or Modbus RTU over TCP ==<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|ASCII|TCP><br />
</pre><br />
In this case no serial interface device is necessary and ModbusAttr connects to the Modbus device directly via TCP using either Modbus TCP or Modbus RTU over TCP.<br />
<br />
Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115 TCP<br />
<br />
attr PWP obj-h256-reading Temp_Wasser_ein<br />
attr PWP obj-h256-unpack n<br />
attr PWP obj-h256-poll 1<br />
<br />
</pre><br />
<br />
defines a Modbus TCP device at ip address 192.168.1.115 with the default port 502 for Modbus TCP.<br />
<br />
Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115:1234 RTU<br />
<br />
attr PWP obj-h256-reading Temp_Wasser_ein<br />
attr PWP obj-h256-unpack n<br />
attr PWP obj-h256-poll 1<br />
</pre><br />
<br />
defines a Modbus device at ip address 192.168.1.115 that communicates with Modbus RTU over the TCP-Port 1234 (probably this is some TCP to RS232 or RS485 converter)<br />
<br />
== Set-Commands ==<br />
can be defined for holding registers and coils by using attributes.<br />
<br />
Every object for which an attribute like <code>obj-xy-set</code> is set to 1 will create a valid set option.<br />
<br />
Additionally the attribute <code>enableControlSet</code> enables the set options <code>interval</code>, <code>stop</code>, <code>start</code>, <code>reread</code> as well as <code>scanModbusObjects</code>, <code>scanStop</code> and <code>scanModbusIds</code> (for devices connected with RTU / ASCII over a serial line).<br />
<br />
;<code>interval &lt;Interval&gt;</code><br />
:modifies the interval that was set during define. <br />
<br />
;<code>stop</code><br />
:stops the interval timer that is used to automatically poll objects through Modbus.<br />
<br />
;<code>start</code><br />
:starts the interval timer that is used to automatically poll objects through Modbus. <br />
:If an interval is specified during the define command then the interval timer is started automatically. <br />
:However if you stop it with the command <code>set &lt;mydevice&gt; stop</code> <br />
:then you can start it again with <code>set &lt;mydevice&gt; start</code>.<br />
<br />
;<code>reread</code><br />
:causes a read of all objects that are set to be polled in the defined interval. The interval timer is not modified.<br />
<br />
;<code>scanModbusObjects &lt;startObj&gt; - &lt;endObj&gt; &lt;reqLen&gt;</code><br />
:scans the device objects and automatically creates attributes for each reply it gets. <br />
:This might be useful for exploring devices without proper documentation. <br />
:The following example starts a scan and queries the holding registers with addresses between 100 and 120. <br />
:<code>set MyModbusAttrDevice scanModbusObjects h100-120</code><br><br />
:For each reply it gets, the module creates a reading like<br />
:<code>scan-h100 hex=0021, string=.!, s=8448, s>=33, S=8448, S>=33</code><br><br />
:the representation of the result as hex is 0021 and<br />
:the ASCII representation is .!. s, s>, S and S> are different representations with their Perl pack-code.<br />
<br />
;<code>scanModbusIds &lt;startId&gt; - &lt;endId&gt; &lt;knownObj&gt;</code><br />
:scans for Modbus Ids on an RS485 Bus. The following set command for example starts a scan:<br><br />
:<code>set Device scanModbusId 1-7 h770</code><br><br />
:since many Modbus devices don't reply at all if an object is requested that does not exist, <br />
:scanModbusId needs the adress of an object that is known to exist.<br />
:If a device with Id 5 replies to a read request for holding register 770, a reading like the following will be created:<br />
:<code>scanId-5-Response-h770 hex=0064, string=.d, s=25600, s>=100, S=25600, S>=100</code><br />
;<code>scanStop</code><br />
:stops any running scans.<br />
<br />
<br />
== Get-Commands ==<br />
Every reading can be manually requested by a Get. <br />
Internally a Get command triggers the corresponding Modbus request to the device and the module then interprets the data and sets the right Fhem readings. To avoid huge option lists in FHEMWEB, the objects visible as Get in FHEMWEB can be defined by setting an attribute <code>obj-xy-showget</code> to 1. <br />
<br />
<br />
== Simple Attributes ==<br />
<br />
Attributes to define data objects start with obj- followed by a code that identifies the type and address<br />
of the data object. <br><br />
<br />
Modbus devices offer the following types of data objects: <br />
;holding registers (16 bit objects that can be read and written)<br />
;input registers (16 bit objects that can only be read)<br />
;coils (single bit objects that can be read and written) <br />
;discrete inputs (single bit objects that can only be read)<br />
<br />
Bigger Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115:502 TCP<br />
<br />
attr PWP obj-h256-reading Temp_Wasser_Ein<br />
attr PWP obj-h256-expr $val/10<br />
<br />
attr PWP obj-h258-reading Temp_Wasser_Aus<br />
attr PWP obj-h258-expr $val/10<br />
<br />
attr PWP obj-h262-reading Temp_Luft<br />
attr PWP obj-h262-expr $val / 10<br />
<br />
attr PWP obj-h770-reading Temp_Soll<br />
attr PWP obj-h770-expr $val / 10<br />
attr PWP obj-h770-set 1<br />
attr PWP obj-h770-setexpr $val * 10<br />
attr PWP obj-h770-max 32<br />
attr PWP obj-h770-min 10<br />
attr PWP obj-h770-hint 8,10,20,25,28,29,30,30.5,31,31.5,32<br />
<br />
attr PWP dev-h-combine 5<br />
attr PWP dev-h-defPoll 1<br />
</pre><br />
<br />
The module uses the first character of Modbus data object types to define attributes. Thus h770 refers to a holding register with the decimal address 770 and c120 refers to a coil with address 120. The address has to be specified as pure decimal number and counting starts at address 0.<br />
<br />
<code>attr PWP obj-h258-reading Temp_Wasser_Aus</code> <br />
defines a reading with the name Temp_Wasser_Aus that is read from the Modbus holding register at address 258.<br />
With the attribute ending on <code>-expr</code> you can define a perl expression to do some conversion or calculation on the raw value read from the device. In the above example the raw value has to be devided by 10 to get the real temperature value.<br />
<br />
An object attribute ending on <code>-set</code> creates a fhem set option. In the above example the reading Temp_Soll can be changed to 12 degrees by the user with the fhem command <code>set PWP Temp_Soll 12</code><br />
The object attributes ending on <code>-min</code> and <code>-max</code> define min and max values for input validation and the attribute ending on <code>-hint</code> will tell fhem to create a selection list so the user can graphically select the defined values.<br />
<br />
To define general properties of the device you can specify attributes starting with <code>dev-</code>. E.g. with <code>dev-timing-timeout</code> you can specify the timeout when waiting for a response from the device. <br />
With <code>dev-h-</code> you can specify several default values or general settings for all holding registers like the function code to be used when reading or writing holding registers. These attributes are optional and the module will use defaults that work in most cases. <br />
<code>dev-h-combine 5</code> for example allows the module to combine read requests to objects having an address that differs 5 or less into one read request. Without setting this attribute the module will start individual read requests for each object. Typically the documentation for the Modbus interface of a given device states the maximum number of objects that can be read in one function code 3 request. <br />
<br />
<br />
== More complex example ==<br />
<pre><br />
define MD ModbusAttr 1 30 192.168.1.115:502 TCP<br />
<br />
attr MD dev-i-combine 10 # combine read for up to 10 adjacent input registers<br />
attr MD dev-i-read 4 # use function code 4 to read input registers (this would be the default anyway)<br />
attr MD dev-i-defLen 2 # input registers define objects that span 2 registers by default<br />
attr MD dev-i-defUnpack f> # default unpack code for all objects that don't specify anything else will be f> (big endian 32 bit float, see perldoc pack)<br />
attr MD dev-i-defFormat %.1f # format values with one digit after the comma by default if nothing else is specified for individual readings<br />
<br />
attr MD dev-h-read 3 # this can be omitted since 3 is the default anyways<br />
attr MD dev-h-write 16 # use function code 16 instead of 6 to write holding registers<br />
attr MD dev-h-defPoll 1 # include all readings for holding registers in the update unless overwritten per object<br />
attr MD dev-h-defShowGet 1 # show a get option in fhemweb for all readings based on holding registers<br />
<br />
attr MD obj-i1010-reading Temp_Wasser_Ein # given the above defaults, this will be <br />
# a float unpacked with f> that spans 2 input registers<br />
attr MD obj-i1020-reading HystMode <br />
attr MD obj-i1020-len 1 # this overrides the default specified with dev-i-defLen above<br />
attr MD obj-i1020-format %s # override default set above<br />
attr MD obj-i1020-map 0:mittig, 1:oberhalb, 2:unterhalb # convert the raw value 0 to "mittig", the value 1 to "oberhalb" and "2" to "unterhalb"<br />
attr MD obj-i1020-pollDelay x10 # only update this reading every 10th iteration<br />
</pre><br />
<br />
== All Attributes ==<br />
<br />
;readingFnAttributes<br />
:the usual Fhem attributes for all devices<br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. <br />
:This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;enableControlSet<br />
:enables the built in set commands like interval, stop, start and reread (see above) <br />
<br />
the following list of attributes can be applied to any data object by specifying the objects type and address in the variable part. <br />
For many attributes you can also specify default values per object type (see dev- attributes later) or you can specify an object attribute without type and address (e.g. obj-len) which then applies as default for all objects:<br />
<br />
;obj-[cdih][1-9][0-9]*-reading <br />
:define the name of a reading that corresponds to the Modbus data object of type c,d,i or h and a decimal address (e.g. obj-h225-reading).<br />
<br />
;obj-[cdih][1-9][0-9]*-name <br />
:defines an optional internal name of this data object (this has no meaning for fhem and serves mainly documentation purposes.<br />
<br />
;obj-[cdih][1-9][0-9]*-set <br />
:if set to 1 then this data object can be changed (works only for holding registers and coils since discrete inputs and input registers can not be modified by definition.<br />
<br />
;obj-[cdih][1-9][0-9]*-min <br />
:defines a lower limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-max <br />
:defines an upper limit to the value that can be written to this data object. This ist just used for input validation.<br />
<br />
;obj-[cdih][1-9][0-9]*-hint <br />
:this is used for set options and tells fhemweb what selection to display for the set option (list or slider etc.)<br />
<br />
;obj-[cdih][1-9][0-9]*-expr <br />
:defines a perl expression that converts the raw value read from the device.<br />
<br />
;obj-[cdih][1-9][0-9]*-ignoreExpr<br />
:defines a perl expression that returns 1 if a value should be ignored and the existing reading should not be modified<br />
<br />
;obj-[cdih][1-9][0-9]*-map <br />
:defines a map to convert values read from the device to more convenient values when the raw value is read from the device or back when the value to write has to be converted from the user value to a raw value that can be written. Example: 0:mittig, 1:oberhalb, 2:unterhalb <br />
<br />
;obj-[cdih][1-9][0-9]*-setexpr <br />
:defines a perl expression that converts the user specified value in a set to a raw value that can be sent to the device. This is typically the inversion of -expr above.<br />
<br />
;obj-[cdih][1-9][0-9]*-format <br />
:defines a format string to format the value read e.g. %.1f<br />
<br />
;obj-[cdih][1-9][0-9]*-len <br />
:defines the length of the data object in registers. It defaults to 1. Some devices store 32 bit floating point values in two registers. In this case you can set this attribute to two.<br />
<br />
;obj-[cdih][1-9][0-9]*-unpack <br />
:defines the unpack code to convert the raw data string read from the device to a reading. For an unsigned integer in big endian format this would be "n", for a signed 16 bit integer in big endian format this would be "s>" and for a 32 bit big endian float value this would be "f>". (see the perl documentation of the pack function).<br />
<br />
;obj-[cdih][1-9][0-9]*-revRegs<br />
:this is only applicable to objects that span several input registers or holding registers.<br />
:when they are read then the order of the registers will be reversed before <br />
:further interpretation / unpacking of the raw register string. <br />
:The same happens before the object is written with a set command.<br />
<br />
;obj-[cdih][1-9][0-9]*-bswapRegs<br />
:this is applicable to objects that span several input or holding registers.<br />
:After the registers have been read and before they are writtem, <br />
:all 16-bit values are treated big-endian and are reversed to little-endian by swapping the two 8 bit bytes. <br />
:This functionality is most likely used for reading (ASCII) strings from the device <br />
:that are stored as big-endian 16-bit values.<br />
:example: original reading is "324d3130203a57577361657320722020". After applying bswapRegs, <br />
:the value will be "4d3230313a2057576173736572202020" which will result in the ASCII string <br />
:"M201: WWasser ". <br />
:Should be used with "(a*)" as -unpack value.<br />
<br />
;obj-[cdih][1-9][0-9]*-decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string <br />
:read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;obj-[cdih][1-9][0-9]*-showget <br />
:every reading can also be requested by a get command. However these get commands are not automatically offered in fhemweb. By specifying this attribute, the get will be visible in fhemweb.<br />
<br />
;obj-[cdih][1-9][0-9]*-poll<br />
:if set to 1 then this obeject is included in the cyclic update request as specified in the define command. If not set, then the object can manually be requested with a get command, but it is not automatically updated each interval. Note that this setting can also be specified as default for all objects with the dev- atributes described later.<br />
<br />
;obj-[cdih][1-9][0-9]*-polldelay <br />
:this attribute allows to poll objects at a lower rate than the interval specified in the define command. you can either specify a time in seconds or number prefixed by "x" which means a multiple of the interval of the define command. if you specify a normal numer then it is interpreted as minimal time between the last read and another automatic read. Please note that this does not create an individual interval timer. Instead the normal interval timer defined by the interval of the define command will check if this reading is due or not yet. So the effective interval will always be a multiple of the interval of the define.<br />
<br />
<br />
;dev-([cdih]-)*read <br />
:specifies the function code to use for reading this type of object. The default is 3 for holding registers, 1 for coils, 2 for discrete inputs and 4 for input registers.<br />
<br />
;dev-([cdih]-)*write <br />
:specifies the function code to use for writing this type of object. The default is 6 for holding registers and 5 for coils. Discrete inputs and input registers can not be written by definition.<br />
<br />
;dev-([cdih]-)*combine <br />
:defines how many adjacent objects can be read in one request. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defLen <br />
:defines the default length for this object type. If not specified, the default is 1<br />
<br />
;dev-([cdih]-)*defFormat <br />
:defines a default format string to use for this object type in a sprintf function on the values read from the device.<br />
<br />
;dev-([cdih]-)*defExpr<br />
:defines a default Perl expression to use for this object type to convert raw values read.<br />
<br />
;dev-([cdih]-)*defIgnoreExpr<br />
:defines a default Perl expression to decide when values should be ignored.<br />
<br />
;dev-([cdih]-)*defUnpack <br />
:defines the default unpack code for this object type. <br />
<br />
;dev-([cdih]-)*defRevRegs<br />
:defines that the order of registers for objects that span several registers will be reversed before <br />
:further interpretation / unpacking of the raw register string<br />
<br />
;dev-([cdih]-)*defBswapRegs<br />
:per device default for swapping the bytes in Registers (see obj-bswapRegs above)<br />
<br />
;dev-([cdih]-)*defDecode<br />
:defines a default for decoding the strings read from a different character set e.g. cp850<br />
<br />
;dev-([cdih]-)*defEncode<br />
:defines a default for encoding the strings read (or after decoding from a different character set) e.g. utf8<br />
<br />
;dev-([cdih]-)*defPoll <br />
:if set to 1 then all objects of this type will be included in the cyclic update by default. <br />
<br />
;dev-([cdih]-)*defShowGet <br />
:if set to 1 then all objects of this type will have a visible get by default. <br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
<br />
;dev-([cdih]-)*allowShortResponses <br />
:if set to 1 the module will accept a response with valid checksum but data lengh < lengh in header<br />
<br />
;dev-timing-timeout <br />
:timeout for the device (defaults to 2 seconds)<br />
<br />
;dev-timing-sendDelay <br />
:delay to enforce between sending two requests to the device. Default ist 0.1 seconds.<br />
<br />
;dev-timing-commDelay <br />
:delay between the last read and a next request. Default ist 0.1 seconds.<br />
<br />
;nextOpenDelay <br />
:delay for Modbus-TCP connections. <br />
:This defines how long the module should wait after a failed TCP connection attempt before the next reconnection attempt. <br />
:This defaults to 60 seconds.<br />
<br />
;openTimeout <br />
:timeout to be used when opening a Modbus TCP connection (defaults to 3)<br />
<br />
;timeoutLogLevel <br />
:log level that is used when logging a timeout. Defaults to 3. <br />
<br />
;silentReconnect <br />
:if set to 1, then it will set the loglevel for "disconnected" and "reappeared" messages to 4 instead of 3<br />
<br />
;maxTimeoutsToReconnect <br />
:this attribute is only valid for TCP connected devices. <br />
:In such cases a disconnected device might stay undetected and lead to timeouts until the TCP connection is reopened. <br />
:This attribute specifies after how many timeouts an automatic reconnect is tried.<br />
<br />
;dev-h-brokenFC3<br />
:workaround for some broken Modbus function code 3 implementations<br />
<br />
;disable<br />
:stop communication with the device while this attribute is set to 1. For Modbus over TCP this also closes the TCP connection.<br />
<br />
<br />
== Links ==<br />
* [http://www.Modbus.org Modbus.org]<br />
* About Modbus ([http://en.wikipedia.org/wiki/Modbus English] / [http://de.wikipedia.org/wiki/Modbus German])<br />
* Perl [http://perldoc.perl.org/functions/pack.html unpack codes]<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=DevelopmentModuleIntro&diff=16625DevelopmentModuleIntro2016-10-15T10:17:17Z<p>StefanStrobel: /* X_Notify */ Tippfehler korrigiert</p>
<hr />
<div>{{Hinweis|Dieser Text ist in Arbeit und muss noch an einigen Stellen ergänzt werden. <br />
Insbesondere beschreibt der Text derzeit nur einstufige Module. Die Abgrenzung zu zweistufigen Modulen und deren Eigenschaften sollte noch ergänzt werden.}}<br />
<br />
<br />
== Einleitung ==<br />
Um neue Geräte in FHEM verfügbar zu machen, kann man ein eigenes Modul in Perl schreiben, das automatisch von FHEM geladen wird, wenn ein passendes Gerät in FHEM definiert wird. Das Modul definiert dann wie mit dem Gerät kommuniziert wird, stellt Werte ("Readings") innerhalb von FHEM zur Verfügung oder erlaubt es das Gerät mit "Set"-Befehlen zu beeinflussen. Dieser Text soll den Einstieg in die Entwicklung eigener Module erleichtern.<br />
<br />
Mit dem FHEM-Befehl "define", der typischerweise in die zentrale Konfigurationsdatei fhem.cfg eingetragen wird, werden Geräte in FHEM definiert. Der Befehl sorgt dafür dass ein neues Modul bei Bedarf geladen wird und die Initialisierungsfunktion des Moduls aufgerufen wird. <br />
<br />
Damit das funktioniert müssen der Name des Geräts, der Name des Moduls und der Name der Initialisierungsfunktion zueinander passen. Das folgende Beispiel soll dies verdeutlichen:<br />
<br />
Ein Jeelink USB-Stick in einer Fritz-Box könnte beispielsweise mit dem Befehl <code>define JeeLink1 JeeLink /dev/ttyUSB0@57600</code> definiert werden.<br />
<br />
In der fhem.pl wird der define-Befehl verarbeitet, geprüft, ob ein Modul mit Namen JeeLink schon geladen ist und falls nicht ein Modul mit Namen XY_JeeLink.pm im Modulverzeichnis (bei einer FritzBox z.B. /var/media/ftp/fhem/FHEM) gesucht und dann geladen. <br />
Danach wird die Funktion JeeLink_Initialize aufgerufen.<br />
Die Moduldatei muss also nach dem Namen des Geräts benannt werden und eine Funktion mit dem Namen des Geräts und einer _initialize Funktion enthalten.<br />
In der Initialisierungsfunktion des Moduls werden dann die Namen der aller weiteren Funktionen des Moduls, die von fhem.pl aus aufgerufen werden, bekannt gemacht. Dazu wird der Hash - das ist die zentrale Datenstruktur für jede Instanz eines Gerätes - mit entsprechenden Werten gefüllt.<br />
<br />
== Der Hash einer Geräteinstanz ==<br />
Eine Besonderheit in Perl sind [http://de.wikipedia.org/wiki/Assoziatives_Array#Perl assoziative Arrays], (nicht ganz richtig als "Hash" bezeichnet) in denen die Adressierung nicht über eine Zählvariable erfolgt, sondern über einen beliebigen String. Die internen Abläufe bei der Adressierung führen dazu, dass die Speicherung in und der Abruf aus Hashes relativ langsam ist.<br />
<br />
Der zentrale Speicherort für Informationen einer Geräteinstanz bei FHEM ist ein solcher Hash, der seinerseits in fhem.pl von einem globalen Hash referenziert wird. <br />
<br />
<code>$defs{''Devicename''}</code> in fhem.pl verweist auf den Hash der Geräteinstanz. Diesen Verweis (also nur die Adresse) bekommen die Funktionen eines Moduls übergeben, das direkt von fhem.pl aufgerufen wird. In dem Hash stehen beispielsweise die internen Werte des Geräts, die im GUI als "Internals" angezeigt werden oder die Readings des Geräts. Beispiele:<br />
*<code>$hash{NAME}</code> enthält den Namen der Geräteinstanz, <br />
*<code>$hash{TYPE}</code> enthält die Typbezeichnung des Geräts <br />
*<code>$hash->{INTERVAL}</code> enthält ein Abfrageintervall<br />
<br />
==Ausführung von Modulen==<br />
FHEM führt Module normalerweise nicht parallel aus. Daher wäre es ungünstig wenn Module Werte von einem Gerät abfragen und dann auf die Antwort des Geräts warten. In dieser Zeit wäre der Rest von FHEM blockiert. Die Ein- und Ausgabe sollte ohne Blockieren erfolgen und die Verarbeitung mehrerer Ein- und Ausgabekanäle quasi parallel ermöglichen. <br />
<br />
Dafür werden in FHEM zwei zentrale Listen gepflegt, in der die Filedeskriptoren der geöffneten Kommunikatonsverbindungen gespeichert sein können. Auf Linux- bzw. Unix-basierten Plattformen wird der select-Befehl des Betriebssystems verwendet und entsprechend gibt es in FHEM eine selectlist, in der die Filedeskriptoren der Geräedateien (z.B. /dev/ttyUSBx etc.) gespeichert sind. <br />
<br />
In der zentralen Schleife von fhem.pl wird mit select überwacht, ob über eine der geöffneten Schnittstellen Daten zum Lesen anstehen. Wenn dies der Fall ist, dann wird die Lesefunktion (X_Read) des zuständigen Moduls aufgerufen, damit es die Daten entgegennimmt und die Schleife wird weiter ausgeführt.<br />
<br />
Auf Windows-Systemen funktioniert dies anders. Hier können USB/Seriell-Geräte nicht per select überwacht werden. In FHEM unter Windows werden daher diese Schnittstellen kontinuierlich abgefragt ob Daten bereitstehen. Dafür müssen Module zusätzlich zur Lesefunktion eine Abfragefunktion (X_Ready) implementieren, die prüft ob Daten zum Lesen anstehen. Auch auf Linux/Unix-Plattformen hat diese Funktion eine Aufgabe. Falls nämlich eine Schnittstelle ausfällt beziehungsweise ein CUL oder USB-zu-Seriell Adapter ausgesteckt wird, dann wird über diese Funktion regelmäßig geprüft ob die Schnittstelle wieder verfügbar wird.<br />
<br />
Innerhalb der eigentlichen Lesefunktion (X_Read) werden dann die Daten vom zugehörigen Gerät gelesen, das nötige Protokoll implementiert und Werte in Readings geschrieben.<br />
<br />
Auch wenn von einem Anwender über ein <code>get</code> Daten aktiv von einem Gerät angefordert werden sollte nicht blockierend gewartet werden. Eine asynchrone Ausgabe ist über asyncOutput möglich. Siehe {{Link2Forum|Topic=43771|Message=357870|LinkText=Beschreibung}} und {{Link2Forum|Topic=43771|Message=360935|LinkText=Beispiel}}. Weitere Anwendungsbeispiele finden sich im {{Link2Forum|Topic=43052|Message=353477|LinkText=PLEX Modul}} und im überarbeiteten und nicht-blockierenden {{Link2Forum|Topic=42771|Message=348498|LinkText= SYSSTAT Modul}}.<br />
<br />
== Readings ==<br />
Werte, die von einem Gerät gelesen werden und in FHEM zur Verfügung stehen werden Readings genannt. Sie werden als Unterstruktur des Hashes der jeweiligen Geräteinstanz gespeichert, beispielsweise <br />
*<code>$hash{READINGS}{Temp}{VAL}</code> für die Temperatur eines Fühlers<br />
*<code>$hash{READINGS}{Temp}{TIME}</code> für den Zeitstempel der Messung<br />
<br />
Für den lesenden Zugriff auf Readings steht die Funktion ReadingsVal($$$) zur Verfügung.<br />
<br />
Readings werden im statefile von FHEM automatisch zwischengespeichert, damit sie nach einem Neustart sofort wieder zur Verfügung stehen, auch bevor sie vom Modul neu gesetzt oder aktualisiert werden.<br />
<br />
Readings, die mit einem Punkt im Namen beginnen, haben eine funktionale Besonderheit. Sie werden im FhemWeb nicht angezeigt und können somit als "Permanentspeicher" innerhalb des Moduls genutzt werden.<br />
<br />
Zum Setzen von Readings sollen <br />
*bei Gruppen von Readings der Funktionsblock <code>readingsBeginUpdate</code>, <code>readingsBulkUpdate</code> (mehrfach wiederholt), <code>readingsEndUpdate</code><br />
*bei einzelnen Updates die Funktion <code>readingsSingleUpdate</code> <br />
aufgerufen werden. Dabei kann man auch angeben, ob dabei ein Event ausgelöst werden soll oder nicht. Events erzeugen spürbare Last auf dem System (siehe NotifyFn), das Ändern von Readings ohne dass dabei Events erzeugt werden jedoch nicht.<br />
<br />
Eine Sequenz zum Setzen von Readings könnte folgendermaßen aussehen:<br />
<br />
<pre><br />
readingsBeginUpdate($hash);<br />
readingsBulkUpdate($hash, $readingName1, $wert1 );<br />
readingsBulkUpdate($hash, $readingName2, $wert2 );<br />
readingsEndUpdate($hash, 1);<br />
</pre><br />
<br />
== Internals ==<br />
Werte, die das Modul intern als Teil des Hashes speichert, die aber keine Readings sind, nennt man Internals. Sie werden ebenfalls als Unterstruktur des Hashes der jeweiligen Geräteinstanz gespeichert, beispielswiese <code>$hash->{INTERVAL}</code> für ein Abfrageintervall, das beim Define-Befehl übergeben wurde und als Internal gespeichert wird. Internals werden jedoch im Gegensatz zu Readings nicht im statefile zwischengespeichert. <br />
<br />
Falls Werte wie das gerade erwähnte Intervall nicht über den Define-Befehl gesetzt werden sollen und im Betrieb einfach änderbar sein sollen, ist eine alternative Möglichkeit die Speicherung in so genannten Attributen. Dann würde man den Define-Befehl so implementieren, dass er kein Intervall übergeben bekommt und statt dessen nach dem Define-Befehl zusätzlich den Befehl <code>attr</code> erwarten.<br />
<br />
== Attribute ==<br />
Parameter einer Geräteinstanz können mit dem Befehl <code>attr</code> als so genannte Attribute gesetzt und damit dem Modul zur Verfügung gestellt werden. Attribute werden zusammen mit der Definition der Geräte beim Speichern der aktuellen Konfiguration von FHEM in die Konfigurationsdatei geschrieben, die auch bei jedem Neustart von FHEM wieder gelesen wird. Zur Laufzeit werden Attribute in der globalen Datenstruktur <code>$attr{$name}</code> gespeichert. Ein Attribut mit dem Namen <code>header</code> würde beispielsweise mit <code>$attr{$name}{header}</code> adressiert (<code>$attr{$name}->{'header'}</code> wäre eine alternative aber unübliche Schreibweise für die selbe Variable). <br />
<br />
Zum Auslesen solcher Attribute sollte die Funktion <code>AttrVal($$$)</code> verwendet werden.<br />
<br />
Welche Attribute ein Modul unterstützt sollte in der Funktion <code>[[#X_Initialize|X_Initialize]]</code> durch Setzen der Variable <code>$hash->{AttrList}</code> bekannt gemacht werden (siehe unten). Wenn beim Setzen von Attributen die Werte geprüft werden sollen oder zusätzliche Funktionalität implementiert werden muss, dann kann dies in der Funktion <code>[[#X_Attr|X_Attr]]</code> ([[#X_Attr|siehe unten]]) implementiert werden.<br />
<br />
== Die wichtigsten Funktionen in einem Modul ==<br />
Eine typische Grundfunktion eines einfachen Moduls ist das Auslesen von Werten von einem physischen Gerät und Bereitstellen dieser Werte innerhalb von FHEM als Readings. Das Geräte könnte beispielsweise an einem USB-Port angeschlossen sein. Folgende Funktionen könnte man beispielsweise in einem Modul mit Namen X implementieren:<br />
* [[#X_Initialize|X_Initialize]] (initialisiert das Modul und gibt de Namen der zusätzlichen Funktionen bekannt)<br />
* [[#X_Define|X_Define]] (wird beim <code>define</code> aufgerufen)<br />
* [[#X_Undef|X_Undef]] (wird beim <code>delete</code>, sowie <code>rereadcfg</code> aufgerufen. Dient zum Abbau von offenen Verbindungen, Timern, etc.)<br />
* [[#X_Delete|X_Delete]] (wird beim <code>delete</code> aufgerufen um das Gerät endgültig zu löschen)<br />
* [[#X_Set|X_Set]] (wird beim Befehl <code>set</code> aufgerufen um Daten an das Gerät zu senden)<br />
* [[#X_Get|X_Get]] (wird beim Befehl <code>get</code> aufgerufen um Daten vom Gerät abzufragen)<br />
* [[#X_Attr|X_Attr]] (wird beim Befehl <code>attr</code> aufgerufen um beispielsweise Werte zu prüfen)<br />
* [[#X_Read|X_Read]] (wird vom globalen select aufgerufen, falls Daten zur Verfuegung stehen)<br />
* [[#X_Parse|X_Parse]] (wird bei zweistufigen Modulen vom Dispatch aufgerufen und muss hier noch beschrieben werden)<br />
* [[#X_Ready|X_Ready]] (wird unter windows als ReadFn-Erstatz benoetigt bzw. um zu pruefen, ob ein Geraet wieder eingesteckt ist)<br />
* [[#X_Notify|X_Notify]] (falls man benachrichtigt werden will)<br />
* [[#X_Rename|X_Rename]] (falls ein Gerät umbenannt wird)<br />
* [[#X_Shutdown|X_Shutdown]] (wird beim Herunterfahren aufgeführt)<br />
<br />
Die Funktionen werden im folgenden beschrieben (soweit diese Seite inzwischen vollständig ist):<br />
<br />
=== X_Initialize ===<br />
<br />
<pre><br />
sub X_Initialize($)<br />
{<br />
my ($hash) = @_;<br />
...<br />
</pre><br />
<br />
Das <code>X</code> im Namen muss dabei auf den Namen des Moduls bzw. des definierten Gerätetyps geändert werden. Im Modul mit der Datei <code>36_JeeLink.pm</code> beispielsweise ist der Name der Funktion <code>JeeLink_Initialize</code>. Die Funktion wird von Fhem.pl nach dem Laden des Moduls aufgerufen und bekommt einen Hash für das Modul als zentrale Datenstruktur übergeben. <br />
<br />
Dieser Hash wird im globalen Hash %modules gespeichert. <code>$modules{$ModulName}</code> wäre dabei der Hash für das Modul mit dem Namen <code>$ModulName</code>. Es handelt sich also nicht um den oben beschriebenen Hash der Geräteinstanzen sondern einen Hash, der je Modul Werte enthält, beispielsweise auch die Namen der Funktionen, die das Modul implementiert und die fhem.pl aufrufen soll. Die Initialize-Funktion setzt diese Funktionsnamen, in den Hash des Moduls:<br />
<br />
<pre><br />
$hash->{DefFn} = "X_Define";<br />
$hash->{UndefFn} = "X_Undef";<br />
$hash->{DeleteFn} = "X_Delete";<br />
$hash->{SetFn} = "X_Set";<br />
$hash->{GetFn} = "X_Get";<br />
$hash->{AttrFn} = "X_Attr";<br />
$hash->{NotifyFn} = "X_Notify";<br />
$hash->{ReadFn} = "X_Read";<br />
$hash->{ReadyFn} = "X_Ready";<br />
$hash->{ShutdownFn} = "X_Shutdown";<br />
</pre><br />
<br />
<code>X</code> ist wieder durch den Modulnamen ohne die vorangestellte Zahl zu ersetzen. <br />
Entsprechend können auch die Funktionen <code>X_Read</code>, <code>X_Parse</code> etc. durch Zuweisung an <code>$hash->{ReadFn}</code> etc. bekannt gemacht werden.<br />
<br />
Darüber hinaus sollten die vom Modul unterstützten Attribute definiert werden:<br />
<br />
<pre><br />
$hash->{AttrList} =<br />
"do_not_notify:1,0 " . <br />
"header " .<br />
$readingFnAttributes; <br />
</pre><br />
<br />
In Fhem.pl werden dann die entsprechenden Werte beim Aufruf eines <code>attr</code>-Befehls in die globale Datenstruktur <code>$attr{$name}</code>, z.B. <code>$attr{$name}{header}</code> für das Attribut <code>header</code> gespeichert. Falls im Modul weitere Aktionen oder Prüfungen beim Setzen eines Attributs nötig sind, dann kann wie im Beispiel oben die Funktion <code>X_Attr</code> implementiert und in der Initialize-Funktion bekannt gemacht werden.<br />
<br />
Die Variable <code>$readingFnAttributes</code>, die im obigen Beispiel an die Liste der unterstützten Attribute angefügt wird, definiert Attributnamen, die dann verfügbar werden wenn das Modul zum Setzen von Readings die Funktionen readingsBeginUpdate, readingsBulkUpdate, readingsEndUpdate oder readingsSingleUpdate verwendet. In diesen Funktionen werden Attribute wie <code>event-min-interval</code> oder auch <code>event-on-change-reading</code> ausgewertet. Für Details hierzu siehe commandref.<br />
<br />
<br />
Des weiteren ist es möglich, das Verhalten von Autocreate zu beeinflussen.<br />
<code>x</code> ist durch den Namen der Geräte zu ersetzen. Legt ihr Geräte mit dem Namen LaCrosse an, dann sollte x durch LaCrosse ersetzt werden.<br />
<pre><br />
$hash->{AutoCreate} =<br />
{ "x.*" => { ATTR => "event-min-interval:.*:300 event-on-change-reading:.*", <br />
FILTER => "%NAME", <br />
GPLOT => "temp4hum4:Temp/Hum,"} };<br />
autocreateThreshold => "<count>:<timeout>" <br />
</pre><br />
Mit <code>ATTR =></code> können Vordefinierte Attribute beim Anlegen definiert werden.<br />
Der Wert von <code>FILTER</code> wird als Regex verwendet, wenn ein Filelog angelegt wird. Damit könnt ihr steuern, welche Events ins Filelog komme. Definiert ihr das Feld <code>FILTER</code> nicht, gebt jedoch andere Felder an, dann kann kein filelog mehr automatisch durch autocreate angelegt werden!<br />
Mit Hilfe von <code>GPLOT</code> kann ein Plot angelegt werden. Mit der Angabe definiert ihr, welcher GPLOT angelegt wird.<br />
Mittels <code>autocreateThreshold</code> wird beeinflusst, wie oft <code>count</code>(default 2) und in welchem Zeitabstand <code>timeout</code> (default 60 Sekunden) die gleiche Nachricht empfangen werden muss, damit ein Gerät per autocreate angelegt wird. <br />
Das Verhalten, kann vom Anwender mittels Attribut <code>autocreateThreshold</code> im device "autocreate" überschrieben werden.<br />
<br />
<br />
Das Parsen der Parameter der define, get und set Kommandos sowie deren Übergabe an die DefFn, GetFn und SetFn lässt sich mit<br />
<pre>$hash->{parseParams} = 1;</pre> beeinflussen. Sobald es gesetzt ist wird automatisch [[DevelopmentModuleAPI#parseParams|parseParams]] aufgerufen und die an X_Define, X_Get und X_Set übergebenen Parameter ändern sich wie weiter unten beschrieben.<br />
<br />
=== X_Define ===<br />
Die Define-Funktion eines Moduls wird von Fhem aufgerufen wenn der Define-Befehl für ein Geräte ausgeführt wird und das Modul bereits geladen und mit der Initialize-Funktion initialisiert ist. Sie ist typischerweise dazu da, die übergebenen Parameter zu prüfen und an geeigneter Stelle zu speichern sowie einen Kommunikationsweg zum Gerät zu öffnen (z.B. TCP-Verbindung, USB-Schnittstelle o.ä.)<br />
Sie beginnt typischerweise mit:<br />
<br />
<pre><br />
sub X_Define($$)<br />
{<br />
my ( $hash, $def ) = @_;<br />
my @a = split( "[ \t][ \t]*", $def );<br />
...<br />
</pre><br />
<br />
Als Übergabeparameter bekommt die Define-Funktion den Hash der Geräteinstanz sowie den Rest der Parameter, die im Befehl angegeben wurden. Welche und wie viele Parameter <br />
akzeptiert werden ist Sache dieser Funktion. Im obigen Beispiel wird alles nach dem übergebenen Hash in ein Array aufgeteilt und so können die vom Modul bzw. der Define-Funktion erwarteten Werte über das Array der Reihe nach verarbeitet werden:<br />
<br />
<pre><br />
my $name = $a[0];<br />
my $url = $a[2];<br />
my $inter = 300;<br />
if(int(@a) == 4) { <br />
$inter = $a[3]; <br />
if ($inter < 5) {<br />
return "interval too small, please use something > 5, default is 300";<br />
}<br />
}<br />
</pre><br />
<br />
<br />
Neu: Zum Aufteilen und Parsen von <code>$def</code> lässt sich [[DevelopmentModuleAPI#parseParams|parseParams]] verwenden. Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird parseParams automatisch aufgerufen und X_Define ändert sich wie folgt:<br />
<pre><br />
sub X_Define($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
<br />
Damit die übergebenen Werte auch anderen Funktionen zur Verfügung stehen und an die jeweilige Geräteinstanz gebunden sind, werden die Werte typischerweise als Internals im Hash der Geräteinstanz gespeichert:<br />
<br />
<pre><br />
$hash->{url} = $url;<br />
$hash->{Interval} = $inter;<br />
</pre><br />
<br />
Wenn eine physische Schnittstelle geöffnet werden soll und dann bei verfügbaren Eingabedaten eine Lese-Funktion von Fhem aufgerufen werden soll, dann kann man in der Define-Funktion die Funktion DevIo_OpenDev aufrufen, die sich um alles weitere kümmert. Sie öffnet die Schnittstelle und fügt den Filedeskriptor an die globale Liste offener Verbindungen (selectlist / readyfnlist) an. Damit kann Fhem in seiner Hauptschleife erkennen, von welchem Gerät Daten bereit stehen und die zuständigen Funktionen aufrufen:<br />
<br />
<pre><br />
my $ret = DevIo_OpenDev( $hash, 0, "X_DevInit" );<br />
</pre><br />
<br />
Die optionale Funktion <code>X_DevInit</code> wird zur weiteren Initialisierung der Verbindung von <code>DevIo_OpenDev</code> aufgerufen. Der zweite Übergabeparameter an <code>DevIo_OpenDev</code> (hier <code>0</code>) steht für reopen und wird benötigt, da die Funktion auch aufgerufen wird, wenn ein USB-Geräte beispielsweise im Betrieb aus- und wieder eingesteckt wird. In diesem Fall wird die Funktion mit <code>1</code> aufgerufen.<br />
<br />
=== X_Undef ===<br />
<br />
Die <code>Undef</code>-Funktion wird aufgerufen wenn ein Gerät mit <code>delete</code> gelöscht wird oder bei der Abarbeitung des Befehls rereadcfg, der ebenfalls alle Geräte löscht und danach das Konfigurationsfile neu abarbeitet. Entsprechend müssen in der Funktion typische Aufräumarbeiten durchgeführt werden wie das saubere Schließen von Verbindungen oder das Entfernen von internen Timern sofern diese im Modul zum Pollen verwendet wurden (siehe später). <br />
<br />
Zugewiesene Variablen im Hash der Geräteinstanz, Internals oder Readings müssen hier nicht gelöscht werden. In fhem.pl werden die entsprechenden Strukturen beim Löschen der Geräteinstanz ohnehin vollständig gelöscht.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Undef($$) <br />
{ <br />
my ( $hash, $name) = @_; <br />
DevIo_CloseDev($hash); <br />
RemoveInternalTimer($hash); <br />
return undef; <br />
}<br />
</pre><br />
<br />
=== X_Delete ===<br />
<br />
Die <code>Delete</code>-Funktion ist das Gegenstück zur <code>Define</code>-Funktion und wird aufgerufen wenn ein Gerät mit <code>delete</code> gelöscht wird. <br />
<br />
Wenn ein Gerät mittels <code>delete</code> gelöscht wird, wird zuerst die <code>[[#X_Undef|Undef]]</code>-Funktion aufgerufen um offene Verbindungen abzubauen, anschließend wird die <code>Delete</code>-Funktion aufgerufen. Diese dient eher zum aufräumen von Dateien, welche durch das Modul evtl. für dieses Gerät spezifisch erstellt worden sind. Es geht hier also eher darum, alle Spuren sowohl im laufenden FHEM-Prozess, als auch Dateien oder Verbindungen zu löschen die mit diesem Gerät zu tun haben.<br />
<br />
Dies kann z.B. folgendes sein:<br />
<br />
* Löschen von Dateien im Dateisystem die während der Nutzung dieses Geräts angelegt worden sind.<br />
* Lösen von evtl. Pairings mit dem physikalischen Gerät <br />
<br />
Beispiel:<br />
<pre><br />
sub X_Delete($$) <br />
{ <br />
my ( $hash, $name ) = @_; <br />
<br />
# Löschen von Geräte-assoziiertem Temp-File<br />
unlink($attr{global}{modpath}."/FHEM/FhemUtils/$name.tmp";)<br />
} <br />
</pre><br />
<br />
=== X_Get ===<br />
Die Get-Funktion wird aufgerufen wenn der Fhem-Befehl <code>get</code> mit einem Gerät dieses Moduls ausgeführt wird. Mit <code>get</code> werden typischerweise Werte von einem Gerät abgefragt. Einige Module verwenden für diese Funktion einen Hash im Modul, der die möglichen <code>get</code>-Optionen mit zusätzlichen Werten definiert:<br />
<br />
<pre><br />
my %X_gets = (<br />
"TempSoll" => "XY",<br />
"Steilheit" => "Z"<br />
);<br />
</pre><br />
In der Get-Funktion selbst werden dann die übergebenen Parameter gegen diesen Hash geprüft.<br />
<br />
Beispiel:<br />
<br />
<pre><br />
sub X_Get($@)<br />
{<br />
my ( $hash, @a ) = @_;<br />
return "\"get X\" needs at least one argument" if ( @a < 2 );<br />
my $name = shift @a;<br />
my $opt = shift @a;<br />
if(!$X_gets{$opt}) {<br />
my @cList = keys %X_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
...<br />
</pre><br />
<br />
Die Ausgabe der Meldung mit <code>unknown ... choose one of ...</code> ist dabei wichtig, da sie im GUI-Modul verwendet wird um die möglichen <code>get</code>-Optionen zu ermitteln und als Auswahl anzubieten. Im weiteren Verlauf der Ger-Funktion könnte man dann mit dem physischen Gerät kommunizieren und den gefragten Wert abfragen und diesen als Return-Wert der Get-Funktion zurückgeben.<br />
<br />
<br />
Neu: Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird [[DevelopmentModuleAPI#parseParams|parseParams]] automatisch aufgerufen und X_Get ändert sich wie folgt:<br />
<pre><br />
sub X_Get($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
=== X_Set ===<br />
Die Set-Funktion ist das Gegenteil zur Get-Funktion. Sie ist dafür gedacht, Werte zum physischen Gerät zu schicken. Falls nur interne Werte im Modul gesetzt werden sollen, so sollte statt Set die Attr-Funktion verwendet werden. Attribute werden bei Save-Config auch in der Fhem.cfg gesichert. Set-Befehle nicht.<br />
<br />
Eine Set-Funktion ist ähnlich aufgebaut wie die Get-Funktion, sie bekommt jedoch nach dem Namen der Option auch den zu setzenden Wert übergeben.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Set($@)<br />
{<br />
my ( $hash, @a ) = @_;<br />
return "\"set X\" needs at least an argument" if ( @a < 2 );<br />
my $name = shift @a;<br />
my $opt = shift @a;<br />
my $value = join("", @a);<br />
<br />
if(!defined($X_sets{$opt})) {<br />
my @cList = keys %X_sets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
</pre><br />
<br />
<br />
Neu: Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird [[DevelopmentModuleAPI#parseParams|parseParams]] automatisch aufgerufen und X_Set ändert sich wie folgt:<br />
<pre><br />
sub X_Set($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
<br />
Das GUI FHEM-Web kann für die einzelnen Set-Optionen, die das Modul versteht auch automatisch Eingabehilfen wie Drop-Down Boxen oder Slider erzeugen. In der Detailansicht des GUI kann der Anwender dann die jeweiligen Werte komfortabel auswählen. Dafür muss die Set-Funktion, wenn sie mit der Option <code>?</code> aufgerufen wird, nicht nur einen Text mit <code>"Unknwon ... choose one of ..."</code> zurückgeben sondern den einzelnen Set-Optionen in diesem Rückgabetext nach einem Doppelpunkt Zusatzinformationen anhängen.<br />
Meist prüft man in den Modulen gar nicht auf die Option <code>?</code> sondern gibt generell bei unbekannten Optionen diesen Text zurück.<br />
<br />
Beispiel:<br />
<pre><br />
if(!defined($X_sets{$opt})) {<br />
return "Unknown argument $opt, choose one of mode:verbose,ultra,relaxed turbo:NoArg";<br />
}<br />
</pre><br />
<br />
Mit Kommata getrennte Werte ergeben eine Drop-Down Liste, mit der der User die Werte auswählen kann<br />
<pre>timer:30,120,300<br />
mode:verbose,ultra,relaxed</pre><br />
<br />
Wird kein Doppelpunkt zum Kommando angegeben, so wird eine Eingabezeile angezeigt, die die freie Eingabe eines Wertes erlaubt.<br />
<br />
Man kann jedoch die Eingabe-/Auswahlmöglichkeiten durch Widgets vereinfachen. Dazu gibt man hinter dem Doppelpunkt einen Widgetnamen und widgetspezifische Parameter an. Es existieren mehrere solcher Widgets in FHEMWEB. Die gebräuchlichsten sind:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Zusatz !! Beispiel !! Beschreibung<br />
|-<br />
| '''noArg''' || <code>reset:noArg</code>|| Es werden keine weiteren Argumente mehr benötigt. In so einem Fall wird bei der Auswahl keine Textbox oder ähnliches angezeigt, da keine weiteren Argumente für diesen Befehl notwendig sind.<br />
|-<br />
| '''slider''':<min>,<step>,<max> || <code>dim:slider,0,1,100</code>|| Es wird ein Schieberegler angezeigt um den Parameter auszuwählen. Dabei werden als Zusatzparameter Minimum, Schrittweite und Maximum angegeben.<br />
|-<br />
| '''colorpicker''' || <code>rgb:colorpicker,RGB</code>|| es wird ein Colorpicker angezeigt, der dem Anwender die Auswahl einer Farbe ermöglicht. Bitte dazu auch den Wiki Artikel zum Colorpicker lesen: [[Color#Colorpicker]]<br />
|-<br />
| '''multiple''' || <code>group:multiple,Telefon,Multimedia,Licht,Heizung</code> || Es erscheint ein Auswahldialog, wo man verschiedene Werte durch klicken auswählen kann. Optional kann man in einem Freitext eigene Werte ergänzen. dieser Dialog wird bspw. bei der Raum-Auswahl (Attribut "room") oder der Gruppen-Auswahl (Attribut "group") in FHEMWEB genutzt. <br />
|-<br />
| '''sortable''' || <code>command:sortable,monday,tuesday,...</code> || Es erscheint ein Auswahldialog, wo man verschiedene Werte auswählen und sortieren kann. Man kann dabei Werte durch Klicken auswählen und durch Drag'n'Drop sortieren.<br />
|}<br />
<br />
Es gibt noch weitere solcher Widgets. Eine genaue Auflistung dazu findet sich in der [http://fhem.de/commandref.html#widgetOverride commandref] unter widgetOverride zu FHEMWEB.<br />
<br />
'''Hinweise'''<br />
<br />
- Damit in einer Eingabe bereits der aktuelle Wert vorbelegt bzw. in einer Auswahlliste der aktuelle Wert vorselektiert ist, muss es im Modul bzw. Gerät ein Reading mit dem gleichen Namen wie die Set-Option geben. Der Wert des gleichnamigen Readings wird dann als Vorbelegung / Vorselektion verwendet. <br />
<br />
- bei den üblichen Kommandos wie on off sollte man auf noArg verzichten, da diese durch FHEMWeb automatisch in der Raumübersicht angezeigt werden. Wenn man hier noArg spezifiziert, so werden diese nicht neben dem Modul in der Raumübersicht angezeigt und der User muss sich diese vie webCmd dann erst selbst definieren, was natürlich unschön ist<br />
<br />
- der User kann sich in der Raumübersicht nach wie vor via webCmd eine entsprechende Steuerung anlegen.<br />
<br />
=== X_Attr ===<br />
Die Attr-Funktion implementiert Prüfungen der bei einem <code>attr</code> übergebenen Werte und eventuell zusätzliche Aktionen wenn ein Attribut gesetzt wird. Die Liste der möglichen Attribute wird in der <code>[[#X_Initialize|X_Initialize]]-Funktion</code> definiert ([[#X_Initialize|siehe oben]]). Fhem ruft bei einem Attr-Befehl die zuständige <code>X-Attr-Funktion</code> auf und wenn diese keine Fehlermeldung sondern <code>undef</code> zurückgibt, dann schreibt fhem.pl die bei <code>attr</code> angegebenen Werte in die jeweilige Datenstruktur <code>$attr{$name}-> ...</code><br />
<br />
Beispiel:<br />
<br />
<pre><br />
X_Attr(@)<br />
{<br />
my ($cmd,$name,$aName,$aVal) = @_;<br />
# $cmd can be "del" or "set"<br />
# $name is device name<br />
# aName and aVal are Attribute name and value<br />
if ($cmd eq "set") {<br />
if ($aName eq "Regex") {<br />
eval { qr/$aVal/ };<br />
if ($@) {<br />
Log3 $name, 3, "X: Invalid regex in attr $name $aName $aVal: $@";<br />
return "Invalid Regex $aVal";<br />
}<br />
}<br />
}<br />
return undef;<br />
}<br />
</pre><br />
<br />
Zusätzlich ist es möglich auch übergebene Attributwerte zu normalisieren und korrigieren, indem man im Parameterhash den ursprünglichen Wert anpasst. Dies erfolgt im Beispiel über die Modifikation des Wertes mit Index 3 im Parameterarray, also <code>$_[3]</code>.<br />
<br />
Die Attr-Funktion bekommt nicht den Hash der Geräteinstanz übergeben, da sie ja auch keine Werte dort speichern muss, sondern den Befehl <code>set</code> oder <code>del</code> je nachdem ob ein Attribut gesetzt oder gelöscht wird, den Namen der Geräteinstanz sowie den Namen des Attributs und seinen Wert.<br />
Im obigen Beispiel wird für ein Attribut mit Namen Regex geprüft ob die Regex fehlerhaft ist. Falls sie ok ist, wird <code>undef</code> zurückgegeben und fhem.pl speichert den Wert des Attributs.<br />
<br />
Falls man Attribute mit Platzhaltern definiert (Wildcard-Attribute), z.B. mit<br />
<pre><br />
$hash->{AttrList} =<br />
"reading[0-9]*Name " .<br />
# usw.<br />
</pre><br />
dann können Anwender Attribute wie reading01Name, reading02Name etc. setzen. Leider funktioniert das bisher nicht durch Klicken, da Fhemweb nicht alle denkbaren Ausprägungen in einem Dropdown anbieten kann. Der Benutzer muß solche Attribute über den <code>attr</code> Befehl eintippen.<br />
<br />
Man kann jedoch in der Attr-Funktion neu gesetzte Ausprägungen von Wildcard-Attributen an die gerätespezifische userattr-Variable anfügen. Dann können bereits gesetzte Attribute in Fhemweb durch Klicken ausgewählt und geändert werden.<br />
Dazu reicht ein Aufruf von <br />
<br />
<pre><br />
addToDevAttrList($name, $aName);<br />
</pre><br />
<br />
in der Attr-Funktion wenn ein Attribut gesetzt wird.<br />
<br />
=== X_Read ===<br />
<br />
Die X_Read-Funktion wird aus der Hauptschleife von FHEM aus aufgerufen wenn das Gerät, für das das Modul zuständig ist, Daten bereit gestellt hat, die gelesen werden können. Im folgenden Beispiel wird über eine serielle Schnittstelle (beziehungsweise über einen USB-To-Seriell-Konverter) von einem angeschlossenen Gerät gelesen. Dazu werden die bisher verfügbaren Daten mit der Funktion <code>DevIo_SimpleRead</code> gelesen. Da die Übertragung möglicherweise noch nicht vollständig ist, kann es sein, dass kurz darauf die X_Read-Funktion wieder aufgerufen wird und ein weiterer Teil oder der Rest der Daten gelesen werden kann.<br />
Die Funktion muss daher prüfen ob schon alle erwarteten Daten angekommen sind und gegebenenfalls die bisher gelesenen Daten zwischenspeichern. Es bietet sich an, dies im Hash der Geräteinstanz zu tun. Im Beispiel ist dies <code>$hash->{buffer}</code> an den die jeweils gelesenen Daten angehängt werden bis die folgende Prüfung ein für das jeweilige Protokoll passendes Frame identifiziert.<br />
<br />
<pre><br />
sub X_Read($)<br />
{<br />
my ($hash) = @_;<br />
my $name = $hash->{NAME};<br />
<br />
# read from serial device<br />
my $buf = DevIo_SimpleRead($hash); <br />
return "" if ( !defined($buf) );<br />
<br />
# convert to hex string to make parsing with regex easier<br />
$hash->{buffer} .= unpack ('H*', $buf); <br />
Log3 $name, 5, "Current buffer content: " . $hash->{buffer};<br />
<br />
# did we already get a full frame?<br />
if ($hash->{buffer} =~ "ff1002(.{4})(.*)1003(.{4})ff(.*)") <br />
...<br />
</pre><br />
<br />
Die zu lesenden Nutzdaten können dann je nach Protokoll des Geräts beispielsweise an einer festgelegten Stelle im Frame (dann in <code>$hash->{buffer}</code>) stehen oder aus dem Kontext mit einem Regex-Match extrahiert werden und in Readings gespeichert werden (siehe unten).<br />
<br />
=== X_Ready ===<br />
<br />
Wird im Main-Loop aufgerufen falls das Modul in <code>@readyfnlist</code> existiert. Prüft, ob das Gerät Daten zum empfangen hat. Beim Initialisieren des Moduls sollte es sich in die Liste eintragen.<br />
<br />
Weiterführende Informationen: [[#KommunikationvomGerätZuLogischenModulen]]<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Ready($)<br />
{<br />
my ($hash) = @_;<br />
return DevIo_OpenDev($hash, 1, undef )<br />
if ( $hash->{STATE} eq "disconnected" );<br />
<br />
# This is relevant for windows/USB only<br />
my $po = $hash->{USBDev};<br />
my ( $BlockingFlags, $InBytes, $OutBytes, $ErrorFlags ) = $po->status;<br />
return ( $InBytes > 0 );<br />
}<br />
</pre><br />
<br />
=== X_Notify ===<br />
<br />
Die X_Notify-Funktion wird aus der Funktion DoTrigger in fhem.pl heraus aufgerufen wenn ein Modul Events erzeugt hat. Damit kann ein Modul auf Events anderer Module reagieren. Typische Beispiele sind das Filelog-Modul oder das Average-Modul. Average reagiert auf Events anderer Module und erweitert diese mit der Berechnung von Tages- und Monats-Durchschnittswerten.<br />
<br />
Die Notify-Funktion bekommt dafür zwei Hash-Referenzen übergeben: den Hash des eigenen Geräts und den Hash des Geräts, das die Events erzeugt hat. <br />
Über den Hash des eigenen Geräts kann die Notify-Funktion beispielsweise auf die Internals oder Attribute des eigenen Geräts zugreifen.<br />
Über den Hash des Gerätes und die <code>deviceEvents</code> Funktion kann auf die aktuellen Events zugegriffen werden. Über den zweiten Parameter dieser Routine lässt sich bestimmen ob für das Reading <code>state</code> ein 'normales' Event (d.h. in der form <code><reading>: <wert></code>) erzeugen soll (Wert: 1) oder ob z.b. aus Gründen der rückwärts Kompatibilität ein Event ohne <code><reading>: </code> erzeugt werden soll. Falls dem Anwender die Wahl des für state verwendeten Formats überlassen werden soll ist hierzu das <code>addStateEvent</code> Attribut vorzusehen.<br />
<br />
Der direkte Zugriff auf <code>$hash->{CHANGED}</code> ist nicht mehr zu empfehlen.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Notify($$)<br />
{<br />
my ($own_hash, $dev_hash) = @_;<br />
my $ownName = $own_hash->{NAME}; # own name / hash<br />
<br />
return "" if(IsDisabled($ownName)); # Return without any further action if the module is disabled<br />
<br />
my $devName = $dev_hash->{NAME}; # Device that created the events<br />
<br />
my $events = deviceEvents($dev,1);<br />
return if( !$events );<br />
<br />
foreach my $event (@{$events}) {<br />
$event = "" if(!defined($event));<br />
<br />
# Examples:<br />
# $event = "readingname: value" <br />
# or<br />
# $event = "INITIALIZED" (for device "global")<br />
#<br />
# processing $event with further code<br />
}<br />
}<br />
</pre><br />
<br />
Da die Notify-Funktion für jedes Gerät mit allen seinen Events aufgerufen wird, muss sie in einer Schleife alle Events prüfen und entscheiden, ob es mit dem jeweiligen Event etwas tun möchte. Ein Gerät, das die Notify-Funktion implementiert sieht dafür typischerweise einen regulären Ausdruck vor, der für die Filterung verwendet wird.<br />
<br />
Wenn man nur gezielt von bestimmten Definitionen Events erhalten will, kann man diese auch in Form einer kommaseparierten Liste von Definitions-Namen in <code>$hash->{NOTIFYDEV}</code> angeben. Bspw. kann man in der Define-Funktion dort diesen Wert setzen. Dadurch wird die Notify-Funktion nur aufgerufen wenn eines der dort gelisteten Definitionen ein Event erzeugt hat. Ein typischer Fall ist die Begrenzung von Events auf "global":<br />
<br />
<pre><br />
in der Define-Funktion:<br />
<br />
$hash->{NOTIFYDEV} = "global";<br />
$hash->{NOTIFYDEV} = "global,Definition_A,Definition_B";<br />
</pre><br />
<br />
Dies schont insbesondere bei grossen Installationen Ressourcen, da die Notify-Funktion nicht sämtliche Events, sondern nur noch Events der hier gelisteten Definitionen erhält. Dadurch erfolgen deutlich weniger Aufrufe der Notify-Funktion, was Systemressourcen schont.<br />
<br />
Als anschauliches Beispiel und für weitere Details eignet sich das Modul 98_Average.pm. Es ist aber (noch) nicht auf deviceEvents umgestellt da es durch das Erzeugen zusätzlicher Events im Quelldevice eine Sonderstellung hat.<br />
<br />
ToDo: NotifyOrderPrefix ?<br />
<br />
=== X_DbLog_splitFn ===<br />
Mit der DbLog_SplitFn kann der Modulautor selbst festlegen, wie die Events des Moduls in die Bestandteile Reading/Value/Unit zerlegt werden um ein korrektes Logging per DbLog zu gewährleisten.<br><br />
Eingangsparameter: <br><br />
1. Das generierte Event<br><br />
2. das eventgenerierende Device<br><br />
Rückgabewerte: Array: Reading/Value/Unit<br />
<br />
Beispiel:<br />
<pre><br />
sub X_DbLog_splitFn($$)<br />
{<br />
my ($event, $device) = @_;<br />
my ($reading, $value, $unit);<br />
my $hash = $defs{$device}<br />
<br />
if($event =~ m/temperature/) {<br />
$reading = 'temperature';<br />
$value = substr($event,12,4);<br />
$unit = '°C';<br />
} <br />
<br />
return ($reading, $value, $unit);<br />
}<br />
</pre><br />
<br />
=== X_Shutdown ===<br />
Mit der X_Shutdown Funktion kann ein Modul bei einem Shutdown von FHEM die geöffneten Ressourcen schließen.<br><br />
Eingangsparameter: Der $hash einer Modul-Instanz<br><br />
<br />
Beispiel:<br />
<pre><br />
sub X_Shutdown($)<br />
{<br />
my ($hash) = @_;<br />
<br />
DevIo_CloseDev($hash);<br />
return undef;<br />
}<br />
</pre><br />
<br />
== Pollen von Geräten ==<br />
Wenn Geräte von sich aus keine Informationen senden sondern abgefragt werden müssen, kann man im Modul die Funktion <code>InternalTimer</code> verwenden. Man übergibt ihr den Zeitpunkt für den nächsten Aufruf, den Namen der Funktion, die aufgerufen werden soll, den zu übergebenden Parameter und ein Flag ob der erste Aufruf verzögert werden soll falls die Initialiserung des Geräts noch nicht abgeschlossen ist. Als zu übergebender Parameter wird üblicherweise der Hash der betroffenen Geräteinstanz verwendet. Damit hat die aufgerufene Funktion Zugriff auf alle wichtigen Daten der Geräteinstanz. Eventuell zusätzlich benötigte Werte können einfach als weitere Internals über den Hash zugänglich gemacht werden.<br />
<br />
Beispielsweise könnte man für das Abfragen eines Geräts in der Define-Funktion den Timer folgendermassen setzen:<br />
<br />
<pre><br />
# initial request after 2 secs, there timer is set to interval for further update<br />
InternalTimer(gettimeofday()+2, "X_GetUpdate", $hash, 0); <br />
</pre><br />
<br />
in der Funktion <code>X_GetUpdate</code> selbst wird dann der Timer neu gesetzt, so dass nach einem Intervall die Funktion erneut aufgerufen wird:<br />
<br />
<pre><br />
sub X_GetUpdate($)<br />
{<br />
my ($hash) = @_;<br />
my $name = $hash->{NAME};<br />
InternalTimer(gettimeofday()+$hash->{Interval}, "X_GetUpdate", $hash, 1);<br />
Log3 $name, 4, "X: GetUpdate called ...";<br />
</pre><br />
<br />
Im weiteren Verlauf der Funktion könnte man dann das Gerät abfragen und die abgefragten Werte in Readings speichern. Falls das Abfragen der Werte jedoch zu einer Verzögerung und damit zu einer Blockade von FHEM führen kann, ist es möglich, in der GetUpdate-Funktion nur die Aufforderung zum Senden bestimmter Daten an das angeschlossene Gerät zu senden und dann das Lesen über die oben beschriebene Read-Funktion zu implementieren, die beim Anstehen von Daten aufgerufen wird.<br />
<br />
== Logging / Debugging ==<br />
Um Innerhalb eines Moduls eine Protokollmeldung in die Fhem-Logdatei zu schreiben, wird die Funktion Log3 aufgerufen:<br />
<pre><br />
Log3 $name, 3, "X: Problem erkannt ...";<br />
</pre><br />
<br />
Die Parameter der Funktion Log3 sind der Name oder der Hash der Geräteinstanz, das Verbose-Level, in dem die Meldung sichtbar sein soll und die Meldung selbst.<br />
Den Namen der Geräteinstanz kann man in den Funktionen, die den Hash übergeben bekommen einfach aus diesem Hash nehmen:<br />
<br />
<pre><br />
my $name = $hash->{NAME};<br />
</pre><br />
<br />
Um für ein neues Modul das Verbose-Level zu erhöhen, ohne gleich für das Gesamte FHEM alle Meldungen zu erzeugen kann man den Befehl <br />
<code>attr gerätename verbose</code> verwenden. Beispielsweise <code>attr PM verbose 5</code><br />
<br />
Damit bietet es sich an im Modul Meldungen, die im normalen Betrieb nicht benötigt werden, beim Aufruf von Log3 mit dem Level 4 oder 5 anzugeben. Wenn man dann bei der Fehlersuche mehr Meldungen sehen möchte, erhöht man mit attr X verbose das Level für das betroffene Gerät.<br />
<br />
<br />
== Zweistufiges Modell für Module ==<br />
<br />
siehe auch<br />
* [http://forum.fhem.de/index.php/topic,18920.msg128100.html#msg128100|The FHEM two-level model]<br />
* [http://forum.fhem.de/index.php/topic,13438.msg83643.html#msg83643|Zum Initialize bei physikalischen und logischen Geräten]<br />
<br />
Das zweistufige Modell besteht aus <br />
* physisches Modul - z.B. für CUL (00_CUL.pm), der mehrer Protokolle empfängt, u.a. FS20<br />
* logische Modul(e) - z.B. das Protokoll FS20 (10_FS20.pm)<br />
<br />
Das physische Modul öffnet die Datenverbindung zum Gerät. <br />
<br />
=== Kommunikation vom Gerät zu den logischen Modulen ===<br />
{{Anker|KommunikationvomGerätZuLogischenModulen}}<br />
Die [[#X_Read|X_Read]]-Funktion wird aus der Hauptschleife von Fhem aufgerufen sobald das Gerät, für das das Modul zuständig ist, Daten bereit gestellt hat, die gelesen werden können.<br />
<br />
Unter Windows funktioniert "select" nur für Geräte, die via TCP verbunden sind. Für alle anderen Geräte ist eine [[#X_Ready|X_Ready]]-Funktion von Nöten, die 10x pro Sekunde das Gerät abfrägt und "true" zurück gibt, sollten Daten bereit stehen.<br />
<br />
Die X_Read-Funktion stellt sicher, dass die Daten<br />
* komplett und<br />
* korrekt<br />
sind und sie ruft die globale Funktion Dispatch() mit einer Nachricht auf.<br />
<br />
Dispatch() sucht nach einem passenden lokalen Modul via <br />
* $hash->{Clients} oder $hash->{MatchList} im physischen Modul<br />
* $hash->{Match} in allen passenden logischen Modulen<br />
und ruft X_Parse in den gefundenen Modulen auf.<br />
<br />
X_Parse <br />
* untersucht die übergebenen Daten (von Dispatch() übergeben)<br />
* setzt alle [[#Readings|readings]] via readings*update Funktionen<br />
* gibt den Namen des logischen Device zurück<br />
<br />
Es findet kein Event-Triggering statt, wenn die readings*update Funktionen <br />
* von X_Parse aufgerufen werden und<br />
* X_Parse wiederum von Dispatch() aufgerufen wurde.<br />
(Im Gegensatz zum direkten Aufrufen der readings*update Funktionen ohne vorhergehendes Dispatch() )<br />
<br />
Dispatch() triggert das Event-Handling für das von X_Parse zurückgegebene logische Device.<br />
<br />
=== Kommunikation von den logischen Modulen zum Gerät ===<br />
<br />
Um von einem logischen Modul an ein physisches Gerät zu senden, wird im logischen Modul das Attribut IODev mit dem namen des physischen Devices gesetzt.<br />
Der Befehl<br />
<code>AssignIoPort($hash);</code><br />
in der X_Define-Funktion des logischen Devices erledigt das.<br />
<br />
Als Befehl zum Schreiben vom logischen ins physische Gerät soll <code>IOWrite()</code> verwendet werden. IOWrite() ruft im physischen Gerät die X_Write-Funktion auf.<br />
<br />
Wenn es keine direkte Kommunikation zwischen dem logischen und dem physischen Gerät gibt(keine direkten Aufrufe von Funktionen, kein direktes überprüfen von $hash Werten,...) so können die Module hintereinander geschaltet werden (z.B. für Routerfunktionen wie in RFR) oder mittels FHEM2FHEM:RAW zwei Fhem Installationen verbunden werden und die logischen Devices werden dennoch funktionieren.<br />
<br />
== Ergänzende Hinweise ==<br />
Die Wahl der vorangestellten Nummer für den Dateinamen eines neuen Moduls hat keine Bedeutung mehr, es sei denn die Nummer ist 99. Module, die mit 99_ beginnen, werden von FHEM automatisch geladen. Module mit einer anderen Nummer nur wenn ein <code>define</code>-Befehl dafür sorgt, dass das Modul geladen wird.<br />
<br />
Wenn ein Modul Initialisierungsdaten benötigt, sollten diese im Modul selbst enthalten sein. Eine zusätzliche Datei oder sogar ein Unterverzeichnis mit mehreren Dateien ist bei FHEM nicht üblich und sollte bei Modulen, die mit FHEM ausgeliefert werden nur in Rücksprache mit Rudolf König angelegt werden, da sie sonst bei einem Update nicht verteilt werden.<br />
<br />
== Weitere Informationen ==<br />
Wenn man weitere Details wissen möchte, ist ein erster sinnvoller Schritt ein Blick in die Datei fhem.pl. Dort sieht man im Perl-Code wie die Module aufgerufen werden, was vorher passiert und was danach. Am Anfang der Datei (ca. ab Zeile 130) findet man beispielsweise eine Liste der globalen Variablen, die den Modulen zur Verfügung stehen sowie Details zu den wichtigen Hashes %modules und %defs. Wer mit Perl noch nicht so gut klar kommt, dem hilft eventuell ein Blick auf die Perldoc Website[http://perldoc.perl.org/] oder in das Perl-Buch seiner Wahl. Auch die FHEM Commandref [http://fhem.de/commandref.html] sollte nicht unterschätzt werden. Es stehen oft mehr interessante Details auch für Modulentwickler darin als man zunächst vermuten könnte.<br />
<br />
<br />
== "Hello World" Beispiel ==<br />
<br />
98_Hello.pm<br />
<br />
<pre><br />
package main;<br />
use strict;<br />
use warnings;<br />
<br />
my %Hello_gets = (<br />
"whatyouwant" => "can't",<br />
"whatyouneed" => "try sometimes",<br />
"satisfaction" => "no"<br />
);<br />
<br />
sub Hello_Initialize($) {<br />
my ($hash) = @_;<br />
<br />
$hash->{DefFn} = 'Hello_Define';<br />
$hash->{UndefFn} = 'Hello_Undef';<br />
$hash->{SetFn} = 'Hello_Set';<br />
$hash->{GetFn} = 'Hello_Get';<br />
$hash->{AttrFn} = 'Hello_Attr';<br />
$hash->{ReadFn} = 'Hello_Read';<br />
<br />
$hash->{AttrList} =<br />
"formal:yes,no "<br />
. $readingFnAttributes;<br />
}<br />
<br />
sub Hello_Define($$) {<br />
my ($hash, $def) = @_;<br />
my @param = split('[ \t]+', $def);<br />
<br />
if(int(@param) < 3) {<br />
return "too few parameters: define <name> Hello <greet>";<br />
}<br />
<br />
my $hash->{name} = $param[0];<br />
my $hash->{greet} = $param[2];<br />
<br />
return undef;<br />
}<br />
<br />
sub Hello_Undef($$) {<br />
my ($hash, $arg) = @_; <br />
# nothing to do<br />
return undef;<br />
}<br />
<br />
sub Hello_Get($@) {<br />
my ($hash, @param) = @_;<br />
<br />
return '"get Hello" needs at least one argument' if (int(@param) < 2);<br />
<br />
my $name = shift @param;<br />
my $opt = shift @param;<br />
if(!$Hello_gets{$opt}) {<br />
my @cList = keys %Hello_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
<br />
if($attr{$name}{formal} eq 'yes') {<br />
return $Hello_gets{$opt}.', sir';<br />
}<br />
return $Hello_gets{$opt};<br />
}<br />
<br />
sub Hello_Set($@) {<br />
my ($hash, @param) = @_;<br />
<br />
return '"set Hello" needs at least one argument' if (int(@param) < 2);<br />
<br />
my $name = shift @param;<br />
my $opt = shift @param;<br />
my $value = join("", @param);<br />
<br />
if(!defined($Hello_gets{$opt})) {<br />
my @cList = keys %Hello_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
$hash->{STATE} = $Hello_gets{$opt} = $value;<br />
<br />
return "$opt set to $value. Try to get it.";<br />
}<br />
<br />
<br />
sub Hello_Attr(@) {<br />
my ($cmd,$name,$attr_name,$attr_value) = @_;<br />
if($cmd eq "set") {<br />
if($attr_name eq "formal") {<br />
if($attr_value !~ /^yes|no$/) {<br />
my $err = "Invalid argument $attr_value to $attr_name. Must be yes or no.";<br />
Log 3, "Hello: ".$err;<br />
return $err;<br />
}<br />
} else {<br />
return "Unknown attr $attr_name";<br />
}<br />
}<br />
return undef;<br />
}<br />
<br />
1;<br />
<br />
=pod<br />
=begin html<br />
<br />
<a name="Hello"></a><br />
<h3>Hello</h3><br />
<ul><br />
<i>Hello</i> implements the classical "Hello World" as a starting point for module development. <br />
You may want to copy 98_Hello.pm to start implementing a module of your very own. See <br />
<a href="http://www.fhemwiki.de/wiki/DevelopmentModuleIntro">DevelopmentModuleIntro</a> for an <br />
in-depth instruction to your first module.<br />
<br><br><br />
<a name="Hellodefine"></a><br />
<b>Define</b><br />
<ul><br />
<code>define &lt;name&gt; Hello &lt;greet&gt;</code><br />
<br><br><br />
Example: <code>define HELLO Hello TurnUrRadioOn</code><br />
<br><br><br />
The "greet" parameter has no further meaning, it just demonstrates<br />
how to set a so called "Internal" value. See <a href="http://fhem.de/commandref.html#define">commandref#define</a> <br />
for more info about the define command.<br />
</ul><br />
<br><br />
<br />
<a name="Helloset"></a><br />
<b>Set</b><br><br />
<ul><br />
<code>set &lt;name&gt; &lt;option&gt; &lt;value&gt;</code><br />
<br><br><br />
You can <i>set</i> any value to any of the following options. They're just there to <br />
<i>get</i> them. See <a href="http://fhem.de/commandref.html#set">commandref#set</a> <br />
for more info about the set command.<br />
<br><br><br />
Options:<br />
<ul><br />
<li><i>satisfaction</i><br><br />
Defaults to "no"</li><br />
<li><i>whatyouwant</i><br><br />
Defaults to "can't"</li><br />
<li><i>whatyouneed</i><br><br />
Defaults to "try sometimes"</li><br />
</ul><br />
</ul><br />
<br><br />
<br />
<a name="Helloget"></a><br />
<b>Get</b><br><br />
<ul><br />
<code>get &lt;name&gt; &lt;option&gt;</code><br />
<br><br><br />
You can <i>get</i> the value of any of the options described in <br />
<a href="#Helloset">paragraph "Set" above</a>. See <br />
<a href="http://fhem.de/commandref.html#get">commandref#get</a> for more info about <br />
the get command.<br />
</ul><br />
<br><br />
<br />
<a name="Helloattr"></a><br />
<b>Attributes</b><br />
<ul><br />
<code>attr &lt;name&gt; &lt;attribute&gt; &lt;value&gt;</code><br />
<br><br><br />
See <a href="http://fhem.de/commandref.html#attr">commandref#attr</a> for more info about <br />
the attr command.<br />
<br><br><br />
Attributes:<br />
<ul><br />
<li><i>formal</i> no|yes<br><br />
When you set formal to "yes", all output of <i>get</i> will be in a<br />
more formal language. Default is "no".<br />
</li><br />
</ul><br />
</ul><br />
</ul><br />
<br />
=end html<br />
<br />
=cut<br />
</pre><br />
<br />
Der HTML-Code zwischen den Tags <code>=pod</code> und <code>=cut</code> dient zur Generierung der commandref.html. Der HTML-Inhalt wird automatisch beim Verteilen des Moduls im Rahmen des Update-Mechanismus aus jedem Modul extrahiert und daraus die Commandref in verschiedenen Sprachen erstellt. Eine detaillierte Beschreibung wie ein Commandref-Abschnitt in einem Modul definiert wird, siehe: [[Guidelines zur Dokumentation]]<br />
<br />
== Noch zu beschreiben ==<br />
* Zweistufiges Modell für Module<br />
* Funktion X_State_Fn: {{Link2Forum|Topic=32680}}, siehe auch [[DevelopmentState]]<br />
* FW_summaryFn (wird von FHEMWEB aufgerufen fuer Raum-Uebersicht)<br />
* FW_detailFn (wird von FHEMWEB aufgerufen fuer Detail-Ansicht)<br />
* DevIO<br />
* AsyncOutputFn / asyncOutput<br />
* SetExtensions / SetExtensionsCancel<br />
* ExceptFn (gleiche wie ReadFn aber EXCEPT_FD anstelle von FD)<br />
* FingerprintFn<br />
* ParseFn<br />
<br />
[[Kategorie:Development]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=16369HTTPMOD2016-09-09T20:49:15Z<p>StefanStrobel: /* Some help with Regular Expressions */ added a little clarification about newlines</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that Fhem HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. Please note that buf might contain special characters like newlines but they are not shown in fhemweb. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Bodenfeuchtesensor&diff=16324Bodenfeuchtesensor2016-09-04T08:01:57Z<p>StefanStrobel: Verweise auf Opus XT300 und Davis LSMS ergänzt</p>
<hr />
<div>Dieser Beitrag beschreibt die Möglichkeiten der Einbindung von Bodenfeuchtesenoren in FHEM<br />
<br />
[[File:1wire_Bodenfeuchtesensor1.jpg|mini|hochkant=2.5|Plot-Beispiel]]<br />
<br />
== Opus XT300 ==<br />
Der [https://www.plantcaretools.com/de/onlineshop/drahtloser-bodenfeuchte-sensor-detail Opus Xt300] Sensor und Sender <br />
wird in diesem [https://forum.fhem.de/index.php?topic=25546.0 Thread] diskutiert. Er kann über einen RFXtrx empfangen werden.<br />
<br />
== Davis Wireless Leaf & Soil Moisture/Temperature Station ==<br />
Ist eine Ergänzungseinheit für Davis Wetterstationen, wird aus einer Solarzelle mit Strom versorgt und kann über einen Jeelink V3c mit erweitertem Davis [https://forum.fhem.de/index.php/topic,44092.0.html Sketch] direkt empfangen werden. Der Sketch gibt die Feuchtigkeitswerte als Saugspannung in kPa bzw. cb aus.<br />
Das Funkprotokoll verwendet Channel-Hopping und hat eine sehr gute Reichweite. <br />
Als Bodenfeuchtesensor verwendet die Station [http://www.irrometer.com/sensors.html Watermark 200SS] [https://de.wikipedia.org/wiki/Tensiometer_(Bodenfeuchte) Tensiometer], die auch einzeln gekauft werden können und so an anderen Sendern (z.B. Panstamps) angeschlossen werden können.<br />
<br />
== Nutzung von panStamps zur Funkbasierten Datenübertragung ==<br />
Die folgende Platine ist eine Modifikation/Weiterentwicklung des von Panstamp.org vertriebenen BatterieBoards. Die Platine ist passend zum G203 Gainta Gehäuse. Der [[panStamp]] ist so angeordnet, dass die "5cm gerade Antenne" aus dem panStamp Shop exakt in das Gehäuse passt. Es sind Anschlüsse für bis zu 4 Bodenfeuchtesensoren vorhanden. Weiterhin ist ein OneWire Anschluss integriert an dem der Anschluss eines DS18B20 zur Bodentemperaturmessung vorgesehen ist. <br />
<br />
Optional kann der Sensor zusätzlich mit einem Solarpanel ausgestatt werden. Dieses lädt den eingesetzten Akku auf, sobald die SolarSpannung > AkkuSpannung ist.<br />
Der Akku MUSS(!) mindestens die zehnfache Leistung des Solarpanels haben, ansonsten ist ein Laderegler notwendig. Bsp: Akku: 2700mA -> Solarpanel: 270mA<br />
Ein für den Gehäusedeckel passendes Solarmodul ist zb. bei ebay unter folgendem Namen zu finden: '''3V 270mA 0,8W 110x55mm Solarmodul Solarzelle Polykristallin vergossen'''. Ist ein Solarpanel angeschlossen, so wird dessen Spannung dem panStamp zugeführt. Damit kann mit entsprechender Sketcherweiterung die solare Einstrahlung gemessen und nach FHEM zusammen mit der Batteriespannung übermittelt werden.<br />
<br />
In Planung ist, dieses mit Silikon sauber auf der Gehäuseoberseite zu verkleben und die benötigten Kabel zuvor durch 2 Löcher im Deckel zu ziehen.<br />
<br />
Achtung: Zur Zeit kann der Sketch nur 2 Bodenfeuchtesensoren bedienen. Die Weiterentwicklung ist in Arbeit.<br />
Die Weiterentwicklung zum Umweltsensor ist hier [[panStamp_Umweltsensor]] beschrieben. Der dortige Sketch kann ebenfalls mittels Konfigurationsanpassung hier verwendet werden. Damit sind nun auch 1wire und der SolarWert in FHEM nutzbar.<br />
<br />
=== Schaltplan und Bauteilliste ===<br />
[[Datei:Arduino_Bodenfeuchtesensor_v0.1_Schaltplan.jpg|200px|thumb|right|Schaltplan]]<br />
[[Datei:Arduino_Bodenfeuchtesensor_v0.1_Platine_Oben.jpg|200px|thumb|right|Platinenlayout Oberseite]]<br />
[[Datei:Arduino_Bodenfeuchtesensor_v0.1_Platine_Unten.jpg|200px|thumb|right|Platinenlayout Unterseite]]<br />
[[Datei:Arduino_Bodenfeuchtesensor_v0.1_Aufbau_Oben.jpg|200px|thumb|right|Platine fertig aufgebaut Oberseite]]<br />
[[Datei:Arduino_Bodenfeuchtesensor_v0.1_Aufbau_Unten.jpg|200px|thumb|right|Platine fertig aufgebaut Unterseite]]<br />
<br />
Bauteilliste:<br />
<br />
{| class="wikitable"<br />
! Bauteil<br />
! Bezeichnung<br />
! Shop<br />
! BauteilNr<br />
|- <br />
| L1<br />
| LQH4C Speicherdrossel <br />
| Mouser.com<br />
Reichelt <br />
<br />
Reichelt<br />
| 81-LQH43CN100K03L<br />
L-1212FPS 10µ<br />
<br />
L-1616FPS 10µ<br />
|- <br />
| C1, C2<br />
| Keramikkondensator Typ:X7R 10uf<br />
| Reichelt<br />
Mouser.com<br />
| X5R-G0805 10/16<br />
810-CGJ4J1X7R0J106AC<br />
|- <br />
| C3<br />
| Keramikkondensator Typ:X7R 1uF<br />
| Reichelt<br />
Mouser.com<br />
| X7R-G0805 1,0/16<br />
810-C2012X5R1C105K-2<br />
|- <br />
| C4<br />
| Keramikkondensator Typ:X7R 100nF<br />
| Reichelt<br />
Mouser.com<br />
| X7R-G0805 100N<br />
581-0805YD104KAT2A<br />
|- <br />
| IC1<br />
| MAX1724 Schaltregler 1.5uA IQ Step-Up DC/DC Converter<br />
| Mouser.com<br />
tme.eu<br />
<br />
Ebay: G&C Supermarket<br />
| MAX1724EZK33T<br />
MAX1724EZK33+T<br />
<br />
MAX1724<br />
|- <br />
| K1,K2,K3,K4,K5<br />
| Anreihklemme 3,5mm 3Pol <br />
| IT-WNS<br />
| AK-3.5-3-GY<br />
|- <br />
| R1<br />
| Widerstandsnetzwerk 100K <br />
| Reichelt<br />
| SIL 5-4 100K<br />
|- <br />
| R2<br />
| SMD Widerstand 1k <br />
| Reichelt<br />
Mouser.com<br />
| SMD-0805 1K<br />
71-CRCW0805-1.0K-E3<br />
|- <br />
| R4<br />
| SMD Widerstand 4k7 <br />
| Reichelt<br />
Mouser.com<br />
| SMD-0805 4,70K<br />
71-CRCW0805-4.7K-E3<br />
|- <br />
| R5<br />
| Widerstandsnetzwerk 100K <br />
| Reichelt<br />
| SIL 7-6 100K<br />
|- <br />
| J1<br />
| Stiftleiste 1x5Pol <br />
| Reichelt<br />
| MPE 087-1-005<br />
|- <br />
| D1<br />
| LED 3mm LowCurrent 2mA<br />
| Reichelt<br />
| LED 3MM 2MA GN<br />
|-<br />
| TASTER<br />
| Reset TASTER 9314 <br />
| Reichelt<br />
| TASTER 9314<br />
|- <br />
| Batteriehalter<br />
| Batteriehalter 1x AA Mignon(Akku)<br />
| Reichelt<br />
| HALTER 1XAAP<br />
|-<br />
| K7<br />
| Anschluss Solarpanel (VERT PCB 2Pin TIN FRICTION LOCK)<br />
| Reichelt<br />
Mouser<br />
| zb. MPE 087-1-002<br />
538-22-23-2021 <br />
|-<br />
| D2<br />
| BAV70 Schaltdiode SMD, SOT-23, 250V, 1A<br />
| Reichelt<br />
| BAV 70 SMD<br />
|-<br />
| T1<br />
| BCW 61 Transistor SMD PNP SOT-23 32V 0,1A 0,25W<br />
| Reichelt<br />
| BCW 61C SMD<br />
|-<br />
| T2<br />
| BC847 Transistor SMD NPN SOT-23 45V 0,1A 0,25W<br />
| Reichelt<br />
| BC 847B SMD<br />
|-<br />
| T3<br />
| BSS138 Transistor SMD N-FET SOT-23 50V 0,22A<br />
| Reichelt<br />
| BSS 138 SMD<br />
|-<br />
| IC<br />
| panStamp AVR<br />
| panstamp.com<br />
| panStamp AVR<br />
|-<br />
| --<br />
| Female Header 2,54 mm, 1X16, straight<br />
| Reichelt<br />
| MPE 094-1-016<br />
|}<br />
<br />
== Onewire Bodenfeuchtesensor ==<br />
Das vorgestellte Platinenlayout dient dazu, an einem DS2450 ([[1-Wire]]) vier Vegetronix Bodenfeuchtesensoren anzuschließen. Die Platine passt in das ELV IP65 Gehäuse G203. Der Anschluss über fünf Kabeldurchführungen PG7 erfolgt von unten.<br />
<br />
=== Schaltplan und Bauteilliste ===<br />
[[Datei:1wire_BF_Schaltplan.jpg|300px|thumb|right|Schaltplan]]<br />
[[Datei:1wire_BF_Platine.jpg|300px|thumb|right|Platinenlayout]]<br />
Bauteilliste:<br />
<br />
{| class="wikitable"<br />
! Bauteil<br />
! Bezeichnung<br />
! Shop<br />
! BauteilNr<br />
|- <br />
| D1,D2<br />
| Diode <br />
| Reichelt<br />
| BAT85<br />
|- <br />
| C1<br />
| Elko 47uF <br />
| Reichelt<br />
| RAD 47/16<br />
|- <br />
| C3<br />
| Keramikkondensator 100N, 10% <br />
| Reichelt<br />
| X7R-5 100N<br />
|- <br />
| K1,K2,K3,K4<br />
| Anreihklemme 3Pol <br />
| Reichelt<br />
| AKL 055-03<br />
|- <br />
| K5<br />
| Anreihklemme 4Pol <br />
| Reichelt<br />
| AKL 055-04<br />
|- <br />
| J1<br />
| Stiftleiste 1x3Pol <br />
| Reichelt<br />
| MPE 087-1-003<br />
|- <br />
| Jumper<br />
| Jumper <br />
| Reichelt<br />
| JUMPER 2,54 SW <br />
|}<br />
Das Platinenlayout (erstellt mit Target3001) ist im unten aufgeführten Forenthread angehängt.<br />
<br />
== Hinweise zum Betrieb mit FHEM ==<br />
Die Einbindung der einzelnen Sensoren ist im [[1-Wire_Feuchtemessung]] Beitrag beschrieben.<br />
<br />
== Weitere Hinweise ==<br />
Keine.<br />
<br />
== Links ==<br />
* [http://sourceforge.net/p/fhem/code/HEAD/tree/trunk/fhem/contrib/arduino/ soilmoisture sketch] auf sourceforge<br />
<br />
[[Kategorie:1-Wire]]<br />
[[Kategorie:panStamp]]<br />
[[Kategorie:Feuchtesensoren]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=16173HTTPMOD2016-08-15T16:11:59Z<p>StefanStrobel: Erläuterungen ergänzt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that Fhem HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negaation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
which will create a reading with the Name "Chlor" (as shown above) and take the value 0.25 from the JSON string.<br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
It might seem very simple at first sight to use extractAllJSON but if you prefer readings with a meaningful name you should instead define these readings with readingXXName and readingXXJSON or getXXName and getXXJSON individually. Of Course it would be possible to create additional user readings outside HTTPMOD but doing calculations, naming and formatting inside HTTPMOD is more efficient.<br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=15769HTTPMOD2016-07-02T09:24:26Z<p>StefanStrobel: Randbemerkung entfernt (aktuelle Version ist eingecheckt), neue Attribute dokumentiert</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
Please also note that Fhem HttpUtils need the global attribute dnsServer to be set in order to work really non blocking even when dns requests can not be answered.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negaation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>reading</code>, <code>internal</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;(get|set|reading)[0-9]+AlwaysNum<br />
:if set to 1 this attributes forces reading names to end with a -1, -01 (depending on the above described AutoNumLen) even if just one value is parsed.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;get|reading[0-9]*DeleteIfUnmatched<br />
:If set to 1 this attribute causes certain readings to be deleted when the parsing of the website does not match the specified reading. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation does not parse this reading again. This is especially useful for parsing that creates several matches / readings and the number of matches can vary from request to request. For example if reading01Regex creates 4 readings in one update cycle and in the next cycle it only matches two times then the readings containing the remaining values from the last round will be deleted.<br />
:Please note that this mechanism will not work in all cases after a restart. Especially when a get definition does not contain its own parsing definition but ExtractAllJSON or relies on HTTPMOD to use all defined reading.* attributes to parse the responsee to a get command, old readings might not be deleted after a restart of fhem.<br />
;get|reading[0-9]*DeleteOnError<br />
:If set to 1 this attribute causes certain readings to be deleted when the website can not be reached and the HTTP request returns an error. Internally HTTPMOD remembers which kind of operation created a reading (update, Get01, Get02 and so on). Specified readings will only be deleted if the same operation returns an error.<br />
The same restrictions as for DeleteIfUnmatched apply regarding a fhem restart.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;alignTime<br />
:Aligns each periodic read request for the defined interval to this base time. This is typcally something like 00:00 (see the Fhem at command)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request as well as UNMATCHED_READINGS and LAST_REQUEST.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=15629HTTPMOD2016-06-23T09:27:38Z<p>StefanStrobel: /* JSON Lists */ added more JSON examples and explanations</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
{{Randnotiz|RNText=An extended / modified version of this module that among other features adds XPath and JSON support is currently under development. <br />
If you want to participate in early tests, please follow this {{Link2Forum|Topic=45176|LinkText=discussion thread}} in FHEM forum.<br />
This page has already been updated to describe most of the new features. So if you need one of the new features described here and yout HTTPMOD is still the old version, load the new module <br />
from the forum and help testing<br />
}}<br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negaation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
then a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=15626HTTPMOD2016-06-23T09:24:10Z<p>StefanStrobel: </p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
{{Randnotiz|RNText=An extended / modified version of this module that among other features adds XPath and JSON support is currently under development. <br />
If you want to participate in early tests, please follow this {{Link2Forum|Topic=45176|LinkText=discussion thread}} in FHEM forum.<br />
This page has already been updated to describe most of the new features. So if you need one of the new features described here and yout HTTPMOD is still the old version, load the new module <br />
from the forum and help testing<br />
}}<br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:attributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands ''interval'', ''stop'', ''start'', ''reread'', ''upgradeAttributes'', ''storeKeyValue''<br />
<br />
== Simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formatting and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups) or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading01regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
<br />
If there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text you are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
<br />
If you have trouble defining a regular expression that matches a certain name, then many complicated characters and then a number, it might be helpful to use a negaation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
=== JSON Lists ===<br />
<br />
imagine the HTTP Response contains:<br />
<br />
<source lang="perl"><br />
{ "power":"0",<br />
"modes":["Off","SimpleColor","RainbowChase"],<br />
"code1":3,<br />
"code2":4<br />
}<br />
</source><br />
<br />
the a configuration like <br />
<br />
<source lang="perl"><br />
attr device reading01JSON modes <br />
attr device reading01Name Mode <br />
</source><br />
<br />
will create a list of Subreadings just like a regex with multiple matches can create multiple subreadings:<br />
<br />
{| class="wikitable"<br />
| Mode-1 || Off<br />
|-<br />
| Mode-2 || SimpleColor<br />
|-<br />
| Mode-3 || RainbowChase <br />
|}<br />
<br />
if you don't want several subreadings but one reading that contains the list of modes, you can specify a recombine expression:<br />
<br />
<source lang="perl"><br />
attr device reading01Name Modes <br />
attr device reading01RecombineExpr join ",", @matchlist <br />
</source><br />
<br />
which will create one reading containing a list:<br />
<br />
{| class="wikitable"<br />
| Modes || Off,SimpleColor,RainbowChase<br />
|}<br />
<br />
JSON parsing specifications also don't Need to match exactly. If there is no exact match for a defined reading, the HTTPMOD will try to Interpret the specification as a regex and look for json object paths that match the specification as a regex. For example:<br />
<br />
<source lang="perl"><br />
attr device reading01Name CodeElem<br />
attr device reading01JSON code<br />
</source><br />
<br />
which will create a list of readings:<br />
<br />
{| class="wikitable"<br />
| CodeElem-1|| 3<br />
|-<br />
| CodeElem-2 || 4<br />
|}<br />
<br />
and of course they could also be recombined into one reading with a RecombineExpr Attribute.<br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid then you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determining how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existence of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an alternative lower update frequency. When the interval defined initially in the define is over and the normal readings are read from the device, the update function will check for additional get parameters that should be included in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
== All attributes ==<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built in cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling cookies should be sufficient and no sidRegex and no manual definition of a cookie header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* Beispiel: [[Pollenflug]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=DevelopmentModuleIntro&diff=15331DevelopmentModuleIntro2016-05-12T17:08:26Z<p>StefanStrobel: /* X_Get */</p>
<hr />
<div>== Einleitung ==<br />
Dieser Text ist in Arbeit und muss noch an einigen Stellen ergänzt werden. <br />
Insbesondere beschreibt der Text derzeit nur einstufige Module. Die Abgrenzung zu zweistufigen Modulen und deren Eigenschaften sollte noch ergänzt werden.<br />
<br />
Um neue Geräte in FHEM verfügbar zu machen, kann man ein eigenes Modul in Perl schreiben, das automatisch von FHEM geladen wird, wenn ein passendes Gerät in FHEM definiert wird. Das Modul definiert dann wie mit dem Gerät kommuniziert wird, stellt Werte ("Readings") innerhalb von FHEM zur Verfügung oder erlaubt es das Gerät mit "Set"-Befehlen zu beeinflussen. Dieser Text soll den Einstieg in die Entwicklung eigener Module erleichtern.<br />
<br />
Mit dem FHEM-Befehl "define", der typischerweise in die zentrale Konfigurationsdatei fhem.cfg eingetragen wird, werden Geräte in FHEM definiert. Der Befehl sorgt dafür dass ein neues Modul bei Bedarf geladen wird und die Initialisierungsfunktion des Moduls aufgerufen wird. <br />
<br />
Damit das funktioniert müssen der Name des Geräts, der Name des Moduls und der Name der Initialisierungsfunktion zueinander passen. Das folgende Beispiel soll dies verdeutlichen:<br />
<br />
Ein Jeelink USB-Stick in einer Fritz-Box könnte beispielsweise mit dem Befehl <code>define JeeLink1 JeeLink /dev/ttyUSB0@57600</code> definiert werden.<br />
<br />
In der fhem.pl wird der define-Befehl verarbeitet, geprüft, ob ein Modul mit Namen JeeLink schon geladen ist und falls nicht ein Modul mit Namen XY_JeeLink.pm im Modulverzeichnis (bei einer FritzBox z.B. /var/media/ftp/fhem/FHEM) gesucht und dann geladen. <br />
Danach wird die Funktion JeeLink_Initialize aufgerufen.<br />
Die Moduldatei muss also nach dem Namen des Geräts benannt werden und eine Funktion mit dem Namen des Geräts und einer _initialize Funktion enthalten.<br />
In der Initialisierungsfunktion des Moduls werden dann die Namen der aller weiteren Funktionen des Moduls, die von fhem.pl aus aufgerufen werden, bekannt gemacht. Dazu wird der Hash - das ist die zentrale Datenstruktur für jede Instanz eines Gerätes - mit entsprechenden Werten gefüllt.<br />
<br />
== Der Hash einer Geräteinstanz ==<br />
Eine Besonderheit in Perl sind [http://de.wikipedia.org/wiki/Assoziatives_Array#Perl assoziative Arrays], (nicht ganz richtig als "Hash" bezeichnet) in denen die Adressierung nicht über eine Zählvariable erfolgt, sondern über einen beliebigen String. Die internen Abläufe bei der Adressierung führen dazu, dass die Speicherung in und der Abruf aus Hashes relativ langsam ist.<br />
<br />
Der zentrale Speicherort für Informationen einer Geräteinstanz bei FHEM ist ein solcher Hash, der seinerseits in fhem.pl von einem globalen Hash referenziert wird. <br />
<br />
<code>$defs{''Devicename''}</code> in fhem.pl verweist auf den Hash der Geräteinstanz. Diesen Verweis (also nur die Adresse) bekommen die Funktionen eines Moduls übergeben, das direkt von fhem.pl aufgerufen wird. In dem Hash stehen beispielsweise die internen Werte des Geräts, die im GUI als "Internals" angezeigt werden oder die Readings des Geräts. Beispiele:<br />
*<code>$hash{NAME}</code> enthält den Namen der Geräteinstanz, <br />
*<code>$hash{TYPE}</code> enthält die Typbezeichnung des Geräts <br />
*<code>$hash->{INTERVAL}</code> enthält ein Abfrageintervall<br />
<br />
==Ausführung von Modulen==<br />
FHEM führt Module normalerweise nicht parallel aus. Daher wäre es ungünstig wenn Module Werte von einem Gerät abfragen und dann auf die Antwort des Geräts warten. In dieser Zeit wäre der Rest von FHEM blockiert. Die Ein- und Ausgabe sollte ohne Blockieren erfolgen und die Verarbeitung mehrerer Ein- und Ausgabekanäle quasi parallel ermöglichen. <br />
<br />
Dafür werden in FHEM zwei zentrale Listen gepflegt, in der die Filedeskriptoren der geöffneten Kommunikatonsverbindungen gespeichert sein können. Auf Linux- bzw. Unix-basierten Plattformen wird der select-Befehl des Betriebssystems verwendet und entsprechend gibt es in FHEM eine selectlist, in der die Filedeskriptoren der Geräedateien (z.B. /dev/ttyUSBx etc.) gespeichert sind. <br />
<br />
In der zentralen Schleife von fhem.pl wird mit select überwacht, ob über eine der geöffneten Schnittstellen Daten zum Lesen anstehen. Wenn dies der Fall ist, dann wird die Lesefunktion (X_Read) des zuständigen Moduls aufgerufen, damit es die Daten entgegennimmt und die Schleife wird weiter ausgeführt.<br />
<br />
Auf Windows-Systemen funktioniert dies anders. Hier können USB/Seriell-Geräte nicht per select überwacht werden. In FHEM unter Windows werden daher diese Schnittstellen kontinuierlich abgefragt ob Daten bereitstehen. Dafür müssen Module zusätzlich zur Lesefunktion eine Abfragefunktion (X_Ready) implementieren, die prüft ob Daten zum Lesen anstehen. Auch auf Linux/Unix-Plattformen hat diese Funktion eine Aufgabe. Falls nämlich eine Schnittstelle ausfällt beziehungsweise ein CUL oder USB-zu-Seriell Adapter ausgesteckt wird, dann wird über diese Funktion regelmäßig geprüft ob die Schnittstelle wieder verfügbar wird.<br />
<br />
Innerhalb der eigentlichen Lesefunktion (X_Read) werden dann die Daten vom zugehörigen Gerät gelesen, das nötige Protokoll implementiert und Werte in Readings geschrieben.<br />
<br />
Auch wenn von einem Anwender über ein <code>get</code> Daten aktiv von einem Gerät angefordert werden sollte nicht blockierend gewartet werden. Eine asynchrone Ausgabe ist über asyncOutput möglich. Siehe {{Link2Forum|Topic=43771|Message=357870|LinkText=Beschreibung}} und {{Link2Forum|Topic=43771|Message=360935|LinkText=Beispiel}}. Weitere Anwendungsbeispiele finden sich im {{Link2Forum|Topic=43052|Message=353477|LinkText=PLEX Modul}} und im überarbeiteten und nicht-blockierenden {{Link2Forum|Topic=42771|Message=348498|LinkText= SYSSTAT Modul}}.<br />
<br />
== Readings ==<br />
Werte, die von einem Gerät gelesen werden und in FHEM zur Verfügung stehen werden Readings genannt. Sie werden als Unterstruktur des Hashes der jeweiligen Geräteinstanz gespeichert, beispielsweise <br />
*<code>$hash{READINGS}{Temp}{VAL}</code> für die Temperatur eines Fühlers<br />
*<code>$hash{READINGS}{Temp}{TIME}</code> für den Zeitstempel der Messung<br />
<br />
Für den lesenden Zugriff auf Readings steht die Funktion ReadingsVal($$$) zur Verfügung.<br />
<br />
Readings werden im statefile von FHEM automatisch zwischengespeichert, damit sie nach einem Neustart sofort wieder zur Verfügung stehen, auch bevor sie vom Modul neu gesetzt oder aktualisiert werden.<br />
<br />
Readings, die mit einem Punkt im Namen beginnen, haben eine funktionale Besonderheit. Sie werden im FhemWeb nicht angezeigt und können somit als "Permanentspeicher" innerhalb des Moduls genutzt werden.<br />
<br />
Zum Setzen von Readings sollen <br />
*bei Gruppen von Readings der Funktionsblock <code>readingsBeginUpdate</code>, <code>readingsBulkUpdate</code> (mehrfach wiederholt), <code>readingsEndUpdate</code><br />
*bei einzelnen Updates die Funktion <code>readingsSingleUpdate</code> <br />
aufgerufen werden. Dabei kann man auch angeben, ob dabei ein Event ausgelöst werden soll oder nicht. Events erzeugen spürbare Last auf dem System (siehe NotifyFn), das Ändern von Readings ohne dass dabei Events erzeugt werden jedoch nicht.<br />
<br />
Eine Sequenz zum Setzen von Readings könnte folgendermaßen aussehen:<br />
<br />
<pre><br />
readingsBeginUpdate($hash);<br />
readingsBulkUpdate($hash, $readingName1, $wert1 );<br />
readingsBulkUpdate($hash, $readingName2, $wert2 );<br />
readingsEndUpdate($hash, 1);<br />
</pre><br />
<br />
== Internals ==<br />
Werte, die das Modul intern als Teil des Hashes speichert, die aber keine Readings sind, nennt man Internals. Sie werden ebenfalls als Unterstruktur des Hashes der jeweiligen Geräteinstanz gespeichert, beispielswiese <code>$hash->{INTERVAL}</code> für ein Abfrageintervall, das beim Define-Befehl übergeben wurde und als Internal gespeichert wird. Internals werden jedoch im Gegensatz zu Readings nicht im statefile zwischengespeichert. <br />
<br />
Falls Werte wie das gerade erwähnte Intervall nicht über den Define-Befehl gesetzt werden sollen und im Betrieb einfach änderbar sein sollen, ist eine alternative Möglichkeit die Speicherung in so genannten Attributen. Dann würde man den Define-Befehl so implementieren, dass er kein Intervall übergeben bekommt und statt dessen nach dem Define-Befehl zusätzlich den Befehl <code>attr</code> erwarten.<br />
<br />
== Attribute ==<br />
Parameter einer Geräteinstanz können mit dem Befehl <code>attr</code> als so genannte Attribute gesetzt und damit dem Modul zur Verfügung gestellt werden. Attribute werden zusammen mit der Definition der Geräte beim Speichern der aktuellen Konfiguration von FHEM in die Konfigurationsdatei geschrieben, die auch bei jedem Neustart von FHEM wieder gelesen wird. Zur Laufzeit werden Attribute in der globalen Datenstruktur <code>$attr{$name}</code> gespeichert. Ein Attribut mit dem Namen <code>header</code> würde beispielsweise mit <code>$attr{$name}{header}</code> adressiert (<code>$attr{$name}->{'header'}</code> wäre eine alternative aber unübliche Schreibweise für die selbe Variable). <br />
<br />
Zum Auslesen solcher Attribute sollte die Funktion <code>AttrVal($$$)</code> verwendet werden.<br />
<br />
Welche Attribute ein Modul unterstützt sollte in der Funktion <code>[[#X_Initialize|X_Initialize]]</code> durch Setzen der Variable <code>$hash->{AttrList}</code> bekannt gemacht werden (siehe unten). Wenn beim Setzen von Attributen die Werte geprüft werden sollen oder zusätzliche Funktionalität implementiert werden muss, dann kann dies in der Funktion <code>[[#X_Attr|X_Attr]]</code> ([[#X_Attr|siehe unten]]) implementiert werden.<br />
<br />
== Die wichtigsten Funktionen in einem Modul ==<br />
Eine typische Grundfunktion eines einfachen Moduls ist das Auslesen von Werten von einem physischen Gerät und Bereitstellen dieser Werte innerhalb von FHEM als Readings. Das Geräte könnte beispielsweise an einem USB-Port angeschlossen sein. Folgende Funktionen könnte man beispielsweise in einem Modul mit Namen X implementieren:<br />
* [[#X_Initialize|X_Initialize]] (initialisiert das Modul und gibt de Namen der zusätzlichen Funktionen bekannt)<br />
* [[#X_Define|X_Define]] (wird beim <code>define</code> aufgerufen)<br />
* [[#X_Undef|X_Undef]] (wird beim <code>delete</code>, sowie <code>rereadcfg</code> aufgerufen. Dient zum Abbau von offenen Verbindungen, Timern, etc.)<br />
* [[#X_Delete|X_Delete]] (wird beim <code>delete</code> aufgerufen um das Gerät endgültig zu löschen)<br />
* [[#X_Set|X_Set]] (wird beim Befehl <code>set</code> aufgerufen um Daten an das Gerät zu senden)<br />
* [[#X_Get|X_Get]] (wird beim Befehl <code>get</code> aufgerufen um Daten vom Gerät abzufragen)<br />
* [[#X_Attr|X_Attr]] (wird beim Befehl <code>attr</code> aufgerufen um beispielsweise Werte zu prüfen)<br />
* [[#X_Read|X_Read]] (wird vom globalen select aufgerufen, falls Daten zur Verfuegung stehen)<br />
* [[#X_Parse|X_Parse]] (wird bei zweistufigen Modulen vom Dispatch aufgerufen und muss hier noch beschrieben werden)<br />
* [[#X_Ready|X_Ready]] (wird unter windows als ReadFn-Erstatz benoetigt bzw. um zu pruefen, ob ein Geraet wieder eingesteckt ist)<br />
* [[#X_Notify|X_Notify]] (falls man benachrichtigt werden will)<br />
* [[#X_Rename|X_Rename]] (falls ein Gerät umbenannt wird)<br />
<br />
Die Funktionen werden im folgenden beschrieben (soweit diese Seite inzwischen vollständig ist):<br />
<br />
=== X_Initialize ===<br />
<br />
<pre><br />
sub X_Initialize($)<br />
{<br />
my ($hash) = @_;<br />
...<br />
</pre><br />
<br />
Das <code>X</code> im Namen muss dabei auf den Namen des Moduls bzw. des definierten Gerätetyps geändert werden. Im Modul mit der Datei <code>36_JeeLink.pm</code> beispielsweise ist der Name der Funktion <code>JeeLink_Initialize</code>. Die Funktion wird von Fhem.pl nach dem Laden des Moduls aufgerufen und bekommt einen Hash für das Modul als zentrale Datenstruktur übergeben. <br />
<br />
Dieser Hash wird im globalen Hash %modules gespeichert. <code>$modules{$ModulName}</code> wäre dabei der Hash für das Modul mit dem Namen <code>$ModulName</code>. Es handelt sich also nicht um den oben beschriebenen Hash der Geräteinstanzen sondern einen Hash, der je Modul Werte enthält, beispielsweise auch die Namen der Funktionen, die das Modul implementiert und die fhem.pl aufrufen soll. Die Initialize-Funktion setzt diese Funktionsnamen, in den Hash des Moduls:<br />
<br />
<pre><br />
$hash->{DefFn} = "X_Define";<br />
$hash->{UndefFn} = "X_Undef";<br />
$hash->{DeleteFn} = "X_Delete";<br />
$hash->{SetFn} = "X_Set";<br />
$hash->{GetFn} = "X_Get";<br />
$hash->{AttrFn} = "X_Attr";<br />
$hash->{NotifyFn} = "X_Notify";<br />
$hash->{ReadFn} = "X_Read";<br />
$hash->{ReadyFn} = "X_Ready";<br />
</pre><br />
<br />
<code>X</code> ist wieder durch den Modulnamen ohne die vorangestellte Zahl zu ersetzen. <br />
Entsprechend können auch die Funktionen <code>X_Read</code>, <code>X_Parse</code> etc. durch Zuweisung an <code>$hash->{ReadFn}</code> etc. bekannt gemacht werden.<br />
<br />
Darüber hinaus sollten die vom Modul unterstützen Attribute definiert werden:<br />
<br />
<pre><br />
$hash->{AttrList} =<br />
"do_not_notify:1,0 " . <br />
"header " .<br />
$readingFnAttributes; <br />
</pre><br />
<br />
In Fhem.pl werden dann die entsprechenden Werte beim Aufruf eines <code>attr</code>-Befehls in die globale Datenstruktur <code>$attr{$name}</code>, z.B. <code>$attr{$name}{header}</code> für das Attribut <code>header</code> gespeichert. Falls im Modul weitere Aktionen oder Prüfungen beim Setzen eines Attributs nötig sind, dann kann wie im Beispiel oben die Funktion <code>X_Attr</code> implementiert und in der Initialize-Funktion bekannt gemacht werden.<br />
<br />
Die Variable <code>$readingFnAttributes</code>, die im obigen Beispiel an die Liste der unterstützten Attribute angefügt wird, definiert Attributnamen, die dann verfügbar werden wenn das Modul zum Setzen von Readings die Funktionen readingsBeginUpdate, readingsBulkUpdate, readingsEndUpdate oder readingsSingleUpdate verwendet. In diesen Funktionen werden Attribute wie <code>event-min-interval</code> oder auch <code>event-on-change-reading</code> ausgewertet. Für Details hierzu siehe commandref.<br />
<br />
<br />
Des weiteren ist es möglich, das Verhalten von Autocreate zu beeinflussen.<br />
<code>x</code> ist durch den Namen der Geräte zu ersetzen. Legt ihr Geräte mit dem Namen LaCrosse an, dann sollte x durch LaCrosse ersetzt werden.<br />
<pre><br />
$hash->{AutoCreate} =<br />
{ "x.*" => { ATTR => "event-min-interval:.*:300 event-on-change-reading:.*", <br />
FILTER => "%NAME", <br />
GPLOT => "temp4hum4:Temp/Hum,"} };<br />
autocreateThreshold => "<count>:<timeout>" <br />
</pre><br />
Mit <code>ATTR =></code> können Vordefinierte Attribute beim Anlegen definiert werden.<br />
Der Wert von <code>FILTER</code> wird als Regex verwendet, wenn ein Filelog angelegt wird. Damit könnt ihr steuern, welche Events ins Filelog komme. Definiert ihr das Feld <code>FILTER</code> nicht, gebt jedoch andere Felder an, dann kann kein filelog mehr automatisch durch autocreate angelegt werden!<br />
Mit Hilfe von <code>GPLOT</code> kann ein Plot angelegt werden. Mit der Angabe definiert ihr, welcher GPLOT angelegt wird.<br />
Mittels <code>autocreateThreshold</code> wird beeinflusst, wie oft <code>count</code>(default 2) und in welchem Zeitabstand <code>timeout</code> (default 60 Sekunden) die gleiche Nachricht empfangen werden muss, damit ein Gerät per autocreate angelegt wird. <br />
Das Verhalten, kann vom Anwender mittels Attribut <code>autocreateThreshold</code> im device "autocreate" überschrieben werden.<br />
<br />
<br />
Das Parsen der Parameter der define, get und set Kommandos sowie deren Übergabe an die DefFn, GetFn und SetFn lässt sich mit<br />
<pre>$hash->{parseParams} = 1;</pre> beeinflussen. Sobald es gesetzt ist wird automatisch [[DevelopmentModuleAPI#parseParams|parseParams]] aufgerufen und die an X_Define, X_Get und X_Set übergebenen Parameter ändern sich wie weiter unten beschrieben.<br />
<br />
=== X_Define ===<br />
Die Define-Funktion eines Moduls wird von Fhem aufgerufen wenn der Define-Befehl für ein Geräte ausgeführt wird und das Modul bereits geladen und mit der Initialize-Funktion initialisiert ist. Sie ist typischerweise dazu da, die übergebenen Parameter zu prüfen und an geeigneter Stelle zu speichern sowie einen Kommunikationsweg zum Gerät zu öffnen (z.B. TCP-Verbindung, USB-Schnittstelle o.ä.)<br />
Sie beginnt typischerweise mit:<br />
<br />
<pre><br />
sub X_Define($$)<br />
{<br />
my ( $hash, $def ) = @_;<br />
my @a = split( "[ \t][ \t]*", $def );<br />
...<br />
</pre><br />
<br />
Als Übergabeparameter bekommt die Define-Funktion den Hash der Geräteinstanz sowie den Rest der Parameter, die im Befehl angegeben wurden. Welche und wie viele Parameter <br />
akzeptiert werden ist Sache dieser Funktion. Im obigen Beispiel wird alles nach dem übergebenen Hash in ein Array aufgeteilt und so können die vom Modul bzw. der Define-Funktion erwarteten Werte über das Array der Reihe nach verarbeitet werden:<br />
<br />
<pre><br />
my $name = $a[0];<br />
my $url = $a[2];<br />
my $inter = 300;<br />
if(int(@a) == 4) { <br />
$inter = $a[3]; <br />
if ($inter < 5) {<br />
return "interval too small, please use something > 5, default is 300";<br />
}<br />
}<br />
</pre><br />
<br />
<br />
Neu: Zum Aufteilen und Parsen von <code>$def</code> lässt sich [[DevelopmentModuleAPI#parseParams|parseParams]] verwenden. Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird parseParams automatisch aufgerufen und X_Define ändert sich wie folgt:<br />
<pre><br />
sub X_Define($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
<br />
Damit die übergebenen Werte auch anderen Funktionen zur Verfügung stehen und an die jeweilige Geräteinstanz gebunden sind, werden die Werte typischerweise als Internals im Hash der Geräteinstanz gespeichert:<br />
<br />
<pre><br />
$hash->{url} = $url;<br />
$hash->{Interval} = $inter;<br />
</pre><br />
<br />
Wenn eine physische Schnittstelle geöffnet werden soll und dann bei verfügbaren Eingabedaten eine Lese-Funktion von Fhem aufgerufen werden soll, dann kann man in der Define-Funktion die Funktion DevIo_OpenDev aufrufen, die sich um alles weitere kümmert. Sie öffnet die Schnittstelle und fügt den Filedeskriptor an die globale Liste offener Verbindungen (selectlist / readyfnlist) an. Damit kann Fhem in seiner Hauptschleife erkennen, von welchem Gerät Daten bereit stehen und die zuständigen Funktionen aufrufen:<br />
<br />
<pre><br />
my $ret = DevIo_OpenDev( $hash, 0, "X_DevInit" );<br />
</pre><br />
<br />
Die optionale Funktion <code>X_DevInit</code> wird zur weiteren Initialisierung der Verbindung von <code>DevIo_OpenDev</code> aufgerufen. Der zweite Übergabeparameter an <code>DevIo_OpenDev</code> (hier <code>0</code>) steht für reopen und wird benötigt, da die Funktion auch aufgerufen wird, wenn ein USB-Geräte beispielsweise im Betrieb aus- und wieder eingesteckt wird. In diesem Fall wird die Funktion mit <code>1</code> aufgerufen.<br />
<br />
=== X_Undef ===<br />
<br />
Die <code>Undef</code>-Funktion wird aufgerufen wenn ein Gerät mit <code>delete</code> gelöscht wird oder bei der Abarbeitung des Befehls rereadcfg, der ebenfalls alle Geräte löscht und danach das Konfigurationsfile neu abarbeitet. Entsprechend müssen in der Funktion typische Aufräumarbeiten durchgeführt werden wie das saubere Schließen von Verbindungen oder das Entfernen von internen Timern sofern diese im Modul zum Pollen verwendet wurden (siehe später). <br />
<br />
Zugewiesene Variablen im Hash der Geräteinstanz, Internals oder Readings müssen hier nicht gelöscht werden. In fhem.pl werden die entsprechenden Strukturen beim Löschen der Geräteinstanz ohnehin vollständig gelöscht.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Undef($$) <br />
{ <br />
my ( $hash, $name) = @_; <br />
DevIo_CloseDev($hash); <br />
RemoveInternalTimer($hash); <br />
return undef; <br />
}<br />
</pre><br />
<br />
=== X_Delete ===<br />
<br />
Die <code>Delete</code>-Funktion ist das Gegenstück zur <code>Define</code>-Funktion und wird aufgerufen wenn ein Gerät mit <code>delete</code> gelöscht wird. <br />
<br />
Wenn ein Gerät mittels <code>delete</code> gelöscht wird, wird zuerst die <code>[[#X_Undef|Undef]]</code>-Funktion aufgerufen um offene Verbindungen abzubauen, anschließend wird die <code>Delete</code>-Funktion aufgerufen. Diese dient eher zum aufräumen von Dateien, welche durch das Modul evtl. für dieses Gerät spezifisch erstellt worden sind. Es geht hier also eher darum, alle Spuren sowohl im laufenden FHEM-Prozess, als auch Dateien oder Verbindungen zu löschen die mit diesem Gerät zu tun haben.<br />
<br />
Dies kann z.B. folgendes sein:<br />
<br />
* Löschen von Dateien im Dateisystem die während der Nutzung dieses Geräts angelegt worden sind.<br />
* Lösen von evtl. Pairings mit dem physikalischen Gerät <br />
<br />
Beispiel:<br />
<pre><br />
sub X_Delete($$) <br />
{ <br />
my ( $hash, $name ) = @_; <br />
<br />
# Löschen von Geräte-assoziiertem Temp-File<br />
unlink($attr{global}{modpath}."/FHEM/FhemUtils/$name.tmp";)<br />
} <br />
</pre><br />
<br />
=== X_Get ===<br />
Die Get-Funktion wird aufgerufen wenn der Fhem-Befehl <code>get</code> mit einem Gerät dieses Moduls ausgeführt wird. Mit <code>get</code> werden typischerweise Werte von einem Gerät abgefragt. Einige Module verwenden für diese Funktion einen Hash im Modul, der die möglichen <code>get</code>-Optionen mit zusätzlichen Werten definiert:<br />
<br />
<pre><br />
my %X_gets = (<br />
"TempSoll" => "XY",<br />
"Steilheit" => "Z"<br />
);<br />
</pre><br />
In der Get-Funktion selbst werden dann die übergebenen Parameter gegen diesen Hash geprüft.<br />
<br />
Beispiel:<br />
<br />
<pre><br />
sub X_Get($@)<br />
{<br />
my ( $hash, @a ) = @_;<br />
return "\"get X\" needs at least one argument" if ( @a < 2 );<br />
my $name = shift @a;<br />
my $opt = shift @a;<br />
if(!$X_gets{$opt}) {<br />
my @cList = keys %X_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
...<br />
</pre><br />
<br />
Die Ausgabe der Meldung mit <code>unknown ... choose one of ...</code> ist dabei wichtig, da sie im GUI-Modul verwendet wird um die möglichen <code>get</code>-Optionen zu ermitteln und als Auswahl anzubieten. Im weiteren Verlauf der Ger-Funktion könnte man dann mit dem physischen Gerät kommunizieren und den gefragten Wert abfragen und diesen als Return-Wert der Get-Funktion zurückgeben.<br />
<br />
<br />
Neu: Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird [[DevelopmentModuleAPI#parseParams|parseParams]] automatisch aufgerufen und X_Get ändert sich wie folgt:<br />
<pre><br />
sub X_Get($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
=== X_Set ===<br />
Die Set-Funktion ist das Gegenteil zur Get-Funktion. Sie ist dafür gedacht, Werte zum physischen Gerät zu schicken. Falls nur interne Werte im Modul gesetzt werden sollen, so sollte statt Set die Attr-Funktion verwendet werden. Attribute werden bei Save-Config auch in der Fhem.cfg gesichert. Set-Befehle nicht.<br />
<br />
Eine Set-Funktion ist ähnlich aufgebaut wie die Get-Funktion, sie bekommt jedoch nach dem Namen der Option auch den zu setzenden Wert übergeben.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Set($@)<br />
{<br />
my ( $hash, @a ) = @_;<br />
return "\"set X\" needs at least an argument" if ( @a < 2 );<br />
my $name = shift @a;<br />
my $opt = shift @a;<br />
my $value = join("", @a);<br />
<br />
if(!defined($X_sets{$opt})) {<br />
my @cList = keys %X_sets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
</pre><br />
<br />
<br />
Neu: Wenn in X_Initialize <pre>$hash->{parseParams} = 1;</pre> gesetzt wurde dann wird [[DevelopmentModuleAPI#parseParams|parseParams]] automatisch aufgerufen und X_Set ändert sich wie folgt:<br />
<pre><br />
sub X_Set($$$)<br />
{<br />
my ( $hash, $a, $h ) = @_;<br />
...<br />
</pre><br />
<br />
<br />
Das GUI FHEM-Web kann für die einzelnen Set-Optionen, die das Modul versteht auch automatisch Eingabehilfen wie Drop-Down Boxen oder Slider erzeugen. In der Detailansicht des GUI kann der Anwender dann die jeweiligen Werte komfortabel auswählen. Dafür muss die Set-Funktion, wenn sie mit der Option <code>?</code> aufgerufen wird, nicht nur einen Text mit <code>"Unknwon ... choose one of ..."</code> zurückgeben sondern den einzelnen Set-Optionen in diesem Rückgabetext nach einem Doppelpunkt Zusatzinformationen anhängen.<br />
Meist prüft man in den Modulen gar nicht auf die Option <code>?</code> sondern gibt generell bei unbekannten Optionen diesen Text zurück.<br />
<br />
Beispiel:<br />
<pre><br />
if(!defined($X_sets{$opt})) {<br />
return "Unknown argument $opt, choose one of mode:verbose,ultra,relaxed turbo:NoArg";<br />
}<br />
</pre><br />
<br />
Mit Kommata getrennte Werte ergeben eine Drop-Down Liste, mit der der User die Werte auswählen kann<br />
<pre>timer:30,120,300<br />
mode:verbose,ultra,relaxed</pre><br />
<br />
Wird kein Doppelpunkt zum Kommando angegeben, so wird eine Eingabezeile angezeigt, die die freie Eingabe eines Wertes erlaubt.<br />
<br />
Man kann jedoch die Eingabe-/Auswahlmöglichkeiten durch Widgets vereinfachen. Dazu gibt man hinter dem Doppelpunkt einen Widgetnamen und widgetspezifische Parameter an. Es existieren mehrere solcher Widgets in FHEMWEB. Die gebräuchlichsten sind:<br />
<br />
{| class="wikitable"<br />
|-<br />
! Zusatz !! Beispiel !! Beschreibung<br />
|-<br />
| '''noArg''' || <code>reset:noArg</code>|| Es werden keine weiteren Argumente mehr benötigt. In so einem Fall wird bei der Auswahl keine Textbox oder ähnliches angezeigt, da keine weiteren Argumente für diesen Befehl notwendig sind.<br />
|-<br />
| '''slider''':<min>,<step>,<max> || <code>dim:slider,0,1,100</code>|| Es wird ein Schieberegler angezeigt um den Parameter auszuwählen. Dabei werden als Zusatzparameter Minimum, Schrittweite und Maximum angegeben.<br />
|-<br />
| '''colorpicker''' || <code>rgb:colorpicker,RGB</code>|| es wird ein Colorpicker angezeigt, der dem Anwender die Auswahl einer Farbe ermöglicht. Bitte dazu auch den Wiki Artikel zum Colorpicker lesen: [[Color#Colorpicker]]<br />
|-<br />
| '''multiple''' || <code>group:multiple,Telefon,Multimedia,Licht,Heizung</code> || Es erscheint ein Auswahldialog, wo man verschiedene Werte durch klicken auswählen kann. Optional kann man in einem Freitext eigene Werte ergänzen. dieser Dialog wird bspw. bei der Raum-Auswahl (Attribut "room") oder der Gruppen-Auswahl (Attribut "group") in FHEMWEB genutzt. <br />
|-<br />
| '''sortable''' || <code>command:sortable,monday,tuesday,...</code> || Es erscheint ein Auswahldialog, wo man verschiedene Werte auswählen und sortieren kann. Man kann dabei Werte durch Klicken auswählen und durch Drag'n'Drop sortieren.<br />
|}<br />
<br />
Es gibt noch weitere solcher Widgets. Eine genaue Auflistung dazu findet sich in der [http://fhem.de/commandref.html#widgetOverride commandref] unter widgetOverride zu FHEMWEB.<br />
<br />
'''Hinweise'''<br />
<br />
- Damit in einer Eingabe bereits der aktuelle Wert vorbelegt bzw. in einer Auswahlliste der aktuelle Wert vorselektiert ist, muss es im Modul bzw. Gerät ein Reading mit dem gleichen Namen wie die Set-Option geben. Der Wert des gleichnamigen Readings wird dann als Vorbelegung / Vorselektion verwendet. <br />
<br />
- bei den üblichen Kommandos wie on off sollte man auf noArg verzichten, da diese durch FHEMWeb automatisch in der Raumübersicht angezeigt werden. Wenn man hier noArg spezifiziert, so werden diese nicht neben dem Modul in der Raumübersicht angezeigt und der User muss sich diese vie webCmd dann erst selbst definieren, was natürlich unschön ist<br />
<br />
- der User kann sich in der Raumübersicht nach wie vor via webCmd eine entsprechende Steuerung anlegen.<br />
<br />
=== X_Attr ===<br />
Die Attr-Funktion implementiert Prüfungen der bei einem <code>attr</code> übergebenen Werte und eventuell zusätzliche Aktionen wenn ein Attribut gesetzt wird. Die Liste der möglichen Attribute wird in der <code>[[#X_Initialize|X_Initialize]]-Funktion</code> definiert ([[#X_Initialize|siehe oben]]). Fhem ruft bei einem Attr-Befehl die zuständige <code>X-Attr-Funktion</code> auf und wenn diese keine Fehlermeldung sondern <code>undef</code> zurückgibt, dann schreibt fhem.pl die bei <code>attr</code> angegebenen Werte in die jeweilige Datenstruktur <code>$attr{$name}-> ...</code><br />
<br />
Beispiel:<br />
<br />
<pre><br />
X_Attr(@)<br />
{<br />
my ($cmd,$name,$aName,$aVal) = @_;<br />
# $cmd can be "del" or "set"<br />
# $name is device name<br />
# aName and aVal are Attribute name and value<br />
if ($cmd eq "set") {<br />
if ($aName eq "Regex") {<br />
eval { qr/$aVal/ };<br />
if ($@) {<br />
Log3 $name, 3, "X: Invalid regex in attr $name $aName $aVal: $@";<br />
return "Invalid Regex $aVal";<br />
}<br />
}<br />
}<br />
return undef;<br />
}<br />
</pre><br />
<br />
Zusätzlich ist es möglich auch übergebene Attributwerte zu normalisieren und korrigieren, indem man im Parameterhash den ursprünglichen Wert anpasst. Dies erfolgt im Beispiel über die Modifikation des Wertes mit Index 3 im Parameterarray, also <code>$_[3]</code>.<br />
<br />
Die Attr-Funktion bekommt nicht den Hash der Geräteinstanz übergeben, da sie ja auch keine Werte dort speichern muss, sondern den Befehl <code>set</code> oder <code>del</code> je nachdem ob ein Attribut gesetzt oder gelöscht wird, den Namen der Geräteinstanz sowie den Namen des Attributs und seinen Wert.<br />
Im obigen Beispiel wird für ein Attribut mit Namen Regex geprüft ob die Regex fehlerhaft ist. Falls sie ok ist, wird <code>undef</code> zurückgegeben und fhem.pl speichert den Wert des Attributs.<br />
<br />
Falls man Attribute mit Platzhaltern definiert (Wildcard-Attribute), z.B. mit<br />
<pre><br />
$hash->{AttrList} =<br />
"reading[0-9]*Name " .<br />
# usw.<br />
</pre><br />
dann können Anwender Attribute wie reading01Name, reading02Name etc. setzen. Leider funktioniert das bisher nicht durch Klicken, da Fhemweb nicht alle denkbaren Ausprägungen in einem Dropdown anbieten kann. Der Benutzer muß solche Attribute über den <code>attr</code> Befehl eintippen.<br />
<br />
Man kann jedoch in der Attr-Funktion neu gesetzte Ausprägungen von Wildcard-Attributen an die gerätespezifische userattr-Variable anfügen. Dann können bereits gesetzte Attribute in Fhemweb durch Klicken ausgewählt und geändert werden.<br />
Dazu reicht ein Aufruf von <br />
<br />
<pre><br />
addToDevAttrList($name, $aName);<br />
</pre><br />
<br />
in der Attr-Funktion wenn ein Attribut gesetzt wird.<br />
<br />
=== X_Read ===<br />
<br />
Die X_Read-Funktion wird aus der Hauptschleife von FHEM aus aufgerufen wenn das Gerät, für das das Modul zuständig ist, Daten bereit gestellt hat, die gelesen werden können. Im folgenden Beispiel wird über eine serielle Schnittstelle (beziehungsweise über einen USB-To-Seriell-Konverter) von einem angeschlossenen Gerät gelesen. Dazu werden die bisher verfügbaren Daten mit der Funktion <code>DevIo_SimpleRead</code> gelesen. Da die Übertragung möglicherweise noch nicht vollständig ist, kann es sein, dass kurz darauf die X_Read-Funktion wieder aufgerufen wird und ein weiterer Teil oder der Rest der Daten gelesen werden kann.<br />
Die Funktion muss daher prüfen ob schon alle erwarteten Daten angekommen sind und gegebenenfalls die bisher gelesenen Daten zwischenspeichern. Es bietet sich an, dies im Hash der Geräteinstanz zu tun. Im Beispiel ist dies <code>$hash->{buffer}</code> an den die jeweils gelesenen Daten angehängt werden bis die folgende Prüfung ein für das jeweilige Protokoll passendes Frame identifiziert.<br />
<br />
<pre><br />
sub X_Read($)<br />
{<br />
my ($hash) = @_;<br />
my $name = $hash->{NAME};<br />
<br />
# read from serial device<br />
my $buf = DevIo_SimpleRead($hash); <br />
return "" if ( !defined($buf) );<br />
<br />
# convert to hex string to make parsing with regex easier<br />
$hash->{buffer} .= unpack ('H*', $buf); <br />
Log3 $name, 5, "Current buffer content: " . $hash->{buffer};<br />
<br />
# did we already get a full frame?<br />
if ($hash->{buffer} =~ "ff1002(.{4})(.*)1003(.{4})ff(.*)") <br />
...<br />
</pre><br />
<br />
Die zu lesenden Nutzdaten können dann je nach Protokoll des Geräts beispielsweise an einer festgelegten Stelle im Frame (dann in <code>$hash->{buffer}</code>) stehen oder aus dem Kontext mit einem Regex-Match extrahiert werden und in Readings gespeichert werden (siehe unten).<br />
<br />
=== X_Ready ===<br />
<br />
muss noch beschrieben werden.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Ready($)<br />
{<br />
my ($hash) = @_;<br />
return DevIo_OpenDev($hash, 1, undef )<br />
if ( $hash->{STATE} eq "disconnected" );<br />
<br />
# This is relevant for windows/USB only<br />
my $po = $hash->{USBDev};<br />
my ( $BlockingFlags, $InBytes, $OutBytes, $ErrorFlags ) = $po->status;<br />
return ( $InBytes > 0 );<br />
}<br />
</pre><br />
<br />
=== X_Notify ===<br />
<br />
Die X_Notify-Funktion wird aus der Funktion DoTrigger in fhem.pl heraus aufgerufen wenn ein Modul Events erzeugt hat. Damit kann ein Modul auf Events anderer Module reagieren. Typische Beispiele sind das Filelog-Modul oder das Average-Modul. Average reagiert auf Events anderer Module und erweitert diese mit der Berechnung von Tages- und Monats-Durchschnittswerten.<br />
<br />
Die Notify-Funktion bekommt dafür zwei Hash-Referenzen übergeben: den Hash des eigenen Geräts und den Hash des Geräts, das die Events erzeugt hat. <br />
Über den Hash des eigenen Geräts kann die Notify-Funktion beispielsweise auf die Internals oder Attribute des eigenen Geräts zugreifen.<br />
Über den Hash des Gerätes und die <code>deviceEvents</code> Funktion kann auf die aktuellen Events zugegriffen werden. Über den zweiten Parameter dieser Routine lässt sich bestimmen ob für ads Reading <code>state</code> ein 'normales' Event (d.h. in der form <code><reading>: <wert></code>) erzeugen soll (Wert: 1) oder ob z.b. aus Gründen der rückwärts Kompatibilität ein Event ohne <code><reading>: </code> erzeugt werden soll. Falls dem Anwender die Wahl des für state verwendeten Formats überlassen werden soll ist hierzu das <code>addStateEvent</code> Attribut vorzusehen.<br />
<br />
Der direkte Zugriff auf <code>$hash->{CHANGED}</code> ist nicht mehr zu empfehlen.<br />
<br />
Beispiel:<br />
<pre><br />
sub X_Notify($$)<br />
{<br />
my ($own_hash, $dev_hash) = @_;<br />
my $ownName = $own_hash->{NAME}; # own name / hash<br />
<br />
return "" if(IsDisabled($ownName)); # Return without any further action if the module is disabled<br />
<br />
my $devName = $dev_hash->{NAME}; # Device that created the events<br />
<br />
my $events = deviceEvents($dev,1);<br />
return if( !$events );<br />
<br />
foreach my $event (@{$events}) {<br />
$event = "" if(!defined($event));<br />
<br />
# Examples:<br />
# $event = "readingname: value" <br />
# or<br />
# $event = "INITIALIZED" (for device "global")<br />
#<br />
# processing $event with further code<br />
}<br />
}<br />
</pre><br />
<br />
Da die Notify-Funktion für jedes Gerät mit allen seinen Events aufgerufen wird, muss sie in einer Schleife alle Events prüfen und entscheiden, ob es mit dem jeweiligen Event etwas tun möchte. Ein Gerät, das die Notify-Funktion implementiert sieht dafür typischerweise einen regulären Ausdruck vor, der für die Filterung verwendet wird.<br />
<br />
Wenn man nur gezielt von bestimmten Definitionen Events erhalten will, kann man diese auch in Form einer kommaseparierten Liste von Definitions-Namen in <code>$hash->{NOTIFYDEV}</code> angeben. Bspw. kann man in der Define-Funktion dort diesen Wert setzen. Dadurch wird die Notify-Funktion nur aufgerufen wenn eines der dort gelisteten Definitionen ein Event erzeugt hat. Ein typischer Fall ist die Begrenzung von Events auf "global":<br />
<br />
<pre><br />
in der Define-Funktion:<br />
<br />
$hash->{NOTIFYDEV} = "global";<br />
$hash->{NOTIFYDEV} = "global,Definition_A,Definition_B";<br />
</pre><br />
<br />
Dies schont insbesondere bei grossen Installationen Ressourcen, da die Notify-Funktion nicht sämtliche Events, sondern nur noch Events der hier gelisteten Definitionen erhält. Dadurch erfolgen deutlich weniger Aufrufe der Notify-Funktion, was Systemressourcen schont.<br />
<br />
Als anschauliches Beispiel und für weitere Details eignet sich das Modul 98_Average.pm. Es ist aber (noch) nicht auf deviceEvents umgestellt da es durch das Erzeugen zusätzlicher Events im Quelldevice eine Sonderstellung hat.<br />
<br />
ToDo: NotifyOrderPrefix ?<br />
<br />
=== X_DbLog_splitFn ===<br />
Mit der DbLog_SplitFn kann der Modulautor selbst festlegen, wie die Events des Moduls in die Bestandteile Reading/Value/Unit zerlegt werden um ein korrektes Logging per DbLog zu gewährleisten.<br><br />
Eingangsparameter: Das generierte Event<br><br />
Rückgabewerte: Array: Reading/Value/Unit<br />
<br />
Beispiel:<br />
<pre><br />
sub X_DbLog_splitFn($)<br />
{<br />
my ($event) = @_;<br />
my ($reading, $value, $unit);<br />
<br />
if($event =~ m/temperature/) {<br />
$reading = 'temperature';<br />
$value = substr($event,12,4);<br />
$unit = '°C';<br />
} <br />
<br />
return ($reading, $value, $unit);<br />
}<br />
</pre><br />
<br />
== Pollen von Geräten ==<br />
Wenn Geräte von sich aus keine Informationen senden sondern abgefragt werden müssen, kann man im Modul die Funktion <code>InternalTimer</code> verwenden. Man übergibt ihr den Zeitpunkt für den nächsten Aufruf, den Namen der Funktion, die aufgerufen werden soll, den zu übergebenden Parameter und ein Flag ob der erste Aufruf verzögert werden soll falls die Initialiserung des Geräts noch nicht abgeschlossen ist. Als zu übergebender Parameter wird üblicherweise der Hash der betroffenen Geräteinstanz verwendet. Damit hat die aufgerufene Funktion Zugriff auf alle wichtigen Daten der Geräteinstanz. Eventuell zusätzlich benötigte Werte können einfach als weitere Internals über den Hash zugänglich gemacht werden.<br />
<br />
Beispielsweise könnte man für das Abfragen eines Geräts in der Define-Funktion den Timer folgendermassen setzen:<br />
<br />
<pre><br />
# initial request after 2 secs, there timer is set to interval for further update<br />
InternalTimer(gettimeofday()+2, "X_GetUpdate", $hash, 0); <br />
</pre><br />
<br />
in der Funktion <code>X_GetUpdate</code> selbst wird dann der Timer neu gesetzt, so dass nach einem Intervall die Funktion erneut aufgerufen wird:<br />
<br />
<pre><br />
sub X_GetUpdate($)<br />
{<br />
my ($hash) = @_;<br />
my $name = $hash->{NAME};<br />
InternalTimer(gettimeofday()+$hash->{Interval}, "X_GetUpdate", $hash, 1);<br />
Log3 $name, 4, "X: GetUpdate called ...";<br />
</pre><br />
<br />
Im weiteren Verlauf der Funktion könnte man dann das Gerät abfragen und die abgefragten Werte in Readings speichern. Falls das Abfragen der Werte jedoch zu einer Verzögerung und damit zu einer Blockade von FHEM führen kann, ist es möglich, in der GetUpdate-Funktion nur die Aufforderung zum Senden bestimmter Daten an das angeschlossene Gerät zu senden und dann das Lesen über die oben beschriebene Read-Funktion zu implementieren, die beim Anstehen von Daten aufgerufen wird.<br />
<br />
== Logging / Debugging ==<br />
Um Innerhalb eines Moduls eine Protokollmeldung in die Fhem-Logdatei zu schreiben, wird die Funktion Log3 aufgerufen:<br />
<pre><br />
Log3 $name, 3, "X: Problem erkannt ...";<br />
</pre><br />
<br />
Die Parameter der Funktion Log3 sind der Name oder der Hash der Geräteinstanz, das Verbose-Level, in dem die Meldung sichtbar sein soll und die Meldung selbst.<br />
Den Namen der Geräteinstanz kann man in den Funktionen, die den Hash übergeben bekommen einfach aus diesem Hash nehmen:<br />
<br />
<pre><br />
my $name = $hash->{NAME};<br />
</pre><br />
<br />
Um für ein neues Modul das Verbose-Level zu erhöhen, ohne gleich für das Gesamte FHEM alle Meldungen zu erzeugen kann man den Befehl <br />
<code>attr gerätename verbose</code> verwenden. Beispielsweise <code>attr PM verbose 5</code><br />
<br />
Damit bietet es sich an im Modul Meldungen, die im normalen Betrieb nicht benötigt werden, beim Aufruf von Log3 mit dem Level 4 oder 5 anzugeben. Wenn man dann bei der Fehlersuche mehr Meldungen sehen möchte, erhöht man mit attr X verbose das Level für das betroffene Gerät.<br />
<br />
<br />
== Zweistufiges Modell für Module ==<br />
siehe auch<br />
* [http://forum.fhem.de/index.php/topic,18920.msg128100.html#msg128100|The FHEM two-level model]<br />
* [http://forum.fhem.de/index.php/topic,13438.msg83643.html#msg83643|Zum Initialize bei physikalischen und logischen Geräten]<br />
<br />
Das zweistufige Modell besteht aus <br />
* physisches Modul - z.B. für CUL (00_CUL.pm), der mehrer Protokolle empfängt, u.a. FS20<br />
* logische Modul(e) - z.B. das Protokoll FS20 (10_FS20.pm)<br />
<br />
Das physische Modul öffnet die Datenverbindung zum Gerät. <br />
<br />
=== Kommunikation vom Gerät zu den logischen Modulen ===<br />
Die [[#X_Read|X_Read]]-Funktion wird aus der Hauptschleife von Fhem aufgerufen sobald das Gerät, für das das Modul zuständig ist, Daten bereit gestellt hat, die gelesen werden können.<br />
<br />
Unter Windows funktioniert "select" nur für Geräte, die via TCP verbunden sind. Für alle anderen Geräte ist eine [[#X_Ready|X_Ready]]-Funktion von Nöten, die 10x pro Sekunde das Gerät abfrägt und "true" zurück gibt, sollten Daten bereit stehen.<br />
<br />
Die X_Read-Funktion stellt sicher, dass die Daten<br />
* komplett und<br />
* korrekt<br />
sind und sie ruft die globale Funktion Dispatch() mit einer Nachricht auf.<br />
<br />
Dispatch() sucht nach einem passenden lokalen Modul via <br />
* $hash->{Clients} oder $hash->{MatchList} im physischen Modul<br />
* $hash->{Match} in allen passenden logischen Modulen<br />
und ruft X_Parse in den gefundenen Modulen auf.<br />
<br />
X_Parse <br />
* untersucht die übergebenen Daten (von Dispatch() übergeben)<br />
* setzt alle [[#Readings|readings]] via readings*update Funktionen<br />
* gibt den Namen des logischen Device zurück<br />
<br />
Es findet kein Event-Triggering statt, wenn die readings*update Funktionen <br />
* von X_Parse aufgerufen werden und<br />
* X_Parse wiederum von Dispatch() aufgerufen wurde.<br />
(Im Gegensatz zum direkten Aufrufen der readings*update Funktionen ohne vorhergehendes Dispatch() )<br />
<br />
Dispatch() triggert das Event-Handling für das von X_Parse zurückgegebene logische Device.<br />
<br />
=== Kommunikation von den logischen Modulen zum Gerät ===<br />
<br />
Um von einem logischen Modul an ein physisches Gerät zu senden, wird im logischen Modul das Attribut IODev mit dem namen des physischen Devices gesetzt.<br />
Der Befehl<br />
<code>AssignIoPort($hash);</code><br />
in der X_Define-Funktion des logischen Devices erledigt das.<br />
<br />
Als Befehl zum Schreiben vom logischen ins physische Gerät soll <code>IOWrite()</code> verwendet werden. IOWrite() ruft im physischen Gerät die X_Write-Funktion auf.<br />
<br />
Wenn es keine direkte Kommunikation zwischen dem logischen und dem physischen Gerät gibt(keine direkten Aufrufe von Funktionen, kein direktes überprüfen von $hash Werten,...) so können die Module hintereinander geschaltet werden (z.B. für Routerfunktionen wie in RFR) oder mittels FHEM2FHEM:RAW zwei Fhem Installationen verbunden werden und die logischen Devices werden dennoch funktionieren.<br />
<br />
== Ergänzende Hinweise ==<br />
Die Wahl der vorangestellten Nummer für den Dateinamen eines neuen Moduls hat keine Bedeutung mehr, es sei denn die Nummer ist 99. Module, die mit 99_ beginnen, werden von FHEM automatisch geladen. Module mit einer anderen Nummer nur wenn ein <code>define</code>-Befehl dafür sorgt, dass das Modul geladen wird.<br />
<br />
Wenn ein Modul Initialisierungsdaten benötigt, sollten diese im Modul selbst enthalten sein. Eine zusätzliche Datei oder sogar ein Unterverzeichnis mit mehreren Dateien ist bei FHEM nicht üblich und sollte bei Modulen, die mit FHEM ausgeliefert werden nur in Rücksprache mit Rudolf König angelegt werden, da sie sonst bei einem Update nicht verteilt werden.<br />
<br />
== Weitere Informationen ==<br />
Wenn man weitere Details wissen möchte, ist ein erster sinnvoller Schritt ein Blick in die Datei fhem.pl. Dort sieht man im Perl-Code wie die Module aufgerufen werden, was vorher passiert und was danach. Am Anfang der Datei (ca. ab Zeile 130) findet man beispielsweise eine Liste der globalen Variablen, die den Modulen zur Verfügung stehen sowie Details zu den wichtigen Hashes %modules und %defs. Wer mit Perl noch nicht so gut klar kommt, dem hilft eventuell ein Blick auf die Perldoc Website[http://perldoc.perl.org/] oder in das Perl-Buch seiner Wahl. Auch die FHEM Commandref [http://fhem.de/commandref.html] sollte nicht unterschätzt werden. Es stehen oft mehr interessante Details auch für Modulentwickler darin als man zunächst vermuten könnte.<br />
<br />
<br />
== "Hello World" Beispiel ==<br />
<br />
98_Hello.pm<br />
<br />
<pre><br />
package main;<br />
use strict;<br />
use warnings;<br />
<br />
my %Hello_gets = (<br />
"whatyouwant" => "can't",<br />
"whatyouneed" => "try sometimes",<br />
"satisfaction" => "no"<br />
);<br />
<br />
sub Hello_Initialize($) {<br />
my ($hash) = @_;<br />
<br />
$hash->{DefFn} = 'Hello_Define';<br />
$hash->{UndefFn} = 'Hello_Undef';<br />
$hash->{SetFn} = 'Hello_Set';<br />
$hash->{GetFn} = 'Hello_Get';<br />
$hash->{AttrFn} = 'Hello_Attr';<br />
$hash->{ReadFn} = 'Hello_Read';<br />
<br />
$hash->{AttrList} =<br />
"formal:yes,no "<br />
. $readingFnAttributes;<br />
}<br />
<br />
sub Hello_Define($$) {<br />
my ($hash, $def) = @_;<br />
my @param = split('[ \t]+', $def);<br />
<br />
if(int(@param) < 3) {<br />
return "too few parameters: define <name> Hello <greet>";<br />
}<br />
<br />
my $hash->{name} = $param[0];<br />
my $hash->{greet} = $param[2];<br />
<br />
return undef;<br />
}<br />
<br />
sub Hello_Undef($$) {<br />
my ($hash, $arg) = @_; <br />
# nothing to do<br />
return undef;<br />
}<br />
<br />
sub Hello_Get($@) {<br />
my ($hash, @param) = @_;<br />
<br />
return '"get Hello" needs at least one argument' if (int(@param) < 2);<br />
<br />
my $name = shift @param;<br />
my $opt = shift @param;<br />
if(!$Hello_gets{$opt}) {<br />
my @cList = keys %Hello_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
<br />
if($attr{$name}{formal} eq 'yes') {<br />
return $Hello_gets{$opt}.', sir';<br />
}<br />
return $Hello_gets{$opt};<br />
}<br />
<br />
sub Hello_Set($@) {<br />
my ($hash, @param) = @_;<br />
<br />
return '"set Hello" needs at least one argument' if (int(@param) < 2);<br />
<br />
my $name = shift @param;<br />
my $opt = shift @param;<br />
my $value = join("", @param);<br />
<br />
if(!defined($Hello_gets{$opt})) {<br />
my @cList = keys %Hello_gets;<br />
return "Unknown argument $opt, choose one of " . join(" ", @cList);<br />
}<br />
$hash->{STATE} = $Hello_gets{$opt} = $value;<br />
<br />
return "$opt set to $value. Try to get it.";<br />
}<br />
<br />
<br />
sub Hello_Attr(@) {<br />
my ($cmd,$name,$attr_name,$attr_value) = @_;<br />
if($cmd eq "set") {<br />
if($attr_name eq "formal") {<br />
if($attr_value !~ /^yes|no$/) {<br />
my $err = "Invalid argument $attr_value to $attr_name. Must be yes or no.";<br />
Log 3, "Hello: ".$err;<br />
return $err;<br />
}<br />
} else {<br />
return "Unknown attr $attr_name";<br />
}<br />
}<br />
return undef;<br />
}<br />
<br />
1;<br />
<br />
=pod<br />
=begin html<br />
<br />
<a name="Hello"></a><br />
<h3>Hello</h3><br />
<ul><br />
<i>Hello</i> implements the classical "Hello World" as a starting point for module development. <br />
You may want to copy 98_Hello.pm to start implementing a module of your very own. See <br />
<a href="http://www.fhemwiki.de/wiki/DevelopmentModuleIntro">DevelopmentModuleIntro</a> for an <br />
in-depth instruction to your first module.<br />
<br><br><br />
<a name="Hellodefine"></a><br />
<b>Define</b><br />
<ul><br />
<code>define &lt;name&gt; Hello &lt;greet&gt;</code><br />
<br><br><br />
Example: <code>define HELLO Hello TurnUrRadioOn</code><br />
<br><br><br />
The "greet" parameter has no further meaning, it just demonstrates<br />
how to set a so called "Internal" value. See <a href="http://fhem.de/commandref.html#define">commandref#define</a> <br />
for more info about the define command.<br />
</ul><br />
<br><br />
<br />
<a name="Helloset"></a><br />
<b>Set</b><br><br />
<ul><br />
<code>set &lt;name&gt; &lt;option&gt; &lt;value&gt;</code><br />
<br><br><br />
You can <i>set</i> any value to any of the following options. They're just there to <br />
<i>get</i> them. See <a href="http://fhem.de/commandref.html#set">commandref#set</a> <br />
for more info about the set command.<br />
<br><br><br />
Options:<br />
<ul><br />
<li><i>satisfaction</i><br><br />
Defaults to "no"</li><br />
<li><i>whatyouwant</i><br><br />
Defaults to "can't"</li><br />
<li><i>whatyouneed</i><br><br />
Defaults to "try sometimes"</li><br />
</ul><br />
</ul><br />
<br><br />
<br />
<a name="Helloget"></a><br />
<b>Get</b><br><br />
<ul><br />
<code>get &lt;name&gt; &lt;option&gt;</code><br />
<br><br><br />
You can <i>get</i> the value of any of the options described in <br />
<a href="#Helloset">paragraph "Set" above</a>. See <br />
<a href="http://fhem.de/commandref.html#get">commandref#get</a> for more info about <br />
the get command.<br />
</ul><br />
<br><br />
<br />
<a name="Helloattr"></a><br />
<b>Attributes</b><br />
<ul><br />
<code>attr &lt;name&gt; &lt;attribute&gt; &lt;value&gt;</code><br />
<br><br><br />
See <a href="http://fhem.de/commandref.html#attr">commandref#attr</a> for more info about <br />
the attr command.<br />
<br><br><br />
Attributes:<br />
<ul><br />
<li><i>formal</i> no|yes<br><br />
When you set formal to "yes", all output of <i>get</i> will be in a<br />
more formal language. Default is "no".<br />
</li><br />
</ul><br />
</ul><br />
</ul><br />
<br />
=end html<br />
<br />
=cut<br />
</pre><br />
<br />
Der HTML-Code zwischen den Tags <code>=pod</code> und <code>=cut</code> dient zur Generierung der commandref.html. Der HTML-Inhalt wird automatisch beim Verteilen des Moduls im Rahmen des Update-Mechanismus aus jedem Modul extrahiert und daraus die Commandref in verschiedenen Sprachen erstellt. Eine detaillierte Beschreibung wie ein Commandref-Abschnitt in einem Modul definiert wird, siehe: [[Guidelines zur Dokumentation]]<br />
<br />
== Noch zu beschreiben ==<br />
* Zweistufiges Modell für Module<br />
* Funktion X_Ready ...<br />
* Funktion X_State_Fn: {{Link2Forum|Topic=32680}}, siehe auch [[DevelopmentState]]<br />
* FW_summaryFn (wird von FHEMWEB aufgerufen fuer Raum-Uebersicht)<br />
* FW_detailFn (wird von FHEMWEB aufgerufen fuer Detail-Ansicht)<br />
* DevIO<br />
* AsyncOutputFn / asyncOutput<br />
* SetExtensions / SetExtensionsCancel<br />
<br />
[[Kategorie:Development]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=14332HTTPMOD2016-02-22T21:23:39Z<p>StefanStrobel: Anpassungen für die neue Version von HTTPMOD</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
{{Randnotiz|RNText=An extended / modified version of this module that among other features adds XPath and JSON support is currently under development. <br />
If you want to participate in early tests, please follow this {{Link2Forum|Topic=45176|LinkText=discussion thread}} in FHEM forum.<br />
This page has already been updated to describe most of the new features. So if you need one of the new features described here and yout HTTPMOD is still the old version, load the new module <br />
from the forum and help testing<br />
}}<br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
<br />
== Define ==<br />
<source lang="perl"><br />
define <name> HTTPMOD <URL> <Interval><br />
</source><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</source><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
;upgradeAttributes<br />
:convert outdated attributes for older HTTPMOD-Versions that are still defined for this device from the old syntax to the new one.<br />
:atributes with the description "this attribute should not be used anymore" or similar will be translated to the new syntax, e.g. readingsName1 to reading01Name.<br />
;storeKeyValue<br />
:stores a key value pair in an obfuscated form in the file system. Such values can then be used in replacements where the mode is "key" e.g. to avoid storing passwords in the configuration in clear text<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?OExpr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?OMap<br />
:Output Map. Defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*OMap.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue<br />
<br />
== simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<source lang="perl"><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</source><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<source lang="perl"><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<source lang="perl"><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</source><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<source lang="perl"><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</source><br />
<br />
the definition could be:<br />
<source lang="perl"><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</source><br />
<br />
<br />
== formating and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OExpr</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?OMap</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Format</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Decode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?Encode</code><br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups)or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<source lang="perl"><br />
reading01Format %.1f<br />
</source><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings thet are parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<source lang="perl"><br />
reading01-2Format %.1f<br />
</source><br />
<br />
Can be used in cases where a regular expression specified as reading1regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<source lang="perl"><br />
readingFormat %.1f<br />
</source><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingOExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
Example for an expression:<br />
<source lang="perl"><br />
attr PM reading03OExpr $val * 10<br />
</source><br />
Just like in the above example of the readingFormat attributes, readingOExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingOMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
Example for a map:<br />
<source lang="perl"><br />
attr PM reading02-3OMap 0:kalt, 1:warm, 2:sehr warm<br />
</source><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<source lang="perl"><br />
attr PM getDecode UTF-8<br />
</source><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
if there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
Be careful if the text your are getting from your device contains special characters like newline. You don't see such special characters in the fhem webinterface as contents of the internal buf but they might cause your regular expression to fail. <br />
If you have trouble defining a regular expression that nmatches a certain name, then many complicated characters and then a number, it might be helpful to use a negaation in matching like <code>temp[^\d]+([\d\.]).*</code>. In this examle <code>[^\d]+</code> means any character that is not a numerical digit, more than once.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<source lang="perl"><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</source><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<source lang="perl"><br />
reading02Name Temp<br />
</source><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<source lang="perl"><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</source><br />
The same notation can be used for formatting attributes like readingXOMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<source lang="perl"><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</source><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<source lang="perl"><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</source><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<source lang="perl"><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</source><br />
<br />
with JSON you can write <br />
<source lang="perl"><br />
attr test2 get01JSON data_34.4008.value <br />
</source><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<source lang="perl"><br />
attr test2 extractAllJSON<br />
</source><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<source lang="perl"><br />
attr test2 get01ExtractAllJSON<br />
</source><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<source lang="xml"><br />
<root xmlns:foo="http://www.foo.org/" xmlns:bar="http://www.bar.org"><br />
<actors><br />
<actor id="1">Peter X</actor><br />
<actor id="2">Charles Y</actor><br />
<actor id="3">John Doe</actor><br />
</actor><br />
</root><br />
</source><br />
<br />
with XPath you can write <br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</source><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<source lang="perl"><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</source><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 || Peter X<br />
|-<br />
| Actor-2 || Charles Y<br />
|-<br />
| Actor-3 || John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* <code>replacement[0-9]*Regex</code><br />
* <code>replacement[0-9]*Mode</code><br />
* <code>replacement[0-9]*Value</code><br />
* <code>[gs]et[0-9]*Replacement[0-9]*Value</code><br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
'''replacement[0-9]*Mode:'''<br />
The way the replacement value is defined can be specified with the replacement mode.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>reading</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of a ''reading'' of the same device or as ''device:reading'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>internal</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as the name of an ''internal'' of the same device or as ''device:internal'' to refer to another device.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>text</code>, then the corresponding <code>replacement[0-9]*Value</code> is interpreted as a static text<br />
* If the <code>replacement[0-9]*Mode</code> is <code>expression</code>, then the corresponding <code>replacement[0-9]*Value</code> is evaluated as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
* If the <code>replacement[0-9]*Mode</code> is <code>key</code>, then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
<br />
Example:<br />
<br />
<source lang="perl"><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
<br />
attr mydevice get05Name profile<br />
attr mydevice get05URL http://www.mydevice.local/getprofile?password=%%password%%<br />
attr mydevice replacement02Mode key<br />
attr mydevice replacement02Regex %%password%%<br />
attr mydevice get05Replacement02Value password<br />
</source> <br />
<br />
defines that <code>%%value%%</code> will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the <code>%%value%%</code> will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
<br />
A second get will look the same except a different name and replacement value.<br />
<br />
With the command <code>set storeKeyValue password geheim</code> you can store the password geheim in an obfuscated form in the file system. <br />
To use this password and send it in a request you can use the above replacement with mode key. The value password will then refer to the ofuscated string stored with the key password.<br />
<br />
The mode <code>expression</code> allows you to define your own replacement syntax:<br />
<source lang="perl"> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</source> <br />
<br />
In this example any <code><nowiki>{{name}}</nowiki></code> in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between <code><nowiki>{{}}</nowiki></code> as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
<br />
HTTPMOD has two built in replacements: One for session Ids and another one for the input value in a set command.<br />
The placeholder $sid is always replaced with the internal <code>$hash->{sid}</code> which contains the session id after it is extracted from a previous HTTP response. <br />
If you don't like to use the placeholder $sid the you can define your own replacement for example like:<br />
<br />
<source lang="perl"><br />
attr mydevice replacement01Mode internal<br />
attr mydevice replacement01Regex %session%<br />
attr mydevice replacement01Value sid<br />
</source> <br />
<br />
Now the internal <code>$hash->{sid}</code> will be used as a replacement for the placeholder %session%.<br />
<br />
In the same way a value that is passed to a set-command can be put into a request with a user defined replacement. <br />
In this case the internal <code>$hash->{value}</code> will contain the value passed to the set command. <br />
<code>$hash->{value}</code> might even be a string containing several values that could be put into several different positions in a request by using user defined replacements.<br />
<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAge</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode</code><br />
* <code>(reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement</code><br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the <code>MaxAge</code> attributes allow. If readings are outdated, the <code>MaxAgeReplacementMode</code> defines how the affected<br />
reading values should be replaced. <code>MaxAgeReplacementMode</code> can be <code>text</code>, <code>expression</code> or <code>delete</code>.<br />
<br />
<code>MaxAge</code> specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
<code>MaxAgeReplacement</code> contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<source lang="perl"><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</source><br />
The variable <code>$val</code> contains the value of the reading before it became outdated.<br />
<br />
Or to show that a device was offline:<br />
<source lang="perl"><br />
attr MyLight reading01Name color<br />
attr MyLight reading01JSON result_02_color<br />
attr MyLight reading01MaxAge 300<br />
attr MyLight reading01MaxAgeReplacement "offline"<br />
attr MyLight reading01MaxAgeReplacementMode text<br />
</source><br />
<br />
<br />
== Note on determinig how to send requests to a special device ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> command and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<source lang="perl"><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</source><br />
<br />
A user command <br />
<source lang="perl"><br />
set MyDevice Licht 1<br />
</source><br />
<br />
will be translated into the http GET request<br />
<source lang="perl"><br />
http://192.168.1.22/switch=1<br />
</source><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<source lang="perl"><br />
attr MyDevive set01IMap 0:off, 1:on<br />
</source><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<source lang="perl"><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</source><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<source lang="html4strict"><br />
{"set" :{"34.3118.value" :"10" }}<br />
</source><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
The HTTP response to such a request will be ignored unless you specify the attribute <code>setParseResponse</code> <br />
for all set commands or <code>set01ParseResponse</code> for the set command with number 01.<br />
If the HTTP response to a set command is parsed then this is done like the parsing of responses to get commands and you can use the attributes ending e.g. on Format, Encode, Decode, OMap and OExpr to manipulate / format the values read.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<source lang="perl"><br />
attr PM set01TextArg<br />
</source><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<source lang="perl"><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</source><br />
<br />
<br />
== Advanced configuration to create a valid session id that might be necessary ==<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<source lang="perl"><br />
http://User:Password@192.168.1.18/something<br />
</source><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
* <code>sid[0-9]*URL</code><br />
* <code>sid[0-9]*Data.*</code><br />
* <code>sid[0-9]*Header.*</code><br />
* <code>sid[0-9]*IgnoreRedirects</code><br />
* <code>idRegex</code><br />
* <code>idJSON</code><br />
* <code>idXPath</code><br />
* <code>idXPath-Strict</code><br />
* <code>(get|set|sid)[0-9]*IdRegex</code><br />
* <code>(get|set|sid)[0-9]*IdJSON</code><br />
* <code>(get|set|sid)[0-9]*IdXPath</code><br />
* <code>(get|set|sid)[0-9]*IdXPath-Strict</code><br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to send a request without <br />
doing the login procedure. If the result contains an error that authentication is necessary, then a login is performed. <br />
To detect such an error in the HTTP response, you can again use a regular expression, JSON or XPath, this time with the attributes <br />
<br />
* <code>reAuthRegex</code><br />
* <code>reAuthJSON</code><br />
* <code>reAuthXPath</code><br />
* <code>reAuthXPath-Strict</code><br />
* <code>(get|set)[0-9]*ReAuthRegex</code><br />
* <code>(get|set)[0-9]*ReAuthJSON</code><br />
* <code>(get|set)[0-9]*ReAuthXPath</code><br />
* <code>(get|set)[0-9]*ReAuthXPath-Strict</code><br />
<br />
reAuthJSON or reAuthXPath typically only extract one piece of data from a response. <br />
If the existance of the specified piece of data is sufficent to start a login procedure, then nothing more needs to be defined to detect this situation. <br />
If however the indicator is a status code that contains different values depending on a successful request and a failed request if a new authentication is needed, <br />
then you can combine things like reAuthJSON with reAuthRegex. In this case the regex is only matched to the data extracted by JSON (or XPath). <br />
This way you can easily extract the status code using JSON parsing and then specify the code that means "authentication needed" as a regular expression.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribute (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<br />
<source lang="perl"><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</source><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
In the special case where a session id is set as a HTTP-Cookie (with the header Set-cookie: in the HTTP response) HTTPMOD offers an even simpler way. With the attribute enableCookies a very basic cookie handling mechanism is activated that stores all cookies that the server sends to the HTTPMOD device and puts them back as cookie headers in the following requests.<br />
<br />
For such cases no sidIdRegex and no $sid in a user defined header is necessary.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
When a get option is defined by attributes, the module allows querying additional values from the device that require individual HTTP-Requests or special parameters to be sent<br />
<br />
Extension to the above example:<br />
<br />
<source lang="perl"><br />
attr PM get01Name MyGetValue <br><br />
attr PM get01URL http://MyPoolManager/cgi-bin/directory/webgui.fcgi?special=1?sid=$sid <br><br />
attr PM getHeader1 Content-Type: application/json <br><br />
attr PM get01Data {"get" :{"30.1234.value"}} <br><br />
</source><br />
<br />
This example defines a get option with the name MyGetValue.<br />
By issuing <code>get PM MyGetValue</code> in FHEM, the defined HTTP request is sent to the device.<br />
The HTTP response is then parsed using the same readingXXName and readingXXRegex attributes as above so<br />
additional pairs will probably be needed there for additional values.<br />
<br />
if you prefer to define the parsing and formatting of readings individually per get command, you can use <br />
attributes like get01Regex, get01XPath, get01Format, get01OMap and so on just like for reading01...<br />
<br />
You can also include parameters / values that are passed to the get command in the request just like for set commands.<br />
The placeholder $val will be replaced with the value given to the get command or you can specify your own replacement as described above.<br />
<br />
If the new get parameter should also be queried regularly, you can define the following optional attributes:<br />
<br />
<source lang="perl"><br />
attr PM get01Poll 1<br />
attr PM get01PollDelay 300<br />
</source><br />
<br />
The first attribute includes this reading in the automatic update cycle and the second defines an<br />
alternative lower update frequency. When the interval defined initially in the define is over and the normal readings<br />
are read from the device, the update function will check for additional get parameters that should be included<br />
in the update cycle.<br />
<br />
If a PollDelay is specified for a get parameter, the update function also checks if the time passed since it has last read this value <br />
is more than the given PollDelay. If not, this reading is skipped and it will be rechecked in the next cycle when <br />
interval is over again. So the effective PollDelay will always be a multiple of the interval specified in the initial define.<br />
<br />
Please note that each defined get command that is included in the regular update cycle will create its own HTTP request. So if you want to extract several values from the same request, it is much more efficient to do this by defining readingXXName and readingXXRegex, XPath or JSON attributes and to specify an interval and a URL in the define of the HTTPMOD device. <br />
<br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<source lang="perl"><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8OMap Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</source><br />
<br />
<br />
== All attributes ==<br />
<br />
;reading[0-9]+Name<br />
:the name of a reading to extract with the corresponding readingRegex, readingJSON, readingXPath or readingXPath-Strict<br />
:Please note that the old syntax <b>readingsName.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|set)[0-9]+Name<br />
:Name of a get or set command<br />
:If the HTTP response that is received after the command is parsed with an individual parse option then this name is also used as a reading name. Please note that no individual parsing needs to be defined for a get or set. If no regex, XPath or JSON is specified for the command, then HTTPMOD will try to parse the response using all the defined readingRegex, reading XPath or readingJSON attributes.<br />
<br />
;(get|set|reading)[0-9]+Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the (get|set|reading)[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading or get or set but without the numbers in the middle, e.g. as getRegex or readingRegex, then it applies to all the other readings / get / set commands where no specific Regex is defined.<br><br />
:The value to extract should be in a capture group / sub expression e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
:Using this attribute for a set command (setXXRegex) only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
:Please note that the old syntax <code>readingsRegex.*</code> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
:If for get or set commands neither a generic Regex attribute without numbers nor a specific (get|set)[0-9]+Regex attribute is specified and also no XPath or JSON parsing specification is given for the get or set command, then HTTPMOD tries to use the parsing definitions for general readings defined in reading[0-9]+Name, reading[0-9]+Regex or XPath or JSON attributes and assigns the Readings that match here.<br />
<br />
;(get|set|reading)[0-9]+RegOpt<br />
:Lets the user specify regular expression modifiers. For example if the same regular expression should be matched as often as possible in the HTTP response, then you can specify RegOpt g which will case the matching to be done as /regex/g<br />
:The results will be trated the same way as multiple capture groups so the reading name will be extended with -number. <br />
:For other possible regular expression modifiers see http://perldoc.perl.org/perlre.html#Modifiers<br />
<br />
;(get|set|reading)[0-9]+XPath<br />
:defines an xpath to one or more values when parsing HTML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get|set|reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more values when parsing XML data (see examples above)<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]+AutoNumLen<br />
:In cases where a regular expression or an XPath results in multiple results and these results are stored in a common reading name with extension -number, then you can modify the format of this number to have a fixed length with leading zeros. AutoNumLen 3 for example will lead to reading names ending with -001 -002 and so on.<br />
<br />
;get|set|reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names. See the above example.<br />
:If you don't know the paths, then start by using extractAllJSON and the use the names of the readings as values for the JSON attribute.<br><br />
:Please don't forget to also specify a name for a reading, get or set. <br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches.<br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. The list of matches will be in the variable @matchlist.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed. <br><br />
:If this attribute is set to 1, then additionally to the matching of get specific regexe (get[0-9]*Regex), XPath or JSON also all the reading names and parse definitions defined in Reading[0-9]+Name and Reading[0-9]+Regex, XPath or JSON attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
:This is automatically done if a get or set command is defined without its own parse attributes.<br />
<br />
;(get|reading)[0-9]*OExpr<br />
:defines an optional expression that is used in an eval to compute / format a readings value after parsing an HTTP response<br />
:The raw value from the parsing will be in the variable $val.<br />
:If specified as readingOExpr then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Expr.<br />
:Please note that the old syntax <b>readingsExpr.*</b> does not work with all features of HTTPMOD and should be avoided. It might go away in a future version of HTTPMOD.<br />
<br />
;(get|reading)[0-9]*Expr<br />
:This is the old syntax for (get|reading)[0-9]*OExpr. It should be replaced by (get|reading)[0-9]*OExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|reading)[0-9]*OMap<br />
:Map that defines a mapping from raw value parsed to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
:If specified as readingOMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br><br />
:The individual options in a map are separated by a komma and an optional space. Spaces are allowed to appear in a visible value however kommas are not possible.<br />
<br />
;(get|reading)[0-9]*Map<br />
:This is the old syntax for (get|reading)[0-9]*OMap. It should be replaced by (get|reading)[0-9]*OMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;(get|set|reading)[0-9]*Format<br />
:Defines a format string that will be used in sprintf to format a reading value.<br />
:If specified without the numbers in the middle e.g. as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br><br />
<br />
;(get|set|reading)[0-9]*Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
;(get|set|reading)[0-9]*Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. <br />
:This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
:If your reading values contain Umlauts and they are shown as strange looking icons then you probably need to use this feature.<br />
:Using this attribute for a set command only makes sense if you want to parse the HTTP response to the HTTP request that the set command sent by defining the attribute setXXParseResponse.<br />
<br />
<br />
;(get|set)[0-9]*URL<br />
:URL to be requested for the set or get command. If this option is missing, the URL specified during define will be used.<br />
<br />
;(get|set)[0-9]*Data<br />
:Data to be sent to the device as POST data when the get or set command is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;(get|set)[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get or set commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;(get|set)[0-9]*Header.*<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;requestHeader.*<br />
:Define an optional additional HTTP Header to set in the HTTP request of the main loop<br />
<br />
;requestData<br />
:optional POST Data to be sent in the request of the main loop. If not defined, it will be an HTTP GET request as defined in HttpUtils which is used by this module<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
<br />
;(get|set)[0-9]*TextArg<br />
:For a get command this defines that the command accepts a text value after the option name. By default a get command doesn't accept optional values after the command name. <br />
:If TextArg is specified and a value is passed after the get name then this value can then be used in a request URL, header or data as replacement for $val or in a user defined replacement that uses the internal "value" ($hash->{value}).<br />
:If used for a set command then it defines that the value to be set doesn't require any validation / conversion. <br />
:The raw value is passed on as text to the device. By default a set command expects a numerical value or a text value that is converted to a numeric value using a map.<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+IExpr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Expr<br />
:This is the old syntax for (get|reading)[0-9]*IExpr. It should be replaced by (get|reading)[0-9]*IExpr. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+IMap<br />
:Map that defines a mapping from raw to input values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the input values.<br />
<br />
;set[0-9]+Map<br />
:This is the old syntax for (get|reading)[0-9]*IMap. It should be replaced by (get|reading)[0-9]*IMap. The set command upgradeAttributes which becomes visible when the attribute enableControlSet is set to 1, can do this renaming automatically.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*NoArg<br />
:Defines that this set option doesn't require arguments. It allows sets like "on" or "off" without further values.<br />
<br />
;set[0-9]*ParseResponse<br />
:defines that the HTTP response to the set will be parsed as if it was the response to a get command.<br />
<br />
<br />
;(get|set)[0-9]*URLExpr<br />
:Defines a Perl expression to specify the HTTP Headers for this request. This overwrites any other header specification and should be used carefully only if needed. The original Header is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*DatExpr<br />
:Defines a Perl expression to specify the HTTP Post data for this request. This overwrites any other post data specification and should be used carefully only if needed. The original Data is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
;(get|set)[0-9]*HdrExpr<br />
:Defines a Perl expression to specify the URL for this request. This overwrites any other URL specification and should be used carefully only if needed. The original URL is availabe as $old. Typically this feature is not needed and it might go away in future versions of HTTPMOD. Please use the "replacement" attributes if you want to pass additional variable data to a web service. <br />
<br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the (get|set)[0-9]*ReAuthRegex attributes.<br />
<br />
;(get|set)[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired during a set operation and a new login needs to be performed.<br />
:It works like the global reAuthRegex but is used for set operations.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
;clearSIdBeforeAuth<br />
:will set the session id to "" before doing the authentication steps<br />
<br />
;authRetries<br />
:number of retries for authentication procedure - defaults to 1<br />
<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, internal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is <code>text</code>, then value is a static text that is used as the replacement.<br />
:If replacementMode is <code>reading</code> then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br />
:If replacementMode is <code>internal</code> the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br />
:If replacementMode is <code>expression</code> the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
:If replacementMode is <code>key</code> then the module will use a value from a key / value pair that is stored in an obfuscated form in the file system with the set storeKeyValue command. This might be useful for storing passwords.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text, expression and delete.<br />
<br />
<br />
;httpVersion<br />
:defines the HTTP-Version to be sent to the server. This defaults to 1.0.<br />
<br />
;sslVersion<br />
:defines the SSL Version for the negotiation with the server. The attribute is evaluated by HttpUtils. If it is not specified, HttpUtils assumes SSLv23:!SSLv3:!SSLv2<br />
<br />
;sslArgs<br />
:defines a list that is converted to a key / value hash and gets passed to HttpUtils. To avoid certificate validation for broken servers you can for example specify <br />
<code>attr myDevice sslArgs SSL_verify_mode,SSL_VERIFY_NONE</code><br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread, upgradeAttributes, storeKeyValue.<br />
<br />
;enableCookies<br />
:enables the built cookie handling if set to 1. With cookie handling each HTTPMOD device will remember cookies that the server sets and send them back to the server in the following requests. <br />
:This simplifies session magamenet in cases where the server uses a session ID in a cookie. In such cases enabling Cookies should be sufficient and no sidRegex and no manual definition of a Cookie Header should be necessary.<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS that contains the names of all readings that could be matched in the last request.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR that contains the error message of the last error returned from HttpUtils. <br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* {{Link2Forum|Topic=29471|LinkText=Thread}} in Fhem Forum that discusses the second major version of this module <br />
* {{Link2Forum|Topic=45176|LinkText=Thread}} in Fhem Forum that discusses the third major version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=13393HTTPMOD2015-12-30T10:25:26Z<p>StefanStrobel: added more examples for defining set options</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
{{Randnotiz|RNText=An extended / modified version of this module that among other features adds XPath and JSON support is currently under development. <br />
If you want to participate in early tests, please follow this {{Link2Forum|Topic=45176|LinkText=discussion thread}} in FHEM forum.<br />
This page has already been updated to describe most of the new features. So if you need one of the new features described here and yout HTTPMOD is still the old version, load the new module <br />
from the forum and help testing<br />
}}<br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
<br />
== Define ==<br />
<pre><br />
define <name> HTTPMOD <URL> <Interval><br />
</pre><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<pre><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</pre><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?Expr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?Map<br />
:defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a sub expression e.g. ([\d\.]+) in the above example<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread<br />
<br />
;(get|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches. <br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. <br />
:The list of matches will be in the variable @matchlist.<br />
<br />
<br />
== simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<pre><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</pre><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<pre><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</pre><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<pre><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</pre><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<pre><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</pre><br />
<br />
the definition could be:<br />
<pre><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</pre><br />
<br />
<br />
== formating and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* (reading|get)[0-9]*(-[0-9]+)?Expr<br />
* (reading|get)[0-9]*(-[0-9]+)?Map<br />
* (reading|get)[0-9]*(-[0-9]+)?Format<br />
* (reading|get)[0-9]*(-[0-9]+)?Decode<br />
* (reading|get)[0-9]*(-[0-9]+)?Encode<br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups)or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<pre><br />
reading01Format %.1f<br />
</pre><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings thet are parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<pre><br />
reading01-2Format %.1f<br />
</pre><br />
<br />
Can be used in cases where a regular expression specified as reading1regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<pre><br />
readingFormat %.1f<br />
</pre><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
=== Example: ===<br />
<pre><br />
attr PM reading03Expr $val * 10<br />
</pre><br />
Just like in the above example of the readingFormat attributes, readingExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
=== Example: ===<br />
<pre><br />
attr PM reading02-3Map 0:kalt, 1:warm, 2:sehr warm<br />
</pre><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<pre><br />
attr PM getDecode UTF-8<br />
</pre><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
if there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<pre><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</pre><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<pre><br />
reading02Name Temp<br />
</pre><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<pre><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</pre><br />
The same notation can be used for formatting attributes like readingXMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<pre><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</pre><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<pre><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</pre><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<pre><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</pre><br />
<br />
with JSON you can write <br />
<pre><br />
attr test2 get01JSON data_34.4008.value <br />
</pre><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<pre><br />
attr test2 extractAllJSON<br />
</pre><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<pre><br />
attr test2 get01ExtractAllJSON<br />
</pre><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<pre><br />
&lt;root xmlns:foo=&quot;http://www.foo.org/&quot; xmlns:bar=&quot;http://www.bar.org&quot;&gt;<br />
&lt;actors&gt;<br />
&lt;actor id=&quot;1&quot;&gt;Peter X&lt;/actor&gt;<br />
&lt;actor id=&quot;2&quot;&gt;Charles Y&lt;/actor&gt;<br />
&lt;actor id=&quot;3&quot;&gt;John Doe&lt;/actor&gt;<br />
&lt;/actors&gt;<br />
&lt;/root&gt;<br />
</pre><br />
<br />
with XPath you can write <br />
<pre><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</pre><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<pre><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</pre><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 | Peter X<br />
|-<br />
| Actor-2 | Charles Y<br />
|-<br />
| Actor-3 | John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* replacement[0-9]*Regex<br />
* replacement[0-9]*Mode<br />
* replacement[0-9]*Value<br />
* [gs]et[0-9]*Replacement[0-9]*Value<br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
The way the replacement value is defined can be specified with the replacement mode. If the mode is reading, then the value is interpreted as the name of a reading of the same device or as device:reading to refer to another device.<br />
If the mode is internal, then the value is interpreted as the name of an internal of the same device or as device:internal to refer to another device.<br />
The mode text will use the value as a static text and the mode expression will evaluate the value as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
<br />
Example:<br />
<br />
<pre><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
</pre> <br />
<br />
defines that %%value%% will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the %%value%% will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
A second get will look the same except a different name and replacement value.<br />
<br />
The mode expression allows you to define your own replacement syntax:<br />
<pre> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</pre> <br />
<br />
In this example any {{name}} in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between {{}} as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAge<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement<br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the MaxAge attributes allow. If readings are outdated, the MaxAgeReplacementMode defines how the affected<br />
reading values should be replaced. MaxAgeReplacementMode can be <code>text</code> or <code>expression</code>.<br />
<br />
MaxAge specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
MaxAgeReplacement contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<pre><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</pre> <br />
The variable $val contains the value of the reading before it became outdated.<br />
<br />
<br />
== Additional notes ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and integrate it into an HTTP-Request that sends the value to the device. The definitions for URL, headers and post data can contain the placeholder $val which will be replaced by the value given to the set command.<br />
<br />
This can be as simple as:<br />
<pre><br />
# No cyclic requests and no main URL needed in this example<br />
define MyDevice none 0<br />
<br />
attr MyDevice set01Name Licht<br />
attr MyDevice set01URL http://192.168.1.22/switch=$val<br />
</pre><br />
<br />
A user command <br />
<pre><br />
set MyDevice Licht 1<br />
</pre><br />
<br />
will be translated into the http GET request<br />
<pre><br />
http://192.168.1.22/switch=1<br />
</pre><br />
<br />
In this example a map would also be helpful, that translates on / off to 0 or 1 and allows the user to select on/of in fhemweb:<br />
<pre><br />
attr MyDevive set01Map 0:off, 1:on<br />
</pre><br />
This also provides input validation to make sure that only on and off can be used with the set command.<br />
<br />
In more complex Scenarios you might need to login before sending a command and the Login might create a session id that has to be part of further requests either in the URL, in headers or in the post data.<br />
<br />
Extension to the above example for a PoolManager 5 where a set needs a session id in the URL and the values have to be passed in JSON strings as post data:<br />
<pre><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</pre><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<pre><br />
{"set" :{"34.3118.value" :"10" }}<br />
</pre><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<pre><br />
attr PM set01TextArg<br />
</pre><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<pre><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</pre><br />
<br />
<br />
=== Advanced configuration to create a valid session id that might be necessary in set options ===<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<pre><br />
http://User:Password@192.168.1.18/something<br />
</pre><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
;sid[0-9]*URL<br />
;sid[0-9]*IDRegex<br />
;sid[0-9]*Data.*<br />
;sid[0-9]*Header.*<br />
;sid[0-9]*IgnoreRedirects<br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to do a set without <br />
doing the login procedure. If the Attribute ReAuthRegex is defined, it will then compare the HTTP Response to the set request with the regular expression from ReAuthRegex. If it matches, then a <br />
login is performed. The ReAuthRegex is meant to match the error page a device returns if authentication or reauthentication is required e.g. because a session timeout has expired.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribue (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<br />
<pre><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</pre><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<pre><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8Map Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</pre><br />
<br />
<br />
=== Advanced attributes ===<br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the [gs]et[0-9]*ReAuthRegex attributes.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
<br />
;set[0-9]+Name<br />
:Name of a set option<br />
<br />
;set[0-9]*URL<br />
:URL to be requested for the set option<br />
<br />
;set[0-9]*Data<br />
:Data to be sent to the device as POST data when the set is executed<br />
<br />
;set[0-9]*Header<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+Expr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Map<br />
:Map that defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the visible values or select a value with a slider.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired an a new login needs to be performed. <br />
<br />
;get[0-9]+Name<br />
:Name of a get option and Reading to be retrieved / extracted<br />
<br />
;get[0-9]*URL<br />
:URL to be requested for the get option. If this option is missing, the URL specified during define will be used.<br />
<br />
;get[0-9]*Data<br />
:optional data to be sent to the device as POST data when the get is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;get[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;get[0-9]*Header<br />
:optional HTTP Headers to be sent to the device when the get is executed<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;get[0-9]*Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the get[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading but as getRegex, then it applies to all get options where no specific Regex is defined.<br />
:If neither a generic getRegex attribute nor a specific get[0-9]+Regex attribute is specified, then HTTPMOD tries all Regex / Reading pairs defined in Reading[0-9]+Name and Reading[0-9]+Regex attributes and assigns the Readings that match.<br />
<br />
;get[0-9]*Expr<br />
:this attribute behaves just like Reading[0-9]*Expr but is applied to a get value. <br />
<br />
;get[0-9]*Map<br />
:this attribute behaves just like Reading[0-9]*Map but is applied to a get value.<br />
<br />
;get[0-9]*Format<br />
:this attribute behaves just like Reading[0-9]*Format but is applied to a get value.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed.<br />
:If this attribute is set to 1, then additionally to any matching of get specific regexes (get[0-9]*Regex), also all the Regex / Reading pairs defined in Reading[0-9]+Name and Reading[0-9]+Regex attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, interbal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is text, then value is a static text that is used as the replacement.<br><br />
:If replacementMode is reading then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br><br />
:If replacementMode is internal the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br><br />
:If replacementMode is expression the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text and expression.<br />
<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS <br />
:that contains the names of all readings that could be matched in the last request.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR <br />
:that contains the error message of the last error returned from HttpUtils. <br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=HTTPMOD&diff=13300HTTPMOD2015-12-21T20:25:57Z<p>StefanStrobel: updated to describe the new HTTPMOD features like XPath, JSON, Aging, Replacements and more.</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information from devices with an HTTP interface (or, more generic, from any URL) or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=HTTPMOD<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_HTTPMOD.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
HTTPMOD provides a generic way to retrieve information from devices with an HTTP Interface and store them in Readings or send information to such devices. It queries a given URL with Headers and data defined by attributes. <br />
<br />
From the HTTP response it extracts readings named in attributes using Regexes, JSON or XPath parsing also defined by attributes.<br />
<br />
In an advanced [[Konfiguration|configuration]] the module can also send information to devices. To do this, a generic <code>set</code> option can be configured using attributes. <br />
<br />
== Availability == <br />
{{Randnotiz|RNText=An extended / modified version of this module that among other features adds XPath and JSON support is currently under development. <br />
If you want to participate in early tests, please follow this {{Link2Forum|Topic=45176|LinkText=discussion thread}} in FHEM forum.<br />
This page has already been updated to describe most of the new features. So if you need one of the new features described here and yout HTTPMOD is still the old version, load the new module <br />
from the forum and help testing<br />
}}<br />
The module is part of the regular FHEM distribution.<br />
<br />
== Prerequisites ==<br />
This module uses the non blocking HTTP function <code>HttpUtils_NonblockingGet</code> provided by FHEM's [[HttpUtils]] in a new version published in December 2013.<br />
If not already installed in your environment, please [[update]] FHEM or install it manually using appropriate commands from your environment.<br />
<br />
== Define ==<br />
<pre><br />
define <name> HTTPMOD <URL> <Interval><br />
</pre><br />
The module connects to the given <code>URL</code> every <code>Interval</code> seconds, sends optional headers and data and then parses the response with regular expressions, xpath or json to set readings.<br />
<br />
URL can be "none" and Interval can be 0 if you prefer to only query data with a get command and not in a defined interval.<br />
<br />
Example:<br />
<pre><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
</pre><br />
<br />
== Set-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
If you set the attribute enableControlSet to 1, the following additional built in set commands are available:<br />
;interval<br />
:set new interval time in seconds and restart the timer<br />
;reread<br />
:request the defined URL and try to parse it just like the automatic update would do it every Interval seconds without modifying the running timer.<br />
;stop<br />
:stop interval timer.<br />
;start<br />
:restart interval timer to call GetUpdate after interval seconds<br />
<br />
== Get-Commands ==<br />
can be defined using attributes, see advanced configuration<br />
<br />
== simple Attributes ==<br />
;do_not_notify<br />
<br />
;readingFnAttributes<br />
<br />
;requestHeader.* <br />
:Define an additional HTTP Header to set in the HTTP request<br />
<br />
;requestData<br />
:POST Data to be sent in the request. If not defined, it will be a GET request as defined in HttpUtils used by this module<br />
<br />
;reading[0-9]+(-[0-9]+)?Name<br />
:the name of a reading to extract with the corresponding readingRegex<br />
<br />
;reading[0-9]*(-[0-9]+)?Expr<br />
:defines an expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
;reading[0-9]*(-[0-9]+)?Map<br />
:defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". If specified as readingMap then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Map.<br />
<br />
;reading[0-9]*(-[0-9]+)?Format<br />
:Defines a format string that will be used in sprintf to format a reading value. If specified as readingFormat then the attribute value is a default for all other readings that don't specify an explicit reading[0-9]*Format.<br />
<br />
;reading[0-9]*(-[0-9]+)?Decode<br />
:defines an encoding to be used in a call to the perl function decode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 instead of utf8.<br />
<br />
;reading[0-9]*(-[0-9]+)?Encode<br />
:defines an encoding to be used in a call to the perl function encode to convert the raw data string read from the device to a reading. This can be used if the device delivers strings in an encoding like cp850 and after decoding it you want to reencode it to e.g. utf8.<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a sub expression e.g. ([\d\.]+) in the above example<br />
<br />
;reading[0-9]+Regex<br />
:defines the regex to be used for extracting the reading. The value to extract should be in a capture group / sub expression <br />
:e.g. ([\d\.]+) in the above example. Multiple capture groups will create multiple readings (see explanation above)<br />
<br />
;reading[0-9]+XPath<br />
:defines an xpath to one or more readings when parsing HTML data (see examples below)<br />
<br />
;reading[0-9]+XPath-Strict<br />
:defines an xpath to one or more readings when parsing XML data (see examples below)<br />
<br />
;reading[0-9]+JSON<br />
:defines a path to the JSON object wanted by concatenating the object names with an underscore as delimiter (see the example below)<br />
<br />
<br />
;noShutdown<br />
:pass the noshutdown flag to HTTPUtils for webservers that need it (some embedded webservers only deliver empty pages otherwise)<br />
<br />
;disable<br />
:stop doing automatic HTTP requests while this attribute is set to 1<br />
<br />
;timeout<br />
:time in seconds to wait for an answer. Default value is 2<br />
<br />
;enableControlSet<br />
:enables the built in set commands interval, stop, start, reread<br />
<br />
;(get|reading)[0-9]*RecombineExpr<br />
:defines an expression that is used in an eval to compute one reading value out of the list of matches. <br />
:This is supposed to be used for regexes or xpath specifications that produce multiple results if only one result that combines them is wanted. <br />
:The list of matches will be in the variable @matchlist.<br />
<br />
<br />
== simple Configuration of HTTP Devices ==<br />
If your device expects special HTTP-headers then specify them as <code>attr requestHeader1</code> to <code>attr requestHeaderX</code>.<br />
If your Device expects an HTTP POST instead of HTTP GET then the POST-data can be specified as <code>attr requestData</code>.<br />
To get the readings, specify pairs of <code>attr readingXName</code> and <code>attr readingXRegex</code>, <code>attr readingXXPath</code>, <code>attr readingXXPath-Strict</code> or <code>attr readingXJSON</code> to define which readings you want to extract from the HTTP response and how to extract them. (The old syntax <code>attr readingsNameX</code> and <code>attr readingsRegexX</code> is still supported but the new one with <code>attr readingXName</code> and <code>attr readingXRegex</code> should be preferred. The actual values to be extracted have to be sub expressions within () in the regex (see example below)<br />
<br />
=== Example for a PoolManager 5: ===<br />
The PoolManager Web GUI can be queried with HTTP POST Requests like this one:<br />
<br />
<pre><br />
POST /cgi-bin/webgui.fcgi HTTP/1.1<br />
Host: 192.168.70.90<br />
Accept: */*<br />
Content-Type: application/json;charset=UTF-8<br />
Content-Length: 60<br />
<br />
{"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value"]}<br />
</pre><br />
<br />
The resulting HTTP Response would look like this:<br />
<br />
<pre><br />
HTTP/1.1 200 OK<br />
Content-type: application/json; charset=UTF-8<br />
Expires: 0<br />
Cache-Control: no-cache<br />
Date: Sun, 12 Jan 2014 12:23:11 GMT<br />
Server: lighttpd/1.4.26<br />
Content-Length: 179<br />
<br />
{<br />
"data": {<br />
"34.4001.value": "7.00",<br />
"34.4008.value": "0.52",<br />
"34.4033.value": "24.8"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</pre><br />
<br />
To configure HTTPMOD for a PoolManager one would first define a PoolManager device with e.g. the name PM, the URL and an interval of e.g. 60 seconds. <br />
<br />
Then the data to be sent in the request needs to be defined because in this example the device expects a POST request so the query is not contained in the URL but in the request data.<br />
<br />
Also as seen above the device expects special HTTP headers in the request so these headers also need to be defined as <code>attr PM requestHeader1</code> and <code>attr PM requestHeader2</code><br />
<br />
Then the names of the readings to be extracted would be set with attributes<br />
<br />
Then for each reading value to be extracted a regular expression needs to be set that will match the value in question within ().<br />
<br />
Example:<br />
<pre><br />
define PM HTTPMOD http://MyPoolManager/cgi-bin/webgui.fcgi 60<br />
attr PM reading01Name PH<br />
attr PM reading01Regex 34.4001.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading02Name CL<br />
attr PM reading02Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM reading03Name3TEMP<br />
attr PM reading03Regex 34.4033.value":[ \t]+"([\d\.]+)"<br />
<br />
attr PM requestData {"get" :["34.4001.value" ,"34.4008.value" ,"34.4033.value", "14.16601.value", "14.16602.value"]}<br />
attr PM requestHeader1 Content-Type: application/json<br />
attr PM requestHeader2 Accept: */*<br />
attr PM stateFormat {sprintf("%.1f Grad, PH %.1f, %.1f mg/l Chlor", ReadingsVal($name,"TEMP",0), ReadingsVal($name,"PH",0), ReadingsVal($name,"CL",0))}<br />
</pre><br />
<br />
<br />
=== Example for AmbientMonitor ===<br />
AmbientMonitor is a webbased visualisation for sensors connected to an Arduino device. Its web interface can also be queried with HTTMOD to grab the data into readings.<br />
<br />
This example was provided by locutus. The hardware configuration is an Arduino + Ethercard with ENC28J60 Controller + DHT22 Sensor and software can be downloaded from https://github.com/lucadentella/AmbientMonitor<br />
<br />
In this example an HTTP GET is sufficent, so no <code>requestData</code> is needed. The device provides temperature and humidity readings in an HTTP response that looks like:<br />
<pre><br />
HTTP/1.0 200 OK <br />
Content-Type: text/html <br />
<br />
myCB({'temperature':22.00,'humidity':46.00})<br />
</pre><br />
<br />
the definition could be:<br />
<pre><br />
define AmbientMonitor HTTPMOD http://192.168.1.221/?callback=? 300<br />
attr AmbientMonitor requestHeader Content-Type: application/json<br />
attr AmbientMonitor reading1Name Temperatur<br />
attr AmbientMonitor reading1Regex temperature':([\d\.]+)<br />
attr AmbientMonitor reading2Name Feuchtigkeit<br />
attr AmbientMonitor reading2Regex humidity':([\d\.]+)<br />
attr AmbientMonitor stateFormat {sprintf("Temperatur %.1f C, Feuchtigkeit %.1f %", ReadingsVal($name,"Temperatur",0), ReadingsVal($name,"Feuchtigkeit",0))}<br />
</pre><br />
<br />
<br />
== formating and manipulating values / readings ==<br />
Values that are parsed from an HTTP response can be further treated or formatted with the following attributes:<br />
<br />
* (reading|get)[0-9]*(-[0-9]+)?Expr<br />
* (reading|get)[0-9]*(-[0-9]+)?Map<br />
* (reading|get)[0-9]*(-[0-9]+)?Format<br />
* (reading|get)[0-9]*(-[0-9]+)?Decode<br />
* (reading|get)[0-9]*(-[0-9]+)?Encode<br />
<br />
They can all be specified for an individual reading, for all readings in one match (e.g. if a regular expression has several capture groups)or for all readings in a get command (defined by getXX) or for all readings in the main reading list (defined by readingXX):<br />
<pre><br />
reading01Format %.1f<br />
</pre><br />
<br />
will format the reading with the name specified by the attribute reading01Name to be numerical with one digit after the decimal point.<br />
If the attribute reading01Regex is used and contains several capture groups then the format will be applied to all readings thet are parsed by this regex unless these readings have their own format specified by reading01-1Format, reading01-2Format and so on.<br />
<br />
<pre><br />
reading01-2Format %.1f<br />
</pre><br />
<br />
Can be used in cases where a regular expression specified as reading1regex contains several capture groups or an xpath specified as reading01XPath creates several readings. <br />
In this case reading01-2Format specifies the format to be applied to the second match.<br />
<br />
<pre><br />
readingFormat %.1f<br />
</pre><br />
<br />
applies to all readings defined by a reading-Attribute that have no more specific format.<br />
<br />
If you need to do some calculation on a raw value before it is used as a reading, you can define the attribute <code>readingExpr</code>.<br />
It defines a Perl expression that is used in an eval to compute the readings value. The raw value will be in the variable $val.<br />
<br />
=== Example: ===<br />
<pre><br />
attr PM reading03Expr $val * 10<br />
</pre><br />
Just like in the above example of the readingFormat attributes, readingExpr and the other following attributes can be applied on several levels.<br />
<br />
To map a numerical value to a name, you can use the readingMap attribute. <br />
It defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb".<br />
<br />
=== Example: ===<br />
<pre><br />
attr PM reading02-3Map 0:kalt, 1:warm, 2:sehr warm<br />
</pre><br />
<br />
To convert character sets, the module can first decode a string read from the device and then encode it again. For example:<br />
<pre><br />
attr PM getDecode UTF-8<br />
</pre><br />
<br />
This applies to all readings defined for Get-Commands.<br />
<br />
<br />
== Some help with Regular Expressions ==<br />
If HTTPMOD seems not to work and the FHEM Logfile contains a message like <br />
:<code>HTTPMOD: Response didn't match Reading ...</code><br />
then you should check if the value you want to extract is read into the internal with the name buf. Internals are visible when you click on the defined HTTPMOD Device. buf is an internal variable that contains the HTTP Response read. If the value is there and you get the mentioned message then probably something is wrong with your regular expression. If you are new to regular expressions then the introduction at http://perldoc.perl.org/perlretut.html might be helpful. <br />
<br />
For a typical HTTPMOD use case where you want to extract a number out of a HTTP-Response you can use something like <code>[\d\.]+</code> to match the number itself. The expression matches the number characters (<code>\d</code>) or a <code>.</code> if one of these characters occurs at least once. <br />
<br />
To tell HTTPMOD that the number is what you want to use for the reading, you have to put the expression in between <code>()</code>. A <code>([\d\.]+)</code> alone would match the longest number in the HTTP Response which is very likely not the number you are looking for so you need to add something to the expression to give it a context and define how to find the number that you are looking for.<br />
if there is a title text before the number or a special text after the number you can put this in the regex. In one of the examples above <code>humidity':([\d\.]+)</code> is looking for the number that immediately follows the text <code>humidity':</code> without any blanks in between.<br />
<br />
=== Regular Expressions with multiple capture Groups ===<br />
The regular expressions used in the above example for a Poolmanager will take the value that matches one capture group. This is the part of the regular expression inside (). In the above example "([\d\.]+)" refers to numerical digits or points between double quotation marks. Only the string consiting of digits and points will match inside (). This piece is assigned to the reading.<br />
<br />
You can also use regular expressions that have several capture groups which might be helpful when parsing tables. In this case an attribute like <br />
<br />
<pre><br />
reading02Regex something[ \t]+([\d\.]+)[ \t]+([\d\.]+)<br />
</pre><br />
<br />
could match two numbers. When you specify only one reading02Name like <br />
<pre><br />
reading02Name Temp<br />
</pre><br />
<br />
the name Temp will be used with the extension -1 and -2 thus giving a reading Temp-1 for the first number and Temp-2 for the second. You can also specify individual names for several readings that get parsed from one regular expression with several capture groups by defining attributes <br />
<br />
<pre><br />
reading02-1Name<br />
reading02-2Name<br />
...<br />
</pre><br />
The same notation can be used for formatting attributes like readingXMap, readingXFormat and so on.<br />
<br />
The usual way to define readings is however to have an individual regular expression with just one capture group per reading as shown in the above example.<br />
<br />
<br />
<br />
== Parsing JSON ==<br />
<br />
If a webservice delivers data in JSON format, HTTPMOD can directly parse JSON which might be easier in this case than definig regular expressions.<br />
The next example shows the data that can be requested from a Poolmanager with the following partial configuration:<br />
<br />
<pre><br />
define test2 HTTPMOD none 0<br />
attr test2 get01Name Chlor<br />
attr test2 getURL http://192.168.70.90/cgi-bin/webgui.fcgi<br />
attr test2 getHeader1 Content-Type: application/json<br />
attr test2 getHeader2 Accept: */*<br />
attr test2 getData {"get" :["34.4008.value"]}<br />
</pre><br />
<br />
The data in the HTTP response looks like this:<br />
<br />
<pre><br />
{<br />
"data": {<br />
"34.4008.value": "0.25"<br />
},<br />
"status": {<br />
"code": 0<br />
},<br />
"event": {<br />
"type": 1,<br />
"data": "48.30000.0"<br />
}<br />
}<br />
</pre><br />
<br />
the classic way to extract the value 0.25 into a reading with the name Chlor with a regex would have been<br />
<pre><br />
attr test2 get01Regex 34.4008.value":[ \t]+"([\d\.]+)"<br />
</pre><br />
<br />
with JSON you can write <br />
<pre><br />
attr test2 get01JSON data_34.4008.value <br />
</pre><br />
<br />
or if you don't care about the naming of your readings, you can simply extract all JSON data with <br />
<pre><br />
attr test2 extractAllJSON<br />
</pre><br />
<br />
which would apply to all data read from this device and create the following readings out of the HTTP response shown above:<br />
<br />
{| class="wikitable"<br />
| data_34.4008.value || 0.25<br />
|-<br />
| event_data || 48.30000.0<br />
|-<br />
| event_type || 1<br />
|-<br />
| status_code || 0<br />
|}<br />
<br />
or you can specify<br />
<pre><br />
attr test2 get01ExtractAllJSON<br />
</pre><br />
which would only apply to all data read as response to the get command defined as get01. <br />
<br />
== Parsing http / XML using xpath ==<br />
Another alternative to regex parsing is the use of XPath to extract values from HTTP responses.<br />
The following example shows how XML data can be parsed with XPath-Strict or HTML Data can be parsed with XPath.<br />
Both work similar and the example uses XML Data parsed with the XPath-Strict option:<br />
<br />
If The XML data in the HTTP response looks like this:<br />
<br />
<pre><br />
&lt;root xmlns:foo=&quot;http://www.foo.org/&quot; xmlns:bar=&quot;http://www.bar.org&quot;&gt;<br />
&lt;actors&gt;<br />
&lt;actor id=&quot;1&quot;&gt;Peter X&lt;/actor&gt;<br />
&lt;actor id=&quot;2&quot;&gt;Charles Y&lt;/actor&gt;<br />
&lt;actor id=&quot;3&quot;&gt;John Doe&lt;/actor&gt;<br />
&lt;/actors&gt;<br />
&lt;/root&gt;<br />
</pre><br />
<br />
with XPath you can write <br />
<pre><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor[2]/text()<br />
</pre><br />
<br />
This will create a reading with the Name "Actor" and the value "Charles Y".<br />
<br />
Since XPath specifications can define several values / matches, HTTPMOD can also interpret these and store them in multiple readings:<br />
<pre><br />
attr htest reading01Name Actor<br />
attr htest reading01XPath-Strict //actor/text()<br />
</pre><br />
<br />
will create the readings <br />
<br />
{| class="wikitable"<br />
| Actor-1 | Peter X<br />
|-<br />
| Actor-2 | Charles Y<br />
|-<br />
| Actor-3 | John Doe<br />
|}<br />
<br />
== Further replacements of URL, header or post data ==<br />
sometimes it might be helpful to dynamically change parts of a URL, HTTP header or post data depending on existing readings, internals or <br />
perl expressions at runtime. This might be needed to pass further variables to a server, a current date or other things. <br />
<br />
To support this HTTPMOD offers generic replacements that are applied to a request before it is sent to the server. A replacement can be defined with the attributes <br />
<br />
* replacement[0-9]*Regex<br />
* replacement[0-9]*Mode<br />
* replacement[0-9]*Value<br />
* [gs]et[0-9]*Replacement[0-9]*Value<br />
<br />
a replacement always replaces a match of a regular expression. <br />
<br />
The way the replacement value is defined can be specified with the replacement mode. If the mode is reading, then the value is interpreted as the name of a reading of the same device or as device:reading to refer to another device.<br />
If the mode is internal, then the value is interpreted as the name of an internal of the same device or as device:internal to refer to another device.<br />
The mode text will use the value as a static text and the mode expression will evaluate the value as a perl expression to compute the replacement. Inside such a replacement expression it is possible to refer to capture groups of the replacement regex.<br />
<br />
Example:<br />
<br />
<pre><br />
attr mydevice getData {"get" :["%%value%%.value"]}<br />
attr mydevice replacement01Mode text<br />
attr mydevice replacement01Regex %%value%%<br />
<br />
attr mydevice get01Name Chlor<br />
attr mydevice get01Replacement01Value 34.4008<br />
<br />
attr mydevice get02Name Something<br />
attr mydevice get02Replacement01Value 31.4024<br />
</pre> <br />
<br />
defines that %%value%% will be replaced by a static text.<br />
<br />
All Get commands will be HTTP post requests of a similar form. Only the %%value%% will be different from get to get.<br />
The first get will set the reading named Chlor and for the request it will take the generic getData and replace %%value%% with 34.4008.<br />
A second get will look the same except a different name and replacement value.<br />
<br />
The mode expression allows you to define your own replacement syntax:<br />
<pre> <br />
attr mydevice replacement01Mode expression<br />
attr mydevice replacement01Regex {{([^}]+)}}<br />
attr mydevice replacement01Value ReadingsVal("mydevice", $1, "")<br />
attr mydevice getData {"get" :["{{temp}}.value"]}<br />
</pre> <br />
<br />
In this example any {{name}} in a URL, header or post data will be passed on to the perl function ReadingsVal <br />
which uses the string between {{}} as second parameter. This way one defined replacement can be used for many different<br />
readings.<br />
<br />
== replacing reading values when they have not been updated / the device did not respond ==<br />
If a device does not respond then the values stored in readings will keep the same and only their timestamp shows that they are outdated. <br />
If you want to modify reading values that have not been updated for a number of seconds, you can use the attributes<br />
<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAge<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacementMode<br />
* (reading|get)[0-9]*(-[0-9]+)?MaxAgeReplacement<br />
<br />
Every time the module tries to read from a device, it will also check if readings have not been updated <br />
for longer than the MaxAge attributes allow. If readings are outdated, the MaxAgeReplacementMode defines how the affected<br />
reading values should be replaced. MaxAgeReplacementMode can be <code>text</code> or <code>expression</code>.<br />
<br />
MaxAge specifies the number of seconds that a reading should remain untouched before it is replaced. <br />
<br />
MaxAgeReplacement contains either a static text that is used as replacement value or a Perl expression that is evaluated to <br />
give the replacement value. This can be used for example to replace a temperature that has not bee updated for more than 5 minutes <br />
with the string "outdated - was 12": <br />
<pre><br />
attr PM readingMaxAge 300<br />
attr PM readingMaxAgeReplacement "outdated - was " . $val<br />
attr PM readingMaxAgeReplacementMode expression<br />
</pre> <br />
The variable $val contains the value of the reading before it became outdated.<br />
<br />
<br />
== Additional notes ==<br />
If you don't know which URLs, headers or POST data your web GUI uses, you might try a local proxy like BurpSuite [http://portswigger.net/burp/ BurpSuite] to track requests and responses<br />
<br />
== Advanced configuration to define a <code>set</code> and send data to a device ==<br />
<br />
When a set option is defined by attributes, the module will use the value given to the set command and translate it into an HTTP-Request that sends the value to the device.<br />
<br />
Extension to the above example for a PoolManager 5:<br />
<pre><br />
attr PM set01Name HeizungSoll<br />
attr PM set01URL http://MyPoolManager/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM set01Hint 6,10,20,30<br />
attr PM set01Min 6<br />
attr PM set01Max 30<br />
attr PM setHeader1 Content-Type: application/json<br />
attr PM set01Data {"set" :{"34.3118.value" :"$val" }}<br />
</pre><br />
<br />
This example defines a set option with the name HeizungSoll.<br />
By issuing <code>set PM HeizungSoll 10</code> in FHEM, the value 10 will be sent in the defined HTTP<br />
Post to URL <code>http://MyPoolManager/cgi-bin/webgui.fcgi</code> in the Post Data as<br />
<br />
<pre><br />
{"set" :{"34.3118.value" :"10" }}<br />
</pre><br />
<br />
The optional attributes set01Min and set01Max define input validations that will be checked in the set function. <br />
The optional attribute set01Hint will define a selection list for the Fhemweb GUI.<br />
<br />
If a parameter to a set command is not numeric but should be passed on to the device as text, then you can specify the attribute setTextArg. For example: <br />
<pre><br />
attr PM set01TextArg<br />
</pre><br />
<br />
If a set command should not require a parameter at all, then you can specify the attribute NoArg. For example: <br />
<pre><br />
attr PM set03Name On<br />
attr PM set03NoArg<br />
</pre><br />
<br />
<br />
=== Advanced configuration to create a valid session id that might be necessary in set options ===<br />
In simple cases logging in works with basic authentication. In the case HTTPMOD accepts a username and password as part of the URL in the form <br />
<pre><br />
http://User:Password@192.168.1.18/something<br />
</pre><br />
<br />
However basic auth is seldom used. If you need to fill in a username and password in a HTML form and the session is then managed by a session id, here is how to configure this:<br />
<br />
when sending data to an HTTP-Device in a set, HTTPMOD will replace any <code>$sid</code> in the URL, Headers and Post data with the internal <code>$hash->{sid}</code>. To authenticate towards the device and give this internal a value, you can use an optional multi step login procedure defined by the following attributes: <br />
<br />
;sid[0-9]*URL<br />
;sid[0-9]*IDRegex<br />
;sid[0-9]*Data.*<br />
;sid[0-9]*Header.*<br />
;sid[0-9]*IgnoreRedirects<br />
<br />
Each step can have a URL, Headers, Post Data pieces and a Regex to extract a resulting Session ID into <code>$hash->{sid}</code>.<br />
HTTPMOD will create a sorted list of steps (the numbers between sid and URL / Data / Header) and the loop through these steps and send the corresponding requests to the device. For each step a $sid in a Header or Post Data will be replaced with the current content of <code>$hash->{sid}</code>.<br />
<br />
Using this feature, HTTPMOD can perform a forms based authentication and send user name, password or other necessary data to the device and save the session id for further requests.<br />
<br />
To determine when this login procedure is necessary, HTTPMOD will first try to do a set without <br />
doing the login procedure. If the Attribute ReAuthRegex is defined, it will then compare the HTTP Response to the set request with the regular expression from ReAuthRegex. If it matches, then a <br />
login is performed. The ReAuthRegex is meant to match the error page a device returns if authentication or reauthentication is required e.g. because a session timeout has expired.<br />
<br />
If for one step not all of the URL, Data or Header Attributes are set, then HTTPMOD tries to use a <br />
<code>sidURL</code>, <code>sidData.*</code> or <code>sidHeader.*</code> Attribue (without the step number after sid). This way parts that are the same for all steps don't need to be defined redundantly.<br />
<br />
=== Example for a multi step login procedure: ===<br />
<br />
<pre><br />
attr PM reAuthRegex /html/dummy_login.htm <br />
attr PM sidURL http://192.168.70.90/cgi-bin/webgui.fcgi?sid=$sid<br />
attr PM sidHeader1 Content-Type: application/json<br />
attr PM sid1IDRegex wui.init\('([^']+)'<br />
attr PM sid2Data {"set" :{"9.17401.user" :"fhem" ,"9.17401.pass" :"password" }}<br />
attr PM sid3Data {"set" :{"35.5062.value" :"128" }}<br />
attr PM sid4Data {"set" :{"42.8026.code" :"pincode" }}<br />
</pre><br />
<br />
In this case HTTPMOD detects that a login is necessary by looking for the pattern /html/dummy_login.htm in the HTTP response. <br />
If it matches, it starts a login sequence. In the above example all steps request the same URL. In step 1 only the defined Header is sent in an HTTP get request. The response will contain a session id that is extraced with the regex wui.init\('([^']+)'.<br />
<br />
In the next step this session id is sent in a post request to the same URL where tha post data contains a username and password. The a third and a fourth request follow that set a value and a code. The result will be a valid and authorized session id that can be used in other requests where $sid is part of a URL, header or post data and will be replaced with the session id extracted above.<br />
<br />
<br />
== Advanced configuration to define a <code>get</code> and request additional data with its own request from a device ==<br />
<br />
The normal automatic HTTP request that is done repeatedly after the defined interval has elapsed works well in cases where all required readings can be requested in one common HTTP request. If however a device needs individual requests with different URLs or different POST data for each value, then another method is necessary. <br />
For such cases a <code>get</code> option can be defined and the user can either issue Fhem <code>get</code> commands each time he needs the reading or the user can set an attribute to request the reading automatically together with the normal iteration.<br />
For each <code>get</code> option attributes define an individual URL, optional headers, and post data as well as individual regular expressions and formatting options. <br />
<br />
Example for a Siemens webserver provided by Lanhydrock:<br />
<pre><br />
define ozw672 HTTPMOD https://192.168.178.8/api/auth/login.json?user=test&pwd=test 300<br />
<br />
attr ozw672 get1Name tempAussen<br />
attr ozw672 get1URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1960<br />
attr ozw672 get1Poll 1<br />
attr ozw672 get1PollDelay 1800<br />
<br />
attr ozw672 get2Name tempAussenGemischt<br />
attr ozw672 get2URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1964<br />
attr ozw672 get2Poll 1<br />
attr ozw672 get2PollDelay 1800<br />
<br />
attr ozw672 get3Name tempTWW<br />
attr ozw672 get3URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1996<br />
attr ozw672 get3Poll 1<br />
<br />
attr ozw672 get4Name tempKesselSoll<br />
attr ozw672 get4URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1910<br />
attr ozw672 get4Poll 1<br />
<br />
attr ozw672 get5Name tempKesselRuecklauf<br />
attr ozw672 get5URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1915<br />
attr ozw672 get5Poll 1<br />
<br />
attr ozw672 get6Name tempKesselRuecklaufSoll<br />
attr ozw672 get6URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1916<br />
attr ozw672 get6Poll 1<br />
<br />
attr ozw672 get7Name anzahlStartsBrenner<br />
attr ozw672 get7URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1927<br />
attr ozw672 get7PollDelay 1800<br />
attr ozw672 get7Poll 1<br />
<br />
attr ozw672 get8Name statusKessel<br />
attr ozw672 get8URL https://192.168.178.8/api/menutree/read_datapoint.json?SessionId=$sid&Id=1898<br />
attr ozw672 get8Poll 1<br />
attr ozw672 get8Regex Value": "([a-zA-Zü ]*)"<br />
attr ozw672 get8Map Aus:0, Nachlauf aktiv:5, Freigegeben für TWW:10, Freigegeben für HK:20, In Teillastbetrieb für TWW:40, In Teillastbetrieb für HK:50, In Betrieb für Trinkwasser:90, In Betrieb für Heizkreis:100<br />
<br />
attr ozw672 getRegex Value": "[ ]*([-.0-9]*)"<br />
<br />
attr ozw672 reAuthRegex .*session not valid.*<br />
attr ozw672 sid1IDRegex .*"(.*-.*-.*-[0-9a-z]*).*<br />
attr ozw672 sid1URL https://192.168.178.8/api/auth/login.json?user=test&pwd=test<br />
</pre><br />
<br />
<br />
=== Advanced attributes ===<br />
<br />
;ReAuthRegex<br />
:regular Expression to match an error page indicating that a session has expired and a new authentication for read access needs to be done. <br />
:This attribute only makes sense if you need a forms based authentication for reading data and if you specify a multi step login procedure based on the sid.. attributes.<br />
:This attribute is used for all requests. For set and get operations you can however specify individual reAuthRegexes with the [gs]et[0-9]*ReAuthRegex attributes.<br />
<br />
;sid[0-9]*URL<br />
:different URLs or one common URL to be used for each step of an optional login procedure. <br />
<br />
;sid[0-9]*IDRegex<br />
:different Regexes per login procedure step or one common Regex for all steps to extract the session ID from the HTTP response<br />
<br />
;sid[0-9]*Data.*<br />
:data part for each step to be sent as POST data to the corresponding URL<br />
<br />
;sid[0-9]*Header.*<br />
:HTTP Headers to be sent to the URL for the corresponding step<br />
<br />
;sid[0-9]*IgnoreRedirects<br />
:Tells the HttpUtils to not follow HTTP Redirects for this Request. Might be needed for some devices that set a session cookie within a 303 Redirect.<br />
<br />
<br />
;set[0-9]+Name<br />
:Name of a set option<br />
<br />
;set[0-9]*URL<br />
:URL to be requested for the set option<br />
<br />
;set[0-9]*Data<br />
:Data to be sent to the device as POST data when the set is executed<br />
<br />
;set[0-9]*Header<br />
:HTTP Headers to be sent to the device when the set is executed<br />
<br />
;set[0-9]+Min<br />
:Minimum value for input validation. <br />
<br />
;set[0-9]+Max<br />
:Maximum value for input validation. <br />
<br />
;set[0-9]+Expr<br />
:Perl Expression to compute the raw value to be sent to the device from the input value passed to the set.<br />
<br />
;set[0-9]+Map<br />
:Map that defines a mapping from raw to visible values like "0:mittig, 1:oberhalb, 2:unterhalb". This attribute atomatically creates a hint for FhemWEB so the user can choose one of the visible values or select a value with a slider.<br />
<br />
;set[0-9]+Hint<br />
:Explicit hint for fhemWEB that will be returned when set ? is seen. Can be used to get a slider or a list of values to choose from.<br />
<br />
;set[0-9]*ReAuthRegex<br />
:Regex that will detect when a session has expired an a new login needs to be performed. <br />
<br />
;get[0-9]+Name<br />
:Name of a get option and Reading to be retrieved / extracted<br />
<br />
;get[0-9]*URL<br />
:URL to be requested for the get option. If this option is missing, the URL specified during define will be used.<br />
<br />
;get[0-9]*Data<br />
:optional data to be sent to the device as POST data when the get is executed. if this attribute is not specified, an HTTP GET method will be used instead of an HTTP POST<br />
<br />
;get[0-9]*NoData<br />
:can be used to override a more generic attribute that specifies POST data for all get commands. With NoData no data is sent and therefor the request will be an HTTP GET.<br />
<br />
;get[0-9]*Header<br />
:optional HTTP Headers to be sent to the device when the get is executed<br />
<br />
;get[0-9]+Poll<br />
:if set to 1 the get is executed automatically during the normal update cycle (after the interval provided in the define command has elapsed)<br />
<br />
;get[0-9]+PollDelay<br />
:if the value should not be read in each iteration (after the interval given to the define command), then a minimum delay can be specified with this attribute. This has only an effect if the above Poll attribute has also been set. Every time the update function is called, it checks if since this get has been read the last time, the defined delay has elapsed. If not, then it is skipped this time.<br />
:PollDelay can be specified as seconds or as x[0-9]+ which means a multiple of the interval in the define command.<br />
<br />
;get[0-9]*Regex<br />
:If this attribute is specified, the Regex defined here is used to extract the value from the HTTP Response and assign it to a Reading with the name defined in the get[0-9]+Name attribute.<br />
:If this attribute is not specified for an individual Reading but as getRegex, then it applies to all get options where no specific Regex is defined.<br />
:If neither a generic getRegex attribute nor a specific get[0-9]+Regex attribute is specified, then HTTPMOD tries all Regex / Reading pairs defined in Reading[0-9]+Name and Reading[0-9]+Regex attributes and assigns the Readings that match.<br />
<br />
;get[0-9]*Expr<br />
:this attribute behaves just like Reading[0-9]*Expr but is applied to a get value. <br />
<br />
;get[0-9]*Map<br />
:this attribute behaves just like Reading[0-9]*Map but is applied to a get value.<br />
<br />
;get[0-9]*Format<br />
:this attribute behaves just like Reading[0-9]*Format but is applied to a get value.<br />
<br />
;get[0-9]*CheckAllReadings<br />
:this attribute modifies the behavior of HTTPMOD when the HTTP Response of a get command is parsed.<br />
:If this attribute is set to 1, then additionally to any matching of get specific regexes (get[0-9]*Regex), also all the Regex / Reading pairs defined in Reading[0-9]+Name and Reading[0-9]+Regex attributes are checked and if they match, the coresponding Readings are assigned as well.<br />
<br />
;replacement[0-9]*Regex<br />
:Defines a replacement to be applied to an HTTP request header, data or URL before it is sent. This allows any part of the request to be modified based on a reading, an internal or an expression.<br />
:The regex defines which part of a header, data or URL should be replaced. The replacement is defined with the following attributes:<br />
<br />
;replacement[0-9]*Mode<br />
:Defines how the replacement should be done and what replacementValue means. Valid options are text, reading, interbal and expression.<br />
<br />
;replacement[0-9]*Value<br />
:Defines the replacement. If the corresponding replacementMode is text, then value is a static text that is used as the replacement.<br><br />
:If replacementMode is reading then Value can be the name of a reading of this device or it can be a reading of a different device referred to by devicename:reading.<br><br />
:If replacementMode is internal the Value can be the name of an internal of this device or it can be an internal of a different device referred to by devicename:internal.<br><br />
:If replacementMode is expression the the Value is treated as a Perl expression that computes the replacement value. The expression can use $1, $2 and so on to refer to capture groups of the corresponding regex that is matched against the original URL, header or post data.<br />
<br />
;[gs]et[0-9]*Replacement[0-9]*Value<br />
:This attribute can be used to override the replacement value for a specific get or set.<br />
<br />
;get|reading[0-9]*MaxAge<br />
:Defines how long a reading is valid before it is automatically overwritten with a replacement when the read function is called the next time.<br />
<br />
;get|reading[0-9]*MaxAgeReplacement<br />
:specifies the replacement for MaxAge - either as a static text or as a perl expression.<br />
<br />
;get|reading[0-9]*MaxAgeReplacementMode<br />
:specifies how the replacement is interpreted: can be text and expression.<br />
<br />
<br />
;showMatched<br />
:if set to 1 then HTTPMOD will create a reading with the name MATCHED_READINGS <br />
:that contains the names of all readings that could be matched in the last request.<br />
<br />
;showError<br />
:if set to 1 then HTTPMOD will create a reading and event with the Name LAST_ERROR <br />
:that contains the error message of the last error returned from HttpUtils. <br />
<br />
;queueDelay<br />
:HTTP Requests will be sent from a queue in order to avoid blocking when several Requests have to be sent in sequence. This attribute defines the delay between calls to the function that handles the send queue. It defaults to one second.<br />
<br />
;queueMax<br />
:Defines the maximum size of the send queue. If it is reached then further HTTP Requests will be dropped and not be added to the queue<br />
<br />
;minSendDelay<br />
:Defines the minimum time between two HTTP Requests.<br />
<br />
== Links ==<br />
* Beispiel: [[Wetter_und_Wettervorhersagen#Wetter_von_Weather_Underground|Wetter von WeatherUnderground auslesen]]<br />
* {{Link2Forum|Topic=17804|LinkText=Thread}} in Fhem Forum that discusses the first version of this module <br />
* [http://perldoc.perl.org/perlretut.html Introduction to regular expressions]<br />
* [http://portswigger.net/burp/ BurpSuite]: Tool (local proxy) to help analyze http traffic<br />
<br />
[[Kategorie:IP Components]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ComfoAir&diff=10749ComfoAir2015-04-02T09:20:35Z<p>StefanStrobel: Bild hinzugefügt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information and control the ventilation devices ComfoAir from Zehnder, WHR930 from StorkAir, G90-380 from Wernig and Santos 370 DC from Paul<br />
|ModType=d<br />
|ModCmdRef=ComfoAir<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_ComfoAir.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ComfoAir module communicates with ComfoAir ventilation systems from Zehnder, especially the ComfoAir 350 (CA350) connected via serial line to the fhem computer. <br />
<br />
Many other ventilation systems use the same communication device and protocol, e.g. WHR930 from StorkAir, G90-380 from Wernig and Santos 370 DC from Paul. They can be controlled with this module as well.<br />
<br />
This module is based on the protocol description at http://www.see-solutions.de/sonstiges/Protokollbeschreibung_ComfoAir.pdf, fixes some small erorrs in this protocol description and copies some ideas from earlier modules for the same devices that were posted in the fhem forum from danhauck(Santos) and Joachim (WHR962).<br />
<br />
The module can be used in two ways depending on how fhem and / or a vendor supplied remote control device like CC Ease or CC Luxe are connected to the system. If a remote control device is connected it is strongly advised that fhem does not send data to the ventilation system as well and only listens to the communication between the vendor equipment. <br />
<br />
The RS232 interface used is not made to support more than two parties communicating and connecting fhem in parallel to a CC Ease or similar device can lead to collisions when sending data which can corrupt the ventilation system.<br />
If connected in parallel, fhem should only passively listen and <Interval> is to be set to 0. <br />
If no remote control device is connected to the ventilation systems then fhem has to take control and actively request data in the interval to be defined. Otherwiese fhem will not see any data. In this case fhem can also send commands to modify settings.<br />
<br />
== Availability == <br />
The module has been checked in<br />
<br />
== Prerequisites ==<br />
This module requires the Device::SerialPort or Win32::SerialPort module.<br />
<br />
== Define ==<br />
<pre>define <name> ComfoAir <device> <Interval></pre><br />
<br />
The module connects to the ventilation system through the given serial device and either passively listens to data that is communicated between the ventialation system and its remote control device (e.g. CC Luxe) or it actively requests data from the ventilation system every <Interval> seconds <br />
<br />
=== Example: ===<br />
<pre>define ZL ComfoAir /dev/ttyUSB1@9600 60</pre><br />
<br />
== Configuration of the module ==<br />
apart from the serial connection and the interval which both are specified in the define command there are several attributes that can optionally be used to modify the behavior of the module. <br />
The module internally gives names to all the protocol messages that are defined in the module and these names can be used in attributes to define which requests are periodically sent to the ventilation device. The same names can also be used with set commands to manually send a request. <br />
Since all messages and readings are generically defined in a data structure in the module, it should be quite easy to add more protocol details if needed without programming.<br />
<br />
The names currently defined are:<br />
<br />
<pre><br />
Bootloader-Version<br />
Firmware-Version<br />
RS232-Modus<br />
Sensordaten<br />
KonPlatine-Version<br />
Verzoegerungen<br />
Ventilation-Levels<br />
Temperaturen<br />
Betriebsstunden<br />
Status-Bypass<br />
Status-Vorheizung<br />
</pre><br />
<br />
The attributes that control which messages are sent / which data is requested every <Interval> seconds are:<br />
<br />
<pre><br />
poll-Bootloader-Version<br />
poll-Firmware-Version<br />
poll-RS232-Modus<br />
poll-Sensordaten<br />
poll-KonPlatine-Version<br />
poll-Verzoegerungen<br />
poll-Ventilation-Levels<br />
poll-Temperaturen<br />
poll-Betriebsstunden<br />
poll-Status-Bypass<br />
poll-Status-Vorheizung<br />
</pre><br />
<br />
if the attribute is set to 1, the corresponding data is requested every <Interval> seconds. If it is set to 0, then the data is not requested. by default <code>Ventilation-Levels</code>, <code>Temperaturen</code> and <code>Status-Bypass</code> are requested if no attributes are set.<br />
<br />
=== Example: ===<br />
<pre><br />
define ZL ComfoAir /dev/ttyUSB1@9600 60<br />
attr ZL poll-Status-Bypass 0<br />
define FileLog_Lueftung FileLog ./log/Lueftung-%Y.log ZL<br />
</pre><br />
<br />
== Set-Commands ==<br />
like with the attributes mentioned above, set commands can be used to send a request for data manually. The following set options are available for this:<br />
<pre><br />
request-Status-Bypass <br />
request-Bootloader-Version <br />
request-Sensordaten<br />
request-Temperaturen <br />
request-Firmware-Version <br />
request-KonPlatine-Version <br />
request-Ventilation-Levels <br />
request-Verzoegerungen <br />
request-Betriebsstunden <br />
request-Status-Vorheizung <br />
</pre><br />
<br />
additionally important fields can be modified with set:<br />
<pre><br />
Temp_Komfort (target temperature for comfort)<br />
Stufe (ventilation level)<br />
</pre><br />
<br />
== Get-Commands ==<br />
All readings that are derived from the responses to protocol requests are also available as Get commands. Internally a Get command triggers the corresponding request to the device and then interprets the data and returns the respective field value. To avoid huge option lists in FHEMWEB, only the most important Get options are visible in FHEMWEB. However this can easily be changed since all the readings and protocol messages are internally defined in the modue in a data structure and to make a Reading visible as Get option only a little option (e.g. <code>showget => 1</code> has to be added to this data structure<br />
<br />
== Readings ==<br />
<br />
The readings created by this module might look like this:<br />
<br />
[[Datei:ComfoAirReadings.jpeg|thumb]]<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;poll-Bootloader-Version <br />
;poll-Firmware-Version <br />
;poll-RS232-Modus <br />
;poll-Sensordaten <br />
;poll-KonPlatine-Version <br />
;poll-Verzoegerungen <br />
;poll-Ventilation-Levels <br />
;poll-Temperaturen <br />
;poll-Betriebsstunden <br />
;poll-Status-Bypass <br />
;poll-Status-Vorheizung <br />
:if any of these attributes are set to 1 the module includes a request for the data belonging to the named group when sending requests every <interval> seconds <br />
<br />
;hide-Bootloader-Version <br />
;hide-Firmware-Version <br />
;hide-RS232-Modus <br />
;hide-Sensordaten <br />
;hide-KonPlatine-Version <br />
;hide-Verzoegerungen <br />
;hide-Ventilation-Levels <br />
;hide-Temperaturen <br />
;hide-Betriebsstunden <br />
;hide-Status-Bypass <br />
;hide-Status-Vorheizung <br />
:if set to 1 the module prevents readings of the named group from being created even if used passively without polling and an external remote control requests this data. Please note that this attribute doesn't delete already existing readings.<br />
<br />
;queueDelay <br />
:modify the delay used when sending requests to the device from the internal queue, defaults to 1 second <br />
;queueMax<br />
:max length of the send queue, defaults to 50<br />
;timeout <br />
:set the timeout for reads, defaults to 2 seconds <br />
<br />
[[Kategorie:Lüftungssteuerung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Datei:ComfoAirReadings.jpeg&diff=10748Datei:ComfoAirReadings.jpeg2015-04-02T09:16:08Z<p>StefanStrobel: Readings des ComfoAir Moduls als Screenshot</p>
<hr />
<div>Readings des ComfoAir Moduls als Screenshot</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Kategorie:L%C3%BCftungssteuerung&diff=10747Kategorie:Lüftungssteuerung2015-04-02T09:02:47Z<p>StefanStrobel: Kategorie Lüftungssteuerungen erstellt</p>
<hr />
<div>Diese Kategorie enthält Lüftungsanlagen, kontrollierte Be- und Entlüftungen etc.<br />
<br />
[[Kategorie:Hardware Typen]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ComfoAir&diff=10744ComfoAir2015-04-02T08:55:01Z<p>StefanStrobel: Description of ComfoAir module created</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Extract information and control the ventilation devices ComfoAir from Zehnder, WHR930 from StorkAir, G90-380 from Wernig and Santos 370 DC from Paul<br />
|ModType=d<br />
|ModCmdRef=ComfoAir<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_ComfoAir.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ComfoAir module communicates with ComfoAir ventilation systems from Zehnder, especially the ComfoAir 350 (CA350) connected via serial line to the fhem computer. <br />
<br />
Many other ventilation systems use the same communication device and protocol, e.g. WHR930 from StorkAir, G90-380 from Wernig and Santos 370 DC from Paul. They can be controlled with this module as well.<br />
<br />
This module is based on the protocol description at http://www.see-solutions.de/sonstiges/Protokollbeschreibung_ComfoAir.pdf, fixes some small erorrs in this protocol description and copies some ideas from earlier modules for the same devices that were posted in the fhem forum from danhauck(Santos) and Joachim (WHR962).<br />
<br />
The module can be used in two ways depending on how fhem and / or a vendor supplied remote control device like CC Ease or CC Luxe are connected to the system. If a remote control device is connected it is strongly advised that fhem does not send data to the ventilation system as well and only listens to the communication between the vendor equipment. <br />
<br />
The RS232 interface used is not made to support more than two parties communicating and connecting fhem in parallel to a CC Ease or similar device can lead to collisions when sending data which can corrupt the ventilation system.<br />
If connected in parallel, fhem should only passively listen and <Interval> is to be set to 0. <br />
If no remote control device is connected to the ventilation systems then fhem has to take control and actively request data in the interval to be defined. Otherwiese fhem will not see any data. In this case fhem can also send commands to modify settings.<br />
<br />
== Availability == <br />
The module has been checked in<br />
<br />
== Prerequisites ==<br />
This module requires the Device::SerialPort or Win32::SerialPort module.<br />
<br />
== Define ==<br />
<pre>define <name> ComfoAir <device> <Interval></pre><br />
<br />
The module connects to the ventilation system through the given serial device and either passively listens to data that is communicated between the ventialation system and its remote control device (e.g. CC Luxe) or it actively requests data from the ventilation system every <Interval> seconds <br />
<br />
=== Example: ===<br />
<pre>define ZL ComfoAir /dev/ttyUSB1@9600 60</pre><br />
<br />
== Configuration of the module ==<br />
apart from the serial connection and the interval which both are specified in the define command there are several attributes that can optionally be used to modify the behavior of the module. <br />
The module internally gives names to all the protocol messages that are defined in the module and these names can be used in attributes to define which requests are periodically sent to the ventilation device. The same names can also be used with set commands to manually send a request. <br />
Since all messages and readings are generically defined in a data structure in the module, it should be quite easy to add more protocol details if needed without programming.<br />
<br />
The names currently defined are:<br />
<br />
<pre><br />
Bootloader-Version<br />
Firmware-Version<br />
RS232-Modus<br />
Sensordaten<br />
KonPlatine-Version<br />
Verzoegerungen<br />
Ventilation-Levels<br />
Temperaturen<br />
Betriebsstunden<br />
Status-Bypass<br />
Status-Vorheizung<br />
</pre><br />
<br />
The attributes that control which messages are sent / which data is requested every <Interval> seconds are:<br />
<br />
<pre><br />
poll-Bootloader-Version<br />
poll-Firmware-Version<br />
poll-RS232-Modus<br />
poll-Sensordaten<br />
poll-KonPlatine-Version<br />
poll-Verzoegerungen<br />
poll-Ventilation-Levels<br />
poll-Temperaturen<br />
poll-Betriebsstunden<br />
poll-Status-Bypass<br />
poll-Status-Vorheizung<br />
</pre><br />
<br />
if the attribute is set to 1, the corresponding data is requested every <Interval> seconds. If it is set to 0, then the data is not requested. by default <code>Ventilation-Levels</code>, <code>Temperaturen</code> and <code>Status-Bypass</code> are requested if no attributes are set.<br />
<br />
=== Example: ===<br />
<pre><br />
define ZL ComfoAir /dev/ttyUSB1@9600 60<br />
attr ZL poll-Status-Bypass 0<br />
define FileLog_Lueftung FileLog ./log/Lueftung-%Y.log ZL<br />
</pre><br />
<br />
== Set-Commands ==<br />
like with the attributes mentioned above, set commands can be used to send a request for data manually. The following set options are available for this:<br />
<pre><br />
request-Status-Bypass <br />
request-Bootloader-Version <br />
request-Sensordaten<br />
request-Temperaturen <br />
request-Firmware-Version <br />
request-KonPlatine-Version <br />
request-Ventilation-Levels <br />
request-Verzoegerungen <br />
request-Betriebsstunden <br />
request-Status-Vorheizung <br />
</pre><br />
<br />
additionally important fields can be modified with set:<br />
<pre><br />
Temp_Komfort (target temperature for comfort)<br />
Stufe (ventilation level)<br />
</pre><br />
<br />
== Get-Commands ==<br />
All readings that are derived from the responses to protocol requests are also available as Get commands. Internally a Get command triggers the corresponding request to the device and then interprets the data and returns the respective field value. To avoid huge option lists in FHEMWEB, only the most important Get options are visible in FHEMWEB. However this can easily be changed since all the readings and protocol messages are internally defined in the modue in a data structure and to make a Reading visible as Get option only a little option (e.g. <code>showget => 1</code> has to be added to this data structure<br />
<br />
== Attributes ==<br />
<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;poll-Bootloader-Version <br />
;poll-Firmware-Version <br />
;poll-RS232-Modus <br />
;poll-Sensordaten <br />
;poll-KonPlatine-Version <br />
;poll-Verzoegerungen <br />
;poll-Ventilation-Levels <br />
;poll-Temperaturen <br />
;poll-Betriebsstunden <br />
;poll-Status-Bypass <br />
;poll-Status-Vorheizung <br />
:if any of these attributes are set to 1 the module includes a request for the data belonging to the named group when sending requests every <interval> seconds <br />
<br />
;hide-Bootloader-Version <br />
;hide-Firmware-Version <br />
;hide-RS232-Modus <br />
;hide-Sensordaten <br />
;hide-KonPlatine-Version <br />
;hide-Verzoegerungen <br />
;hide-Ventilation-Levels <br />
;hide-Temperaturen <br />
;hide-Betriebsstunden <br />
;hide-Status-Bypass <br />
;hide-Status-Vorheizung <br />
:if set to 1 the module prevents readings of the named group from being created even if used passively without polling and an external remote control requests this data. Please note that this attribute doesn't delete already existing readings.<br />
<br />
;queueDelay <br />
:modify the delay used when sending requests to the device from the internal queue, defaults to 1 second <br />
;queueMax<br />
:max length of the send queue, defaults to 50<br />
;timeout <br />
:set the timeout for reads, defaults to 2 seconds <br />
<br />
[[Kategorie:Lüftungssteuerung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ArduCounter&diff=10718ArduCounter2015-04-01T09:38:47Z<p>StefanStrobel: Description of the module ArduCounter created</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Count pulses, calculate time between pulses and convert this to readings for e.g. power consumption of Energy meters<br />
|ModType=contrib<br />
|ModCmdRef=ArduCounter<br />
|ModForumArea=Sonstiges<br />
|ModFTopic=19285<br />
|ModTechName=98_ArduCounter.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
The ArduCounter module works together with an arduino board connected via usb that runs the provided ArduCounter.ino sketch to count impulses from energy meters with an S0 interface or similar devices in configurable intervals. <br />
<br />
The arduino board counts impulses on every configured input pin using pin change interrupts. It reports the counted number of impulses per pin together with the time between the first and last impulse per pin.<br />
The module converts this to readings like power consumption and it is not affected by delays or the load of your fhem server.<br />
Counters are configured using attributes that define which Arduino pins should count pulses and in which intervals the counted numbers should be reported.<br />
<br />
== Availability == <br />
The module has been checked in to the contrib area. <br />
<br />
The corresponding arduino sketch can also be found under contrib in the subdirectory arduino/.<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module as well as an arduino uno, nano, jeenode or similar device that runs the ArduCounter sketch.<br />
<br />
== Define ==<br />
<pre>define <name> ArduCounter <device></pre><br />
<br />
<device> specifies the serial port to communicate with the Arduino.<br />
<br />
The name of the serial-device depends on your distribution. You can also specify a baudrate if the device name contains the @ character, e.g.: /dev/ttyUSB0@9600<br />
<br />
=== Example:=== <br />
<pre>define AC ArduCounter /dev/ttyUSB2@9600</pre><br />
<br />
== Configuration of the module ==<br />
Specify the pins where S0 interfaces are connected to as <br />
<pre>attr AC pinX rising pullup</pre> <br />
The X in pinX can be an Arduino pin number with or without the letter D e.g. pin4, pin5, pinD4, pinD6 ...<br />
After the pin ypu can define if rising or falling edges of the signals should be counted. The optional keyword pullup activates the pullup resistor for the given Arduino Pin.<br />
<br />
== Example: ==<br />
<pre><br />
define AC ArduCounter /dev/ttyUSB2@9600<br />
attr AC factor 1000<br />
attr AC interval 60 300<br />
attr AC pinD4 rising pullup<br />
attr AC pinD5 rising pullup<br />
</pre><br />
this defines two counters connected to the pins D4 and D5, each with the pullup resistor activated. <br />
Impulses will be counted when the signal changes from 0 to 1.<br />
The ArduCounter sketch which must be loaded on the Arduino implements this using pin change interrupts,<br />
so all avilable input<br />
<br />
== Get-Commands ==<br />
;info <br />
:send the internal command <code>show</code> to the Arduino board to get current counts. This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Set-Commands ==<br />
;raw <br />
:send the value to the Arduino board so you can directly talk to the sketch using its commands. This is not needed for normal operation but might be useful sometimes for debugging.<br />
<br />
== Supported readings ==<br />
;pinX<br />
:the counted total of impulses for pin X<br />
<br />
;powerX<br />
:the result of (delta count) / (delta time) * factor.<br />
:the number of pulses counted during the last reporting interval diveded by the time between the first pulse and the last pulse in the interval, multiplied by a configurabe factor.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
: ...<br />
;pin.* <br />
:Define a pin of the Arduino board as input. This attribute expects either <code>rising</code>, <code>falling</code> or <code>change</code> as value, followed by on optional <code>pullup</code>.<br />
;interval <br />
:Define the reporting interval after which the Arduino board should hand over the count and the time from first to last impulse per pin. <br />
:This Attribute expects two numbers as value. The first is the minimal interval, the second the maximal interval. <br />
:Nothing is reported during the minimal interval. The Arduino board just counts and remembers the time between the first impulse and the last impulse for each pin.<br />
:After the minimal interval the Arduino board reports count and time for those pins where impulses were encountered. <br />
:If no impulses were encountered, the pin is not reported until the second interval is over. <br />
:The default intervals are 60 seconds as minimal time and 5 minutes as maximum interval.<br />
;factor <br />
:Define a multiplicator for calculating the power from the impulse count and the time between the first and the last impulse<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Energieverbrauchsmessung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Modbus&diff=10640Modbus2015-03-26T20:15:29Z<p>StefanStrobel: Links ergänzt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Library or physical device to extract information from devices with a Modbus interface or send information to such devices <br />
|ModType=d<br />
|ModCmdRef=Modbus<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_Modbus.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
<br />
<br />
Modbus defines a physical modbus interface and library functions to be called from other logical modules / devices.<br />
This low level module takes care of the communication with modbus devices and provides Get, Set and cyclic polling of Readings as well as formatting and input validation functions.<br />
<br />
The logical device modules for individual machines only need to define the supported modbus function codes and objects of the machine with the modbus interface in data structures. These data structures are then used by this low level module to implement Set, Get and automatic updateing of readings in a given interval.<br />
<br />
The Modbus module supports Modbus RTU over serial / RS485 lines as well as Modbus TCP and Modbus RTU over TCP. It defines read / write functions for Modbus holding registers, input registers, coils and discrete inputs.<br />
<br />
See [[ModbusAttr]] if you don't want to use a library to develop your own module and if you are looking for a generic Modbus Module instead that can be configured with attributes.<br />
<br />
== Availability == <br />
The module has been checked in.<br />
<br />
== Prerequisites ==<br />
This module requires the Device::SerialPort or Win32::SerialPort module if you want to communicate with modbus devices over a serial line.<br />
<br />
== Define of a modbus interface device for serial communication ==<br />
<pre><br />
define <name> Modbus <device><br />
</pre><br />
<br />
A define of a physical device based on this module is only necessary if a shared physical device like a RS485 USB adapter is used. <br />
In the case Modbus TCP this module will be used as a library for other modules that define all the data objects and no define of the base module is needed.<br />
<br />
Example:<br />
<pre><br />
define ModBusLine Modbus /dev/ttyUSB1@9600<br />
</pre><br />
<br />
In this example the module opens the given serial interface and other logical modules like [[ModbusAttr]] or [[ModbusSET]] can access several Modbus devices connected to this bus concurrently.<br />
<br />
== Set-Commands ==<br />
this low level device module doesn't provide set commands for itself but implements set <br />
for logical device modules that make use of this module as a library. See ModbusSET for example.<br />
<br />
== Get-Commands ==<br />
this low level device module doesn't provide get commands for itself but implements get <br />
for logical device modules that make use of this module as a library.<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;queueDelay<br />
:modify the delay used when sending requests to the device from the internal queue, defaults to 1 second<br />
<br />
;queueMax<br />
:max length of the send queue, defaults to 100<br><br />
<br />
== Writing modules for devices using this module as a library ==<br />
<br />
Writing a module for a physical device with modbus interface is easy when you use the 98_Modbus.pm module as a library. <br />
To use this module as a library for other fhem modules you only have to define a data structure that defines the mapping between modbus data objects (holding registers, input registers, coils or discrete inputs) and fhem readings.<br />
Additionally the module needs to contain a few <code>package</code> and <code>use</code> statements and an initialize function at the beginning, that assigns a few special variables to point to functions of the Modbus base module. <br />
<br />
Example for a module that is called ModbusSET:<br />
<pre><br />
package main;<br />
use strict;<br />
use warnings;<br />
<br />
sub ModbusSET_Initialize($)<br />
{<br />
my ($modHash) = @_;<br />
require "$attr{global}{modpath}/FHEM/98_Modbus.pm";<br />
$modHash->{parseInfo} = \%SET10parseInfo; # defines registers, inputs, coils etc. for this Modbus Defice<br />
$modHash->{deviceInfo} = \%SET10deviceInfo; # defines properties of the device, defaults and supported function codes<br />
ModbusLD_Initialize($modHash); # Generic function of the Modbus module does the rest<br />
$modHash->{AttrList} = $modHash->{AttrList}; # Standard Attributes like IODEv etc <br />
}<br />
</pre><br />
<br />
The name of the initialize-Function has to match the name of the module. In the above example this is <code>ModbusSET_Initialize</code>. Most of the steps needed in an initialize function are provided by the library function <code>ModbusLD_Initialize</code>. This function tells fhem to use the library functions for <code>define</code>, <code>set</code>, <code>get</code> and other typical functions in a module. See [[DevelopmentModuleIntro]] for more background information on writing fhem modules if you are curious.<br />
<br />
=== Introduction to the parseInfo structure ===<br />
<br />
Typically the data structure to map between data objects of the modbus device and fhem readings is named <code>parseInfo</code> with a part of the name of the module itself as prefix. In the example of the module 98_ModbusSET.pm which uses Modbus.pm to implement a module for SET Silent 10 heat pumps, the structure is called <code>SET10parseInfo</code>.<br />
<br />
As an example a very simple definition of a parseInfo structure for a heat pump could look like this:<br />
<br />
<pre><br />
my %XYparseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
},<br />
"h258" => { reading => "Temp_Wasser_Aus",<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
set => 1, # this value can be set<br />
}<br />
);<br />
</pre><br />
<br />
This would be the main part of the module and map from holding register 256 to a fhem reading named Temp_Wasser_Ein, holding register 258 to Temp_Wasser_Aus and 770 to Temp_Soll. <br />
<br />
All readings will be read from the device in an interval that the user can specify when he issues the define command for your module.<br />
The meaning of <code>set => 1</code> is that the holding register 770 can also be written to with a set command. Fhem will check that the value written is not smaller than 10 and not bigger than 32 as specified above.<br />
<br />
More complex example:<br />
<pre><br />
my %SET10parseInfo = (<br />
"h256" => { reading => "Temp_Wasser_Ein", # name of the reading for this value<br />
name => "Pb1", # internal name of this register in the hardware doc<br />
expr => '$val / 10', # conversion of raw value to visible value <br />
len => 1,<br />
},<br />
"h770" => { reading => "Temp_Soll", <br />
name => "ST03",<br />
expr => '$val / 10', # convert raw value to readable temp<br />
setexpr => '$val * 10', # expression to convert a set value to the internal value <br />
min => 10, # input validation for set: min value<br />
max => 32, # input validation for set: max value<br />
hint => "8,10,20,25,28,29,30,30.5,31,31.5,32",<br />
set => 1, # this value can be set<br />
},<br />
"h771" => { reading => "Hysterese", # Hex Adr 303<br />
name => "ST04",<br />
expr => '$val / 10',<br />
setexpr => '$val * 10',<br />
poll => "x10", # only poll every 10th iteration.<br />
min => 0.5,<br />
max => 3,<br />
set => 1,<br />
},<br />
"h777" => { reading => "Hyst_Mode", # Hex Adr 0309<br />
name => "ST10",<br />
map => "0:mittig, 1:oberhalb, 2:unterhalb", <br />
poll => "once", # only poll once (or after a set)<br />
set => 1,<br />
},<br />
"i800" => { reading => "Voltage", # Input register <br />
unpack => "f>", # this value is a float<br />
len => 2, # the float occupies two input registers, 800 and 801<br />
},<br />
);<br />
</pre><br />
<br />
There are many more options that can be specified for each data object / reading. If these options are not specified, the base module assumes defaults that typically make sense. However if you want to modify the defaults, you can either define explicit values in the parseInfo structure or you can define another data structure typically called deviceInfo. <br />
<br />
=== Introduction to the deviceInfo structure ===<br />
<br />
The deviceInfo structure is completely optional. If you don't define it in your module, the base module takes default values that work in ost cases. If you only want to override a few of the defaults, you can just define them and leave other options or sections out.<br />
A simple device info structure that modifies some defaults could look like this:<br />
<br />
<pre><br />
my %SET10deviceInfo = (<br />
"timing" => {<br />
timeout => 3, # timeout is 3 seconds /default would be 2<br />
commDelay => 0.7, # wait 0.7 seconds before sending after receiving<br />
sendDelay => 0.7, # wait at least 0.7 seconds for another send<br />
}, <br />
"c" => { <br />
read => 1, # function code 1 to read coils (this could be omitted because it is the default anyways<br />
write => 5, # dito<br />
},<br />
"h" => { <br />
read => 3, <br />
write => 6, <br />
defLen => 1, # default legth is 1 object<br />
combine => 5, <br />
defShowGet => 1, <br />
defUnpack => "s>", # default data format is a signed 16 bit integer for holding registers <br />
},<br />
);<br />
</pre><br />
<br />
The deviceInfo structure contains five optional parts. Timing defines timing values and the remaining parts define settings or defaults for coils (c), discrete inputs (d), input registers (i) and holding registers (h). <br />
<br />
for each modbus object type you can change what function code should be used to read or write to the object. This is completely optional and if nothing is specified, the base module assumes function codes 1,2,3 and 4 for reading as well as 5 and 6 for writing which works for many modbus devices. If you prefer to use function code 16 for writing to holding registers, you can specify "write => 16" in the "h" part.<br />
<br />
=== usage of a module created this way ===<br />
<br />
a logical module written this way will have a define command that can work in two ways. <br />
If your module would be called ModbusSET and it is using a serial line connection (Modbus RTU over RS485 oer over RS232):<br />
<br />
<pre><br />
define <iodevice> Modbus /dev/device@baudrate<br />
define <name> ModbusSET <Id> <Interval> </code><br />
</pre><br />
<br />
In this case, a physical serial interface device is defined first using the Modbus module. Then a device based on your module (ModbusSET in the example) is defined for each physical modbus device connected to the serial line. For a RS485 bus, several devices with different Ids can be connected to the same bus.<br />
<br />
Example:<br />
<pre><br />
define ModbusRS485 Modbus /dev/rs485@9600<br />
define PWP ModbusAttr 5 60<br />
</pre><br />
<br />
this defines the device and it will use the readings that you coded in the parseInfo data structure.<br />
<br />
Alternatively your module would also support Modbus TCP or Modbus RTU over TCP with the following define syntax:<br />
<br />
<pre><br />
define <name> ModbusAttr <Id> <Interval> <Address:Port> <RTU|TCP><br />
</pre><br />
In this case no serial interface device is necessary and your module connects to the modbus device directly via TCP using either Modbus TCP or Modbus RTU over TCP.<br />
<br />
Example:<br />
<pre><br />
define PWP ModbusAttr 1 30 192.168.1.115:502 TCP<br />
</pre><br />
<br />
=== General information about data objects ===<br />
<br />
Modbus devices can use many different ways to encode values in their data objects. A temperature might be stored multiplied with 10 as a 16 bit integer value in one holding register so you have to read the integer and divide it by 10 to get the real temperature value back. It might also be stored as a 32 bit float data type that spans two adjacent input registers.<br />
The modbus base module implements a very generic way to handle different encodings without real programming: It lets you define the Perl unpack code to convert a raw data string to a Perl value, a Perl expression to do further computation and a length in data objects. <br />
<br />
This way a temperature stored in a 16 bit signed integer as the value multiplied by 10 can be described with the unpack code "s>" and the expression "$val / 10". A float value spanning 2 registers would be described with an unpack code "f>" and a len of 2. No expression is needed in this case. See Perldoc on the pack function for a detailed explation of pack and unpack codes.<br />
<br />
The idea here is that you should be able to define any mapping, encoding, transformation or formatting of data objects without programming by simpy describing them.<br />
<br />
=== All options in parseInfo ===<br />
<br />
Most options here are optional and can be used if there is a need but they can also be omitted. If most readings require the same options and the option is different from the default, it is also possible to define a different default per modbus data object type in another data structure (see deviceInfo).<br />
<br />
;reading <br />
:name of the reading to be used in Fhem e.g. Temp_Wasser_ein<br />
<br />
;expr <br />
:perl expression to convert a string after it has been read. The original value is in $val e.g. $val / 10<br />
<br />
;map <br />
:a map string to convert an value from the device to a more readable output string or to convert a user input to the machine representation e.g. "0:mittig, 1:oberhalb, 2:unterhalb" <br />
<br />
;format <br />
:a format string for sprintf to format a value read, e.g. %.1f<br />
<br />
;len <br />
:number of Registers this value spans, can be 2 for a 32 bit float which is stored in 2 registers<br />
<br />
;unpack <br />
:defines the translation between data in the module and in the communication frame see the documentation of the perl pack function for details. example: "n" for an unsigned 16 bit value or "f>" for a float that is stored in two registers or "s>" for signed 16 bit integer in big endian format<br />
<br />
;showget <br />
:can be set to 1 to allow a Fhem get command to read this value from the device. All defined objects can be used in a get command that is issued on the command line. This parameter only controls if fhemweb will offer a get command for the object.<br />
<br />
;poll <br />
:defines if this value is included in the read that the module does every defined interval this can be changed by a user with an attribute<br />
<br />
;polldelay <br />
:if a value should not be read in each iteration (after interval has passed), this value can be set to an explicit time in seconds. The update function will then verify if this delay has elapsed since the last read of this object. If not, the read is skipped.<br />
<br />
;set <br />
:can be set to 1 to allow writing this value with a Fhem set-command<br />
<br />
;min <br />
:min value for input validation in a set command. If the user issues e.g. set Device Temp_Soll 10, Fhem will check if the given value 10 is bigger or equal the defined min and smaller or equal the defined max.<br />
<br />
;max <br />
:max value for input validation in a set command<br />
<br />
;hint <br />
:string for fhemweb to create a selection or slider<br />
<br />
;setexpr <br />
:per expression to convert an input string to the machine format before writing this is typically the reverse of the above expr, e.g. $val * 10<br />
<br />
;name<br />
:optional internal name of the value in the modbus documentation of the physical device, e.g. pb1<br />
<br />
<br />
=== All options in deviceInfo ===<br />
<br />
Keys in the timing section:<br />
<br />
;timeout <br />
:how long to wait for a response from the device, can be overwritten by attribute timeout in logical device. Defaults to two seconds if this is not specified<br />
<br />
;commDelay <br />
:minimal delay in secounds between two communications e.g. a read a the next write, can be overwritten with attribute commDelay if added to AttrList in _Initialize below defaults to 0.1 seconds if not specified<br />
<br />
;sendDelay <br />
:minimal delay in seconds between two sends, can be overwritten with the attribute sendDelay if added to AttrList in _Initialize function below. Defaults to 0.1 seconds if not specified<br />
<br />
Keys per object type (c = coil, d = discrete input, i = input register, h = holding register)<br />
<br />
;read <br />
:function code to use for reading this object type (e.g. 3 for holding registers) defaults to function codes 1-4 depending on the object types if nothing else is specified (3 to read holding register, 1 to read coils and so on)<br />
<br />
;write <br />
:function code to use for writing this object type (e.g. 6 or 16 for holding registers) defaults to function codes 5 and 6 depending on the object types if nothing else is specified (6 to read holding register, 5 to write coils and so on)<br />
<br />
;defLen <br />
:default len for objects using this type (e.g. can be set to 2 if the device mainly provides float values that span 2 registers (2 times 16 Bit) can be overwritten in parseInfo per reading by specifying the key "len" defaults to 1 if not specified<br />
<br />
;defFormat <br />
:format string to do sprintf with the value can be overwritten in parseInfo per reading by specifying the key "format" if no format is specified here and none in parseInfo, the the reading is set without further formatting (which is typically fine)<br />
<br />
;defUnpack <br />
:default pack / unpack code to convert raw values, e.g. "n" for a 16 bit integer or "f>" for a big endian float can be overwritten in parseInfo per reading by specifying the key "unpack" if not specified here and not in parseInfo, then the raw value is interpreted as "n" which is 16 bit unsigned integer in big endian format<br />
<br />
;defPoll <br />
:defines that objects of this type should be polled by default unless specified otherwise in parseInfo or by attributes can be overwritten in parseInfo per reading by specifying the key defaultpoll if not specified here or in parseInfo, the object is not polled<br />
<br />
;defShowGet <br />
:defines that Fhemweb shows a Get option (by returning it as reslut to get ?) for objects of this type can be overwritten in parseInfo per reading by specifying the key showget defaults to 0.<br />
<br />
;combine <br />
:max number of registers that the device is willing to deliver in one read request. The modbus application layer protocol specification allows for more than 100 but most devices limit this to 5, 10 or some other number. This option defaults to 1 if not specified.<br />
<br />
For an example of a full module that is based on the mechanisms described here see 98_ModbusSET.pm.<br />
<br />
=== Attributes of your module ===<br />
<br />
a module based on the base module / library 98_Modbus.pm can also allow the end user to modify properties of each reading if you want to allow it. All you have to do is to offer an attribute by adding its name to the variable $modHash->{AttrList} in your initialize function.<br />
<br />
If for example you want to allow the user to modify the maximum value for the reading Temp_Soll, you can add "Temp_Soll-max " to this variable and the use can the set this attribute. The attribute takes precedence over the max potentially already defined in your parseInfo structure.<br />
<br />
there are two ways that the base module accepts such readings. One is the reading name followed by "-" and the option to override, the alternative syntax is "obj-" followed by the first letter of an object type (c/d/h/i) and a decimal address just like the main key of an object in the parseInfo structure. <br />
<br />
Instaed of allowing the attribute Temp_Soll-max for the max value of reading Temp_Soll which corresponds to holding register 770, you can alternatively add the attribute name "obj-h770-min " to $modHash->{AttrList}.<br />
<br />
If the user is allowd to specify such attributes solely depends on the contents of the $modHash->{AttrList} variable. All the processing is already built into the base module.<br />
<br />
If you want to allow the user the override the formatting of readings then you can add "obj-[cdih][1-9][0-9]*-format " as a regular expression that allows format specifications for all possible data objects.<br />
<br />
The module 98_ModbusAttr for example is also based on 98_Modbus.pm and allows all possible attributes so the user can completely define his device with attributes and without a parseInfo or deviceInfo structure.<br />
<br />
In the same way you can allow the user to override the device specific options and defaults with attributes that start with "dev-", followed by the section of the deviceInfo and the name of the option. If you want to allow the user to modify the function code to be used for writing holding registers, you can add the attribute "obj-h-write " and the user can then set this attribute to 6 or 16 as he prefers.<br />
It is up to the module author to decide if this makes sense.<br />
<br />
An assignment that allows most options to the user could be:<br />
<br />
<pre><br />
$modHash->{AttrList} = $modHash->{AttrList} . " " .<br />
"obj-[cdih][1-9][0-9]*-reading " .<br />
"obj-[cdih][1-9][0-9]*-name " .<br />
"obj-[cdih][1-9][0-9]*-set " .<br />
"obj-[cdih][1-9][0-9]*-min " .<br />
"obj-[cdih][1-9][0-9]*-max " .<br />
"obj-[cdih][1-9][0-9]*-hint " .<br />
"obj-[cdih][1-9][0-9]*-expr " .<br />
"obj-[cdih][1-9][0-9]*-map " .<br />
"obj-[cdih][1-9][0-9]*-setexpr " .<br />
"obj-[cdih][1-9][0-9]*-format " .<br />
"obj-[cdih][1-9][0-9]*-len " .<br />
"obj-[cdih][1-9][0-9]*-unpack " .<br />
"obj-[cdih][1-9][0-9]*-showget " .<br />
<br />
"obj-[cdih][1-9][0-9]*-poll " .<br />
"obj-[cdih][1-9][0-9]*-polldelay " .<br />
"poll-.* " .<br />
"polldelay-.* " .<br />
<br />
"dev-([cdih]-)*read " .<br />
"dev-([cdih]-)*write " .<br />
"dev-([cdih]-)*combine " .<br />
"dev-([cdih]-)*defLen " .<br />
"dev-([cdih]-)*defFormat " .<br />
"dev-([cdih]-)*defUnpack " .<br />
"dev-([cdih]-)*defPoll " .<br />
"dev-([cdih]-)*defShowGet " .<br />
"dev-timing-timeout " .<br />
"dev-timing-sendDelay " .<br />
"dev-timing-commDelay ";<br />
}<br />
</pre><br />
<br />
== Examples for logical device modules that use this base module ==<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM220M]<br />
;[http://forum.fhem.de/index.php/topic,25315.60.html SDM630M]<br />
:modules for energy meters from B+G E-Tech & EASTON written by Roger<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG103]<br />
;[http://forum.fhem.de/index.php/topic,25315.45.html UMG604]<br />
:modules for the UMG103 and UMG604 meters from Janitza<br />
;[[ModbusSET]]<br />
:module for the set silent heat pumps from Schmidt Energie Technik<br />
;[[ModbusAttr]]<br />
:generic modbus device module where the data objects, addresses, display formats, function codes and other things can be configured using Fhem attributes similar to HTTPMOD<br />
<br />
<br />
[[Kategorie:Interfaces]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Benutzer:StefanStrobel&diff=10638Benutzer:StefanStrobel2015-03-26T19:59:38Z<p>StefanStrobel: /* Benutzer Stefan Strobel */ Ergänzungen</p>
<hr />
<div>== Benutzer Stefan Strobel ==<br />
Wikiseiten in Arbeit: <br />
* [[DevelopmentModuleIntro|Einführung in die Entwicklung von Modulen für FHEM]]<br />
* detailliertere Beschreibung von [[HTTPMOD]] (siehe [http://forum.fhem.de/index.php/topic,17804.0.html])<br />
* Beschreibung des generischen Moduls [[ModbusAttr]] für Geräte mit Modbus-Interface. Readings werden wie bei [[HTTPMOD]] per Attr spezifiziert. (basiert auf dem Modul [[Modbus]])<br />
* Beschreibung des generischen Basis-Moduls [[Modbus]] für Geräte mit Modbus-Interface<br />
* Beschreibung des Moduls [[ModbusSET]] für Wärmepumpen der Reihe SET Silent von Schmidt Energie Technik mit Modbus-Interface<br />
* Beschreibung des Moduls FReplacer - z.B. Für die Integration eines [[Kindle_Display|Kindle als Fhem-Display]]<br />
* Doku zum Modul für den Zugriff auf [[Waterkotte_heat_pump_with_Resümat_CD4|Wärempumpen von Waterkotte mit Steuerung Resümat CD4]]<br />
* Ausführlichere Doku zum Modul für Lüftungsanlagen wie Zehnder [[ComfoAir]] und baugleiche Geräte<br />
* Beschreibung des Moduls [[ArduCounter]] (generischer Impulszähler auf Arduino-Basis z.B. für Stromzähler mit S0 Schnittstelle), siehe [[http://forum.fhem.de/index.php/topic,19285.0.html]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Waterkotte_heat_pump_with_Res%C3%BCmat_CD4&diff=10637Waterkotte heat pump with Resümat CD42015-03-26T19:53:19Z<p>StefanStrobel: Seite für ältere Wärmepumpen von Waterkotte erstellt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Control / Extract information from Waterkotte Heat pumps with Resümat CD4 controller<br />
|ModType=x<br />
|ModCmdRef=WKRCD4<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_WKRCD4.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
WKRCD4 allows to control / communicate with older heat pumps from Waterkotte that have a control unit called Resümat CD4 with an RS232 serial interface. Newer heat pumps from Waterkotte typically have a modbus interface that can be controlled with the Module [[ModbusAttr]].<br />
<br />
== Availability == <br />
The module has been checked in to the contrib area<br />
<br />
== Prerequisites ==<br />
This module requires Device::SerialPort or Win32::SerialPort module.<br />
<br />
== Define ==<br />
<pre><br />
define <name> WKRCD4 <devicename@speed> <Interval><br />
</pre><br />
The module connects to the heat pump through a serial interface and actively requests data from the heat pump every <Interval> seconds<br />
<br />
=== Example:===<br />
<br />
<pre><br />
define WP WKRCD4 /dev/dualser0@9600 60<br />
attr WP stateFormat Status<br />
attr WP event-min-interval .*:3600<br />
attr WP event-on-change-reading .*<br />
</pre><br />
<br />
== Configuration of the module ==<br />
<br />
apart from the serial interface and the interval there is nothing that needs to be defined.<br />
<br />
== Get-Commands ==<br />
<br />
There are only two parameters for which an explicit GET has been implemented. Most values are just readings that are automatically requested:<br />
<pre><br />
Hzg-TempBasisSoll<br />
WW-Temp-Soll<br />
</pre><br />
<br />
== Set-Commands ==<br />
<br />
There are only one parameter that can be changed:<br />
<pre><br />
Hzg-TempBasisSoll<br />
</pre><br />
<br />
== Supported readings ==<br />
<br />
the module extracts the following readings from the heat pump:<br />
<pre><br />
Versions-Nummer <br />
Temp-Aussen <br />
Temp-Ruecklauf-Soll <br />
Temp-Ruecklauf <br />
Temp-Vorlauf <br />
Temp-WW-Soll <br />
Temp-WW <br />
Temp-Raum <br />
Temp-WQuelle-Ein <br />
Temp-WQuelle-Aus <br />
Temp-Verdampfer <br />
Temp-Kondensator <br />
Temp-Saugleitung <br />
Druck-Verdampfer <br />
Druck-Kondensator <br />
Hzg-TempEinsatz <br />
Hzg-TempBasisSoll <br />
Hzg-KlSteilheit <br />
Hzg-KlBegrenz <br />
Hzg-TempRlSoll <br />
Hzg-TempRlIst <br />
Hzg-TmpRaumSoll <br />
Hzg-RaumEinfluss <br />
Hzg-ExtAnhebung <br />
Hzg-Zeit-Ein <br />
Hzg-Zeit-Aus <br />
Hzg-AnhebungEin <br />
Hzg-AnhebungAus <br />
Hzg-St2Begrenz <br />
Hzg-Hysterese <br />
Hzg-PumpenNachl <br />
Klg-Abschaltung <br />
Klg-Temp-Einsatz <br />
Klg-TeBasisSoll <br />
Klg-KlSteilheit <br />
Klg-KlBegrenz <br />
Klg-KlSollwert <br />
Klg-Temp-Rl <br />
Ww-Abschaltung <br />
Ww-Zeit-Ein <br />
Ww-Zeit-Aus <br />
Ww-Temp-Ist <br />
Ww-Temp-Soll <br />
Ww-Hysterese <br />
Uhrzeit <br />
Datum <br />
BetrStundenKompressor <br />
BetrStundenHzgPu <br />
BetrStundenWwPu <br />
BetrStundenSt2 <br />
Zeit <br />
SetBetriebsMode <br />
Display-Zeile-1 <br />
Display-Zeile-2 <br />
Status-Gesamt <br />
Status-Heizung <br />
Status-Kuehlung <br />
Mode-Heizung <br />
Mode-Kuehlung <br />
Mode-Warmwasser <br />
Heizung <br />
Kuehlung <br />
Warmwasser <br />
</pre><br />
<br />
== Attributes ==<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Heizungssteuerung]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Benutzer:StefanStrobel&diff=10636Benutzer:StefanStrobel2015-03-26T19:52:40Z<p>StefanStrobel: kleinere Korrekturen</p>
<hr />
<div>== Benutzer Stefan Strobel ==<br />
Wikiseiten in Arbeit: <br />
* [[DevelopmentModuleIntro|Einführung in die Entwicklung von Modulen für FHEM]]<br />
* detailliertere Beschreibung von [[HTTPMOD]] (siehe [http://forum.fhem.de/index.php/topic,17804.0.html])<br />
* Beschreibung des generischen Moduls [[ModbusAttr]] für Geräte mit Modbus-Interface<br />
* Beschreibung des Moduls [[ModbusSET]] für Wärmepumpen der Reihe SET Silent von Schmidt Energie Technik mit Modbus-Interface<br />
* Beschreibung des Moduls FReplacer - z.B. Für die Integration eines [[Kindle_Display|Kindle als Fhem-Display]]<br />
* Doku zum Modul für den Zugriff auf [[Waterkotte_heat_pump_with_Resümat_CD4|Wärempumpen von Waterkotte mit Steuerung Resümat CD4]]<br />
* Ausführlichere Doku zum Modul für Lüftungsanlagen wie Zehnder [[ComfoAir]] und baugleiche Geräte<br />
* Beschreibung des Moduls [[ArduCounter]] (generischer Impulszähler auf Arduino-Basis z.B. für Stromzähler mit S0 Schnittstelle), siehe [[http://forum.fhem.de/index.php/topic,19285.0.html]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=Benutzer:StefanStrobel&diff=10635Benutzer:StefanStrobel2015-03-26T19:35:42Z<p>StefanStrobel: Links ergänzt</p>
<hr />
<div>== Benutzer Stefan Strobel ==<br />
Wikiseiten in Arbeit: <br />
* [[DevelopmentModuleIntro|Einführung in die Entwicklung von Modulen für FHEM]]<br />
* detailliertere Beschreibung von [[HTTPMOD]] (siehe [http://forum.fhem.de/index.php/topic,17804.0.html])<br />
* Beschreibung des generischen Moduls [[ModbusAttr]] für Geräte mit Modbus-Interface<br />
* Beschreibung des Moduls [[ModbusSET]] für Wärmepumpen der Reihe SET Silent von Schmidt Energie Technik mit Modbus-Interface<br />
* Beschreibung des Moduls FReplacer - z.B. Für die Integration eines [[Kindle_Display|Kindle als Fhem-Display]]<br />
* Doku zum Modul für den Zugriff auf [[WKRCD4|Wärempumpen von Waterkotte mit Steuerung Resümat CD4]]<br />
* Ausführlichere Doku zum Modul für Lüftungsanlagen wie Zehnder [[ComfoAir]] und baugleiche Geräte<br />
* Beschreibung des Moduls [[ArduCounter]] (generischer Impulszähler auf Arduino-Basis z.B. für Stromzähler mit S0 Schnittstelle), siehe [[http://forum.fhem.de/index.php/topic,19285.0.html]]</div>StefanStrobelhttp://wiki.fhem.de/w/index.php?title=ModbusSET&diff=10634ModbusSET2015-03-26T19:29:28Z<p>StefanStrobel: Kategorien ergänzt</p>
<hr />
<div>{{Infobox Modul<br />
|ModPurpose=Control / Extract information from SET Silent Pool Heat pumps<br />
|ModType=d<br />
|ModCmdRef=ModbusSET<br />
|ModForumArea=Sonstiges<br />
|ModTechName=98_ModbusSET.pm<br />
|ModOwner=StefanStrobel ({{Link2FU|3960|Forum}} / [[Benutzer:StefanStrobel|Wiki]])<br />
}}<br />
ModbusSET allows to control / communicate with Silent 10 pool heat pumps from Schmidt Energie Technik (SET). These heat pumps provide a Modbus interface over RS485. The module probably works with other heat pumps from SET as well and since the control device used in these heat pumps is an iChill IC121 from Dixell, it could even work for other heat pumps with this controller as well or with few changes. It defines the modbus holding registers for the temperature sensors and reads them in a defined interval.<br />
<br />
The module can also be used as a programming example for the usage of the base [[Modbus]] module.<br />
<br />
== Availability == <br />
The module has been checked in<br />
<br />
== Prerequisites ==<br />
This module requires the basic Modbus module which itsef requires Device::SerialPort or Win32::SerialPort module.<br />
<br />
== Define ==<br />
<pre><br />
define <name> ModbusSET <Id> <Interval><br />
</pre><br />
The module connects to the heat pump with Modbus Id <Id> through an already defined modbus device and actively requests data from the heat pump every <Interval> seconds<br />
<br />
=== Example:===<br />
<br />
<pre><br />
define WP ModbusSET 1 60<br />
</pre><br />
<br />
== Configuration of the module ==<br />
<br />
apart from the modbus id and the interval which both are specified in the define command there is nothing that needs to be defined.<br />
However there are some attributes that can optionally be used to modify the behavior of the module.<br />
<br />
The attributes that control which messages are sent / which data is requested every <Interval> seconds are:<br />
<br />
<pre><br />
poll-Hyst_Mode<br />
poll-Temp_Luft<br />
poll-Temp_Wasser_Aus_Off<br />
poll-Temp_Wasser_Ein_Off<br />
poll-Temp_Wasser_Aus<br />
poll-Hysterese<br />
poll-Temp_Wasser_Ein<br />
poll-Temp_Soll<br />
poll-Temp_Luft_Off<br />
poll-Temp_Verdampfer<br />
poll-Temp_Verdampfer_Off<br />
</pre><br />
<br />
if the attribute is set to 1, the corresponding data is requested every <Interval> seconds. If it is set to 0, then the data is not requested.<br />
<br />
by default the temperatures are requested if no attributes are set.<br />
if some readings should be polled, but less frequently than the normal interval, you can specify a pollDelay-Attribute for the reading.<br />
The pollDelay attribute allows to poll objects at a lower rate than the interval specified in the define command. you can either specify a time in seconds or number prefixed by "x" which means a multiple of the interval of the define command.<br />
if you specify a normal number then it is interpreted as minimal time between the last read and another automatic read. Please note that this does not create an individual interval timer. Instead the normal interval timer defined by the interval of the define command will check if this reading is due or not yet. So the effective interval will always be a multiple of the interval of the define.<br />
<br />
=== Example:===<br />
<pre><br />
define WP ModbusSET 1 60<br />
attr WP poll-Temp_Soll 0<br />
attr WP pollDelay-Hysterese 300<br />
</pre><br />
<br />
== Set-Commands ==<br />
<br />
The following set options are available:<br />
<pre><br />
Hysterese (defines the hysterese in Kelvin)<br />
Hyst_Mode (defines the interpretation of hysterese for the heating and can be set to mittig, oberhalb or unterhalb)<br />
Temp_Wasser_Aus_Off (offset of sensor in Kelvin - used to kalibrate)<br />
Temp_Wasser_Ein_Off (offset of sensor in Kelvin - used to kalibrate)<br />
Temp_Luft_Off (offset of sensor in Kelvin - used to kalibrate)<br />
Temp_Verdampfer_Off (offset of sensor in Kelvin - used to kalibrate)<br />
Temp_Soll (target temperature of the heating pump)<br />
</pre><br />
<br />
== Get-Commands ==<br />
All readings are also available as Get commands. Internally a Get command triggers the corresponding <br />
request to the device and then interprets the data and returns the right field value. To avoid huge option lists in FHEMWEB, only the most important Get options<br />
are visible in FHEMWEB. However this can easily be changed since all the readings and protocol messages are internally defined in the modue in a data structure <br />
and to make a Reading visible as Get option only a little option (e.g. <code>showget => 1</code> has to be added to this data structure<br />
<br />
== Attributes ==<br />
;do_not_notify<br />
;readingFnAttributes<br />
<br />
;poll-Hyst_Mode <br />
;poll-Temp_Luft <br />
;poll-Temp_Wasser_Aus_Off <br />
;poll-Temp_Wasser_Ein_Off <br />
;poll-Temp_Wasser_Aus <br />
;poll-Hysterese <br />
;poll-Temp_Wasser_Ein <br />
;poll-Temp_Soll <br />
;poll-Temp_Luft_Off <br />
;poll-Temp_Verdampfer <br />
;poll-Temp_Verdampfer_Off <br />
:include a read request for the corresponding registers when sending requests every interval seconds <br />
<br />
;pollDelay-* <br />
:set a delay for polling individual Readings. In case some readings should be polled less frequently than the normal delay specified during define. Specifying a pollDelay will not create an individual timer for polling this reading but check if the delay is over when the normal update interval is handled.<br />
:You can either specify a time in seconds or number prefixed by "x" which means a multiple of the interval of the define command. If you specify a normal numer then it is interpreted as minimal time between the last read and another automatic read. Please note that this does not create an individual interval timer. Instead the normal interval timer defined by the interval of the define command will check if this reading is due or not yet. So the effective interval will always be a multiple of the interval of the define.<br />
<br />
;dev-timing-timeout <br />
:set the timeout for reads, defaults to 2 seconds <br />
;dev-timing-minSendDelay <br />
:minimal delay between two requests sent to this device<br />
;dev-timing-minCommDelay <br />
:minimal delay between requests or receptions to/from this device<br />
<br />
[[Kategorie:Other Components]]<br />
[[Kategorie:Heizungssteuerung]]</div>StefanStrobel