ThingsML
A generic payload protocol by KPN, also suitable for LoRa.
Last updated
A generic payload protocol by KPN, also suitable for LoRa.
Last updated
ThingsML is an extension of the SenML specification, in a way SenML is designed to be extended (as described in Section 4.4 of the SenML specification).
ThingsML provides a more compact way of sending measurements based on SenML by indexing commonly used measurements and allowing to only send the measurement index instead of the full name and unit. It is inspired by the way developers try to cope with the small number of bytes available when using LoRaWAN for data transmission. They often employ context-based encoding schemes, meaning you need to know a lot of specifics to be able to decode a message. ThingsML tries to find the middle ground between bloaty generic protocols and niche specific protocols.
The ThingsML specification only covers the JSON and CBOR data types.
In KPN Things we use ThingsML as a generic decoding and encoding service for the communication with Devices. SDKs from KPN Things are available to further help you implement ThingsML in your device.
Currently only the following use cases are supported in KPN Things:
ThingsML for LoRa for uplink messages from own LoRa devices.
Support for M2M and downlink message will follow. For M2M communication we currently use SenML, and for LoRa downlink messages only raw payload communication is available.
ThingsML is part of our Device SDK (written in C). With the Device SDK you can easily setup communication between your Device and KPN Things. Learn more about the Things Device SDK.
With the ThingsML playground you can compare SenML with ThingsML, as well as JSON with CBOR. Visit the ThingsML playground.
This section explains the way ThingsML works.
In SenML the type of measurement present in a record is expressed by its name and its unit. You will find almost all temperature measurements described with name temperature and unit Cel. In ThingsML every of these very common types of measurements is indexed in the measurement index list. This means every combination of SenML name and unit gets a unique number.
The measurement index value starts with -24, because this value is finetuned on the CBOR specification to allow for optimal usage of this protocol for LoRaWAN applications. In CBOR a number from -24 to 23 can be encoded without additional bytes. For more information read the Major types section in the CBOR specifications https://tools.ietf.org/html/rfc7049#section-2.1.
So now, if you want to put a measurement in SenML, you only need to add the measurement index to the record instead of the full name and unit, saving you a lot of bytes. This measurement index is stored in a new field called the measurement index field and it has the label name i_.
If you want to send a measurement using ThingsML, but you want to send a measurement that is not in the measurement index list, you still can. Just add the name and unit to the record as you are used to, and a ThingsML decoder will accept it nonetheless. ThingsML is fully compatible with SenML, since it is a valid extension of the latter.
To experiment with ThingsML, please visit the online ThingsML Playground:
This section explains the fields that ThingsML introduces or uses.
Since the measurement index field is a custom field that is mandatory to understand to correctly process the pack, it must have a label name that ends with the "_" character [SenML, Section 12.2].
For every update of the measurement index list we will up the version number. The version number of the list you implemented can be added to the ThingsML pack in the bver field. This guarantees that your ThingsML pack will not be interpreted incorrectly.
We aim to only extend and not modify the measurement index list for backwards compatibility.
Also, we aim to keep the ThingsML Decoder in KPN Things up-to-date with the most recent version of the measurement index list. So, when using ThingsML in combination with KPN Things, you would not need to use the bver field.
This section explains more about ThingsML in JSON data format.
The difference between SenML and ThingsML in JSON is illustrated with the example below.
This section explains more about ThingsML in CBOR data format.
CBOR stands for Concise Binary Object Representation and is a standard data format: https://tools.ietf.org/html/rfc7049. It is the more compact counterpart of JSON. That means each JSON object has its equal in CBOR and the other way around. In the SenML specification additional translation steps from JSON to CBOR are proposed to further decrease the size of SenML packs in CBOR. More information about SenML in CBOR can be found here: https://tools.ietf.org/html/rfc8428#section-6.
ThingsML has an integer CBOR representation for the label of the measurement index field. Officially, only default SenML labels should have an integer as CBOR representation. But this would save us 2 extra bytes per measurement in the SenML record, giving is more compactness. Since the measurement index is a regular field, the CBOR representation should be a positive integer.
The example below shows a measurement list in both SenML and ThingsML in CBOR hexadecimal representation and CBOR diagnostic notation. The hexadecimal representation could be copy-pasted into the CBOR tool at http://cbor.me.
This example encodes all decimal numbers as double point float representations. Further payload size optimization can be achieved by using single point or half point float representations.
ThingsML for LoRa follows the following additional aspects:
ThingsML for LoRa only supports CBOR data representation, since this is the most compact way of sending ThingsML.
You may omit the base name and base time values from your measurement. The LoRaWAN network layer already provides this information to KPN Things, so your device doesn't have to.
This section gives the measurement index list and the way it is moderated. This list can be found on the common measurements page.
If you have a measurement name and unit you think should be in the ThingsML measurement index list? Please contact the author for a request for change. Proposed records will be added if they are generic and unique in the list. For instance Temperature in Fahrenheit shall not be added to the list, since translation to Celsius can be trivially implemented in the device.
We plan to support use case specific measurement tables, enabling you to define your own measurement index list.
Specification by Paul Marcelis. Email: pjmarcelis@gmail.com
SenML
277 bytes
ThingsML
203 bytes (-30%)
[
{"bn":"urn:dev:DEVEUI:0123456789012345:","bt":1555257479, "n":"latitude","v":51.95255,"u":"lat"},
{"n":"longitude","v":5.32949,"u":"lon"},
{"n":"temperature","v":29.75,"u":"Cel"},
{"n":"batteryVoltage","v":3.59,"u":"V"},
{"n":"extTemp","v":17.22,"u":"Cel"},
{"n":"io","vb":false}
]
[
{"bn":"urn:dev:DEVEUI:0123456789012345:","bt":1555257479, "v":51.95255,"i_":-22},
{"v":5.32949,"i_":-21},
{"v":29.75,"i_":-24},
{"v":3.59,"i_":6},
{"n":"extTemp","v":17.22,"u":"Cel"},
{"n":"io","vb":false}
]
Name
Label
CBOR Label
Measurement index
i_
23
SenML
142 bytes
ThingsML
66 bytes (-55%)
85a300686c6174697475646502fb4049f9ed288ce70401636c6174a3
00696c6f6e67697475646502fb40155165d3996fa801636c6f6ea300
6b74656d706572617475726502fb403dc00000000000016343656ca3
006e62617474657279566f6c7461676502fb400cb851eb851eb80161
56a3006b74656d706572617475726502fb40313851eb851eb8016343
656c
85a202fb4049f9ed288ce7041735a202fb40155165d3996fa81734a2
02fb403dc000000000001737a202fb400cb851eb851eb81706a202fb
40313851eb851eb81737
[
{0: "latitude", 2: 51.95255, 1: "lat"},
{0: "longitude", 2: 5.32949, 1: "lon"},
{0: "temperature", 2: 29.75, 1: "Cel"},
{0: "batteryVoltage", 2: 3.59, 1: "V"},
{0: "temperature", 2: 17.22, 1: "Cel"}
]
[
{2: 51.95255, 23: -22},
{2: 5.32949, 23: -21},
{2: 29.75, 23: -24},
{2: 3.59, 23: 6},
{2: 17.22, 23: -24}
]
SenML
187 bytes
ThingsML
128 bytes (-30%)
87a221782075726e3a6465763a4445564555493a3031323334353637
38393031323334353a221a5cb35887a300686c6174697475646502fb
4049f9ed288ce70401636c6174a300696c6f6e67697475646502fb40
155165d3996fa801636c6f6ea3006b74656d706572617475726502fb
403dc00000000000016343656ca3006e62617474657279566f6c7461
676502fb400cb851eb851eb8016156a3006765787454656d7002fb40
313851eb851eb8016343656ca20062696f04f4
87a221782075726e3a6465763a4445564555493a3031323334353637
38393031323334353a221a5cb35887a202fb4049f9ed288ce7041735
a202fb40155165d3996fa81734a202fb403dc000000000001737a202
fb400cb851eb851eb81706a3006765787454656d7002fb40313851eb
851eb8016343656ca20062696f04f4
[
{-2: "urn:dev:DEVEUI:0123456789012345:", -3: 1555257479},
{0: "latitude", 2: 51.95255, 1: "lat"},
{0: "longitude", 2: 5.32949, 1: "lon"},
{0: "temperature", 2: 29.75, 1: "Cel"},
{0: "batteryVoltage", 2: 3.59, 1: "V"},
{0: "extTemp", 2: 17.22, 1: "Cel"},
{0: "io", 4: false}
]
[
{-2: "urn:dev:DEVEUI:0123456789012345:", -3: 1555257479},
{2: 51.95255, 23: -22},
{2: 5.32949, 23: -21},
{2: 29.75, 23: -24},
{2: 3.59, 23: 6},
{0: "extTemp", 2: 17.22, 1: "Cel"},
{0: "io", 4: false}
]
Date
Description
26-11-2019
First version
8-1-2020
Updated text
27-1-2020
Added 3 measurements to the index list
Added change log table
7-4-2020
Specification released publicly
18-11-2020
Made introduction and ThingsML for LoRa more clear.
