A generic payload protocol by KPN, also suitable for LoRa
ThingsML is an extension of the SenML specification, in a way SenML is designed to be extended (as described in Section 4.4 of the SenML specification).
ThingsML provides a more compact way of sending measurements based on SenML by indexing commonly used measurements and allowing to only send the measurement index instead of the full name and unit. It is inspired by the way developers try to cope with the small number of bytes available when using LoRaWAN for data transmission. They often employ context-based encoding schemes, meaning you need to know a lot of specifics to be able to decode a message. ThingsML tries to find the middle ground between bloaty generic protocols and niche specific protocols.
The ThingsML specification only covers the JSON and CBOR data types.
JSON is a suitable data type for Internet and M2M communication.
CBOR is a suitable data type for LoRa communication.
Currently only the following use cases are supported in KPN Things:
ThingsML for LoRa for uplink messages from own LoRa devices.
Support for M2M and downlink message will follow. For M2M communication we currently use SenML, and for LoRa downlink messages only raw payload communication is available.
ThingsML is part of our Device SDK (written in C). With the Device SDK you can easily setup communication between your Device and KPN Things. Learn more about the Things Device SDK.
With the ThingsML playground you can compare SenML with ThingsML, as well as JSON with CBOR. Visit the ThingsML playground.
This section explains the way ThingsML works.
In SenML the type of measurement present in a record is expressed by its name and its unit. You will find almost all temperature measurements described with name temperature and unit Cel. In ThingsML every of these very common types of measurements is indexed in the measurement index list. This means every combination of SenML name and unit gets a unique number.
The measurement index value starts with -24, because this value is finetuned on the CBOR specification to allow for optimal usage of this protocol for LoRaWAN applications. In CBOR a number from -24 to 23 can be encoded without additional bytes. For more information read the Major types section in the CBOR specifications https://tools.ietf.org/html/rfc7049#section-2.1.
So now, if you want to put a measurement in SenML, you only need to add the measurement index to the record instead of the full name and unit, saving you a lot of bytes. This measurement index is stored in a new field called the measurement index field and it has the label name i_.
If you want to send a measurement using ThingsML, but you want to send a measurement that is not in the measurement index list, you still can. Just add the name and unit to the record as you are used to, and a ThingsML decoder will accept it nonetheless. ThingsML is fully compatible with SenML, since it is a valid extension of the latter.
To experiment with ThingsML, please visit the online ThingsML Playground:
This section explains the fields that ThingsML introduces or uses.
Since the measurement index field is a custom field that is mandatory to understand to correctly process the pack, it must have a label name that ends with the "_" character [SenML, Section 12.2].
For every update of the measurement index list we will up the version number. The version number of the list you implemented can be added to the ThingsML pack in the bver field. This guarantees that your ThingsML pack will not be interpreted incorrectly.
We aim to only extend and not modify the measurement index list for backwards compatibility.
Also, we aim to keep the ThingsML Decoder in KPN Things up-to-date with the most recent version of the measurement index list. So, when using ThingsML in combination with KPN Things, you would not need to use the bver field.
This section explains more about ThingsML in JSON data format.
The difference between SenML and ThingsML in JSON is illustrated with the example below.
SenML
277 bytes
ThingsML
203 bytes (-30%)
This section explains more about ThingsML in CBOR data format.
CBOR stands for Concise Binary Object Representation and is a standard data format: https://tools.ietf.org/html/rfc7049. It is the more compact counterpart of JSON. That means each JSON object has its equal in CBOR and the other way around. In the SenML specification additional translation steps from JSON to CBOR are proposed to further decrease the size of SenML packs in CBOR. More information about SenML in CBOR can be found here: https://tools.ietf.org/html/rfc8428#section-6.
ThingsML has an integer CBOR representation for the label of the measurement index field. Officially, only default SenML labels should have an integer as CBOR representation. But this would save us 2 extra bytes per measurement in the SenML record, giving is more compactness. Since the measurement index is a regular field, the CBOR representation should be a positive integer.
Name
Label
CBOR Label
Measurement index
i_
23
The example below shows a measurement list in both SenML and ThingsML in CBOR hexadecimal representation and CBOR diagnostic notation. The hexadecimal representation could be copy-pasted into the CBOR tool at http://cbor.me.
SenML
142 bytes
ThingsML
66 bytes (-55%)
85a300686c6174697475646502fb4049f9ed288ce70401636c6174a3
00696c6f6e67697475646502fb40155165d3996fa801636c6f6ea300
6b74656d706572617475726502fb403dc00000000000016343656ca3
006e62617474657279566f6c7461676502fb400cb851eb851eb80161
56a3006b74656d706572617475726502fb40313851eb851eb8016343
656c
85a202fb4049f9ed288ce7041735a202fb40155165d3996fa81734a2
02fb403dc000000000001737a202fb400cb851eb851eb81706a202fb
40313851eb851eb81737
This example encodes all decimal numbers as double point float representations. Further payload size optimization can be achieved by using single point or half point float representations.
SenML
187 bytes
ThingsML
128 bytes (-30%)
87a221782075726e3a6465763a4445564555493a3031323334353637
38393031323334353a221a5cb35887a300686c6174697475646502fb
4049f9ed288ce70401636c6174a300696c6f6e67697475646502fb40
155165d3996fa801636c6f6ea3006b74656d706572617475726502fb
403dc00000000000016343656ca3006e62617474657279566f6c7461
676502fb400cb851eb851eb8016156a3006765787454656d7002fb40
313851eb851eb8016343656ca20062696f04f4
87a221782075726e3a6465763a4445564555493a3031323334353637
38393031323334353a221a5cb35887a202fb4049f9ed288ce7041735
a202fb40155165d3996fa81734a202fb403dc000000000001737a202
fb400cb851eb851eb81706a3006765787454656d7002fb40313851eb
851eb8016343656ca20062696f04f4
ThingsML for LoRa follows the following additional aspects:
ThingsML for LoRa only supports CBOR data representation, since this is the most compact way of sending ThingsML.
You may omit the base name and base time values from your measurement. The LoRaWAN network layer already provides this information to KPN Things, so your device doesn't have to.
This section gives the measurement index list and the way it is moderated. This list can be found on the common measurements page.
If you have a measurement name and unit you think should be in the ThingsML measurement index list? Please contact the author for a request for change. Proposed records will be added if they are generic and unique in the list. For instance Temperature in Fahrenheit shall not be added to the list, since translation to Celsius can be trivially implemented in the device.
We plan to support use case specific measurement tables, enabling you to define your own measurement index list.
Date
Description
26-11-2019
First version
8-1-2020
Updated text
27-1-2020
Added 3 measurements to the index list
Added change log table
7-4-2020
Specification released publicly
18-11-2020
Made introduction and ThingsML for LoRa more clear.
Specification by Paul Marcelis. Email: pjmarcelis@gmail.com
These changes only apply to Freemium projects.
KPN Things currently rounds the SenML bt and t fields in its SenML output to whole seconds. We are going to remove or change this rounding to allow for more accurate time values. Although time values always use seconds as their unit, it means that these values may contain fractional seconds.
For example, "t" may contain the value 0.001 or -33.333333 whereas before they would be rounded as 0 or -33 respectively. Depending on the device, a "t" value may change from "t": 1.659571200E9 to "t": 1.6595711995E9. So this changes in this example from Thursday, August 4, 2022 0:00:00 to Wednesday, August 3, 2022 23:59:59.500.
Check your device specification and your implementation to see if this has an impact on you. You may be affected if your software assumes the time values are always integers.
The base values sent by KPN Things are currently presented as a separate SenML record. As each record represents a measurement, this record containing only the base values implies a measurement of "v": 0 for "n": "". This is incorrect, as such a measurement was never made. In the future, KPN Things will no longer generate a separate SenML record with only the base name and base time fields, but instead will follow the SenML specification and store the base values as needed in existing records. Note: base values can appear in any SenML record, not just the first. To properly resolve the records in a SenML pack care must be taken to follow the specification to decompress the pack correctly.
To read more on this topic see Section 4 of the SenML standard
base values can appear in any SenML record, not just the first. To properly resolve the records in a SenML pack, care must be taken to follow the specification to decompress the pack correctly.
Currently the system sends SenML like this:
In the first measurement, the "v" and "u" fields of the first measures are missing (which is allowed in SenML). But this indicates a sensor with the name urn:dev:DEVEUI:0123456789ABCDEF: that has no units or values, which is not a representation of reality.
In the new situation this will no longer occur. The base values will be part of the first measurement, and will look like this:
Check the software on your end to see if it can handle this.
The primary aim of this document is to provide a comprehensive guide on how to correctly interpret Sensor Measurement List (SenML) data within the KPN Things platform. This guidance is essential for ensuring accurate data processing and utilization on the receiving end of the platform.
To get the most out of SenML data, it is important to interpret it correctly. Not doing so may result in missing or incorrect data in your downstream applications. As SenML data structures can be complex and varied, understanding the nuances is vital for reliable data handling.
SenML data is structured in JSON format to ensure uniformity and readability. Following are examples of correctly structured SenML records. All the records in this paragraph represent the same SenML data, this will help you understand what the SenML that arrives at your Destination might look like.
In the above example, we can see unresolved SenML. The SenML contains a base name (bn) and a base time (bt). A SenML Record is referred to as "resolved" if it does not contain any base values, i.e., labels starting with the character b, (except for Base Version fields), and has no relative times. The message above, when "resolved", looks like this:
As mentioned earlier, all the SenML in this paragraph represents the same data. So when it is resolved, it matches the SenML records in the previous example.
The records should be in chronological order in the resolved SenML, however in this example the time values are equal so switching the order still represents the same SenML:
Numeric values should be valid JSON numbers, which means that they can have a decimal point and an exponent. Therefore the above example is equal to the following example:
Time values are relative to the base time. So if measurement records have time values, they are relative to the value of the previous base time record. If you look at the example below, the base time should be added to the time fields of the measurements to resolve the time of the measurement.
The SenML data is the same as all the previous SenML messages.
It is possible to have multiple base value records. The base value applies to all successive measurements until another of that base value is found. In the example below, a single base name is used for all the records. On the contrary, the base time is altered in the second record. Since in both record the base time and the time elements add up to the same value, this data still equals the previous SenML examples.
Working with SenML requires an understanding of the SenML data structures. Simple parsing strategies, such as string splitting, might work with one SenML representation, but could fail when dealing with a different one. When SenML is sent out to your destination, we give no guarantees on the representation.
It is advised that incoming data should be parsed and normalised to an appropriate internal representation of the resolved SenML records, before any further processing is attempted.
To assist in correctly interpreting SenML, several libraries are available for different programming languages:
These libraries are designed to handle the complexities of the SenML data and provide more reliable parsing and data extraction.
When in doubt you can always use the RFC: rfc8428 as a reference. Or take a look at the senml-spec GitHub page.
At KPN Things, we strive to adhere as closely as we can to the SenML standard. This dedication sometimes necessitates modifications on our end to align more precisely with these standards. We recommend our customers to also embrace this approach, ensuring customer applications remain robust and future-proof by closely following the SenML standard.
Understanding and correctly interpreting SenML data is crucial for accurate and reliable data processing when interacting with the KPN Things platform. We urge our users to follow best practices and consult the official SenML specification. Adhering to these guidelines not only guarantees data integrity but also maximizes the benefits derived from using the KPN Things platform.
Introduction on the SenML data representation
For Freemium projects, the way KPN Things processes device data to SenML changed (per December 13th 2022). Read all the details about the Changes in KPN SenML.
SenML is the default data format for device data in KPN Things. Currently KPN Things only supports SenML in JSON. We plan to support SenML in CBOR in the future.
A SenML message is officially called a SenML pack. One SenML pack consists of one or more SenML records. One SenML record expresses a single measurement. That way a SenML pack expresses a set of measurements
The first record in a SenML pack always contains base information. Base information expresses from which device the measurements are from (base name) and the moment the measurements were taken (base time).
There are different type of SenML records:
Number record - for communicating a number measurement
Boolean record - for communicating an on/off measurement
String record - for communicating a string measurement
Data record - for communicating raw data measurement
Below you find a non-exhaustive example of a SenML pack:
Line nr.
Explanation
2
First line with base attributes, stating measurement source and timestamp.
3
Number record
4
Number record with relative time
5
Boolean record
6
String record
7
Data record
This section explains the attributes you can use to express your measurements in SenML
When the base name and/or base time values are present in the pack, they are available in the first measurement of the pack and applicable for all the measurements in the pack.
The base name of a SenML pack states which device the measurements are from. It should contain a unique resource name. The parts of the base name, which are colon separated, are as following:
urn stands for unique resource name.
dev means we are identifying a device.
DEVEUI is the type of device identifier used to identify the device.
01234567890ABCDEF is the DevEUI, the device identifier.
In the example above we identify a LoRa device by its DEVEUI. If we would have an M2M device, we would identify it by its IMEI and the base name would be as following: urn:dev:IMEI:0123456789012345:.
A base name always ends with a colon :!
The timestamp of the measurement in UNIX timestamp format. If KPN Things receives a measurement without base time, the time of receiving the measurement is taken as base time. SenML being sent by KPN Things will always contain a base time.
Name of the individual measurement.
The (..) name MUST consist only of characters out of the set "A" to "Z", "a" to "z", and "0" to "9", as well as "-", ":", ".", "/", and "_"; furthermore, it MUST start with a character out of the set "A" to "Z", "a" to "z", or "0" to "9". - The SenML spec, RFC8428 section 4.5.1
When designing your SenML measurement format for M2M and Internet devices, you should take into account our Common measurements list. By using as much name and unit pairs from that list, you achieve higher compatibility with current and future data processing features.
Unit of the measurement. Write your u-values in camel case, so start with a lower case character.
The timestamp of an individual record relative to the base time in seconds.
Value of the number measurement.
Boolean value for a boolean measurement. Should be true or false.
String value for a string measurement.
Data value for a data measurement. Should be a base64 encoded string.
There are more attributes in the SenML specification, but these values are not used by KPN Things. You can read about them in the Sensor Measurements Lists specification:
For easier integration with KPN Things we have created client libraries in C and Python for you to use in your devices.
To make SenML more suitable for communication over LoRa, we have introduced ThingsML. ThingsML is a valid extension of SenML that introduces measurement indices. We have listed several common measurements, allowing you to only send the measurement index number instead of sending the complete name and unit of the measurement. Learn more about ThingsML.
A generic payload protocol by KPN, also suitable for LoRa.
In KPN Things we use ThingsML as a generic decoding and encoding service for the communication with Devices. SDKs from KPN Things are available to further help you implement ThingsML in your device.
We use SenML within KPN Things and when forwarding the data to a destination.
