ThingsML

A generic payload protocol by KPN, also suitable for LoRa.

ThingsML is an extension of the SenML specification, in a way SenML is designed to be extended (as described in Section 4.4 of the SenML specification).

Introduction

ThingsML provides a more compact way of sending measurements based on SenML by indexing commonly used measurements and allowing to only send the measurement index instead of the full name and unit. It is inspired by the way developers try to cope with the small number of bytes available when using LoRaWAN for data transmission. They often employ context-based encoding schemes, meaning you need to know a lot of specifics to be able to decode a message. ThingsML tries to find the middle ground between bloaty generic protocols and niche specific protocols.

The ThingsML specification only covers the JSON and CBOR data types.

JSON is a suitable data type for Internet and M2M communication.
CBOR is a suitable data type for LoRa communication.

In KPN Things

Currently only the following use cases are supported in KPN Things:

ThingsML for LoRa for uplink messages from own LoRa devices.

Support for M2M and downlink message will follow. For M2M communication we currently use SenML, and for LoRa downlink messages only raw payload communication is available.

Additional sources

Device SDK

ThingsML is part of our Device SDK (written in C). With the Device SDK you can easily setup communication between your Device and KPN Things. Learn more about the Things Device SDK.

ThingsML playground

With the ThingsML playground you can compare SenML with ThingsML, as well as JSON with CBOR. Visit the ThingsML playground.

Working

This section explains the way ThingsML works.

Measurement index list

In SenML the type of measurement present in a record is expressed by its name and its unit. You will find almost all temperature measurements described with name temperature and unit Cel. In ThingsML every of these very common types of measurements is indexed in the measurement index list. This means every combination of SenML name and unit gets a unique number.

The measurement index value starts with -24, because this value is finetuned on the CBOR specification to allow for optimal usage of this protocol for LoRaWAN applications. In CBOR a number from -24 to 23 can be encoded without additional bytes. For more information read the Major types section in the CBOR specifications https://tools.ietf.org/html/rfc7049#section-2.1.

Measurement index field

So now, if you want to put a measurement in SenML, you only need to add the measurement index to the record instead of the full name and unit, saving you a lot of bytes. This measurement index is stored in a new field called the measurement index field and it has the label name i_.

Compatible with SenML

If you want to send a measurement using ThingsML, but you want to send a measurement that is not in the measurement index list, you still can. Just add the name and unit to the record as you are used to, and a ThingsML decoder will accept it nonetheless. ThingsML is fully compatible with SenML, since it is a valid extension of the latter.

Online playground

To experiment with ThingsML, please visit the online ThingsML Playground:

ThingsML Playground

Fields

This section explains the fields that ThingsML introduces or uses.

New field: measurement index field "i_"

Since the measurement index field is a custom field that is mandatory to understand to correctly process the pack, it must have a label name that ends with the "_" character [SenML, Section 12.2].

Used field: base version "bver"

For every update of the measurement index list we will up the version number. The version number of the list you implemented can be added to the ThingsML pack in the bver field. This guarantees that your ThingsML pack will not be interpreted incorrectly.

We aim to only extend and not modify the measurement index list for backwards compatibility.

Also, we aim to keep the ThingsML Decoder in KPN Things up-to-date with the most recent version of the measurement index list. So, when using ThingsML in combination with KPN Things, you would not need to use the bver field.

JSON

This section explains more about ThingsML in JSON data format.

Example

The difference between SenML and ThingsML in JSON is illustrated with the example below.

SenML

277 bytes

ThingsML

203 bytes (-30%)

[
  {"bn":"urn:dev:DEVEUI:0123456789012345:","bt":1555257479, "n":"latitude","v":51.95255,"u":"lat"},
  {"n":"longitude","v":5.32949,"u":"lon"},
  {"n":"temperature","v":29.75,"u":"Cel"},
  {"n":"batteryVoltage","v":3.59,"u":"V"},
  {"n":"extTemp","v":17.22,"u":"Cel"},
  {"n":"io","vb":false}
]

[
  {"bn":"urn:dev:DEVEUI:0123456789012345:","bt":1555257479, "v":51.95255,"i_":-22},
  {"v":5.32949,"i_":-21},
  {"v":29.75,"i_":-24},
  {"v":3.59,"i_":6},
  {"n":"extTemp","v":17.22,"u":"Cel"},
  {"n":"io","vb":false}
]

CBOR

This section explains more about ThingsML in CBOR data format.

About CBOR

CBOR stands for Concise Binary Object Representation and is a standard data format: https://tools.ietf.org/html/rfc7049. It is the more compact counterpart of JSON. That means each JSON object has its equal in CBOR and the other way around. In the SenML specification additional translation steps from JSON to CBOR are proposed to further decrease the size of SenML packs in CBOR. More information about SenML in CBOR can be found here: https://tools.ietf.org/html/rfc8428#section-6.

CBOR representation for measurement index field

ThingsML has an integer CBOR representation for the label of the measurement index field. Officially, only default SenML labels should have an integer as CBOR representation. But this would save us 2 extra bytes per measurement in the SenML record, giving is more compactness. Since the measurement index is a regular field, the CBOR representation should be a positive integer.

Name

Label

CBOR Label

Measurement index

Example 1: fully encodable measurements

The example below shows a measurement list in both SenML and ThingsML in CBOR hexadecimal representation and CBOR diagnostic notation. The hexadecimal representation could be copy-pasted into the CBOR tool at http://cbor.me.

SenML

142 bytes

ThingsML

66 bytes (-55%)

85a300686c6174697475646502fb4049f9ed288ce70401636c6174a3 00696c6f6e67697475646502fb40155165d3996fa801636c6f6ea300 6b74656d706572617475726502fb403dc00000000000016343656ca3 006e62617474657279566f6c7461676502fb400cb851eb851eb80161 56a3006b74656d706572617475726502fb40313851eb851eb8016343 656c

85a202fb4049f9ed288ce7041735a202fb40155165d3996fa81734a2 02fb403dc000000000001737a202fb400cb851eb851eb81706a202fb 40313851eb851eb81737

[
  {0: "latitude", 2: 51.95255, 1: "lat"},
  {0: "longitude", 2: 5.32949, 1: "lon"},
  {0: "temperature", 2: 29.75, 1: "Cel"},
  {0: "batteryVoltage", 2: 3.59, 1: "V"},
  {0: "temperature", 2: 17.22, 1: "Cel"}
]

[
  {2: 51.95255, 23: -22},
  {2: 5.32949, 23: -21},
  {2: 29.75, 23: -24},
  {2: 3.59, 23: 6},
  {2: 17.22, 23: -24}
]

This example encodes all decimal numbers as double point float representations. Further payload size optimization can be achieved by using single point or half point float representations.

Example 2: some custom measurements

SenML

187 bytes

ThingsML

128 bytes (-30%)

87a221782075726e3a6465763a4445564555493a3031323334353637 38393031323334353a221a5cb35887a300686c6174697475646502fb 4049f9ed288ce70401636c6174a300696c6f6e67697475646502fb40 155165d3996fa801636c6f6ea3006b74656d706572617475726502fb 403dc00000000000016343656ca3006e62617474657279566f6c7461 676502fb400cb851eb851eb8016156a3006765787454656d7002fb40 313851eb851eb8016343656ca20062696f04f4

87a221782075726e3a6465763a4445564555493a3031323334353637 38393031323334353a221a5cb35887a202fb4049f9ed288ce7041735 a202fb40155165d3996fa81734a202fb403dc000000000001737a202 fb400cb851eb851eb81706a3006765787454656d7002fb40313851eb 851eb8016343656ca20062696f04f4

[
  {-2: "urn:dev:DEVEUI:0123456789012345:", -3: 1555257479},
  {0: "latitude", 2: 51.95255, 1: "lat"},
  {0: "longitude", 2: 5.32949, 1: "lon"},
  {0: "temperature", 2: 29.75, 1: "Cel"},
  {0: "batteryVoltage", 2: 3.59, 1: "V"},
  {0: "extTemp", 2: 17.22, 1: "Cel"},
  {0: "io", 4: false}
]

[
  {-2: "urn:dev:DEVEUI:0123456789012345:", -3: 1555257479},
  {2: 51.95255, 23: -22},
  {2: 5.32949, 23: -21},
  {2: 29.75, 23: -24},
  {2: 3.59, 23: 6},
  {0: "extTemp", 2: 17.22, 1: "Cel"},
  {0: "io", 4: false}
]

ThingsML for LoRa

ThingsML for LoRa follows the following additional aspects:

ThingsML for LoRa only supports CBOR data representation, since this is the most compact way of sending ThingsML.
You may omit the base name and base time values from your measurement. The LoRaWAN network layer already provides this information to KPN Things, so your device doesn't have to.

Measurement index list

This section gives the measurement index list and the way it is moderated. This list can be found on the common measurements page.

Request for change

If you have a measurement name and unit you think should be in the ThingsML measurement index list? Please contact the author for a request for change. Proposed records will be added if they are generic and unique in the list. For instance Temperature in Fahrenheit shall not be added to the list, since translation to Celsius can be trivially implemented in the device.

Custom measurement index list

We plan to support use case specific measurement tables, enabling you to define your own measurement index list.

Changelog

Date	Description
26-11-2019	First version
8-1-2020	Updated text
27-1-2020	Added 3 measurements to the index list Added change log table
7-4-2020	Specification released publicly
18-11-2020	Made introduction and ThingsML for LoRa more clear.

Specification by Paul Marcelis. Email: pjmarcelis@gmail.com

Last updated 1 month ago