Information Management Design

by

The design of the new Device and Thing information model you design is great. It off-loads the application layer a lot of code managing the aggregation of devices.  – Watson IoT Platform sponsor.
Getting Started with Watson IoT Platform Data Management
DevZone Quick Lab at Think 2018 on Data Management.

In 2016 and 2017 I contributed to the design concepts and UX design for Information Management in the Watson IoT Platform.

What is Information Management

Watson IoT Platform Information Management provides an abstract model of identifiable assets that will allow IoT applications to be decoupled from the complexities of how specific devices are connected. Devices and logical composites often form a complex hierarchical relationship between maintainable assets & connected sensors and actuation devices. By providing a definition of a state, type schemas for data, abstract interfaces to state and behaviour and transformation of raw event data as part of the IoT Platform service the application layer benefits from decoupling of the instrumentation of the real-world and the management of the information model.

User Research

In our research, we observe that IoT solutions have

  • Multiple devices and sensors instrumenting connected assets
  • Devices integrated into an IoT solution come from an ecosystem of manufacturers
  • The message format defined by devices are often unique to the device type. Limited data standardisation on formats and units exists. 
  • Devices form complex hierarchical relationship between the assets and its sensors and actuation devices. 
  • Device relations may be static by configuration, or dynamic by condition.
  • Asset state depends on the conditional state of the related devices. 

An IoT solution developer needs access to an abstract model of identified assets that will allow IoT applications to be decoupled from the complexities of how specific devices are connected.

Our research uncovers that organizations need to

  • Improve productivity and speed of delivery on customer IoT deployment projects
  • Bring down development cost due to impact from variability across device models and variants
  • Reduce custom code required to create and manage Things on top of devices and their events and commands
  • Reduce steps to configuration and deploy a customer IoT system of devices

Organizations also need

  • A simple programming model and an abstract interface model for devices and things to insulate from variability
  • The platform to take the load off developers and relationships, behavior, state, history, simple transformations, complex enrichment, and access control into a consistent Things design
  • A DEVOPS practice to run and manage the information model processing pipeline

Decomposing the needs for actions on data in the information management processing pipeline we find the following

  • Parsing – Devices send data in various formats, like text, binary, encrypted formats. Binary formats are used to reduce transmission costs. Applications need data in JSON format.
  • Normalise – Devices come in various models and versions. Each model uses its specific data format and schema. Applications need to interact with them all in the same way.
  • Transform – Devices use various data units. For example, temperatures may be sent in Fahrenheit, Celsius, or Kelvin degrees. Applications need to interact with using one defined unit.
  • Filter – Only data that are relevant should be exposed. For example, only average data per hour may be required.
  • Enrich – Device data may be combined with from some other external source. For example weather at the device location.
  • Abstract – Devices operator in an event mode sending continuous data updates. My programming model may prefer a REST-based state model or an change based event-model.

Hills

Device Abstractions – Donna the information operator can manage the complexities of an IoT ecosystem while keeping the application insulated from data change

Aggregation into Things – Chris the developer can reference and work with manageable assets without being exposed to the individual instrumentation of them

Design Concepts

IoT Platform Information Management ingest, transform and aggregate data from your IoT devices, diverse data sources and platforms into asset-based data structures.

  • Information Management manages reusable abstractions of devices and aggregations of things.
  • Information Management provides an event-based processing pipeline to transform, enrich and aggregate the state of devices and things.
  • Information Management provides APIs and User Interfaces to the manage the information model and interact with the state model of device and thing instances

Devices connect to the IoT Platform and submit events with device data. The information model declares schemas for the event types to specify the payload data types. The model also transforms the physical device events to a logical property model with readable property names, JSON data types, REST APIs to property values, state change events.

im1

Simple information mode with a TEMPERATURE device sending “t:37.5” MQTT messages that get exposed to Watson IoT Platform Platform consumers as property “temperature” of type Number with value 37.5 C.

The information management model also provides an abstraction of device interfaces. Each Application Interface interface declares properties mapped to the physical interface. Abstract interfaces may be used to implement common standards normalising the interface across sets of similar device types. Application Interfaces may be used to manage versions of device interfaces, hence adding new properties and deprecating old properties in a new interface. Application Interfaces may also implement open standards. 

im2

Binary messages from the LED light is transformed using DFDL to JSON.
The Application Interfaces are mapping to the event types and type schemas in the Physical Interface. 

A pain point called out by our sponsors are the effort put on the application layer to manage, and keep up with, changes and additions to the set of devcie types. Device types may be declaring different event types and applications needs to be updated when new device types are added to the solution. 

With the Information Management model devices may now be mapped from single abstract Application Interfaces to the specific Physical Interfaces individually for each devcie type, hence making the application layer independent of the variability across physical devices.  

im3

Any LED devcie type supports the colour property by mapping to the specific event type on each device type.  

The concepts can be summarised as

  1. Model custom device data events and provide an information model
  2. Multiple reusable abstract application interfaces to insulate applications
    from variability across device types, sensor models, variants and versions
  3. Use abstract application interfaces to insulate applications from variability across device types, sensor models, variants and versions

UX design

The use of Information Management in Watson IoT Platform is an optional capability.

  • Operators can view raw event data in JSON format sent by devices to the IoT platform 
  • Developers can build IoT applications that depend on the custom JSON message payload retrieved from the last event cache using the IoT platform APIs

However, using Information Management in Watson IoT Platform developers can get the benefits of parsing, transforming, normalising, enriching and abstracting data using the Information Management data processing pipeline. The Operator or Developer configures Information Management using the Interface section on a Device Type.

InformationManagement02

The user experience with the Information Management user experience is separated into two modes; a simple flow and an advanced flow.

The user experience with the Information Management user experience is separated into two modes; a simple flow and an advanced flow. The simple flow uses a simple interface editor and lets users select and filter the data properties that represent the device state. The advanced flow uses an advanced interface editor and provides richer capabilities to define an event type schema. Developers can also perform filtering and transformation of event data to multiple reusable interfaces that normalise device state across multiple device types.

Designing a Simple User Experience for Device State Properties

The simple interface editor appeals to early adaptors getting started with IoT, Devices and Device State. Its essential to quickly get started, connect devices, identify the properties exposed by the devices and make them available to other capabilities in the IoT platform. The simple model allows users to look at incoming events, select the data that is of interest, and create readable name and data types for the exposed properties. 

The simple interface editor user experience lets users view the message payload a device event. The user then selects the properties that should be exposed and chooses a data type for the data property.

The Information Management processing pipeline will receive the event, filter the data, and expose the state properties. The state is exposed in the IoT platform dashboard and accessible through the IoT platform APIs.

Using the simple interface editor to expose a temperature property “t” from the TemperatureSensor device.

Designing an Advanced User Experience using Physical and Logical Interfaces

The advanced interface editor exposed a richer set of capability and flexibility in defining the information model. Two additional concepts are exposed to the user in the user experience; physical interfaces and logical interfaces.

A physical interface is used to model the interface between a physical device and Watson IoT Platform. Event types can be associated with physical interface. The physical interface editor has been designed to support multiple ways of defining an event type schema. Developers can

  • Let the IoT platform listen for events from devices and reverse engineer an event type schema from the event payload.
  • Manually edit an event type schema by adding event properties
  • Import an existing event type schema from a file

Using the simple interface editor to define a ‘t’ : Number temperature state property for the TemperatureSensor device type

A logical interface is used to define the normalized view onto the device state in Watson IoT Platform and is usually consumed by applications. A logical interface must be associated with a logical interface schema that defines the state properties. The state is updated in response to inbound device events. The advanced interface editor provides a guided user experience in creating a logical interface.

As logical interfaces are highly reusable across device types is recommended to use a interface name that states the purpose of the abstraction, rather than the name of the device type. For example, its recommended to use names like ‘iTemperature’ and ‘iHumidity’ for logical interfaces than ‘TiSenstorTag-Temperature’ that makes it less reusable across an ecosystem of temperature sensor products.  The advanced editor also exposes the library of schemas and interfaces for reuse.

A state property in a logical interface is bound to event properties using a mapping expression. Such a mapping expression may be as simple as just referencing the event property. It may also be as complex as a JSONata expression that computes a typed value using event properties, other state property values, constant values. JSONata expression provides arithmetic functions, string, array and time functions and other utilities.

Using the advanced interface editor to define a ‘iTemperature’ logical interface with an exposed temperature state property “Temperature” from a TiSensorTag device.

Designing an Advanced User Experience for Multiple Logical Interfaces

Developers using the advanced interface editor can add multiple logical interfaces to a device type. Such added interfaces can be new definitions, or reused interfaces selected from the Interface Library. A shared logical interface provides a fixed set of state properties, and a new set of mapping expressions that binds the fixed state properties to the event properties defined in the physical interface of the device type.

Using the advanced interface editor to define a second ‘iHumidity’ logical interface with an exposed temperature state property “Humidity” from a TiSensorTag device.

Designing an Advanced User Experience for Activating Draft Interfaces

The advanced interface editor will save changes to physical and logical interfaces as Draft versions of the resources. A draft resource needs to be Activated to be added to the Information Management processing pipeline. The design of the advanced interface editor clearly indicates the state of an interface resource. The time stamp of the activation (deployment) is shown in the user interface.

InformationManagementLogicalInterfaces06

Updating a resource will create a draft version.

InformationManagement04

Changes to interface resources have to be activated to be deployed into the Information Management processing pipeline.

Designing an Advanced User Experience for Mapping Expressions

A state property in a logical interface is bound to event properties using a mapping expression. Such a mapping expression may be as simple as just referencing the event property. It may also be as complex as a JSONata expression that computes a typed value using event properties, other state property values, constant values. JSONata expression provides arithmetic functions, string, array and time functions and other utilities.

The expression editor user experience design provides two editor modes; an expression builder and a code editor mode. The expression builder provides a graphical interface for selecting expression elements like payload properties, values, arithmetical operators. The code editor design provides a text editor for JSONata expressions. Expressions use keywords like $event to reference physical interface event properties by name, or $state to reference logical interface state properties by name. For example, “= $event.d.ambientTemp” to reference the ambient temperature event property in the TiSensorTag status event.

Using the Expression Editor to bind state properties to event properties.

Note, a state property may be an object of values. In the example above a Temperature property exposes the Celsius value. The state object exposed computed temperature values converted to Fahrenheit, Celsius and Kelvin using mapping expressions all bound to the TiSensorTag ambient temperature value.

Related Designs

New designs for Information Management are currently developed

  • Design for Things Types and Thing Instance user experience
  • Design for an Interface Library user experience
  • Design for new Types vs Instance navigation user experience
Getting Started with Watson IoT Platform Data Management
DevZone Quick Lab at Think 2018 on Data Management.
PageLines