Data Stream Designer

⌘K
  2. Home
  4. Docs
  6. Data Stream Designer
  8. Concepts
  10. Collections and Stream Hosts

Collections and Stream Hosts

It is recommended that you read this documentation along with the articles listed below to improve your understanding in how stream hosts and collections work.

Overview
Stream Hosts

A stream host is an application that can either be installed as a Windows Service or as a Console Application. Stream hosts enable data streams to run and execute actions and are also responsible for getting the configurations of non-virtual agents. In other words, the stream host application needs to be running for agents in a data stream to be able to process the data, according to the design of the stream.

Both virtual and non-virtual agents essentially run on a stream host, but how they are handled at design-time differs greatly. Non-virtual agents require a stream host to be online for Data Stream Designer to be able to get their configurations, but virtual agents don’t need a stream host to be running.

Collections

Stream hosts are grouped into different collections, which are created and maintained in Data Stream Designer. A collection can be defined as a category that contains a set of stream hosts that run the same data streams. Each stream host is associated with a collection by keeping the ID of the collection in the appsettings.json file of the stream host, which can be found in the location where the stream host has been installed. Thus, all the stream hosts that have the same collection ID stored in this file, will automatically fall under the collection that has that ID.

After a collection has been created within Data Stream Designer, you will be able to associate a data stream with a default collection. Unless changes are made for individual agents in your data stream, each agent will use this collection to perform the actions it was designed to perform. However, Data Stream Designer allows you to build a stream in which some agents use one collection and other agents use a different collection. For more information on how to configure this, click here.

Along with the collection ID, the appsettings.json file contains other configurations for the stream host. Each of these configurations are listed and described in the table below.

Property Description
ID Uniquely identifies the stream host.
Name Name of the stream host.
Collection ID ID of the collection associated with the stream host.
Rank Indicator of the preference given to the stream host. This especially applies when more than one stream host with the same collection ID is running. The stream host with the lowest rank will be used to get the configurations of non-virtual agents. The rank has a default value of 0. See stream host rank.
Secret Used to verify the connection between the stream host and the collection. This field needs to correspond to the Key that is stored for a collection in Data Stream Designer. If this key is revoked/replaced, all stream hosts will be disconnected.
Server URL URL that the stream host needs to use to connect to Data Stream Designer.
Cryptography key Key that the stream host will use to encrypt or decrypt secure user settings, for example, a SQL Server password.

The table below contains a list of the configurations stored for a collection along with a description for each. This information can be found by opening the Collections page from the left-hand menu in Data Stream Designer and selecting your collection from the list. Please note that the only field you will not be able to change, is the ID of the collection.

Property Description
ID Uniquely identifies the collection.
Key Used to verify the connection between the host and Data Stream Designer. If this key is revoked/replaced, all stream hosts will be disconnected.
Name Name of the collection.
Remote Receiver See remote receivers and publishers.
Remote Publisher See remote receivers and publishers.
Metadata Tags that have been added to the collection.

On this page

Overview
Remote receivers and publishers
Connection profiles
Stream host installation
Stream host rank
Usage: Create a new collection
Usage: Replace a collection key
Usage: Delete a collection

Remote Receivers and Publishers

The remote receiver and remote publisher properties are used to bridge the gap between agents in a stream that are associated with different collections, for example:

Consider having a stream that has two agents:

  • Listener_A, which is associated with Collection_A, and
  • Action_Agent_B, which is associated with Collection_B

Listener_A needs to communicate to Action_Agent_B, but they belong to different collections. As a result, the engine will automatically use the remote receiver and publisher to pass data from one collection to another.

Please note that only MQTT is currently being supported for this functionality

Example:

Consider having a stream that has the following agents:

  1. Event Simulator
  2. Event Printer

For the purpose of this example, the Event Simulator will be named ES_1 and the Event Printer EP_1, where ES_1 is using Collection_A and EP_1 is using Collection_B. Since these two agents are using two different collections, data flow from ES_1 to EP_1 is not possible as there is no physical connection (the stream hosts in these collections can be on different machines in different parts of the world).

To solve this problem and allow data flow from ES_1 to EP_1, the collection EP_1 is using needs to be set up as follows:

  • The Remote Receiver for Collection_B needs to be set to MQTT and the broker address needs to be specified.
  • The Remote Publisher for Collection_B needs to be set to MQTT and the broker address needs to be specified.

When the stream is published now, the data simulated by ES_1 will be published to MQTT, which will enable the engine to get the data EP_1 expects by listening for it on MQTT. Thus, data flow between ES_1 and EP_1 has been established.

The below image is an example of the configuration for Collection_B.

Connection Profiles
Overview

A connection profile is a file, containing certain details about a collection, that can be downloaded from Data Stream Designer. The purpose of this file is to make the installation process of stream hosts easier by providing you with an option to upload this file to the stream host installer instead of manually specifying the collection information. This file contains the values described in the table below.

Property Description
Device Name Name of the stream host. This name will be the same as the device name that needs to be specified when downloading a collection profile. If you install the stream host as a Windows Service, the service name and device name will be the same.
Collection ID The ID of the collection the stream host will be associated with.
Secret Used to verify the connection between the stream host and the collection. If this key is revoked/replaced, all stream hosts will be disconnected.
Server URL URL that the stream host needs to use to connect to Data Stream Designer. This is the URL of the instance of Data Stream Designer that has been installed on the server.
Key Key used to encrypt or decrypt user settings.
Download a connection profile

To download an installation profile, follow the steps below:

  1. Open the Collections page from the left-hand menu.
  2. Select the collection you would like to install the stream host for.
  3. Click Connection Profile.
  4. Enter a Device Name.
  5. Click OK. The download will start automatically.
Stream Host Installation

To install a stream host, please refer to these instructions.

Stream Host Rank

All the stream hosts contained in a collection will be used to run the data streams associated with that collection. However, you can choose which stream host should be responsible for getting the configurations of non-virtual agents. The rank given to a stream host defines what preference the stream host will be given in such a scenario. The stream host with the lowest rank will be picked by the system. All stream hosts will always have a default rank value of 0.

Example

Consider having a data stream with the following agents:

  • MQTT listener
  • Filter transformation
  • SQL Server Writer

The purpose of this data stream is to get temperature readings from a sensor on a machine, filter the readings that are higher than 120oC and write these values to a table in a SQL Server database. All three agents are configured to use stream hosts in the “Mine 09” collection to run (see image below).

However, to get the user settings for the SQL Server Writer action agent, which is a non-virtual agent, Data Stream Designer will pick and use the first stream host available. This can be changed by setting the rank of the stream host you want to use to be lower than the other stream hosts that are available.

Usage: Create a new collection

Before a data stream can be created, a collection needs to be available for use. To create a collection, follow the steps below:

  1. Open the Collections page from the left-hand menu.
  2. Click on New.
  3. Choose a name for the new collection.
  4. You may also include tags if needed by choosing tags from the drop down that will open as soon as you click on the field or by typing a new value and pressing Enter.
  5. Click Ok.
Usage: Replace a collection key

It might be needed to replace or change the key that is associated with a collection from time to time. To replace a key with a new one, follow the steps below:

  1. Open the Collections page from the left-hand menu.
  2. Select the collection from the list.
  3. Click on Revoke Key.
  4. Confirm that you would like to revoke the key. After clicking Revoke, the key will be replaced with a new key.
As soon as the key for a collection is revoked, all the stream hosts in that collection will be disconnected.
Usage: Delete a collection

To remove a collection, follow the steps below:

  1. Open the Collections page from the left-hand menu.
  2. Select the collection you would like to remove.
  3. Click on the Delete button.
  4. Confirm that you would like to delete the selected collection.