Skip to content

Data Transformation Using Ruby

Jump to bottom
Dave Wickelhaus edited this page Dec 15, 2024 · 1 revision

Agent 2.0.0.2 release introduced an extensible plug-in architecture for incoming and outgoing protocols; and data transformations.

Data Transformation Pipeline

The MTConnect Agent uses a data transformation pipeline to provide a flexible and mutable mechanism for processing incoming data from various sources and allowing for reusability of common transform components. Learn more about the architecture at MTConnect Agent Pipeline Architecture.

RubyTransform and Useful Methods

Constructor Method

A Transform defined for a SHDR pipeline MUST have the transform's name and MAY have a filter. See the constructor method in the Fix Execution example.

  • filter can be a specific type of Entity only to which the transform shall be applied. Examples of common filters:
    • :Event
    • :Sample
    • :Condition
classFixExecution < MTConnect::RubyTransformdefinitialize(name,filter)super(name,filter)endend

On the other hand, a Transform defined for an MQTT pipeline MUST additionally have a Guard defined. Learn more about Guardhere.

  • A guardMUST always return one of :RUN, :CONTINUE or :SKIP.

  • See the MQTT examples at MQTT Pipeline.

classMapMqttData < MTConnect::RubyTransformdefinitializesuper("MapMqttData")guard=lambda{ |e|
ife.name == "JsonMessage"return:RUNelsereturn:CONTINUEend}endend

Transform Method

Typically transformation is performed on an Entity in the pipeline. For example, the transform method as seen in the "fix execution" example takes an Observation Entity, obs, as an argument.

  • Name of an entity: entity.name

  • Properties of an Entity: entity.properties

    • Example: Get dataItemId of Observation: observation.properties[:dataItemId]
    • Properties defined for any entity type are same as defined by the standard.

  • Value of an Entity: entity.value

deftransform(entity)puts"*** received #{entity.name} with value: #{entity.value}"end
  • Learn more about different types of entities used in the pipelines here.

MTConnect device metadata can be accessed as shown:

deftransform(obs)dataItemId=obs.properties[:dataItemId]device=MTConnect.agent.default_device# Device Metadatadataitem_of_obs=device.data_item(dataItemId)end
  • In case of multiple devices, devices can be used instead of default_device. See Ruby Agent.

Creating a new Observation:

# obs is the incoming observation# {"dataItemId":"execution","timestamp":"2023-01-11T21:36:06.371Z","value":"IDLE"}device=MTConnect.agent.default_devicedataitem_of_obs=device.data_item[obs.properties[:dataItemId]]new_obs=MTConnect::Event.new(dataitem_of_obs,'NOT_READY')

Creating a new Observation with the timestamp of the Observation to be transformed:

# obs is the incoming observationdevice=MTConnect.agent.default_devicedataitem_of_obs=device.data_item[obs.properties[:dataItemId]]new_obs=MTConnect::Observation.new(dataitem_for_obs,'<transformed_value>',obs.properties[:timestamp])# For current timestamp: Time.now
  • Similarly MTConnect::Event, MTConnect::Sample and MTConnect::Condition can be created. See examples.

Forwarding an Entity:

  • New, transformed or old Entity may be passed on to the pipeline by using: forward(obs).

Splicing the Pipeline

Accessing the pipelines of all the data sources:

MTConnect.agent.sources.eachdo |s|
pipe=s.pipeline

Splicing the pipeline:

transform=YourTransform.new('YourTransform',:Event)# filtering only Eventspipe.splice_before('DeliverObservation',transform)# See the note below
  • See Pipelines to understand different methods defined to modify the pipeline.

Configuration

For data transformation using Ruby, add the path to the Ruby module in the agent config file as shown below:

 Ruby {
module = path/to/module.rb
}

The module specified at the path given will be loaded.

The current functionality is limited to the pipeline transformations from the adapters. Future changes will include adding sources and sinks.

Following examples will elucidate how to write a Ruby Transform module.

Examples

Template

# You may replace the name of the class <UseCaseName> with the custom use case at hand.classUseCaseName < MTConnect::RubyTransform# Constructor methoddefinitialize(name,filter)super(name,filter)end# Tranformation methoddeftransform(obs)# Transformation code goes here.# Please see examples listed below.forward(obs)endend# Splicing the pipeline of each data source for transformationMTConnect.agent.sources.eachdo |s|
pipe=s.pipelineputs"Splicing the pipeline"# The arguments may differ depending upon the initialization. See examples below to see howtrans=UseCaseName.new('UseCaseName',:Entity)# The method called to modify the pipeline may differ depending upon the usecase. See examples below to see howpipe.splice_before('DeliverObservation',trans)end

Fix Execution state of a device

An example of when a transformation can be done is when an adapter incorrectly outputs Execution state of NOT_READY as IDLE, and of WAIT as WAITING instead. IDLE and WAITING are not MTConnect semantics. Hence can be transformed to NOT_READY and WAIT as shown below.

classFixExecution < MTConnect::RubyTransformdefinitialize(name,filter)@cache=Hash.newsuper(name,filter)end@@count=0deftransform(obs)@@count += 1if@@count % 10000 == 0puts"---------------------------"puts"> #{ObjectSpace.count_objects}"puts"---------------------------"end# Get dataItemId of the observationdataItemId=obs.properties[:dataItemId]# check if the dataitemId is that of `Execution` observationifdataItemId == 'execution'# get the value of `Execution` observation@cache[dataItemId]=obs.value# get the device infodevice=MTConnect.agent.default_device# get the `Execution` dataitem from the deviceexecution=device.data_item(dataItemId)# Using case statement to create and forward transformed valuescase@cache[dataItemId]when'IDLE'# creating and forwarding new observation with value NOT_READY isntead of IDLEnewobs=MTConnect::Observation.new(execution,'NOT_READY')forward(newobs)when'WAITING'newobs=MTConnect::Observation.new(execution,'WAIT')forward(newobs)else# Forwarding original Execution observations only if no transformation requiredforward(obs)endelse# Forwarding observations that are not Executionforward(obs)endendendMTConnect.agent.sources.eachdo |s|
pipe=s.pipelineputs"Splicing the pipeline"# Updated the dataitem type to :Eventtrans=FixExecution.new('FixExecution',:Event)putstranspipe.splice_before('DeliverObservation',trans)end

MQTT Pipeline: Fix Execution state of a Device

When using the mruby embedded language, one can write dynamic scripted transformation to support quick corrections or protocol transformations from JSON representations via MQTT.

An example of a ruby transform takes some data with the topic data and converts 1 to READY and 2 to ACTIVE. The transform is added as the first transform after the Start (the first transform).

classMapMqttData < MTConnect::RubyTransformdefinitializesuper("MapMqttData")guard=lambda{ |e|
pe.topicife.topic =~ /^\/data/return:RUNelsereturn:CONTINUEend}enddeftransform(entity)puts"*** received #{entity.name} with value: #{entity.value}"value="UNAVAILABLE"caseentity.valuewhen"1"value="READY"when"2"value="ACTIVE"endputs"**** Setting execution to #{value}"puts"Creating timestamped"obs=MTConnect::Timestamped.new("Timestamped",{VALUE: value})obs.timestamp=Time.nowobs.tokens=["execution",value]forward(obs)endendMTConnect.agent.sources.eachdo |s|
ifs.name =~ /^mqtt/MTConnect::Logger.info"Splcing token mapper for #{s.name}"pipe=s.pipelinetrans=MapMqttData.new()pipe.first_after("Start",trans)mapper,=pipe.find("ShdrTokenMapper")trans.bind(mapper)endend

The second example is interprestation of MQTT data. This replaces the dummy JsonMapper and the guard runs only on JsonMessages. The data in Json format is easily converted to a ruby Hash by just evaluating it.

classMapMqttData < MTConnect::RubyTransformdefinitializesuper("MapMqttData")guard=lambda{ |e|
ife.name == "JsonMessage"return:RUNelsereturn:CONTINUEend}enddeftransform(entity)# {"dataItemId":"execution","timestamp":"2023-01-11T21:36:06.371Z","value":"STOPPED"}puts"*** received #{entity.name} with value: #{entity.value}"data=evalentity.valuepdatadata_item=MTConnect.agent.default_device.data_item(data[:dataItemId])if !data_itemMTConnect::Logger.warning"cannot find data item for #{data[:dataItemId]}"returnnilendcat=data_item['category']puts"DataItem category: #{cat}"obs=nilcasecatwhen'EVENT'obs=MTConnect::Event.new(data_item,{"VALUE"=>data[:value]},Time.now)# data[:timestamp])when'SAMPLE'obs=MTConnect::Sample.new(item,{"VALUE"=>data[:value]},data[:timestamp])elseMTConnect::Logger.warning"Not doing conditions at the moment"endifobsforward(obs)elsenilendendendMTConnect.agent.sources.eachdo |s|
ifs.name =~ /^mqtt/MTConnect::Logger.info"Splcing token mapper for #{s.name}"pipe=s.pipelinetrans=MapMqttData.new()pipe.replace("JsonMapper",trans)endend

Clone this wiki locally