# Geneos and Open Telemetry Best Practise

**URL:** <https://community.itrsgroup.com/t/geneos-and-open-telemetry-best-practise/284>\
**Category:** Geneos\
**Tags:** opentelemetry, otel, collection-agent\
**Created:** [23 May 2024 16:03 UTC](https://community.itrsgroup.com/t/geneos-and-open-telemetry-best-practise/284 "2024-05-23T16:03:15Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![pgalbavy](https://dub1.discourse-cdn.com/flex013/user_avatar/community.itrsgroup.com/pgalbavy/32/15_2.png) [@pgalbavy](https://community.itrsgroup.com/u/pgalbavy)\
**Post date:** [23 May 2024 16:03 UTC](https://community.itrsgroup.com/t/geneos-and-open-telemetry-best-practise/284/1 "2024-05-23T16:03:15Z")

</div>

# Geneos and Open Telemetry Best Practise

## Introduction

The examples in this document are intended to form the initial basis for best practise advice to Geneos administrators on how to integrate with Open Telemetry instrumented applications.

[Open Telemetry](https://opentelemetry.io/docs/) is a widely used, vendor-neutral, open-source framework for instrumenting and reporting telemetry data in the form of signals, which include traces, metrics and logs.

We will use the common short form OTel to refer to Open Telemetry.

## Audience

These examples are intended for experienced Geneos administrators who have been tasked with integrating Open Telemetry instrumented applications into their existing Geneos estates.

If you are, rather, experienced with OTel and have been asked to integrate it with Geneos, with which you may not have extensive experience, then we suggest the following introductory documents first:

- [Geneos Architecture](https://docs.itrsgroup.com/docs/geneos/current/getting-started/architecture/index.html)
- [Introduction to Collection Agent](https://docs.itrsgroup.com/docs/geneos/current/collection/collection-agent/intro-to-ca/index.html)

## Assumptions

You will have new or existing Open Telemetry instrumented applications and we cannot expect to change how this instrumentation is configured. The best we can hope for is that Open Telemetry common/best practises have been followed.

As a part of this we are going to assume that the practises in the OTel demo environment are considered as good practise: [https://opentelemetry.io/docs/demo/](https://opentelemetry.io/docs/demo/)

## Geneos Overview

As an experienced Geneos user you may still be new to the Collection Agent, which is distributed as part of the Netprobe, and is typically managed as a separate process via the Netprobe configuration. If you need to know more please take a look at [Introduction to Collection Agent](https://docs.itrsgroup.com/docs/geneos/current/collection/collection-agent/intro-to-ca/index.html).

## Open Telemetry Overview

A good place to start with OTel is to work through their documentation starting with the [Observability Primer](https://opentelemetry.io/docs/concepts/observability-primer/)

Their [demo environment](https://opentelemetry.io/docs/demo/) exercises the various signal types and language support for OTel as well as providing a good set of UIs to see what is happening in real time.

## Challenges

OTel places a lot of emphasis on distributed tracing, while Geneos is more focused on exception monitoring. At the time of writing Geneos, through the Collection Agent, only supports Metrics and Logs fully and extracts only timing metrics from root spans in traces.

## Approach

We are going to start with the OTel demo environment and work through, step-by-step, integrating it with Geneos. If your OTel instrumented application follows the patterns used by the OTel demo environment then many of the examples will work with minimal changes.

## Let’s Get Going

### Geneos Setup

First, let’s make sure there is a suitable Geneos set-up to run an up-to-date Collection Agent. All of the examples we use were tested with release 6.7.1 of both Gateway and Netprobe. You can use an existing Gateway and Netprobe if you choose, or for testing you can create new ones (this assumes you use [`geneos`](https://github.com/ITRS-Group/cordial/tree/main/tools/geneos) from the `cordial` toolset):

```bash
$ geneos deploy gateway otel-test -S
certificate created for gateway "otel-test" (expires 2025-04-10 00:00:00 +0000 UTC)
gateway "otel-test" added, port 7102
gateway "otel-test" started with PID 78905
$ geneos deploy netprobe otel-test -S
certificate created for netprobe "otel-test" (expires 2025-04-10 00:00:00 +0000 UTC)
netprobe "otel-test" added, port 7103
netprobe "otel-test" started with PID 79166

```

⚠ Please note that the above example also creates new certificates as the TLS subsystem has been previously initialised. All the example Gateway configurations assume secure connections between Geneos components. Please adjust accordingly if you are not using TLS. The OTLP gRPC port we use is not secured using TLS, but this can be done by using the certificate and key created for the Netprobe and adjusting the configurations accordingly.

### Installing the OTel Demo

Install the demo in your environment as per the documentation. We will use docker, so it’s a case of following the start of the OTel demo documentation for [Bring Your Own Backend](https://opentelemetry.io/docs/demo/docker-deployment/):

```bash
git clone https://github.com/open-telemetry/opentelemetry-demo.git
cd opentelemetry-demo

```

To ensure that everything will work you can start the demo at this point and try out the UIs before moving on to connect it to the Geneos Collection Agent. Follow the appropriate instructions in the above docs.

If you have any problems with the demo at this point please see the [Troubleshooting](#troubleshooting) section below and also the OTel demo documentation for further help.

### Direct OTel Signals to Collection Agent

Follow the instructions in [Bring Your Own Backend](https://opentelemetry.io/docs/demo/docker-deployment/) to add the Collection Agent OTel endpoint as an `oltpgrpc` URL. Remember to stop and restart the demo after making change to the file mentioned below.

First, add an `otlp` exporter (for OTLP over gRPC, as opposed to `otlphttp` for HTTP) and to give it a unique name add the `/ca` for Collection Agent. If you want to extend the set-up to use multiple Collection Agents then you can use other names, such a `/ca1` or `/caprimary` etc. just make sure you also adjust the other settings further on.

Edit `src/otelcollector/otelcol-config-extras.yml` in the opentelemetry-demo directory:

```yaml
exporters:
  otlp/ca:
    endpoint: "CAHOST:4317"
    tls:
      insecure: true

```

You should replace `CAHOST` with the name or IP address of the Collection Agent host.

Next, add a `service` section but remember to include the default exporters, as the array entries (those in `[...]`) overwrite the defaults, not append to them:

```yaml
service:
  pipelines:
    traces:
      exporters: [otlp, debug, spanmetrics, otlp/ca]
    metrics:
      exporters: [otlphttp/prometheus, debug, otlp/ca]
    logs:
      exporters: [opensearch, debug, otlp/ca]

```

As you can see above, the arrays for each exporter have `otlp/ca` appended. The other YAML settings do not have to be included, and are left unchanged.

### Configure Collection Agent and Gateway

You should now set-up the Netprobe that will run the Collection Agent and connect it to your Gateway.

⚠ Note that for the Collection Agent to correctly start-up and to report Self-Monitoring you have to set two environment variables:

- `JAVA_HOME` must point to the top level of the Java runtime you will use

- `HOSTNAME` must be the hostname of your system so that the self-monitoring can identify itself as a Dynamic Entity with that name

You can set both of these like this:

```bash
$ geneos set netprobe otel-test -e JAVA_HOME=/usr/lib/jvm/java-17-openjdk-amd64 -e HOSTNAME=myHostName
$ geneos restart netprobe otel-test
netprobe "otel-test" started with PID 98945

```

In the Gateway config set-up a Probe entry with the `Dynamic Entities` tab filled in. If you use your own names remember to also make changes to the other examples below.

⚠ As for all these XML configuration section, copy them using the icon in the top right (if visible) or manually selevct the XML and CTRL+C. Then, in the GSE select the parent configuration section and right click and select “Paste XML”

```xml
<probe name="otel-test">
    <hostname>localhost</hostname>
    <port>7036</port>
    <secure>true</secure>
    <dynamicEntities>
        <mappingType ref="otel-mapping-type"/>
        <collectionAgentParameters ref="otel-ca"/>
    </dynamicEntities>
</probe>

```

⚠ Update the `hostname` and `port` above to match the Netprobe you have either created or have already selected.

Now, under the `Dynamic Entities` section of the GSE you have to create one configuration in each of the four section:

- Mapping:

- Collectors:

- Mapping Types:

- Collection Agent Parameters:

### Start the Collection Agent

Start the Collection Agent (and Netprobe) and make sure that you see the Self-Monitoring data after about 30 seconds:

 ![Netprobe Info](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/ca7fd58a7c2791163624a78ead69a904190d9ed9.png)

⚠ Note in the latest versions of Geneos the `Entities` view only shows Dynamic Entities in a Netprobe that are in an errored state. Use the headline, as indicated above, to check that one Dynamic Entity has been accepted. You can change this behaviour to show a row-per-entity by overriding the default sampler used. See the [Dynamic Entities Health](https://docs.itrsgroup.com/docs/geneos/current/collection/dynamic-entities-health/index.html#Heading-introduction) plugin documentation for more.

You should now have a Managed Entity with the same name as your `HOSTNAME` environment variable set in the Netprobe environment:

 ![CA Self-Monitoring](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/bc19a6afe663d8428f328faf29d3dcecc4824319.png)

Once the OTel demo is connected and streaming to the Geneos Collection Agent then you will also see a large number of entries in `InvalidMetrics` and `InvalidStreams` under the Netprobe Info entity, like this:

 ![InvalidMetrics](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/ddf49a6205d4cfc1f853a3726da4faabd7103433.png)

 ![InvalidStreams](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/3d725142f991ed5f8d090ead53ae2548fc4a38b4.png)

This is not a bad thing as such, in fact it means that the OTel demo is successfully sending data to the Collection Agent. The next step is to start mapping the OTel signal to useable Geneos data.

### Mappings

Geneos supports [Dynamic Entities](https://docs.itrsgroup.com/docs/geneos/current/processing/configuration/dynamic_entities/index.html) through mappings that allow you to make statements about the meaning of the signals you receive in the Collection Agent and how those should be displayed in Geneos.

OTel has the concept of [semantic conventions](https://opentelemetry.io/docs/concepts/semantic-conventions/) to help ensure that signals reported by OTel can be used to add meaning to the collected data. The current specification is here: [https://opentelemetry.io/docs/specs/semconv/](https://opentelemetry.io/docs/specs/semconv/)

Let’s use these two things to build some simple Geneos configuration that can display signals from the OTel demo in a meaningful way.

You could use the `InvalidMetrics` and `InvalidStreams` Dataview to do this, but the Collection Agent logs are a much better place to start.

If you are using the latest `geneos` from the `cordial` repo, then you can do something like this:

```bash
geneos logs netprobe otel-test -C

```

This will output the last 10 lines from both the main Netprobe log file and also the Collection Agent log. You have to look at both log files as they each contain information you can use to create your mappings.

The main Netprobe logfile has entries like this:

```log
===> netprobe "otel-test" /home/peter/geneos/netprobe/netprobes/otel-test/netprobe.log <===
2024-04-09 10:29:50.727+0100 ERROR: DynamicEntities No mapper matches the metric 'grpc.oteldemo.CurrencyService/Convert_latency|host.arch=amd64|host.name=9f0167b27676|os.type=linux|os.version=6.5.0-21-generic|process.command=/app/server.js|process.command_args=/usr/local/bin/node_--require_./Instrumentation.js_/app/server.js|process.executable.name=node|process.executable.path=/usr/local/bin/node|process.owner=nextjs|process.pid=17|process.runtime.description=Node.js|process.runtime.name=nodejs|process.runtime.version=18.20.1|service.name=frontend'. It will be ignored
2024-04-09 10:29:50.727+0100 ERROR: DynamicEntities No mapper matches the metric 'grpc.oteldemo.ProductCatalogService/ListProducts_latency|host.arch=amd64|host.name=9f0167b27676|os.type=linux|os.version=6.5.0-21-generic|process.command=/app/server.js|process.command_args=/usr/local/bin/node_--require_./Instrumentation.js_/app/server.js|process.executable.name=node|process.executable.path=/usr/local/bin/node|process.owner=nextjs|process.pid=17|process.runtime.description=Node.js|process.runtime.name=nodejs|process.runtime.version=18.20.1|service.name=frontend'. It will be ignored
2024-04-09 10:30:28.849+0100 ERROR: DynamicEntities No mapper matches the metric 'process.runtime.dotnet.gc.committed_memory.size|container.id=bb51603d87178c21eeb07715c49f6ec375638972569f195341d0af2e250289cb|host.name=bb51603d8717|service.name=cartservice'. It will be ignored
2024-04-09 10:30:28.849+0100 ERROR: DynamicEntities No mapper matches the metric 'process.runtime.dotnet.gc.heap.size|container.id=bb51603d87178c21eeb07715c49f6ec375638972569f195341d0af2e250289cb|host.name=bb51603d8717|service.name=cartservice'. It will be ignored
2024-04-09 10:30:28.849+0100 ERROR: DynamicEntities No mapper matches the metric 'process.runtime.dotnet.gc.heap.fragmentation.size|container.id=bb51603d87178c21eeb07715c49f6ec375638972569f195341d0af2e250289cb|host.name=bb51603d8717|service.name=cartservice'. It will be ignored

```

As you scroll across the lines you will see some common names, which are using the OTel semantic conventions. The two that jump out most often are `host.name` and `service.name`.

It may be tempting to immediately use these values as an Entity and Dataview in Geneos, but this isn’t always the case. The `host.name` value is the docker container ID and is not very descriptive **and** more importantly is dynamic and will change from one run of the demo environment to the next. `service.name` may be a better candidate. Also, `service.namespace` makes a good Geneos Attribute.

Let’s try this as a new Collector:

```xml
<mapping name="otel-basic">
    <custom>
        <geneosItems>
            <geneosItem>
                <label>service.name</label>
                <entity>
                    <required>true</required>
                    <useInDisplayName>true</useInDisplayName>
                </entity>
            </geneosItem>
            <geneosItem>
                <label>service.namespace</label>
                <attribute/>
            </geneosItem>
        </geneosItems>
    </custom>
    <localAttributes>
        <localAttribute>
            <name>COMPONENT</name>
            <value>
                <data>Astronomy Shop</data>
            </value>
        </localAttribute>
    </localAttributes>
</mapping>

```

Add this new Collector to your Mapping Type, replacing the initial one above with:

```xml
<mappingType name="otel-mapping-type">
    <collectors>
        <collector ref="otel-collector"/>
    </collectors>
    <mappings>
        <mapping ref="ca-self-monitor"/>
        <mapping ref="otel-basic"/>
    </mappings>
</mappingType>

```

If you save this you should now start seeing new Managed Entities pop up after about 30 seconds. But wait! There’s something wrong. The Dataview names are very long and include too much unwanted information. Similarly, in each Dataview there is are one or two rows with a very convoluted names instead of two column of data, like we would expect.

This is because all of the semantic properties from OTel are being converted to Geneos dimensions (and some Properties).

What we want to do is to stop all the attributes, except for the ones we are interested in, from being turned into Dimensions. We do this in the `otel-collector` configuration from above; If you remember we left the YAML part empty? Now, let’s fill it in:

Paste this into the YAML part of the `otel-collector` configuration. Be careful with spacing as YAML is highly sensitive to indentation:

```yaml
resource-attributes:
  metrics:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  logs:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  traces:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id

data-attributes:
  metrics:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  logs:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  traces:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id

```

What this configuration does is to change the default behaviour of the `resource-attributes` from including all semantic names and the `data-attributes` from including all names to both only passing up the three attributes we have set. Everything else becomes a “Property”, which means that it is available to Geneos but not used for naming.

If you now save the configuration you should see useful data flowing through!

⚠ Note: Because of the flow of data during the configuration changes, you may see attributes other than those above in the Dataview and row names. To clean this out, restart the Collection Agent (and Netprobe):

```bash
geneos restart netprobe otel-test

```

 ![State Tree](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/b10de87b5552f063dfacd8356d11491a9a98f68c.png)

 ![adservice Dataview](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/7c5f0e8cce68c77d5d50cf860a52a17b1f91e74a.png)

 ![adservice histogram](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/4e49f5955e2d7ed70b843c5c09ad50cd6df01e29.png)

If you now right-click on a cell and run the `Show Dynamic Mappings` command you will see something like this:

 ![dynamic mappings](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/dee0ff74ff963631f364dd9cdd450b12b0c18201.png)

Here you can see that only two `Dimensions` has made it through and all the other OTel attributes are Properties. The Geneos Attributes include both the `service.name` and the `service.namespace`.

### Logs

The final piece of the puzzle is to be able to process the log entries being sent from OTel. To do this we need to add an FKM sampler to the `otel-mapping-type`.

First, we create the FKM sampler that subscribes to all streams, so that we can see what logs are being published:

```xml
<sampler name="logs">
    <var-group>
        <data>logs</data>
    </var-group>
    <plugin>
        <fkm>
            <files>
                <file>
                    <source>
                        <stream>
                            <data>*</data>
                        </stream>
                    </source>
                </file>
            </files>
        </fkm>
    </plugin>
</sampler>

```

Then, we add a reference to this sampler in `otel-mapping-type`, like this:

```xml
<mappingType name="otel-mapping-type">
    <collectors>
        <collector ref="otel-collector"/>
    </collectors>
    <mappings>
        <mapping ref="ca-self-monitor"/>
        <mapping ref="otel-basic"/>
    </mappings>
    <samplers>
        <sampler ref="logs"/>
    </samplers>
</mappingType>

```

If you now locate one of the Entities that contains a `logs` Dataview, you should be able to right-click on one of the streams data cells and run the `View File` command. If you select `Continuous` monitoring you will be able to watch the file stream updating in real-time. You can then begin using this to build FKM key tables to identify error conditions as normal. If you already know the likely log entries that need to be identified as error or exception conditions then you can go ahead and implement those too.

 ![watch file stream](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/4473805f8791c9f21d31aa10763c7893a3bb4392.png)

To use separate FKM samplers for different OTel log sources, so that you can use different FKM key tables, you can add multiple FKM samplers and as long as the stream name only matches the OTel log name then only that sampler will be applied to the log entries, e.g. setting up a new FKM sampler for `kafka.*` logs:

```xml

<sampler name="kafka-logs">
    <var-group>
        <data>logs</data>
    </var-group>
    <plugin>
        <fkm>
            <files>
                <file>
                    <source>
                        <stream>
                            <data>kafka.*</data>
                        </stream>
                    </source>
                </file>
            </files>
        </fkm>
    </plugin>
</sampler>

```

and adding this to the `otel-mapping-type` in addition to the catch-all `logs` sampler:

```xml

<mappingType name="otel-mapping-type">
    <collectors>
        <collector ref="otel-collector"/>
    </collectors>
    <mappings>
        <mapping ref="ca-self-monitor"/>
        <mapping ref="otel-basic"/>
    </mappings>
    <samplers>
        <sampler ref="logs"/>
        <sampler ref="kafka-logs"/>
    </samplers>
</mappingType>

```

Now a new sampler appears but only in those entities that publish logs with names beginning `kafka.`:

 ![kafka logs](https://europe1.discourse-cdn.com/flex013/uploads/itrsgroup/original/1X/a5ba52e42b0832e5438f7eb9600aebea7f173c78.png)

Unless you also change the original `logs` sampler, the same logs will continue to also appear there.

## Troubleshooting

- OTel demo in WSL with docker

- Missing Log Datapoints

- Other Errors in `collection-agent.log`

## Example Configuration

Below is a complete Gateway configuration include file with all of the above examples.

Please remember to change the Probe hostname and port and Secure flag to match your local instances.

```xml
<?xml version="1.0" encoding="ISO-8859-1"?>
<gateway compatibility="1" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="http://schema.itrsgroup.com/GA6.7.1-240223/gateway.xsd">
    <probes>
        <probe name="otel-test">
            <hostname>localhost</hostname>
            <port>7036</port>
            <secure>true</secure>
            <dynamicEntities>
                <mappingType ref="otel-mapping-type"/>
                <collectionAgentParameters ref="otel-ca"/>
            </dynamicEntities>
        </probe>
    </probes>
    <dynamicEntities>
        <simpleMappings>
            <mapping name="ca-self-monitor">
                <builtIn>
                    <name>Collection Agent V3</name>
                </builtIn>
            </mapping>
            <mapping name="otel-basic">
                <custom>
                    <geneosItems>
                        <geneosItem>
                            <label>service.name</label>
                            <entity>
                                <required>true</required>
                                <useInDisplayName>true</useInDisplayName>
                            </entity>
                        </geneosItem>
                        <geneosItem>
                            <label>service.namespace</label>
                            <attribute/>
                        </geneosItem>
                    </geneosItems>
                </custom>
                <localAttributes>
                    <localAttribute>
                        <name>COMPONENT</name>
                        <value>
                            <data>Astronomy Shop</data>
                        </value>
                    </localAttribute>
                </localAttributes>
            </mapping>
        </simpleMappings>
        <collectors>
            <collector name="otel-collector">
                <plugin>
                    <open-telemetry>
                        <OpenTelemetryCollector>
                            <yaml>
                                <data>resource-attributes:
  metrics:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  logs:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  traces:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id

data-attributes:
  metrics:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  logs:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
  traces:
    base: none
    include:
      - service.namespace
      - service.name
      - service.instance.id
</data>
                            </yaml>
                        </OpenTelemetryCollector>
                    </open-telemetry>
                </plugin>
            </collector>
        </collectors>
        <mappingTypes>
            <mappingType name="otel-mapping-type">
                <collectors>
                    <collector ref="otel-collector"/>
                </collectors>
                <mappings>
                    <mapping ref="ca-self-monitor"/>
                    <mapping ref="otel-basic"/>
                </mappings>
                <samplers>
                    <sampler ref="logs"/>
                    <sampler ref="kafka-logs"/>
                </samplers>
            </mappingType>
        </mappingTypes>
        <collectionAgentParameters>
            <collectionAgentParameters name="otel-ca">
                <managed>
                    <yaml>
                        <managedYAML>
                            <selfMonitoring>true</selfMonitoring>
                        </managedYAML>
                    </yaml>
                </managed>
            </collectionAgentParameters>
        </collectionAgentParameters>
    </dynamicEntities>
    <samplers>
        <sampler name="logs">
            <var-group>
                <data>logs</data>
            </var-group>
            <plugin>
                <fkm>
                    <files>
                        <file>
                            <source>
                                <stream>
                                    <data>*</data>
                                </stream>
                            </source>
                        </file>
                    </files>
                </fkm>
            </plugin>
        </sampler>
        <sampler name="kafka-logs">
            <var-group>
                <data>logs</data>
            </var-group>
            <plugin>
                <fkm>
                    <files>
                        <file>
                            <source>
                                <stream>
                                    <data>kafka.*</data>
                                </stream>
                            </source>
                        </file>
                    </files>
                </fkm>
            </plugin>
        </sampler>
    </samplers>
</gateway>

```