ZRTMC results monitor – phase 2: components and self healing

The ZRTMC analytics pipeline is made up of several components and can be configured to send several different types of data. An individual component could fail and cause data to not be available, or a specific type of data may not be available. As more features are added, it becomes more and more involved to know when the ZRTMC monitoring and analytics is failing to send or analyze data.

It is desirable to add to the ZRTMC analytics pipeline a “results monitor – phase 2: components and self healing”. Its purpose is to monitor the individual components, attempt to selfheal and alert administrators to the conditions and failures to selfheal. This will attempt to automate the resolution of any issues and help administrators to resolve failures more quickly.

The solution should include the following features discussed with customers:

• Consider using OpenTelemetry to monitor all the components in the framework.

• Notify administrators when redundancy configuration/fall back components are being used. This allows administrators to correct conditions that caused the primary components to fail such that if the secondary fails, it can fall back to the primary instead of resulting in an outage.

• Look at the entire chain from z/TPF to Grafana and surface what is not functioning properly or are dead.

• Automate recovery of components that are not functioning.

• Avoid false positives like IPLs or slow starts of the ZRTMC component chain after an IPL.