Skip to main content

Monitor stream health

This guide provides instructions and information about using stream metrics:

  • Monitor any active stream for issues that may impact the quality of your stream
  • Understand the metrics for operating a livestreaming or user generated content platform
  • View health status of your livestream (API and Dashboard No-Code Option)

Metrics with Source Segments Duration​

The value of sourceSegmentsDuration returned is the duration in seconds of the asset source processed by Livepeer Studio. On the parent stream object, this value equates to the total amount of source video ingested by Livepeer Studio all time. On the session object, this value is the length of the live stream session (or the session recording).

Request​

GET /stream/{id} to retrieve a stream object

curl -H 'authorization: Bearer {api-key}' \
"https://livepeer.studio/api/stream/{id}"

GET /session/{id} to retrieve a session object

curl -H 'authorization: Bearer {api-key}' \
"https://livepeer.studio/api/session/{id}"

Response​

  • Returns a 200 response with the following object:
{
    "name":"test_stream",
    "id":"ijkl61f3-95bd-4971-a7b1-4dcb5d39e78a",
    "createdAt":1596081229373,
    "streamKey":"abcd-uimq-jtgy-x98v",
    "playbackId":"efghb2mxupongp5k",
    "isActive":"true",
    "lastSeen":1596119750455",
    "sourceSegments":1360,
    "transcodedSegments":5480,
    "sourceSegmentsDuration":2630.53200000000004,
    "transcodedSegmentsDuration":10620.1280000000004,
    {other asset object keys, like profiles and parentId}
}

Global Health Status​

  • Healthy indicates that all the Health Checks are in a Healthy state.
  • Unhealthy indicates that one or more of the Health Checks are in an Unhealthy state
  • Idle indicates that the stream is not currently active

Health Checks​

  • Transcoding — A Healthy status indicates that video transcoding is occurring, and the multiple profiles of your stream are available for playback. An Unhealthy status indicates that we have failed to generate the profiles , so only the original video will be available. Check the event log below for any actionable errors, or check the Status page for platform-wide issues.
  • Real-time — This indicates whether our system transcoded the stream faster than in real-time. In other words, if the video transcodes, latency is lower than the video duration. Unconventional configurations may cause an Unhealthy status.
  • Multistreaming — This health check indicates whether all configured Multistream Targets are connected. If no targets exist, this will appear blank and not be considered for the global health state. If this shows up as Unhealthy, you can also check the status of the individual Multistream Targets in the Overivew tab, where the targets can be either Idle, Pending, Online'or' Offline:
    • An Idle status indicates that the stream is not currently active, so neither is multistreaming.
    • A Pending status indicates that our system tries connecting to the respective Multistream Target.
    • An Online status indicates that the Multistream Target is successfully connected.
    • An Offline status means that we've received an error when trying to connect to the target. To try again, check the ingest URL and stream key of your destination service, re-configure your Multistream Target and restart your stream.

Logs​

Logs will surface informational alerts (ex., stream has started/stopped, multistream destinations have connected/disconnected) or any fatal errors from the transcoding process. Here is an example of a fatal error:

Transcode error from ewr-prod-livepeer-orchestrator-0 for segment 0:
**Unsupported input pixel format**

If you run into this or any similar errors, check the configuration in your streaming software (e.g., OBS) and restart the stream.

Session Ingest Rate​

Session ingest rate tells you the bitrate of the video received by Livepeer ingest servers, updated every 10 seconds. A high bitrate is suggestive that streams into the Livepeer system are high quality, your encoder is working correctly, and your internet connection is good. A fluctuating or low bitrate may suggest that your encoder is misconfigured or that the streamer internet connection isn't as strong as it should be to deliver high-quality streams to your viewers.

Monitoring Stream Health Using The Dashboard​

In the dashboard, there will be a health tab on the right side

image

During any Active stream, components on this page will display a variety of health indicators.

image

image

At the bottom of the page, a chart will show the ingest rate of a livestream

image

Monitor Stream Health Using The API​

Note: The Stream Health API is currently in beta and some elements may change as we mature the product. Before you start building with it, let us know at help@livepeer.studio so we can give you a heads up before any breaking changes.

The health information about a stream can also be queried using a separate Stream Health API resides on a different root path /data. To use it, you need to include the same API key as the one used for the regular API. The API consists of the following endpoints:

  • Health Status (/data/stream/{streamId}/health): Grabs the latest health status info about a given stream, referenced by its Stream ID. This API powers the dashboard health checks and multistream status.
  • Events (/data/stream/{streamId}/events): Provides direct access to the low-level events sent by Livepeer video-processing services. It also allows subscribing via SSE to receive new events and is useful for debugging. This API powers the log feed in the dashboard.

Now let's go into more detail about each of the mentioned API endpoints.

Health Status​

Request​

The stream ID is the same one as used in the Livestream API.

GET livepeer.studio/data/stream/{id}/health

Content-Type: application/json

curl 'https://livepeer.studio/data/stream/{id}/health' \
-H 'authorization: Bearer {api-key}'

Response​

Returns an object with a 200 OK response

{
  "id": "{id}",
  "healthy": {
    "status": true,
    "frequency": { "1m": 1, "10m": 0.9655, "60m": 0.9655 },
    "lastProbeTime": 1635993921603,
    "lastTransitionsTime": 1635993751824
  },
  "conditions": [
    {
      "type": "Active",
      "status": true,
      "extraData": {
        "nodeId": "prod-mist-blue-server-1",
        "region": "lon"
      },
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993739354,
      "lastTransitionsTime": 1635993739354
    },
    {
      "type": "Transcoding",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "TranscodeRealTime",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "TranscodeNoErrors",
      "status": true,
      "frequency": { "1m": 1, "10m": 1, "60m": 1 },
      "lastProbeTime": 1635993920641,
      "lastTransitionsTime": 1635993751824
    },
    {
      "type": "Multistreaming",
      "status": null,
      "lastProbeTime": null,
      "lastTransitionsTime": null
    }
  ],
  "metrics": {
    "MediaTimeMillis": [
      {
        "name": "MediaTimeMillis",
        "dimensions": { "nodeId": "prod-mist-blue-server-1" },
        "last": [1635993921603, 182528]
      }
    ],
    "TranscodeRealtimeRatio": [
      {
        "name": "TranscodeRealtimeRatio",
        "dimensions": {
          "nodeId": "prod-livepeer-broadcaster-6emu-97d8bf496-7zh9f"
        },
        "last": [1635993920641, 3.088169642857143]
      }
    ]
  }
}

In the above response payload:

  • All timestamps are represented in milliseconds since the Unix epoch, which is the default representation of timestamps across all the Livepeer API.
  • The healthy top-level condition is a computed state based on some specific stream conditions. As of writing this doc, a stream is healthy if it's both Active, Transcoding in realtime (TranscodeRealTime), and if multistream targets are configured, they are also healthy (Multistreaming).
  • The conditions array contains more specific information about the stream health state, with the type field specifying what the condition means.
  • More generally, a Condition represents a specific state of the stream health.
    • The status field represents the value of the last reading for that condition, and the lastProbeTime specified when that was.
    • The lastTransitionsTime and the frequency fields provide some insight into how it's changing over time.
    • The lastTransitionTime is the timestamp of the last time where the condition status changed, and;
    • The frequency is the ratio of successful probes (status: true) in the specified time ranges preceding lastProbeTime.
  • The metrics are still an experimental API and are not recommended for production systems. But the last field is a tuple containing the timestamp and the value of the last reading for that metric.

Events​

Request​

The stream ID is the same one used in the Livestream API.

GET livepeer.studio/data/stream/{id}/events

Content-Type: text/event-stream

curl 'https://livepeer.studio/data/stream/{id}/events' \
-H 'authorization: Bearer {api-key}'

Response​

Returns an object with a 200 OK response

retry: 10000

id: b6e0d6a4-0718-4db4-b2ec-31bb23a9fde0
event: lp_event
data: {"type":"media_server_metrics","id":"b6e0d6a4-0718-4db4-b2ec-31bb23a9fde0","timestamp":1635994871604,"streamId":"{id}","nodeId":"prod-mist-blue-server-1","region":"lon","stats":{"mediaTimeMs":1132566},"multistream":[]}

id: 7a0e6f9e-baa1-42c4-a92f-4dfa2f50ac05
event: lp_event
data: {"type":"transcode","id":"7a0e6f9e-baa1-42c4-a92f-4dfa2f50ac05","timestamp":1635994875678,"streamId":"{id}","nodeId":"prod-livepeer-broadcaster-6emu-97d8bf496-7zh9f","segment":{"name":"","seqNo":292,"duration":8.334,"byteSize":2270100},"startTime":1635994874469,"latencyMs":1209,"success":true,"attempts":[{"orchestrator":{"address":"0x1cd98ad89a7d143847f62d2249e4005d09e10648","transcodeUri":"https://vno2-prod-livepeer-orchestrator-7.livepeer.studio:443"},"latencyMs":1209,"error":null}]}

id: fd782a49-4986-4b3c-b3b4-9088ae76c152
event: lp_event
data:
{"type":"media_server_metrics","id":"fd782a49-4986-4b3c-b3b4-9088ae76c152","timestamp":1635994881604,"streamId":"{id}","nodeId":"prod-mist-blue-server-1","region":"lon","stats":{"mediaTimeMs":1142550},"multistream":[]}
...

The /events API response implements the Server-Sent Events protocol using any SSE client. On JavaScript, we recommend using the eventsource NPM package to be able to add an Authorization header to the API request.

The individual events sent over the SSE stream are all JSON objects and are precisely the same low-level objects as published by the services running in the Livepeer infrastructure. Some fields are always present, specifically:

  • type: the unique type of the event, which defines the structure of the rest of the JSON object.
  • id: a unique identifier for the event, which SSE clients automatically use to recover a connection without losing events.
  • timestamp: the timestamp of the event in milliseconds since the Unix epoch.

You can also get historical events within a time range by specifying the from and to query parameters with Unix Millis timestamps. For example:

  • /data/stream/{id}/events?from=1635437394000 to get all events since Oct 28 16:09:54Z 2021 (exclusively) and stay connected for live events.
  • /data/stream/{id}/events?from=1635437394000&to=1635447394000 to get all events between Oct 28 16:09:54Z 2021 (exclusively) and Oct 28 18:56:34Z 2021 (inclusively).

Finally, we recommend using the events API primarily for debugging purposes, for example, to get all available information about a live stream to find any issues. The specific events that are published and their respective schemas will keep evolving, so they are still subject to changes.