Metrics API¶

The ASR server provides a set of metrics related to speech recognition than can be accessed via two interfaces: HTTP and PROMETHEUS. Through these metrics, combined with metrics that can be collected from the operating system itself (CPU, memory, I/O, etc.), the administrator can evaluate the level of system utilization and performance.

Warning

Although several metrics are returned by ASR, only a certain set should be considered for monitoring speech recognition. Metrics not presented here should not be used. Some of these are still undergoing development and can be modified or even removed. Other metrics are internal and are not directly related to ASR.

The following types of metrics exist:

Counter: A value that can be incremented or subtracted (e.g. Number of requests processed).
Histogram: Measures the statistical distribution in a sequence of values. We have the maximum, minimum, mean and standard deviation values, besides the quantis. This measure corresponds to the most recent samples, within a window of approximately 5 minutes. This window only moves when there are new samples, and does not consider time per se.
Timer: A function’s execution rate and the distribution of its length.
Gauge: Instantaneous value of a given measure.
Meter: Measures the occurrence rate of an event in an interval of time. It registers the mean rate in a mobile time window (-1, -5, -15 minutes). This window only moves when there are new samples, and does not consider time per se.

ASR metrics¶

Recognition sessions.¶

gauges/br.com.cpqd.asr.session.open
Number of ongoing recognition sessions (open).

timers/br.com.cpqd.asr.session.duration
Session length histogram and session creation rate.

Recognition requests¶

gauges/br.com.cpqd.asr.recognition.active
Number of ongoing recognition sessions. Currently, corresponds to the number of license channels in use.

counters/br.com.cpqd.asr.recognition.requests
Accumulated value of the total number of recognitions performed so far.

Input audio¶

counters/br.com.cpqd.asr.engine.audio.duration
Accumulated length of input audio in milliseconds.

counters/br.com.cpqd.asr.engine.speech.duration
Accumulated length of speech contained in the input audio, in milliseconds.

Recognition time¶

counters/br.com.cpqd.asr.engine.recognition.time
Accumulated value of the recognition time in milliseconds.

histograms/br.com.cpqd.asr.recognition.time
Histogram of the recognition time in milliseconds.

Recognition RTF¶

histograms/br.com.cpqd.asr.engine.rtf100.grammar
Histogram of the RTF percentage for grammars. A value of 80 displayed here corresponds to an RTF=0,8.

histograms/br.com.cpqd.asr.engine.rtf100.slm
Histogram of the RTF percentage for free speech models. A value of 120 displayed here corresponds to an RTF=1,2.

Recognition results¶

counters/br.com.cpqd.asr.recognition.result.RECOGNIZED
Accumulated value of the number of recognition requests that completed successfully, in other words, with a recognized sentence. Successfully recognized does not mean a”correct result”

counters/br.com.cpqd.asr.recognition.result.CANCELED
Accumulated number of recognition requests that were “canceled”

counters/br.com.cpqd.asr.recognition.result.ERR_MAX_LICENSES
Accumulated number of recognition requests declined for surpassing the maximum number of license channels.

counters/br.com.cpqd.asr.recognition.result.FAILURE
Accumulated number of recognition requests that “failed”

counters/br.com.cpqd.asr.recognition.result.MAX_SPEECH
Accumulated value of the number of recognition requests that were ended for exceeding the internal maximum speech limit.

counters/br.com.cpqd.asr.recognition.result.NO_INPUT_TIMEOUT
Accumulated number of recognition requests that were ended because the system did not detect any speech for a time longer than the configured limit (noInputTimeout.value parameter).

counters/br.com.cpqd.asr.recognition.result.NO_MATCH
Accumulated number of recognition requests that were ended because the system could not recognize the input audio or the confidence rate was lower than the configured threshold (decoder.confidenceThreshold parameter).

counters/br.com.cpqd.asr.recognition.result.NO_SPEECH
Accumulated number of recognition requests that were ended because the system did not detect any input audio.

counters/br.com.cpqd.asr.recognition.result.RECOGNITION_TIMEOUT
Accumulated number of recognition requests that were ended because the system was not able to perform the recognition within the configured time limit (recognitionTimeout.value parameter).

HTTP Interface¶

Using the HTTP interface, ASR metrics can be read by sending a HTTP GET request to URL: http://ASR_HOST:8081/metrics, where ASR_HOST is the IP of the ASR server to be read. As an example, open a console on the ASR Server machine and run:

curl http://localhost:8027/actuator/metrics

The result should be similar to the following:

$ curl http://localhost:8027/actuator/metrics

{
  "names": [
      "application.ready.time",
      "application.started.time",
      "br.com.cpqd.asr.engine.audio.duration",
      "br.com.cpqd.asr.engine.recognition.time",
      "br.com.cpqd.asr.engine.rtf100",
      "br.com.cpqd.asr.engine.speech.duration",
      "br.com.cpqd.asr.license.usage",
      "br.com.cpqd.asr.recognition.active",
      "br.com.cpqd.asr.recognition.requests",
      "br.com.cpqd.asr.recognition.result",
      "br.com.cpqd.asr.recognition.result.ERR_MAX_LICENSES",
      "br.com.cpqd.asr.redis.saveaudio.time",
      "br.com.cpqd.asr.session.duration",
      "br.com.cpqd.asr.session.open",
      "disk.free",
      "disk.total",
      "executor.execution",
      "executor.idle",
      "grpc.server.processing.duration",
      "grpc.server.requests.received",
      "grpc.server.responses.sent",
      "jvm.buffer.count",
      "jvm.buffer.memory.used",
      "jvm.buffer.total.capacity",
      "jvm.classes.loaded",
      "jvm.classes.unloaded",
      "jvm.gc.live.data.size",
      "jvm.gc.max.data.size",
      "jvm.gc.memory.allocated",
      "jvm.gc.memory.promoted",
      "jvm.gc.overhead",
      "jvm.gc.pause",
      "jvm.memory.committed",
      "jvm.memory.max",
      "jvm.memory.usage.after.gc",
      "jvm.memory.used",
      "jvm.threads.daemon",
      "jvm.threads.live",
      "jvm.threads.peak",
      "jvm.threads.states",
      "logback.events",
      "process.cpu.usage",
      "process.files.max",
      "process.files.open",
      "process.start.time",
      "process.uptime",
      "system.cpu.count",
      "system.cpu.usage",
      "system.load.average.1m",
      "tomcat.sessions.active.current",
      "tomcat.sessions.active.max",
      "tomcat.sessions.alive.max",
      "tomcat.sessions.created",
      "tomcat.sessions.expired",
      "tomcat.sessions.rejected"
  ]
}

To query each metric use the URL/<nome da métrica>. See the following example:

$ curl http://localhost:8027/actuator/metrics/br.com.cpqd.asr.session.open

O resultado será o seguinte:

{
  "name":"br.com.cpqd.asr.session.open",
  "description":"Open sessions",
  "baseUnit":null,
  "measurements":[
    {
      "statistic":"VALUE",
      "value":0.0
    }
  ],
  "availableTags":[
  ]
}

PROMETHEUS interface¶

ASR metrics can be read via the PROMETHEUS interface by sending a request to URL http://ASR_HOST:8027/actuator/prometheus, where ASR_HOST is the desired ASR Server IP.