Metrics API

The CPQD ASR provides a set of metrics related to speech recognition that can be accessed through two interfaces: HTTP and JMX. Through these metrics, combined with the metrics that can be collected from the operating system itself (CPU, memory I/O, etc.), the administrator can evaluate the usage level and performance of the system.

Warning

Although several metrics are returned by the CPQD ASR, only a certain set of them must be taken into consideration for speech recognition monitoring. Metrics not listed here should not be used. Some of them are still under development and can be modified or even removed. Other metrics are internal and are not directly related to the ASR.

The following types of metrics exist:

Counter

A value that can be incremented or subtracted (e.g. Number of requests processed).

Histogram

Measures the statistical distribution in a sequence of values. We have the maximum, minimum, mean and standard deviation values, besides the quantis. This measure corresponds to the most recent samples, within a window of approximately 5 minutes. This window only moves when there are new samples, and does not consider time per se.

Timer

A function’s execution rate and the distribution of its length.

Gauge

Instantaneous value of a given measure.

Meter

Measures the occurrence rate of an event in an interval of time. It registers the mean rate in a mobile time window (-1, -5, -15 minutes). This window only moves when there are new samples, and does not consider time per se.

ASR metrics

Recognition sessions.

gauges/br.com.cpqd.asr.session.open

Number of ongoing recognition sessions (open).

timers/br.com.cpqd.asr.session.duration

Session length histogram and session creation rate.

Recognition requests

gauges/br.com.cpqd.asr.recognition.active

Number of ongoing recognition sessions. Currently, corresponds to the number of license channels in use.

counters/br.com.cpqd.asr.recognition.requests

Accumulated value of the total number of recognitions performed so far.

Input audio

counters/br.com.cpqd.asr.engine.audio.duration

Accumulated length of input audio in milliseconds.

counters/br.com.cpqd.asr.engine.speech.duration

Accumulated length of speech contained in the input audio, in milliseconds.

Recognition time

counters/br.com.cpqd.asr.engine.recognition.time

Accumulated value of the recognition time in milliseconds.

histograms/br.com.cpqd.asr.recognition.time

Histogram of the recognition time in milliseconds.

Recognition RTF

histograms/br.com.cpqd.asr.engine.rtf100.grammar

Histogram of the RTF percentage for grammars. A value of 80 displayed here corresponds to an RTF=0,8.

histograms/br.com.cpqd.asr.engine.rtf100.slm

Histogram of the RTF percentage for free speech models. A value of 120 displayed here corresponds to an RTF=1,2.

Recognition results

counters/br.com.cpqd.asr.recognition.result.RECOGNIZED

Accumulated value of the number of recognition requests that completed successfully, in other words, with a recognized sentence. Successfully recognized does not mean a”correct result”

counters/br.com.cpqd.asr.recognition.result.CANCELED

Accumulated number of recognition requests that were “canceled”

counters/br.com.cpqd.asr.recognition.result.ERR_MAX_LICENSES

Accumulated number of recognition requests declined for surpassing the maximum number of license channels.

counters/br.com.cpqd.asr.recognition.result.FAILURE

Accumulated number of recognition requests that “failed”

counters/br.com.cpqd.asr.recognition.result.MAX_SPEECH

Accumulated value of the number of recognition requests that were ended for exceeding the internal maximum speech limit.

counters/br.com.cpqd.asr.recognition.result.NO_INPUT_TIMEOUT

Accumulated number of recognition requests that were ended because the system did not detect any speech for a time longer than the configured limit (noInputTimeout.value parameter).

counters/br.com.cpqd.asr.recognition.result.NO_MATCH

Accumulated number of recognition requests that were ended because the system could not recognize the input audio or the confidence rate was lower than the configured threshold (decoder.confidenceThreshold parameter).

counters/br.com.cpqd.asr.recognition.result.NO_SPEECH

Accumulated number of recognition requests that were ended because the system did not detect any input audio.

counters/br.com.cpqd.asr.recognition.result.RECOGNITION_TIMEOUT

Accumulated number of recognition requests that were ended because the system was not able to perform the recognition within the configured time limit (recognitionTimeout.value parameter).

HTTP Interface

Using the HTTP interface, ASR metrics can be read by sending a HTTP GET request to URL: http://ASR_HOST:8081/metrics, where ASR_HOST is the IP of the ASR server to be read. As an example, open a console on the ASR Server machine and run:

curl http://localhost:8081/metrics?pretty=true

The result should be something like this: (irrelevant lines have been removed to simplify the example):

$ curl http://localhost:8081/metrics?pretty=true

{
  "gauges" : {
    "br.com.cpqd.asr.recognition.active" : {
      "value" : 0
    },
    "br.com.cpqd.asr.session.open" : {
      "value" : 0
    },
  },
  "counters" : {
    "br.com.cpqd.asr.engine.audio.duration" : {
      "count" : 0
    },
    "br.com.cpqd.asr.engine.recognition.time" : {
      "count" : 0
    },
    "br.com.cpqd.asr.engine.speech.duration" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.requests" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.CANCELED" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.ERR_MAX_LICENSES" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.FAILURE" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.MAX_SPEECH" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.NO_INPUT_TIMEOUT" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.NO_MATCH" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.NO_SPEECH" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.RECOGNITION_TIMEOUT" : {
      "count" : 0
    },
    "br.com.cpqd.asr.recognition.result.RECOGNIZED" : {
      "count" : 0
    },
  },
  "histograms" : {
    "br.com.cpqd.asr.engine.rtf100.grammar" : {
      "count" : 0,
      "max" : 0,
      "mean" : 0.0,
      "min" : 0,
      "p50" : 0.0,
      "p75" : 0.0,
      "p95" : 0.0,
      "p98" : 0.0,
      "p99" : 0.0,
      "p999" : 0.0,
      "stddev" : 0.0
    },
    "br.com.cpqd.asr.engine.rtf100.slm" : {
      "count" : 0,
      "max" : 0,
      "mean" : 0.0,
      "min" : 0,
      "p50" : 0.0,
      "p75" : 0.0,
      "p95" : 0.0,
      "p98" : 0.0,
      "p99" : 0.0,
      "p999" : 0.0,
      "stddev" : 0.0
    },
    "br.com.cpqd.asr.recognition.time" : {
      "count" : 0,
      "max" : 0,
      "mean" : 0.0,
      "min" : 0,
      "p50" : 0.0,
      "p75" : 0.0,
      "p95" : 0.0,
      "p98" : 0.0,
      "p99" : 0.0,
      "p999" : 0.0,
      "stddev" : 0.0
    }
  },
  "timers" : {
    "br.com.cpqd.asr.session.duration" : {
      "count" : 0,
      "max" : 0.0,
      "mean" : 0.0,
      "min" : 0.0,
      "p50" : 0.0,
      "p75" : 0.0,
      "p95" : 0.0,
      "p98" : 0.0,
      "p99" : 0.0,
      "p999" : 0.0,
      "stddev" : 0.0,
      "m15_rate" : 0.0,
      "m1_rate" : 0.0,
      "m5_rate" : 0.0,
      "mean_rate" : 0.0,
      "duration_units" : "seconds",
      "rate_units" : "calls/second"
    },
  }
}

JMX Interface

The returned metrics can be read with the JMX Interface, using port 9999. By default, the JXM Interface is disabled on the CPQD ASR. Use the following instructions to enable the JMX Interface:

  1. Define the variable ASR_JMX_ENABLED=true using the command systemctl set-environment

  2. Check the configured variable using the command systemctl show-environment

  3. Restart the ASR server

See the following example:

$ systemctl set-environment ASR_JMX_ENABLED=true

$ systemctl show-environment

ASR_JMX_ENABLED=true
LANG=en_US.UTF-8
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin

$ systemctl restart asr-server

Check if the ASR server was successfully started and observe how the JMX access has been configured, as shown in the following example:

$ systemctl status asr-server

● asr-server.service - CPqD Speech Recognition Server
   Loaded: loaded (/usr/lib/systemd/system/asr-server.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-06-15 20:24:24 UTC; 7s ago
     Docs: https://speechweb.cpqd.com.br/asr/docs/latest
 Main PID: 1320 (java)
   CGroup: /docker/a6645a54c667c4bf2d7e49d08e80ddc60fad1383cc2e698740a967cc135fe91f/system.slice/asr-server.service
           └─1320 java -Xms128m -Xmx256m -XX:MaxDirectMemorySize=512m -XX:ErrorFile=/var/log/cpqd/asr/server/hs_err_pid%p.log
            -Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.m...

Jun 15 20:24:24 a6645a54c667 systemd[1]: Started CPqD Speech Recognition Server.
Jun 15 20:24:24 a6645a54c667 systemd[1]: Starting CPqD Speech Recognition Server...
Jun 15 20:24:24 a6645a54c667 start-server[1320]: Starting process asr-server...

Test the JMX access using a machine with Java (Oracle) installed, using the ‘VisualVM tool <https://visualvm.github.io/>`_, with the command jvisualvm --openjmx HOST_IP:9999, where HOST_IP is the IP of CPQD ASR server. Fig. 7 shows the MBeans query window.

../_images/visualvm-jmx.png

Fig. 7 JMX Access using the VisualVM tool

JMX Connection problems

Sometimes, Java fails to correctly identify the machine IP and the JMX connection returns an error (Cannot connect to ... using service:jmx:rmi ...). In this case, configure the IP using the following command. Replace HOST_IP for the correct IP of the machine running the CPQD ASR server. Remember to :ref:` restart the ASR server <restart_server>‘ after the configuration.

systemctl set-environment ASR_JMX_OPTS="-Dcom.sun.management.jmxremote.port=9999 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Djava.rmi.server.hostname=HOST_IP"