Prometheus Probe

The prometheus probe allows users to run Prometheus queries and match the resulting output against specific conditions. The intent behind this probe is to allow users to define metrics-based SLOs in a declarative way and determine the experiment verdict based on its success. The probe runs the query on a Prometheus server defined by the endpoint, and checks whether the output satisfies the specified criteria. It can be executed by setting type as promProbe inside .spec.experiments[].spec.probe.

View the prometheus probe schema

Field	`.name`
Description	Flag to hold the name of the probe
Type	Mandatory
Range	n/a (type: string)
Notes	The `.name` holds the name of the probe. It can be set based on the usecase

Field	`.type`
Description	Flag to hold the type of the probe
Type	Mandatory
Range	`httpProbe`, `k8sProbe`, `cmdProbe`, `promProbe`
Notes	The `.type` supports four type of probes. It can one of the `httpProbe`, `k8sProbe`, `cmdProbe`, `promProbe`

Field	`.mode`
Description	Flag to hold the mode of the probe
Type	Mandatory
Range	`SOT`, `EOT`, `Edge`, `Continuous`, `OnChaos`
Notes	The `.mode` supports five modes of probes. It can one of the `SOT`, `EOT`, `Edge`, `Continuous`, `OnChaos`

Field	`.promProbe/inputs.endpoint`
Description	Flag to hold the prometheus endpoints for the promProbe
Type	Mandatory
Range	n/a {type: string}
Notes	The `.promProbe/inputs.endpoint` contains the prometheus endpoints

Field	`.promProbe/inputs.query`
Description	Flag to hold the promql query for the promProbe
Type	Mandatory
Range	n/a {type: string}
Notes	The `.promProbe/inputs.query` contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint

Field	`.promProbe/inputs.queryPath`
Description	Flag to hold the path of the promql query for the promProbe
Type	Optional
Range	n/a {type: string}
Notes	The `.promProbe/inputs.queryPath` This field is used in case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR.

Field	`.promProbe/inputs.comparator.criteria`
Description	Flag to hold criteria for the comparision
Type	Mandatory
Range	it supports {>=, <=, ==, >, <, !=, oneOf, between} criteria
Notes	The `.promProbe/inputs.comparator.criteria` contains criteria of the comparision, which should be fulfill as part of comparision operation.

Field	`.promProbe/inputs.comparator.value`
Description	Flag to hold value for the comparision
Type	Mandatory
Range	n/a {type: string}
Notes	The `.promProbe/inputs.comparator.value` contains value of the comparision, which should follow the given criteria as part of comparision operation.

Field	`.runProperties.probeTimeout`
Description	Flag to hold the timeout for the probes
Type	Mandatory
Range	n/a {type: integer}
Notes	The `.runProperties.probeTimeout` represents the time limit for the probe to execute the specified check and return the expected data

Field	`.runProperties.retry`
Description	Flag to hold the retry count for the probes
Type	Mandatory
Range	n/a {type: integer}
Notes	The `.runProperties.retry` contains the number of times a check is re-run upon failure in the first attempt before declaring the probe status as failed.

Field	`.runProperties.interval`
Description	Flag to hold the interval for the probes
Type	Mandatory
Range	n/a {type: integer}
Notes	The `.runProperties.interval` contains the interval for which probes waits between subsequent retries

Field	`.runProperties.probePollingInterval`
Description	Flag to hold the polling interval for the probes(applicable for `Continuous` mode only)
Type	Optional
Range	n/a {type: integer}
Notes	The `.runProperties.probePollingInterval` contains the time interval for which continuous probe should be sleep after each iteration

Field	`.runProperties.initialDelaySeconds`
Description	Flag to hold the initial delay interval for the probes
Type	Optional
Range	n/a {type: integer}
Notes	The `.runProperties.initialDelaySeconds` represents the initial waiting time interval for the probes.

Field	`.runProperties.stopOnFailure`
Description	Flags to hold the stop or continue the experiment on probe failure
Type	Optional
Range	false {type: boolean}
Notes	The `.runProperties.stopOnFailure` can be set to true/false to stop or continue the experiment execution after probe fails

Common Probe Tunables¶

Refer the common attributes to tune the common tunables for all the probes.

Prometheus Query(query is a simple)¶

It contains the promql query to extract out the desired prometheus metrics via running it on the given prometheus endpoint. The prometheus query can be provided in the query field. It can be executed by setting .promProbe/inputs.query field.

Use the following example to tune this:

# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-delete-sa
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "check-probe-success"
        type: "promProbe"
        promProbe/inputs:
          # endpoint for the promethus service
          endpoint: "<prometheus-endpoint>"
          # promql query, which should be executed
          query: "<promql-query>"
          comparator:
            # criteria which should be followed by the actual output and the expected output
            #supports >=,<=,>,<,==,!= comparision
            criteria: "==" 
            # expected value, which should follow the specified criteria
            value: "<value-for-criteria-match>"
        mode: "Edge"
        runProperties:
          probeTimeout: 5
          interval: 5
          retry: 1

Prometheus Query(query is a complex¶

In case of complex queries that spans multiple lines, the queryPath attribute can be used to provide the path to a file consisting of the same. This file can be made available to the experiment pod via a ConfigMap resource, with the ConfigMap name being defined in the ChaosEngine OR the ChaosExperiment CR. It can be executed by setting promProbe/inputs.queryPath field.

NOTE: It is mutually exclusive with the query field. If query is set then it will use the query field otherwise, it will use the queryPath field.

Use the following example to tune this:

# contains the prom probe which execute the query and match for the expected criteria
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  appinfo:
    appns: "default"
    applabel: "app=nginx"
    appkind: "deployment"
  chaosServiceAccount: pod-delete-sa
  experiments:
  - name: pod-delete
    spec:
      probe:
      - name: "check-probe-success"
        type: "promProbe"
        promProbe/inputs:
          # endpoint for the promethus service
          endpoint: "<prometheus-endpoint>"
          # the configMap should be mounted to the experiment which contains promql query
          # use the mounted path here
          queryPath: "<path of the query>"
          comparator:
            # criteria which should be followed by the actual output and the expected output
            #supports >=,<=,>,<,==,!= comparision
            criteria: "==" 
            # expected value, which should follow the specified criteria
            value: "<value-for-criteria-match>"
        mode: "Edge"
        runProperties:
          probeTimeout: 5
          interval: 5
          retry: 1