Cassandra Pod Delete

Introduction¶

It causes (forced/graceful) pod failure of specific/random replicas of an cassandra statefulset
It tests cassandra sanity (replica availability & uninterrupted service) and recovery workflow of the cassandra statefulset.

Scenario: Deletes cassandra pod

Uses¶

View the uses of the experiment

coming soon

Prerequisites¶

Verify the prerequisites

Ensure that Kubernetes Version > 1.16
Ensure that the Litmus Chaos Operator is running by executing kubectl get pods in operator namespace (typically, litmus).If not, install from here
Ensure that the cassandra-pod-delete experiment resource is available in the cluster by executing kubectl get chaosexperiments in the desired namespace. If not, install from here

Default Validations¶

View the default validations

Cassandra pods are healthy before chaos injection
The load should be distributed on the each replicas.

Minimal RBAC configuration example (optional)¶

NOTE

If you are using this experiment as part of a litmus workflow scheduled constructed & executed from chaos-center, then you may be making use of the litmus-admin RBAC, which is pre installed in the cluster as part of the agent setup.

View the Minimal RBAC permissions

---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
rules:
  # Create and monitor the experiment & helper pods
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create","delete","get","list","patch","update", "deletecollection"]
  # Performs CRUD operations on the events inside chaosengine and chaosresult
  - apiGroups: [""]
    resources: ["events"]
    verbs: ["create","get","list","patch","update"]
  # Create and manage the liveness and target application services
  - apiGroups: [""]
    resources: ["services"]
    verbs: ["create","delete","get","list", "deletecollection"]
  # Fetch configmaps & secrets details and mount it to the experiment pod (if specified)
  - apiGroups: [""]
    resources: ["secrets","configmaps"]
    verbs: ["get","list",]
  # Track and get the runner, experiment, and helper pods log 
  - apiGroups: [""]
    resources: ["pods/log"]
    verbs: ["get","list","watch"]  
  # for creating and managing to execute comands inside target container
  - apiGroups: [""]
    resources: ["pods/exec"]
    verbs: ["get","list","create"]
  # for deriving the parent/owner details of the pod   
  - apiGroups: ["apps"]
    resources: ["deployments","statefulsets"]
    verbs: ["list","get"]
  # for configuring and monitor the experiment job by the chaos-runner pod
  - apiGroups: ["batch"]
    resources: ["jobs"]
    verbs: ["create","list","get","delete","deletecollection"]
  # for creation, status polling and deletion of litmus chaos resources used within a chaos workflow
  - apiGroups: ["litmuschaos.io"]
    resources: ["chaosengines","chaosexperiments","chaosresults"]
    verbs: ["create","list","get","patch","update","delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: cassandra-pod-delete-sa
  namespace: default
  labels:
    name: cassandra-pod-delete-sa
    app.kubernetes.io/part-of: litmus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: cassandra-pod-delete-sa
subjects:
- kind: ServiceAccount
  name: cassandra-pod-delete-sa
  namespace: default

Use this sample RBAC manifest to create a chaosServiceAccount in the desired (app) namespace. This example consists of the minimum necessary role permissions to execute the experiment.

Experiment tunables¶

check the experiment tunables

Optional Fields

Variables	Description	Notes
CASSANDRA_SVC_NAME	Cassandra Service Name	Defaults value: cassandra
KEYSPACE_REPLICATION_FACTOR	Value of the Replication factor for the cassandra liveness deploy	It needs to create keyspace while checking the livenss of cassandra
CASSANDRA_PORT	Port of the cassandra statefulset	Defaults value: 9042
CASSANDRA_LIVENESS_CHECK	It allows to check the liveness of the cassandra statefulset	It can be `enabled` or `disabled`
CASSANDRA_LIVENESS_IMAGE	Image of the cassandra liveness deployment	Default value: litmuschaos/cassandra-client:latest
SEQUENCE	It defines sequence of chaos execution for multiple target pods	Default value: parallel. Supported: serial, parallel
TOTAL_CHAOS_DURATION	The time duration for chaos insertion (seconds)	Defaults to 15s
PODS_AFFECTED_PERC	The Percentage of total pods to target	Defaults to 0% (corresponds to 1 replica)
CHAOS_INTERVAL	Time interval b/w two successive pod failures (sec)	Defaults to 5s
LIB	The chaos lib used to inject the chaos	Defaults to `litmus`. Supported `litmus` only
FORCE	Application Pod deletion mode. `False` indicates graceful deletion with default termination period of 30s. `true` indicates an immediate forceful deletion with 0s grace period	Default to `true`, With `terminationGracePeriodSeconds=0`
RAMP_TIME	Period to wait before injection of chaos in sec

Experiment Examples¶

Common Experiment Tunables¶

Refer the common attributes to tune the common tunables for all the experiments.

Cassandra App Details¶

It tunes the cassandra service name at CASSANDRA_SVC_NAME and cassandra port at CASSANDRA_PORT.

Use the following example to tune this:

## contains details of cassandra application
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "cassandra"
    applabel: "app=cassandra"
    appkind: "statefulset"
  chaosServiceAccount: casssandra-pod-delete-sa
  experiments:
  - name: casssandra-pod-delete
    spec:
      components:
        env:
        # name of the cassandra service
        - name: CASSANDRA_SVC_NAME
          value: 'cassandra'
        # name of the cassandra port
        - name: CASSANDRA_PORT
          value: '9042'
        # percentage of cassandra replicas with matching labels
        - name: PODS_AFFECTED_PERC
          value: '100'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Force Delete¶

The cassandra pod can be deleted forcefully or gracefully. It can be tuned with the FORCE env. It will delete the pod forcefully if FORCE is provided as true and it will delete the pod gracefully if FORCE is provided as false.

Use the following example to tune this:

## force env provided to forcefully or gracefully delete the pod
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "cassandra"
    applabel: "app=cassandra"
    appkind: "statefulset"
  chaosServiceAccount: casssandra-pod-delete-sa
  experiments:
  - name: casssandra-pod-delete
    spec:
      components:
        env:
        # deletes the cassandra pod forcefully or gracefully
        # supports: true, false. default: false
        - name: FORCE
          value: 'true'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Liveness check of cassandra¶

The cassandra liveness can be tuned with CASSANDRA_LIVENESS_CHECK env. Provide CASSANDRA_LIVENESS_CHECK as enabled to enable the liveness check and provide CASSANDRA_LIVENESS_CHECK as disabled to skip the liveness check. The default value is disabled.
The cassandra liveness image can be provided at CASSANDRA_LIVENESS_IMAGE.
The cassandra liveness pod performs the CRUD operations to verify the liveness of cassandra. It creates the keyspace with KEYSPACE_REPLICATION_FACTOR keyspace factor.

Use the following example to tune this:

## enable the cassandra liveness check, while injecting chaos
## it continuosly performs cassandra database operations(with cqlsh command) to vefify the liveness status
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "cassandra"
    applabel: "app=cassandra"
    appkind: "statefulset"
  chaosServiceAccount: casssandra-pod-delete-sa
  experiments:
  - name: casssandra-pod-delete
    spec:
      components:
        env:
        # checks the liveness of cassandra while injecting chaos
        # supports: enabled, disabled. default: disabled
        - name: CASSANDRA_LIVENESS_CHECK
          value: 'enabled'
        # image of the cassandra liveness deployment
        - name: CASSANDRA_LIVENESS_IMAGE
          value: 'litmuschaos/cassandra-client:latest'
        # keyspace replication factor, needed for liveness check
        - name: KEYSPACE_REPLICATION_FACTOR
          value: '3'
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'

Multiple Iterations Of Chaos¶

The multiple iterations of chaos can be tuned via setting CHAOS_INTERVAL ENV. Which defines the delay between each iteration of chaos.

Use the following example to tune this:

# defines delay between each successive iteration of the chaos
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: engine-nginx
spec:
  engineState: "active"
  annotationCheck: "false"
  appinfo:
    appns: "cassandra"
    applabel: "app=cassandra"
    appkind: "statefulset"
  chaosServiceAccount: casssandra-pod-delete-sa
  experiments:
  - name: casssandra-pod-delete
    spec:
      components:
        env:
        # delay between each iteration of chaos
        - name: CHAOS_INTERVAL
          value: '15'
        # time duration for the chaos execution
        - name: TOTAL_CHAOS_DURATION
          VALUE: '60'