Install/Upgrade notes for GetVisibility Essentials chart
- 1 4.5.145
- 1.1 Redis volume
- 2 4.5.130
- 2.1 CRD PostgreSQL
- 3 4.5.119
- 4 4.2.1
- 5 4.0.1
- 6 4.0.0
- 7 3.2.0
- 8 3.1.2
- 9 3.1.1
- 10 3.1.0
- 10.1 FIXES AND ENHANCEMENTS
- 10.2 KNOWN ISSUES
4.5.145
Redis volume
Rolled back Redis volume size tom 8 GB. If tenant had 4.5.144 installed with 16 GB, k3s can block upgrade with message:
cannot patch "gv-essentials-redis-master" with kind StatefulSet: StatefulSet.apps "gv-essentials-redis-master" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
If that is the case, delete Redis PVC redis-data-gv-essentials-redis-master-0
and retry the upgrade.
4.5.130
CRD PostgreSQL
Could be that there is an error while upgrading to that version.
helm upgrade --history-max=5 --install=true --namespace=default --timeout=10m0s --values=/home/shell/helm/values-gv-essentials-4.5.130.yaml --version=4.5.130 --wait=true gv-essentials /home/shell/helm/gv-essentials-4.5.130.tgz
Error: UPGRADE FAILED: error validating "": error validating data: [ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "enable_finalizers" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "enable_owner_references" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "enable_persistent_volume_claim_deletion" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "enable_secrets_deletion" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "pdb_master_label_selector" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.kubernetes): unknown field "persistent_volume_claim_retention_policy" in do.zalan.acid.v1.OperatorConfiguration.configuration.kubernetes, ValidationError(OperatorConfiguration.configuration.logical_backup): unknown field "logical_backup_cronjob_environment_secret" in do.zalan.acid.v1.OperatorConfiguration.configuration.logical_backup, ValidationError(OperatorConfiguration.configuration.logical_backup): unknown field "logical_backup_s3_bucket_prefix" in do.zalan.acid.v1.OperatorConfiguration.configuration.logical_backup]
Please then install Support Tools (gv-debug) at least in version 0.2.32 and make sure you checked:
Expected output would be then
>>>>>>>>>>> Upgrade PostgreSQL CRDs
Warning: resource customresourcedefinitions/operatorconfigurations.acid.zalan.do is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/operatorconfigurations.acid.zalan.do configured
Warning: resource customresourcedefinitions/postgresqls.acid.zalan.do is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/postgresqls.acid.zalan.do configured
Warning: resource customresourcedefinitions/postgresteams.acid.zalan.do is missing the kubectl.kubernetes.io/last-applied-configuration annotation which is required by kubectl apply. kubectl apply should only be used on resources created declaratively by either kubectl create --save-config or kubectl apply. The missing annotation will be patched automatically.
customresourcedefinition.apiextensions.k8s.io/postgresteams.acid.zalan.do configured
Then re-install gv-essentails
4.5.119
FIX in gv-debug
Elastic search CRD under helm control. If cluster does not get updates from fleet follow this guide
CRD elastic issue root cause
In the past we used to install CRD for elastic manually, now we want them being managed by the helm chart eck-operator 3.0.0 · elastic/elastic
However they are missing some labels, we added a job to the client-config from where fleet should push it to the cluster. Sometimes clusters are paused and not receiving fleet updates. Then when you try to upgrade to latest gv-essential you will have an error
Error: UPGRADE FAILED: rendered manifests contain a resource that already exists. Unable to continue with update: CustomResourceDefinition "agents.agent.k8s.elastic.co" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; label validation error: missing key "app.kubernetes.io/managed-by": must be set to "Helm"; annotation validation error: missing key "meta.helm.sh/release-name": must be set to "gv-essentials"; annotation validation error: missing key "meta.helm.sh/release-namespace": must be set to "default"
Fix - relabel CRDs
To label CRDs please install gv-support-tool chart.
Make sure to check customize helm chart
On default page make sure you enabled
Install and check logs
Pod which runs the script will be named like
k logs gv-support-tool-XXXX
logs should look like that
Processing CRD: agents.agent.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/agents.agent.k8s.elastic.co not labeled
Processing CRD: apmservers.apm.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/apmservers.apm.k8s.elastic.co not labeled
Processing CRD: beats.beat.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/beats.beat.k8s.elastic.co not labeled
Processing CRD: elasticmapsservers.maps.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticmapsservers.maps.k8s.elastic.co not labeled
Processing CRD: elasticsearchautoscalers.autoscaling.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticsearchautoscalers.autoscaling.k8s.elastic.co not labeled
Processing CRD: elasticsearches.elasticsearch.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/elasticsearches.elasticsearch.k8s.elastic.co not labeled
Processing CRD: enterprisesearches.enterprisesearch.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/enterprisesearches.enterprisesearch.k8s.elastic.co not labeled
Processing CRD: kibanas.kibana.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/kibanas.kibana.k8s.elastic.co not labeled
Processing CRD: logstashes.logstash.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/logstashes.logstash.k8s.elastic.co not labeled
Processing CRD: stackconfigpolicies.stackconfigpolicy.k8s.elastic.co
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co annotate
customresourcedefinition.apiextensions.k8s.io/stackconfigpolicies.stackconfigpolicy.k8s.elastic.co not labeled
Successfully processed all CRDs.
That should be it, then you can upgrade gv-essential
4.2.1
Defined CPU consumption limit for the following services
Kafka
Elasticsearch
PR reference: https://github.com/GetVisibility/rancher-charts/pull/136
4.0.1
FIXES AND ENHANCEMENTS
Elasticsearch configuration UI improvements and bugfixes
4.0.0
ADDED
Enabled Elasticsearch for all clusters
Elasticsearch is planned to replace Postgres as the main data store for all products (including Synergy). We’re working to provide some degree of automation to update many clusters at once, but feel free to use latest 4.x version for manual upgrades if needed.
KNOWN ISSUES
Elasticsearch kind missing
When upgrading gv-essentials
installed manually with Helm to version 4.x you may get the error Error: UPGRADE FAILED: unable to recognize "": no matches for kind "Elasticsearch"...
.
This is because Helm cannot determine whether upgrading a CRD will break other things in the cluster, therefore, Helm v3 by design can create CRDs but is not able not upgrade them.
To address this issue install the CRDs manually:
kubectl apply -f https://assets.master.k3s.getvisibility.com/k3s/misc/elastic.yaml
In airgapped deployments, you can upload the elastic.yaml and run the following command before upgrading the chart:
kubectl apply -f elastic.yaml
Not enough memory
Wheen upgrading gv-essentials on a machine with 32 GB RAM via helm (without Rancher), Elasticsearch can get misconfigured to use 20 GB RAM, and the cluster will run out of memory. In this case, please amke sure you adjust the resource variables manually:
helm upgrade --install gv-essentials gv_stable/gv-essentials --wait \
--timeout=10m0s --kubeconfig /etc/rancher/k3s/k3s.yaml \
--set eck-operator.enabled=true --set eck-operator.settings.cpu=1 \
--set eck-operator.settings.memory=1
Decreasing storage size is not supported: an attempt was made to decrease storage size for claim elasticsearch-data
A volume for ES data was already created in this cluster, and it’s larger than entered size. Current size can be checked in Storage -> Persistent Volumes -> elasticsearch-data-gv-es-default-0
or via kubectl get pv
, but usually it's 160 GB.
For helm installations, use --set eck-operator.settings.disk=160 --set eck-operator.settings.storage=160
3.2.0
FIXES AND ENHANCEMENTS
Configured SaaS mode
3.1.2
FIXES AND ENHANCEMENTS
Actually disable heap dumps on OOM in Elasticsearch
3.1.1
FIXES AND ENHANCEMENTS
DIsabled heap dumps on OOM in Elasticsearch
3.1.0
FIXES AND ENHANCEMENTS
Upgrade Zalando Postgres Operator to version
1.10.0
Make chart compatible with Kubernetes 1.25.x
KNOWN ISSUES
When upgrading gv-essentials
to version 3.1.x you’ll get the error Error: UPGRADE FAILED: error validating "": error validating data...
.
This is because Helm cannot determine whether upgrading a CRD will break other things in the cluster, therefore, Helm v3 by design can create CRDs but is not able not upgrade them.
To address this issue run the following command before upgrading the chart:
kubectl apply -f https://assets.master.k3s.getvisibility.com/k3s/misc/crd_operatorconfigurations_v1_10_0.yml
Or you can download the file from the internet, save it locally and run the command kubectl apply -f
crd_operatorconfigurations_v1_10_0.yml
from within the cluster.
Classified as Getvisibility - Partner/Customer Confidential