Autoscaling In Openshift
So you have scaled pods up manually using the web interface or:
oc scale --replicas=3 dc/<my-project>
but now you want to ensure that this autoscales for when you aren’t around and will increase and decrease based on load
HorizontalPodAutoscaler#
A HorizontalPodAutoscaler object needs to be added to your project
The horizontal pod autoscaler computes the ratio of the current metric utilization with the desired metric utilization, and scales up or down accordingly
You should ensure your pods has gone through readiness checks to ensure it is ready for scaling
You should know the max and min pods to run beforehand:
oc autoscale dc/<project> --min 1 --max 10 --cpu-percent=80
Get the horizontal pod autoscalers
oc get hpa
It should show you min, max, target and current
$ oc get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
blog DeploymentConfig/blog <unknown>/80% 1 10 3 1m
More detailed info:
oc describe hpa/blog
If you get errors like:
ScalingActive False FailedGetResourceMetric the HPA was unable to compute the replica count: unable to get metrics for resource cpu: unable to fetch metrics from resource metrics API: the server could not find the requested resource (get pods.metrics.k8s.io)
...failed to get cpu utilization: unable to get metrics for resource cpu
then you should request that your Openshift cluster admin enable cluster metrics
Once you have set that up by modifying the metrics cluster install configuration
oc adm diagnostics MetricsApiProxy --loglevel 10
Debugging no access to the metrics API#
To check the deployed services use:
kubectl get apiservices
We are looking for metrics.k8s.io in the output, if it is not there then it might not be registered
Get the metrics related pods with:
kubectl get pods -n kube-system
To check resource consumption you should be able to:
kubectl top
but in my case it would just give help info and this message
This command requires Heapster to be correctly configured and working on the server.
Using the docs#
You can verify the metrics were installed correctly with:
oc adm top node
oc adm top pod
You can also get all the apis with
oc get --raw /apis/
which I think is the same as
kubectl get apiservices
I decided to redeploy the metrics API with ansible and didn’t set a specific hostname
oc adm diagnostics MetricsApiProxy
I still get this error:
ERROR: [DClu4003 from diagnostic MetricsApiProxy@openshift/origin/pkg/oc/cli/admin/diagnostics/diagnostics/cluster/metrics.go:89]
Unable to access the metrics API Proxy endpoint /api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/metrics:
(*errors.StatusError) the server could not find the requested resource
The Horizontal Pod Autoscaler is not able to retrieve metrics to drive scaling.
You can check that you can get the metrics for a pod with:
oc adm top pod --heapster-namespace='openshift-infra' --heapster-scheme='https' -n demo
you don’t need those flags though:
oc adm top pod -n demo
Configure a 80 milicore limit for CPU requests
oc patch dc/guestbook -p '{"spec":{"template":{"spec":{"containers":[{"name":"guestbook","resources":{"limits":{"cpu":"80m"}}}]}}}}'
oc autoscale dc/guestbook --min 1 --max 3 --cpu-percent=20
Get the object like any other openshift object
oc get hpa guestbook -o yaml -n myproject
In the pod artificial stress is added with:
seq 3 | xargs -0 -n1 timeout -t 60 md5sum /dev/zero
This reddit post describes the same issue and answers declare that it is insecure certs
To view the logs on openshift I used:
kubectl logs hawkular-metrics-ks6sb --namespace=openshift-infra
It seems to still be the api aggregation issue…that it was not setup correctly.
metrics.k8s.io
Can check diagnostics of the entire cluster with:
oc adm diagnostics
Get metrics pods with:
oc get pods -n openshift-infra
Checking the pod:
oc describe pod hawkular-metrics-ks6sb -n openshift-infra
Gave me a bunch of errors:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 1h default-scheduler Successfully assigned openshift-infra/hawkular-metrics-ks6sb to openshift.example.co.za
Normal Pulled 1h kubelet, openshift.example.co.za Container image "docker.io/openshift/origin-metrics-hawkular-metrics:v3.11.0" already present on machine
Normal Created 1h kubelet, openshift.example.co.za Created container
Normal Started 1h kubelet, openshift.example.co.za Started container
Warning Unhealthy 1h kubelet, openshift.example.co.za Liveness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>.
Traceback (most recent call last):
File "/opt/hawkular/scripts/hawkular-metrics-liveness.py", line 48, in <module>
if int(uptime) < int(timeout):
ValueError: invalid literal for int() with base 10: ''
Warning Unhealthy 1h (x3 over 1h) kubelet, openshift.example.co.za Readiness probe failed: Failed to access the status endpoint : <urlopen error [Errno 111] Connection refused>. This may be due to Hawkular Metrics not being ready yet. Will try again.
Warning Unhealthy 1h kubelet, openshift.example.co.za Readiness probe failed: Failed to access the status endpoint : timed out. This may be due to Hawkular Metrics not being ready yet. Will try again.
Warning Unhealthy 1h (x3 over 1h) kubelet, openshift.example.co.za Readiness probe failed: The MetricService is not yet in the STARTED state [STARTING]. We need to wait until its in the STARTED state.
So I try to access heapster directly with:
curl -X GET https://${KUBERNETES_MASTER}/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/metrics
And I get back:
{
"kind": "Status",
"apiVersion": "v1",
"metadata": {
},
"status": "Failure",
"message": "services \"https:heapster:\" is forbidden: User \"system:anonymous\" cannot proxy services in the namespace \"openshift-infra\": proxy verb changed to unsafeproxy\nno RBAC policy matched, proxy verb changed to unsafeproxy",
"reason": "Forbidden",
"details": {
"name": "https:heapster:",
"kind": "services"
},
"code": 403
}
The redhat forums also highlight this issue and give the following diagnosis steps:
oc adm top pod -n hooks
oc get --raw="/api/v1/proxy/namespaces/openshift-infra/services/https:heapster:/api/v1/model/metrics"
oc get --raw="/api/v1/namespaces/openshift-infra/services/https:heapster:/proxy/api/v1/model/metrics"
The second request works, furthermore the bugzilla report notes that the incorrect url is used.
So the solution is to upgrade to a later version, use this guide and ensure to reboot the host:
ansible-playbook playbooks/byo/openshift-cluster/upgrades/v3_11/upgrade.yml -e openshift_certificate_expiry_warning_days=30
So now it works.
oc describe hpa
Now gives successful
Type Status Reason Message
---- ------ ------ -------
AbleToScale True ReadyForNewScale the last scale time was sufficiently old as to warrant a new scale
ScalingActive True ValidMetricFound the HPA was able to successfully calculate a replica count from cpu resource utilization (percentage of request)
ScalingLimited True TooFewReplicas the desired replica count is increasing faster than the maximum scale rate
and the hpa now shows the current cpu usage:
[root@openshift ~]# oc get hpa
NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE
blog DeploymentConfig/blog 0%/50% 1 10 1 2d
You need to set CPU request and CPU Limit.
Sources#
Remember to get the correct version of the documentation - the same as the system you are running