Elasticsearch and InfluxDB backing service management actions may time out

Incident Report for GOV.UK Platform as a Service (PaaS)

Resolved

This incident has been resolved.

Posted Jul 10, 2020 - 18:00 BST

Monitoring

You should now be able to make changes to your Elasticsearch and InfluxDB backing services as normal.

As of 3pm this afternoon, Aiven's API latency returned to normal, and we are no longer observing requests timing out.

We expect to resolve the incident by 6pm this evening, or once we hear that Aiven have resolved their incident.

We’re sorry for any inconvenience this has caused.

If you need to contact us, please email us via gov-uk-paas-support@digital.cabinet-office.gov.uk

Regards,

Toby Lorne Welch-Richards
Senior Site Reliability Engineer
GOV.UK PaaS team

Posted Jul 10, 2020 - 16:04 BST

Identified

Aiven.io is a service provider we use to provide Elasticsearch and InfluxDB backing services.

The Aiven API is currently returning responses with increased latency, which result in timed out requests.

Your applications will continue to work as normal, your Elasticsearch and InfluxDB backing services are working as normal.

However:
- Creating a new Elasticsearch or InfluxDB backing service may fail with a timeout
- Updating or deleting an existing Elasticsearch or InfluxDB backing service may fail with a timeout
- Binding/unbinding an application to/from an existing Elasticsearch or InfluxDB backing service may fail with a timeout (this may prevent application deploys, when using the autopilot plugin)
- Newly created Elasticsearch instances will not have metrics available until incident is resolved

We are monitoring the availability of the Aiven.io API. We will update when we know more.

Posted Jul 10, 2020 - 11:38 BST

This incident affected: Ireland (Elasticsearch Backing Service - availability of platform-provided services via Aiven, InfluxDB Backing Service - availability of platform-provided services via Aiven, Metrics - availability of platform-provided metrics) and London (Elasticsearch Backing Service - availability of platform-provided services via Aiven, InbfluxDB Backing Service - availability of platform-provided services via Aiven, Metrics - availability of platform-provided metrics).