We recently released a feature which allowed users to create read-only service bindings for Postgres databases. This change prevented users from restoring databases from backups without modifying the "search_path" setting within the backup. This change also prevented applications from starting if they overrode the "search_path" setting to not include "public".
We have updated our broker to support these use cases, and now we are confident that the broker is working as normal. Bindings will work as expected before the update. If your applications were affected by the "search_path" issue mentioned above, then you should unbind and rebind them to the service.
We plan on resolving this incident tomorrow morning.
If you have noticed any irregular behaviour with the service broker which is not covered by the two cases mentioned above, please contact support via https://admin.london.cloud.service.gov.uk/support
Regards,
Toby Lorne GOV.UK PaaS team
Posted Sep 22, 2020 - 16:28 BST
Monitoring
Hello,
We are still working to resolve the issue caused by a change in how we manage Postgres permissions.
Since our last update, we are updating our documentation on how to take backups and restore data to Postgres databases - in this case it will require new tenant action before restoring.
We’ll continue to update you as we know more and have updated the documentation.
Once again, we’re sorry for the inconvenience that this has caused to you and your users.
We have identified that this issue is caused by a change in how we manage Postgres permissions.
Dumping and restoring data to Postgres databases is broken at present for all users. However, app deploys have only failed for one particular tenant.
We will carry on addressing this issue tomorrow. In the meantime, we suggest not performing unnecessary deploys. If you do need to deploy, it is very unlikely your deploy will be affected. It should not lead to downtime if you are deploying in zero-downtime way, such as:
- Blue-green deploy (creating a new app, and only sending traffic to the new app if the deploy succeeds.) You are using this approach if using the Autopilot cf plugin
- V3 zero-downtime-deploys. You are using this approach if passing the "--strategy=rolling" argument to "cf push"
Fixing this issue is our priority. We will be doing everything we can to address this issue tomorrow. Due to its complexity and the limited impact we do not want to risk fixing it out-of-hours.
We’ll continue to update you as we know more, and we’ll let you know as soon as the problem has been resolved. We’re sorry for the inconvenience that this may cause to your team.
If you need to contact us for help or anything else, please email us via gov-uk-paas-support@digital.cabinet-office.gov.uk. If this has a critical impact to a live service out-of-hours please use the emergency contact details.
Regards, Miki Mokrysz GOV.UK PaaS team
Posted Sep 21, 2020 - 17:12 BST
Identified
Hello,
We are aware of and investigating a problem with GOV.UK PaaS.
Recent changes to how we automate Postgres have caused problems for some users. This may affect your ability to:
- Binding apps to Postgres databases, or redeploying apps is failing for a small number of users - Dumping and restoring the contents of Postgres databases does not work - Unless you encounter the above problems your users should be unaffected
We’re looking into this as a matter of urgency and will update you as soon as we know more.
Regards,
Miki Mokrysz GOV.UK PaaS team
Posted Sep 21, 2020 - 16:13 BST
This incident affected: Ireland (Apps - availability of tenant applications to end users, Postgres Backing service - availability of platform-provided services) and London (Apps - availability of tenant applications to end users, Postgres Backing service - availability of platform-provided services).