[arin-tech-discuss] RPKI RRDP Service Degradation incident - 11 August 2022

[Brad Gorman]

RPKI RRDP Service Degradation

Incident window : 11:20 AM - 12:50 PM ET on 11 August 2022

This morning, ARIN renewed SSL certificates within our infrastructure that caused
suboptimal performance of the RPKI RRDP services run by ARIN.

• At 11:20 AM, a configuration management change installed a new certificate and
keys on nodes that serve the RPKI RRDP repository.  A subset of these nodes
received a mismatched CA certificate and key. This triggered the degraded
performance of the RPKI RRDP services.

• At 11:45 AM, repository generation was paused during the process of diagnosing
the issue.

• At 12:05 PM, the misconfigured nodes were identified and removed from DNS
rotation.

• At 12:40 PM, new CA certificates and keys had been pushed to the impacted
systems and they were returned to DNS rotation.

• At 12:50 PM, after confirmation that the systems were running normally, the
repository generation was restarted and full functionality of the RPKI RRDP
services was restored

RPKI rsync services were fully functional throughout the incident.
The publication of 6 ROAs were delayed during the incident.

Please note that ARIN has a Services Status page which can be found
at https://arin.statuspage.io/ or via the link in the footer of ARIN’s website. This link is
also visible when logged in to your ARIN Online account. We encourage our customers
to subscribe to the Services Status page to receive notifications on service-impacting
issues.

Regards,

Brad Gorman
Senior Product Owner, Routing Security
American Registry for Internet Numbers




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.arin.net/pipermail/arin-tech-discuss/attachments/20220811/3c6e49d5/attachment.htm>