<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Menlo;
panose-1:2 11 6 9 3 8 4 2 2 4;}
@font-face
{font-family:"Times New Roman \(Body CS\)";
panose-1:2 2 6 3 5 4 5 2 3 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:Menlo;
color:windowtext;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:10.5pt;font-family:Menlo;color:black">RPKI RRDP Service Degradation<span class="apple-converted-space"> </span><br>
<br>
Incident window : 11:20 AM - 12:50 PM ET on 11 August 2022</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
This morning, ARIN renewed SSL certificates within our infrastructure that caused<span class="apple-converted-space"> </span><br>
suboptimal performance of the RPKI RRDP services run by ARIN.<br>
<br>
</span><span style="font-size:10.5pt;font-family:Symbol;color:black">·</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:black"> </span></span><span style="font-size:10.5pt;font-family:Menlo;color:black">At 11:20
AM, a configuration management change installed a new certificate and<span class="apple-converted-space"> </span><br>
keys on nodes that serve the RPKI RRDP repository. A subset of these nodes<span class="apple-converted-space"> </span><br>
received a mismatched CA certificate and key. This triggered the degraded<span class="apple-converted-space"> </span><br>
performance of the RPKI RRDP services.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
</span><span style="font-size:10.5pt;font-family:Symbol;color:black">·</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:black"> </span></span><span style="font-size:10.5pt;font-family:Menlo;color:black">At 11:45
AM, repository generation was paused during the process of diagnosing<span class="apple-converted-space"> </span><br>
the issue.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
</span><span style="font-size:10.5pt;font-family:Symbol;color:black">·</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:black"> </span></span><span style="font-size:10.5pt;font-family:Menlo;color:black">At 12:05
PM, the misconfigured nodes were identified and removed from DNS<span class="apple-converted-space"> </span><br>
rotation.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
</span><span style="font-size:10.5pt;font-family:Symbol;color:black">·</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:black"> </span></span><span style="font-size:10.5pt;font-family:Menlo;color:black">At 12:40
PM, new CA certificates and keys had been pushed to the impacted<span class="apple-converted-space"> </span><br>
systems and they were returned to DNS rotation.</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
</span><span style="font-size:10.5pt;font-family:Symbol;color:black">·</span><span class="apple-converted-space"><span style="font-size:10.5pt;font-family:Menlo;color:black"> </span></span><span style="font-size:10.5pt;font-family:Menlo;color:black">At 12:50
PM, after confirmation that the systems were running normally, the<span class="apple-converted-space"> </span><br>
repository generation was restarted and full functionality of the RPKI RRDP<span class="apple-converted-space"> </span><br>
services was restored<br>
<br>
RPKI rsync services were fully functional throughout the incident.<span class="apple-converted-space"> </span><br>
The publication of 6 ROAs were delayed during the incident. </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.5pt;font-family:Menlo;color:black"><br>
Please note that ARIN has a Services Status page which can be found<span class="apple-converted-space"> </span><br>
at <a href="https://arin.statuspage.io/" title="https://arin.statuspage.io/"><span style="font-family:"Calibri",sans-serif">https://arin.statuspage.io/</span></a><span class="apple-converted-space"> </span>or via the link in the footer of ARIN’s website. This
link is<span class="apple-converted-space"> </span><br>
also visible when logged in to your ARIN Online account. We encourage our customers<span class="apple-converted-space"> </span><br>
to subscribe to the Services Status page to receive notifications on service-impacting<span class="apple-converted-space"> </span><br>
issues.<br>
<br>
Regards,<br>
<br>
Brad Gorman<br>
Senior Product Owner, Routing Security<br>
American Registry for Internet Numbers</span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.0pt;font-family:Menlo;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="font-size:10.0pt;font-family:Menlo;color:black"> </span><span style="color:black"><o:p></o:p></span></p>
<p class="MsoNormal" style="caret-color: rgb(0, 0, 0);font-variant-caps: normal;orphans: auto;text-align:start;widows: auto;-webkit-text-size-adjust: auto;-webkit-text-stroke-width: 0px;word-spacing:0px">
<span style="color:black"> <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:Menlo"><o:p> </o:p></span></p>
</div>
</body>
</html>