<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.apple-tab-span
{mso-style-name:apple-tab-span;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="color:#1F497D">Greetings,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D">We have not seen a reply from ARIN yet on this ticket, I’m sending to the tech-discuss list to see if anyone else has experienced this today and can offer some guidance? Please note that our NOC isn’t on this
distro, so as you reply, please copy </span><span style="color:black"><a href="mailto:noc@softlayer.com">noc@softlayer.com</a></span><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:black">----<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:black">Dani Roisman<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:black">VP, Network Operations<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="color:black"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:black">SoftLayer, an IBM Company<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:black">315 Capitol Street Suite 205, Houston, TX 77002
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> ARIN Ticketing System [mailto:do-not-reply@arin.net]
<br>
<b>Sent:</b> Saturday, November 16, 2013 10:33<br>
<b>To:</b> Ticket Logger<br>
<b>Cc:</b> NOC - SoftLayer<br>
<b>Subject:</b> Re:[ARIN-20131116.48] SERVFAIL replies in 174.in-addr.arpa. - Ref Ticket 7871618<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">There appears to have been some sort of key roll over issue with 174.in-addr.arpa. This is causing SERVFAIL replies from validating DNS resolvers for anything
in 174.in-addr.arpa.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">The in-addr.arpa zone contains a single DS key with the ID 7353. This is true for the zone that is currently being served. Truncated dig(1) output:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">174.in-addr.arpa.<span class="apple-tab-span">
</span> 86400<span class="apple-tab-span"> </span>IN</span><span class="apple-tab-span"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">DS<span class="apple-tab-span">
</span>7353 5 1 2096BD9C5612836870ADCCBDD68A957CB6487F34</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">;; Received 70 bytes from 2001:500:87::87#53(2001:500:87::87) in 60 ms<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">This same record appears on line 946 (byte offset: 85175) in the in-addr.arpa zone file that's available
<a href="ftp://ftp.internic.net">ftp.internic.net</a> (time stamp: Nov 16 04:34):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">174.in-addr.arpa.<span class="apple-tab-span">
</span> 86400<span class="apple-tab-span"> </span>IN</span><span class="apple-tab-span"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">DS<span class="apple-tab-span">
</span>7353 5 1 2096BD9C5612836870ADCCBDD68A957CB6487F34</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Unfortunately, the two keys that are available in the 174.in-addr.arpa zone, as served by z.arin.net, have id's of 19034 and 39420:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">174.in-addr.arpa.<span class="apple-tab-span">
</span> 14400 IN DNSKEY<span class="apple-tab-span"> </span>256 3 5 (<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">BQEAAAABtReA/SI3OKJpKhRDt5FfaXAsrQSO9a2zpxVY<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">ZqP0z+ONdbw7aBaibwSC5Itly1foVXeZbTx3TkF8AAT3<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">0Aa/8au4KV26TDQ6o3awY087f7rspZld8OR/fh44HV3o<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">puzIkeyh8PEb91rkCkAjkxqNavuEbL1OzT8wf8x+2L3p<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">x5c=<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">) ; key id = 19034<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">174.in-addr.arpa.<span class="apple-tab-span">
</span> 14400 IN DNSKEY<span class="apple-tab-span"> </span>257 3 5 (<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">BQEAAAABqELZSV55GgyN/BvBbtLlz7xzP341SyAcBqkI<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">87QVGiFt3PzKdNlw7NU5BeZ9Oo7aMih/5afr2RfZB50M<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">cREP1TUhCOf2WSoCA2l7VFpC/eG0NO1EFSiqmxJVOJ61<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">XNZG0c0yPvmvz6jVbwLUGeIzEOHQy2bZNYc9lzP7fbKQ<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">4WLIwqQnS7iwkVaZtmub0WI+2ybCe27Xr+W/b1CwhTww<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">mgCJegmb/xQMGwF3zR059i+adsPy8meveGKczf7R26mV<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">pyynDJHFDTdBzVQH2ubGCZ3Qfnby1xq6xoNkcMUGMopQ<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">mHDcPQFW2IZf667ijh2oSrns18vugXlFPtT33IFMNw==<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">) ; key id = 39420<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">I have verified my findings with two DNSSEC validation tools:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Sandia National Labs:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://dnsviz.net/d/174.in-addr.arpa/dnssec/">http://dnsviz.net/d/174.in-addr.arpa/dnssec/</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Verisign:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://dnssec-debugger.verisignlabs.com/174.in-addr.arpa">http://dnssec-debugger.verisignlabs.com/174.in-addr.arpa</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Finally, ICANN's in-addr.arpa DNSSEC Report for 15 and 16 Nov indicate an issue with the zone (Signed NO, DS Key YES):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">15th:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://stats.research.icann.org/dns/inaddr_report/archive/20131115.003001.html">http://stats.research.icann.org/dns/inaddr_report/archive/20131115.003001.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">16th:<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://stats.research.icann.org/dns/inaddr_report/archive/20131116.003001.html">http://stats.research.icann.org/dns/inaddr_report/archive/20131116.003001.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">While the report from the 14th says everything was working fine (Signed YES, DS Key YES):<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><a href="http://stats.research.icann.org/dns/inaddr_report/archive/20131114.003002.html">http://stats.research.icann.org/dns/inaddr_report/archive/20131114.003002.html</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">I don't have historical copies of the zones to know for sure what has happened, but my educated guess is that the DNSKEY records changed in 174.in-addr.arpa during
key roll-over, but the DS key in the parent zone hasn't been updated. <o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">If that is what has happened, then either publishing a new DS key in the parent (in-addr.arpa.) zone, or publishing 7353 dns key and re-signing 174.in-addr.arpa
zone correct the issue.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif";color:black">Thank you for your time and attention to this matter.<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="color:black">---<o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="color:black">Christopher Price<o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D">Network Operations Center </span>
<span style="font-size:12.0pt;font-family:"Times New Roman","serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><b><span style="color:black"><o:p> </o:p></span></b></p>
<p class="MsoNormal"><b><span style="color:#1F497D">SoftLayer, an IBM Company</span></b><b><span style="font-size:12.0pt;color:black"><o:p></o:p></span></b></p>
<p class="MsoNormal"><span style="color:#1F497D">+1(281)714-3555 direct | +1(214)442-0600 main |
<a href="mailto:noc@softlayer.com">noc@softlayer.com</a></span><span style="font-family:"Times New Roman","serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>