[arin-tech-discuss] FW: [ARIN-20131116.48] SERVFAIL replies in 174.in-addr.arpa. - Ref Ticket 7871618

[Dani Roisman]

Greetings,

We have not seen a reply from ARIN yet on this ticket, I'm sending to the tech-discuss list to see if anyone else has experienced this today and can offer some guidance?  Please note that our NOC isn't on this distro, so as you reply, please copy noc at softlayer.com<mailto:noc at softlayer.com>


----
Dani Roisman
VP, Network Operations

SoftLayer, an IBM Company
315 Capitol Street Suite 205, Houston, TX 77002


From: ARIN Ticketing System [mailto:do-not-reply at arin.net]
Sent: Saturday, November 16, 2013 10:33
To: Ticket Logger
Cc: NOC - SoftLayer
Subject: Re:[ARIN-20131116.48] SERVFAIL replies in 174.in-addr.arpa. - Ref Ticket 7871618

There appears to have been some sort of key roll over issue with 174.in-addr.arpa.  This is causing SERVFAIL replies from validating DNS resolvers for anything in 174.in-addr.arpa.

The in-addr.arpa zone contains a single DS key with the ID 7353.  This is true for the zone that is currently being served.  Truncated dig(1) output:

174.in-addr.arpa.  86400 IN
DS 7353 5 1 2096BD9C5612836870ADCCBDD68A957CB6487F34
;; Received 70 bytes from 2001:500:87::87#53(2001:500:87::87) in 60 ms

This same record appears on line 946 (byte offset: 85175) in the in-addr.arpa zone file that's available ftp.internic.net<ftp://ftp.internic.net> (time stamp: Nov 16 04:34):

174.in-addr.arpa.  86400 IN
DS 7353 5 1 2096BD9C5612836870ADCCBDD68A957CB6487F34


Unfortunately, the two keys that are available in the 174.in-addr.arpa zone, as served by z.arin.net, have id's of 19034 and 39420:

174.in-addr.arpa.  14400 IN DNSKEY 256 3 5 (
BQEAAAABtReA/SI3OKJpKhRDt5FfaXAsrQSO9a2zpxVY
ZqP0z+ONdbw7aBaibwSC5Itly1foVXeZbTx3TkF8AAT3
0Aa/8au4KV26TDQ6o3awY087f7rspZld8OR/fh44HV3o
puzIkeyh8PEb91rkCkAjkxqNavuEbL1OzT8wf8x+2L3p
x5c=
) ; key id = 19034
174.in-addr.arpa.  14400 IN DNSKEY 257 3 5 (
BQEAAAABqELZSV55GgyN/BvBbtLlz7xzP341SyAcBqkI
87QVGiFt3PzKdNlw7NU5BeZ9Oo7aMih/5afr2RfZB50M
cREP1TUhCOf2WSoCA2l7VFpC/eG0NO1EFSiqmxJVOJ61
XNZG0c0yPvmvz6jVbwLUGeIzEOHQy2bZNYc9lzP7fbKQ
4WLIwqQnS7iwkVaZtmub0WI+2ybCe27Xr+W/b1CwhTww
mgCJegmb/xQMGwF3zR059i+adsPy8meveGKczf7R26mV
pyynDJHFDTdBzVQH2ubGCZ3Qfnby1xq6xoNkcMUGMopQ
mHDcPQFW2IZf667ijh2oSrns18vugXlFPtT33IFMNw==
) ; key id = 39420
I have verified my findings with two DNSSEC validation tools:

Sandia National Labs:
http://dnsviz.net/d/174.in-addr.arpa/dnssec/

Verisign:
http://dnssec-debugger.verisignlabs.com/174.in-addr.arpa


Finally, ICANN's in-addr.arpa DNSSEC Report for 15 and 16 Nov indicate an issue with the zone (Signed NO, DS Key YES):

15th:
http://stats.research.icann.org/dns/inaddr_report/archive/20131115.003001.html

16th:
http://stats.research.icann.org/dns/inaddr_report/archive/20131116.003001.html

While the report from the 14th says everything was working fine (Signed YES, DS Key YES):
http://stats.research.icann.org/dns/inaddr_report/archive/20131114.003002.html

I don't have historical copies of the zones to know for sure what has happened, but my educated guess is that the DNSKEY records changed in 174.in-addr.arpa during key roll-over, but the DS key in the parent zone hasn't been updated.

If that is what has happened, then either publishing a new DS key in the parent (in-addr.arpa.) zone, or publishing 7353 dns key and re-signing 174.in-addr.arpa zone correct the issue.


Thank you for your time and attention to this matter.

---
Christopher Price
Network Operations Center

SoftLayer, an IBM Company
+1(281)714-3555 direct | +1(214)442-0600 main | noc at softlayer.com<mailto:noc at softlayer.com>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.arin.net/pipermail/arin-tech-discuss/attachments/20131117/b74106b1/attachment.html>