[arin-tech-discuss] FW: [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today..

[Nate Davis]

ARIN's DNS process moves DNS data from the internal database to a Secure64
DNSSEC appliance to a hidden distribution master. From the hidden
distribution master, zones are fetched to name server constellations from
ARIN,
VeriSign, and PCH.

About two weeks ago a script was run that reset the serial on a zone in
the database. This script was run to accommodate an inter-RIR network
transfer, and is not executed during the normal course of operations. It
reset the serial in our database in an unexpected way, and consequently
zone transfers from the Secure64 to our distribution master did not occur.

This script was cumbersome and error prone, and had already been
identified to be replaced in the upcoming, planned deployment this weekend.

This incident exposed a gap in our monitoring that we are fixing. Our
current, legacy monitoring system does not adequately identify the serial
number inconsistencies between the DNS nodes, nor does it adequately
handle issues with DNSSEC signature validation. We have work underway to
replace our old monitoring system with a new system that solves these
problems.

This update is being posted to both arin-ppml and arin-tech-discuss.  To
avoid non-policy related discussion on PPML, we encourage follow up
discussion on arin-tech-discuss, a public mailing list that ARIN¹s
engineering team
monitors.  For those not familiar with arin-tech-discuss, please subscribe
here:
http://lists.arin.net/mailman/listinfo/arin-tech-discuss

Regards,

Nate Davis


>On 3/8/16, 11:05 AM, "arin-ppml-bounces at arin.net on behalf of Chris
>Woodfield" <arin-ppml-bounces at arin.net on behalf of chris at semihuman.com>
>wrote:
>
>>Agreed with Chris¹ sentiment. I¹m a firm believer in the blameless
>>post-mortem particularly when paired with action items to avoid repeat
>>occurrences, and I¹d hope that others can learn from the technical issues
>>involved.
>>
>>On top of that, everyone loves a good war story :)
>>
>>Thanks,
>>
>>-C
>>
>>> On Mar 8, 2016, at 7:45 AM, Christopher Morrow
>>><christopher.morrow at gmail.com> wrote:
>>> 
>>> Also, i'd be super awesome if there would be a pretty detailed
>>> post-mortem document published about what happened, how it happened
>>> and how it was discovered/repaired.
>>> 
>>> I believe ARIN isn't the only one having these issues, so publishing
>>> so other folk can learn would be great!
>>> 
>>> -crhis
>>> 
>>> On Mon, Mar 7, 2016 at 10:28 PM,  <frnkblk at iname.com> wrote:
>>>> Nate,
>>>> 
>>>> Please let us know if ARIN monitors all their zones for DNSSEC
>>>>signature
>>>> expiration.
>>>> 
>>>> Frank
>>>> 
>>>> -----Original Message-----
>>>> From: arin-ppml-bounces at arin.net [mailto:arin-ppml-bounces at arin.net]
>>>>On
>>>> Behalf Of Nate Davis
>>>> Sent: Monday, March 07, 2016 7:59 PM
>>>> To: Michael Peddemors <michael at linuxmagic.com>; arin-ppml at arin.net
>>>> Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. )
>>>>outages
>>>> today..
>>>> 
>>>> Michael - thanks for reporting the issue.
>>>> 
>>>> ARIN Engineering resolved the DNSSEC failure shortly after you
>>>>reported
>>>> the issue. They are currently looking into the cause of the failure.
>>>>All
>>>> DNSSEC functions should be operating properly at this time.
>>>> 
>>>> Regards,
>>>> 
>>>> Nate Davis
>>>> Chief Operating Officer
>>>> American Registry for Internet Numbers
>>>> 
>>>> 
>>>> 
>>>> 
>>>> On 3/7/16, 6:14 PM, "arin-ppml-bounces at arin.net on behalf of Michael
>>>> Peddemors" <arin-ppml-bounces at arin.net on behalf of
>>>> michael at linuxmagic.com> wrote:
>>>> 
>>>>> We had a flurry of reports from various customers, problems with
>>>>>reverse
>>>>> DNS lookups..
>>>>> 
>>>>> Limited to the 65/8 IPv4, and from apparent reports, related to a
>>>>> failure to update a DNSSEC signature..
>>>>> 
>>>>> Reported: Anyone with a DNSSEC enforced name server will have
>>>>>problems
>>>>> with PTR queries for that range.
>>>>> 
>>>>> Someone with more inside knowledge can provide more details, I am
>>>>>sure..
>>>>> 
>>>>> 
>>>>> 
>>>>> --
>>>>> "Catch the Magic of Linux..."
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>-
>>>>>-
>>>>> Michael Peddemors, President/CEO LinuxMagic Inc.
>>>>> Visit us at http://www.linuxmagic.com @linuxmagic
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>-
>>>>>-
>>>>> A Wizard IT Company - For More Info http://www.wizard.ca
>>>>> "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices
>>>>>Ltd.
>>>>> 
>>>>>----------------------------------------------------------------------
>>>>>-
>>>>>-
>>>>> 604-682-0300 Beautiful British Columbia, Canada
>>>>> 
>>>>> This email and any electronic data contained are confidential and
>>>>>intended
>>>>> solely for the use of the individual or entity to which they are
>>>>> addressed.
>>>>> Please note that any views or opinions presented in this email are
>>>>>solely
>>>>> those of the author and are not intended to represent those of the
>>>>> company.
>>>>> 
>>>>> _______________________________________________
>>>>> PPML
>>>>> You are receiving this message because you are subscribed to
>>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net).
>>>>> Unsubscribe or manage your mailing list subscription at:
>>>>> http://lists.arin.net/mailman/listinfo/arin-ppml
>>>>> Please contact info at arin.net if you experience any issues.
>>>> 
>>>> _______________________________________________
>>>> PPML
>>>> You are receiving this message because you are subscribed to
>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net).
>>>> Unsubscribe or manage your mailing list subscription at:
>>>> http://lists.arin.net/mailman/listinfo/arin-ppml
>>>> Please contact info at arin.net if you experience any issues.
>>>> 
>>>> 
>>>> _______________________________________________
>>>> PPML
>>>> You are receiving this message because you are subscribed to
>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net).
>>>> Unsubscribe or manage your mailing list subscription at:
>>>> http://lists.arin.net/mailman/listinfo/arin-ppml
>>>> Please contact info at arin.net if you experience any issues.
>>> _______________________________________________
>>> PPML
>>> You are receiving this message because you are subscribed to
>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net).
>>> Unsubscribe or manage your mailing list subscription at:
>>> http://lists.arin.net/mailman/listinfo/arin-ppml
>>> Please contact info at arin.net if you experience any issues.
>>> 
>>
>>_______________________________________________
>>PPML
>>You are receiving this message because you are subscribed to
>>the ARIN Public Policy Mailing List (ARIN-PPML at arin.net).
>>Unsubscribe or manage your mailing list subscription at:
>>http://lists.arin.net/mailman/listinfo/arin-ppml
>>Please contact info at arin.net if you experience any issues.
>