From pete at arin.net Thu Mar 3 15:53:47 2016 From: pete at arin.net (Pete Toscano) Date: Thu, 3 Mar 2016 20:53:47 +0000 Subject: [arin-tech-discuss] OT&E Refresh Scheduled for 3 March In-Reply-To: <34CC7F34-346C-4B5D-B131-A4A168759718@arin.net> References: <34CC7F34-346C-4B5D-B131-A4A168759718@arin.net> Message-ID: The update that was planned for today has been postponed. The database refresh will happen on its normal day, Monday, 7 March. Regards, Pete > On Feb 29, 2016, at 2:46 PM, Pete Toscano wrote: > > On Thursday, 3 March, ARIN's Operational Test and Evaluation environment (OT&E) will be updated to mirror production data. > > This is a one-time change to the OT&E refresh date. Further OT&E refreshes will normally occur on the first Monday of each month. For more OT&E information, visit: > > https://www.arin.net/resources/ote.html > > Please send any questions, comments, or issues to hostmaster at arin.net. > > Regards, > > Pete Toscano > Network Operations Manager > American Registry for Internet Numbers (ARIN) > _______________________________________________ > arin-tech-discuss mailing list > arin-tech-discuss at arin.net > http://lists.arin.net/mailman/listinfo/arin-tech-discuss From ndavis at arin.net Wed Mar 9 12:30:37 2016 From: ndavis at arin.net (Nate Davis) Date: Wed, 9 Mar 2016 17:30:37 +0000 Subject: [arin-tech-discuss] FW: [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today.. In-Reply-To: References: <56DE0B3C.60903@linuxmagic.com> <000001d178ea$958fe190$c0afa4b0$@iname.com> <37E922BF-B613-4248-96AF-83EB0B8990D2@semihuman.com> Message-ID: ARIN's DNS process moves DNS data from the internal database to a Secure64 DNSSEC appliance to a hidden distribution master. From the hidden distribution master, zones are fetched to name server constellations from ARIN, VeriSign, and PCH. About two weeks ago a script was run that reset the serial on a zone in the database. This script was run to accommodate an inter-RIR network transfer, and is not executed during the normal course of operations. It reset the serial in our database in an unexpected way, and consequently zone transfers from the Secure64 to our distribution master did not occur. This script was cumbersome and error prone, and had already been identified to be replaced in the upcoming, planned deployment this weekend. This incident exposed a gap in our monitoring that we are fixing. Our current, legacy monitoring system does not adequately identify the serial number inconsistencies between the DNS nodes, nor does it adequately handle issues with DNSSEC signature validation. We have work underway to replace our old monitoring system with a new system that solves these problems. This update is being posted to both arin-ppml and arin-tech-discuss. To avoid non-policy related discussion on PPML, we encourage follow up discussion on arin-tech-discuss, a public mailing list that ARIN?s engineering team monitors. For those not familiar with arin-tech-discuss, please subscribe here: http://lists.arin.net/mailman/listinfo/arin-tech-discuss Regards, Nate Davis >On 3/8/16, 11:05 AM, "arin-ppml-bounces at arin.net on behalf of Chris >Woodfield" >wrote: > >>Agreed with Chris? sentiment. I?m a firm believer in the blameless >>post-mortem particularly when paired with action items to avoid repeat >>occurrences, and I?d hope that others can learn from the technical issues >>involved. >> >>On top of that, everyone loves a good war story :) >> >>Thanks, >> >>-C >> >>> On Mar 8, 2016, at 7:45 AM, Christopher Morrow >>> wrote: >>> >>> Also, i'd be super awesome if there would be a pretty detailed >>> post-mortem document published about what happened, how it happened >>> and how it was discovered/repaired. >>> >>> I believe ARIN isn't the only one having these issues, so publishing >>> so other folk can learn would be great! >>> >>> -crhis >>> >>> On Mon, Mar 7, 2016 at 10:28 PM, wrote: >>>> Nate, >>>> >>>> Please let us know if ARIN monitors all their zones for DNSSEC >>>>signature >>>> expiration. >>>> >>>> Frank >>>> >>>> -----Original Message----- >>>> From: arin-ppml-bounces at arin.net [mailto:arin-ppml-bounces at arin.net] >>>>On >>>> Behalf Of Nate Davis >>>> Sent: Monday, March 07, 2016 7:59 PM >>>> To: Michael Peddemors ; arin-ppml at arin.net >>>> Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. ) >>>>outages >>>> today.. >>>> >>>> Michael - thanks for reporting the issue. >>>> >>>> ARIN Engineering resolved the DNSSEC failure shortly after you >>>>reported >>>> the issue. They are currently looking into the cause of the failure. >>>>All >>>> DNSSEC functions should be operating properly at this time. >>>> >>>> Regards, >>>> >>>> Nate Davis >>>> Chief Operating Officer >>>> American Registry for Internet Numbers >>>> >>>> >>>> >>>> >>>> On 3/7/16, 6:14 PM, "arin-ppml-bounces at arin.net on behalf of Michael >>>> Peddemors" >>> michael at linuxmagic.com> wrote: >>>> >>>>> We had a flurry of reports from various customers, problems with >>>>>reverse >>>>> DNS lookups.. >>>>> >>>>> Limited to the 65/8 IPv4, and from apparent reports, related to a >>>>> failure to update a DNSSEC signature.. >>>>> >>>>> Reported: Anyone with a DNSSEC enforced name server will have >>>>>problems >>>>> with PTR queries for that range. >>>>> >>>>> Someone with more inside knowledge can provide more details, I am >>>>>sure.. >>>>> >>>>> >>>>> >>>>> -- >>>>> "Catch the Magic of Linux..." >>>>> >>>>>---------------------------------------------------------------------- >>>>>- >>>>>- >>>>> Michael Peddemors, President/CEO LinuxMagic Inc. >>>>> Visit us at http://www.linuxmagic.com @linuxmagic >>>>> >>>>>---------------------------------------------------------------------- >>>>>- >>>>>- >>>>> A Wizard IT Company - For More Info http://www.wizard.ca >>>>> "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices >>>>>Ltd. >>>>> >>>>>---------------------------------------------------------------------- >>>>>- >>>>>- >>>>> 604-682-0300 Beautiful British Columbia, Canada >>>>> >>>>> This email and any electronic data contained are confidential and >>>>>intended >>>>> solely for the use of the individual or entity to which they are >>>>> addressed. >>>>> Please note that any views or opinions presented in this email are >>>>>solely >>>>> those of the author and are not intended to represent those of the >>>>> company. >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact info at arin.net if you experience any issues. >>>> >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact info at arin.net if you experience any issues. >>>> >>>> >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact info at arin.net if you experience any issues. >>> _______________________________________________ >>> PPML >>> You are receiving this message because you are subscribed to >>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>> Unsubscribe or manage your mailing list subscription at: >>> http://lists.arin.net/mailman/listinfo/arin-ppml >>> Please contact info at arin.net if you experience any issues. >>> >> >>_______________________________________________ >>PPML >>You are receiving this message because you are subscribed to >>the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>Unsubscribe or manage your mailing list subscription at: >>http://lists.arin.net/mailman/listinfo/arin-ppml >>Please contact info at arin.net if you experience any issues. > From ndavis at arin.net Wed Mar 9 13:20:48 2016 From: ndavis at arin.net (Nate Davis) Date: Wed, 9 Mar 2016 18:20:48 +0000 Subject: [arin-tech-discuss] FW: [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today.. In-Reply-To: References: <56DE0B3C.60903@linuxmagic.com> <000001d178ea$958fe190$c0afa4b0$@iname.com> <37E922BF-B613-4248-96AF-83EB0B8990D2@semihuman.com> Message-ID: On 3/9/16, 11:34 AM, "Christopher Morrow" wrote: >Thanks! >(I have a few questions, which may not be answerable here, I suppose.. >if they can be answered that'd be cool though) > >On Tue, Mar 8, 2016 at 12:59 PM, Nate Davis wrote: >> >> ARIN's DNS process moves DNS data from the internal database to a >>Secure64 >> DNSSEC appliance to a hidden distribution master. From the hidden >> distribution >> master, zones are fetched to name server constellations from ARIN, >> VeriSign, and PCH. >> >> About two weeks ago a script was run that reset the serial on a zone in >> the database. This script was run to accommodate an inter-RIR network > >This script sounds like something that should/would happen >periodically? (whenever there's an xfer I guess?) is that correct? > >> transfer, and is not executed during the normal course of operations. It >> reset the serial in our database in an unexpected way, and consequently >> zone transfers from the Secure64 to our distribution master did not >>occur. >> > >'unexpected way' was decreased the serial? made it a string not an >integer? other? >(ie: Can I dork up my zone by setting the serial in the same fashion? >what should I look for?) > >> This script was cumbersome and error prone, and had already been >> identified to be replaced in the upcoming, planned deployment this >>weekend. >> > >neat, ok. > >> This incident exposed a gap in our monitoring that we are fixing. Our > >is/was the gap: "Make sure serial is monotonically increasing" >or is/was it: "If you are going to backup the serial, be sure to force >a reload on all masters via process X" > >(ie: If I make a serial change, what other things should I look for? >what monitoring gap do I also have?) > >> current, legacy monitoring system does not adequately identify the >>serial >> number inconsistencies between the DNS nodes, nor does it adequately >> handle issues with DNSSEC signature validation. We have work underway to >> replace our old monitoring system with a new system that solves these >> problems. > >The legacy/current system should be doing the moral equivalane of: > for s in $(dig +short NS zone); do > dig SOA +short zone @${s} > done > >and make sure that all servers agree that the serial/soa is the same... >right? >Was there other verification that was happening? (or not) >is the above too naive? should we be looking for other things? > >For dnssec I suppose you'd be doing the above but pulling rrsig for >the SOA and making sure they are all the same. > >> This update is being posted to both arin-ppml and arin-tech-discuss. To >> avoid non-policy related discussion on PPML, we encourage follow up >> discussion >> on arin-tech-discuss, a public mailing list that ARIN?s engineering team >> monitors. For those not >> familiar with arin-tech-discuss, please subscribe here: >> http://lists.arin.net/mailman/listinfo/arin-tech-discuss >> > >oh :) > >> Regards, >> >> Nate Davis >> >> >> On 3/8/16, 11:05 AM, "arin-ppml-bounces at arin.net on behalf of Chris >> Woodfield" >> wrote: >> >>>Agreed with Chris? sentiment. I?m a firm believer in the blameless >>>post-mortem particularly when paired with action items to avoid repeat >>>occurrences, and I?d hope that others can learn from the technical >>>issues >>>involved. >>> >>>On top of that, everyone loves a good war story :) >>> >>>Thanks, >>> >>>-C >>> >>>> On Mar 8, 2016, at 7:45 AM, Christopher Morrow >>>> wrote: >>>> >>>> Also, i'd be super awesome if there would be a pretty detailed >>>> post-mortem document published about what happened, how it happened >>>> and how it was discovered/repaired. >>>> >>>> I believe ARIN isn't the only one having these issues, so publishing >>>> so other folk can learn would be great! >>>> >>>> -crhis >>>> >>>> On Mon, Mar 7, 2016 at 10:28 PM, wrote: >>>>> Nate, >>>>> >>>>> Please let us know if ARIN monitors all their zones for DNSSEC >>>>>signature >>>>> expiration. >>>>> >>>>> Frank >>>>> >>>>> -----Original Message----- >>>>> From: arin-ppml-bounces at arin.net [mailto:arin-ppml-bounces at arin.net] >>>>>On >>>>> Behalf Of Nate Davis >>>>> Sent: Monday, March 07, 2016 7:59 PM >>>>> To: Michael Peddemors ; arin-ppml at arin.net >>>>> Subject: Re: [arin-ppml] Just so it is recorded here (DNSSEC.. ) >>>>>outages >>>>> today.. >>>>> >>>>> Michael - thanks for reporting the issue. >>>>> >>>>> ARIN Engineering resolved the DNSSEC failure shortly after you >>>>>reported >>>>> the issue. They are currently looking into the cause of the failure. >>>>>All >>>>> DNSSEC functions should be operating properly at this time. >>>>> >>>>> Regards, >>>>> >>>>> Nate Davis >>>>> Chief Operating Officer >>>>> American Registry for Internet Numbers >>>>> >>>>> >>>>> >>>>> >>>>> On 3/7/16, 6:14 PM, "arin-ppml-bounces at arin.net on behalf of Michael >>>>> Peddemors" >>>> michael at linuxmagic.com> wrote: >>>>> >>>>>> We had a flurry of reports from various customers, problems with >>>>>>reverse >>>>>> DNS lookups.. >>>>>> >>>>>> Limited to the 65/8 IPv4, and from apparent reports, related to a >>>>>> failure to update a DNSSEC signature.. >>>>>> >>>>>> Reported: Anyone with a DNSSEC enforced name server will have >>>>>>problems >>>>>> with PTR queries for that range. >>>>>> >>>>>> Someone with more inside knowledge can provide more details, I am >>>>>>sure.. >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> "Catch the Magic of Linux..." >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> Michael Peddemors, President/CEO LinuxMagic Inc. >>>>>> Visit us at http://www.linuxmagic.com @linuxmagic >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> A Wizard IT Company - For More Info http://www.wizard.ca >>>>>> "LinuxMagic" a Registered TradeMark of Wizard Tower TechnoServices >>>>>>Ltd. >>>>>> >>>>>>--------------------------------------------------------------------- >>>>>>-- >>>>>>- >>>>>> 604-682-0300 Beautiful British Columbia, Canada >>>>>> >>>>>> This email and any electronic data contained are confidential and >>>>>>intended >>>>>> solely for the use of the individual or entity to which they are >>>>>> addressed. >>>>>> Please note that any views or opinions presented in this email are >>>>>>solely >>>>>> those of the author and are not intended to represent those of the >>>>>> company. >>>>>> >>>>>> _______________________________________________ >>>>>> PPML >>>>>> You are receiving this message because you are subscribed to >>>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>>>> Unsubscribe or manage your mailing list subscription at: >>>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>>> Please contact info at arin.net if you experience any issues. >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact info at arin.net if you experience any issues. >>>>> >>>>> >>>>> _______________________________________________ >>>>> PPML >>>>> You are receiving this message because you are subscribed to >>>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>>> Unsubscribe or manage your mailing list subscription at: >>>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>>> Please contact info at arin.net if you experience any issues. >>>> _______________________________________________ >>>> PPML >>>> You are receiving this message because you are subscribed to >>>> the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>> Unsubscribe or manage your mailing list subscription at: >>>> http://lists.arin.net/mailman/listinfo/arin-ppml >>>> Please contact info at arin.net if you experience any issues. >>>> >>> >>>_______________________________________________ >>>PPML >>>You are receiving this message because you are subscribed to >>>the ARIN Public Policy Mailing List (ARIN-PPML at arin.net). >>>Unsubscribe or manage your mailing list subscription at: >>>http://lists.arin.net/mailman/listinfo/arin-ppml >>>Please contact info at arin.net if you experience any issues. >> From markk at arin.net Wed Mar 9 15:50:14 2016 From: markk at arin.net (Mark Kosters) Date: Wed, 9 Mar 2016 20:50:14 +0000 Subject: [arin-tech-discuss] FW: [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today.. In-Reply-To: References: <56DE0B3C.60903@linuxmagic.com> <000001d178ea$958fe190$c0afa4b0$@iname.com> <37E922BF-B613-4248-96AF-83EB0B8990D2@semihuman.com> Message-ID: Hi Chris Answers in-line: On 3/9/16, 1:20 PM, "arin-tech-discuss-bounces at arin.net on behalf of Nate Davis" wrote: >On 3/9/16, 11:34 AM, "Christopher Morrow" >wrote: > >>Thanks! >>(I have a few questions, which may not be answerable here, I suppose.. >>if they can be answered that'd be cool though) >> >>On Tue, Mar 8, 2016 at 12:59 PM, Nate Davis wrote: >>> >>> ARIN's DNS process moves DNS data from the internal database to a >>>Secure64 >>> DNSSEC appliance to a hidden distribution master. From the hidden >>> distribution >>> master, zones are fetched to name server constellations from ARIN, >>> VeriSign, and PCH. >>> >>> About two weeks ago a script was run that reset the serial on a zone in >>> the database. This script was run to accommodate an inter-RIR network >> >>This script sounds like something that should/would happen >>periodically? (whenever there's an xfer I guess?) is that correct? Not even that frequently. It only needs to be run when we initially set up a /8 for out-of-region transfers. This marks the /8 in our system so that we can start doing things like retrieving, validating, and aggregating the RIR snippets to put into our published zone file, and eventually do the right things to work with RPKI and so on. Of course, after this weekend?s deploy, we will no longer need to run this script as the system will automatically detect this and mark the zone. >>> This incident exposed a gap in our monitoring that we are fixing. Our >> >>is/was the gap: "Make sure serial is monotonically increasing" >>or is/was it: "If you are going to backup the serial, be sure to force >>a reload on all masters via process X" >> >>(ie: If I make a serial change, what other things should I look for? >>what monitoring gap do I also have?) No, it was the soa checking went from the distribution master out to the anycast cloud. We have had incidents in the past where various nodes where not fetching the latest zone within a reasonable interval. So, we added checks that would make sure the soa would update within a "reasonable interval". If the node did not update within a reasonable interval, on-call people got notified to escalate. Unfortunately, we did not do the same monitoring going on internally within our provisioning flow. We did not monitor appropriately for our internal nodes. That has now been fixed. >>For dnssec I suppose you'd be doing the above but pulling rrsig for >>the SOA and making sure they are all the same. What we want to do is to catch it before the sig expires. Do you have any ideas? Thanks, Mark From jcurran at arin.net Thu Mar 10 07:15:30 2016 From: jcurran at arin.net (John Curran) Date: Thu, 10 Mar 2016 12:15:30 +0000 Subject: [arin-tech-discuss] [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today.. In-Reply-To: References: <56DE0B3C.60903@linuxmagic.com> <000001d178ea$958fe190$c0afa4b0$@iname.com> <37E922BF-B613-4248-96AF-83EB0B8990D2@semihuman.com> Message-ID: On Mar 9, 2016, at 8:50 PM, Mark Kosters wrote: > ... >>> For dnssec I suppose you'd be doing the above but pulling rrsig for >>> the SOA and making sure they are all the same. > > What we want to do is to catch it before the sig expires. Do you have any > ideas? Mark - How often is that refreshed and what the is signature lifetime? /John From markk at arin.net Thu Mar 10 10:22:37 2016 From: markk at arin.net (Mark Kosters) Date: Thu, 10 Mar 2016 15:22:37 +0000 Subject: [arin-tech-discuss] [arin-ppml] Just so it is recorded here (DNSSEC.. ) outages today.. In-Reply-To: References: <56DE0B3C.60903@linuxmagic.com> <000001d178ea$958fe190$c0afa4b0$@iname.com> <37E922BF-B613-4248-96AF-83EB0B8990D2@semihuman.com>

Message-ID: Hi John On 3/10/16, 7:15 AM, "John Curran" wrote: >On Mar 9, 2016, at 8:50 PM, Mark Kosters wrote: >> ... >>>> For dnssec I suppose you'd be doing the above but pulling rrsig for >>>> the SOA and making sure they are all the same. >> >> What we want to do is to catch it before the sig expires. Do you have >>any >> ideas? > >Mark - > > How often is that refreshed and what the is signature lifetime? In the normal course of operations, zones are generated six times a day to accommodate zone snippets from other RIRs. These snippets are included in the zone, signed, and pushed out to the authoritative servers from the distribution master. Any changes made to the zone between the zone generation intervals are pushed out by ixfr. Regardless if what time it is, if you make any delegation changes within ARIN Online, these changes are normally reflected on our authoritative servers within five minutes. The DNSSEC signatures are currently set to expire 14 days and 1 hour from the time signed by the Secure64 box. Thanks, Mark