[arin-tech-discuss] Issue for Delegated Users within ARIN's RPKI Repository - Outage Report

[Mark Kosters]

On 11/24/20, 6:30 AM, "Job Snijders" <job at ntt.net> wrote:

    Dear Mark,
    
    On Mon, Nov 23, 2020 at 09:32:53PM +0000, Mark Kosters wrote:
    > On Nov 19 at 2:30PM EST (UTC-5), ARIN updated the software that generates the RPKI repository.
    > On Nov 20 at 9:48PM EST (UTC-5), we were notified by a 3rd party that validators no longer were fetching ROAs from organizations that had selected the delegated option.
    
    Can you elaborate on why it appears there was a delay between the
    software update having taken place, and the problem becoming visible?
    
    From my measurements the problem became visible at 19:22 UTC on November
    20nd. The RPKI stack from an end-to-end perspective is an interesting
    waterfall of timers, the above question is for my own edification on how
    this all works.

If I got my timing right, looks like you must have received the updated repository as we were pushing out the software updates.
    
    > Upon review, ARIN Engineering discovered that a certificate was not included in the manifest for each delegated organization.
    > The fix was to include that certificate in the manifest for each delegated organization was deployed at 1:20AM EST (UTC-5) on Nov 21.
    
    A fix was deployed on November ***22nd***, right?

Good catch.
    
Thanks,
Mark