[arin-tech-discuss] Issue for Delegated Users within ARIN's RPKI Repository - Outage Report

[Job Snijders]

Dear Mark,

On Mon, Nov 23, 2020 at 09:32:53PM +0000, Mark Kosters wrote:
> On Nov 19 at 2:30PM EST (UTC-5), ARIN updated the software that generates the RPKI repository.
> On Nov 20 at 9:48PM EST (UTC-5), we were notified by a 3rd party that validators no longer were fetching ROAs from organizations that had selected the delegated option.

Can you elaborate on why it appears there was a delay between the
software update having taken place, and the problem becoming visible?

>From my measurements the problem became visible at 19:22 UTC on November
20nd. The RPKI stack from an end-to-end perspective is an interesting
waterfall of timers, the above question is for my own edification on how
this all works.

> Upon review, ARIN Engineering discovered that a certificate was not included in the manifest for each delegated organization.
> The fix was to include that certificate in the manifest for each delegated organization was deployed at 1:20AM EST (UTC-5) on Nov 21.

A fix was deployed on November ***22nd***, right?

> After Action Items
> 
> ARIN will add additional delegated repository tests to prevent this
> type of operational issue to happen again. Additionally, as planned,
> ARIN will be adding additional improvements to its external monitoring
> that uses various validators to ensure that the repository is working
> as intended.

This is welcome news!

Kind regards,

Job