[arin-tech-discuss] Issue for Delegated Users within ARIN's RPKI Repository - Outage Report

[Christopher Morrow]

thanks mark! :)

On Mon, Nov 23, 2020 at 4:34 PM Mark Kosters <markk at arin.net> wrote:
>
> Summary
>
> On Nov 19 at 2:30PM EST (UTC-5), ARIN updated the software that generates the RPKI repository.   On Nov 20 at 9:48PM EST (UTC-5), we were notified by a 3rd party that validators no longer were fetching ROAs from organizations that had selected the delegated option.  Upon review, ARIN Engineering discovered that a certificate was not included in the manifest for each delegated organization. The fix was to include that certificate in the manifest for each delegated organization was deployed at 1:20AM EST (UTC-5) on Nov 21.  At that time, ROAs from the affected delegated repositories could then again be fetched and validated.
>
> ARIN's hosted RPKI customers were not affected by this outage in any way.
>
> Root Cause
>
> The root cause of this failure was a software bug that was introduced by the RPKI repository generator.
>
> Scope of Issue
>
> This bug meant that validators would not fetch information from the delegated repositories during the affected period.  ARIN has nine delegated organizations and affected approximately 180 ROAs that may have disappeared from the global RPKI system for approximately 35 hours and 40 minutes starting on Nov 19 at 2:30PM EST (UTC-5). Depending on how validation is setup by the ISPs who use RPKI, the route origins associated with these 180 ROA’s may have remained in the secure state or became unsecure during this period.
>

I think 'unknown' and maybe (if there were ROA for supernets possibly
invalid :( oops.

> After Action Items
>
> ARIN will add additional delegated repository tests to prevent this type of operational issue to happen again. Additionally, as planned, ARIN will be adding additional improvements to its external monitoring that uses various validators to ensure that the repository is working as intended.
>

Do you all document what is tested? so either RP software folk can
integrate similar tests (or suggest others), and/or so actual ARIN
users of the RPKI can test their (or instrument their) repository
collections?

> Regards,
> Mark Kosters
> ARIN CTO
>
> _______________________________________________
> arin-tech-discuss mailing list
> arin-tech-discuss at arin.net
> https://lists.arin.net/mailman/listinfo/arin-tech-discuss