[arin-tech-discuss] (1) Some serious issues with the REST service and (2) Strange inconsistencies in the database mappings
Andy Newton
andy at arin.net
Tue Dec 31 14:42:08 EST 2013
Eric,
Thanks for your message and sorry about the confusion.
First let me address your primary need, which is to gather data based on an IP address. While /ip/XXXX/orgs is not a supported query, /ip/XXXX/pft is and it will return the information you seek on both the organization and the POCs. Using the example address you gave, http://whois.arin.net/rest/ip/209.144.225.62/pft.txt will return the network registration, the associated organization, and the abuse contact you seek all in one query. Hopefully that will reduce the complexity of your code and address the bulk of your frustration.
With regard to the issue of HTML being returned when JSON, TXT, or XML are requested, this is a known bug in the REST library upon which we rely. However, under these conditions our REST service does produce a proper HTTP status code of 404. When using curl, the –f flag will produce a 404 and suppress the output, which should ease the burden on your script.
Finally, our current plans for Whois information over HTTP/REST include the adoption of the RDAP protocol under development in the IETF. It is currently envisioned that all 5 RIRs and many domain registrars will deploy this service. When that time comes, programmatically retrieving data should be simpler and more uniform and thus reduce the complexity of your codebase.
I hope this helps.
Happy New Year!
Andy Newton,
Chief Engineer, ARIN
From: Eric Dynamic <ecsd at transbay.net<mailto:ecsd at transbay.net>>
Date: Tuesday, December 31, 2013 11:54 AM
To: "arin-tech-discuss at arin.net<mailto:arin-tech-discuss at arin.net>" <arin-tech-discuss at arin.net<mailto:arin-tech-discuss at arin.net>>
Subject: [arin-tech-discuss] (1) Some serious issues with the REST service and (2) Strange inconsistencies in the database mappings
The REST info page:
4.3.3. MIME Types and File Extensions
The following table lists the data types and their associated MIME types and file extensions:
Data type Current Version MIME Type Version 1 MIME Type File Extension
XML application/xml application/arin.whoisrws-v1+xml xml
JSON application/json application/arin.whoisrws-v1+json json
plain text text/plain txt
HTML text/html html
Problems:
(1) formatting issue.
http://whois.arin.net/rest/org/ATL-83/pocs.txt
returns
http://whois.arin.net/rest/poc/NOC3056-ARINhttp://whois.arin.net/rest/poc/NOC3056-ARINhttp://whois.arin.net/rest/poc/JCL171-ARINhttp://whois.arin.net/rest/poc/NOC3056-ARIN
As opposed to what one can "swipe" from the web page:
Abuse NOC3056-ARIN (NOC3056-ARIN<http://whois.arin.net/rest/poc/NOC3056-ARIN.html>)
NOC NOC3056-ARIN (NOC3056-ARIN<http://whois.arin.net/rest/poc/NOC3056-ARIN.html>)
Tech NOC3056-ARIN (NOC3056-ARIN<http://whois.arin.net/rest/poc/NOC3056-ARIN.html>)
Admin JCL171-ARIN (JCL171-ARIN<http://whois.arin.net/rest/poc/JCL171-ARIN.html>)
So, the TXT output in this case has no line breaks and omits the function descriptions. Other similar TXT output more closely resembles what the web page shows.
(2) missing information.
Via the web, the URL http://whois.arin.net/rest/ip/209.144.225.62/orgs returns:
NetRange 209.144.225.0 - 209.144.225.255
CIDR 209.144.225.0/24
Name SAVV-S218155-6
Handle NET-209-144-225-0-1
Parent SAVVIS4 (NET-209-144-0-0-1<http://whois.arin.net/rest/net/NET-209-144-0-0-1.html>)
Net Type Reallocated
Origin AS
Organization AppServe Technologies, LLC (ATL-83<http://whois.arin.net/rest/org/ATL-83.html>)
Registration Date 2007-09-20
Last Updated 2007-09-20
Comments
RESTful Link http://whois.arin.net/rest/net/NET-209-144-225-0-1
See Also Related POC records.<http://whois.arin.net/rest/net/NET-209-144-225-0-1/pocs>
See Also Related organization's POC records.<http://whois.arin.net/rest/org/ATL-83/pocs>
See Also Related delegations.<http://whois.arin.net/rest/net/NET-209-144-225-0-1/rdns>
However, from the Unix command line
curl -H "Accept: text/plain" http://whois.arin.net/rest/ip/209.144.225.62.txt
returns this data:
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
NetRange: 209.144.225.0 - 209.144.225.255
CIDR: 209.144.225.0/24
OriginAS:
NetName: SAVV-S218155-6
NetHandle: NET-209-144-225-0-1
Parent: NET-209-144-0-0-1
NetType: Reallocated
RegDate: 2007-09-20
Updated: 2007-09-20
Ref: http://whois.arin.net/rest/net/NET-209-144-225-0-1
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
Notice that there is no Organization Name. Ouch! for my purposes.
(3A) Incorrect formats returned;
(4A)Dead-end maze to get Organization Name due to apparent database mapping inconsistencies:
I am trying to write a program which, given an IP address, returns
* the associated NetRange, e.g. "209.144.225.0 - 209.144.225.255"
* the Abuse POC associated with that netrange.
Not getting the Organization Name in the command line call is making this a maze to accomplish, a dead-end as we'll see:
If I do this:
# curl -H "Accept: text/plain" http://whois.arin.net/rest/ip/209.144.225.62.txt
#
# ARIN WHOIS data and services are subject to the Terms of Use
# available at: https://www.arin.net/whois_tou.html
#
NetRange: 209.144.225.0 - 209.144.225.255
CIDR: 209.144.225.0/24
OriginAS:
NetName: SAVV-S218155-6
NetHandle: NET-209-144-225-0-1
Parent: NET-209-144-0-0-1
NetType: Reallocated
RegDate: 2007-09-20
Updated: 2007-09-20
Ref: http://whois.arin.net/rest/net/NET-209-144-225-0-1
And then this:
# curl -H "Accept: text/plain" http://whois.arin.net/rest/net/NET-209-144-225-0-1/pocs.txt
http://whois.arin.net/rest/poc/JJ772-ARIN
And then this:
curl -H "Accept: text/plain" http://whois.arin.net/rest/poc/JJ772-ARIN/orgs.txt
This is the output I get:
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01//EN" "http://www.w3.org/TR/html4/strict.dtd"<http://www.w3.org/TR/html4/strict.dtd>>
<html xmlns:rdns="http://www.arin.net/whoisrws/rdns/v1"<http://www.arin.net/whoisrws/rdns/v1> xmlns:pft="http://www.arin.net/whoisrws/pft/v1"<http://www.arin.net/whoisrws/pft/v1> xmlns:whois="http://www.arin.net/whoisrws/core/v1"<http://www.arin.net/whoisrws/core/v1>>
<head>
<META http-equiv="Content-Type" content="text/html; charset=UTF-8">
<title>Whois-RWS</title>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta content="Virginia" name="geo.placename">
<meta content="USA" name="geo.country">
<meta name="viewport" content="width=990">
<meta name="Keywords" content="">
<meta name="Description" content="">
<style type="text/css">
... VERY MUCH HTML SUPPRESSED ...
<div id="content">
<h2 class="noauth">Whois-RWS </h2>
<div id="maincontent">
<p style="margin:0;padding:0;">
<table>
<tr>
<th>No Related Resources</th>
</tr>
<tr>
<td>No related resources were found for the handle provided.</td>
</tr>
</table>
... MORE HTML SUPPRESSED ...
As you can see: I ASKED FOR TEXT and got HTML;
and despite NET-209-144-225-0-1 belonging to "AppServe Technologies, LLC (ATL-83<http://whois.arin.net/rest/org/ATL-83.html>)"and having POC JJ772-ARIN,
asking for the organization associated with JJ772-ARIN says "AIN'T NO SUCH THING." --> ?????
So trying to get the organization from the command line, I am screwed.
So I wonder if other formats can help ...
(3B) Incorrect formats returned
Asking for ip/#.#.#.#.txt did not return the ORGANIZATION NAME, as I pointed out above.
However, asking for ip/#.#.#.#.json does.
However!
All of these calls
curl http://whois.arin.net/rest/poc/JJ772-ARIN/orgs.json > org.json
and
curl -H "Accept: application/json" http://whois.arin.net/rest/poc/JJ772-ARIN/orgs.json > org.json
and
curl -H "Accept: application/xml" http://whois.arin.net/rest/poc/JJ772-ARIN/orgs.xml > org.xml
Return HTML.
Remember that "no information" was returned for JJ772-ARIN; maybe that has something to do with this bug.
(4B) Apparent database mapping inconsistencies:
There is this oddness with the database, that although
NET=X ----> ORG=Y
and
NET=X ----> POC=Z
we get
POC=Z ----> ORG="NO RESULT", which is inconsistent.
You would think that the information would be transitive and reflexive; that
NET=X <----> ORG=Y is bidirectional,
NET=X <----> POC=Z is bidirectional,
and therefore
POC=Z ----> NET=X ----> ORG=Y
and not ORG = "NO RESULT", as demonstrated above.
(5) In summary:
It would appear you have not debugged your "format matrix" properly;
1. Different requested formats selectively omit various information.
2. Requests for specific formats are in various cases ignored.
May or may not depend on whether a query returned a result.
and
3. There may well be more going on underneath than meets my eye, but it seems
the database mapping paths are not fully developed properly, since if
A relates to B and A relates to C; then B relates to C; but you say it does not!
(6) Suggested changes in offerings and documentation
1. The documentation is a little misleading. It says
4.4. ARIN's RESTful Resources
In the examples from the previous sections, the resources being queried had the URL http://whois.arin.net/rest/poc/KOSTE-ARIN. Conceptually, this can be broken into a base URL and the resource identifier:
* base URL = http://whois.arin.net/rest
* resource = /poc/KOSTE-ARIN
The base URL is just where ARIN is hosting the service. The real interesting parts are the bits that identify the resources:
* /poc - this indicates the resource is a Point of Contact.
* /KOSTE-ARIN - this is the unique identifier for the Point of Contact, or the POC Handle.
ARIN's data model has six types of addressable resources. These are reflected in Whois-RWS in the following way:
1. /poc - point of contact.
2. /org - organization
3. /net - a network
4. /asn - autonomous system number(s)
5. /customer - a customer of an organization
6. /rdns - delegation
(underlining mine to call attention to a claim, bolding below mine to call attention to the rebuttal.)
But this is not a complete list, because below you go on to say
4.4.3. IP Addresses and Networks
In the ARIN data model, an IP "network" is a set of associated IP address blocks and the information related to these IP address blocks and the set as a whole. A network can be composed of one IP address block or of multiple IP address blocks. Networks are also hierarchical, meaning that a network can have a parent and can have children. An IP address or range of IP addresses can therefore fall within the IP address block of multiple networks.
To get the networks that a particular IP address may fall within, one can use the /ip/XX.XX.XX.XX resource, where XX.XX.XX.XX signifies the IP address.
(and)
Networks may be looked up as well by supplying the full CIDR notation of a range. To use the full CIDR notation, the resource looks like /cidr/XX.XX.XX.XX/YY where XX.XX.XX.XX is the IP address prefix and YY is the CIDR length.
So you actually have EIGHT resources, not SIX. And since you DO have the "ip" and "cidr" resources, why not expand this list:
4.4.1. Resources Related To Resources
In the ARIN Whois data model, resources have relationships to other resources. Getting references to these resources can be accomplished by addressing the resource in question and applying a resource type qualifier.
...
Here is the list of relationships (where "XXXX" signifies a handle):
* /poc/XXXX
* /orgs ==> /poc/XXXX/orgs - lists the organizations associated with a given POC.
* /asns ==> /poc/XXXX/asns - lists the ASNs associated with a given POC.
* /nets ==> /poc/XXXX/nets - lists the networks associated with a given POC.
* /org/XXXX
* /pocs ==> /org/XXXX/pocs - lists the POCs associate with a given organization.
* /asns ==> /org/XXXX/asns - lists the ASNs associated with a given organization.
* /nets ==> /org/XXXX/nets - lists the networks associated with a given organization.
* /asn/XXXX
* /pocs ==> /asn/XXXX/pocs - lists the POCs associated with a given ASN.
* /net/XXXX
* /pocs ==> /net/XXXX/pocs - lists the POCs associated with a given network.
* /parent ==> /net/XXXX/parent - lists the parent network of a given network.
* /children ==> /net/XXXX/children - lists the child networks of a given network.
* /rdns ==> /net/XXXX/rdns – lists the delegations of a given network.
* /rdns/DELEGATION_NAME
* /nets ==> /rdns/DELEGATION_NAME/nets - lists networks related to a give delegation.
I had wished to be able to do
/ip/XXXX/orgs
directly. Seems obvious; why not?
I suggest making that (and similar extensions) possible; and I suggest including the "ip" and "cidr" as named resources that can be used along with the other six - that is largely a documentation fix.
Thanks for your time and attention.
Eric Dynamic (ED244-ARIN @ org/UCTELE-2)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.arin.net/pipermail/arin-tech-discuss/attachments/20131231/0fa4105a/attachment.html>
More information about the arin-tech-discuss
mailing list