General Guidelines for Automated Queries

Latest Revision: 2015 July 20

The ability of NED to support and keep pace with automated access, involving large data volumes and/or high query rates is limited. Such support is provided on an as-available, best-effort basis. Many users of NED services are now being conscientious and considerate, mindful that limited computing and network bandwidth resources are shared by thousands of users, and too many simultaneous queries can make NED services unresponsive for everyone.

Our goal is to provide fair use and responsive access to NED resources for the entire astronomical community as well as the general public. NED plans significant changes to operation and access to it's services, including discontinuing some services, and enforcing guideline and restrictions on others. (See Discontinuing Services.)

Even as we implement these changes, we continue to request user community consideration when planning, developing or performing automated queries. The term Automated Query refers to requests made in lieu of, or as proxy to a human user; or by computer controlled processes such as data pipe-line processing systems. Builders and operators of such automated query systems, are requested to contact the NED Team via the NED Helpdesk: Website or direct email to ned-helpdesk@lists.ipac.caltech.edu and to ensure that their systems conform to the following guidelines in accessing to NED. Exceptions to the General Guidelines can be accommodated by prior arrangement. Whether you are an individual researcher, a power user, represent a data-center, archive site, or operate a mediation/proxy service for multiple users, the NED Team is striving to meet your needs. We ask that you contact us to make arrangements for any usage that may exceed the bounds presented here. This will also help us minimize the impact of future changes on your operation and your users.

General Guidelines for Automated use of NED services:

• Please contact the NED Helpdesk before starting any set of queries that might extend over more than 6 hours; even those that follow these guidelines.
• Do not query NED more frequently than 1 per second; wait at least 1 second between receiving a reply and making a new query.
• Do not run multiple concurrent automated queries; single-thread interactions only.
• Use XML-VOTable output format whenever possible as HTML and other formats may change at any time.
• On queries that support optional image retrieval set img_stamp=NO
• On queries that support detailed information lists, set list_limit=0
• If you need multiple XML VOTable outputs, consider using of=xml_all
• Do NOT automate image retrieval -- please contact the NED Helpdesk if this is essential to you.
• Do NOT masquerade automated queries to make them 'look like' they are coming from a web browser.
• For further information about automating queries to NED, see "NED for a New Era" (Mazzarella et al. 2007, ASPC, 376, 153).
• Specific sample query URLs, which can be accessed via utilities such as wget and curl or from programs written in Perl, Python, or Java, can be found in the book chapter "NED VO Services" (Mazzarella et al. 2008).

NED is committed to providing services to the best benefit of our users, both individual and institutional, within available resources. Very unlike the situation when NED created these services, some over 20 years ago, the internet and our community is now flush with frameworks, protocols, conventions and even standards. Coordination and collaboration with other data providers and centers of excellence in applicable technologies is one of the best ways to do this, but NED is also requesting direct input from the community of users.

The best way to provide input to NED, on any of our services or plans, is to contact us via the NED Helpdesk Website or direct email to ned-helpdesk@lists.ipac.caltech.edu