Skip to content

The LunaCompliance crawler

If you found this User-Agent in your server logs, this page explains it:

LunaCompliance/1.0 (+https://lunacompliance.io/bot; tye@lunacompliance.io)

The crawler belongs to Luna Compliance, Inc., which makes compliance software for registered investment advisers.

What it does

An investment adviser has to keep what its website says in line with what it tells the SEC in its Form ADV. Luna is building a check that compares the two. The crawler reads the pages of the adviser's own public website and keeps their text so that Luna can make that comparison for the adviser.

The rest of this page describes how the crawler behaves when it runs.

Which sites it reads

  • It reads a website when a firm that uses Luna has confirmed the site as its own. The addresses usually come from the firm's Form ADV.
  • It stays on that site's domain. From www.example.com it may read example.com and blog.example.com, and nothing on another domain.
  • It notes links to other websites but does not follow them.

How it behaves

  • It reads robots.txt before anything else and follows it. The User-Agent token to use is LunaCompliance.
  • It makes one request at a time, at most one every 2 seconds. If your robots.txt sets a longer Crawl-delay, it uses that.
  • It reads up to 50 pages in one check. A check usually runs once a month.
  • It reads HTML pages as your server sends them. It does not run JavaScript or render pages in a browser, and it does not download images, scripts or style sheets.
  • It never logs in and never fills in or submits a form.
  • It skips login pages, admin pages, feeds, pages marked noindex and links marked nofollow.
  • It stops when it is turned away. A 401, 403, 406, 429 or 503 response, or a bot check such as a CAPTCHA, ends the check for that site. It does not retry under another name, from another address or through a proxy, and it does not try to solve the challenge. The next attempt is the next scheduled check.

How to block it

To stop the crawler reading any part of your site, add this to your robots.txt:

User-agent: LunaCompliance
Disallow: /

To keep it out of one folder, name that folder instead:

User-agent: LunaCompliance
Disallow: /private/

The crawler reads robots.txt at the start of every check, so a change applies from the next check. If a site blocks it, the firm that asked for the check can send Luna saved copies of its pages instead.

Contact

For questions about the crawler, or to ask us to stop reading your site, write to support@lunacompliance.io. You can also use the address in the User-Agent, tye@lunacompliance.io. Please tell us the domain and, if you can, the time of the requests.

The crawler is operated by Luna Compliance, Inc., c/o Harvard Business Services, Inc., 16192 Coastal Highway, Lewes, DE 19958, United States.