hclubsBot

hclubsBot is the crawler behind hclubs. It reads public Harvard directory pages and the public websites of student organizations so club information can link back to its source.

How it behaves

• It identifies itself honestly with a user agent that starts with hclubsBot/.
• It obeys robots.txt, including Crawl-delay, and re-reads it daily.
• It fetches one page at a time per site, at least two seconds apart. It backs off on errors and honors Retry-After.
• It crawls shallowly: a handful of pages per club site (home, about, join, recruiting). It uses conditional requests so unchanged pages cost almost nothing, and skips cart, account and login pages.
• It rechecks a page only when it's due: club home pages about monthly, Harvard directories weekly, and joining pages more often while clubs are recruiting.
• For the few pages that show nothing without JavaScript, it may render the page in a browser — with the same user agent, after the same robots.txt check.
• For application forms linked from a club's site, it reads only the form's public title and description. It never fills in or submits a form.
• It never logs in or submits forms, and never solves CAPTCHAs. It doesn't rotate addresses or disguise itself. If a site blocks it, it stops.
• It does not crawl Instagram, LinkedIn, Facebook, TikTok, or X.
• It keeps a private copy of each page it reads, as evidence. That copy is never published.

What appears on hclubs

Short facts about each organization (how to join, deadlines, links, a one-line description), each linked to the page it came from, plus page titles in the club's source list. Not the text of your pages, and never member rosters, officer names, or personal email or phone numbers.

Opting out

To stop the crawler, add this to your site's robots.txt:

User-agent: hclubsBot
Disallow: /

Pages it can no longer fetch stay cited as they were when we last read them, with that date. To have information about your organization corrected or removed, contact hclubsapp@gmail.com.