Our data collector, livedata-collector

How to recognise it

Every request it sends carries this user agent: livedata-collector/0.1 (+thingstojoin.com/bot)

It runs on our servers and never switches to other addresses to get around a block.

What it fetches

Public data files and APIs that their publishers make available, and a few public web pages that publish such data. It asks only for what we need, never for a whole website.

For Things to join, the public page of each group we list, about every 30 days, to see whether the page is still there and shows a dated meeting; after a failed check it tries again 7 days later. On Meetup it only checks that a group's page still exists, and it opens no group page on Facebook, Instagram, Reddit or any other platform we do not check.

In each hourly run it asks any one website for at most one group page, plus that website's robots.txt and any redirect the page answers with; on Meetup, up to 15 group pages in a run, at least 10 seconds apart. It keeps to a fixed pace for each website, mostly one request at a time, waits when a website asks it to slow down, and pauses a website that keeps failing.

robots.txt

Before it fetches any group's page, and almost every other web page it asks for, it reads that website's robots.txt and follows its Allow and Disallow rules for livedata-collector, or the rules for all robots when no rule names it. It uses a robots.txt it has read for at most 24 hours, so a new rule applies within a day. A group whose page robots.txt keeps it from is listed without a check.

Data files and APIs that publishers offer by their exact address are fetched without reading robots.txt on some hosts, since many publishers serve their files from hosts whose robots.txt is meant for web crawlers. If you publish such files and want it to stop, write to us.

What it never does

• Sign in to see what the public cannot see, create accounts, or accept terms on anyone's behalf.
• Send a form, except a public download form that a data publisher offers for its files.
• Try to get around a block. A refusal, a sign-in wall or a bot check ends the visit; a group's page is asked for again only at its next regular check, weeks later.
• Look for member lists, profiles or anyone's contact details, or take them into our records.
• Keep text from group pages in our records: only when we checked, the answer's status and the date of any meeting found. Copies of downloaded group pages are left out of our backups, stored in hourly batches, and deleted by a daily clean-up once 30 days have passed since a check last downloaded any page of the batch. A Meetup page is not kept at all.

How to opt out

To keep it off the web pages of your website, group pages included, add these two lines to your robots.txt. For data files and APIs, write to us as well.

• User-agent: livedata-collector
• Disallow: /

Or write to us

Write to us through thingstojoin.com/contact with your website's address and we will stop fetching it. To have a group removed from Things to join, use the same page.