Crawler policy
Where Roster collects public web data (a supplementary source — official APIs and creator consent come first), the crawler behaves as follows, enforced in code:
- Identifies itself as RosterBot (+https://internetchicks.sa.com/crawler/)
- Respects robots.txt, honours 429/503 with exponential backoff, and rate-limits conservatively
- Never authenticates, never uses borrowed sessions or cookies, never evades blocks via proxies, never solves CAPTCHAs — a block means stop
- Collects only public, professional-context signals; never private accounts or personal contact data
- Checks the permanent suppression list before any write, so removed people stay removed
Site operators: blocking the UA above in robots.txt is honoured. Individuals: use the removal form — it outranks everything else.