Our Crawlers
Last Updated: October 30, 2024
User Agent Information
All data source crawlers use the following User Agent structure:
MinimalDashboardFetcher/1.0.0 (+https://www.minimaldashboard.com/crawler)
(Versions and OS will change, but MinimalDashboardFetcher will remain constant.)
Description
Our crawler uses published API endpoints in conjungtion with credentials supplied by our users to access their data in third party services for the means of generating metrics (KPIs) and time series.
Access Frequency
During setup an initial sync of data is performed based on data availability as published in API documentation.
Once the initial sync is complete we perform one data sync request per hour to using any continuation token or paging cursor provided with the last request. A data sync request may consist of multiple http requests if there are more than one page of new data available.
We aim not to fetch previously fetched data again if avoidable.
Rate Limiting
We respect 429 rate limiting header responses and use documented cool off periods or fallback to 1 hour if none are specified in the public API documentation.
Contact
We strive to get the information we need while minimizing our impact on your servers and bandwidth. If you have any concerns, please reach out to [email protected].