CRAWL STRATEGY
How should we optimize our public site crawl frequency for political and critical government sites
by John Doe
3 comments May 9, 2025 6:52 AM
With the 2024 US National Election coming up, how often should we run crawls across high-importance political and governmental websites. I propose we increase frequency on critical websites with current political changes being implemented now.
Recent Comments
3 comments
J. Smith replied 8 minutes ago
Government sites should be updated at minimum every 24 hours in real-time, as information is constantly being updated in real-time with current events happening quickly.
Mark as helpful
Upvote