Robotexclusionrulesparser An alternative to Python's robotparser module
Download

Robotexclusionrulesparser Ranking & Summary

Robotexclusionrulesparser Tags

Python library Python parser robot robot parser

Robotexclusionrulesparser Description

An alternative to Python's robotparser module Robotexclusionrulesparser is an alternative to the Python standard library module robotparser. It fetches and parses robots.txt files and can answer questions as to whether or not a given user agent is permitted to visit a certain URL., respect for Expires headers and understanding of Crawl-delay and Sitemap directives and wildcard syntax in path names.Complete documentation (including a comparison with the standard library module robotparser) is available in ReadMe.html. Requirements: · Python

Robotexclusionrulesparser Related Software