I found an issue when using this library, when parsing robots.txt files that are relatively large, the memory used by the library started to grow exponentially.
I pinpointed the issue here:
'rules' here is being updated at every iteration without filtering duplicates, and every iteration contains the past rules plus the new ones. The 'rules' structure ends up doubling its size at every iteration, eventually eating all system memory.
As a quick fix, I just wrapped it into a set to remove duplicates at every iteration, and that worked for me.
rules = list(set(
existing_rules + rules
)) # Combine rules if agent found several times`
And all unit tests pass succesfully with this change.
However, I'm unsure if the original intention was to do this, or to use an independent variable within the context of the for loop.
You can reproduce the issue with the attached robots.txt file and running the parser as:
import robots
with open('robots.txt','rt') as f:
robots_data = f.read()
rp = robots.RobotsParser.from_string(robots_data)
robots.txt
This is just one of many robots.txt that caused this issue for me, the issue just appears as long as the robots.txt is long enough, the memory just explodes exponentially.
I found an issue when using this library, when parsing robots.txt files that are relatively large, the memory used by the library started to grow exponentially.
I pinpointed the issue here:
robotspy/robots/parser.py
Line 295 in d0840f3
'rules' here is being updated at every iteration without filtering duplicates, and every iteration contains the past rules plus the new ones. The 'rules' structure ends up doubling its size at every iteration, eventually eating all system memory.
As a quick fix, I just wrapped it into a set to remove duplicates at every iteration, and that worked for me.
And all unit tests pass succesfully with this change.
However, I'm unsure if the original intention was to do this, or to use an independent variable within the context of the for loop.
You can reproduce the issue with the attached robots.txt file and running the parser as:
robots.txt
This is just one of many robots.txt that caused this issue for me, the issue just appears as long as the robots.txt is long enough, the memory just explodes exponentially.