Details of the Inquiry
I built a simple web crawler in Python, and it worked as intended.
However, when I run this program with proxychains to hide the source IP, the chain between the proxies is denied and I cannot retrieve the data…?
When I searched for related posts, many developers seemed to have run into the same problem. Is there a way to resolve it?
First, as a developer who has been through the same difficulty, I am glad to hear from you. ^^
When you build a crawler for a website organized around page navigation, using Selenium WebDriver for the web crawler makes it easy to build.
These WebDrivers crawl the target website through the server’s browser and browser driver, so they need a host and use localhost. Because of that, they appear to conflict with proxychains, which uses socks4 127.0.0.1 9050 as its default setting, and return denied or timeout.
I also confirmed that the program ran correctly, and then ran into the same problem as you when I finally ran it on top of proxychains.
I searched every likely place, including reddit and stackoverflow, and found many posts describing the same problem, but no one offered a clear solution.
After a day of trying different approaches and switching webdrivers, I happened to find a method.
Instead of running the web crawler program on top of proxychains, set the proxy server option directly on the WebDriver inside the web crawler.
options = webdriver.ChromeOptions()
options.add_argument('--proxy-server=socks4://localhost:9050') # Use the default proxychains setting socks4 127.0.0.1 9050
browser = webdriver.Chrome(driver_path, options=options)
The exact cause will need further testing, but the site currently running with selenium webdriver and proxychains settings using this method, https://vul.wikisecurity.net is working very well.
Here is one more tip for using proxychains.
According to the proxychains manual, editing the configuration file proxychains.conf and changing it to random chain is supposed to change the source IP each time you use it. This method does not work and produces a timeout error 99% of the time. There is no such thing as a free lunch.
One workaround I use is to change the setting to dynamic chain and register a cron job so that the tor service restarts periodically.
This can be a reasonable alternative that lets you use proxychains cleanly while changing the source IP.