Dynamic Proxy Management via Bash and Scheduled Tasks > 자유게시판

Managing web scrapers or automated requests often requires rotating proxy servers to avoid IP bans and detection.

A simple, resource-efficient solution for proxy rotation involves bash scripting and cron scheduling.

This method is especially suited for low-resource systems where simplicity and reliability matter more than feature bloat.

Start by creating a list of proxy servers in a plain text file.

Format each proxy as "ip port" or "ip port username password" depending on whether authentication is required.

Valid entries might look like 192.168.10.5:8080 or 192.168.1.20 9090 john doe.

Regularly refresh your proxy list to remove inactive or banned entries.

Next, write a simple bash script to select a random proxy from the list and update a configuration file or environment variable that your scraper uses.

The script will read more on hackmd.io the proxies.txt file, count the number of lines, pick one at random, and write it to a file called current_proxy.txt.

Here’s a sample implementation:.

bash.

PROXY_FILE=.

var.

if [[! -e "$PROXY_FILE" ]]; then.

echo "Proxy configuration file does not exist".

exit 1.

fi.

LINE_COUNT=$(wc -l < "$PROXY_FILE" 2>.

$LINE_COUNT -eq 0 ]]; then.

echo "Proxy list is empty".

exit 1.

fi.

RANDOM_LINE=$(((RANDOM % LINE_COUNT) + 1)).

tail -n 1 > "$OUTPUT_FILE".

Grant execute rights via: chmod u+x rotate_proxy.sh.

rotate_proxy.sh and check that current_proxy.txt contains a single valid proxy line.

Configure a cron job to trigger the proxy rotation script periodically.

opt.

The proxy changes once per hour with this setup.

Adjust the timing based on your needs—for example, every 30 minutes would be 0,30, or every 15 minutes would be .

Rotating too often may break live sessions—pace rotations to align with your scraper's request intervals.

Finally, make sure your scraper reads the current_proxy.txt file each time it makes a request.

Curl, wget, and similar utilities support proxy configuration via file input or environment variables.

to.

Continuously validate your proxies and remove those that fail to respond.

You can even extend the bash script to ping each proxy and remove unresponsive ones automatically.

Enable logging by appending a timestamped entry: echo "$(date): $(cat $OUTPUT_FILE)" >> .

It offers a minimal yet powerful solution that grows with your needs.

It avoids the overhead of complex proxy management libraries and works well on any Unixlike system.

A simple bash script and a cron entry are all you need to sustain uninterrupted, undetected scraping.