09
Aug
08

New Purge in Production

The purge version of the proxy resurrection code is now in production and will run seven days per week at 6:45AM EDT.

The performance in the VM was excellent and the purge ran in approximately 10 minutes for ~1000 proxies.  This is a tremendous improvement over the old purge, which typically took about 45 minutes for ~450 proxies.  The difference is concurrent testing.  The old purge checked each proxy sequentially.  The AMD box was able to do 80 concurrent checks but I limited the VM to 30.  I could probably kick that up to about 50 without a problem but I can live with a 10 minute processing time.

However, the junk filter is still killing me.

The resurrection code will probably be run on an as-needed basis from now on, or at least monthly.  Or perhaps weekly, in chunks of about 3000 or so, although that would put it in perpetual competition with the purge. 

Or  – and I’m thinking out loud here – test a thousand or so random offline proxies with every purge.   Eventually they’d all get checked and it would add a touch of equilibrium to The List.

Now there’s an idea!

Advertisements

0 Responses to “New Purge in Production”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


Archives

Advertisements

%d bloggers like this: