31
Aug
08

Back On Schedule

We now have 1,089,613 proxies in the database, which is astounding considering two days ago there were 890,000.

That’s slighlty less than 100,000 proxies per day  for the last two days, or about 4000+ per hour.  And as usual most of them were dead, but there were enough live ones to slow down publishing the list.

In fact there were only two page refreshes yesterday since it was taking so long to go through them all.

But we’re all caught up now and back on the usual publishing schedule.

Meanwhile I’m working on the page refresh to make it faster so days like yesterday don’t happen again.  Proxies come and go so fast you need the freshest possible data.  A twice-a-day refresh is not going to cut it.

Instead of moving the newly found proxies from the main table into the “gold” table before every page run, I will be putting them in both the main and “gold” table.  This way I can run a modified version of the resurrection code on the new proxies, which will run much faster than the old sequential method.

For now we’re on the old method because things have died dwon a bit.  The file at IS-1 stopped refreshing last night.  If it starts back up you will notice some page delays if I don’t get this code working today.

Advertisements

0 Responses to “Back On Schedule”



  1. Leave a Comment

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s


Archives

Advertisements

%d bloggers like this: