Hinky’s Ultimate Proxy Regex

 

[1-2]\?[0-9]\{1,3\}\.[1-2]\?[0-9]
\{1,3\}\.[1-2]\?[0-9]\{1,3\}\.[1-2]
\?[0-9]\{1,3\}[:\ ][1-9]\?[0-9]\{1,5\}

It’s wrapped, but you’ll figure it out.

Doesn’t look like much but it does the trick at identifying address:port in any page no matter what, unless it is otherwise obfuscated.

Some dickhead proxy list maintainers like to keep each octet three characters wide and to pad the port with zeros, like this:

010.001.010.066:0080

That is stupid and most correctly written shell tools will treat those octets as octal numbers since they start with a zero (that’s just the way it is – the address above is equal to 8.1.8.54 in decimal and the port would be an illegal octal number with no decimal value at all).  The regex should not recognize this format, but if you want these funky addresses, change all the occurences of [1-2] with [0-2].  However, additional post-processing will be needed to get rid of the leading zeros or you will end up with junk IP addresses.

(Revised 7/26/2008 to remove leading zeros from :port spec)

How can you tell if it’s supposed  to be octal?  Easy.  Nobody uses octal.  Padding with zeroes is just a way of making a list look “pretty” to anal retentive idiots and spreadsheet jockeys.  Unfortunately your shell tools don’t know that.

 

General Use

Here’s how it’s done:

  • Grab the page and save it to a file (using wget, curl, html2text, links2 or equivalent)
  • Strip out all HTML (unless you used html2text or links2 in the first place).
  • Turn any character that’s greater than ASCII 127 into a space.  This can be a pain in the ass.
  • Strip out all newlines/CRLFs, making the file One Big Line.
  • Strip out all multiple spaces down to one space (sed).
  • Break out the address:port using the regex (sed) by adding a newline before and after the regex match.
  • Replace all remaining spaces with a colon (sed).
  • Remove all lines that don’t begin with a number (grep)

Essentially you simply turn a page into a single line and break the data you need down into separate lines, removing junk in the process. 

After this you have nothing but address:port combinations in your file.  Remove the octal crap as needed.

This will allow a few FQDNs (Fully Qualified Domain Names) through, which I don’t use.  They tend to be expired names anyway.  In my setup they generate an error and are ignored.

One problem with html2text is that it can add a lot of garbage.  For some reason you get this crap:

[char][0x08][char]

This prints a character, rubs it out (ASCII 8, a.k.a “backspace”), then prints the same character.  Looks fine on a terminal but junk gets in your file.

 

Notes on Proxy Obfuscation

Obfuscation takes various forms.  So far the most popular I’ve run into is Javascript code that scrambles the order of the octets.

At least two sites that I am aware of use an image of the address and port in GIF format.  This is not an unsurmountable problem and there are various Open Source utilities available for OCR (Optical Character Recognition).

The idiots who use these methods are trying to keep page-scrapers like the Dinkster from “stealing” their already stolen data.  It’s an entirely useless practice because all the data is usually available somewhere else and the proxies are dead anyway.

Advertisements

8 Responses to “proxy regex”


  1. 1 Oghenero
    May 20, 2011 at 11:52 am

    Hello,

    Here is mine ([0-9]{1,3}\.)+[0-9]{1,3}\:[0-9]{1,4}

  2. 2 Regex
    November 7, 2011 at 12:00 am

    lol Oghenero, your Regex is funny…

    999.999:0 – matches
    123.123.123.123:12345 – doesnt match

  3. January 21, 2012 at 5:50 pm

    Hi l’m trying to run your regex against this string:

    And it doesn’t match anything… what am l doing wrong?

  4. May 5, 2012 at 5:39 pm

    ehmmm….
    here is mine,.
    \b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\:[0-9]{1,5}\b
    i use it for grab proxy from web…

  5. July 9, 2013 at 11:10 pm

    Movies of the genre can be obtained from Buried and Prowl in horror to gems
    like No Strings Attached in romantic comedies. Workplaces block Facebook
    and You – Tube access for some users. RANK 3: Site: This site features a web based torrent
    applet.

  6. October 10, 2015 at 1:50 pm

    How about that:

    ((?:1?\d{2}|2[0-4]\d|25[0-5]|\d{1,2})\.(?:1?\d{2}|2[0-4]\d|25[0-5]|\d{1,2})\.(?:1?\d{2}|2[0-4]\d|25[0-5]|\d{1,2})\.(?:1?\d{2}|2[0-4]\d|25[0-5]|\d{1,2})|(?:[a-zA-Z]|[a-zA-Z][a-zA-Z0-9\-]*[a-zA-Z0-9]\.)+(?:AAA|ABB|ABBOTT|ABOGADO|AC|ACADEMY|ACCENTURE|ACCOUNTANT|ACCOUNTANTS|ACO|ACTIVE|ACTOR|AD|ADS|ADULT|AE|AEG|AERO|AF|AFL|AG|AGENCY|AI|AIG|AIRFORCE|AIRTEL|AL|ALLFINANZ|ALSACE|AM|AMICA|AMSTERDAM|ANDROID|AO|APARTMENTS|APP|AQ|AQUARELLE|AR|ARCHI|ARMY|ARPA|AS|ASIA|ASSOCIATES|AT|ATTORNEY|AU|AUCTION|AUDIO|AUTO|AUTOS|AW|AX|AXA|AZ|AZURE|BA|BAND|BANK|BAR|BARCELONA|BARCLAYCARD|BARCLAYS|BARGAINS|BAUHAUS|BAYERN|BB|BBC|BBVA|BCN|BD|BE|BEER|BENTLEY|BERLIN|BEST|BET|BF|BG|BH|BHARTI|BI|BIBLE|BID|BIKE|BING|BINGO|BIO|BIZ|BJ|BLACK|BLACKFRIDAY|BLOOMBERG|BLUE|BM|BMS|BMW|BN|BNL|BNPPARIBAS|BO|BOATS|BOM|BOND|BOO|BOOTS|BOUTIQUE|BR|BRADESCO|BRIDGESTONE|BROKER|BROTHER|BRUSSELS|BS|BT|BUDAPEST|BUILD|BUILDERS|BUSINESS|BUZZ|BV|BW|BY|BZ|BZH|CA|CAB|CAFE|CAL|CAMERA|CAMP|CANCERRESEARCH|CANON|CAPETOWN|CAPITAL|CAR|CARAVAN|CARDS|CARE|CAREER|CAREERS|CARS|CARTIER|CASA|CASH|CASINO|CAT|CATERING|CBA|CBN|CC|CD|CEB|CENTER|CEO|CERN|CF|CFA|CFD|CG|CH|CHANEL|CHANNEL|CHAT|CHEAP|CHLOE|CHRISTMAS|CHROME|CHURCH|CI|CIPRIANI|CISCO|CITIC|CITY|CK|CL|CLAIMS|CLEANING|CLICK|CLINIC|CLOTHING|CLOUD|CLUB|CLUBMED|CM|CN|CO|COACH|CODES|COFFEE|COLLEGE|COLOGNE|COM|COMMBANK|COMMUNITY|COMPANY|COMPUTER|CONDOS|CONSTRUCTION|CONSULTING|CONTRACTORS|COOKING|COOL|COOP|CORSICA|COUNTRY|COUPONS|COURSES|CR|CREDIT|CREDITCARD|CRICKET|CROWN|CRS|CRUISES|CSC|CU|CUISINELLA|CV|CW|CX|CY|CYMRU|CYOU|CZ|DABUR|DAD|DANCE|DATE|DATING|DATSUN|DAY|DCLK|DE|DEALS|DEGREE|DELIVERY|DELTA|DEMOCRAT|DENTAL|DENTIST|DESI|DESIGN|DEV|DIAMONDS|DIET|DIGITAL|DIRECT|DIRECTORY|DISCOUNT|DJ|DK|DM|DNP|DO|DOCS|DOG|DOHA|DOMAINS|DOOSAN|DOWNLOAD|DRIVE|DURBAN|DVAG|DZ|EARTH|EAT|EC|EDU|EDUCATION|EE|EG|EMAIL|EMERCK|ENERGY|ENGINEER|ENGINEERING|ENTERPRISES|EPSON|EQUIPMENT|ER|ERNI|ES|ESQ|ESTATE|ET|EU|EUROVISION|EUS|EVENTS|EVERBANK|EXCHANGE|EXPERT|EXPOSED|EXPRESS|FAGE|FAIL|FAITH|FAMILY|FAN|FANS|FARM|FASHION|FEEDBACK|FI|FILM|FINAL|FINANCE|FINANCIAL|FIRMDALE|FISH|FISHING|FIT|FITNESS|FJ|FK|FLIGHTS|FLORIST|FLOWERS|FLSMIDTH|FLY|FM|FO|FOO|FOOTBALL|FOREX|FORSALE|FORUM|FOUNDATION|FR|FRL|FROGANS|FUND|FURNITURE|FUTBOL|FYI|GA|GAL|GALLERY|GAME|GARDEN|GB|GBIZ|GD|GDN|GE|GEA|GENT|GENTING|GF|GG|GGEE|GH|GI|GIFT|GIFTS|GIVES|GIVING|GL|GLASS|GLE|GLOBAL|GLOBO|GM|GMAIL|GMO|GMX|GN|GOLD|GOLDPOINT|GOLF|GOO|GOOG|GOOGLE|GOP|GOV|GP|GQ|GR|GRAPHICS|GRATIS|GREEN|GRIPE|GROUP|GS|GT|GU|GUGE|GUIDE|GUITARS|GURU|GW|GY|HAMBURG|HANGOUT|HAUS|HEALTHCARE|HELP|HERE|HERMES|HIPHOP|HITACHI|HIV|HK|HM|HN|HOCKEY|HOLDINGS|HOLIDAY|HOMEDEPOT|HOMES|HONDA|HORSE|HOST|HOSTING|HOTELES|HOTMAIL|HOUSE|HOW|HR|HSBC|HT|HU|HYUNDAI|IBM|ICBC|ICE|ICU|ID|IE|IFM|IINET|IL|IM|IMMO|IMMOBILIEN|IN|INDUSTRIES|INFINITI|INFO|ING|INK|INSTITUTE|INSURE|INT|INTERNATIONAL|INVESTMENTS|IO|IPIRANGA|IQ|IR|IRISH|IS|IST|ISTANBUL|IT|ITAU|IWC|JAVA|JCB|JE|JETZT|JEWELRY|JLC|JLL|JM|JO|JOBS|JOBURG|JP|JPRS|JUEGOS|KAUFEN|KDDI|KE|KG|KH|KI|KIA|KIM|KINDER|KITCHEN|KIWI|KM|KN|KOELN|KOMATSU|KP|KR|KRD|KRED|KW|KY|KYOTO|KZ|LA|LACAIXA|LANCASTER|LAND|LASALLE|LAT|LATROBE|LAW|LAWYER|LB|LC|LDS|LEASE|LECLERC|LEGAL|LEXUS|LGBT|LI|LIAISON|LIDL|LIFE|LIGHTING|LIMITED|LIMO|LINDE|LINK|LIVE|LIXIL|LK|LOAN|LOANS|LOL|LONDON|LOTTE|LOTTO|LOVE|LR|LS|LT|LTD|LTDA|LU|LUPIN|LUXE|LUXURY|LV|LY|MA|MADRID|MAIF|MAISON|MAN|MANAGEMENT|MANGO|MARKET|MARKETING|MARKETS|MARRIOTT|MBA|MC|MD|ME|MEDIA|MEET|MELBOURNE|MEME|MEMORIAL|MEN|MENU|MG|MH|MIAMI|MICROSOFT|MIL|MINI|MK|ML|MM|MMA|MN|MO|MOBI|MODA|MOE|MOI|MOM|MONASH|MONEY|MONTBLANC|MORMON|MORTGAGE|MOSCOW|MOTORCYCLES|MOV|MOVIE|MOVISTAR|MP|MQ|MR|MS|MT|MTN|MTPC|MTR|MU|MUSEUM|MV|MW|MX|MY|MZ|NA|NADEX|NAGOYA|NAME|NAVY|NC|NE|NEC|NET|NETBANK|NETWORK|NEUSTAR|NEW|NEWS|NEXUS|NF|NG|NGO|NHK|NI|NICO|NINJA|NISSAN|NL|NO|NOKIA|NP|NR|NRA|NRW|NTT|NU|NYC|NZ|OBI|OFFICE|OKINAWA|OM|OMEGA|ONE|ONG|ONL|ONLINE|OOO|ORACLE|ORANGE|ORG|ORGANIC|OSAKA|OTSUKA|OVH|PA|PAGE|PANERAI|PARIS|PARTNERS|PARTS|PARTY|PE|PET|PF|PG|PH|PHARMACY|PHILIPS|PHOTO|PHOTOGRAPHY|PHOTOS|PHYSIO|PIAGET|PICS|PICTET|PICTURES|PINK|PIZZA|PK|PL|PLACE|PLAY|PLUMBING|PLUS|PM|PN|POHL|POKER|PORN|POST|PR|PRAXI|PRESS|PRO|PROD|PRODUCTIONS|PROF|PROPERTIES|PROPERTY|PROTECTION|PS|PT|PUB|PW|PY|QA|QPON|QUEBEC|RACING|RE|REALTOR|REALTY|RECIPES|RED|REDSTONE|REHAB|REISE|REISEN|REIT|REN|RENT|RENTALS|REPAIR|REPORT|REPUBLICAN|REST|RESTAURANT|REVIEW|REVIEWS|RICH|RICOH|RIO|RIP|RO|ROCKS|RODEO|RS|RSVP|RU|RUHR|RUN|RW|RYUKYU|SA|SAARLAND|SAKURA|SALE|SAMSUNG|SANDVIK|SANDVIKCOROMANT|SANOFI|SAP|SARL|SAXO|SB|SC|SCA|SCB|SCHMIDT|SCHOLARSHIPS|SCHOOL|SCHULE|SCHWARZ|SCIENCE|SCOR|SCOT|SD|SE|SEAT|SECURITY|SEEK|SENER|SERVICES|SEVEN|SEW|SEX|SEXY|SG|SH|SHIKSHA|SHOES|SHOW|SHRIRAM|SI|SINGLES|SITE|SJ|SK|SKI|SKY|SKYPE|SL|SM|SN|SNCF|SO|SOCCER|SOCIAL|SOFTWARE|SOHU|SOLAR|SOLUTIONS|SONY|SOY|SPACE|SPIEGEL|SPREADBETTING|SR|SRL|ST|STADA|STARHUB|STATOIL|STC|STCGROUP|STOCKHOLM|STUDIO|STUDY|STYLE|SU|SUCKS|SUPPLIES|SUPPLY|SUPPORT|SURF|SURGERY|SUZUKI|SV|SWATCH|SWISS|SX|SY|SYDNEY|SYSTEMS|SZ|TAIPEI|TATAMOTORS|TATAR|TATTOO|TAX|TAXI|TC|TD|TEAM|TECH|TECHNOLOGY|TEL|TELEFONICA|TEMASEK|TENNIS|TF|TG|TH|THD|THEATER|THEATRE|TICKETS|TIENDA|TIPS|TIRES|TIROL|TJ|TK|TL|TM|TN|TO|TODAY|TOKYO|TOOLS|TOP|TORAY|TOSHIBA|TOURS|TOWN|TOYOTA|TOYS|TR|TRADE|TRADING|TRAINING|TRAVEL|TRUST|TT|TUI|TV|TW|TZ|UA|UBS|UG|UK|UNIVERSITY|UNO|UOL|US|UY|UZ|VA|VACATIONS|VC|VE|VEGAS|VENTURES|VERSICHERUNG|VET|VG|VI|VIAJES|VIDEO|VILLAS|VIN|VIRGIN|VISION|VISTA|VISTAPRINT|VIVA|VLAANDEREN|VN|VODKA|VOTE|VOTING|VOTO|VOYAGE|VU|WALES|WALTER|WANG|WATCH|WEBCAM|WEBSITE|WED|WEDDING|WEIR|WF|WHOSWHO|WIEN|WIKI|WILLIAMHILL|WIN|WINDOWS|WINE|WME|WORK|WORKS|WORLD|WS|WTC|WTF|XBOX|XEROX|XIN|XN–11B4C3D|XN–1QQW23A|XN–30RR7Y|XN–3BST00M|XN–3DS443G|XN–3E0B707E|XN–3PXU8K|XN–42C2D9A|XN–45BRJ9C|XN–45Q11C|XN–4GBRIM|XN–55QW42G|XN–55QX5D|XN–6FRZ82G|XN–6QQ986B3XL|XN–80ADXHKS|XN–80AO21A|XN–80ASEHDB|XN–80ASWG|XN–90A3AC|XN–90AIS|XN–9DBQ2A|XN–9ET52U|XN–B4W605FERD|XN–C1AVG|XN–C2BR7G|XN–CG4BKI|XN–CLCHC0EA0B2G2A9GCD|XN–CZR694B|XN–CZRS0T|XN–CZRU2D|XN–D1ACJ3B|XN–D1ALF|XN–EFVY88H|XN–ESTV75G|XN–FHBEI|XN–FIQ228C5HS|XN–FIQ64B|XN–FIQS8S|XN–FIQZ9S|XN–FJQ720A|XN–FLW351E|XN–FPCRJ9C3D|XN–FZC2C9E2C|XN–GECRJ9C|XN–H2BRJ9C|XN–HXT814E|XN–I1B6B1A6A2E|XN–IMR513N|XN–IO0A7I|XN–J1AEF|XN–J1AMH|XN–J6W193G|XN–KCRX77D1X4A|XN–KPRW13D|XN–KPRY57D|XN–KPUT3I|XN–L1ACC|XN–LGBBAT1AD8J|XN–MGB9AWBF|XN–MGBA3A4F16A|XN–MGBAAM7A8H|XN–MGBAB2BD|XN–MGBAYH7GPA|XN–MGBBH1A71E|XN–MGBC0A9AZCG|XN–MGBERP4A5D4AR|XN–MGBPL2FH|XN–MGBX4CD0AB|XN–MK1BU44C|XN–MXTQ1M|XN–NGBC5AZD|XN–NODE|XN–NQV7F|XN–NQV7FS00EMA|XN–NYQY26A|XN–O3CW4H|XN–OGBPF8FL|XN–P1ACF|XN–P1AI|XN–PGBS0DH|XN–PSSY2U|XN–Q9JYB4C|XN–QCKA1PMC|XN–RHQV96G|XN–S9BRJ9C|XN–SES554G|XN–T60B56A|XN–TCKWE|XN–UNUP4Y|XN–VERMGENSBERATER-CTB|XN–VERMGENSBERATUNG-PWB|XN–VHQUV|XN–VUQ861B|XN–WGBH1C|XN–WGBL6A|XN–XHQ521B|XN–XKC2AL3HYE2A|XN–XKC2DL3A5EE0H|XN–Y9A3AQ|XN–YFRO4I67O|XN–YGBI2AMMX|XN–ZFR164B|XPERIA|XXX|XYZ|YACHTS|YAMAXUN|YANDEX|YE|YODOBASHI|YOGA|YOKOHAMA|YOUTUBE|YT|ZA|ZIP|ZM|ZONE|ZUERICH|ZW))(?:[^0-9]+?)([0-9]{1,4}|[1-5][0-9]{4}|6[0-4][0-9]{3}|65[0-4][0-9]{2}|655[0-2][0-9]|6553[0-5])(?:[^0-9]|$)

  7. January 17, 2017 at 5:27 pm

    どうも~コン と申します 。このサイトとんでもなく
    楽しい ですね。ふいに 訪問しただけなんですけど刺激になります 。感謝してます。
    ところで、ぼくはなかなか結婚できない独身 の若者のためにコン活 アプリ を組織してます。
    そのうち一押しのコンカツ アプリ をお披露目することがあります。
    世の中様々なタイプの結婚活動 サイト があるんですよ。例えば離婚者 の高齢者のための婚活 サイト や娘あり の人のためのコンカツ サービス 。他には若い人 向けのものであったり、逆に60歳以上 向けのサービスであったりです。
    でもとても一般的なの は40代 の男性 ですね。意外ですか?
    僕 のアンケート調査 ではそういう検証データ が報告されています
    。結婚活動 サービスは費用もまちまち でお金は不要 で利用できるものから結構 高額な 入会金のものまで多種多様 です。
    また、すでに幸せな結婚をされた年配に聞き取り調査して 馴れ初め や結婚の告白 がどんな感じだったかを記事 で案内しています。
    かなり心温まる お話や不思議な ご縁の話、エキサイトする体験もあってメチャクチャ楽しいです。
    気が向いたら 是非 私のアプリ をチェックしてちょうだいね。
    それでは今後も凄い サイトを楽しみにしています!楽しみにしています!失礼します


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s




Archives

Advertisements

%d bloggers like this: