Welcome! Log In Create A New Profile

Advanced

Auto build PeerBlock lists from Command line Utility

Posted by Anonymous User 
Anonymous User
Auto build PeerBlock lists from Command line Utility
January 27, 2013 10:32AM
Hi and thanks to those who develop PeerBlock.

I became a little obsessed with building block lists and am a software developer, so I built a little utility to help me, which others may find of use. I could not find the same functionality anywhere else; but that does not mean there isn't any.

Basically its a windows command line exe that can be automated to build files (block or allow) in PeerBlock format. It converts a number of possible entries or sources into PeerBlock's file format.

For example, if you feed it a domain name, it will query the DNS server(s) to return all the IP's that should be blocked. Currently, it converts:
* Host list
* IP List
* IP range in CIDR format
* Downloads source files in any accepted format (ie hosts, ip, ip range etc)
* Includes of a local file
* Downloads HTML using a start and end tag to section off the file
and 1 or two other little experimental things

For example:
ConvertToPeerBlockList /l "http://winhelp2002.mvps.org/hosts.txt" /o C:\temp\MVPS_hosts.p2b /d 8.8.8.8,129.250.35.250,129.250.35.251,4.2.2.1,4.2.2.2
will download the MVP Hosts file & resolve all the IP's using a number of public DNS server. NB: This is a BIG list so be ready to press CTRL-C when you get bored to stop it.

In the ZIp file I have a directory called Example, which has some slightly modified versions of "conversion scripts" and a Windows Command prompt file which I have scheduled to occur every day automatically at 4am OR I can click on a shortcut to run it "now" if I have made any changes to the source files. NB: This will not run on your computer as you will be missing a number of the "include" files I have pre-created (using this utility). I also need to run it at administrator level as I access protected directories. Also note, that the CMD file uses stuff I think only came out in Windows 7.

I hope someone else finds this of use.

===================================
edits:
11 Feb 2013 removed - ConvertToPeerBlockList_v1.zip (356.2 KB ) added - ConvertToPeerBlockList_v2.zip (58.1 KB )
19 Mar 2013 removed - ConvertToPeerBlockList_v2.zip (356.2 KB ) added - ConvertToPeerBlockList_v3.zip (59,485 bytes )



Edited 2 time(s). Last edit at 03/19/2013 09:58AM by JohnK.
Attachments:
open | download - ConvertToPeerBlockList_v3.zip (56.9 KB)
Re: Auto build PeerBlock lists from Command line Utility
February 05, 2013 07:51PM
Sounds great, but crashes with PC beeping.

Win7Pro
Anonymous User
Re: Auto build PeerBlock lists from Command line Utility
February 11, 2013 10:57AM
Things to check:
1. you are running from a console window

2. the following 3 files are in the current directory (or in the PATH) see the text file on how / where to get the two dll's
i) ConvertToPeerBlockList.exe
ii) JHSoftware.DnsClient.dll
iii) LukeSkywalker.IPNetwork.dll

3. You have .Net framework 2.0 or higher (NB v1 actually required 4 or 4.5 of the framework)

To test it on your PC:
Copy all the 3 files above to c:\temp\
Open a console window (ie start cmd.exe), and type each of the following lines, followed by the enter key at the end of each.

c:
CD TEMP
ConvertToPeerBlockList /l 1.2.3.4

It should either print out a PeerBlock line or crash. The messages on the screen should give the reason it did not work. Post these up, and maybe I can help you.
Re: Auto build PeerBlock lists from Command line Utility
February 12, 2013 03:55PM
Thanks, it's a really good tool and now works fine. It does everything written in 'converttopeerblock help.txt'.
Also country lists will be generated using e.g. 'ConvertToPeerBlockList /l "http://ipinfodb.com/country_query.php?country=AF # Afganistan"'.

But I've got e.g. this
Z:\ConvertToPeerBlockList_v2>ConvertToPeerBlockList.exe /l www.google.com
Started...
1.1.1 www.google.com:173.194.44.144-173.194.44.144
1.2.2 www.google.com:173.194.44.145-173.194.44.145
1.3.3 www.google.com:173.194.44.146-173.194.44.146
1.4.4 www.google.com:173.194.44.147-173.194.44.147
1.5.5 www.google.com:173.194.44.148-173.194.44.148

What IPs does it take, because Google is more like this:

Google 1:64.18.0.1-64.18.15.254
Google 2:64.233.160.1-64.233.191.254
Google 3:66.102.0.1-66.102.15.254
Google 4:66.249.80.1-66.249.95.254
Google 5:72.14.192.1-72.14.255.254
Google 6:74.125.0.1-74.125.255.254
Anonymous User
Re: Auto build PeerBlock lists from Command line Utility
February 13, 2013 09:42AM
WhiteHat Wrote:
-------------------------------------------------------
> But I've got e.g. this
> Z:\ConvertToPeerBlockList_v2>ConvertToPeerBlockList.exe /l www.google.com
> Started...
> 1.1.1 www.google.com:173.194.44.144-173.194.44.144
> 1.2.2 www.google.com:173.194.44.145-173.194.44.145
> 1.3.3 www.google.com:173.194.44.146-173.194.44.146
> 1.4.4 www.google.com:173.194.44.147-173.194.44.147
> 1.5.5 www.google.com:173.194.44.148-173.194.44.148
>
> What IPs does it take, because Google is more like
> this:
>
> Google 1:64.18.0.1-64.18.15.254
> Google 2:64.233.160.1-64.233.191.254
> Google 3:66.102.0.1-66.102.15.254
> Google 4:66.249.80.1-66.249.95.254
> Google 5:72.14.192.1-72.14.255.254
> Google 6:74.125.0.1-74.125.255.254

If you do not supply a DNS (or a list of DNS's) it uses Google's public DNS server. So the IP's it returned I would expect to be what you would get if you typed the following at the command prompt.

NSLOOKUP www.google.com 8.8.8.8

However, if you use a number of DNS servers, you will get a broader range of IP's. for example:
ConvertToPeerBlockList.exe /l www.google.com /d 202.83.95.227;208.67.220.220;208.67.222.222;216.87.84.211;66.244.95.20;69.164.196.21;8.20.247.20;8.26.56.26;87.118.100.175;94.75.228.29;4.2.2.1;4.2.2.2;4.2.2.3;4.2.2.4;4.2.2.5;4.2.2.6;8.8.4.4;8.8.8.8;156.154.70.1;156.154.71.1;129.250.35.251

Returns a list of 94 unique IP addresses for google.

The list of DNS server above are a list of public ones. Dont forget to add your currently configured DNS server (use IPCONFIG /all) to the list.

Also, the utility only retrieves IP addresses, not ranges from a domain name; NB: it will return a range if supplied a CIDR. I would not be suprised if the ranges you show above, also include Googles advertising IP addresses as well as their web search; which is what I block in my black list.
Re: Auto build PeerBlock lists from Command line Utility
February 13, 2013 02:39PM
I see - great tool.
Re: Auto build PeerBlock lists from Command line Utility
March 18, 2013 07:41PM
Very Nice work John, Nice work!

This may not help me, but the idea was original - It was good, and it works. Good work smiling smiley
Anonymous User
Re: Auto build PeerBlock lists from Command line Utility
March 19, 2013 10:11AM
Added a new version, which addressed a few bugs.

A feature I use extensively, is the ability to resolve a large host file in multiple parallel streams. for example I have a batch file which contains:

START ConvertToPeerBlockList /s 1 /e 30 /h /i C:\Program Files\PeerBlock\lists\Built\Source\Domain\virustracker.info_Blocklist_combined.src /a /o g:\temp\peerblock\virustracker.info_Blocklist_combined.txt /d %DNS_LIST%
START ConvertToPeerBlockList /s 31 /e 60 /h /i C:\Program Files\PeerBlock\lists\Built\Source\Domain\virustracker.info_Blocklist_combined.src /a /o g:\temp\peerblock\virustracker.info_Blocklist_combined.txt /d %DNS_LIST%
.
.
START ConvertToPeerBlockList /s 9211 /e 9240 /h /i C:\Program Files\PeerBlock\lists\Built\Source\Domain\virustracker.info_Blocklist_combined.src /a /o g:\temp\peerblock\virustracker.info_Blocklist_combined.txt /d %DNS_LIST%
START ConvertToPeerBlockList /s 9241 /e 9270 /h /i C:\Program Files\PeerBlock\lists\Built\Source\Domain\virustracker.info_Blocklist_combined.src /a /o g:\temp\peerblock\virustracker.info_Blocklist_combined.txt /d %DNS_LIST%
START ConvertToPeerBlockList /s 9271 /h /i C:\Program Files\PeerBlock\lists\Built\Source\Domain\virustracker.info_Blocklist_combined.src /a /o g:\temp\peerblock\virustracker.info_Blocklist_combined.txt /d %DNS_LIST%

DNS_LIST is a list of public DNS servers I use, set in a called batch:

SET DNS_LIST=106.186.17.181;109.235.51.12;109.69.8.34;109.69.8.51;111.67.16.202;129.250.35.250;129.250.35.251;139.130.4.4;156.154.70.1;156.154.71.1;176.58.117.25;178.217.184.36;184.154.13.11;184.154.13.186;190.111.81.23;192.71.245.208;193.33.179.203;195.46.245.59;198.153.192.40;198.153.194.40;202.83.95.227;205.210.42.205;208.111.40.37;208.67.220.220;208.67.222.222;208.71.35.137;208.76.50.50;209.244.0.3;209.244.0.4;213.138.101.252;216.167.252.195;216.167.252.196;216.167.252.197;216.87.84.211;217.171.190.12;217.6.34.47;217.6.34.48;4.2.2.1;4.2.2.2;4.2.2.3;4.2.2.4;4.2.2.5;4.2.2.6;46.182.18.228;5.9.28.125;50.116.23.211;50.116.38.157;63.243.164.219;64.0.55.201;64.81.79.2;66.244.95.20;67.212.90.199;69.164.196.21;70.33.226.128;72.14.183.109;74.207.247.4;8.20.247.20;8.26.56.26;8.8.4.4;8.8.8.8;85.126.4.170;87.118.100.175;91.236.239.11;92.243.8.139;93.95.226.146;94.75.228.29;176.31.159.150;192.121.86.100;72.232.162.195;78.138.97.33

I run this at up to 300 separate threads on my PC (4 hyperthreaded CPU's with 8GB ram 400 threads crash) and because its hidden, I have no real knowledge its running in the background. It turns converting a file that would take a week or two, down to an overnight process.

The application is smart enough to share the writes to the same output file



Edited 1 time(s). Last edit at 03/20/2013 07:46AM by JohnK.
Re: Auto build PeerBlock lists from Command line Utility
February 26, 2017 07:09PM
Is there a Linux version of this utility?
Sorry, only registered users may post in this forum.

Click here to login