Looking for advice / benchmarks on sources

[nohehf]

Hey, are there any existing benchmarks on the various sources that can be used in Subfinder?
I want to fine-tune subfinder to achieve a nice speed/results ratio, and I'm considering using private sources (that need an API key); however, it is pretty hard to tell which sources are with it or not. Would you have any advice on working with the said sources?
Thank you!

[MetzinAround]

hi @nohehf.
using subfinder -ls will list all the sources used by subfinder.

[MetzinAround]

https://github.com/projectdiscovery/subfinder#post-installation-instructions

[nohehf]

I was not looking for the list of sources, but for a performance/cost etc. comparison, if any had been done, to know which options are worth considering. I've done a little digging on my own. I'll clean up and publish the results in that discussion next week :)

[nohehf]

Benchmark

So I benchmarked the "private" sources, comparing them with the default ones on 3 company domains: one small, one medium, one large.

Here is the rough table I came up with:

I marked “somewhat interesting sources” with 🟠, and the most interesting sources with ⭐️ (in terms of performance).

Some tests were not ran because the performance was already too poor on the small domain.

source	notes	results on “small domain”	results on “medium domain”	results on “large domain”
bevigil		0	1
binaryedge		1 - 900ms
bufferover		0
c99	no free trial but 5$ - not enterprise but might be worth to try
censys	no free trial - seems very enterprise
🟠 sslmate / certspotter	https://sslmate.com/ct_search_api/ Free trial → 10req / hour Pricing: 50$/mo →100req/h 500$ → 1000req/h	154 - 3.667s	187 - 2.2s	910 - 35s
⭐️ chaos - by projectdiscovery	No info on the limits / paid plan	134 - 393ms	87 - 835ms	6560 - 816ms
chinaz	chineese, no idea on how to get the api key
dnsdb → domaintools	very enterprise, no pricing, no trial
fofa	non-comercial free tier → no api access but seems powerful from using the platform
🟠 fullhunt	enterprise pricing	539ms 28	920ms 77	4.341s 784
github	no idea on github api limits & commercial use policy. veryyyy slow	8 - 3m
intelX		0
passivetotal → riskiq	microsoft login, no idea on free tier
quake		3 - 4s
robtex	google sso only. no idea on the limits. needs “credits”
⭐️ security trails	free trial 50req/mo then 500$/mo. nice but really expensive	894ms 146	767ms 151	1.438s 2000
shodan	from 69$/mo (also $49 one time). not usable on free tier	0
threatbook	seems down
🟠 virustotal	free quota seems resonable, then it is custom enterprise pricing	703ms 17	405ms 12	946ms 100
⭐️ whoisxmlapi	pricing: https://whois.whoisxmlapi.com/pricing. from 30$ / mo	1.223s 126	1.291s 254	2.498s 5566
zoomeye	needs sms verification
dnsrepo	Please contact mailto:[email protected] to get started.
hunter	needs phone number
facebook	no idea on how to get the api key
All default sources		3,8s 144 alienvault 1.944s 324 0 anubis 647ms 91 0 crtsh 6.311s 1726 0 digitorus 377ms 3 0 dnsdumpster 2.457s 80 0 hackertarget 849ms 38 0 leakix 314ms 10 0 riddler 179ms 0 1 virustotal 0s 0 0	6,3s 106	1552 - 7,1s
default + chaos, securitytrails & whoisxmlapi		2,4s 158 alienvault 1.944s 225 0 anubis 653ms 0 1 chaos 480ms 134 0 crtsh 569ms 1291 0 digitorus 739ms 15 0 dnsdumpster 2.45s 57 0 hackertarget 894ms 60 0 leakix 315ms 10 0 riddler 226ms 0 1 securitytrails 973ms 146 0 virustotal 0s 0 0 whoisxmlapi 872ms 126 0	4387s 335 alienvault 4.295s 324 0 anubis 215ms 91 0 chaos 330ms 87 0 crtsh 1.103s 1726 0 digitorus 388ms 3 0 dnsdumpster 2.389s 80 0 hackertarget 619ms 0 0 leakix 289ms 10 0 riddler 205ms 0 1 securitytrails 766ms 151 0 virustotal 0s 0 0 whoisxmlapi 1.608s 254 0	2,4s - 7084 alienvault 1.436s 502 0 anubis 350ms 0 1 chaos 387ms 6562 0 crtsh 1.263s 7802 0 digitorus 330ms 0 0 dnsdumpster 2.376s 213 0 hackertarget 638ms 0 0 leakix 371ms 10 0 riddler 164ms 0 1 securitytrails 1.637s 2000 0 virustotal 0s 0 0 whoisxmlapi 2.447s 5566 0

My conclusion is that the 3 most interesting sources are:
- Chaos
- SecurityTrails
- WhoisXML API

Sorry for the roughness of these results. It was meant to be personal at first. I think it might be useful to publish it. Let me know if it's not clear enough.