Tool not moving on with no events in queue?

[felipewarrener]

Maybe I'm confused about how this tool is supposed to work:

Some scans just run forever without moving on after putting zero items in the queue, and I'm not sure why, I've checked the docs and I can't see a timeout value for the entire scan to move on upon having the queue empty. This is a big deal for me as I am getting out of memory errors when running a single instance of bbot with 12,000+ domains - so I resorted to invoking it 12,000 times using xargs, but xargs will not move onto the next instance until the current fork is killed, so this halts my loop.

For example: bbot -t 100daysinthecloud.com -rf passive -f subdomain-enum

For me, this will just loop telling me there's no events in queue, over and over:

[INFO] golden_bombadil: No events in queue
[DNS_NAME]              100daysinthecloud.com   TARGET  (a-error, aaaa-error, cname-error, domain, in-scope, mx-error, ns-error, soa-error, srv-error, target, txt-error, unresolved)
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue
[INFO] golden_bombadil: Modules running (incoming:processing:outgoing) anubisdb(0:1:0), certspotter(0:1:0), columbus(0:1:0), crt(0:1:0), digitorus(0:1:0), dnscommonsrv(0:1:0), dnsdumpster(0:1:0), github_codesearch(0:1:0), hackertarget(0:1:0), leakix(0:1:0), massdns(0:1:0), myssl(0:1:0), otx(0:1:0), postman(0:1:0), rapiddns(0:1:0), riddler(0:1:0), shodan_dns(0:1:0), sitedossier(0:1:0), subdomaincenter(0:1:0), threatminer(0:1:0), urlscan(0:1:0), virustotal(0:1:0), wayback(0:1:0)
[INFO] golden_bombadil: Events produced so far: SCAN: 1, DNS_NAME: 1
[INFO] golden_bombadil: No events in queue

[TheTechromancer]

Hmm, this scan completes for me in under 20 seconds. Can you run it with -c http_debug=true dns_debug=true --debug, and attach your debug.log?

[danielgh94]

Hello,
The same problem occurred with the following command:
bbot -t domains.txt -f cloud-enum -y -c http_debug=true dns_debug=true --debug -n bbot -o .
Please find attached debug.log.
Hopefully it will be able to solve this issue globally.
If you search for "No events in queue" you will see it repeats endlessly.
Also, the debug log is too large for github, it's 59 MB while the limit is 25 MB.
For that reason, I removed the first half of the log. I hope it's ok.

debug.log

[TheTechromancer]

@danielgh94 thanks for the log, that's really helpful. In this case it looks like baddns is the misbehaving module:

2024-06-21 22:55:24,770 [DEBUG] bbot.scanner scanner.py:895             tasks:
2024-06-21 22:55:24,770 [DEBUG] bbot.scanner scanner.py:895                 - baddns.handle_event(DNS_NAME("hyundaimobil.co.id", module=TARGET, tags={'target', 'ns-record', 'soa-record', 'resolved', 'a-record', 'domain', 'in-scope'})) running for 3 minutes, 29 seconds:
2024-06-21 22:55:24,771 [DEBUG] bbot.scanner scanner.py:895                 - baddns.handle_event(DNS_NAME("www.hyundaimobil.co.id", module=excavate, tags={'subdomain', 'a-record', 'resolved', 'in-scope'})) running for 1 minute, 13 seconds:
2024-06-21 22:55:24,771 [DEBUG] bbot.scanner scanner.py:895                 - baddns.handle_event(DNS_NAME("www.hyundai-mk.mk", module=excavate, tags={'ns-record', 'in-scope', 'soa-record', 'resolved', 'txt-record', 'subdomain', 'a-record', 'mx-record', 'cname-record'})) running for 42 seconds:

It's taking upwards of 1 minute per domain.

@liquidsec can you take a look at this one?

[TheTechromancer]

@felipewarrener in your case, was 100daysinthecloud.com the domain that it got stuck on?

[TheTechromancer]

Baddns slowness fixed in #1502 (BBOT 2.0)