Assertion failure in dns.c, possibly connected to UDP DoS attack
I run a full exit node, and today I was hit by a udp denial of service. Standard botnet garbage, really. But something surprising happened:
Jan 7 10:54:51 testbed Tor[4289]: Circuit handshake stats since last time: 92457/94799 TAP, 209527/209991 NTor. Jan 7 15:28:03 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:28:03 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:28:03 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:28:03 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:28:48 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:28:48 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:28:48 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:28:48 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:29:01 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:29:01 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:29:01 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:29:01 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:29:23 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:29:23 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:29:23 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:29:23 testbed Tor[4289]: eventdns: Nameserver 8.8.8.8:53 is back up Jan 7 15:30:04 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 15:30:04 testbed Tor[4289]: eventdns: All nameservers have failed Jan 7 16:06:15 testbed Tor[4289]: tor_assertion_failed_(): Bug: src/or/dns.c:1136: connection_dns_remove: Assertion 0 failed; aborting. Jan 7 16:06:15 testbed Tor[4289]: tor_assertion_failed_(): Bug: src/or/dns.c:1136: connection_dns_remove: Assertion 0 failed; aborting. Jan 7 16:06:15 testbed Tor[4289]: Bug: Assertion 0 failed in connection_dns_remove at src/or/dns.c:1136. Stack trace: Jan 7 16:06:15 testbed Tor[4289]: Bug: Assertion 0 failed in connection_dns_remove at src/or/dns.c:1136. Stack trace: Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(log_backtrace+0x29) [0x50dbe9] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(log_backtrace+0x29) [0x50dbe9] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_assertion_failed_+0x7a) [0x51af6a] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_assertion_failed_+0x7a) [0x51af6a] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(connection_dns_remove+0x240) [0x4fa6e0] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(connection_dns_remove+0x240) [0x4fa6e0] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x428ed9] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x428ed9] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x429209] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x429209] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/lib64/libevent-2.0.so.5(event_base_loop+0x40c) [0x3758b7d8f5c] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/lib64/libevent-2.0.so.5(event_base_loop+0x40c) [0x3758b7d8f5c] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(do_main_loop+0x215) [0x42afd5] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(do_main_loop+0x215) [0x42afd5] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_main+0x15d5) [0x42dc05] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor(tor_main+0x15d5) [0x42dc05] Jan 7 16:06:15 testbed Tor[4289]: Bug: /lib64/libc.so.6(__libc_start_main+0xf5) [0x3758abecab5] Jan 7 16:06:15 testbed Tor[4289]: Bug: /lib64/libc.so.6(__libc_start_main+0xf5) [0x3758abecab5] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x427e31] Jan 7 16:06:15 testbed Tor[4289]: Bug: /usr/bin/tor() [0x427e31]
I have been having issues with DNS connectivity due to the high packet (~100k pps) rate, which means tor has been without consistent DNS for about a half hour before this died.
Having tor crash was extremely surprising.
I am running tor on Gentoo, tor version:
tor --version
Tor version 0.2.6.1-alpha (git-5a601dd2901644a5).
My suspicion is that DNS requests piled up in an internal tor buffer of some sort, and that got maxed out resulting in an oops.
I'll try to keep an eye on this. Let me know if more information is needed for debugging.
Trac:
Username: jowr