Appveyor fails test_ntor.sh and test_hs_ntor.sh

FAIL: src/test/test_ntor.sh
2571===========================
2572
2573Traceback (most recent call last):
2574  File "C:/projects/tor/i686-w64-mingw32/../src/test/ntor_ref.py", line 402, in <module>
2575    test_tor()
2576  File "C:/projects/tor/i686-w64-mingw32/../src/test/ntor_ref.py", line 368, in test_tor
2577    c2s_msg, c_state = tor_client1(node_id, pubkey_B)
2578  File "C:/projects/tor/i686-w64-mingw32/../src/test/ntor_ref.py", line 347, in tor_client1
2579    stdout=subprocess.PIPE)
2580  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 709, in __init__
2581    restore_signals, start_new_session)
2582  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 971, in _execute_child
2583    args = list2cmdline(args)
2584  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 461, in list2cmdline
2585    needquote = (" " in arg) or ("\t" in arg) or not arg
2586TypeError: a bytes-like object is required, not 'str'
2587OK
2588FAIL src/test/test_ntor.sh (exit status: 1)
2589
2590

FAIL: src/test/test_hs_ntor.sh
2591==============================
2592
2593DONE: python dance [b'Cc\x0e"k\'*\xc4\x8a\x18\xc2\xfcN\xac\x8e(\xaa\x14\xb1\xccqqCN"\x9f\x9b\xd5W@\x94O']
2594Traceback (most recent call last):
2595  File "C:/projects/tor/i686-w64-mingw32/../src/test/hs_ntor_ref.py", line 424, in <module>
2596    do_little_t_tor_ntor_test()
2597  File "C:/projects/tor/i686-w64-mingw32/../src/test/hs_ntor_ref.py", line 310, in do_little_t_tor_ntor_test
2598    client_ephemeral_enc_privkey, subcredential)
2599  File "C:/projects/tor/i686-w64-mingw32/../src/test/hs_ntor_ref.py", line 248, in tor_client1
2600    stdout=subprocess.PIPE)
2601  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 709, in __init__
2602    restore_signals, start_new_session)
2603  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 971, in _execute_child
2604    args = list2cmdline(args)
2605  File "C:\msys64\mingw32\lib\python3.6\subprocess.py", line 461, in list2cmdline
2606    needquote = (" " in arg) or ("\t" in arg) or not arg
2607TypeError: a bytes-like object is required, not 'str'
2608FAIL src/test/test_hs_ntor.sh (exit status: 1)
2609
2610

changed milestone to %Tor: 0.3.4.x-final

added 034-must appveyor component::core tor/tor milestone::Tor: 0.3.4.x-final owner::nickm priority::high python3 resolution::fixed severity::normal status::closed tor-ci type::defect labels

Trac:
Component: Core Tor/TorDNSEL to Core Tor/Tor

I wonder if this is related to our recent change in #26372 (moved).

If so we might need to backport a fix for this.

Trac:
Priority: Medium to High

Yeah, python3 considers str to be Unicode, and bytes to be, well, bytes.

I'm surprised that this isn't affecting other platforms, though. Why would this be windows-only?

Replying to nickm:

I'm surprised that this isn't affecting other platforms, though. Why would this be windows-only? Maybe python2, not python3, is installed as python on most non-Windows platforms by default? (or at least the ones we run CI on?)

FWIW, this code works for me on Linux with both Python2 and Python3. So whatever is going on here, it's a Windows-specific python thing.

So ahf confirms that Python 3 on Windows does not accept the bytes() type on subprocess.Popen. (Python 3.6.5.) And I can confirm that on Linux it does.

Trac:
Owner: N/A to nickm
Status: new to accepted

Possible fixes in bug26535_029 and bug26535_032. Respective PRs at https://github.com/torproject/tor/pull/198 and https://github.com/torproject/tor/pull/199 .

Trac:
Status: accepted to needs_review

Trac:
Keywords: tor-ci appveyor deleted, tor-ci appveyor 034-must python3 added

These patches have passed the Windows appveyor CI, and they look good to me. Linux CI on travis also passed.

Trac:
Status: needs_review to merge_ready

Merged to maint-0.2.9 and maint-0.3.2 respectively, then forward.

Trac:
Status: merge_ready to closed
Resolution: N/A to fixed

closed

moved to tpo/core/tor#26535 (closed)