Implement proposal 315: "Updating the list of fields required in directory documents"

Proposal 315 describes a general policy for making once-optional fields in directory documents mandatory.

Here are the steps I mean to take to implement it:

• Review all the fields in dir-spec.txt to see whether the list of fields to be made required in prop315 is in fact accurate
• Review to see whether the list of fields is complete.
• Verify that all (or nearly all) of the descriptors in today's network have all the required fields.
• Mark each of the fields as required in src/features/dirparse. (Since there are no onion-service fields on the list, this can be done in one step.)
• Simplify code in src/features/dirparse that treated these fields as optional.
• Update unit tests to pass when these fields are required.
• Update dir-spec.txt to mark the fields as required as of the version that first required them.
• Mark hidden-service-dir as assumed-present.

changed milestone to %Tor: 0.4.5.x-freeze

added Backlog label

@tpo/core , @arma -- The proposal describes a two step approach for making fields required in routerdescs: it says that first we should make authorities require the fields, and only after that is done should we make relays or clients require the fields. I think that it might not be worthwhile to do it this way; the possible partitioning attack(*) seems pretty weak, and pretty detectable. What do you think?

(*) The attack would be to deliberately upload descriptors that the authorities will accept, but which some clients won't accept, in hopes that those clients will stand out on the network. This attack is kinda weak because you can't do it without making many tor instances spew warnings about your relays.

mentioned in issue #40133 (closed)

Yeah I'm not too worried about this attack. I just read the phased process of the proposal and this means 3-4 years before completing it which seems quite intense for such a weak attack.

Agreed with @dgoulet !

added Next label and removed Backlog label

added Doing label and removed Next label

marked the checklist item Review all the fields in dir-spec.txt to see whether the list of fields to be made required in prop315 is in fact accurate as completed

marked the checklist item Verify that all (or nearly all) of the descriptors in today's network have all the required fields. as completed

marked the checklist item Mark each of the fields as required in src/features/dirparse. (Since there are no onion-service fields on the list, this can be done in one step.) as completed

changed the description

added 30m of time spent at 2020-10-15

marked the checklist item Update unit tests to pass when these fields are required. as completed

marked the checklist item Review to see whether the list of fields is complete. as completed

Updating the unit tests was not trivial. We had a bunch of hardwired objects throughout the tests -- and some of them required that the signatures be correct! Generating correctly signed objects make me dust off an old python script, port it to python3, and reconstruct the missing template files that were used to generate its outputs in the first place.

Doing this took up a lot of time. But with any luck, it will make our tests a bit more flexible going forward.

added 6h of time spent at 2020-10-16

mentioned in merge request !174 (merged)

Merge request at !174 (merged).

Spec merge request at torspec!14 (merged).

marked the checklist item Update dir-spec.txt to mark the fields as required as of the version that first required them. as completed