Backend for an async publish-subcribe messaging system for cross-module communications

In Mexico City I walked with Teor and Catalyst about a generic publish/subscribe system. Thanks for feedback from the network team, I have an initial API I like, and I've tried to do an implementation.

changed milestone to %Tor: 0.4.1.x-final

added actualpoints::18 asn-merge component::core tor/tor milestone::Tor: 0.4.1.x-final network-team-roadmap-2019-Q1Q2 owner::nickm points::8 priority::medium resolution::fixed reviewer::catalyst severity::normal sponsor31-maybe sponsor::31-must status::closed teor-merge tor-pubsub type::enhancement labels

See branch messaging in my public repository, with PR at https://github.com/torproject/tor/pull/452

Trac:
Status: assigned to needs_review

Next steps for me here:

• send the design document to tor-dev
• Write the API for testing modules that use this
• write the subsystem implementation.

(Declaring this new subsystem "tor-pubsub" :)

Trac:
Reviewer: N/A to catalyst
Keywords: N/A deleted, tor-pubsub added

Tor 0.3.6.x has been renamed to 0.4.0.x.

Trac:
Milestone: Tor: 0.3.6.x-final to Tor: 0.4.0.x-final

I've pushed a few more commits based on your comments, to try to improve naming and documentation.

Trac:
Sponsor: Sponsor8 to N/A
Keywords: N/A deleted, s31-maybe added

Trac:
Keywords: s31-maybe deleted, sponsor31-maybe added

I've made a new branch as messaging_v2, with a PR at https://github.com/torproject/tor/pull/648.

It refactors the old branch considerably, and incorporates a bunch of your suggestions.

(It does not switch from using "auxdata" to using a subtype of "msg_t" for additional data as we had suggested, since I am not sure about that yet. We can do that part later if need be.)

Trac:
Milestone: Tor: 0.4.0.x-final to Tor: 0.4.1.x-final

Trac:
Sponsor: N/A to Sponsor31-can

Note from Catalyst: In "Add MESG as a new log domain.", N_LOGGING_DOMAINS needs to be incremented.

Partial review so far, by commit:

b7358da11 namemap:: seems ok. I have a few misgivings about the type punning used for the temporary object, but it might be safe d26a213c9 LD_MESG:: seems ok, except for needing to bump the total as mentioned above 749456940 semicolon eater:: seems ok. Maybe we want to uses this in ht.h, but that means changing all its users, so maybe not in this ticket 33c20c70d smartlist_grow:: seems ok, assuming the sizes bounds check in smartlist_ensure_capacity() is correct, which from a quick glance it seems to be

Trac:
Sponsor: Sponsor31-can to Sponsor31-must
Keywords: N/A deleted, network-team-roadmap-2019-Q1Q2 added
Points: N/A to 8

Made some comments on github; still reviewing.

I wrote a proof of concept hooking up part of the new bootstrap reporting code to pubsub. https://github.com/tlyu/tor/tree/bootpubsub

It builds but doesn't pass tests, because I haven't yet figured out how to hook up the pubsub mainloop stuff.

I commented in the pull request about most of difficulties I experienced with the code and consuming-developer documentation. There are a few more, but I think they're minor documentation issues.

I have some comments on architecture and design that it might not be worth delaying integration to address. I'll write them up soon.

How much work is it to finish hooking up pubsub to the mainloop and initialization code? Would you be willing to add commits to do that? It would be great to be able to test my POC. Thanks.

Trac:
Status: needs_review to needs_revision

Trac:
Status: needs_revision to needs_review

The branch is now squashed as messaging_v3, with a new PR for CI purposes at https://github.com/torproject/tor/pull/844

Rebased and checked my bootpubsub branch against the new pull request; seems good!

Right now I think these changes are good enough to merge. I still have comments about technical debt and long-term risks, but I'll put those in a separate ticket. (Some of these overlap with stuff Nick has sent me in private feedback.) It's probably better to clean those up later now that we have a working proof of concept of a consumer of this API.

Trac:
Status: needs_review to merge_ready