Improve Latests TBB Format to be easily parsable

added TorBrowserTeam201509R component::applications/gettor owner::boklm priority::medium resolution::fixed severity::normal status::closed type::enhancement labels

Replying to naif:

This ticket is to make up a new version of RecommendedTBBVersion in a simplified format, containing always and only 2 versions:

Stable one

Alpha one

We don't have one stable version and one alpha version recommended all the time for at least two reasons:

Releasing a Tor Browser which does not fix critical security bugs should not trigger the need for updates. Thus, it is perfectly fine to have, say, two versions of Tor Browser that are okay at a given moment in time.
The release process is organized in a way that we remove old versions only after some delay in order to give users the chance to get a Firefox update notification before their TorButton is blinking weirdly.

Thanks for the clarification gk. For what I understand, having more than one (stable and alpha) version recommended is a normal behaviour. In other words, it's not bad to use an old version of Tor Browser, as long as it is present in RecommendedTBBVersion. With this in mind, do you still consider it a problem, naif?

@ilv We can live with that but i do think it's not a good idea to keep it that way, given that a developer is looking at this file to get a simple answer to the question "Which is the latest version of Tor Browser?" .

Answering that way is misleading, adding parsing logic complexity to any third party implementation willing to take action based on the answer of that file.

I feel that an over-simplified version would be suitable, probably enriched with the URL where to download the version of the file that's reported in that file, so any third party developer will only get 1 answer and can immediately do a request to fetch it.

I agree, Tor Browser Launcher has this exact problem. At the moment it loads https://www.torproject.org/projects/torbrowser/RecommendedTBBVersions and then tries to parse it. There have been a few times in the past where RecommendedTBBVersions changed its format slightly, which entirely broken TBL.

This quickly becomes a complicated issue because of all of the different pieces of information that end up defining a URL: version (stable or experimental), platform, architecture, and language.

So the URLs to download the latest stable for Windows in English are:

https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe.asc

And the URLs to download the latest experimental for 32-bit Linux in Spanish are:

https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux32-5.0a3_es-ES.tar.xz https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux32-5.0a3_es-ES.tar.xz.asc

Whatever the ultimate solution is, it would be good to figure out how to encapsulate all of this information. It could be a single JSON object, something like this:

{
  "stable": {
    "version": "4.5.3",
    "downloads": {
      "en-US": {
        "windows": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/torbrowser-install-4.5.3_en-US.exe.asc"
        },
        "macos": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/TorBrowser-4.5.3-osx64_en-US.dmg",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/TorBrowser-4.5.3-osx64_en-US.dmg.asc"
        },
        "linux32": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux32-4.5.3_en-US.tar.xz",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux32-4.5.3_en-US.tar.xz.asc"
        },
        "linux64": {
          "binary": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz",
          "sig": "https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz.asc"
        }
      },
      "ar": ...,
      "de": ...,
      ...
    }
  },
  "experimental": {
    "version": "5.0a3",
    "downloads": ...
  }
}

Or it could be some sort of API, so these requests would return these responses:

/version/stable: 4.5.3

/version/experimental: 5.0a3

/stable/bin/linux64/en-US: https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz

/stable/sig/linux64/en-US: https://www.torproject.org/dist/torbrowser/4.5.3/tor-browser-linux64-4.5.3_en-US.tar.xz.asc

/experimental/bin/linux64/en-US: https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux64-5.0a3_en-US.tar.xz

/experimental/sig/linux64/en-US: https://www.torproject.org/dist/torbrowser/5.0a3/tor-browser-linux64-5.0a3_en-US.tar.xz.asc

Trac:
Cc: evilaliv3, ilv to evilaliv3, ilv, micahlee

@micahlee I'd really love to see implemented your proposed data structure by Tor!

It would be even cooler to see it implemented as part of CheckTor service, that way an active-client-side-component will be able in a single request to know if it's coming from Tor and which is the latests version of Tor Browser

An option could be to use the URLs used by the Firefox updater: https://dist.torproject.org/torbrowser/update_2/release/Linux_x86_64-gcc3/x/en-US https://dist.torproject.org/torbrowser/update_2/alpha/Linux_x86_64-gcc3/x/en-US

The appVersion attribute has the version number.

Trac:
Cc: evilaliv3, ilv, micahlee to evilaliv3, ilv, micahlee, boklm

Trac:
latest_tbversion.txt

Sample JSON output

boklm: nice! how are those file generated, by hand?

naif, micahlee: I did something for the first idea (based on this), with some minor changes in the structure. I'm attaching a sample of the JSON generated. You can check the script that generated it here. It would be interesting to know how RecommendedTBBVersion is generated, or the URLs mentioned by boklm; if these are generated by hand maybe there is a chance that someone could run the script on tpo on every TB release (or maybe a cronjob?). I've been working on something similar in #16601 (moved).

Replying to ilv:

boklm: nice! how are those file generated, by hand?

It is generated by the update_responses script in this directory: https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/tools/update-responses

Using the informations from this config.yml file: https://gitweb.torproject.org/builders/tor-browser-bundle.git/tree/tools/update-responses/config.yml

I think we can extend the script to generate a json file with download information, which would be updated at the same time a new version is made available in the updater.

Trac:
0001-Bug-16551-generate-a-downloads.json-file.patch

patch for update_responses to generate a downloads.json file

Trac:
downloads.json

Example downloads.json generated by update_responses

I attached a proposal for a patch to update_responses to make it generate a downloads.json file, in the same directory as the xml responses for the updater, and an example json file that it created. The URLs for this file would be: https://dist.torproject.org/torbrowser/update_2/release/downloads.json and https://dist.torproject.org/torbrowser/update_2/alpha/downloads.json

Replying to boklm:

I attached a proposal for a patch to update_responses to make it generate a downloads.json file, in the same directory as the xml responses for the updater, and an example json file that it created. The URLs for this file would be: https://dist.torproject.org/torbrowser/update_2/release/downloads.json and https://dist.torproject.org/torbrowser/update_2/alpha/downloads.json

Thank you boklm, this looks great. I think we are almost done here. I guess the next step should be to open a new ticket to apply these changes, but I'm not sure under what component, Service-dist maybe?

Replying to ilv:

Thank you boklm, this looks great. I think we are almost done here. I guess the next step should be to open a new ticket to apply these changes, but I'm not sure under what component, Service-dist maybe?

The next step is tagging the ticket with the TorBrowserTeam201508R keyword (I'm doing it now) to get the patch reviewed.

Trac:
Type: defect to enhancement
Cc: evilaliv3, ilv, micahlee, boklm to evilaliv3, ilv, micahlee, boklm, tbb-team
Keywords: N/A deleted, TorBrowserTeam201508R added
Status: new to needs_review

well done blklm/ilv/micahlee

i've simply a concern: your all comments are still missing to answer the first comment by gk that if i'm not wrong can be summarized as:

multiple stables (the one that during time has not become vulnerable and so should not be listed at all) may exist
generically as 1. multiple alphas/beta/something may exist at a time.

i think that your gerarchical data format for describing a precise version is perfect, but we miss a good wrapping object that would allow to access the relevant information in O(1).

here is my proposal (written fastly as i'm reading but i think it should really be considered and improved):

tor_versions { categories_list: ['stable', 'alpha', 'whatever'], category_summary: { 'stable': { latest: '1.1', versions[1.1, 1.2, ...] }, 'alpha;: { latest: '1.X', versions[1.X, 1.Y, ...] }, 'whatever': { ... } }, versions = { '1.1' = { your self describing data structure for a version}, '1.2' = { }, '1.X' = { }, } }

benefits of the wrapper object described above:

is it possible to have in O(1) the list of the categories
is it possible to have in O(1) the list of the version in a category
is it possible to get in O(1) the latest version of a category or in general the descriptor of a known version.

what do you think?

Improve Latests TBB Format to be easily parsable

Child items 0

Activity