Support other media types besides apps

Currently, repomaker only supports apps via APK files. In the future, it should also support other types of media such as videos, audios, books, etc.

Other media types are includes in the top-level JSON array apps (even if they are not apps) like so:

{
    "apps": [
        {
            "added": 1491606000000,
            "categories": [
                "System"
            ],
            "description": "<p><b>Note:</b> F-Droid will need root privileges to install this app as a system app.</p>",
            "lastUpdated": 1493247600000,
            "name": "org.fdroid.fdroid.privileged.ota_2040.zip",
            "packageName": "org.fdroid.fdroid.privileged.ota",
            "suggestedVersionCode": "2040",
            "suggestedVersionName": "0.2.4",
            "summary": "Help F-Droid acquire system privileges",
            "webSite": "https://f-droid.org"
        },
    ],
}

So it essentially supports all fields that apps also support, even suggestedVersionCode and suggestedVersionName.

The packageName can be freely chosen as long as it doesn't conflict with other apps in the index. It can be the sha265 hash of a file, the file name or something supplied by the user. Important is that the packageName is a key in the packages JSON object with at least one entry which looks like this:

{
    "packages": {
        "org.fdroid.fdroid.privileged.ota": [
            {
                "added": 1493247600000,
                "apkName": "org.fdroid.fdroid.privileged.ota_2040.zip",
                "hash": "6929cd908aaf72c0fe41887001bd9c636b5c6aba4b6899a2f0d29b0b46e6077e",
                "hashType": "sha256",
                "packageName": "org.fdroid.fdroid.privileged.ota",
                "size": 3679840,
                "versionCode": 2040,
                "versionName": "6929cd908aaf72c0fe41887001bd9c636b5c6aba4b6899a2f0d29b0b46e6077e"
            },
            {
                "added": 1492038000000,
                "apkName": "org.fdroid.fdroid.privileged.ota_2030.zip",
                "hash": "cafea902e34ec161e01ca6ec7c9b259180745fc210aa0b488eb1b3b62c6e61ec",
                "hashType": "sha256",
                "packageName": "org.fdroid.fdroid.privileged.ota",
                "size": 3677402,
                "versionCode": 2030,
                "versionName": "cafea902e34ec161e01ca6ec7c9b259180745fc210aa0b488eb1b3b62c6e61ec"
            }
        ],
    },
}

• The packageName needs to match the parent key and be the same for packages of that "app".
• The versionCode needs to be an integer and could automatically be set to the same value of added to ensure increasing values.
• The versionName is a string and can be set by the user. It might default to the string representation of added as well.
• The apkName is exactly the filename of the package in the repository (even if the package is not actually an APK).

@eighthave based on the example testy repo media files in the index seem to be only present in the packages section, but not in the apps section of the index. So I wonder where their metadata is living then?

The sha256 hash of the file seems to be used for the package ID, the packageName and versionName:

        "c501f949d766e7b56cc51bd924638887605ea58135a675d138dbdb7219a55f09": [
            {
                "added": 1493829765669,
                "apkName": "FDroidPrivilegedExtension-0.2-release.zip",
                "hash": "c501f949d766e7b56cc51bd924638887605ea58135a675d138dbdb7219a55f09",
                "hashType": "sha256",
                "packageName": "c501f949d766e7b56cc51bd924638887605ea58135a675d138dbdb7219a55f09",
                "size": 33448,
                "versionName": "c501f949d766e7b56cc51bd924638887605ea58135a675d138dbdb7219a55f09"
            }
        ],
        "c723f0a20bd2b3bb1dbf0b603edf98d6987f7909829bb4cdf5be9eb47f3d0f98": [
            {
                "added": 1493829764384,
                "apkName": "PanicKit Demo with Ripple and Orweb-mS1gstS6YS8.mp4",
                "hash": "c723f0a20bd2b3bb1dbf0b603edf98d6987f7909829bb4cdf5be9eb47f3d0f98",
                "hashType": "sha256",
                "packageName": "c723f0a20bd2b3bb1dbf0b603edf98d6987f7909829bb4cdf5be9eb47f3d0f98",
                "size": 10844682,
                "versionName": "c723f0a20bd2b3bb1dbf0b603edf98d6987f7909829bb4cdf5be9eb47f3d0f98"
            }
        ],

Also if the hash of a single file is used as the package_id how are updates of that file handled? I can easily imagine non-Android software or e-Books with several revisions being distributed that should show up as updates. The hash package_id doesn't seem to support that or am I missing something?

Its a bug in fdroidserver if there are packages that have no matching entry in apps.

The SHA-256 hash is the default packageName for non-APKs. In the future, the idea is to figure out a way for repo managers to set that, if they want. But maybe that's not really necessary, since there isn't really anything for fdroidclient to do in terms of "upgrading" if it is not an APK.

Maybe it makes more sense to use the current file name as the default packageName, since the filesystem where the repo lives will guarantee that it is unique. Its a big, open question. I don't think we need to answer it to provide useful media handling now.

Its a bug in fdroidserver if there are packages that have no matching entry in apps.

Good to know. So non-apps are actually part of the apps dictionary. That is a bit confusing, but fine with me if that design decision is final.

there isn't really anything for fdroidclient to do in terms of "upgrading" if it is not an APK

If it is an ebook or a spreedsheet, fdroidclient might want to replace the original file by the update. Or it at least wants to add the new version next to the old one, maybe by using a version suffix.

Using the hash as packageName seems to make all upgrade scenarios impossible for the future. In repomaker, we could easily let the user set that name or take it from the used file-name instead. It looks like the official repo's app org.fdroid.fdroid.privileged.ota uses a custom packageName as well, so I guess there's no real restriction on what the packageName can be.

Using hash as version name

The hash is very long, longer than most versions are. So the UI is not really prepared for it and even if it would be, hashes are not really suitable to be presented to users as version name. Here's how that currently looks like for org.fdroid.fdroid.privileged.ota:

So if allowed, I would prefer a different version name. If you don't want to support upgrades at all (which is strange because this non-app already has updates), we could just hide the versions completely. Alternatively, we could allow proper version names to be supplied instead of the hash.

Media Types

How are media types determined. Do you think it is sufficient to just check the extension of apkName (which is a strange name for a non-apk btw.) and determine the mime-type from that?

Or should we include a new field where one of the supported media types are specified?

I'm not saying I don't want to support updates. I'm saying it's a hard problem that can be solved later. We can start with useful cases that work fine without having full update support. That OTA ZIP is a good example: F-Droid can update that via the normal APK updating channel, so we don't need to figure out how to tell if one ZIP file is a standalone or an update. APKs have that info built in.

The version naming should be easy enough to improve now. To start with, seems like fdroidclient should trim that string to what it can cleanly display. versionName in APKs can also be very long, and its user-generated (i.e. manually set by the developer).

There are some old variable names in there that have not been changed yet. apps in JSON matches org.fdroid.fdroid.data.App in Java. apkName is another example of that. I think some of those names will have to stay as is, because it'll be a lot of work to change them, and they'll be hidden away in specific parts of the implementation. Other names we can start changing once things have settled down and people find time to do it.

@eighthave thanks for your explanations! I updated the description of this ticket with a short description of the format. It would be great if you could check whether it is documented correctly there.

It seems like there was a misunderstanding about the media types. My question was not so much about the outdated field names, but more about how to detect media types.

Do we want to support arbitrary files or just a white-list of file types?

In any case, how should we detect the file-type? Do you think it is sufficient to just check the extension of apkName and determine the mime-type from that? Or should we actually peek into the file?

It might be nice to have a field for the file-type in the index, so consumers of the index know right away what kind of "app" they are dealing with. If the apps JSON array is being kept and no ebooks or 'photos' arrays are introduced for example, then a type field might provide some guidance to consumers. What do you think?

FYI, you can do some kind of generic file updating right now, if you just follow the APK naming scheme of packageName_versionCode.ext. For example, Novnain_Whistles_at.or.at_2002.mp3 will become:

packageName: Novnain_Whistles_at.or.at
versionCode: 2002

changed milestone to %1.0

added dev label

assigned to @grote

added Doing and removed ~2112488 labels

mentioned in merge request !105 (merged)

closed via merge request !105 (merged)

removed Doing label