Store categories in separate category table

Currently, the category that an app is in is recorded in the database via the fdroid_app.categories column, containing a comma separated list of strings. This makes it hard to query. The existing code to get a list of categories was pretty bad as a result.

This moves to a different data model whereby categories are stored in a separate table. Each repo is free to specify that an app is in arbitray caregories (as with before). This is represented by a join table between categories and app metadata.

The end result is that categories are much more a first class citizen than before, and they will be able to be queried easier - which is important for the new UI.

Note that the categories table need never be emptied, it can keep being appended to. The reason is that if there are no apps in a particular category (represented by no corresponding rows in the join table) then the category will not be shown to the user.

Added 1 commit:

• b02d113e - Use the Query object rather than the SQLiteDatabase#query() method.

Compare with previous version

looks good to me, feel free to merge once the build passes, or ping me to merge it.

As a sidenote, I know that the whole database API is ContentProviders but I wonder whether it would be worthwhile to incrementally migrate to a simpler method that works for internal-only APIs like the F-Droid database stuff. ContentProviders add a lot of overhead conceptually since they are meant for sharing data between apps, etc.

Added 18 commits:

• b02d113e...dd5f84f2 - 7 commits from branch fdroid:master
• 21b91f69 - Removed unused category view in app details.
• 450c0e5a - Comments for the Schema.PackageTable.
• bab09067 - Added "category" table and "category to app metadata" join table.
• d393fa61 - Added category provider.
• 993cc26b - Added category to keyword search.
• d3e03eba - When querying based on category, use join table.
• 1ec7bb1e - Refactored categories field from column in metadata table to join table.
• b7bd1424 - Move category helper functions to CategoryProvider
• 4ed7c5a2 - Make category helper functions query new category table
• 3143f2cc - Make it explicit that the CATEGORIES column is not for selecting from.
• a46dd270 - Use the Query object rather than the SQLiteDatabase#query() method.

Compare with previous version

Added 14 commits:

• a46dd270...6f0b33a0 - 2 commits from branch fdroid:master
• 19ca68cb - Removed unused category view in app details.
• bb7cfc14 - Comments for the Schema.PackageTable.
• 99f7cab6 - Added "category" table and "category to app metadata" join table.
• 8757acca - Added category provider.
• 3e3fdd5c - Added category to keyword search.
• b2d5bcc9 - When querying based on category, use join table.
• a7a7f77b - Refactored categories field from column in metadata table to join table.
• 634fe108 - Move category helper functions to CategoryProvider
• 68f66668 - Make category helper functions query new category table
• 354f0a9b - Make it explicit that the CATEGORIES column is not for selecting from.
• 97855369 - Use the Query object rather than the SQLiteDatabase#query() method.
• 2263352c - Add test for miscellanious update from f-droid.org metadata

Compare with previous version

As a sidenote, I know that the whole database API is ContentProviders but I wonder whether it would be worthwhile to incrementally migrate to a simpler method that works for internal-only APIs like the F-Droid database stuff. ContentProviders add a lot of overhead conceptually since they are meant for sharing data between apps, etc.

I think you're right. These last few refactors which added a CategoriesProvider and PackageProvider show how much overhead there is. There are various solutions around, and I'll continue to keep an eye out. It wont be an easy change, but it probably could be done slowly over time.

After a bit fo a searching around, I've come up with a list of possible alternatives which I'll put in an issue rather than this MR.

Unmarked this merge request as a Work In Progress

Could I please request a CR of the top commit in this branch? I found a bug that was only hit during the update of an existing repository, rather than the original update of a repository. That made it clear that there was not sufficient test coverage, and so I added a test to verify that updating the official f-droid metadata works as expected. I have two copies of the metadata from about 10 days apart, hopefully to cover multiple scenarios (e.g. changing descriptions/links/other metadata for an app) which are not covered by other multi-repo tests.

The bug was that I wasn't passing query.getArgs() as the second argument here. It is now passed through correctly and rebased on master again.

This seems to be related to #795 and #754

Should this go into 0.102? Or wait for 0.103?

Status changed to merged