Repository index update does not remove applications that are not available any more

Hi,

we're creating the index.jar dynamically for each user (hence the HTTP Basic Auth merge request), but if an application becomes unavailable for a user, it is not removed from the list of available applications after updating the repository index.

The same happens if I add applications after the initial update, no new applications show up until I re-create the repo.

The etag of the index.jar changes with every modification to the index, as well as the timestamp attribute of the <repo> element in the index.xml.

How can I force and update of the repository index to the client?

Thanks and best regards, Christian

This should be working, and is definitely working for our repo.

Look at the logcat to see if the client is really updating the repo. If it is and the index is large, the process could take seconds or minutes.

Will do, thank you!

Does the client do caching at all, and if yes, based on what information?

I can now confirm that an application will only be removed from the list of available apps if I disable the repo and enable it again. Also, if I remove and re-add an application, it is shown with a slightly lighter text color, and the version number is missing.

I suspect that the client does not completely remove the app from its datastore, hence the question on what information the client bases its caching (if any).

The client keeps an sqlite database, which is updated whenever a repo update happens. Disabling the repo forces all apks belonging to it to be removed, and we remove apps without any apks from the database afterwards.

If this was a public repo, I would test myself and confirm this. But I wouldn't be able to, since I'm assuming this is a private repo with auth.

Otherwise, if you give two sample indexes we can use, that should work. E.g. one index with apps A and B, and another with apps A. Then B should remain in the app list, if what you say is correct.

Also, what I said above about "this works for us" might not be true, as we don't remove apps very often. But to the best of my knowledge it should work.

Thanks for the bug report @cmorgner. I had a look, and it seems that you are onto something. Check out how the RepoPersister initialises the temporary table so that it can cache the results of an index update.

It takes a copy of the existing app and apk tables (note ane app has multiple apks, potentially from different repositories):

TempAppProvider.Helper.init(context);
TempApkProvider.Helper.init(context);

At this point, it has an entire copy of all apks it previously knew about. It then proceeds to add information about new apks, or update metadata about existing apks as it sees them in the metdata (insertOrUpdateApks). However it doesn't seem to ever be removing apks.

This used to happen in v0.97 when we managed the update process in memory before persisting it all to the database in one hit. This caused memory problems for low end devices, so v0.98 now manages the update process in temporary tables before copying the temporary tables to the real tables used by F-Droid (hence the slower update process in v0.98):

Here is the code from v0.97, that I accidentally didn't port across to the new update process in v0.98

    /**
     * If a repo was updated (i.e. it is in use, and the index has changed
     * since last time we did an update), then we want to remove any apks that
     * belong to the repo which are not in the current list of apks that were
     * retrieved.
     */
    private void removeApksNoLongerInRepo(List<Apk> apksToUpdate, List<Repo> updatedRepos) {

        long startTime = System.currentTimeMillis();
        List<Apk> toRemove = new ArrayList<>();

        final String[] fields = {
            ApkProvider.DataColumns.APK_ID,
            ApkProvider.DataColumns.VERSION_CODE,
            ApkProvider.DataColumns.VERSION,
        };

        for (final Repo repo : updatedRepos) {
            final List<Apk> existingApks = ApkProvider.Helper.findByRepo(this, repo, fields);
            for (final Apk existingApk : existingApks) {
                if (!isApkToBeUpdated(existingApk, apksToUpdate)) {
                    toRemove.add(existingApk);
                }
            }
        }

        long duration = System.currentTimeMillis() - startTime;
        Utils.debugLog(TAG, "Found " + toRemove.size() + " apks no longer in the updated repos (took " + duration + "ms)");

        if (toRemove.size() > 0) {
            ApkProvider.Helper.deleteApks(this, toRemove);
        }
    }

This worked in the past because we had a list of all apks from the index in memory. However now we don't, because we stream the data from the index into the database then throw it away to avoid memory problems. However, it may be even better with this new setup. Instead of:

• Querying database for IDs of all existing apks
• Iterating over apks from the new index
• Asking if the existing apk is in the index
• Remov it if it is

We can now do the following:

• LEFT JOIN the temp apk table onto the real apk table
• Only fetch apks from the real apk table that have no record in the temp table
• Delete based on this result.

Here is an example from stack overflow (which is freakishly similar to our case of a cache + main table :)):

DELETE FROM cache
WHERE id IN (
  SELECT cache.id
  FROM cache
  LEFT JOIN main ON cache.id = main.id
  WHERE main.id IS NULL
);

The join will be more complex for our case, because we only want to remove things from the cache that belong to the repo we are currently updating.

I'm happy to prepare a fix for this to go into a future v0.99 release, but be aware that although I think I know how to fix it, it will take some time. This is because I want to create some tests for it to add to the test suite. This will likely involve creating some mock indexes that will cause the problem described. From my experience adding various tests to F-Droid, this always takes longer than I'd hope.

Milestone changed to 0.99

Added regression label

Added bug label

Added update label

I was about to tag as regression and for the upcoming release - you were faster than me!

Milestone removed

Dropping this off 0.99 since we want to do a release really soon and don't have the time to fix this with the proper testing support. Even though it's a regression, it's not a terrible one.

Milestone changed to 1.0

mentioned in merge request !232 (merged)

Milestone changed to 1.0

mentioned in issue #1071 (closed)

mentioned in issue #1028 (closed)

changed milestone to %1.0-alpha0

added bazaar label

changed milestone to %1.0

assigned to @pserwylo

@pserwylo Can you take a look at this? Seems important with Repomaker coming along. I looked through it a bit, but it seems the fix is SQL which I'm definitely weak at.

mentioned in merge request !564 (merged)

closed via merge request !564 (merged)