Skip to content
Snippets Groups Projects
New issue look: Off

MySQL Error when updating repo: Incorrect string value

    • Open Issue created by username-removed-26331 @grote

    When updating the F-Droid.org repo with a MySQL database, we get this error:

    Jun 09 14:02:20 repomaker.grobox.de python3[21288]: ERROR:background_task.tasks:Rescheduling maker.tasks.update_remote_repo
Jun 09 14:02:20 repomaker.grobox.de python3[21288]: Traceback (most recent call last):
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/local/lib/python3.5/dist-packages/django/db/backends/utils.py", line 65, in execute
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     return self.cursor.execute(sql, params)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/local/lib/python3.5/dist-packages/django/db/backends/mysql/base.py", line 101, in execute
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     return self.cursor.execute(query, args)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 226, in execute
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     self.errorhandler(self, exc, value)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     raise errorvalue
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 217, in execute
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     res = self._query(query)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 378, in _query
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     rowcount = self._do_query(q)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/cursors.py", line 341, in _do_query
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     db.query(q)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:   File "/usr/lib/python3/dist-packages/MySQLdb/connections.py", line 280, in query
Jun 09 14:02:20 repomaker.grobox.de python3[21288]:     _mysql.connection.query(self, query)
Jun 09 14:02:20 repomaker.grobox.de python3[21288]: _mysql_exceptions.OperationalError: (1366, "Incorrect string value: '\\xF0\\x9F\\x9A\\x80' for column 'l_summary' at row 1")

    We should of course fix this error, but maybe also catch this exception so only this problematic app isn't added and background task processing isn't stopped.

    Ideally, we also run our tests on a MySQL database.

    Designs

    Child items 0

    • No child items are currently open.

    Activity

    • username-removed-26331 assigned to @grote

      assigned to @grote

    • username-removed-26331
      username-removed-26331 @grote ·
      Author Maintainer

      The database tables need to be set up with character encoding utf8mb4 to support the characters that cause this issue. Problem is that with this character set, django-background-tasks can not be used since the character limit for database indexes is shorter with utf8mb4, shorter than the limit that django-background-tasks wants.

      There's a similar issue with django-allauth, but this can be worked around with. For now, the workaround I use is to manually change the character set of the relevant tables only:

      # Fix character sets :(
mysql -u ${MYSQL_USER} -p${MYSQL_PASS} -e "USE repomaker; ALTER TABLE `maker_app` CHANGE `summary` `summary` VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL, CHANGE `description` `description` LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;"
mysql -u ${MYSQL_USER} -p${MYSQL_PASS} -e "USE repomaker; ALTER TABLE `maker_app_translation` CHANGE `l_summary` `l_summary` VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL, CHANGE `l_description` `l_description` LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;"
mysql -u ${MYSQL_USER} -p${MYSQL_PASS} -e "USE repomaker; ALTER TABLE `maker_remoteapp` CHANGE `summary` `summary` VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL, CHANGE `description` `description` LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;"
mysql -u ${MYSQL_USER} -p${MYSQL_PASS} -e "USE repomaker; ALTER TABLE `maker_remoteapp_translation` CHANGE `l_summary` `l_summary` VARCHAR(255) CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL, CHANGE `l_description` `l_description` LONGTEXT CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci NOT NULL;"

      The Django database configuration also needs to include:

              'OPTIONS': {
            'charset': 'utf8mb4',
        },

      I also created a ticket for django-background-tasks. If that is fixed, repomaker can run normally on (and requires) a utf8mb4 MySQL database (or PostgresSQL)

      Edited by username-removed-26331
    • username-removed-26331 added Doing and removed ~2112488 labels

      added Doing and removed ~2112488 labels

    • username-removed-26331
      username-removed-26331 @grote ·
      Author Maintainer

      Some more context:

      Jun 15 16:35:11 repomaker.grobox.de python3[28900]: Traceback (most recent call last):
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/tasks.py", line 51, in update_remote_repo
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     remote_repo.update_index()
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/repository.py", line 402, in update_index
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     raise e
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/repository.py", line 398, in update_index
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     self._update(repo_index, update_apps)  # also saves at the end
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/repository.py", line 444, in _update
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     self._update_apps(repo_index['apps'], repo_index['packages'])
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/repository.py", line 476, in _update_apps
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     changed = app_obj.update_from_json(app)  # this also saves the app_obj
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/app.py", line 285, in update_from_json
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     self._update_translations(app['localized'])
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/app.py", line 327, in _update_translations
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     self.apply_translation(language_code, translation)
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/var/local/repomaker/maker/models/app.py", line 344, in apply_translation
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     self.save()
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:   File "/usr/local/lib/python3.5/dist-packages/hvad/models.py", line 271, in save
Jun 15 16:35:11 repomaker.grobox.de python3[28900]:     translation.save(*args, **tkwargs)
    • username-removed-26331 added bazaar label

      added bazaar label

    Please register or sign in to reply