Skip to content

common: don't assume build script output is utf-8

username-removed-22388 requested to merge mvdan/fdroidserver:popen-nonutf8 into master

FDroidPopen is used for running many commands - from git to gradle to custom commands via flags like build=. When any of these invoke calls to custom build systems or upstream programs/scripts, it's not safe to assume that the output will be utf8.

Unfortunately, this currently leads to crashes and failed builds:

ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
    options.onserver, options.refresh):
  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
    build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
    p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
  File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
    result.output = result.output.decode('utf-8')
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte

One way to fix this would be to use one of the python libraries that guess an encoding. But a much safer option is to tell the decode method to ignore non-utf8 bytes, as opposed to crashing on them.

Merge request reports