common: don't assume build script output is utf-8

FDroidPopen is used for running many commands - from git to gradle to
custom commands via flags like build=. When any of these invoke calls to
custom build systems or upstream programs/scripts, it's not safe to
assume that the output will be utf8.

Unfortunately, this currently leads to crashes and failed builds:

	ERROR: Could not build app org.kiwix.kiwixmobile due to unknown error: Traceback (most recent call last):
	  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 1155, in main
	    options.onserver, options.refresh):
	  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 951, in trybuild
	    build_local(app, build, vcs, build_dir, output_dir, srclib_dir, extlib_dir, tmp_dir, force, onserver, refresh)
	  File "/home/vagrant/fdroidserver/fdroidserver/build.py", line 648, in build_local
	    p = FDroidPopen(['bash', '-x', '-c', cmd], cwd=root_dir)
	  File "/home/vagrant/fdroidserver/fdroidserver/common.py", line 1786, in FDroidPopen
	    result.output = result.output.decode('utf-8')
	UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb7 in position 5397290: invalid start byte

One way to fix this would be to use one of the python libraries that
guess an encoding. But a much safer option is to tell the decode method
to ignore non-utf8 bytes, as opposed to crashing on them.

diff --git a/fdroidserver/common.py b/fdroidserver/common.py
index 1d2275f19503eeb47bff3ca96ccdae75e71f3304..b8a506a55c5495ca2d7dacb11ce8d46d80adb0b7 100644 (file)
--- a/fdroidserver/common.py
+++ b/fdroidserver/common.py
@@ -1798,7 +1798,7 @@ def FDroidPopen(commands, cwd=None, output=True, stderr_to_stdout=True):
     :returns: A PopenResult.
     """
     result = FDroidPopenBytes(commands, cwd, output, stderr_to_stdout)
-    result.output = result.output.decode('utf-8')
+    result.output = result.output.decode('utf-8', 'ignore')
     return result