chiark / gitweb /
fdroidserver.git
6 years agowiki: log build start/stop time, command line, RAM, and processor count
Hans-Christoph Steiner [Mon, 22 Jan 2018 13:00:16 +0000 (14:00 +0100)]
wiki: log build start/stop time, command line, RAM, and processor count

6 years agowiki: log server start/stop times and command line
Hans-Christoph Steiner [Wed, 17 Jan 2018 21:21:15 +0000 (22:21 +0100)]
wiki: log server start/stop times and command line

6 years agobuildserver: force no auto updates of package lists or upgrades
Hans-Christoph Steiner [Wed, 17 Jan 2018 20:04:08 +0000 (21:04 +0100)]
buildserver: force no auto updates of package lists or upgrades

6 years agojenkins-test: include repo_pubkey in config.py for BUILD test
Hans-Christoph Steiner [Wed, 17 Jan 2018 19:01:51 +0000 (20:01 +0100)]
jenkins-test: include repo_pubkey in config.py for BUILD test

The BUILD machine does not have a keyring on it, only the public key for
the index signing key.  This is a very rudementary test for that.

6 years agowiki: move checkupdates wiki log to separate function
Hans-Christoph Steiner [Wed, 17 Jan 2018 20:11:32 +0000 (21:11 +0100)]
wiki: move checkupdates wiki log to separate function

6 years agolog installed android sdk versions for update and checkupdates
Hans-Christoph Steiner [Wed, 17 Jan 2018 16:17:26 +0000 (17:17 +0100)]
log installed android sdk versions for update and checkupdates

6 years agomove get_android_tools_versions functions to common
Hans-Christoph Steiner [Wed, 17 Jan 2018 16:13:12 +0000 (17:13 +0100)]
move get_android_tools_versions functions to common

6 years agowiki: log update start/stop time and command line
Hans-Christoph Steiner [Wed, 17 Jan 2018 14:21:59 +0000 (15:21 +0100)]
wiki: log update start/stop time and command line

6 years agowiki: fix bug updating Repository Maintenance
Hans-Christoph Steiner [Wed, 17 Jan 2018 14:22:48 +0000 (15:22 +0100)]
wiki: fix bug updating Repository Maintenance

site.pages doesn't seem to exist anywhere, site.Pages is used throughout.

6 years agowiki: log appids as checkupdates goes through them
Hans-Christoph Steiner [Wed, 17 Jan 2018 14:21:16 +0000 (15:21 +0100)]
wiki: log appids as checkupdates goes through them

6 years agowiki: log checkupdates start/stop time and command line for each run
Hans-Christoph Steiner [Wed, 17 Jan 2018 14:18:05 +0000 (15:18 +0100)]
wiki: log checkupdates start/stop time and command line for each run

6 years agocommon.get_wiki_timestamp() for posting timestamps to wiki log pages
Hans-Christoph Steiner [Wed, 17 Jan 2018 13:39:54 +0000 (14:39 +0100)]
common.get_wiki_timestamp() for posting timestamps to wiki log pages

6 years agoMerge branch 'iconfix' into 'master'
Hans-Christoph Steiner [Thu, 18 Jan 2018 10:53:34 +0000 (10:53 +0000)]
Merge branch 'iconfix' into 'master'

fix "cannot identify image file" with XML icons

See merge request fdroid/fdroidserver!435

6 years agoMerge branch 'gitlab-mirrors-reorder' into 'master'
Hans-Christoph Steiner [Thu, 18 Jan 2018 09:01:17 +0000 (09:01 +0000)]
Merge branch 'gitlab-mirrors-reorder' into 'master'

Reorder the gitlab mirrors so GitLab Pages comes before "raw".

See merge request fdroid/fdroidserver!438

6 years agoReorder the gitlab mirrors so GitLab Pages comes before "raw".
Peter Serwylo [Wed, 17 Jan 2018 21:02:07 +0000 (08:02 +1100)]
Reorder the gitlab mirrors so GitLab Pages comes before "raw".

GitLab storage provides two mirrors by default:
 * https://gitlab.com/user/repo/raw/master/fdroid/repo
 * https://user.gitlab.io/repo/fdroid/repo

While the F-Droid client will happily fetch the index*.jar files and
parse them from either of these two mirrors, only the GitLab Pages
mirror will serve files with the correct mime type. Many repos
tend to put index.html files (and associated .css/.js/image files) in
the root of a repository to provide information about that repo.

One example is RepoMaker. The way in which RepoMaker decides the public
URL of a repo, is to take the first mirror in the list. This means that
the URL which RepoMaker directs people to for GitLab storage returns a
.html document in text/plain, which means that it is not rendered.

We could change RepoMaker so that it takes the last mirror, and then it
woruld work. However there is something nice about the first mirror in a
list being the most authoritative (even though the mirror order doesn't
- and perhaps shouldn't have any specific meaning).

6 years agosimplifying fix for "cannot identify image file" with XML icons
Izzy [Wed, 17 Jan 2018 15:48:08 +0000 (16:48 +0100)]
simplifying fix for "cannot identify image file" with XML icons

6 years agofix "cannot identify image file" with XML icons
Izzy [Fri, 12 Jan 2018 21:12:27 +0000 (22:12 +0100)]
fix "cannot identify image file" with XML icons

6 years agobuild: bump max_apps_per_run to 50
Hans-Christoph Steiner [Thu, 11 Jan 2018 22:25:31 +0000 (23:25 +0100)]
build: bump max_apps_per_run to 50

With this at 10, it seems that there are often runs that produce no builds
at all.  That's bad.

6 years agojenkins-build-all: don't fail if max build limit caused no builds
Hans-Christoph Steiner [Thu, 11 Jan 2018 22:25:24 +0000 (23:25 +0100)]
jenkins-build-all: don't fail if max build limit caused no builds

6 years agojenkins-test: ensure gpg is starting from a clean and proper place
Hans-Christoph Steiner [Thu, 11 Jan 2018 19:55:36 +0000 (20:55 +0100)]
jenkins-test: ensure gpg is starting from a clean and proper place

There have been frequent failures on import, some bugs suggest that it
might be because these dirs are missing.  They would get wiped by a
`git clean -fdx`.

6 years agojenkins-build-all: use local mediawiki if available
Hans-Christoph Steiner [Thu, 11 Jan 2018 15:47:49 +0000 (16:47 +0100)]
jenkins-build-all: use local mediawiki if available

6 years agobuild: buildserverid must always be str not bytes
Hans-Christoph Steiner [Thu, 11 Jan 2018 11:56:28 +0000 (12:56 +0100)]
build: buildserverid must always be str not bytes

6 years agobuild: fix str vs. bytes error in buildserverid
Hans-Christoph Steiner [Thu, 11 Jan 2018 11:28:49 +0000 (12:28 +0100)]
build: fix str vs. bytes error in buildserverid

ERROR: Could not build app org.fdroid.fdroid due to unknown error: Traceback (most recent call last):
  File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 1202, in main
    options.onserver, options.refresh):
  File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 972, in trybuild
    build_server(app, build, vcs, build_dir, output_dir, log_dir, force)
  File "/var/lib/jenkins/userContent/reproducible/reproducible_setup_fdroid_build_environment/fdroidserver/build.py", line 82, in build_server
    logging.debug(_('Fetched buildserverid from VM: ') + buildserverid)
TypeError: Can't convert 'bytes' object to str implicitly

6 years agoMerge branch 'cleaner-clean' into 'master'
Hans-Christoph Steiner [Wed, 10 Jan 2018 19:14:49 +0000 (19:14 +0000)]
Merge branch 'cleaner-clean' into 'master'

build: clean up only known subdirectories in build/*

Closes #438

See merge request fdroid/fdroidserver!432

6 years agobuild: clean up only known subdirectories in build/*
relan [Wed, 10 Jan 2018 18:06:32 +0000 (21:06 +0300)]
build: clean up only known subdirectories in build/*

We remove the whole "build" directory while cleaning source code tree
because Gradle can leave there files even after "gradle clean". But some
projects (Mozilla Fennec) actually have useful stuff checked into VCS
under the "build" directory.

Remove only those subdirectories that we known for sure are leftovers
from Gradle.

Fixes fdroid/fdroidserver#438.

6 years agoMerge branch 'gradle-4.4.1' into 'master'
Marcus [Wed, 10 Jan 2018 18:40:36 +0000 (18:40 +0000)]
Merge branch 'gradle-4.4.1' into 'master'

makebuildserver: add Gradle 4.4.1

See merge request fdroid/fdroidserver!433

6 years agomakebuildserver: add Gradle 4.4.1
relan [Wed, 10 Jan 2018 18:31:59 +0000 (21:31 +0300)]
makebuildserver: add Gradle 4.4.1

6 years agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Fri, 5 Jan 2018 13:47:48 +0000 (13:47 +0000)]
Merge branch 'master' into 'master'

new script to audit the FDroid.apk on https://f-droid.org

See merge request fdroid/fdroidserver!431

6 years agonew script to audit the FDroid.apk on https://f-droid.org
Hans-Christoph Steiner [Fri, 5 Jan 2018 13:08:14 +0000 (14:08 +0100)]
new script to audit the FDroid.apk on https://f-droid.org

This makes sure its signed by the F-Droid key.

6 years agoREADME: fix name on CI badges
Hans-Christoph Steiner [Fri, 5 Jan 2018 12:39:18 +0000 (13:39 +0100)]
README: fix name on CI badges

6 years agouse 1.0.0 for release version to make pypi happy 1.0.0
Hans-Christoph Steiner [Wed, 3 Jan 2018 20:41:35 +0000 (21:41 +0100)]
use 1.0.0 for release version to make pypi happy

6 years agorelease: compile_catalog must be run before register
Hans-Christoph Steiner [Wed, 3 Jan 2018 20:26:39 +0000 (21:26 +0100)]
release: compile_catalog must be run before register

6 years agoBump to 1.0!!
Hans-Christoph Steiner [Wed, 3 Jan 2018 14:52:22 +0000 (15:52 +0100)]
Bump to 1.0!!

6 years agoMerge branch 'weblate' into 'master'
Hans-Christoph Steiner [Wed, 3 Jan 2018 15:40:05 +0000 (15:40 +0000)]
Merge branch 'weblate' into 'master'

Weblate

See merge request fdroid/fdroidserver!430

6 years agoWeblate
Hans-Christoph Steiner [Wed, 3 Jan 2018 15:40:03 +0000 (15:40 +0000)]
Weblate

6 years agoMerge branch 'build-log-data-points' into 'master'
Hans-Christoph Steiner [Wed, 3 Jan 2018 14:39:23 +0000 (14:39 +0000)]
Merge branch 'build-log-data-points' into 'master'

add data points to the build log to add debugging

See merge request fdroid/fdroidserver!429

6 years agoget minimum aapt version from fdroidserver/common.py for CI tests
Hans-Christoph Steiner [Wed, 3 Jan 2018 13:32:16 +0000 (14:32 +0100)]
get minimum aapt version from fdroidserver/common.py for CI tests

6 years agobuild: include buildserverid on wiki build logs, if it exists
Hans-Christoph Steiner [Wed, 3 Jan 2018 12:58:06 +0000 (13:58 +0100)]
build: include buildserverid on wiki build logs, if it exists

When `fdroid build` is run using the buildserver, it should fetch the
buildserverid on the first build.

Seems this was really a silly bug in 837fc99d74f7694a64f014f3d38a8b07d9e8b3bd

6 years agobuild: log the start time of the current build session
Hans-Christoph Steiner [Wed, 3 Jan 2018 11:16:20 +0000 (12:16 +0100)]
build: log the start time of the current build session

Since `fdroid build --all` can run a long time, knowing when that command
was started will be very useful information for figuring out what the build
server is doing.

6 years agojenkins.debian.net is Debian/stretch, its build-tools are too old
Hans-Christoph Steiner [Fri, 29 Dec 2017 15:06:25 +0000 (16:06 +0100)]
jenkins.debian.net is Debian/stretch, its build-tools are too old

6 years agogitlab-ci: and one last stoopid error in debian_testing target
Hans-Christoph Steiner [Fri, 29 Dec 2017 14:49:20 +0000 (15:49 +0100)]
gitlab-ci: and one last stoopid error in debian_testing target

6 years agogitlab-ci: fix syntax error
Hans-Christoph Steiner [Fri, 29 Dec 2017 14:34:57 +0000 (15:34 +0100)]
gitlab-ci: fix syntax error

6 years agogitlab-ci: temp disable some tests on Debian/testing
Hans-Christoph Steiner [Fri, 29 Dec 2017 14:28:51 +0000 (15:28 +0100)]
gitlab-ci: temp disable some tests on Debian/testing

build-tools needs an update before these tests will work again.

6 years agogitlab-ci: set Fedora build to use a supported build-tools version
Hans-Christoph Steiner [Fri, 29 Dec 2017 14:12:36 +0000 (15:12 +0100)]
gitlab-ci: set Fedora build to use a supported build-tools version

Using 25.0.2, not all of the permissions were being output:

--- /builds/fdroid/fdroidserver/tests/repo/index.xml 2017-12-28 22:33:53.624704459 +0000
+++ repo/index.xml 2017-12-28 22:41:52.207849097 +0000
@@ -35,7 +35,6 @@
  <added>2017-12-22</added>
  <sig>2d337e40aef77564bf62781ac424595c</sig>
  <permissions>ACCESS_NETWORK_STATE,ACCESS_WIFI_STATE,CHANGE_WIFI_MULTICAST_STATE,INTERNET,READ_EXTERNAL_STORAGE,WRITE_EXTERNAL_STORAGE</permissions>
- <uses-permission maxSdkVersion="18" name="android.permission.READ_EXTERNAL_STORAGE"/>
  <uses-permission maxSdkVersion="18" name="android.permission.WRITE_EXTERNAL_STORAGE"/>
  <uses-permission-sdk-23 maxSdkVersion="27" name="android.permission.REQUEST_IGNORE_BATTERY_OPTIMIZATIONS"/>
  </package>

6 years agoMerge branch 'ever-more-fixes' into 'master'
Hans-Christoph Steiner [Thu, 28 Dec 2017 22:33:25 +0000 (22:33 +0000)]
Merge branch 'ever-more-fixes' into 'master'

Ever more fixes

Closes #395 and #306

See merge request fdroid/fdroidserver!423

6 years agoMerge branch 'fix_no-refresh' into 'master'
Hans-Christoph Steiner [Thu, 28 Dec 2017 22:08:29 +0000 (22:08 +0000)]
Merge branch 'fix_no-refresh' into 'master'

build: fix --no-refresh

See merge request fdroid/fdroidserver!428

6 years agotests: correct package names in stats/known_apks.txt
Hans-Christoph Steiner [Fri, 22 Dec 2017 22:19:06 +0000 (23:19 +0100)]
tests: correct package names in stats/known_apks.txt

This doesn't seem to affect the tests either way, but it is good to have
things correct there.

6 years agocommon: fix bug in new SHA-256 signatures for >= android-18
Hans-Christoph Steiner [Fri, 22 Dec 2017 16:49:36 +0000 (17:49 +0100)]
common: fix bug in new SHA-256 signatures for >= android-18

Luckily, this is only used in `fdroid nightly` so far.

6 years agoaapt 26.0.0 is required to properly parse permissions and label
Hans-Christoph Steiner [Fri, 22 Dec 2017 16:28:25 +0000 (17:28 +0100)]
aapt 26.0.0 is required to properly parse permissions and label

#236

closes #395
aapt 26.0.0 outputs the permissions correctly

closes #306
aapt 26.0.0 now outputs:  application-label:'K-9 Mail'

6 years agonightly: support arbitrary keystore files for setup
Hans-Christoph Steiner [Thu, 21 Dec 2017 11:16:06 +0000 (12:16 +0100)]
nightly: support arbitrary keystore files for setup

GitHub only allows an SSH key to be used as a Deploy Key for a single repo.
That means, each nightly build repo on GitHub/Travis must have its own
debug keystore.

6 years agojenkins: try to prevent the build node from running out of disk space
Hans-Christoph Steiner [Thu, 28 Dec 2017 21:51:55 +0000 (22:51 +0100)]
jenkins: try to prevent the build node from running out of disk space

6 years agoMerge branch 'master' into 'master'
Marcus [Wed, 27 Dec 2017 19:31:50 +0000 (19:31 +0000)]
Merge branch 'master' into 'master'

gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line

See merge request fdroid/fdroidserver!426

6 years agoAdd Nextcloud and DavDroid test case
mimi89999 [Sun, 24 Dec 2017 10:55:56 +0000 (11:55 +0100)]
Add Nextcloud and DavDroid test case

6 years agobuild: fix --no-refresh
Marcus Hoffmann [Mon, 25 Dec 2017 23:20:17 +0000 (00:20 +0100)]
build: fix --no-refresh

This was broken by the in-source-tree .fdroid.yml file support.
Also support this for building on the buildserver.

6 years agoMerge branch 'platform-27' into 'master'
Marcus [Sun, 24 Dec 2017 18:39:01 +0000 (18:39 +0000)]
Merge branch 'platform-27' into 'master'

makebuildserver: remove platform-27

See merge request fdroid/fdroidserver!427

6 years agomakebuildserver: remove platform-27
Marcus Hoffmann [Sun, 24 Dec 2017 17:51:28 +0000 (18:51 +0100)]
makebuildserver: remove platform-27

The hash changes all the time and gradle will install this on the fly
anyway when missing.

6 years agogradle file: use flavour specific versionCode/versionName, fall back to parsing line...
mimi89999 [Sat, 23 Dec 2017 13:36:38 +0000 (14:36 +0100)]
gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line

6 years agoMerge branch 'master' into 'master'
Michel Le Bihan [Sat, 23 Dec 2017 12:01:12 +0000 (12:01 +0000)]
Merge branch 'master' into 'master'

Revert: gradle file: use flavour specific versionCode/versionName, fall back to…

See merge request fdroid/fdroidserver!425

6 years agoRevert: gradle file: use flavour specific versionCode/versionName, fall back to parsi...
mimi89999 [Sat, 23 Dec 2017 11:43:16 +0000 (12:43 +0100)]
Revert: gradle file: use flavour specific versionCode/versionName, fall back to parsing line by line

6 years agoMerge branch 'limit-build-all' into 'master'
Marcus [Fri, 22 Dec 2017 13:09:16 +0000 (13:09 +0000)]
Merge branch 'limit-build-all' into 'master'

build: limit --all to 10 apps at a time

See merge request fdroid/fdroidserver!420

6 years agoMerge branch 'typo' into 'master'
Hans-Christoph Steiner [Fri, 22 Dec 2017 08:50:49 +0000 (08:50 +0000)]
Merge branch 'typo' into 'master'

Fix a typo in vmtools.py

See merge request fdroid/fdroidserver!421

6 years agoFix a typo in vmtools.py
Pierre Rudloff [Fri, 22 Dec 2017 00:14:29 +0000 (00:14 +0000)]
Fix a typo in vmtools.py

6 years agobuild: limit --all to 10 apps at a time
Hans-Christoph Steiner [Thu, 21 Dec 2017 22:15:08 +0000 (23:15 +0100)]
build: limit --all to 10 apps at a time

This needed now because the buildserver is hanging so often, that we are
often going a week or more without any builds published.  Perhaps this is
only temporary, or maybe we will want to flush this feature out more as a
standard thing.  But we really need it for now to at least get some builds
out on a daily basis.

Since the website deploy is also triggered by this cycle, making the build
finish more often means the website will be published more often.

6 years agoMerge branch 'additional_tests' into 'master'
Hans-Christoph Steiner [Thu, 21 Dec 2017 09:18:13 +0000 (09:18 +0000)]
Merge branch 'additional_tests' into 'master'

readme: add note about additional CI tests

See merge request fdroid/fdroidserver!419

6 years agoreadme: add note about additional CI tests
Marcus Hoffmann [Wed, 20 Dec 2017 23:51:26 +0000 (00:51 +0100)]
readme: add note about additional CI tests

[ci-skip]

6 years agoMerge branch 'nail-down-tests' into 'master'
Hans-Christoph Steiner [Wed, 20 Dec 2017 23:16:03 +0000 (23:16 +0000)]
Merge branch 'nail-down-tests' into 'master'

Nail down tests

Closes #432

See merge request fdroid/fdroidserver!418

6 years agotravis-ci: install babel with pip to stop easy_install
Hans-Christoph Steiner [Wed, 20 Dec 2017 17:04:37 +0000 (18:04 +0100)]
travis-ci: install babel with pip to stop easy_install

easy_install just fails, trying to install into ./.eggs
https://travis-ci.org/fdroidtravis/fdroidserver/jobs/319144754

6 years agotravis-ci: include new android-sdk-license on OSX
Hans-Christoph Steiner [Wed, 20 Dec 2017 16:04:19 +0000 (17:04 +0100)]
travis-ci: include new android-sdk-license on OSX

6 years agotests: only generate keystores when that is actually being tested
Hans-Christoph Steiner [Wed, 20 Dec 2017 16:03:48 +0000 (17:03 +0100)]
tests: only generate keystores when that is actually being tested

Generating a keystore is quite slow since it means a new RSA key is created.
That only needs to happen in the tests that check that it actually happened,
otherwise the test can just reuse the stored test keystore.

closes #432

6 years agoREADME: document test suite (closes #432)
Hans-Christoph Steiner [Wed, 20 Dec 2017 15:33:39 +0000 (16:33 +0100)]
README: document test suite (closes #432)

6 years agotravis-ci: show sdkmanager logs
Hans-Christoph Steiner [Wed, 20 Dec 2017 12:18:31 +0000 (13:18 +0100)]
travis-ci: show sdkmanager logs

6 years agoupdate: support working with old versions of PIL/Pillow
Hans-Christoph Steiner [Wed, 20 Dec 2017 10:38:54 +0000 (11:38 +0100)]
update: support working with old versions of PIL/Pillow

Image.close() was added in Pillow 2.4 or so.

6 years agobuild: use dpkg to purge sudo, for less spammy debug logs
Hans-Christoph Steiner [Mon, 18 Dec 2017 09:02:19 +0000 (10:02 +0100)]
build: use dpkg to purge sudo, for less spammy debug logs

apt is quite verbose:
DEBUG: buildserver > DEBUG: > sudo SUDO_FORCE_REMOVE=yes apt-get -y purge sudo
DEBUG: buildserver > Reading package lists...

DEBUG: buildserver > Building dependency tree...
DEBUG: buildserver > Reading state information...
DEBUG: buildserver > The following package was automatically installed and is no longer required:
DEBUG: buildserver >   libasprintf0c2
DEBUG: buildserver > Use 'apt-get autoremove' to remove it.
DEBUG: buildserver > The following packages will be REMOVED:
DEBUG: buildserver >   sudo*

DEBUG: buildserver > 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
DEBUG: buildserver > After this operation, 2,391 kB disk space will be freed.
DEBUG: buildserver > (Reading database ...
(Reading database ... 5%
(Reading database ... 10%
(Reading database ... 15%
(Reading database ... 20%
(Reading database ... 25%
(Reading database ... 30%
(Reading database ... 35%
(Reading database ... 40%
(Reading database ... 45%
(Reading database ... 50%
(Reading database ... 55%
(Reading database ... 60%
(Reading database ... 65%
(Reading database ... 70%
(Reading database ... 75%
(Reading database ... 80%
(Reading database ... 85%
(Reading database ... 90%
(Reading database ... 95%
(Reading database ... 100%
(Reading database ... 73055 files and directories currently installed.)
DEBUG: buildserver > Removing sudo (1.8.10p3-1+deb8u5) ...
DEBUG: buildserver > Purging configuration files for sudo (1.8.10p3-1+deb8u5) ...
DEBUG: buildserver > Processing triggers for man-db (2.7.0.2-5) ...

6 years agoalways hide PIL.PngImagePlugin's "STREAM" debug messages
Hans-Christoph Steiner [Thu, 14 Dec 2017 19:30:00 +0000 (20:30 +0100)]
always hide PIL.PngImagePlugin's "STREAM" debug messages

Otherwise, enabling verbose messages gives tons of these messages:
DEBUG: STREAM b'IHDR' 16 13
DEBUG: STREAM b'IDAT' 41 32768

6 years agoRevert "revert reverted checksum of platform-27_r01.zip (google tampered there again)"
Hans-Christoph Steiner [Wed, 20 Dec 2017 09:44:10 +0000 (10:44 +0100)]
Revert "revert reverted checksum of platform-27_r01.zip (google tampered there again)"

This reverts commit 7f13675b8c321d350d49ccc4bbbfab89d2df4f8b.

6 years agojenkins-test: import secret key into test GNUPGHOME
Hans-Christoph Steiner [Wed, 20 Dec 2017 12:35:20 +0000 (13:35 +0100)]
jenkins-test: import secret key into test GNUPGHOME

Can't run `fdroid gpgsign` without a secret key!

6 years agoscanner: fix tests so they work on all tested platforms
Hans-Christoph Steiner [Tue, 19 Dec 2017 21:51:03 +0000 (22:51 +0100)]
scanner: fix tests so they work on all tested platforms

The standard test configuration is needed to make the tests reliably. Also,
these tests used some odd yield logic.  Who knows what exactly failed, but
these tests should be reliable.

* https://gitlab.com/fdroid/fdroidserver/-/jobs/44984595
* https://gitlab.com/fdroid/fdroidserver/-/jobs/44984596
* https://travis-ci.org/f-droid/fdroidserver/builds/318071369

6 years agoMerge branch 'whitelist-firebase' into 'master'
Hans-Christoph Steiner [Mon, 18 Dec 2017 12:30:30 +0000 (12:30 +0000)]
Merge branch 'whitelist-firebase' into 'master'

Whitelist some open-source firebase libs

See merge request fdroid/fdroidserver!411

6 years agoMerge branch '430-UnboundLocalError-local-variable-im-referenced-before-assignment...
Hans-Christoph Steiner [Mon, 18 Dec 2017 08:56:27 +0000 (08:56 +0000)]
Merge branch '430-UnboundLocalError-local-variable-im-referenced-before-assignment' into 'master'

fix handling unreadable images in update.extract_apk_icons

Closes #430

See merge request fdroid/fdroidserver!416

6 years agofix handling unreadable images in update.extract_apk_icons
Michael Pöhn [Sat, 16 Dec 2017 21:06:20 +0000 (22:06 +0100)]
fix handling unreadable images in update.extract_apk_icons

6 years agoMerge branch '431-Invalid-checksum-platform-27_r01.zip' into 'master'
Marcus [Sat, 16 Dec 2017 15:46:10 +0000 (15:46 +0000)]
Merge branch '431-Invalid-checksum-platform-27_r01.zip' into 'master'

revert reverted checksum of platform-27_r01.zip (google tampered there again)

Closes #431

See merge request fdroid/fdroidserver!415

6 years agorevert reverted checksum of platform-27_r01.zip (google tampered there again)
Michael Pöhn [Sat, 16 Dec 2017 15:23:06 +0000 (16:23 +0100)]
revert reverted checksum of platform-27_r01.zip (google tampered there again)

6 years agoMerge branch 'build-tools_r27.0.2' into 'master'
Marcus [Fri, 15 Dec 2017 12:14:00 +0000 (12:14 +0000)]
Merge branch 'build-tools_r27.0.2' into 'master'

makebuildserver: add build-tools_r27.0.2

See merge request fdroid/fdroidserver!413

6 years agoMerge branch 'gradle-4.4' into 'master'
Marcus [Fri, 15 Dec 2017 11:55:13 +0000 (11:55 +0000)]
Merge branch 'gradle-4.4' into 'master'

makebuildserver: add Gradle 4.4

See merge request fdroid/fdroidserver!412

6 years agomakebuildserver: add build-tools_r27.0.2
relan [Fri, 15 Dec 2017 11:34:40 +0000 (14:34 +0300)]
makebuildserver: add build-tools_r27.0.2

6 years agomakebuildserver: add Gradle 4.4
relan [Fri, 15 Dec 2017 11:32:34 +0000 (14:32 +0300)]
makebuildserver: add Gradle 4.4

6 years agoMerge branch 'security-fixes' into 'master'
Hans-Christoph Steiner [Fri, 15 Dec 2017 11:22:56 +0000 (11:22 +0000)]
Merge branch 'security-fixes' into 'master'

security fixes for Janus and image metadata exploits

See merge request fdroid/fdroidserver!409

6 years agoConvert to string
Jan Berkel [Thu, 14 Dec 2017 23:47:57 +0000 (00:47 +0100)]
Convert to string

6 years agoAdd a simple test for scanner
Jan Berkel [Thu, 14 Dec 2017 23:29:38 +0000 (00:29 +0100)]
Add a simple test for scanner

6 years agowhitelist some open-source firebase libs
Jan Berkel [Thu, 14 Dec 2017 20:58:06 +0000 (21:58 +0100)]
whitelist some open-source firebase libs

6 years agobuild: force purging of sudo, ignore error message
Hans-Christoph Steiner [Thu, 14 Dec 2017 13:42:09 +0000 (14:42 +0100)]
build: force purging of sudo, ignore error message

Fixes bb758d3f, spotted by @bubu:
DEBUG: buildserver > DEBUG: > sudo apt-get -y purge sudo
DEBUG: buildserver > Reading package lists...
DEBUG: buildserver > Building dependency tree...
DEBUG: buildserver > Reading state information...
DEBUG: buildserver > The following packages will be REMOVED:
DEBUG: buildserver >   sudo*
DEBUG: buildserver > 0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
DEBUG: buildserver > After this operation, 2,391 kB disk space will be freed.
(Reading database ... 68491 files and directories currently installed.)
DEBUG: buildserver > Removing sudo (1.8.10p3-1+deb8u4) ...
DEBUG: buildserver > You have asked that the sudo package be removed,
DEBUG: buildserver > but no root password has been set.
DEBUG: buildserver > Without sudo, you may not be able to gain administrative privileges.
DEBUG: buildserver >
DEBUG: buildserver > If you would prefer to access the root account with su(1)
DEBUG: buildserver > or by logging in directly,
DEBUG: buildserver > you must set a root password with "sudo passwd".
DEBUG: buildserver >
DEBUG: buildserver > If you have arranged other means to access the root account,
DEBUG: buildserver > and you are sure this is what you want,
DEBUG: buildserver > you may bypass this check by setting an environment variable
DEBUG: buildserver > (export SUDO_FORCE_REMOVE=yes).
DEBUG: buildserver >
DEBUG: buildserver > Refusing to remove sudo.
DEBUG: buildserver > dpkg: error processing package sudo (--purge):
DEBUG: buildserver >  subprocess installed pre-removal script returned error exit status 1
DEBUG: buildserver > Errors were encountered while processing:
DEBUG: buildserver >  sudo
DEBUG: buildserver > E: Sub-process /usr/bin/dpkg returned an error code (1)

6 years agoupdate: do not crash if AndroidManifest.xml in APK has invalid date
Hans-Christoph Steiner [Thu, 14 Dec 2017 09:58:02 +0000 (10:58 +0100)]
update: do not crash if AndroidManifest.xml in APK has invalid date

This crash actually blocked a Janus exploit APK from being added to the
repo, but crashing isn't really the appropriate way to do that.

6 years agoupdate: close unclosed Image instance
Hans-Christoph Steiner [Wed, 13 Dec 2017 11:28:11 +0000 (12:28 +0100)]
update: close unclosed Image instance

6 years agoupdate: strip all metadata from PNGs
Hans-Christoph Steiner [Wed, 13 Dec 2017 10:51:34 +0000 (11:51 +0100)]
update: strip all metadata from PNGs

This strips metadata and optimizes the compression of all PNGs copied
from the app's source repo as well as all the icons extracted from the
APKs.  There have been exploits delivered via image metadata, and
F-Droid isn't using it all, so its best to just remove it.

This unfortunately uncompresses and recompresses the files.  Luckily,
that's a lossless procedure with PNGs, and we might end up with
smaller files.  The only tool I could find that strips without
changing the image data is exiftool, but that is written in Perl.

6 years agoupdate: strip EXIF data from all JPEGs
Hans-Christoph Steiner [Wed, 13 Dec 2017 10:57:36 +0000 (11:57 +0100)]
update: strip EXIF data from all JPEGs

EXIF data can be abused to exploit systems a lot easier than the JPEG image
data can.  The F-Droid ecosystem does not use the EXIF data, so keep things
safe and strip it all away.  There is a chance that some images might rely
on the rotation to be set by EXIF, but I think having a safe system is more
important.

If needed, only the rotation data could be saved.  But that then makes it
hard to tell which images have been stripped.  This way, if there is no
EXIF, it has been stripped.  And if there is EXIF data, then it is suspect.

https://securityaffairs.co/wordpress/51043/mobile-2/android-cve-2016-3862-flaw.html
https://threatpost.com/google-shuts-down-potentially-massive-android-bug/120393/
https://blog.sucuri.net/2013/07/malware-hidden-inside-jpg-exif-headers.html

The big downside of this is that it decompresses and recompresses the
image data.  That should be replaced by a technique from jhead,
exiftool, ObscuraCam, etc. that only strips the metadata.

6 years agoupdate: reject APKs with invalid file sig, probably Janus exploits
Hans-Christoph Steiner [Mon, 11 Dec 2017 17:36:21 +0000 (18:36 +0100)]
update: reject APKs with invalid file sig, probably Janus exploits

This just checks the first four bytes of the APK file, aka the "file
signature", to make sure it is the ZIP signature and not the DEX signature.
This was checked against the test APK, and I ran it against some known
malware and all of f-droid.org to make sure it works.

All valid ZIP files (therefore APK files) should start with the ZIP
Local File Header of four bytes.

https://www.guardsquare.com/en/blog/new-android-vulnerability-allows-attackers-modify-apps-without-affecting-their-signatures

6 years agoupdate: print warnings for all KnownVulns found
Hans-Christoph Steiner [Mon, 11 Dec 2017 16:56:04 +0000 (17:56 +0100)]
update: print warnings for all KnownVulns found

Some baby steps towards making the KnownVuln stuff more visible.

6 years agoupdate: switch tests to using standardized setUp() method
Hans-Christoph Steiner [Thu, 14 Dec 2017 10:06:22 +0000 (11:06 +0100)]
update: switch tests to using standardized setUp() method

6 years agoMerge branch 'fixFlavor' into 'master'
Hans-Christoph Steiner [Thu, 14 Dec 2017 15:56:01 +0000 (16:56 +0100)]
Merge branch 'fixFlavor' into 'master'

Regex only for flavor blocks: flavor { ... }

See merge request fdroid/fdroidserver!407

6 years agoadd Conversations as gradle flavor test case
Hans-Christoph Steiner [Thu, 14 Dec 2017 15:52:02 +0000 (16:52 +0100)]
add Conversations as gradle flavor test case