chiark / gitweb /
fdroidserver.git
11 months agoMerge branch 'master' into 'master' master t
Hans-Christoph Steiner [Fri, 4 May 2018 12:58:29 +0000 (12:58 +0000)]
Merge branch 'master' into 'master'

Fix applicationIdSuffix / versionNameSuffix not understood #455

Closes #455

See merge request fdroid/fdroidserver!484

11 months agoFix applicationIdSuffix / versionNameSuffix #455
J-Jamet [Mon, 26 Mar 2018 17:45:06 +0000 (19:45 +0200)]
Fix applicationIdSuffix / versionNameSuffix #455

11 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Thu, 3 May 2018 12:11:48 +0000 (12:11 +0000)]
Merge branch 'master' into 'master'

Three random fixes

See merge request fdroid/fdroidserver!496

11 months agoimplement common.get_apk_id() using androguard
Hans-Christoph Steiner [Thu, 3 May 2018 11:33:37 +0000 (13:33 +0200)]
implement common.get_apk_id() using androguard

11 months agofix intermittent test failure
Hans-Christoph Steiner [Thu, 3 May 2018 11:30:03 +0000 (13:30 +0200)]
fix intermittent test failure

For some reason, the parser stopped working intermittently, even
though the format has been the same since aapt 23 or earlier.  Then
also, some of the test cases pointed to symlinks that were no longer
generated, and one test app now has a blank versionName.

Strange that this wasn't caught in the gitlab-ci runs.  !484

FAIL: test_get_api_id_aapt (__main__.CommonTest)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./common.TestCase", line 578, in testA_get_api_id_aapt
    self.assertEqual(versionName, vn)
AssertionError: '0.1' != "0.1' platformBuildVersionName='4.3.1-1425645"
- 0.1
+ 0.1' platformBuildVersionName='4.3.1-1425645

11 months agodelete .java.security after checking MD5 signatures
Hans-Christoph Steiner [Thu, 3 May 2018 11:46:36 +0000 (13:46 +0200)]
delete .java.security after checking MD5 signatures

This file is written freshly each time before use, so it does not need
to be ekpt around.  It was the only file making the fdroiddata.git
repo dirty on the f-droid.org infrastructure.

This also adds stricter file permissions to avoid an attacker changing
those settings during operation.

11 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Thu, 26 Apr 2018 14:36:07 +0000 (14:36 +0000)]
Merge branch 'master' into 'master'

fix bad parsing of maxSdkVersion as a @string resource

See merge request fdroid/fdroidserver!494

11 months agoreusable method for checking if a value is a resId or not
Hans-Christoph Steiner [Thu, 26 Apr 2018 11:20:43 +0000 (13:20 +0200)]
reusable method for checking if a value is a resId or not

I wrote this for the previous commit, but since aapt does not recognize
SDK Version values that are string resources, I made this do the same thing

11 months agohandle bad SDK Version values in APKs
Hans-Christoph Steiner [Thu, 26 Apr 2018 11:17:25 +0000 (13:17 +0200)]
handle bad SDK Version values in APKs

Even though it is invalid to have *SdkVersion in AndroidManifest.xml set as
anything but an integer, sometimes people manage to get something in there.
`fdroid update` needs to be able to handle all that.

* https://developer.android.com/guide/topics/manifest/uses-sdk-element#min
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/AndroidManifest.xml#L29
* https://gitlab.com/souch/SMSbypass/blob/v0.9/app/src/main/res/values/strings.xml#L27

admin#65

11 months agoignore index files generated as part of manual testing
Hans-Christoph Steiner [Thu, 26 Apr 2018 10:33:44 +0000 (12:33 +0200)]
ignore index files generated as part of manual testing

it is quite useful to run `../fdroid update --pretty` directly in tests/

11 months agoindex: ensure index.xml timestamp is in UTC, closes #481
Hans-Christoph Steiner [Wed, 25 Apr 2018 12:20:19 +0000 (14:20 +0200)]
index: ensure index.xml timestamp is in UTC, closes #481

use verbose but safe syntax so this works with all versions of sed, e.g. osx

12 months agoMerge branch 'gradle_4.7' into 'master'
Marcus [Sun, 22 Apr 2018 18:00:33 +0000 (18:00 +0000)]
Merge branch 'gradle_4.7' into 'master'

makebuildserver: add gradle 4.7

See merge request fdroid/fdroidserver!493

12 months agomakebuildserver: add gradle 4.7
Marcus Hoffmann [Fri, 20 Apr 2018 15:55:51 +0000 (17:55 +0200)]
makebuildserver: add gradle 4.7

12 months agofix scan_apk_androguard when parsing versionName without a dot (e.g. 5)
Hans-Christoph Steiner [Wed, 18 Apr 2018 10:31:48 +0000 (12:31 +0200)]
fix scan_apk_androguard when parsing versionName without a dot (e.g. 5)

fixes de35f1b05bb864c2d51b278f933cfd9a39a96b16:

Traceback (most recent call last):
  File "/home/hans/code/fdroid/server/fdroid", line 164, in <module>
    main()
  File "/home/hans/code/fdroid/server/fdroid", line 138, in main
    mod.main()
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1932, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1459, in process_apks
    use_date_from_apk, ada, True)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1332, in process_apk
    apk = scan_apk(apkfile)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1051, in scan_apk
    scan_apk_androguard(apk, apk_file)
  File "/export/share/code/fdroid/server/fdroidserver/update.py", line 1220, in scan_apk_androguard
    res_id = arsc.get_id(apk['packageName'], res_id)[1]
TypeError: 'NoneType' object is not subscriptable

For example https://f-droid.org/archive/com.abitsinc.andr_5.apk:
$ aapt dump badging archive/com.abitsinc.andr_5.apk |head -1
package: name='com.abitsinc.andr' versionCode='5' versionName='5' platformBuildVersionName='2.3.3'

12 months agoMerge branch 'blank-versionName' into 'master'
Hans-Christoph Steiner [Tue, 17 Apr 2018 13:02:55 +0000 (13:02 +0000)]
Merge branch 'blank-versionName' into 'master'

handle APKs with a blank versionName

Closes #477, #478, fdroidclient#1416, fdroidclient#1417, and fdroidclient#1418

See merge request fdroid/fdroidserver!492

12 months agoupdate: handle APKs with a blank versionName
Hans-Christoph Steiner [Tue, 17 Apr 2018 10:15:51 +0000 (12:15 +0200)]
update: handle APKs with a blank versionName

Instead of just crashing, first try to use the versionName as written in the
build metadata, otherwise just let it be blank.  A blank versionName will
cause fdroidclient < 1.3 to crash.  Blank versionNames are not allowed in
the .txt metadata format, only .yml.

closes #477
closes #478
closes fdroidclient#1416
closes fdroidclient#1417
closes fdroidclient#1418
fdroiddata!3061

12 months agoupdate: use only 7 chars of SHA256 for non-APK version name
Hans-Christoph Steiner [Mon, 16 Apr 2018 21:36:12 +0000 (23:36 +0200)]
update: use only 7 chars of SHA256 for non-APK version name

#460

12 months agoupdate index metadata tests for 673a3517d337e9e9de736459c13b0126883b8fba
Hans-Christoph Steiner [Mon, 16 Apr 2018 09:47:48 +0000 (11:47 +0200)]
update index metadata tests for 673a3517d337e9e9de736459c13b0126883b8fba

12 months agoadd test of Android TV "activity banner" graphic
Hans-Christoph Steiner [Mon, 16 Apr 2018 09:35:30 +0000 (11:35 +0200)]
add test of Android TV "activity banner" graphic

!491

https://developer.android.com/guide/topics/manifest/activity-element.html#banner
https://developer.android.com/training/tv/start/start.html#banner

12 months agoMerge branch 'patch-1' into 'master'
Hans-Christoph Steiner [Mon, 16 Apr 2018 09:37:42 +0000 (09:37 +0000)]
Merge branch 'patch-1' into 'master'

fdroidserver/update.py: APK_LABEL_ICON_PAT icon pattern should not be greedy

See merge request fdroid/fdroidserver!491

12 months agofdroidserver/update.py: APK_LABEL_ICON_PAT icon pattern should not be greedy
Phoenix09 [Sat, 14 Apr 2018 22:30:23 +0000 (23:30 +0100)]
fdroidserver/update.py: APK_LABEL_ICON_PAT icon pattern should not be greedy

12 months agoMerge branch 'wiki-page-name' into 'master'
Marcus [Wed, 11 Apr 2018 13:35:42 +0000 (13:35 +0000)]
Merge branch 'wiki-page-name' into 'master'

Filter another invalid mediawiki page title character

See merge request fdroid/fdroidserver!490

12 months agoFilter another invalid mediawiki page title character
Ciaran Gultnieks [Wed, 11 Apr 2018 13:32:55 +0000 (14:32 +0100)]
Filter another invalid mediawiki page title character

12 months agojenkins-test: remove redundant `fdroid update` test
Hans-Christoph Steiner [Mon, 9 Apr 2018 08:27:29 +0000 (10:27 +0200)]
jenkins-test: remove redundant `fdroid update` test

This test ran before config.py was setup, so it didn't respect the test
setup.  Things like "make_current_version_link = False" are needed so that
the checkupdates tests work like on the official infrastructure.  Also,
./tests/run-tests has many `fdroid update` tests, so having it here just
wastes time.  The tests that are directly in this script are meant to test
the separate build/sign setup.

12 months agoMerge branch 'mwclient-dep' into 'master'
Marcus [Sun, 8 Apr 2018 11:53:26 +0000 (11:53 +0000)]
Merge branch 'mwclient-dep' into 'master'

Replace deprecated mwclient method

See merge request fdroid/fdroidserver!489

12 months agoReplace deprecated mwclient method
Ciaran Gultnieks [Sun, 8 Apr 2018 08:37:48 +0000 (09:37 +0100)]
Replace deprecated mwclient method

It's functionally the same, just a different name.

12 months agoMerge branch 'patch-1' into 'master'
Hans-Christoph Steiner [Fri, 6 Apr 2018 20:24:54 +0000 (20:24 +0000)]
Merge branch 'patch-1' into 'master'

Fix a typo in README.md

See merge request fdroid/fdroidserver!488

12 months agoFix a typo in README.md
scde [Fri, 6 Apr 2018 17:16:56 +0000 (17:16 +0000)]
Fix a typo in README.md

12 months agojenkins-test: turn off link generation for checkupdates test
Hans-Christoph Steiner [Thu, 5 Apr 2018 19:01:59 +0000 (21:01 +0200)]
jenkins-test: turn off link generation for checkupdates test

These links will make checkupdates fail since they will make the git repo
dirty.  There is also a mystery "cache/" subdir, but I don't know what is
making that.

12 months agoMerge branch 'gradle_plugin_3.1' into 'master'
relan [Thu, 5 Apr 2018 12:34:47 +0000 (12:34 +0000)]
Merge branch 'gradle_plugin_3.1' into 'master'

add gradle plugin 3.1 which requires gradle 4.4

See merge request fdroid/fdroidserver!487

12 months agoadd gradle plugin 3.1 which requires gradle 4.4
Marcus Hoffmann [Thu, 5 Apr 2018 12:11:28 +0000 (14:11 +0200)]
add gradle plugin 3.1 which requires gradle 4.4

12 months agoMerge branch 'java-gentoo' into 'master'
Hans-Christoph Steiner [Wed, 4 Apr 2018 17:04:40 +0000 (17:04 +0000)]
Merge branch 'java-gentoo' into 'master'

added java selection criteria for gentoo linux

See merge request fdroid/fdroidserver!486

12 months agoMerge branch 'fix-icon-crash' into 'master'
Hans-Christoph Steiner [Wed, 4 Apr 2018 17:04:24 +0000 (17:04 +0000)]
Merge branch 'fix-icon-crash' into 'master'

Fix crash when icon_name is None

See merge request fdroid/fdroidserver!485

12 months agoadded java selection criteria for gentoo linux
lb@lb520 [Tue, 26 Sep 2017 08:21:38 +0000 (10:21 +0200)]
added java selection criteria for gentoo linux

12 months agoAdd test for when apk icon src could not be found
Jonas Kalderstam [Tue, 27 Mar 2018 16:39:59 +0000 (18:39 +0200)]
Add test for when apk icon src could not be found

An APK (Netflix) was found to have the following icon filename:

    \u2003\u2009\n.xml

This breaks the aapt dump parsing because it iterates line by line and
this filename goes across two lines. Consequently, icon_src will be
None (default value) when it is passed to the icons parser.

12 months agoFix crash when icon_name is None
Jonas Kalderstam [Tue, 27 Mar 2018 12:17:08 +0000 (14:17 +0200)]
Fix crash when icon_name is None

Fixes the following crash:

```
$ fdroid update --create-metadata --rename-apks
WARNING: Using Java's jarsigner, not recommended for verifying APKs! Use apksigner
CRITICAL: Unknown exception found!
Traceback (most recent call last):
  File "/home/jonas/miniconda3/bin/fdroid", line 164, in <module>
    main()
  File "/home/jonas/miniconda3/bin/fdroid", line 138, in main
    mod.main()
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1927, in main
    apks, cachechanged = process_apks(apkcache, repodirs[0], knownapks, options.use_date_from_apk)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1454, in process_apks
    use_date_from_apk, ada, True)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1327, in process_apk
    apk = scan_apk(apkfile)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1056, in scan_apk
    scan_apk_aapt(apk, apk_file)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1184, in scan_apk_aapt
    apk['icons_src'] = _get_apk_icons_src(apkfile, icon_name)
  File "/home/jonas/miniconda/lib/python3.6/site-packages/fdroidserver/update.py", line 1089, in _get_apk_icons_src
    density_re = re.compile('^res/(.*)/' + icon_name + '\.(png|xml)$')
TypeError: must be str, not NoneType
```

13 months agobump to 1.0.3 1.0.3
Hans-Christoph Steiner [Thu, 22 Mar 2018 22:00:10 +0000 (23:00 +0100)]
bump to 1.0.3

13 months agoupdate: strip all whitespace from buildserverid
Hans-Christoph Steiner [Thu, 22 Mar 2018 16:16:09 +0000 (17:16 +0100)]
update: strip all whitespace from buildserverid

13 months agoMerge branch 'weblate' into 'master'
Hans-Christoph Steiner [Thu, 22 Mar 2018 22:21:06 +0000 (22:21 +0000)]
Merge branch 'weblate' into 'master'

Weblate

See merge request fdroid/fdroidserver!483

13 months agoWeblate
Hans-Christoph Steiner [Thu, 22 Mar 2018 22:21:06 +0000 (22:21 +0000)]
Weblate

13 months agoMerge branch 'gradle-4.6' into 'master'
Marcus [Fri, 16 Mar 2018 08:33:52 +0000 (08:33 +0000)]
Merge branch 'gradle-4.6' into 'master'

makebuildserver: add Gradle 4.6

See merge request fdroid/fdroidserver!482

13 months agomakebuildserver: add Gradle 4.6
relan [Fri, 16 Mar 2018 04:39:44 +0000 (07:39 +0300)]
makebuildserver: add Gradle 4.6

13 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Thu, 15 Mar 2018 15:23:55 +0000 (15:23 +0000)]
Merge branch 'master' into 'master'

update: handle AndroidManifest XML namespace named other than 'android'

See merge request fdroid/fdroidserver!481

13 months agoupdate: handle AndroidManifest XML namespace named other than 'android'
Hans-Christoph Steiner [Thu, 15 Mar 2018 14:56:30 +0000 (15:56 +0100)]
update: handle AndroidManifest XML namespace named other than 'android'

https://stackoverflow.com/questions/5856719/why-do-we-specify-namespace-in-android-xml-file

https://f-droid.org/repo/org.moire.ultrasonic_60.apk has:
xmlns:a="http://schemas.android.com/apk/res/android"

13 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Wed, 14 Mar 2018 18:09:19 +0000 (18:09 +0000)]
Merge branch 'master' into 'master'

update: handle renameManifestPackage when extracting icons with androguard

See merge request fdroid/fdroidserver!480

13 months agoupdate: handle renameManifestPackage when extracting icons with androguard
Hans-Christoph Steiner [Wed, 14 Mar 2018 17:42:06 +0000 (18:42 +0100)]
update: handle renameManifestPackage when extracting icons with androguard

aapt --rename-manifest-package changes the applicationId for an app without
changing the packageName listed in AndroidManifest.xml under
<application android:package="">

13 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Tue, 13 Mar 2018 16:12:52 +0000 (16:12 +0000)]
Merge branch 'master' into 'master'

update: support all official DPIs when extracting icons

See merge request fdroid/fdroidserver!479

13 months agoupdate: support all official DPIs when extracting icons
Hans-Christoph Steiner [Tue, 13 Mar 2018 15:47:07 +0000 (16:47 +0100)]
update: support all official DPIs when extracting icons

https://developer.android.com/training/multiscreen/screendensities.html

13 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Tue, 13 Mar 2018 11:36:43 +0000 (11:36 +0000)]
Merge branch 'master' into 'master'

two update parsing fixes  when using androguard

See merge request fdroid/fdroidserver!477

13 months agoupdate: do not crash on android-26+ XML icon
Hans-Christoph Steiner [Tue, 13 Mar 2018 11:13:23 +0000 (12:13 +0100)]
update: do not crash on android-26+ XML icon

For example:
res/drawable-v26/icon.xml

Here's the full range of possibilities, we have a lot of work to do:

$ for f in */*.apk; do unzip -l $f |grep -F icon.| grep -Eo 'res/drawable[^/]*'; done | sort -u
res/drawable
res/drawable-anydpi-v21
res/drawable-anydpi-v26
res/drawable-hdpi
res/drawable-hdpi-v11
res/drawable-hdpi-v12
res/drawable-hdpi-v4
res/drawable-hdpi-v5
res/drawable-hdpi-v6
res/drawable-hdpi-v9
res/drawable-large
res/drawable-large-hdpi-v11
res/drawable-large-hdpi-v4
res/drawable-large-ldpi-v4
res/drawable-large-mdpi
res/drawable-large-mdpi-v11
res/drawable-large-v4
res/drawable-large-xhdpi-v11
res/drawable-large-xhdpi-v4
res/drawable-large-xxhdpi-v11
res/drawable-large-xxhdpi-v4
res/drawable-ldpi
res/drawable-ldpi-v11
res/drawable-ldpi-v4
res/drawable-ldpi-v5
res/drawable-ldpi-v6
res/drawable-ldpi-v9
res/drawable-ldrtl-v17
res/drawable-mdpi
res/drawable-mdpi-v11
res/drawable-mdpi-v12
res/drawable-mdpi-v4
res/drawable-mdpi-v5
res/drawable-mdpi-v6
res/drawable-mdpi-v9
res/drawable-nodpi
res/drawable-nodpi-v4
res/drawable-tvdpi-v4
res/drawable-v11
res/drawable-v14
res/drawable-v21
res/drawable-xhdpi
res/drawable-xhdpi-v11
res/drawable-xhdpi-v12
res/drawable-xhdpi-v4
res/drawable-xhdpi-v9
res/drawable-xlarge-hdpi
res/drawable-xlarge-hdpi-v4
res/drawable-xlarge-mdpi
res/drawable-xlarge-mdpi-v4
res/drawable-xlarge-v4
res/drawable-xlarge-xhdpi-v4
res/drawable-xxhdpi
res/drawable-xxhdpi-v11
res/drawable-xxhdpi-v21
res/drawable-xxhdpi-v4
res/drawable-xxhdpi-v9
res/drawable-xxxhdpi
res/drawable-xxxhdpi-v11
res/drawable-xxxhdpi-v4

13 months agoupdate: do not crash on APKs without icons when using androguard
Hans-Christoph Steiner [Tue, 13 Mar 2018 10:19:19 +0000 (11:19 +0100)]
update: do not crash on APKs without icons when using androguard

13 months agoupdate: handle bad XML when using androguard
Hans-Christoph Steiner [Sun, 11 Mar 2018 21:10:46 +0000 (22:10 +0100)]
update: handle bad XML when using androguard

repo/ch.swift.willi_417101.apk had a C/Java comment in the
AndroidManifest.xml rather than an XML comment:

<uses-sdk android:minSdkVersion="16" android:targetSdkVersion="26">
</uses-sdk>
// Remove permissions introduced by the appsflyer library
<uses-permission android:name="android.permission.ACCESS_COARSE_LOCATION">
</uses-permission>

13 months agoupdate: do not crash with androguard when working with apps without icons
Hans-Christoph Steiner [Sun, 11 Mar 2018 21:09:09 +0000 (22:09 +0100)]
update: do not crash with androguard when working with apps without icons

For example, SpeedoMeterApp.main_1.apk

13 months agoMerge branch 'fix_config_doc' into 'master'
Marcus [Mon, 12 Mar 2018 12:37:43 +0000 (12:37 +0000)]
Merge branch 'fix_config_doc' into 'master'

Fix misleading java_path example

See merge request fdroid/fdroidserver!478

13 months agoFix misleading java_path example
Gioacchino Mazzurco [Mon, 12 Mar 2018 11:01:16 +0000 (12:01 +0100)]
Fix misleading java_path example

As you can see in fdroidserver/common.py:219
for java_version in ('7', '8', '9'):
the code look for java version without the 1. in front, after getting a
bunch of error message that JDK could't be found, investigating the code
and documentation I discovered my configuration was ignored because of
this and realized the example was wrong

13 months agojenkins-test: run checkupdates as final stage of test run
Hans-Christoph Steiner [Sun, 11 Mar 2018 20:36:41 +0000 (21:36 +0100)]
jenkins-test: run checkupdates as final stage of test run

13 months agoMerge branch 'master' into 'master'
Marcus [Fri, 9 Mar 2018 20:13:51 +0000 (20:13 +0000)]
Merge branch 'master' into 'master'

checkupdates: Save app data to correct metadata format

Closes #461

See merge request fdroid/fdroidserver!476

13 months agocheckupdates: Save app data to correct metadata format
mimi89999 [Fri, 9 Mar 2018 19:05:11 +0000 (20:05 +0100)]
checkupdates: Save app data to correct metadata format

Closes #461

13 months agojenkins-test: remove tmp hack now that androguard is available
Hans-Christoph Steiner [Fri, 9 Mar 2018 12:45:51 +0000 (13:45 +0100)]
jenkins-test: remove tmp hack now that androguard is available

Does the same thing as 9f553186e8d24559ab8a9ce2745c041fc094dac4 but for
jenkins.debian.net's profitbricks-build7-amd64 setup.

13 months agogitlab-ci: make ubuntu_lts job test of fdroid/fdroidserver PPA
Hans-Christoph Steiner [Fri, 9 Mar 2018 08:44:38 +0000 (09:44 +0100)]
gitlab-ci: make ubuntu_lts job test of fdroid/fdroidserver PPA

Now that androguard is working, there should be no need for a specific aapt
version.  The aapt included in Ubuntu LTS should always work fine when
androguard handles the bulk of the work.

13 months agoMerge 'fix-scanner' into 'master'
Hans-Christoph Steiner [Thu, 8 Mar 2018 12:56:14 +0000 (13:56 +0100)]
Merge 'fix-scanner' into 'master'

scanner: support Gradle plugin 3.0 syntax for dependencies

fdroid/fdroidserver!475

13 months agofix PEP8 formatting issue
Hans-Christoph Steiner [Thu, 8 Mar 2018 12:55:07 +0000 (13:55 +0100)]
fix PEP8 formatting issue

fdroidserver/scanner.py:35:34: E241 multiple spaces after ','
fdroidserver/scanner.py:36:35: E241 multiple spaces after ','
fdroidserver/scanner.py:37:30: E241 multiple spaces after ','
fdroidserver/scanner.py:38:41: E241 multiple spaces after ','
fdroidserver/scanner.py:39:30: E241 multiple spaces after ','
fdroidserver/scanner.py:40:38: E241 multiple spaces after ','
fdroidserver/scanner.py:41:38: E241 multiple spaces after ','

13 months agoscanner: support Gradle plugin 3.0 syntax for dependencies
relan [Thu, 8 Mar 2018 12:12:58 +0000 (15:12 +0300)]
scanner: support Gradle plugin 3.0 syntax for dependencies

See https://developer.android.com/studio/build/gradle-plugin-3-0-0-migration.html#new_configurations

13 months agotravis-ci: build against latest OSX 10.12 and 10.13 images
Hans-Christoph Steiner [Wed, 7 Mar 2018 14:15:28 +0000 (15:15 +0100)]
travis-ci: build against latest OSX 10.12 and 10.13 images

13 months agotravis-ci: support old versions of OSX
Hans-Christoph Steiner [Wed, 7 Mar 2018 14:14:30 +0000 (15:14 +0100)]
travis-ci: support old versions of OSX

13 months agotravis: brew's Python3 is now called just 'python'
Hans-Christoph Steiner [Wed, 7 Mar 2018 13:14:29 +0000 (14:14 +0100)]
travis: brew's Python3 is now called just 'python'

https://discourse.brew.sh/t/brew-install-python3-fails/1756

13 months agotests: checkupdates now requires a clean git repo state
Hans-Christoph Steiner [Tue, 6 Mar 2018 19:50:23 +0000 (20:50 +0100)]
tests: checkupdates now requires a clean git repo state

13 months agoMerge branch 'checkupdates' into 'master'
Hans-Christoph Steiner [Tue, 6 Mar 2018 19:20:38 +0000 (19:20 +0000)]
Merge branch 'checkupdates' into 'master'

tighten up checkupdates

See merge request fdroid/fdroidserver!474

13 months agobuild/checkupdates/update: log current fdroiddata commit to wiki
Hans-Christoph Steiner [Mon, 5 Mar 2018 20:47:19 +0000 (21:47 +0100)]
build/checkupdates/update: log current fdroiddata commit to wiki

13 months agocheckupdates: exit with error if fdroiddata git repo is dirty
Hans-Christoph Steiner [Mon, 5 Mar 2018 20:44:38 +0000 (21:44 +0100)]
checkupdates: exit with error if fdroiddata git repo is dirty

One key security property of the F-Droid ecosystem is that the sensitive
code is all stored forever in git repos and source tarballs.  That means
we can easily go back and see if there where exploits and where they came
from.  Therefore, checkupdates should require everything in fdroiddata be
committed to git before running.

This provides --allow-dirty to override that behavior.

13 months agoMerge branch 'more-security-fixes' into 'master'
Hans-Christoph Steiner [Mon, 5 Mar 2018 09:10:57 +0000 (09:10 +0000)]
Merge branch 'more-security-fixes' into 'master'

More security fixes

See merge request fdroid/fdroidserver!471

13 months agoVercodeOperation: only allow simple math expresssions and %c
Hans-Christoph Steiner [Fri, 2 Mar 2018 11:50:48 +0000 (12:50 +0100)]
VercodeOperation: only allow simple math expresssions and %c

13 months agohg: use /bin/false to clarify that it is an executable
Hans-Christoph Steiner [Fri, 2 Mar 2018 10:06:26 +0000 (11:06 +0100)]
hg: use /bin/false to clarify that it is an executable

13 months agoSVN: only allow redirects to HTTPS
Hans-Christoph Steiner [Fri, 2 Mar 2018 09:21:55 +0000 (10:21 +0100)]
SVN: only allow redirects to HTTPS

"SVN follows HTTP 301 redirects to svn+ssh:// URLs. As a result, an
innocent looking HTTP URL can be used to trigger a Command Execution with a
301 redirect."
https://blog.recurity-labs.com/2017-08-10/scm-vulns.html#third-round-svn-and-mercurial

I scanned fdroiddata and found no suspicious redirects.  Here's how:

grep -A1 '^Repo *Type: *git-svn' *.txt *.yml| sed -n 's,.*Repo:\(.*\),\1,p' > /tmp/urls.txt

import requests
with open('/tmp/urls.txt') as fp:
    for line in fp:
        try:
            r = requests.head(line.strip())
            print(r.status_code, line)
        except requests.exceptions.SSLError:
            print('SSLError', line)

13 months agoMerge branch 'lint_fix' into 'master'
Marcus [Fri, 2 Mar 2018 22:59:26 +0000 (22:59 +0000)]
Merge branch 'lint_fix' into 'master'

lint: fix update check data https check

See merge request fdroid/fdroidserver!473

13 months agolint: fix update check data https check
Marcus Hoffmann [Fri, 2 Mar 2018 22:56:49 +0000 (23:56 +0100)]
lint: fix update check data https check

The urlver field can be '.', this was not considered in
26bfd7fb28163abbf8f599609ce57e2bd10a9eed.

13 months agoMerge branch 'tighten-up-UpdateCheckData' into 'master'
Marcus [Fri, 2 Mar 2018 11:45:53 +0000 (11:45 +0000)]
Merge branch 'tighten-up-UpdateCheckData' into 'master'

Tighten up UpdateCheckData

See merge request fdroid/fdroidserver!470

13 months agocheckupdates: require UpdateCheckData has valid HTTPS URL
Hans-Christoph Steiner [Thu, 1 Mar 2018 22:51:36 +0000 (23:51 +0100)]
checkupdates: require UpdateCheckData has valid HTTPS URL

13 months agolint: require UpdateCheckData to contain only valid HTTPS URLs
Hans-Christoph Steiner [Thu, 1 Mar 2018 22:29:38 +0000 (23:29 +0100)]
lint: require UpdateCheckData to contain only valid HTTPS URLs

13 months agolint: tighten up HTTPS checks on URLs
Hans-Christoph Steiner [Thu, 1 Mar 2018 22:24:00 +0000 (23:24 +0100)]
lint: tighten up HTTPS checks on URLs

13 months agoMerge branch 'master' into 'master'
Hans-Christoph Steiner [Wed, 28 Feb 2018 21:42:06 +0000 (21:42 +0000)]
Merge branch 'master' into 'master'

update: make icon extraction less dependent on aapt

Closes fdroid-website#192

See merge request fdroid/fdroidserver!469

13 months agolint: ban all dangerous HTML tags
Hans-Christoph Steiner [Tue, 27 Feb 2018 11:09:54 +0000 (12:09 +0100)]
lint: ban all dangerous HTML tags

* https://en.wikipedia.org/wiki/HTML_sanitization
* https://asostack.com/enhance-your-google-play-store-description-with-rich-formatting-and-emojis-5f50ff354e5f

13 months agoupdate: make icon extraction less dependent on aapt
Hans-Christoph Steiner [Mon, 26 Feb 2018 22:43:42 +0000 (23:43 +0100)]
update: make icon extraction less dependent on aapt

For androguard, @thezero already developed a way to get all the icons after
only extracting the icon name.  So this uses that for the aapt-based scans
also, to make them less brittle.

This should fix the problem where `fdroid update` was choosing the XML icon
for apps that include one, like NewPipe.

closes fdroid/fdroid-website#192

13 months agogitlab-ci: set metadata_v0 test to use 1.0.2 as the baseline
Hans-Christoph Steiner [Fri, 23 Feb 2018 21:48:44 +0000 (22:48 +0100)]
gitlab-ci: set metadata_v0 test to use 1.0.2 as the baseline

13 months agorewritemeta: fix proper_format() so lint works with .yml files
Hans-Christoph Steiner [Fri, 23 Feb 2018 21:42:46 +0000 (22:42 +0100)]
rewritemeta: fix proper_format() so lint works with .yml files

13 months agoMerge branch 'allow-dashes-and-underscores-in-signature-file-names-when-checking...
Hans-Christoph Steiner [Thu, 22 Feb 2018 23:23:50 +0000 (23:23 +0000)]
Merge branch 'allow-dashes-and-underscores-in-signature-file-names-when-checking-for-reproducability' into 'master'

allow dashes and underscores in signature file names when checking for reproducability

See merge request fdroid/fdroidserver!468

13 months agoallow dashes and underscores in signature file names when checking for reproducability
Michael Pöhn [Thu, 22 Feb 2018 22:29:30 +0000 (23:29 +0100)]
allow dashes and underscores in signature file names when checking for reproducability

13 months agobump to 1.0.2 to placate pypi 1.0.2
Hans-Christoph Steiner [Thu, 22 Feb 2018 20:23:19 +0000 (21:23 +0100)]
bump to 1.0.2 to placate pypi

I mistakenly uploaded the dist tarball to pypi without the PGP signature.
So I deleted the release, thinking I could reupload it.  It is not possible:
https://github.com/pypa/packaging-problems/issues/74

So this is really just a bump so I can reupload to pypi.

13 months agocommon: tighten up regexs when searching for version name/code and appid
Hans-Christoph Steiner [Thu, 22 Feb 2018 20:15:41 +0000 (21:15 +0100)]
common: tighten up regexs when searching for version name/code and appid

This should have less of a change of matching bad things.
thanks to @stf for the report.  I ran tests comparing the original vs these
new patterns, and it was a 100% match. So at least it didn't make things
worse.

Here's the test script:
#!/usr/bin/env python3

import os
import re

old_vcsearch_g = re.compile(r'''.*[Vv]ersionCode[ =]+["']*([0-9]+)["']*''').search
old_vnsearch_g = re.compile(r'.*[Vv]ersionName *=* *(["\'])((?:(?=(\\?))\3.)*?)\1.*').search
old_psearch_g = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new_vcsearch_g = re.compile(r'''.*[Vv]ersionCode\s*=?\s*["']*([0-9]+)["']*''').search
new_vnsearch_g = re.compile(r'''.*[Vv]ersionName\s*=?\s*(["'])((?:(?=(\\?))\3.)*?)\1.*''').search
new_psearch_g = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search

old = re.compile(r'.*(packageName|applicationId) *=* *["\']([^"]+)["\'].*').search
new = re.compile(r'''.*(packageName|applicationId)\s*=*\s*["']([^"']+)["'].*''').search

for root, dirs, files in os.walk('build'):
    for f in files:
        if f.endswith('.gradle'):
            with open(os.path.join(root, f)) as fp:
                for line in fp:
                    for old, new in ((old_vcsearch_g, new_vcsearch_g),
                                     (old_vnsearch_g, new_vnsearch_g),
                                     (old_psearch_g, new_psearch_g)):
                        found_old = old(line)
                        found_new = new(line)
                        oldresult = None
                        newresult = None
                        if found_old or found_new:
                            if found_old:
                                oldresult = found_old.groups()
                                #print('OLD', oldresult)
                            if found_new:
                                newresult = found_new.groups()
                                #print('NEW', newresult)
                            if oldresult != newresult:
                                print('--------------------------------')
                                print(f, oldresult, newresult)

13 months agoBump to 1.0.1! 1.0.1
Hans-Christoph Steiner [Thu, 22 Feb 2018 19:01:43 +0000 (20:01 +0100)]
Bump to 1.0.1!

13 months agoMerge branch 'weblate' into 'master'
Hans-Christoph Steiner [Thu, 22 Feb 2018 14:59:32 +0000 (14:59 +0000)]
Merge branch 'weblate' into 'master'

Weblate

See merge request fdroid/fdroidserver!467

13 months agoWeblate
Hans-Christoph Steiner [Thu, 22 Feb 2018 14:59:32 +0000 (14:59 +0000)]
Weblate

13 months agoMerge branch 'androguard-fixes' into 'master'
Hans-Christoph Steiner [Thu, 22 Feb 2018 14:39:34 +0000 (14:39 +0000)]
Merge branch 'androguard-fixes' into 'master'

Androguard fixes

See merge request fdroid/fdroidserver!466

13 months agosignatures: future-proof fetching app ID info from APK
Hans-Christoph Steiner [Thu, 22 Feb 2018 13:30:39 +0000 (14:30 +0100)]
signatures: future-proof fetching app ID info from APK

We're not using platformBuildVersionName and it might go away just like it
appeared: with no good reason or announcement.

13 months agoupdate: switch to improved androguard detection
Hans-Christoph Steiner [Thu, 22 Feb 2018 12:39:41 +0000 (13:39 +0100)]
update: switch to improved androguard detection

13 months agogitlab-ci: remove apt-get progress dumps from build log
Hans-Christoph Steiner [Thu, 22 Feb 2018 09:32:27 +0000 (10:32 +0100)]
gitlab-ci: remove apt-get progress dumps from build log

13 months agogitlab-ci: try to download PPA keys until they succeed
Hans-Christoph Steiner [Thu, 22 Feb 2018 09:27:10 +0000 (10:27 +0100)]
gitlab-ci: try to download PPA keys until they succeed

This download occasionally fails, so this keeps retrying till it succeeds.
The CI job has a time limit, so no need to figure out an exit condition.

13 months agotests: do not automatically run install.TestCase, its troublesome
Hans-Christoph Steiner [Wed, 21 Feb 2018 13:08:03 +0000 (14:08 +0100)]
tests: do not automatically run install.TestCase, its troublesome

`fdroid install` is rarely used, if at all, and the test frequently fails
for no reason in gitlab-ci, because it can't start the adb server.

13 months agogitlab-ci: test against latest build-tools 27.0.3
Hans-Christoph Steiner [Wed, 21 Feb 2018 12:59:34 +0000 (13:59 +0100)]
gitlab-ci: test against latest build-tools 27.0.3

13 months agogitlab-ci: androguard from Debian/unstable til it stabilizes
Hans-Christoph Steiner [Thu, 22 Feb 2018 13:40:03 +0000 (14:40 +0100)]
gitlab-ci: androguard from Debian/unstable til it stabilizes