From: Hans-Christoph Steiner Date: Wed, 26 Aug 2015 12:44:36 +0000 (+0000) Subject: Merge branch 'support-vagrant-cachier' into 'master' X-Git-Tag: 0.5.0~180 X-Git-Url: http://www.chiark.greenend.org.uk/ucgi/~ianmdlvl/git?a=commitdiff_plain;h=99d0c55fe95a354bb1ba468339cec468a9c226b5;hp=-c;p=fdroidserver.git Merge branch 'support-vagrant-cachier' into 'master' Add optional support for vagrant-cachier plugin Building the basebox is excruciating for people on slow connections. I'm particularly sensitive to this after living in Central America for awhile :) This won't affect anyone who hasn't installed the plugin. For those who do, it creates a persistent shared folder for each box (ie. testing23.box) and detects directories to cache between VM builds (apt, gems, pip, chef cache, etc.) (The only downside is that, for those following server setup does who are not aware what vagrant-cachier does, it might be unexpected that artifacts persist between vagrant destroys.) See merge request !25 --- 99d0c55fe95a354bb1ba468339cec468a9c226b5 diff --combined docs/fdroid.texi index 4e87eaf3,dacf6cc9..3ac3927c --- a/docs/fdroid.texi +++ b/docs/fdroid.texi @@@ -8,7 -8,7 +8,7 @@@ @copying This manual is for the F-Droid repository server tools. -Copyright @copyright{} 2010, 2011, 2012, 2013 Ciaran Gultnieks +Copyright @copyright{} 2010, 2011, 2012, 2013, 2014, 2015 Ciaran Gultnieks Copyright @copyright{} 2011 Henrik Tunedal, Michael Haas, John Sullivan @@@ -82,8 -82,6 +82,8 @@@ intended usage. At the very least, you' GNU/Linux @item Python 2.x +To be sure of being able to process all apk files without error, you need +2.7.7 or later. See @code{http://bugs.python.org/issue14315}. @item The Android SDK Tools and Build-tools. Note that F-Droid does not assume that you have the Android SDK in your @@@ -115,8 -113,11 +115,10 @@@ VirtualBox (debian package virtualbox @item Ruby (debian packages ruby and rubygems) @item -Vagrant (unpackaged) Be sure to use 1.3.x because 1.4.x is completely broken -(at the time of writing, the forthcoming 1.4.3 might work) +Vagrant (unpackaged, tested on v1.4.3) @item + vagrant-cachier plugin (unpackaged): `vagrant plugin install vagrant-cachier` + @item Paramiko (debian package python-paramiko) @item Imaging (debian package python-imaging) @@@ -456,7 -457,7 +458,7 @@@ following them). In fact, you can stand command, without changing the functional content, by running: @example -fdroid rewritemetadata +fdroid rewritemeta @end example The following sections describe the fields recognised within the file. @@@ -470,7 -471,6 +472,7 @@@ * Web Site:: * Source Code:: * Issue Tracker:: +* Changelog:: * Donate:: * FlattrID:: * Bitcoin:: @@@ -635,16 -635,6 +637,16 @@@ applications have one This is converted to (@code{}) in the public index file. +@node Changelog +@section Changelog + +@cindex Changelog + +The URL for the application's changelog. Optional, since not all +applications have one. + +This is converted to (@code{}) in the public index file. + @node Donate @section Donate @@@ -787,6 -777,11 +789,6 @@@ root dir Here's an example of a complex git-svn Repo URL: http://svn.code.sf.net/p/project/code/svn;trunk=trunk;tags=tags;branches=branches -For a Subversion repo that requires authentication, you can precede the repo -URL with username:password@ and those parameters will be passed as @option{--username} -and @option{--password} to the SVN checkout command. (This now works for both -svn and git-svn) - If the Repo Type is @code{srclib}, then you must specify the name of the according srclib .txt file. For example if @code{scrlibs/FooBar.txt} exist and you want to use this srclib, then you have to set Repo to @@@ -840,9 -835,7 +842,9 @@@ As for 'prebuild', but runs on the sour takes place. You can use $$SDK$$, $$NDK$$ and $$MVN3$$ to substitute the paths to the -android SDK and NDK directories, and maven 3 executable respectively. +android SDK and NDK directories, and maven 3 executable respectively. The +following per-build variables are available likewise: $$VERSION$$, +$$VERCODE$$ and $$COMMIT$$. @item oldsdkloc=yes The sdk location in the repo is in an old format, or the build.xml is @@@ -895,7 -888,7 +897,7 @@@ which architecture or platform the apk If specified, the package version code in the AndroidManifest.xml is replaced with the version code for the build. See also forceversion. -@item rm=relpath1,relpath2,... +@item rm=[,,...] Specifies the relative paths of files or directories to delete before the build is done. The paths are relative to the base of the build directory - i.e. the root of the directory structure checked out from @@@ -905,7 -898,7 +907,7 @@@ AndroidManifest.xml Multiple files/directories can be specified by separating them with ','. Directories will be recursively deleted. -@item extlibs=a,b,... +@item extlibs=[,,...] Comma-separated list of external libraries (jar files) from the @code{build/extlib} library, which will be placed in the @code{libs} directory of the project. @@@ -956,11 -949,9 +958,11 @@@ the @code{srclib} directory for detail You can use $$SDK$$, $$NDK$$ and $$MVN3$$ to substitute the paths to the android SDK and NDK directories, and Maven 3 executable respectively e.g. -for when you need to run @code{android update project} explicitly. +for when you need to run @code{android update project} explicitly. The +following per-build variables are available likewise: $$VERSION$$, $$VERCODE$$ +and $$COMMIT$$. -@item scanignore=path1,path2,... +@item scanignore=[,,...] Enables one or more files/paths to be excluded from the scan process. This should only be used where there is a very good reason, and probably accompanied by a comment explaining why it is necessary. @@@ -968,7 -959,7 +970,7 @@@ When scanning the source tree for problems, matching files whose relative paths start with any of the paths given here are ignored. -@item scandelete=path1,path2,... +@item scandelete=[,,...] Similar to scanignore=, but instead of ignoring files under the given paths, it tells f-droid to delete the matching files directly. @@@ -982,9 -973,7 +984,9 @@@ mvn or gradle will be executed to clea build= (or the final build) is run. You can use $$SDK$$, $$NDK$$ and $$MVN3$$ to substitute the paths to the -android SDK and NDK directories, and Maven 3 executable respectively. +android SDK and NDK directories, and maven 3 executable respectively. The +following per-build variables are available likewise: $$VERSION$$, +$$VERCODE$$ and $$COMMIT$$. @item buildjni=[yes|no|] Enables building of native code via the ndk-build script before doing @@@ -1002,41 -991,23 +1004,41 @@@ actually not required or used, remove t isn't used nor built will result in an error saying that native libraries were expected in the resulting package. -@item gradle= -Build with Gradle instead of Ant, specifying what flavour to assemble. -If is 'yes' or 'main', no flavour will be used. Note -that this will not work on projects with flavours, since it will build -all flavours and there will be no 'main' build. +@item ndk= +Version of the NDK to use in this build. Defaults to the latest NDK release +that included legacy toolchains, so as to not break builds that require +toolchains no longer included in current versions of the NDK. + +The buildserver supports r9b with its legacy toolchains and the latest release +as of writing this document, r10e. You may add support for more versions by +adding them to 'ndk_paths' in your config file. + +@item gradle=[,,...] +Build with Gradle instead of Ant, specifying what flavours to use. Flavours +are case sensitive since the path to the output apk is as well. + +If only one flavour is given and it is 'yes' or 'main', no flavour will be +used. Note that for projects with flavours, you must specify at least one +valid flavour since 'yes' or 'main' will build all of them separately. @item maven=yes[@@] Build with Maven instead of Ant. An extra @@ tells f-droid to run Maven inside that relative subdirectory. Sometimes it is needed to use @@.. so that builds happen correctly. -@item preassemble= -Space-separated list of Gradle tasks to be run before the assemble task -in a Gradle project build. +@item preassemble=[,,...] +List of Gradle tasks to be run before the assemble task in a Gradle project +build. -@item antcommand=xxx -Specify an alternate Ant command (target) instead of the default +@item gradleprops=[,,...] +List of Gradle properties to pass via the command line to Gradle. A property +can be of the form @code{foo} or of the form @code{key=value}. + +For example: @code{gradleprops=enableFoo,someSetting=bar} will result in +@code{gradle -PenableFoo -PsomeSetting=bar}. + +@item antcommands=[,,...] +Specify an alternate set of Ant commands (target) instead of the default 'release'. It can't be given any flags, such as the path to a build.xml. @item output=path/to/output.apk @@@ -1063,7 -1034,8 +1065,7 @@@ Another example, using extra parameters This is optional - if present, it contains a comma-separated list of any of the following values, describing an anti-feature the application has. -Even though such apps won't be displayed unless a settings box is ticked, -it is a good idea to mention the reasons for the anti-feature(s) in the +It is a good idea to mention the reasons for the anti-feature(s) in the description: @itemize @bullet @@@ -1083,21 -1055,15 +1085,21 @@@ are impossible to replace or that the r without major changes to the app. @item -@samp{NonFreeAdd} - the application promotes non-Free add-ons, such that the +@samp{NonFreeAdd} - the application promotes non-free add-ons, such that the app is effectively an advert for other non-free software and such software is not clearly labelled as such. @item -@samp{NonFreeDep} - the application depends on a non-Free application (e.g. +@samp{NonFreeDep} - the application depends on a non-free application (e.g. Google Maps) - i.e. it requires it to be installed on the device, but does not include it. +@item +@samp{UpstreamNonFree} - the application is or depends on non-free software. +This does not mean that non-free software is included with the app: Most +likely, it has been patched in some way to remove the non-free code. However, +functionality may be missing. + @end itemize @node Disabled @@@ -1259,10 -1225,6 +1261,10 @@@ specify the package name to search for name but change it programmatically in some app flavors, by e.g. appending ".open" or ".free" at the end of the package name. +You can also use @code{Ignore} to ignore package name searching. This should +only be used in some specific cases, for example if the app's build.gradle +file does not contain the package name. + @node Update Check Data @section Update Check Data @@@ -1330,9 -1292,6 +1332,9 @@@ which version should be recommended This field is normally automatically updated - see Update Check Mode. +If not set or set to @code{0}, clients will recommend the highest version they +can, as if the @code{Current Version Code} was infinite. + This is converted to (@code{}) in the public index file. @node No Source Since @@@ -1436,7 -1395,7 +1438,7 @@@ applications @section Setting up a build server In addition to the basic setup previously described, you will also need -a Vagrant-compatible Debian Testing base box called 'testing32' (or testing64 +a Vagrant-compatible Debian Testing base box called 'jessie32' (or jessie64 for a 64-bit VM, if you want it to be much slower, and require more disk space). @@@ -1446,16 -1405,10 +1448,16 @@@ working copies of source trees are move for example, having subversion v1.6 on the host and v1.7 on the guest would fail. -Unless you're very trusting. you should create one of these for yourself -from verified standard Debian installation media. However, you could skip -over the next few paragraphs (and sacrifice some security) by downloading -@url{https://f-droid.org/testing32.box}. +@subsection Creating the Debian base box + +The output of this step is a minimal Debian VM that has support for remote +login and provisioning. + +Unless you're very trusting, you should create one of these for yourself +from verified standard Debian installation media. However, by popular +demand, the @code{makebuildserver} script will automatically download a +prebuilt image unless instructed otherwise. If you choose to use the +prebuilt image, you may safely skip the rest of this section. Documentation for creating a base box can be found at @url{http://docs.vagrantup.com/v1/docs/base_boxes.html}. @@@ -1479,9 -1432,8 +1481,9 @@@ boot, you need to set @code{GRUB_RECORD -1 in @code{/etc/grub/default} and then run @code{update-grub}. @end enumerate +@subsection Creating the F-Droid base box -With this base box available, you should then create @code{makebs.config.py}, +The next step in the process is to create @code{makebs.config.py}, using @code{./examples/makebs.config.py} as a reference - look at the settings and documentation there to decide if any need changing to suit your environment. There is a path for retrieving the base box if it doesn't exist, and an apt @@@ -1509,23 -1461,7 +1511,23 @@@ provisioning scripts detect these, the running the android tools. For example, if you have @code{buildserver/addons/cache/platforms/android-19.tar.gz} that will be used when installing the android-19 platform, instead of re-downloading it -using @code{android update sdk --no-ui -t android-19}. +using @code{android update sdk --no-ui -t android-19}. It is possible to +create the cache files of this additions from a local installation of the +SDK including these: + +@example +cd /path/to/android-sdk/platforms +tar czf android-19.tar.gz android-19 +mv android-19.tar.gz /path/to/buildserver/addons/cache/platforms/ +@end example + +If you have already built a buildserver it is also possible to get this +files directly from the buildserver: + +@example +vagrant ssh -- -C 'tar -C ~/android-sdk/platforms czf android-19.tar.gz android-19' +vagrant ssh -- -C 'cat ~/android-sdk/platforms/android-19.tar.gz' > /path/to/fdroidserver/buildserver/cache/platforms/android19.tar.gz +@end example Once it's complete you'll have a new base box called 'buildserver' which is what's used for the actual builds. You can then build packages as normal, diff --combined makebuildserver index 5eba8049,1c9ef248..5b4862f3 --- a/makebuildserver +++ b/makebuildserver @@@ -62,9 -62,9 +62,9 @@@ if not os.path.exists(cachedir) os.mkdir(cachedir) cachefiles = [ - ('android-sdk_r23.0.2-linux.tgz', - 'https://dl.google.com/android/android-sdk_r23.0.2-linux.tgz', - 'a86741fee9140c340b60fe545566db7c0a43a0963f3c7e64d07b4d05ebbe89f4'), + ('android-sdk_r24.3.4-linux.tgz', + 'https://dl.google.com/android/android-sdk_r24.3.4-linux.tgz', + '886412375d8fe6e49a1583e57a8a36a47943666da681701ba9ad1ab7236e83ea'), ('gradle-1.4-bin.zip', 'https://services.gradle.org/distributions/gradle-1.4-bin.zip', 'cd99e85fbcd0ae8b99e81c9992a2f10cceb7b5f009c3720ef3a0078f4f92e94e'), @@@ -89,34 -89,13 +89,34 @@@ ('gradle-1.12-bin.zip', 'https://services.gradle.org/distributions/gradle-1.12-bin.zip', '8734b13a401f4311ee418173ed6ca8662d2b0a535be8ff2a43ecb1c13cd406ea'), + ('gradle-2.1-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.1-bin.zip', + '3eee4f9ea2ab0221b89f8e4747a96d4554d00ae46d8d633f11cfda60988bf878'), + ('gradle-2.2.1-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.2.1-bin.zip', + '420aa50738299327b611c10b8304b749e8d3a579407ee9e755b15921d95ff418'), + ('gradle-2.3-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.3-bin.zip', + '010dd9f31849abc3d5644e282943b1c1c355f8e2635c5789833979ce590a3774'), + ('gradle-2.4-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.4-bin.zip', + 'c4eaecc621a81f567ded1aede4a5ddb281cc02a03a6a87c4f5502add8fc2f16f'), + ('gradle-2.5-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.5-bin.zip', + '3f953e0cb14bb3f9ebbe11946e84071547bf5dfd575d90cfe9cc4e788da38555'), + ('gradle-2.6-bin.zip', + 'https://services.gradle.org/distributions/gradle-2.6-bin.zip', + '18a98c560af231dfa0d3f8e0802c20103ae986f12428bb0a6f5396e8f14e9c83'), ('Kivy-1.7.2.tar.gz', 'https://pypi.python.org/packages/source/K/Kivy/Kivy-1.7.2.tar.gz', - '0485e2ef97b5086df886eb01f8303cb542183d2d71a159466f99ad6c8a1d03f1') - ] + '0485e2ef97b5086df886eb01f8303cb542183d2d71a159466f99ad6c8a1d03f1'), +] if config['arch64']: cachefiles.extend([ + ('android-ndk-r10e-linux-x86_64.bin', + 'https://dl.google.com/android/ndk/android-ndk-r10e-linux-x86_64.bin', + '102d6723f67ff1384330d12c45854315d6452d6510286f4e5891e00a5a8f1d5a'), ('android-ndk-r9b-linux-x86_64.tar.bz2', 'https://dl.google.com/android/ndk/android-ndk-r9b-linux-x86_64.tar.bz2', '8956e9efeea95f49425ded8bb697013b66e162b064b0f66b5c75628f76e0f532'), @@@ -125,9 -104,6 +125,9 @@@ 'de93a394f7c8f3436db44568648f87738a8d09801a52f459dcad3fc047e045a1')]) else: cachefiles.extend([ + ('android-ndk-r10e-linux-x86.bin', + 'https://dl.google.com/android/ndk/android-ndk-r10e-linux-x86.bin', + '92b07d25aaad9b341a7f2b2a62402d508e948bf2dea3ee7b65a6aeb18bca7df5'), ('android-ndk-r9b-linux-x86.tar.bz2', 'https://dl.google.com/android/ndk/android-ndk-r9b-linux-x86.tar.bz2', '748104b829dd12afb2fdb3044634963abb24cdb0aad3b26030abe2e9e65bfc81'), @@@ -168,21 -144,22 +168,28 @@@ for f, src, shasum in cachefiles # Generate an appropriate Vagrantfile for the buildserver, based on our # settings... vagrantfile = """ -Vagrant::Config.run do |config| +Vagrant.configure("2") do |config| + if Vagrant.has_plugin?("vagrant-cachier") + config.cache.scope = :box + config.cache.auto_detect = false + config.cache.enable :apt + config.cache.enable :chef + end + config.vm.box = "{0}" config.vm.box_url = "{1}" - config.vm.customize ["modifyvm", :id, "--memory", "{2}"] + config.vm.provider "virtualbox" do |v| + v.customize ["modifyvm", :id, "--memory", "{2}"] + v.customize ["modifyvm", :id, "--cpus", "{3}"] + end config.vm.provision :shell, :path => "fixpaths.sh" -""".format(config['basebox'], config['baseboxurl'], config['memory']) +""".format(config['basebox'], + config['baseboxurl'], + config['memory'], + config.get('cpus', 1)) if 'aptproxy' in config and config['aptproxy']: vagrantfile += """ config.vm.provision :shell, :inline => 'sudo echo "Acquire::http {{ Proxy \\"{0}\\"; }};" > /etc/apt/apt.conf.d/02proxy && sudo apt-get update'