[termux-packages] / README.md

termux-packages
===============
This project contains scripts and patches to cross compile and package packages for
the [Termux](http://termux.com/) Android application.

NOTE: This is in a rough state - be prepared for some work and frustrations, and give
feedback if you find incorrect our outdated things! Building is only supported on
ubuntu 15.04 - execute `ubuntu-setup.sh` to install required packages as well as
setting up the `/data/` folder.

Building a package
==================
In a non-rooted Android device an app such as Termux may not write to system locations,
which is why every package is installed inside the private file area of the Termux app:

    PREFIX=/data/data/com.termux/files/usr

For simplicity while developing and building, the build scripts here assume that a /data
folder is reserved for use on the host builder.

The basic flow is then to run "./build-package.sh $PKG", which:
* Sets up a patched stand-alone Android NDK toolchain

* Reads packages/$PKG/build.sh to find out where to find the source code of the  package and how to build it.

* Applies all patches in packages/$PKG/\*.patch

* Builds the package and installs it to $PREFIX

* Packages the package in one or more .dpkg files for distribution
Reading and following build-package.sh is the best way to understand what's going on here.


Additional utilities
====================
* build-all.sh: used for building all packages in the correct order (using buildorder.py)

* check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+

* detect-hardlinks.sh: Used for finding if any packages uses hardlinks, which does not work on Android M

* check-versions.sh: used for checking for package updates

* clean-rebuild-all.sh: used for doing a clean rebuild of all packages (takes a couple of hours)
	
* list-packages.sh: used for listing all packages with a one-line summary


Resources about cross-compiling packages
========================================
* [Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/index.html)

* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/)

* [Cross-Compiled Linux From Scratch](http://cross-lfs.org/view/svn/x86_64-64/)

* [OpenWrt](https://openwrt.org/), an embedded Linx distribution, contains [patches and build scripts](https://dev.openwrt.org/browser/packages)

* http://dan.drown.org/android contains [patches for cross-compiling to Android](http://dan.drown.org/android/src/) as well as [work notes](http://dan.drown.org/android/worknotes.html), including a modified dynamic linker to avoid messing with LD_LIBRARY_PATH.

* [CCTools](http://cctools.info/index.php?title=Main_Page) is an Android native IDE containing [patches for several programs](https://code.google.com/p/cctools/source/browse/#svn%2Ftrunk%2Fcctools-repo%2Fpatches) and [a bug tracker](https://code.google.com/p/cctools/issues/list).

* [BotBrew](http://botbrew.com/) was a package manager for rooted devices with [sources on github](https://github.com/jyio/botbrew). Based on opkg and was transitioning to apt.

* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/recipes) contains recipes for building packages for Android.


Common porting problems
=======================
* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A package from the NDK, libandroid-support,
contains these and may be used by all packages.

* "error: z: no archive symbol table (run ranlib)" usually means that the build machines libz is used instead of the one for cross compilation, due to the builder library -L path being setup incorrectly

* rindex(3) is defined in &lt;strings.h&gt; but does not exist in NDK, but strrchr(3) from &lt;string.h&gt; is preferred anyway

* &lt;sys/termios.h&gt; does not exist, but &lt;termios.h&gt; is the standard location.

* &lt;sys/fcntl.h&gt; does not exist, but &lt;fcntl.h&gt; is the standard location.

* glob(3) system function (glob.h) - not in bionic, but use the `libandroid-glob` package

* undefined reference to 'rpl_malloc' and/or 'rpl_realloc': These functions are added by some autoconf setups
  when it fails to detect 0-safe malloc and realloc during cross-compilating. Avoided by defining
  "ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes".
  See http://wiki.buici.com/xwiki/bin/view/Programing+C+and+C%2B%2B/Autoconf+and+RPL_MALLOC

* cmake and cross compiling: http://www.cmake.org/Wiki/CMake_Cross_Compiling
  CMAKE_FIND_ROOT_PATH=$TERMUX_PREFIX to search there.
  CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY and
  CMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY
  for only searching there and don't fall back to build machines

* Android is removing sys/timeb.h because it was removed in POSIX 2008, but ftime(3) can be replaced with gettimeofday(2)

* mempcpy(3) is a GNU extension. We have added it to &lt;string.h&gt; provided TERMUX_EXPOSE_MEMPCPY is defined,
  so use something like CFLAGS+=" -DTERMUX_EXPOSE_MEMPCPY=1" for packages expecting that function to exist.


dlopen() and RTLD&#95;&#42; flags
=================================
&lt;dlfn.h&gt; declares

    enum { RTLD_NOW=0, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=2,       RTLD_NOLOAD=4}; // 32-bit
    enum { RTLD_NOW=2, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=0x00100, RTLD_NOLOAD=4}; // 64-bit

These differs from glibc ones in that

1. They are not preprocessor #define:s so cannot be checked for with #ifdef RTLD_GLOBAL (dln.c in ruby does this)
2. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
3. They are missing some values (RTLD_BINDING_MASK, RTLD_NOLOAD, ...)


RPATH, LD_LIBRARY_PATH AND RUNPATH
==================================
On desktop linux the linker searches for shared libraries in:

1. RPATH - a list of directories which is linked into the executable, supported on most UNIX systems. It is ignored if RUNPATH is present.
2. LD_LIBRARY_PATH - an environment variable which holds a list of directories
3. RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX systems

The Android linker (/system/bin/linker) does not support RPATH or RUNPATH, so we set LD_LIBRARY_PATH=$USR/lib and try to avoid building useless rpath entries with --disable-rpath configure flags. Another option to avoid depending on LD_LIBRARY_PATH would be supplying a custom linker - this is not done due to the overhead of maintaining a custom linker.


Warnings about unused DT entries
================================
Starting from 5.1 the Android linker warns about VERNEED (0x6FFFFFFE) and VERNEEDNUM (0x6FFFFFFF) ELF dynamic sections:

    WARNING: linker: $BINARY: unused DT entry: type 0x6ffffffe arg ...
    WARNING: linker: $BINARY: unused DT entry: type 0x6fffffff arg ...
These may come from version scripts in a Makefile such as:

    -Wl,--version-script=$(top_srcdir)/proc/libprocps.sym
The termux-elf-cleaner utilty is run from build-package.sh and should normally take care of that problem.

Obtaining shell access on an emulator
=====================================
First install and start sshd on device:

    apt install openssh
    sshd
Then follow the below steps:

    # Find out the linux user for the package to use in the chown command later:
    adb shell dumpsys package com.termux | grep userId=
    # Push your public ssh key:
    adb push $HOME/.ssh/id_dsa.pub /data/data/com.termux/files/home/.ssh/authorized_keys
    # Use the linux user for the package, 10053 below, to set ownerhip and permissions:
    adb shell chown -R 10053 /data/data/com.termux/files/home/.ssh/
    adb shell chmod -R 0700 /data/data/com.termux/files/home/.ssh/
    # Forward port 8022 to the emulator:
    adb forward tcp:8022 tcp:8022
    # Finally connect with ssh:
    ssh -p 8022 localhost

Bootstrapping on device
=======================
To get files on device one option is:

    udpsvd -vE 0.0.0.0 8069 tftpd -c . # Run on device. -c arg to allow file uploading
    printf "mode binary\nput out.md\nquit" | tftp 192.168.0.12 8069 # on computer
Another is with ftp:

    tcpsvd -vE 0.0.0.0 8021 ftpd -w . # Run on device. -w arg to allow file uploading
    printf "put tmp.c\nquit" | ftp -n 192.168.0.12 8021 # Run on computer. -n arg to use anonymous login
NOTE: The ftpd and tftpd programs has been patched to run without chroot. This means that the directory serving is only the starting point and clients may cd out of if the access the whole system!
Commit	Line	Data
798619b9 FF	1	termux-packages
	2	===============
	3	This project contains scripts and patches to cross compile and package packages for
	4	the [Termux](http://termux.com/) Android application.
	5
6dab2062 FF	6	NOTE: This is in a rough state - be prepared for some work and frustrations, and give
	7	feedback if you find incorrect our outdated things! Building is only supported on
	8	ubuntu 15.04 - execute `ubuntu-setup.sh` to install required packages as well as
	9	setting up the `/data/` folder.
798619b9	10
9ecaa95d FF	11	Building a package
9ecaa95d FF	12	==================
798619b9 FF	13	In a non-rooted Android device an app such as Termux may not write to system locations,
798619b9 FF	14	which is why every package is installed inside the private file area of the Termux app:
3b8f4313 FF	15
3b8f4313 FF	16	PREFIX=/data/data/com.termux/files/usr
798619b9 FF	17
798619b9 FF	18	For simplicity while developing and building, the build scripts here assume that a /data
6dab2062	19	folder is reserved for use on the host builder.
798619b9	20
3b8f4313 FF	21	The basic flow is then to run "./build-package.sh $PKG", which:
3b8f4313 FF	22	* Sets up a patched stand-alone Android NDK toolchain
9ecaa95d FF	23
	24	* Reads packages/$PKG/build.sh to find out where to find the source code of the package and how to build it.
	25
	26	* Applies all patches in packages/$PKG/\*.patch
	27
	28	* Builds the package and installs it to $PREFIX
	29
	30	* Packages the package in one or more .dpkg files for distribution
798619b9 FF	31	Reading and following build-package.sh is the best way to understand what's going on here.
798619b9 FF	32
9ecaa95d FF	33
	34	Additional utilities
	35	====================
	36	* build-all.sh: used for building all packages in the correct order (using buildorder.py)
	37
	38	* check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+
	39
	40	* detect-hardlinks.sh: Used for finding if any packages uses hardlinks, which does not work on Android M
	41
	42	* check-versions.sh: used for checking for package updates
	43
	44	* clean-rebuild-all.sh: used for doing a clean rebuild of all packages (takes a couple of hours)
	45
	46	* list-packages.sh: used for listing all packages with a one-line summary
798619b9 FF	47
	48
	49	Resources about cross-compiling packages
	50	========================================
	51	* [Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/index.html)
	52
	53	* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/)
	54
	55	* [Cross-Compiled Linux From Scratch](http://cross-lfs.org/view/svn/x86_64-64/)
	56
	57	* [OpenWrt](https://openwrt.org/), an embedded Linx distribution, contains [patches and build scripts](https://dev.openwrt.org/browser/packages)
	58
	59	* http://dan.drown.org/android contains [patches for cross-compiling to Android](http://dan.drown.org/android/src/) as well as [work notes](http://dan.drown.org/android/worknotes.html), including a modified dynamic linker to avoid messing with LD_LIBRARY_PATH.
	60
	61	* [CCTools](http://cctools.info/index.php?title=Main_Page) is an Android native IDE containing [patches for several programs](https://code.google.com/p/cctools/source/browse/#svn%2Ftrunk%2Fcctools-repo%2Fpatches) and [a bug tracker](https://code.google.com/p/cctools/issues/list).
	62
	63	* [BotBrew](http://botbrew.com/) was a package manager for rooted devices with [sources on github](https://github.com/jyio/botbrew). Based on opkg and was transitioning to apt.
	64
	65	* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/recipes) contains recipes for building packages for Android.
	66
	67
	68	Common porting problems
	69	=======================
	70	* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A package from the NDK, libandroid-support,
	71	contains these and may be used by all packages.
	72
	73	* "error: z: no archive symbol table (run ranlib)" usually means that the build machines libz is used instead of the one for cross compilation, due to the builder library -L path being setup incorrectly
	74
	75	* rindex(3) is defined in <strings.h> but does not exist in NDK, but strrchr(3) from <string.h> is preferred anyway
	76
	77	* <sys/termios.h> does not exist, but <termios.h> is the standard location.
	78
	79	* <sys/fcntl.h> does not exist, but <fcntl.h> is the standard location.
	80
3b8f4313	81	* glob(3) system function (glob.h) - not in bionic, but use the `libandroid-glob` package
798619b9 FF	82
	83	* undefined reference to 'rpl_malloc' and/or 'rpl_realloc': These functions are added by some autoconf setups
	84	when it fails to detect 0-safe malloc and realloc during cross-compilating. Avoided by defining
	85	"ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes".
	86	See http://wiki.buici.com/xwiki/bin/view/Programing+C+and+C%2B%2B/Autoconf+and+RPL_MALLOC
	87
	88	* cmake and cross compiling: http://www.cmake.org/Wiki/CMake_Cross_Compiling
	89	CMAKE_FIND_ROOT_PATH=$TERMUX_PREFIX to search there.
	90	CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY and
	91	CMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY
	92	for only searching there and don't fall back to build machines
	93
	94	* Android is removing sys/timeb.h because it was removed in POSIX 2008, but ftime(3) can be replaced with gettimeofday(2)
	95
	96	* mempcpy(3) is a GNU extension. We have added it to <string.h> provided TERMUX_EXPOSE_MEMPCPY is defined,
	97	so use something like CFLAGS+=" -DTERMUX_EXPOSE_MEMPCPY=1" for packages expecting that function to exist.
	98
	99
	100	dlopen() and RTLD_* flags
	101	=================================
	102	<dlfn.h> declares
	103
9ecaa95d FF	104	enum { RTLD_NOW=0, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=2, RTLD_NOLOAD=4}; // 32-bit
9ecaa95d FF	105	enum { RTLD_NOW=2, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=0x00100, RTLD_NOLOAD=4}; // 64-bit
798619b9 FF	106
	107	These differs from glibc ones in that
	108
	109	1. They are not preprocessor #define:s so cannot be checked for with #ifdef RTLD_GLOBAL (dln.c in ruby does this)
	110	2. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
	111	3. They are missing some values (RTLD_BINDING_MASK, RTLD_NOLOAD, ...)
	112
	113
	114	RPATH, LD_LIBRARY_PATH AND RUNPATH
	115	==================================
	116	On desktop linux the linker searches for shared libraries in:
1ddf1728	117
798619b9 FF	118	1. RPATH - a list of directories which is linked into the executable, supported on most UNIX systems. It is ignored if RUNPATH is present.
798619b9 FF	119	2. LD_LIBRARY_PATH - an environment variable which holds a list of directories
1ddf1728 FF	120	3. RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX systems
	121
	122	The Android linker (/system/bin/linker) does not support RPATH or RUNPATH, so we set LD_LIBRARY_PATH=$USR/lib and try to avoid building useless rpath entries with --disable-rpath configure flags. Another option to avoid depending on LD_LIBRARY_PATH would be supplying a custom linker - this is not done due to the overhead of maintaining a custom linker.
798619b9 FF	123
	124
	125	Warnings about unused DT entries
	126	================================
	127	Starting from 5.1 the Android linker warns about VERNEED (0x6FFFFFFE) and VERNEEDNUM (0x6FFFFFFF) ELF dynamic sections:
9ecaa95d FF	128
	129	WARNING: linker: $BINARY: unused DT entry: type 0x6ffffffe arg ...
	130	WARNING: linker: $BINARY: unused DT entry: type 0x6fffffff arg ...
798619b9	131	These may come from version scripts in a Makefile such as:
9ecaa95d FF	132
9ecaa95d FF	133	-Wl,--version-script=$(top_srcdir)/proc/libprocps.sym
798619b9 FF	134	The termux-elf-cleaner utilty is run from build-package.sh and should normally take care of that problem.
798619b9 FF	135
a9abb520 FF	136	Obtaining shell access on an emulator
	137	=====================================
	138	First install and start sshd on device:
9ecaa95d FF	139
	140	apt install openssh
	141	sshd
	142	Then follow the below steps:
	143
	144	# Find out the linux user for the package to use in the chown command later:
	145	adb shell dumpsys package com.termux \| grep userId=
	146	# Push your public ssh key:
	147	adb push $HOME/.ssh/id_dsa.pub /data/data/com.termux/files/home/.ssh/authorized_keys
	148	# Use the linux user for the package, 10053 below, to set ownerhip and permissions:
	149	adb shell chown -R 10053 /data/data/com.termux/files/home/.ssh/
	150	adb shell chmod -R 0700 /data/data/com.termux/files/home/.ssh/
	151	# Forward port 8022 to the emulator:
	152	adb forward tcp:8022 tcp:8022
	153	# Finally connect with ssh:
	154	ssh -p 8022 localhost
	155
	156	Bootstrapping on device
	157	=======================
798619b9	158	To get files on device one option is:
798619b9	159
9ecaa95d FF	160	udpsvd -vE 0.0.0.0 8069 tftpd -c . # Run on device. -c arg to allow file uploading
	161	printf "mode binary\nput out.md\nquit" \| tftp 192.168.0.12 8069 # on computer
	162	Another is with ftp:
e419d02d	163
9ecaa95d FF	164	tcpsvd -vE 0.0.0.0 8021 ftpd -w . # Run on device. -w arg to allow file uploading
	165	printf "put tmp.c\nquit" \| ftp -n 192.168.0.12 8021 # Run on computer. -n arg to use anonymous login
	166	NOTE: The ftpd and tftpd programs has been patched to run without chroot. This means that the directory serving is only the starting point and clients may cd out of if the access the whole system!