[termux-packages] / README.md

termux-packages
===============
This project contains scripts and patches to cross compile and package packages for
the [Termux](http://termux.com/) Android application.


Building a package
==================
In a non-rooted Android device an app such as Termux may not write to system locations,
which is why every package is installed inside the private file area of the Termux app:
	PREFIX=/data/data/com.termux/files/usr

For simplicity while developing and building, the build scripts here assume that a /data
folder is reserved for use on the host builder, which requires setup:
	sudo mkdir /data
	sudo chown $USER /data

The basic flow is then to run "./build-package.sh $PKG", which
- Sets up a patched stand-alone Android NDK toolchain

* Reads packages/$PKG/build.sh to find out where to find the source code of the  package and how to build it.

* Applies all patches in packages/$PKG/\*.patch

* Builds the package and installs it to $PREFIX

* Packages the package in one or more .dpkg files for distribution
Reading and following build-package.sh is the best way to understand what's going on here.


Additional utilities
====================
* build-all.sh: used for building all packages in the correct order (using buildorder.py)

* check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+

* detect-hardlinks.sh: Used for finding if any packages uses hardlinks, which does not work on Android M

* check-versions.sh: used for checking for package updates

* clean-rebuild-all.sh: used for doing a clean rebuild of all packages (takes a couple of hours)
	
* list-packages.sh: used for listing all packages with a one-line summary


Resources about cross-compiling packages
========================================
* [Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/index.html)

* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/)

* [Cross-Compiled Linux From Scratch](http://cross-lfs.org/view/svn/x86_64-64/)

* [OpenWrt](https://openwrt.org/), an embedded Linx distribution, contains [patches and build scripts](https://dev.openwrt.org/browser/packages)

* http://dan.drown.org/android contains [patches for cross-compiling to Android](http://dan.drown.org/android/src/) as well as [work notes](http://dan.drown.org/android/worknotes.html), including a modified dynamic linker to avoid messing with LD_LIBRARY_PATH.

* [CCTools](http://cctools.info/index.php?title=Main_Page) is an Android native IDE containing [patches for several programs](https://code.google.com/p/cctools/source/browse/#svn%2Ftrunk%2Fcctools-repo%2Fpatches) and [a bug tracker](https://code.google.com/p/cctools/issues/list).

* [BotBrew](http://botbrew.com/) was a package manager for rooted devices with [sources on github](https://github.com/jyio/botbrew). Based on opkg and was transitioning to apt.

* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/recipes) contains recipes for building packages for Android.


Common porting problems
=======================
* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A package from the NDK, libandroid-support,
contains these and may be used by all packages.

* "error: z: no archive symbol table (run ranlib)" usually means that the build machines libz is used instead of the one for cross compilation, due to the builder library -L path being setup incorrectly

* rindex(3) is defined in &lt;strings.h&gt; but does not exist in NDK, but strrchr(3) from &lt;string.h&gt; is preferred anyway

* &lt;sys/termios.h&gt; does not exist, but &lt;termios.h&gt; is the standard location.

* &lt;sys/fcntl.h&gt; does not exist, but &lt;fcntl.h&gt; is the standard location.

* glob(3) system function (glob.h) - not in bionic, but use the libglob package

* undefined reference to 'rpl_malloc' and/or 'rpl_realloc': These functions are added by some autoconf setups
  when it fails to detect 0-safe malloc and realloc during cross-compilating. Avoided by defining
  "ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes".
  See http://wiki.buici.com/xwiki/bin/view/Programing+C+and+C%2B%2B/Autoconf+and+RPL_MALLOC

* cmake and cross compiling: http://www.cmake.org/Wiki/CMake_Cross_Compiling
  CMAKE_FIND_ROOT_PATH=$TERMUX_PREFIX to search there.
  CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY and
  CMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY
  for only searching there and don't fall back to build machines

* Android is removing sys/timeb.h because it was removed in POSIX 2008, but ftime(3) can be replaced with gettimeofday(2)

* mempcpy(3) is a GNU extension. We have added it to &lt;string.h&gt; provided TERMUX_EXPOSE_MEMPCPY is defined,
  so use something like CFLAGS+=" -DTERMUX_EXPOSE_MEMPCPY=1" for packages expecting that function to exist.


dlopen() and RTLD&#95;&#42; flags
=================================
&lt;dlfn.h&gt; declares

    enum { RTLD_NOW=0, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=2,       RTLD_NOLOAD=4}; // 32-bit
    enum { RTLD_NOW=2, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=0x00100, RTLD_NOLOAD=4}; // 64-bit

These differs from glibc ones in that

1. They are not preprocessor #define:s so cannot be checked for with #ifdef RTLD_GLOBAL (dln.c in ruby does this)
2. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
3. They are missing some values (RTLD_BINDING_MASK, RTLD_NOLOAD, ...)


RPATH, LD_LIBRARY_PATH AND RUNPATH
==================================
On desktop linux the linker searches for shared libraries in:
1. RPATH - a list of directories which is linked into the executable, supported on most UNIX systems. It is ignored if RUNPATH is present.
2. LD_LIBRARY_PATH - an environment variable which holds a list of directories
3. RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX systems, e.g. on most current Linux systems
The Android linker (/system/bin/linker) does not support RPATH or RUNPATH, so we set LD_LIBRARY_PATH=$USR/lib and try to avoid building
useless rpath entries with --disable-rpath configure flags.
Another option to avoid depending on LD_LIBRARY_PATH would be supplying a custom linker - this is not done due to the overhead of maintaining a custom linker.


Warnings about unused DT entries
================================
Starting from 5.1 the Android linker warns about VERNEED (0x6FFFFFFE) and VERNEEDNUM (0x6FFFFFFF) ELF dynamic sections:

    WARNING: linker: $BINARY: unused DT entry: type 0x6ffffffe arg ...
    WARNING: linker: $BINARY: unused DT entry: type 0x6fffffff arg ...
These may come from version scripts in a Makefile such as:

    -Wl,--version-script=$(top_srcdir)/proc/libprocps.sym
The termux-elf-cleaner utilty is run from build-package.sh and should normally take care of that problem.

Obtaining shell access on an emulator
=====================================
First install and start sshd on device:

    apt install openssh
    sshd
Then follow the below steps:

    # Find out the linux user for the package to use in the chown command later:
    adb shell dumpsys package com.termux | grep userId=
    # Push your public ssh key:
    adb push $HOME/.ssh/id_dsa.pub /data/data/com.termux/files/home/.ssh/authorized_keys
    # Use the linux user for the package, 10053 below, to set ownerhip and permissions:
    adb shell chown -R 10053 /data/data/com.termux/files/home/.ssh/
    adb shell chmod -R 0700 /data/data/com.termux/files/home/.ssh/
    # Forward port 8022 to the emulator:
    adb forward tcp:8022 tcp:8022
    # Finally connect with ssh:
    ssh -p 8022 localhost

Bootstrapping on device
=======================
To get files on device one option is:

    udpsvd -vE 0.0.0.0 8069 tftpd -c . # Run on device. -c arg to allow file uploading
    printf "mode binary\nput out.md\nquit" | tftp 192.168.0.12 8069 # on computer
Another is with ftp:
    tcpsvd -vE 0.0.0.0 8021 ftpd -w . # Run on device. -w arg to allow file uploading
    printf "put tmp.c\nquit" | ftp -n 192.168.0.12 8021 # Run on computer. -n arg to use anonymous login
NOTE: The ftpd and tftpd programs has been patched to run without chroot. This means that the directory serving is only the starting point and clients may cd out of if the access the whole system!
Commit	Line	Data
798619b9 FF	1	termux-packages
	2	===============
	3	This project contains scripts and patches to cross compile and package packages for
	4	the [Termux](http://termux.com/) Android application.
	5
	6
9ecaa95d FF	7	Building a package
9ecaa95d FF	8	==================
798619b9 FF	9	In a non-rooted Android device an app such as Termux may not write to system locations,
	10	which is why every package is installed inside the private file area of the Termux app:
	11	PREFIX=/data/data/com.termux/files/usr
	12
	13	For simplicity while developing and building, the build scripts here assume that a /data
	14	folder is reserved for use on the host builder, which requires setup:
	15	sudo mkdir /data
	16	sudo chown $USER /data
	17
	18	The basic flow is then to run "./build-package.sh $PKG", which
9ecaa95d FF	19	- Sets up a patched stand-alone Android NDK toolchain
	20
	21	* Reads packages/$PKG/build.sh to find out where to find the source code of the package and how to build it.
	22
	23	* Applies all patches in packages/$PKG/\*.patch
	24
	25	* Builds the package and installs it to $PREFIX
	26
	27	* Packages the package in one or more .dpkg files for distribution
798619b9 FF	28	Reading and following build-package.sh is the best way to understand what's going on here.
798619b9 FF	29
9ecaa95d FF	30
	31	Additional utilities
	32	====================
	33	* build-all.sh: used for building all packages in the correct order (using buildorder.py)
	34
	35	* check-pie.sh: Used for verifying that all binaries are using PIE, which is required for Android 5+
	36
	37	* detect-hardlinks.sh: Used for finding if any packages uses hardlinks, which does not work on Android M
	38
	39	* check-versions.sh: used for checking for package updates
	40
	41	* clean-rebuild-all.sh: used for doing a clean rebuild of all packages (takes a couple of hours)
	42
	43	* list-packages.sh: used for listing all packages with a one-line summary
798619b9 FF	44
	45
	46	Resources about cross-compiling packages
	47	========================================
	48	* [Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/index.html)
	49
	50	* [Beyond Linux From Scratch](http://www.linuxfromscratch.org/blfs/view/svn/)
	51
	52	* [Cross-Compiled Linux From Scratch](http://cross-lfs.org/view/svn/x86_64-64/)
	53
	54	* [OpenWrt](https://openwrt.org/), an embedded Linx distribution, contains [patches and build scripts](https://dev.openwrt.org/browser/packages)
	55
	56	* http://dan.drown.org/android contains [patches for cross-compiling to Android](http://dan.drown.org/android/src/) as well as [work notes](http://dan.drown.org/android/worknotes.html), including a modified dynamic linker to avoid messing with LD_LIBRARY_PATH.
	57
	58	* [CCTools](http://cctools.info/index.php?title=Main_Page) is an Android native IDE containing [patches for several programs](https://code.google.com/p/cctools/source/browse/#svn%2Ftrunk%2Fcctools-repo%2Fpatches) and [a bug tracker](https://code.google.com/p/cctools/issues/list).
	59
	60	* [BotBrew](http://botbrew.com/) was a package manager for rooted devices with [sources on github](https://github.com/jyio/botbrew). Based on opkg and was transitioning to apt.
	61
	62	* [Kivy recipes](https://github.com/kivy/python-for-android/tree/master/recipes) contains recipes for building packages for Android.
	63
	64
	65	Common porting problems
	66	=======================
	67	* The Android bionic libc does not have iconv and gettext/libintl functionality built in. A package from the NDK, libandroid-support,
	68	contains these and may be used by all packages.
	69
	70	* "error: z: no archive symbol table (run ranlib)" usually means that the build machines libz is used instead of the one for cross compilation, due to the builder library -L path being setup incorrectly
	71
	72	* rindex(3) is defined in <strings.h> but does not exist in NDK, but strrchr(3) from <string.h> is preferred anyway
	73
	74	* <sys/termios.h> does not exist, but <termios.h> is the standard location.
	75
	76	* <sys/fcntl.h> does not exist, but <fcntl.h> is the standard location.
	77
	78	* glob(3) system function (glob.h) - not in bionic, but use the libglob package
	79
	80	* undefined reference to 'rpl_malloc' and/or 'rpl_realloc': These functions are added by some autoconf setups
	81	when it fails to detect 0-safe malloc and realloc during cross-compilating. Avoided by defining
	82	"ac_cv_func_malloc_0_nonnull=yes ac_cv_func_realloc_0_nonnull=yes".
	83	See http://wiki.buici.com/xwiki/bin/view/Programing+C+and+C%2B%2B/Autoconf+and+RPL_MALLOC
	84
	85	* cmake and cross compiling: http://www.cmake.org/Wiki/CMake_Cross_Compiling
	86	CMAKE_FIND_ROOT_PATH=$TERMUX_PREFIX to search there.
	87	CMAKE_FIND_ROOT_PATH_MODE_LIBRARY=ONLY and
	88	CMAKE_FIND_ROOT_PATH_MODE_INCLUDE=ONLY
	89	for only searching there and don't fall back to build machines
	90
	91	* Android is removing sys/timeb.h because it was removed in POSIX 2008, but ftime(3) can be replaced with gettimeofday(2)
	92
	93	* mempcpy(3) is a GNU extension. We have added it to <string.h> provided TERMUX_EXPOSE_MEMPCPY is defined,
	94	so use something like CFLAGS+=" -DTERMUX_EXPOSE_MEMPCPY=1" for packages expecting that function to exist.
	95
	96
	97	dlopen() and RTLD_* flags
	98	=================================
	99	<dlfn.h> declares
	100
9ecaa95d FF	101	enum { RTLD_NOW=0, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=2, RTLD_NOLOAD=4}; // 32-bit
9ecaa95d FF	102	enum { RTLD_NOW=2, RTLD_LAZY=1, RTLD_LOCAL=0, RTLD_GLOBAL=0x00100, RTLD_NOLOAD=4}; // 64-bit
798619b9 FF	103
	104	These differs from glibc ones in that
	105
	106	1. They are not preprocessor #define:s so cannot be checked for with #ifdef RTLD_GLOBAL (dln.c in ruby does this)
	107	2. They differ in value from glibc ones, so cannot be hardcoded in files (DLFCN.py in python does this)
	108	3. They are missing some values (RTLD_BINDING_MASK, RTLD_NOLOAD, ...)
	109
	110
	111	RPATH, LD_LIBRARY_PATH AND RUNPATH
	112	==================================
	113	On desktop linux the linker searches for shared libraries in:
	114	1. RPATH - a list of directories which is linked into the executable, supported on most UNIX systems. It is ignored if RUNPATH is present.
	115	2. LD_LIBRARY_PATH - an environment variable which holds a list of directories
	116	3. RUNPATH - same as RPATH, but searched after LD_LIBRARY_PATH, supported only on most recent UNIX systems, e.g. on most current Linux systems
	117	The Android linker (/system/bin/linker) does not support RPATH or RUNPATH, so we set LD_LIBRARY_PATH=$USR/lib and try to avoid building
	118	useless rpath entries with --disable-rpath configure flags.
	119	Another option to avoid depending on LD_LIBRARY_PATH would be supplying a custom linker - this is not done due to the overhead of maintaining a custom linker.
	120
	121
	122	Warnings about unused DT entries
	123	================================
	124	Starting from 5.1 the Android linker warns about VERNEED (0x6FFFFFFE) and VERNEEDNUM (0x6FFFFFFF) ELF dynamic sections:
9ecaa95d FF	125
	126	WARNING: linker: $BINARY: unused DT entry: type 0x6ffffffe arg ...
	127	WARNING: linker: $BINARY: unused DT entry: type 0x6fffffff arg ...
798619b9	128	These may come from version scripts in a Makefile such as:
9ecaa95d FF	129
9ecaa95d FF	130	-Wl,--version-script=$(top_srcdir)/proc/libprocps.sym
798619b9 FF	131	The termux-elf-cleaner utilty is run from build-package.sh and should normally take care of that problem.
798619b9 FF	132
a9abb520 FF	133	Obtaining shell access on an emulator
	134	=====================================
	135	First install and start sshd on device:
9ecaa95d FF	136
	137	apt install openssh
	138	sshd
	139	Then follow the below steps:
	140
	141	# Find out the linux user for the package to use in the chown command later:
	142	adb shell dumpsys package com.termux \| grep userId=
	143	# Push your public ssh key:
	144	adb push $HOME/.ssh/id_dsa.pub /data/data/com.termux/files/home/.ssh/authorized_keys
	145	# Use the linux user for the package, 10053 below, to set ownerhip and permissions:
	146	adb shell chown -R 10053 /data/data/com.termux/files/home/.ssh/
	147	adb shell chmod -R 0700 /data/data/com.termux/files/home/.ssh/
	148	# Forward port 8022 to the emulator:
	149	adb forward tcp:8022 tcp:8022
	150	# Finally connect with ssh:
	151	ssh -p 8022 localhost
	152
	153	Bootstrapping on device
	154	=======================
798619b9	155	To get files on device one option is:
798619b9	156
9ecaa95d FF	157	udpsvd -vE 0.0.0.0 8069 tftpd -c . # Run on device. -c arg to allow file uploading
	158	printf "mode binary\nput out.md\nquit" \| tftp 192.168.0.12 8069 # on computer
	159	Another is with ftp:
	160	tcpsvd -vE 0.0.0.0 8021 ftpd -w . # Run on device. -w arg to allow file uploading
	161	printf "put tmp.c\nquit" \| ftp -n 192.168.0.12 8021 # Run on computer. -n arg to use anonymous login
	162	NOTE: The ftpd and tftpd programs has been patched to run without chroot. This means that the directory serving is only the starting point and clients may cd out of if the access the whole system!