%include "default.mgp"
%deffont "standard" xfont "helvetica-medium-r"
%deffont "thick" xfont "helvetica-bold-r"
%deffont "typewriter" xfont "courier-medium-r"
%deffont "thick-tw" xfont "courier-bold-r"
%default 1 leftfill, size 2, fore "white", back "black", font "thick"
%default 2 size 7, vgap 10, prefix " "
%default 3 size 2, bar "gray70", vgap 10
%default 4 size 5, fore "white", vgap 10, prefix " ", font "standard"
%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%nodefault
%back "blue"
%fore "yellow"
%bquality 10
%font "standard"
%size 10
%center
#%bgrad 0 0 128 0 1 "black" "blue"

%vgap 40
Jigdo
Spreading the load of 
CD and DVD downloads

%size 5
Steve McIntyre <steve@einval.com>

%vgap 60
UKUUG Conference, Sunday 7th August 2005

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Agenda

	The Problem
	Pseudo-Image Kit (PIK)
	Jigsaw Download (jigdo)
	Creating jigdo files using jigdo
	Improving performance - patch mkisofs
	Other changes
	"Nasty C++"
	Jigdoofus
	Further work
	Downloads / more information

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

The Problem
    
Debian ships lots of CD & DVD images

%pause
	2.2 (Potato): 24 CDs

%pause
	3.0 (Woody): 89 CDs

%pause
	3.1 (Sarge): 164 CDs / 26 DVDs

%pause, fore "red"
That's a lot of disk and bandwidth!

%pause, fore "white"
How can we spread the load?

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Pseudo-Image Kit (PIK)

%pause
	Written by Anne Bezemer
    
%pause
	Generate a list of files on the CD

%pause
	Stick them together to form a "pseudo-image"

%pause
	Run rsync against the original image

%pause, fore "red"
	Server still needs to keep all the images for this rsync step

%pause, fore "red"
	Client needs net access to finalise the image

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Jigsaw Download (jigdo)

	Written by Richard Atterer
    
%pause
	Pick out files from the ISO image
		Reference them by md5sum
		"Jigdo" file contains lookup table of md5sum -> filename

%pause
	Gzip the rest of the contents of the image
		Store the data as a binary lump in the "template" file

%pause
	Works well for reducing size
		Sarge CDs/DVDs take ~210 GB as ISO images
		Sarge CDs/DVDs take ~500 MB as jigdo files

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Creating jigdo files using jigdo

%pause
	Create a list of all the files likely to be in the ISO image

%pause
	Create the ISO image as normal

%pause
	Feed the ISO image into jigdo, along with the file list

%pause
	Jigdo calculates checksums for all the listed files

%pause
	Jigdo then walks a byte at a time through the image
		If a checksum match is found, add to the jigdo file
		If not, compress data and add to the template file

%pause, fore "red"
INCREDIBLY slow


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Improving performance - patch mkisofs

%pause
	JTE (Jigdo Template Engine), replace brute-force code
		Performance increase of 20-30x!

%pause
	Added boot support for several extra architectures

%pause
	Used for the Debian sarge release
		Took ~24 hours to create all the CD & DVD jigdo files for the release
		Would have taken nearly a month to do the release using the old jigdo program!

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Other changes

%pause
	Get away from the nasty C++

%pause
	Re-implement most of the jigdo tools
		mkimage
		jigdump
		jigit
		jigsum
		iso-image.pl

%pause
	Add bzip2 support for the template data

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

"Nasty C++"

Inner loop in mktemplate.cc:

%pause, font "typewriter", size 3
do {
    const vector<FilePart*>* hashEntry;

%pause
 #  define x ; \\
    rsum.removeFront(buf[rsumBack], blockLength); \\
    rsum.addBack(buf[data]); \\
    ++data; ++rsumBack; \\
    hashEntry = &block[rsum.getHi() & blockMask]; \\
    if (hashEntry->size() > 1) break; \\
    if (hashEntry->size() == 1) { \\
      FilePart* file = (*hashEntry)[0]; \\
      const RsyncSum64* fileSum = file->getRsyncSum(cache); \\
    if (fileSum != 0 && *fileSum == rsum) break; \\
    }

%pause, fore "yellow", font "thick-tw"
    x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x x

%pause, fore "white", font "typewriter"
 #  undef x
    dataOld = data; // Special case: "Did not break out of loop"
    } while (false);


%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Jigdoofus

%pause
	A jigdo-based filesystem

%pause
	Recreate ISO images on the fly

%pause
	Mount a directory full of template files
		Each template file shows up as an ISO file too

%pause
	Uses FUSE (Filesystems in USErland)
		Very simple read-only filesystem

%pause
	Backend SQLite database for information caching

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Jigdoofus - jigdo filesystem (cont.)

%pause
	Several uses
		Burn CDs/DVDs without creating the image
		Allow mirrors to serve ISO images
		Allow mirrors to act as Bittorrent seeds

%pause
	Early alpha version in testing now
		Creates correct ISO images
		Still some bugs to fix
		Needs better usability & documentation :-)

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%
%page
%back "blue"
%fore "yellow"

Downloads / more information

Jigsaw Download:
	http://atterer.net/jigdo/

JTE / jigdoofus:
	http://www.einval.com/~steve/software/JTE/

Debian CDs/DVDs:
	http://cdimage.debian.org/debian-cd/