1 From ijackson Mon Sep 26 15:37:19 +0100 2016
2 X-VM-v5-Data: ([nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil nil]
3 [nil "Monday" "26" "September" "2016" "15:37:19" "+0100" "Ian Jackson" "ijackson@chiark.greenend.org.uk" nil nil "Intent to commit craziness - source package unpacking" "^From:" nil nil "9" nil nil nil nil nil nil nil nil nil nil]
6 X-Mozilla-Status2: 00000000
8 Content-Type: text/plain; charset=us-ascii
9 Content-Transfer-Encoding: 7bit
10 Message-ID: <22505.12959.668142.478444@chiark.greenend.org.uk>
11 X-Mailer: VM 8.2.0b under 24.4.1 (i586-pc-linux-gnu)
12 From: Ian Jackson <ijackson@chiark.greenend.org.uk>
13 To: debian-dpkg@lists.debian.org,
14 Guido Guenther <agx@debian.org>,
15 Bernhard R. Link <brlink@debian.org>,
16 vcs-pkg-discuss@lists.alioth.debian.org
17 Subject: Intent to commit craziness - source package unpacking
18 Date: Mon, 26 Sep 2016 15:37:19 +0100
22 * dpkg developers, please tell me whether I am making assumptions
23 that are likely to become false. Particularly, on the behaviour of
24 successive runs of dpkg-source --before-build with successively
27 * git-buildpackage and git-dpm developers, please point me to
28 information about what metadata to put into the commit message for
29 a git commit which represents a dpkg-source quilt patch. I would
30 like these commits to be as convenient for gbp and git-dpm users as
36 Currently when dgit needs to import a .dsc into git, it just uses
37 dpkg-source -x, and git-add. The result is a single commit where the
38 package springs into existence fully formed. This is not as good as
39 it could be. I would like to represent (in the git pseudohistory) the
40 way that the resulting tree is constructed from the input objects.
42 In particular, I would like to: represent the input tarballs as a
43 commit each (which all get merged together as if by git merge -s
44 subtree), and for quilt packages, each patch as a commit. But I want
45 to avoid (as much as possible) reimplementing the package extraction
46 algorithm in dpkg-source.
48 dpkg-source does not currently provide interfaces that look like they
49 are intended for what I want to do. And dgit wants to work with old
50 versions of dpkg, so I don't want to block on getting such interfaces
51 added (even supposing that a sane interface could be designed, which
54 So I intend to do as follows. (Please hold your nose.)
56 * dgit will untar each input tarball (other than the Debian tarball).
58 This will be done by scanning the .dsc for things whose names look
59 like (compressed) tarballs, and using the interfaces provided by
60 Dpkg::Compression to get at the tarball.
62 Each input tarball unpack will be done separately, and will be
63 followed by git-add and git-write tree, to obtain a git tree object
64 corresponding to the tarball contents.
66 That tree object will be made into a commit object with no parents.
67 (The package changelog will be searched for the earliest version
68 with the right upstream version component, and the information found
69 there used for the commit object's metadata.)
71 * dgit will then run dpkg-source -x --skip-patches.
73 Again, git plumbing will be used to make this into a tree and a
74 commit. The commit will have as parents all the tarballs previous
75 mentioned. The metadata will come from the .dsc and/or the
76 final changelog entry.
78 * dgit will look to see if the package is `3.0 (quilt)' and if so
79 whether it has a series file. (dgit already rejects packages with
80 distro-specific series files, so we need worry only about a single
81 debian/patches/series file.)
83 If there is a series file, dgit will read it into memory. It will
84 then iterate over the series file, and each time:
85 - write into its playground a series file containing one
86 more non-comment non-empty line to previously
87 - run dpkg-source --before-build (which will apply that
89 - make git tree and commit objects, using the metadata from
90 the relevant patch file to make the commit (if available)
91 - each commit object has as a parent the previous commit
92 (either the previous commit, or the commit resulting from
95 After this the series file has been completely rewritten.
97 * dgit will then run one final invocation of dpkg-source
98 --before-build. This ought not to produce any changes, but if
99 it does, they will be represented as another commit.
101 * As currently, there will be a final no-change-to-the-tree
102 pseudomerge commit which stitches the package into the relevant dgit
103 suite branch; ie something that looks as if it was made with git
106 * As currently, dgit will take steps so that none of the git trees
107 discussed above contain a .pc directory.
110 This has the following properties:
112 * Each input tarball is represented by a different commit; in usual
113 cases these commits will be the same for every upload of the same
116 * For `3.0 (quilt)' each patch's changes to the upstream files appears
117 as a single git commit (as is the effect of the debian tarball).
118 For `1.0' non-native, the effect of the diff is represented as a
119 commit. So eg `git blame' will show synthetic commits corresponding
120 to the correct parts of the input source package.
122 * It is possible to `git-cherry-pick' etc. commits representing `3.0
123 (quilt)' patches. It is even possible fish out the patch stack as
124 git branch and rebase it elsewhere etc., since the patch stack is
125 represented as a contiguous series of commits which make only the
126 relevant upstream changes.
128 * Every orig tarball in the source package is decompressed twice, but
129 disk space for only one extra copy of its unpacked contents is
130 needed. (The converse would be possible in principle but would be
131 very hard to arrange with the current interfaces provided by the
134 * No back doors into the innards of dpkg-source (nor changes to
135 dpkg-dev) are required.
137 * dgit does grow a dependency on Dpkg::Compression.
139 * Knowledge of the source format embedded in dgit is is restricted to
140 iterating over tarballs and manipulating debian/patches/series,
141 which dgit already does.
143 * dgit now depends on dpkg-source --before-build idempotently applying
144 patches as they successively appear on debian/patches/series.
146 * Perhaps the git commits generated by dgit to represent patches can
147 be made to round-trip nicely into tools like git-dpm and
150 I have found the information about tags in gbp-dch(1), but that
151 doesn't seem like it's applicable.
153 I have also found the information about tags in gbp-pq(1). From
154 that it looks like I ought to generate "Gbp-Pq: Name" and "Gbp-Pq:
157 * The scheme I describe avoids introducing a dependency from dgit to
158 git-buildpackage. I might be able to replace the
159 successive-patch-application part with an appropriate invocation of
160 gbp-pq. Would that be better ?
162 Bear in mind that because the output of gbp-pq import doesn't
163 contain debian/patches, I would need to rewrite its output (perhaps
164 with git-filter-branch).
167 Comments welcome. Please be quick - this is very close to the top of
176 Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my own.
178 If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
179 a private address which bypasses my fierce spamfilter.
181 From ijackson Wed Sep 28 10:50:49 +0100 2016
182 X-VM-v5-Data: ([nil nil nil nil t nil nil nil nil nil nil nil nil nil nil nil]
183 [nil "Wednesday" "28" "September" "2016" "10:50:49" "+0100" "Ian Jackson" "ijackson@chiark.greenend.org.uk" "<22507.37497.633622.843659@chiark.greenend.org.uk>" nil "Re: Intent to commit craziness - source package unpacking" "^From:" nil nil "9" nil nil nil nil nil nil nil nil nil nil]
185 X-Mozilla-Status: 0003
186 X-Mozilla-Status2: 00000000
188 Content-Type: text/plain; charset=iso-8859-1
189 Content-Transfer-Encoding: quoted-printable
190 Message-ID: <22507.37497.633622.843659@chiark.greenend.org.uk>
191 In-Reply-To: <20160928010117.nqe2prbsbaqkbjza@gaara.hadrons.org>
192 References: <22505.12959.668142.478444@chiark.greenend.org.uk>
193 <20160928010117.nqe2prbsbaqkbjza@gaara.hadrons.org>
194 X-Mailer: VM 8.2.0b under 24.4.1 (i586-pc-linux-gnu)
195 From: Ian Jackson <ijackson@chiark.greenend.org.uk>
196 To: Guillem Jover <guillem@debian.org>
197 Cc: debian-dpkg@lists.debian.org,
198 Guido Guenther <agx@debian.org>,
199 "Bernhard R. Link" <brlink@debian.org>,
200 vcs-pkg-discuss@lists.alioth.debian.org
201 Subject: Re: Intent to commit craziness - source package unpacking
202 Date: Wed, 28 Sep 2016 10:50:49 +0100
204 Guillem Jover writes ("Re: Intent to commit craziness - source package =
206 > On Mon, 2016-09-26 at 15:37:19 +0100, Ian Jackson wrote:
209 > > * dpkg developers, please tell me whether I am making assumptions
210 > > that are likely to become false. Particularly, on the behaviour=
212 > > successive runs of dpkg-source --before-build with successively
213 > > longer series files.
215 > For format =AB3.0 (quilt)=BB, that seems fine, to the point I'm fine =
217 > documenting this, which I can probably do for 1.18.11.
221 > For other formats, such as =AB2.0=BB, I don't think that's true, but =
223 > assume you don't care about that one anyway. But just mentioning
224 > because this behavior is probably format-specific. For =AB2.0=BB I
225 > think it could be fixed, and should not be too hard (not sure if it's=
229 I think the right approach is perhaps to use --skip-patches and
230 --before-build only with 3.0 (quilt). The that would leave 2.0 (or
231 other strange or future formats) producing a correct (although
232 possibly sub-optimal) import.
234 > > dpkg-source does not currently provide interfaces that look like th=
236 > > are intended for what I want to do. And dgit wants to work with ol=
238 > > versions of dpkg, so I don't want to block on getting such interfac=
240 > > added (even supposing that a sane interface could be designed, whic=
244 > Even then I'm still interested in a decription of what you'd need
245 > ideally, to take into account when having a pass at cleaning up that
246 > part of the interface. I think you could be interested in a cleaner
247 > Dpkg::Source::* hierarchy, for the mid/long-term?
249 For `3.0 (quilt)' explicit interfaces for applying and unapplying
250 individual patches would help. But really IMO such an interface ought
251 to be exposed on the command line rather than (or as well as) via a
254 Beyond that I find it hard to see what could make dgit's life easier.
255 Since dgit wants to construct a commit graph representing the source
256 package's innards, unless dpkg-source explicitly provides an interface
257 along those lines ("please output a graph of unpacked source tree
258 states and corresponding commit messages") dgit is still going to have
259 to know specially about most of the source package formats.
261 > > * dgit will untar each input tarball (other than the Debian tarball=
264 > > This will be done by scanning the .dsc for things whose names loo=
266 > > like (compressed) tarballs, and using the interfaces provided by
267 > > Dpkg::Compression to get at the tarball.
269 > Hmm, Dpkg::Source::Archive is currently private, but I might have a
270 > look at making it public if that would be helpful here.
272 I think the amount of logic I would have to replicate is minimal.
274 > > * As currently, dgit will take steps so that none of the git trees
275 > > discussed above contain a .pc directory.
277 > As long as the directory does not disappear from the working tree,
280 Right, indeed it won't.
282 Thanks for your comments. I feel unblocked :-).
287 Ian Jackson <ijackson@chiark.greenend.org.uk> These opinions are my o=
290 If I emailed you from an address @fyvzl.net or @evade.org.uk, that is
291 a private address which bypasses my fierce spamfilter.