site: Make return value of transforms be an enum

We are going to need to distinguish more cases.  It was always bad to
have these hardcoded values.

transform_apply_seqrange is, right now, returned even when the problem
is that the packet is recent but is a duplicate.  This is wrong.

No functional change.

Signed-off-by: Ian Jackson <ijackson@chiark.greenend.org.uk>