site.c: Cope with failure of transform `setkey' method.

The `setkey' method can fail, and indicates this by returning False.
Indeed, the `serpent-cbc256' transform will fail if the shared secret
it's given is too short.

Change `set_new_transform' and its callers to propagate failures
properly.

Signed-off-by: Mark Wooding <mdw@distorted.org.uk>