util: rework cunescape(), improve error handling

util: rework cunescape(), improve error handling

Change cunescape() to return a normal error code, so that we can
distuingish OOM errors from parse errors.

This also adds a flags parameter to control whether "relaxed" or normal
parsing shall be done. If set no parse failures are generated, and the
only reason why cunescape() can fail is OOM.