You could try
zlib compression, for example using the
miniz implementation. The miniz decompressor (tinfl.c) seems to be sufficiently self-contained that it should not be a problem to compile it for 68000. As for in-place decompression, I am not sure, but my guess is that it will not be a problem if you give it some margin.