A.D.A. Amiga Demoscene Archive

open up the source, release the Mc680x0 executable ;)

Don't worry, I will. In that order. ;)

CALL FOR TEST DATA

To benchmark and tune the cruncher, I need all of you to send me uncompressed versions of your released (and/or unreleased) intros - 4k, 40k, 64k, whatever.

For extra appreciation, include information about which cruncher you used originally, and what it crunched down to. But if that is too much bother, just send the intros - that's the important part.

(my handle) at loonies dot dk

Thanks in advance! :-D

I'll send you a zip package today.

Aske, I have just sent you a larger zip file - let me know, if you haven't received it in your loonies mail. I have changed the .zip into a .zi_ file, but maybe it can still get stuck in a filter, dunno.

Happy new year everybody!

I am very happy to present: Shrinkler 4.0! :-D

So, what happened to version 3, you might ask...?

Version 3 was an internal ad-hoc cruncher used for a number of 4k intros, starting with Rapo Diablo. Version 4 is basically the hunk processing code from version 2 combined with (a much further developed version of) the compression algorithm from version 3.

My tests (big thanks to Corial for a comprehensive test suite) indicate that it wins around 8-10% compared to ExeCruncher2 in the typical case. It is also much, much slower - with default options, it is typically around 20-30 times slower. Run it on a 100 times faster computer and you'll be fine. ;)

Like its predecessor, Shrinkler has a number of options controlling the compression, though in contrast to ExeCruncher2, higher values (almost) always result in better compression. It also retains the special mini mode for 4k intros.

For those of you interested in the technology behind it all, the source code is available at Bitbucket. For an overview of the compression algorithm, read LZEncoder.h.

Let me know how it works for you. And most important of all: Enjoy, and make some cool intros! :-D

Awesome! I'll keep running it on the real hardware and just get drunk while it does its magic :D

Very cool. Great work!

Cool work,Aske.Really looking forward to try it out.

Outstanding work and überthanks for your endless effort!
I'll give it a try during the "intro making session" this weekend :)

Blueberry, it doesn't produce an executable that can run.

I did not have ixemul.library but I found version 63 (or was it 68?) through Google.

I run Shrinkler like this:
shrinkler myintro.obj myintro.exe

When I try to run myintro.exe (in Shell) it tells me that myintro.exe: file is not executable

This is not the case if I use Execruncher2.

Could you please investigate? On a sidenote: Execruncher2 produced a filesize of 63436, Shrinkler produced a filesize of 59484. That's awesome!

Corial, it worked out of the box for me when trying to compress an old Amiga intro of mine on my Win32 PC (and it won more than 10k over cru (!), my former #1 choice). Was "myintro.obj" a valid executable?

Noname,ok that's odd. My object file is however valid,but I'll try to build a new one and let shrinkler do its shrinkling business. I also didn't reset my emulator after having put the ix library in the emulator's libs folder. Will give it a try.

Ok, something is completely wrong. Before running the new tests, I downloaded ixemul in version 48 (which shrinkler says it the minimum) and chose to use "ixemul-040.library", which I renamed to "ixemul.library" (stated in the readme.txt) and put into my libs folder of my Cloanto Amiga Forever installation.

Tried another valid object file: same result. Tried one of the object files, which I provided as testinput for Blueberry and during the shrinkler run it halts and gives a:
"terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc"

Execruncher2, however, crunches the file nicely and produces an executable.

I'll contact Blueberry on mail....

The version I'm using is:
(from the Amiga Shrinkler folder)

05-01-2014 21:58. Filesize = 119kb (read via Windows Explorer)

There seems to be an issue with the Amiga version that it sometimes does not set the executable bit on the produced executable. Try to run
on the output file from the crunching.

As for the bad_alloc error message, it means Shrinkler ran out of memory. I just tried crunching the same file and it gobbled up 40MB of memory, so if you don't have that much on your Amiga, it explains it.

Excessive memory usage and lack of proper memory handling (giving an understandable "out of memory" error) are some of the known "rough edges" of the first version. Your report is actually sort of good news, as I had my fears it would simply crash the machine if it tried to allocate too much memory. :)

I will look into these issues for the next version. Until then, a workaround for the memory usage is to split up your big sections into smaller sections, since the memory usage is proportional to the largest section (as it compresses one section at a time).

Thanks,Aske - will try setting the bit later today. And yes,I don't really have that many sections in the code,thus they tend to get big.Thanks for the help.

If you split up the sections in such a way that you put similar data into the same section, it might actually improve compression in some cases.

And just to round this one off. In Shell, the solution is (as Blueberry suggests) to type "protect myIntro.exe +e" after having crunched with Shrinkler.

For you Mac users (like me) i have added Shrinkler to my brew tap:

https://github.com/tditlu/homebrew-amiga/

I finally got to test the Shrinkler4.0 on the current assembled project.
The project assembles to about 35k.

Ye olde reliable Stonecracker gave me, after including its library: ~13k

Execruncher2.1 crunched it down to ~11k

Shrinkler obliterated it to a whopping ~10k

All tested with default settings. Nevertheless, the decrunch performance of Stonecracker on the A500 could not be matched.
Since I'm a dud in packer-things, are there some options in Shrinkler that are especially useful in terms of 64k intros ?

If you want Shrinkler to crunch harder, the best place to start is usually to raise the skip length (-s option) and add another iteration (-i 3).

To push it even harder, increase the -a and -c values as well.

Finally, -l is good for cramming out the last few bytes, but can also be quite costly.

The -r option only makes a difference if the "References discarded" metric printed by Shrinkler is larger than 0, and raising it increases the memory requirements during crunching proportionally.


The decompression speed is not affected by any options, and there is not much to do about it for this level of compression (and compact decompression code). I measured the decrunch performance using The Tetris Intro (40k), which decrunched in 14 seconds, so this is well within my goal of less than 30 seconds for a 64k intro. :)

But, speaking of decrunching, one of the features that I am contemplating (based on discussions here and there) is to have some way for the decruncher to tell that it is decrunching. This could be by printing a text, like TITANICS-Cruncher's "jeg er en ged, Og jeg vil d0...", or the traditional flashing of the mouse cursor (or some other hardware register of your preference).

Any interest in such a feature?

Totally :)
Especially on the A500 there should be a visible sign that the program is indeed doing something.

To celebrate that DATASTORM is just one week away, here is a new version of the cruncher, with bug fixes and new features!

Download Shrinkler 4.1 now! :-D

The output file is now marked executable, so you don't have to run the protect command manually.

And the cryptic bad_alloc error message is now replaced by a more friendly out-of-memory message.

I fixed a couple of nasty bugs in the range coder which could result in corrupt output files in some rare (and not so rare) cases. Big thanks to Axis for providing test files which triggered the bugs.

Motivated by these bugs, I added a verification pass to the compression process. If something goes wrong, the cruncher will now abort with an internal error instead of just giving you a bad output file. This should improve the reliability of the cruncher tremendously.

Finally, I added the decrunching feedback options I mentioned in my last post:

You can specify a text to be embedded into the intro and printed prior to decrunching, TITANICS-Cruncher style. You can also include the text from a file - let your ASCII artist go crazy! ;)

The text embedding option costs 56 bytes plus the size of the text itself.

And, for the classic decrunching feeling, you can also ask the decruncher to bang a particular hardware register with "random" values during decrunching. DFF180, DFF182 and DFF1A2 are classic choices. DFF102 and DFF090 also give some interesting effects. DFF100, DFF108 and DFF140 are for the more chaotically minded. The suicidal use DFF058.

The option costs 4 bytes, and is available both in the normal and mini crunching modes.

Enjoy! :-D

Thumbs up for decrunching text and colour effects alone :)

Fabulous,simply have to try that "Titanics" feature :)

Awesome. Many thanks for this Blueberry. (^_^)

Blueberry, very interesting topic. Pack ratio looks very promising. Would be possible to add compression for data files and provide decruncher for 68k in asm?

Sometimes ago I did a small comparison for different data depackers:
http://www.atari-forum.com/viewtopic.php?f=68&t=26825&p=257375#p257158
Depackers for PFT, LZ4, LZ77, LZ78, ARJ4 are in 68000 and are max 176 bytes long (fits into 030 cache).
Original file size was 32034 bytes.

After compression:
Packer | Size | Pack Ratio
LZ78 | 31804 | 99.3%
LZ77 | 30103 | 94.0%
LZ4 | 28527 | 89.1%
ARJ4 | 28095 | 87.7%
PFT | 26267 | 82.0%
ARJ7 | 25145 | 78.5%

Depack Time:
Packer | Sec.Milisec | vs LZ77 | vs PFT
LZ78 | 1.348 | 1078.4% | 22.5%
LZ77 | 0.125 | 100.0% | 2.1%
LZ4 | 0.135 | 108.0% | 2.3%
ARJ4 | 0.751 | 600.8% | 12.5%
PFT | 5.988 | 4790.4% | 100.0%

I've just checked Shrinkler (SHR) vs other packers. Original file was 66328 bytes size.

Packer | Size | Pack Ratio
SHR | 3832 | 5.8%
PFL | 3663 | 5.5%
ARJ4 | 5276 | 8.0%
ARJ7 | 4433 | 6.7%
LZ78 | 6571 | 9.9%
LZ77 | 12480 | 18.8%


Shrinkler has really good compression ratio

Interesting comparison.

I hadn't heard of PackFire before (I'm not much into Atari stuff). Its LZMA-derived packer seems to compress somewhat better that Shrinkler, but the decompression code is also much larger (680 bytes versus Shrinkler's 180).

What is included in each of the numbers? Is this the output file size (including decrunch header) or just the compressed data? Which compression options did you use for Shrinkler?

It is probably on the slow side for use as a data compressor for most purposes. On an A500, it can decompress around 2k/s of compressed data. Then again, if speed is not an issue...

If you want to have a go at the source code, the packData function in Pack.h is the one you want. And then just chop away all the hunk and reloc stuff from Header.S ;)