A.D.A. Amiga Demoscene Archive

  Welcome guest! Please register a new account or log in

  

  

  

Demos Amiga Demoscene Archive Forum / Coding / My Amiga Cruncher
 Page:  ««  1  2  3  4  5  »» 
Author Message
sp_
Member
#1 - Posted: 21 Dec 2013 04:51
Reply Quote
open up the source, release the Mc680x0 executable ;)
Blueberry
Member
#2 - Posted: 22 Dec 2013 00:50
Reply Quote
Don't worry, I will. In that order. ;)
Blueberry
Member
#3 - Posted: 26 Dec 2013 23:59 - Edited
Reply Quote
CALL FOR TEST DATA

To benchmark and tune the cruncher, I need all of you to send me uncompressed versions of your released (and/or unreleased) intros - 4k, 40k, 64k, whatever.

For extra appreciation, include information about which cruncher you used originally, and what it crunched down to. But if that is too much bother, just send the intros - that's the important part.

(my handle) at loonies dot dk

Thanks in advance! :-D
Corial
Member
#4 - Posted: 27 Dec 2013 07:44
Reply Quote
I'll send you a zip package today.
Corial
Member
#5 - Posted: 27 Dec 2013 10:03
Reply Quote
Aske, I have just sent you a larger zip file - let me know, if you haven't received it in your loonies mail. I have changed the .zip into a .zi_ file, but maybe it can still get stuck in a filter, dunno.
Blueberry
Member
#6 - Posted: 5 Jan 2014 21:14
Reply Quote
Happy new year everybody!

I am very happy to present: Shrinkler 4.0! :-D

So, what happened to version 3, you might ask...?

Version 3 was an internal ad-hoc cruncher used for a number of 4k intros, starting with Rapo Diablo. Version 4 is basically the hunk processing code from version 2 combined with (a much further developed version of) the compression algorithm from version 3.

My tests (big thanks to Corial for a comprehensive test suite) indicate that it wins around 8-10% compared to ExeCruncher2 in the typical case. It is also much, much slower - with default options, it is typically around 20-30 times slower. Run it on a 100 times faster computer and you'll be fine. ;)

Like its predecessor, Shrinkler has a number of options controlling the compression, though in contrast to ExeCruncher2, higher values (almost) always result in better compression. It also retains the special mini mode for 4k intros.

For those of you interested in the technology behind it all, the source code is available at Bitbucket. For an overview of the compression algorithm, read LZEncoder.h.

Let me know how it works for you. And most important of all: Enjoy, and make some cool intros! :-D
bstrr
Member
#7 - Posted: 5 Jan 2014 21:29
Reply Quote
Awesome! I'll keep running it on the real hardware and just get drunk while it does its magic :D
nemas
Member
#8 - Posted: 5 Jan 2014 22:00
Reply Quote
Very cool. Great work!
Corial
Member
#9 - Posted: 5 Jan 2014 23:07
Reply Quote
Cool work,Aske.Really looking forward to try it out.
d0DgE
Member
#10 - Posted: 8 Jan 2014 09:05
Reply Quote
Outstanding work and überthanks for your endless effort!
I'll give it a try during the "intro making session" this weekend :)
Corial
Member
#11 - Posted: 11 Jan 2014 21:30
Reply Quote
Blueberry, it doesn't produce an executable that can run.

I did not have ixemul.library but I found version 63 (or was it 68?) through Google.

I run Shrinkler like this:
shrinkler myintro.obj myintro.exe

When I try to run myintro.exe (in Shell) it tells me that myintro.exe: file is not executable

This is not the case if I use Execruncher2.

Could you please investigate? On a sidenote: Execruncher2 produced a filesize of 63436, Shrinkler produced a filesize of 59484. That's awesome!
noname
Member
#12 - Posted: 12 Jan 2014 00:28
Reply Quote
Corial, it worked out of the box for me when trying to compress an old Amiga intro of mine on my Win32 PC (and it won more than 10k over cru (!), my former #1 choice). Was "myintro.obj" a valid executable?
Corial
Member
#13 - Posted: 12 Jan 2014 07:44
Reply Quote
Noname,ok that's odd. My object file is however valid,but I'll try to build a new one and let shrinkler do its shrinkling business. I also didn't reset my emulator after having put the ix library in the emulator's libs folder. Will give it a try.
Corial
Member
#14 - Posted: 12 Jan 2014 09:07
Reply Quote
Ok, something is completely wrong. Before running the new tests, I downloaded ixemul in version 48 (which shrinkler says it the minimum) and chose to use "ixemul-040.library", which I renamed to "ixemul.library" (stated in the readme.txt) and put into my libs folder of my Cloanto Amiga Forever installation.

Tried another valid object file: same result. Tried one of the object files, which I provided as testinput for Blueberry and during the shrinkler run it halts and gives a:
"terminate called after throwing an instance of 'std::bad_alloc' what(): std::bad_alloc"

Execruncher2, however, crunches the file nicely and produces an executable.

I'll contact Blueberry on mail....
Corial
Member
#15 - Posted: 12 Jan 2014 09:20
Reply Quote
The version I'm using is:
(from the Amiga Shrinkler folder)

05-01-2014 21:58. Filesize = 119kb (read via Windows Explorer)
Blueberry
Member
#16 - Posted: 12 Jan 2014 09:55 - Edited
Reply Quote
There seems to be an issue with the Amiga version that it sometimes does not set the executable bit on the produced executable. Try to run
protect file +e
on the output file from the crunching.

As for the bad_alloc error message, it means Shrinkler ran out of memory. I just tried crunching the same file and it gobbled up 40MB of memory, so if you don't have that much on your Amiga, it explains it.

Excessive memory usage and lack of proper memory handling (giving an understandable "out of memory" error) are some of the known "rough edges" of the first version. Your report is actually sort of good news, as I had my fears it would simply crash the machine if it tried to allocate too much memory. :)

I will look into these issues for the next version. Until then, a workaround for the memory usage is to split up your big sections into smaller sections, since the memory usage is proportional to the largest section (as it compresses one section at a time).
Corial
Member
#17 - Posted: 12 Jan 2014 10:32
Reply Quote
Thanks,Aske - will try setting the bit later today. And yes,I don't really have that many sections in the code,thus they tend to get big.Thanks for the help.
Blueberry
Member
#18 - Posted: 12 Jan 2014 11:11
Reply Quote
If you split up the sections in such a way that you put similar data into the same section, it might actually improve compression in some cases.
Corial
Member
#19 - Posted: 12 Jan 2014 15:23
Reply Quote
And just to round this one off. In Shell, the solution is (as Blueberry suggests) to type "protect myIntro.exe +e" after having crunched with Shrinkler.
Todi
Member
#20 - Posted: 13 Jan 2014 20:56
Reply Quote
For you Mac users (like me) i have added Shrinkler to my brew tap:

https://github.com/tditlu/homebrew-amiga/
d0DgE
Member
#21 - Posted: 20 Jan 2014 08:47
Reply Quote
I finally got to test the Shrinkler4.0 on the current assembled project.
The project assembles to about 35k.

Ye olde reliable Stonecracker gave me, after including its library: ~13k

Execruncher2.1 crunched it down to ~11k

Shrinkler obliterated it to a whopping ~10k

All tested with default settings. Nevertheless, the decrunch performance of Stonecracker on the A500 could not be matched.
Since I'm a dud in packer-things, are there some options in Shrinkler that are especially useful in terms of 64k intros ?
Blueberry
Member
#22 - Posted: 20 Jan 2014 15:21
Reply Quote
If you want Shrinkler to crunch harder, the best place to start is usually to raise the skip length (-s option) and add another iteration (-i 3).

To push it even harder, increase the -a and -c values as well.

Finally, -l is good for cramming out the last few bytes, but can also be quite costly.

The -r option only makes a difference if the "References discarded" metric printed by Shrinkler is larger than 0, and raising it increases the memory requirements during crunching proportionally.


The decompression speed is not affected by any options, and there is not much to do about it for this level of compression (and compact decompression code). I measured the decrunch performance using The Tetris Intro (40k), which decrunched in 14 seconds, so this is well within my goal of less than 30 seconds for a 64k intro. :)

But, speaking of decrunching, one of the features that I am contemplating (based on discussions here and there) is to have some way for the decruncher to tell that it is decrunching. This could be by printing a text, like TITANICS-Cruncher's "jeg er en ged, Og jeg vil d0...", or the traditional flashing of the mouse cursor (or some other hardware register of your preference).

Any interest in such a feature?
d0DgE
Member
#23 - Posted: 20 Jan 2014 15:50
Reply Quote
Blueberry:
Any interest in such a feature?

Totally :)
Especially on the A500 there should be a visible sign that the program is indeed doing something.
Blueberry
Member
#24 - Posted: 8 Feb 2014 15:27
Reply Quote
To celebrate that DATASTORM is just one week away, here is a new version of the cruncher, with bug fixes and new features!

Download Shrinkler 4.1 now! :-D

The output file is now marked executable, so you don't have to run the protect command manually.

And the cryptic bad_alloc error message is now replaced by a more friendly out-of-memory message.

I fixed a couple of nasty bugs in the range coder which could result in corrupt output files in some rare (and not so rare) cases. Big thanks to Axis for providing test files which triggered the bugs.

Motivated by these bugs, I added a verification pass to the compression process. If something goes wrong, the cruncher will now abort with an internal error instead of just giving you a bad output file. This should improve the reliability of the cruncher tremendously.

Finally, I added the decrunching feedback options I mentioned in my last post:

You can specify a text to be embedded into the intro and printed prior to decrunching, TITANICS-Cruncher style. You can also include the text from a file - let your ASCII artist go crazy! ;)

The text embedding option costs 56 bytes plus the size of the text itself.

And, for the classic decrunching feeling, you can also ask the decruncher to bang a particular hardware register with "random" values during decrunching. DFF180, DFF182 and DFF1A2 are classic choices. DFF102 and DFF090 also give some interesting effects. DFF100, DFF108 and DFF140 are for the more chaotically minded. The suicidal use DFF058.

The option costs 4 bytes, and is available both in the normal and mini crunching modes.

Enjoy! :-D
noname
Member
#25 - Posted: 8 Feb 2014 15:35
Reply Quote
Thumbs up for decrunching text and colour effects alone :)
Corial
Member
#26 - Posted: 9 Feb 2014 07:35
Reply Quote
Fabulous,simply have to try that "Titanics" feature :)
Lonewolf10
Member
#27 - Posted: 3 Apr 2014 22:50
Reply Quote

Awesome. Many thanks for this Blueberry. (^_^)
Cyprian
Member
#28 - Posted: 29 Aug 2014 15:12
Reply Quote
Blueberry, very interesting topic. Pack ratio looks very promising. Would be possible to add compression for data files and provide decruncher for 68k in asm?

Sometimes ago I did a small comparison for different data depackers:
http://www.atari-forum.com/viewtopic.php?f=68&t=26825&p=257375#p257158
Depackers for PFT, LZ4, LZ77, LZ78, ARJ4 are in 68000 and are max 176 bytes long (fits into 030 cache).
Original file size was 32034 bytes.

After compression:
Packer | Size | Pack Ratio
LZ78 | 31804 | 99.3%
LZ77 | 30103 | 94.0%
LZ4 | 28527 | 89.1%
ARJ4 | 28095 | 87.7%
PFT | 26267 | 82.0%
ARJ7 | 25145 | 78.5%

Depack Time:
Packer | Sec.Milisec | vs LZ77 | vs PFT
LZ78 | 1.348 | 1078.4% | 22.5%
LZ77 | 0.125 | 100.0% | 2.1%
LZ4 | 0.135 | 108.0% | 2.3%
ARJ4 | 0.751 | 600.8% | 12.5%
PFT | 5.988 | 4790.4% | 100.0%
Cyprian
Member
#29 - Posted: 29 Aug 2014 15:14
Reply Quote
I've just checked Shrinkler (SHR) vs other packers. Original file was 66328 bytes size.

Packer | Size | Pack Ratio
SHR | 3832 | 5.8%
PFL | 3663 | 5.5%
ARJ4 | 5276 | 8.0%
ARJ7 | 4433 | 6.7%
LZ78 | 6571 | 9.9%
LZ77 | 12480 | 18.8%


Shrinkler has really good compression ratio
Blueberry
Member
#30 - Posted: 31 Aug 2014 00:38
Reply Quote
Interesting comparison.

I hadn't heard of PackFire before (I'm not much into Atari stuff). Its LZMA-derived packer seems to compress somewhat better that Shrinkler, but the decompression code is also much larger (680 bytes versus Shrinkler's 180).

What is included in each of the numbers? Is this the output file size (including decrunch header) or just the compressed data? Which compression options did you use for Shrinkler?

It is probably on the slow side for use as a data compressor for most purposes. On an A500, it can decompress around 2k/s of compressed data. Then again, if speed is not an issue...

If you want to have a go at the source code, the packData function in Pack.h is the one you want. And then just chop away all the hunk and reloc stuff from Header.S ;)
 Page:  ««  1  2  3  4  5  »» 

  Please register a new account or log in to comment

  

  

  

 

A.D.A. Amiga Demoscene Archive, Version 3.0