A.D.A. Amiga Demoscene Archive

Sounds like a job for a script of some kind. Maybe it's time to take up AREXX again. It's been ages since I touched that. :)

I have a new version of the cruncher in the works (first used for Rapo Diablo) which is self-tuning, i.e. all of its explicit parameters are for size/compression speed tradeoff. It is not quite stable yet and only exists in a proof-of-concept PC version, so there is no knowing when, if ever, it will be properly done. So better count on using ExeCruncher2 for now. I have a 4k to make as well, you know. ;)

I got an idea to improve lz compression on small files by using the ROM. I have tested several WB 3.0 - 3.5 ROM files and found that they share the same bytes with various lengths at various offsets. To make sure this will work I need to write a small program that searh for equal bytes in all available ROMfiles. When I have this I can start to copy the longest matching sequences down to a memoryblock. Then a lzcompression algorithm can search trough these bytes to improve compression. The decompressor needs to copy the same bytes from the rom before decompressing.

Lz won't crunch very in the beginning of files, but with this teqnique I think 4k intros can be shrinked more.¨

The ROM file consist mostly of code and some data and this is perfect for finding lz matches in a 4k intro.

I have a collection of 21 ROM files Kickstart v3.0 - kickstart 3.5 I downloaded from the internet.

What do you think?
Is it worth the effort?

It will only pay off if there are actually significant sequences of code in the ROM which are identical to code sequences in the intro. This is unlikely to happen by accident, so in order for this to have any benefit, the coder will have to be familiar with the available code snippets and use them conciously.

Even then, I doubt there is much to be gained by integrating this directly into the LZ compressor. The extra complexity will take up a fair amount of space in the decompression header, and the possibility of selecting from ROM will increase the size of the compressed data as well.

But it might certainly be useful as a preprocessing pass before compression. In other words, precede the code with a small piece of code which unfolds a list of ROM offsets, code offsets and lengths into the code. One advantage of this is that the unfolding code will also be compressed.

So yes, I think it is worth the effort. It is definitely an interesting idea to try out. Some really useful pieces might turn up - who knows? And if some of them end in RTS they are even easier to use. :)

@Blueberry:

I've got a Guru report for running an executable crunched with the packer on an OCS system. Dunno why, some weeks ago the stuff worked.

The ID the 500 gave me after refusing to start the program (Task held while loading) was: #00000005.00C091E0

I couldn't find a suitable reference to this in the Guru book.

After a short research in ze intarwebz I found this site where a #00000005.00XXXXXX guru was classified as follows:


If you get a Guru Meditation number like #00000005.00C13870 it means: A program, at the address C13870, tried to devide a value by zero. Since you now know what the error was you only need to look at the places where your program can be forced to divide a value by zero, and it should not take to long time to find the bug.


Curious that the same program (packed at a lower efficiency) with stonecracker runs flawlessly on the 500.
On the big machine (the 1260) it also runs fine.

Can I circumvent this occasion with one of these options ?

dODgE: Are you using the MINI option perhaps? Executables compressed using this option require Kickstart 2.0 or newer, since they finish up by calling CacheClearU. The normal (non-MINI) mode checks for the kickstart version and only calls CacheClearU if it is supported by the kickstart.

No options used. Only the blank program call with the source prog and destination name.
(your 4k-Jedi trick didn't work on me :D )
I can remember one thing I changed between the testing periods.
I replaced a bss range entry with a constant that is declared before the startup e.g.
<pre>
SIZE equ 4096

...blabla...lame code...

section chipdata,bss_c
copperbuffer: ds.w SIZE
</pre>

Perhaps the glitch lies somewhere there ?
Before the buffer was declared directly in the copperlist like:
<pre>
copperlist:
...pretty much usual copper commands...
copprg: dcb.w 4096
dc.w $ffff,$fffe
dc.w $ffff,$fffe
</pre>
Now there is only the reference (dc.l copperbuffer) at that label in the listing.

Hmm, that shouldn't change anything. And it certainly shouldn't make a difference between 500 and 1260.

If you send me the (uncompressed) executable you are trying to compress, I can take a look at it.

(handle) at loonies dot dk

done...kthx in advance.

Perhaps Stingray didn't report the bug i found a long time ago in your depacker but there is or was a 68020 instruction in it; A 16 bit displacement at offset 0x12a in packed executables (normal depacker): lea (pc,d0.w,$0072.w),a5 using a 8 bit one will make it work on a 68000 based machines and it'll get shorter by a grand 2 bytes.

dODgE: The company hosting our domain (needhost) went bankrupt and closed everything down, so I haven't received your mail. Please send it again. :)

hitchhikr, you are absolutely right. Thanks a lot for pointing this out.

While developing the decompression header code, I have worked inside AsmOne/AsmPro, but for the actual cruncher, the code is assembled using the SAS/C assembler. It turns out that this assembler, when told to generate code for 68060, changes x(pc,d0.w) into (pc,d0.w,x.w).

All I had to do to fix it was to set SAS/C to compile for 68000. Note that this does not mean that the cruncher itself will run on 68000, since there are some parts of it that are written in assembler using some 68020+ instructions.

I have uploaded a new version of the cruncher that generates executables which should work on 68000.

for the record I'd like to add, that the new version indeed works properly on the real 68000 thingie... kthnx for the fix

@blueberry
can your cruncher works with large files which done by vbcc or sasc ? I just try last version, and:

Not knowing anything about the cruncher at hands, it seems like it doesn't support symbol hunks (03f0 in the SAS/C log). Symbol hunks are usually removed by crunchers, so you could try crunching/decrunching the SASC file with another cruncher and then retry with Blueberry's cruncher.

Indeed the Cruncher does not support SYMBOL hunks. It skips UNIT, NAME and DEBUG hunks, and ought to do the same for SYMBOL and EXT hunks. Presumably (I wrote that code more than 10 years ago so I don't remember clearly), I didn't have sufficient documentation on SYMBOL and EXT hunks to know how to skip them. Unlike most other hunk types, they do not contain a size specifier at the start.

I looked around for some documentation just now and found this very nice document which others here might find interesting as well: Amiga Binary File Structure

This means I should be able to make the cruncher remove SYMBOL and EXT hunks. Is this a desirable feature? ;)

kasie: Can you send me the files you tried to compress so I have something to test on? Preferably something that gives an easily recognizable result when run so I can see if I break anything.

blueberry, yes, sure it desirable :) i just tryed tons of crunchers, most of them provide bins which work over os3.x, but not works on os4.x for example (or mos). Just try to found somethink stable which will packed file very good to reduce some size :)

On which email i can send test stuff ?

@Blueberry:
Thanks for the ABFS pdf link. I ripped the arexx script. :D

kasie: You can reach me at (my handle) at loonies dot dk.

Blueberry: Did you make any progress in adding the remove SYMBOL and EXT hunks while crunching? I would appreciate such a feature as well. :)

Currently I do use stc to crunch and decrunch and then ExeCruncher2 to crunch the better way. ;)

As an alternative, is there some tool like strip to remove the unnecessary hunks from an executable or is there a way to tell vasmm68k_mot -Fhunkexe to don't even add them while assembling?

with -nosym option vasm doesn't write HUNK_SYMBOL debugging symbols to final executable.

Any chance you turning this into a data cruncher?

endo: Oh, great, that does the trick! Now that i know what it is called i found it in the albeit very good and detailed vasm manual. Thanks. :)

Hi there!

I have released a new version of my Cruncher with some minor enhancements, mainly to make it easier to target 68000 / KS1.3 machines. Full list of new features:

- The MINI header checks the exec version and only calls the CacheClearU function if available. This makes it possible to use MINI mode for KS1.3 intros. Useful for your oldskool 4k intro for Revision. ;)
- A new option - NOKSCHECK - omits the Kickstart check from the header. This saves 8 bytes, at the cost of only working on KS2.04 or newer.
- The cruncher itself now runs on all CPUs. It still requires Kickstart 2.0 or newer, though (because it relies on ReadArgs for argument passing). If there is desire for a KS1.3 compatible version, I can work on rewriting the argument passing to avoid the use of ReadArgs.
- As per popular request, the hunk processor now properly skips SYMBOL hunks. EXT hunks are still unsupported, but it is my impression that these are unlikely to be found in executable files.
- The text output has been reformatted to a more compact and readable layout.

The download link is the same as always.

Bug reports, requests and other comments are more than welcome! :-D

Hi everybody!

I am working on a new version of my cruncher, and thought I would put out a teaser here, so you have something to look forward to. :)

Planned features include:
- Better compression. I know a lot more about data compression now than I did back when I designed the compression algorithm for the current ExeCruncher. The new algorithm draws inspiration from many sources, including Crinkler and LZMA. I have been experimenting a lot with the format and found what seems to be a good sweet spot between compression efficiency and decompression code size.
- Faster decompression, though still much slower than most Amiga crunchers. My goal for the speed is something like half a minute to decompress a 64k intro on an A500.
- Cruncher executables for multiple platforms. No reason not to use all that CPU power that you have on your desk anyway.
- Open source. Nothing is up yet, but I will probably open up the development some time before the first fully functional version for all the really curious ones out there. I'll post a link here when that happens.
- Maybe a proper name. I am contemplating "Shrinkler" to sort of stay in my usual style of names. :)
- No more arcane options. The cruncher will do the hard work for you.

Of course, there will be some drawbacks. The following are some expected shortcomings compared to the current version:
- Slower compression. The cruncher will think a lot harder about how to make the file small. It will also use smarter data structures to get the job done, so there will generally be more bang for the buck. Anyway, the compression speed should really only be an issue if you are running the cruncher on a real Amiga.
- Very memory hungry compression. Crunching a 4k intro on a machine with 1MB of memory will be a stretch (not to mention that it will take ages), and a 64k will be out of the question. The decompressor will only need a few kilobytes of memory (plus the compressed data) on top of the memory for the executable, and both of these will be deallocated right after decompression (as does the current version).
- No more saving of all registers in the decompression header. It will of course save A0 and D0, so commandline arguments work.
- No more arcane options. I know some people consider this a feature. :) Well, there will be options, don't worry, but they be more of the "compressed size vs. compression time/memory" kind.

Anyway... All kinds of feedback on any of this information is more than welcome. There is still a lot of work to do, so the new version will not be ready for FAP, but it will likely be ready for DATASTORM.

Sad that crunching a 64k is out of the question,but luckily there's still the traditional execruncher to use.Good luck with the project,Aske.

It is only crunching the intro on the machine itself which is out of the question. You can still move the executable to a different machine (a bigger Amiga or a PC) and crunch it there. The resulting exe will run on the 1MB machine just fine (the decruncher only has a small memory overhead).

This is actually an issue that I am very interested in feedback on: Is it a desirable feature for the cruncher itself to run on a low-end Amiga? If it is, I will look into having a special low-memory mode, which will run in a reasonable amount of time (say, less than an hour for a 64k) at the expense of the compression ratio.

This sounds really nice. For me personally it isn't really important that the decruncher works on a low-end Amiga.

Good luck with the project and keep us updated.

Obviously it's important that the *de*cruncher works on low-end machines, but that already seems to be a given from Blueberry's description. :)

As for the ability to crunch on a low-end machine I don't care at all (and even if there had been a custom version for that I wouldn't have used it if it compromised crunch ratio). Cross-crunching all the way! :)

crunching a project on, say, an A500 1mb is ... masochism it even was back in the days, wasn't it ?

It is indeed. But sometimes the borderline between masochism and retro demo coding is a bit blurry. :)