| Author |
Message |
Cyprian
Member |
PackFire has two variants Large and Tiny, the second one is only 138 bytes long.
I did another test for Amiga executable file 4628bytes size . Column 'Size' contains figures for packed data only. And actually those compressors are only "data packers".
Packer | Size | Ratio
PFL | 4022 | 86.9%
SHR mini | 4212 | 91.0%
ARJ7 | 4237 | 91.6%
PFT | 4239 | 91.6%
SHR | 4296 | 92.8%
LZ4 | 4590 | 99.2%
ARJ4 | 4733 | 102.3%
LZ77 | 4978 | 107.6%
LZ78 | 5485 | 118.5%
SHR" was packed without any parameters and "SHR mini" with "--mini". Really good figures, and would be better with excluded depacker. Would be cool to have option for packing TOS executable
btw. under Win7 Shrinkler shows a lot of debug messages: "[0.0%]←[6D[1.0%]←[6D[1.1%]←[6D[1.2%]←[6D[1.3%]←[6D[1.4%]←[6D[1.5%]←[6D[1.6%]←[6D[1.7%]←[6D[1.8%]←[6D[1.9%]←[6D[2.0"
|
Blueberry
Member |
Those are not debug messages - they are progress output. Use a terminal which understands ANSI escape codes. ;)
Do the PFL and PFT numbers include the depackers?
|
Cyprian
Member |
hehe, you right
no, they don't include the depacker
|
Blueberry
Member |
Wow, given that Shrinkler mini has a total overhead (hunk structure + depacker code) of 228 bytes, this means that it actually compresses slightly better than PFL (even when taking into account that the PFL size includes the original hunk structure), with a depacker less than a third of the size. :-D
You are not the first one to express interest in an Atari port (Frequent asked for one when I talked to him at Revision this year). I don't know anything about the Atari TOS executable format, but I will happily answer any questions about the source if someone wants to attempt a port.
Data file compression should be easy to add. I will put it on my to-do list.
|
Blueberry
Member |
Christmas has arrived early this year. Shrinkler 4.2 is here! From conversations with various people, and from snooping around in online cruncher discussions, I have gathered that the memory overhead of Shrinkler could be an issue to some. In its normal crunching mode, the Shrinkler decruncher uses memory equal to the statically allocated memory (including uncompressed code, data and bss) plus the decrunch code plus the compressed size. When decrunching is completed, the decrunch code and compressed data are deallocated. The new version adds an option to overlap the compressed and uncompressed data in memory, which reduces the memory usage during decrunching to the statically allocated memory plus the decrunch code plus the compressed representation of just the relocation information. This usually results in a total memory overhead of less than 1k (depending on the number of relocations). Shrinkler now prints the memory overhead during and after decrunching when you crunch. These numbers are highly dependent on the new --overlap option as well as the good old --mini option. Experiment with these options to see the difference. Enjoy! :-D |
Lonewolf10
Member |
Blueberry: Christmas has arrived early this year. Shrinkler 4.2 is here! Awesome, I wondered whether you had abandoned it or not, as the thead on it over on English Amiga Board appeared to have been abandoned since my last post on it (from a year or so back). Will have a play with this tomorrow ;) (No work for me this week, except on Wednesday 31st Dec).
|
Blueberry
Member |
Which thread are you referring to? I have read through all EAB threads I could find on exe crunching on several occasions, but most of them are quite old, and none of them mention Shrinkler by name (searching gives zero hits). Do you mean one of the threads that mention ExeCruncher2?
|
Lonewolf10
Member |
Sincere apologies Blueberry, I was getting mixed up with The Nibbler by Photon. I did attempt to use the latest version of The Shrinkler today (4.2), but couldn't get it to work. I tested it on WinUAE 2.0.1 with A600 setup (68000 CPU, 8MB chipRAM, 0MB fastRAM, ECS chipset): ram:Shrinkler -m ram:File ram:CrunchedFile
Gives two errors (Bus Error and Illegal Instruction). ram:File = 192256 bytes in length Tried with a smaller file (27243 bytes) and got the same error. ram:Shrinkler -i 3 ram:File ram:CrunchedFile
Same errors as above. (trying -i3 results in illegal instruction error) Does it needs fastRAM to work? I didn't get around to checking that today. |
Blueberry
Member |
Indeed it seems to not work on low-end configs.
Most likely it requires a newer kickstart. I have only tested it with my kickstart 3 setup, as it is sort of pointless to run it on slow machines anyway. :)
Run it on a high-end Amiga, in a high-end fastest-possible emulated config (better), or directly on your PC (best). The resulting executable will of course work on all Amigas in any case.
|
Moerder
Member |
I tried building shrinkler with VS 2013, but the resulting binaries crash. In LZParser::parse at the beginning it does which leaves edges_to_pos with a size of 0, but later there's the following loop
for (int pos = 1 ; pos <= data_length ; pos++) {
// Assimilate edges ending here
for (map<int, RefEdge*>::iterator it = edges_to_pos[pos].begin() ; it != edges_to_pos[pos].end() ; it++) {
This accesses edges_to_pos (which has a size of 0) at index 1. I can build working Cygwin binaries, but I can't help and think this is a problem in the code itself. |
Blueberry
Member |
Yes, you are right. The line should be removed. The intent seemingly was to clear all maps in edges_to_pos, but this is actually not necessary, as they are individually cleared after use on (currently) line 294: edges_to_pos[pos].clear(); and new edges are only added to entries strictly higher than pos (since all added edges are starting at pos and have a length of at least 2). Given that the code works on the targets that I compile and test for, it seems the memory for the vector is not freed on a clear, and the destructed maps are valid as empty maps. These conditions are then apparently not fulfilled for VS. Thank you for the precise report. |
Lonewolf10
Member |
I had some time today to do some testing on WinUAE 2.0.1 using my normal A600 config. (which ran out of memory - see below) and a much beefier A4000 config. (which completed successfully after an hour or so!), as well as the Windows (32bit) version (which was completed in around 5 seconds or less!). Amiga 600 config (kickstart 37.350, ECS chipset, 68040 CPU, 2MB chipRAM, 8MB fastRAM) > ram:Shrinkler ram:DD_L-Strikes ram:CrunchedFile Hunk Mem Type Mem Size Data Size Data Sum Relocs
0 CHIP DEBUG (skipped)
CODE 168356 168356 f7b24472 1294
SYMBOL (216 entries)
DEBUG (skipped)
1 CHIP BSS 1122600 1122600
SYMBOL (118 entries)
Crunching...
Hunk Original After 1st Pass After 2nd Pass Relocs
0 168356 [82.0%]
Out of memory
Amiga 4000 config (kickstart 39.106, AGA chipset, 68040 CPU, 2MB chipRAM, 0MB fastRAM, 32MB Z3 fastRAM): > ram:Shrinkler ram:DD_L-Strikes ram:CrunchedFile Hunk Mem Type Mem Size Data Size Data Sum Relocs
0 CHIP DEBUG (skipped)
CODE 168356 168356 f7b24472 1294
SYMBOL (216 entries)
DEBUG (skipped)
1 CHIP BSS 1122600 1122600
SYMBOL (118 entries)
Crunching...
Hunk Original After 1st Pass After 2nd Pass Relocs
0 168356 990056.982 98525.852 733.967
1 0 6.875 6.875 0.479
Verifying... OK
References considered: 40398
References discarded: 0
Hunk Mem Type Mem Size Data Size Data Sum Relocs
0 CHIP CODE 168356 24 35f96e58
1 CHIP BSS 1122600 1122600
2 ANY CODE 99508 99504 6a4e6d09
Memory overhead during decrunching: 99520
Memory overhead after decrunching: 0
Saving file ram:CrunchedFile
Final file size: 99588
It used 12MB of Z3 fastRAM to do the crunching. Would you expect it to use such a large amount of memory? It only updates '1st pass' (and '2nd pass'?) status once it has reached 1.0% or higher. A quick comparison with LHA 1.42: CrunchedFile.lha is 118212 bytes in length. So Shrinkler is better than LHA on my test file :) Next, I tried the Windows version. No major issues, just this minor one: Shrinkler executable file compressor by Blueberry - version 4.2 (2014-12-16)
Loading file DD_L-Strikes...
Hunk Mem Type Mem size Data size Data sum Relocs
0 CHIP DEBUG (skipped)
CODE 168356 168356 f7b24472 1294
SYMBOL (216 entries)
DEBUG (skipped)
1 CHIP BSS 1122600 1122600
SYMBOL (118 entries)
Crunching...
Hunk Original After 1st pass After 2nd pass Relocs
0 168356 [0.0%]←[6D[1.0%]←[6D[1.1%]←[6D[1.2%]←[6D[1.3%]←[6D[1.4%]←[6D[1.5
%]←[6D[1.6%]←[6D[1.7%]←[6D[1.8%]←[6D[1.9%]←[6D[2.0%]←[6D[2.1%]←[6D[2.2%]←[6D[2.3
The status doesn't update properly in the Windows (2000 SP4) DOS prompt, as you can see above. I'm guessing the DOS prompt doesn't like ANSI codes? Both the Windows (32bit) and Amiga version of Shrinkler created compressed files that I was able to auto-decrunch and run by double-clicking on them. Poking DFF180 worked a treat during decrunching too. Great work Blueberry. |
Moerder
Member |
Blueberry: Given that the code works on the targets that I compile and test for, it seems the memory for the vector is not freed on a clear, and the destructed maps are valid as empty maps. These conditions are then apparently not fulfilled for VS. Yes. The standard intentionally doesn't say what happens with allocated memory when clear() is called. A container implementation may choose to keep its memory for later use, and std::vector's index operator isn't required (or even allowed?) to perform index range checking. This can lead to situations where a program crashes when built with one toolset and seemingly works when built with another. The more recent debug libraries from Microsoft on the other hand are pretty good at finding such problems since they perform all sorts of checks. It certainly didn't take a long time to find this particular problem.
|
Blueberry
Member |
Lonewolf10: It used 12MB of Z3 fastRAM to do the crunching. Would you expect it to use such a large amount of memory? The current version is quite memory hungry. 12MB for a 168k file sounds about right. :) You can reduce the memory consumption using the -r option (try e.g. -r 10000), at some cost to the compression rate (but you can compensate using the other compression options, at the expense of compression time). Lonewolf10: It only updates '1st pass' (and '2nd pass'?) status once it has reached 1.0% or higher. Oops. Will fix. :) Lonewolf10: A quick comparison with LHA 1.42: CrunchedFile.lha is 118212 bytes in length. So Shrinkler is better than LHA on my test file :) LHA is using an ancient LZ+Huffman compression. To beat Shrinkler you need to go to a more modern compressor, such as 7-Zip (LZMA-based). Lonewolf10: I'm guessing the DOS prompt doesn't like ANSI codes? Correct. There are many alternative consoles for Windows that do support ANSI escape codes. Myself I use Cygwin, but that is probably overkill if all you want is ANSI support. :) Thanks for your experience report. It is always good to know what people think.
|
Blueberry
Member |
Moerder: Yes. The standard intentionally doesn't say what happens with allocated memory when clear() is called. I think most implementations will keep the memory around (at least in release mode), since the common use case for clear() is to re-fill the vector afterwards. I am more surprised that the destructed maps are in a valid state for adding elements. Presumably the first memory allocation is done lazily, and the destruction of a newly created map is a no-op.
|
Moerder
Member |
Probably. Tbh, I didn't really check whether it was the the map or already the vector that crashed. I'm happy it works with VS now =)
|
Blueberry
Member |
Shrinkler 4.3 has hit the streets, with a few minor fixes and tweaks based on the discussions in this thread: The bogus clear() that Moerder found is fixed. While I was at it, I tried out the source code in Visual Studio and fixed a couple of other things that were in the way for compiling it there. It now compiles out of the box in VS2010 and also in VS2013 after you disable that "secure runtime" crap. The bug in the progress indicator that Lonewolf10 mentioned is also fixed, so now there is also progress in 0.1% increments between 0% and 1%. :) And speaking of progress indication, the printing of progress can now be disabled (using the new --no-progress / -p option), which makes the output far more readable if you run it in a terminal without ANSI escape code support. The value printed for "References discarded" previously showed the number of references discarded in the last pass of the last hunk, which would often be 0 even when references were discarded in earlier hunks/passes. It now shows the maximum number of references discarded over all passes, similarly to what "References considered" shows. Thus you can use this number to see whether changing the number of references might affect the compression (often it doesn't even when references were discarded, but if no references were discarded, it definitely doesn't). The help text has been slightly reformatted and reformulated to stay within 77 columns, so that it does not wrap when shown in a 640 pixel wide Amiga shell with Topaz 8. Finally, yesterday was the one-year anniversary of the initial 4.0 version and the day of Shrinkler's Pouet debut after a one-year ADA exclusivity period. Wonderful comments so far. ;)
|
noname
Member |
Good stuff. Just wanted to quickly mention that the URL you gave in the Pouet nfo is currently broken. Would be good to fix that so as to attract more people back to ADA :) |
Angry Retired Bastard
Member |
Hah, the overlay option was exactly what I needed to turn Rubzilla into a sub-64k-intro (a couple of months ago)... I guess this'll now replace good old crunchmania as my go-to cruncher.
|
Moerder
Member |
Blueberry: Data file compression should be easy to add. I will put it on my to-do list. A raw file compression mode would be totally awesome, not only for data compression but also executable compression for platforms other than Amiga, witout having to hack platform support into Shrinkler itself.
|
Blueberry
Member |
noname: Good stuff. Just wanted to quickly mention that the URL you gave in the Pouet nfo is currently broken. Would be good to fix that so as to attract more people back to ADA :) Thanks for the tip. This is now fixed in the nfo as well as in the archive and repository.
|
Blueberry
Member |
Moerder: A raw file compression mode would be totally awesome, not only for data compression but also executable compression for platforms other than Amiga, witout having to hack platform support into Shrinkler itself. Good point. I have been thinking that it is too slow for practical use (it certainly is for on-the-fly decompression), and that I would rather design a new algorithm which compresses almost as well but decompresses much faster (at the cost of significantly larger decompression code). But it does make sense for launch-time decompression of small amounts of data. Manual executable compression based on a data cruncher can be somewhat cumbersome, though (been there, done that), as you need to handle executable format details, segmentation, relocation, cache flushing and the like by yourself. I have been looking a bit into the Atari executable format recently, and it is likely that Atari support will be added to Shrinkler at some point. Are there any significant 68000-based demo platforms apart from Amiga and Atari?
|
Moerder
Member |
Blueberry: Manual executable compression based on a data cruncher can be somewhat cumbersome, though (been there, done that), as you need to handle executable format details, segmentation, relocation, cache flushing and the like by yourself. Of course. Personally I'm mostly thinking about ROM based platforms (not necessarily 68k based), so there's not much of an executable file format to worry about. You see, I tend to write a prod or so for a particular platform and then move on, so I've got absolutely no problem with that kind of hack because I'm only doing it once (so yes, I'm not an Amiga guy, I just stumbled in here because Shrinkler is so awesome).
|
Lonewolf10
Member |
I finally got some time to test out the latest version today... Blueberry: The bug in the progress indicator that Lonewolf10 mentioned is also fixed, so now there is also progress in 0.1% increments between 0% and 1%. :) Thanks :) Shrinkler43 Windows32 tests (on my Windows 2K SP4 laptop, with Pentium 3m CPU), using DOS prompt: - using -f ddf180 still works, including when used at the same time as those below. - using -p worked a treat - using -T <textfile> works nicely - using -t works too! I was trying to reproduce a bug where -fdff180 was used as textfile contents when using -T option (I later realised I had used -t by accident), when this happened: C:Documents and Settings/user/My Documents/Amiga-HD/PC-TEMP/Shrinkler>Shrinkler
43-Windows32 -T -fdff18233 DD_L-Strikes2 TTTT1
Shrinkler executable file compressor by Blueberry - version 4.3 (2015-01-05)
Error: Could not open text file -fdff18233
Produced an 'Application Error' pop-up message: The instruction at "0x77f8206b" referenced memory at "0x00000034". The memory could not be "read". It seems that it happens if the textfile required by the -T option is either incorrectly typed, or non-existant. Clicking on "Ok" to close the application just appears to abort the Shrinkler operation and doesn't close the DOS command prompt. Shrinkler43 Amiga tests (using A4000 config. in WinUAE): - using -f still works, even when used in conjunction with the -T option, below. - didn't try -p as 1) it takes ages to crunch my testfile; and 2) Amiga supports ANSI! :) - using -T <textfile> works nicely ;) Thanks for the improved version Blueberry. |
Blueberry
Member |
Ah yes, there's an exit missing after the text file error message. Thanks!
|
Blueberry
Member |
Hi guys! When I put together the first version of Shrinkler, the focus was on functionality, and many of the algorithms and data structures used were far from optimal. Optimizations were planned... I finally got around to implementing all those optimizations. The result is Shrinkler 4.4. With the compression now running much, much faster, I upped some of the default values to what now seems reasonable compromises. All in all, with default options, on the tests I have run, Shrinkler 4.4 typically runs 5-20 times faster and compresses 1-3% better than Shrinkler 4.3. By popular request, I have also added an option for raw data compression, and included decompression source code for it. Let me know if you find it useful. |
corial
Member |
4.4!! You algorithm-machine, you!
|
Moerder
Member |
Cool. I totally realize Shrinkler is targeted primarily at Amigas, but for usage with other CPUs and raw data compression, an option to select endianness of output data would also be useful.
|
Blueberry
Member |
You can easily modify the decompression algorithm to handle the big-endian compressed data independently of the machine endian simply by reading a byte at a time instead of a longword.
Just initialize D4 to $80 instead of $80000000 and change the three instructions using D4 (on lines 136, 138 and 139) from .l to .b. That should do it.
|
Moerder
Member |
Yes, that will do for some cases.
Not sure about ARM (my target CPU), which can do 8/16 bit load/store, but most other operations are 32 bit only, IIRC. This matters since I need a size optimized depacker.
Anyway, I can always post-process compressed data myself.
|
