A.D.A. Amiga Demoscene Archive

  Welcome guest! Please register a new account or log in

  

  

  

Demos Amiga Demoscene Archive Forum / Coding / My Amiga Cruncher
 Page:  ««  1  2  3  4  5  
Author Message
Lonewolf10
Member
#1 - Posted: 24 Jan 2015 16:54 - Edited
Reply Quote
I have tested (the Win32 version) of Shrinkler 4.4 and found one issue with the crunched files.

I get an error 80000004 (illegal instruction?) on 68000 CPU when running crunched file from workbench. It works fine straight from floppy disk at boot-up (before workbench is loaded).

I also retested Shrinkler 4.3 with the same files and that works from workbench or from floppy disk at boot-up without any issues.

Edit:It seems that after more testing Shrinkler 4.3 seems to have the same issue. I can't seem to put my finger on what is causing it, as sometimes the files will decompress from workbench without an error and sometimes they won't.
Any ideas Blueberry?
Blueberry
Member
#2 - Posted: 25 Jan 2015 20:59
Reply Quote
That doesn't sound good...

Does it have a .info file or are you running it via Show All Files?

Which Kickstart and Workbench versions are you running?

Are you running any non-standard programs that might interfere? If you rename WBStartup to something else (if you are on 2.0 or newer), then start with no startup sequence and simply run LoadWB and EndCLI, does the problem still occur?
Lonewolf10
Member
#3 - Posted: 26 Jan 2015 23:09
Reply Quote

I'm using my usual A600 setup (68000 CPU, 8MB chipRAM, 0MB fastRAM, ECS chipset) in WinUAE 2.0.1:

Kickstart=37.350
workbench=37.71

No other programs running in background (unless you count workbench itself!)


When the files don't have any icons, the files decompress and run without any issues.
When the files have icons (tool icon on my 3 test files), I get the error mentioned in my previous post.


I don't understand why having an icon would cause them to crash. The icon file (.info) is completely separate, right?

Blueberry
Member
#4 - Posted: 27 Jan 2015 13:34 - Edited
Reply Quote
When a program is run from Workbench using a tool icon, the Workbench sends a message to the program containing information about tooltypes, icons selected and such. The program must receive that message, or it might crash. See for instance this explanation.

Do your programs work if they are not crunched? You might need to test both short-running and longer-running (at least a few seconds) programs to be sure.
Lonewolf10
Member
#5 - Posted: 29 Jan 2015 23:27 - Edited
Reply Quote
Ok, I found some time to do some more testing today using Shrinkler 4.4 and WinUAE 2.0.1

The original 3 test files were created in Blitz Basic by someone else. The file used in the tests below, was created using DevPac 3.18 by me:

Compression - No Icon - works
Compression - Icon - Project - fails with 'no default tool' message from Workbench.
Compression - Icon - Tool - fails

No Compression - No Icon - works /
No Compression - Icon - Project - fails with 'no default tool' message from Workbench.
No Compression - Icon - Tool - works /


Blueberry:
Are you running any non-standard programs that might interfere? If you rename WBStartup to something else (if you are on 2.0 or newer), then start with no startup sequence and simply run LoadWB and EndCLI, does the problem still occur?


I dumped the files onto an emulated floppy (ADF) along with LoadWB and Dir in the "C" drawer (I forgot about EndCLI at the time). Created a "startup-sequence" file that runs LoadWB and dumped it in the "S" drawer. I then rebooted with the disk in df0: and once WB loaded, I simply typed DIR df0: and then typed the name of the file to load. The results are as follows:

Compression - Icon (Project) works
Compression - Icon (Tool) works
Compression - No Icon - works

No Compression - Icon (Project) works
No Compression - Icon (Tool) works
No Compression - No Icon - works


Now I went back to the original 3 files that caused issues, and did the same thing:

File 1 with Icon (Tool) - works
File 2 with Icon (Tool) - works
File 3 with Icon (Tool) - works


I looked at my startup-sequence.HD and user-startup files, but can't see anything that I think might cause issues. I can list both files (or upload to my website) if you want to take a look at them.

Blueberry:
Do your programs work if they are not crunched? You might need to test both short-running and longer-running (at least a few seconds) programs to be sure.


When I say they work, they can run for as long as I let them (e.g. minutes or for an hour).

Hannibal
Member
#6 - Posted: 14 Dec 2015 07:36
Reply Quote
I have a minor bug report. Running with WinUAE's builtin Aros kickstart, and running as an A4000-level machine, shrinkler-compressed executables hang after decompressing somehow (I see in the winuae debugger that the program counter is pointing to garbage). I've tried with multiple different kinds of executables, and they seem to all behave the same. The same executables run fine on Aros without compression.

An easy way to test this is to use my toolchain and use runaros.bat with for example Octorubber.

I don't know if you care enough to try to fix this, or figure out why aros doesn't behave the same as all other platforms, but I wanted to at least mention it to you.
Blueberry
Member
#7 - Posted: 25 Dec 2015 09:34 - Edited
Reply Quote
I have not looked at the AROS executable launcher code, but my guess is it leaves registers in a different state than the AmigaOS launcher. Shrinkler assumes that A3 points to the loaded segment list (4 bytes before the entry point) which is true for all Amiga kickstarts but might not be for AROS.

If A3 points to garbage, it could give the behavior you describe for mini-compressed executables (such as Octorubber) but executables crunched using the normal or overlap modes would likely crash earlier on.
Hannibal
Member
#8 - Posted: 28 Dec 2015 23:19
Reply Quote
Toni has made a fix in the replacement rom so for future releases, a3 will match the behavior - problem solved :-)
dalton
Member
#9 - Posted: 23 Aug 2016 12:55
Reply Quote
1)
Is there a preferred file extension for a shrinkled data file?

2)
A request for the addition of an option to add an extra longword at the end of the crunched data file, and possibly also an option to pad the file size to the next multiple of 16.

Cheers
Blueberry
Member
#10 - Posted: 26 Aug 2016 23:18 - Edited
Reply Quote
1) I haven't really thought about that. It's not like it was meant as an interchange format. But of course it is nice to tell the shrinkled files apart from other files easily. My first thought would be ".shr". Or maybe ".shrinkled" because this is Amiga and three-letter extensions are so MS-DOS. What do you think?

2) I am not sure I understand why you would want Shrinkler to do that, rather than just padding the file when you load it. Can you tell a bit more about your use case?
dalton
Member
#11 - Posted: 29 Aug 2016 09:14
Reply Quote
1) My thought exactly. I settled for .shr to keep it reasonably short. Strict filename conventions help when writing makefile recipes.

2) My idea was to concatenate several shrinkled files into one, and keep good data alignment. But then it struck me that I don't really benefit from having the shrinkled data aligned. It's the decrunched data I wan't to have aligned, and that's another story. So forget this request... I didn't think it through.


Another question:

Does the decompression routine support in-place operation? If I put the shrinkled data in the end of the buffer, and decompress to the the beginning?
blakkhar
Member
#12 - Posted: 29 Aug 2016 13:44
Reply Quote
I would vote for ".shrinkled" because it is more clear then ".shr" :-)
Blueberry
Member
#13 - Posted: 30 Aug 2016 16:28 - Edited
Reply Quote
dalton:
Does the decompression routine support in-place operation? If I put the shrinkled data in the end of the buffer, and decompress to the the beginning?

Sometimes it will work, but sometimes it will need a safety margin (shrinkled data extending a few bytes beyond the end of the decompressed data). This margin is actually computed for each hunk when crunching in overlap mode, so Shrinkler could also compute it and print it when compressing as data.

The margin will usually not be very big, unless you are trying to compress completely random data. A 16 byte margin should be safe. Remember that when decompressing on 68000, the shrinkled data must be word aligned. On 68020+ it doesn't matter.
dalton
Member
#14 - Posted: 31 Aug 2016 12:57
Reply Quote
cool, thanks!
losso
Member
#15 - Posted: 5 Dec 2016 10:40
Reply Quote
I'm wondering about a specific behavior when using data compression (-d option). The compressed data grows in 4-byte steps, and sometimes I get an output file with four trailing zeroes. For Example:

$ util/shrinkler44/Linux64/Shrinkler -d meat.bin meat.shrinkler 
Shrinkler executable file compressor by Blueberry - version 4.4 (2015-01-18)

Loading file DH0/meat.bin...

Crunching...

Original After 1st pass After 2nd pass
280172 842.674 835.943

Verifying... OK

References considered: 1068
References discarded: 0

Saving file meat.shrinkler...

Final file size: 840


And:

$ hexdump -C meat.shrinkler | tail -3
00000330 65 27 b7 d2 1c 25 83 6f d0 6f 76 fc c9 df 4c 87 |e'...%.o.ov...L.|
00000340 ea 67 81 72 00 00 00 00 |.g.r....|
00000348


Is this some rounding error when writing out the data file (since "835.943 bytes" sound that it might fit into 836 bytes) and I can safely truncate those 00 bytes?
krabob
Member
#16 - Posted: 17 Dec 2016 11:15
Reply Quote
Hello, maybe too late for testing ?

I'm currently trying shrinkler from crinkler.net/shrinkler40.zip
2 problems to report:
The option "-h" for merging hunks: return
"Merging hunks ... EMT Trap"... with an exe where's there is nothing to merge, so ok.

My startup code that GetMsg the eventual Workbench message and return it seems to doesn't work when crinkled... the crunched executable is able to launch from dos, crash at start from WB icon.
Sad !
Blueberry
Member
#17 - Posted: 28 Dec 2016 23:52
Reply Quote
losso:
Is this some rounding error when writing out the data file (since "835.943 bytes" sound that it might fit into 836 bytes) and I can safely truncate those 00 bytes?

The decompressor reads the compressed data one longword at a time. This is why the compressed size is always divisible by 4. As the sole data block inside an executable, this doesn't matter, as executable sizes are rounded up to a multiple of 4 anyway.

It consumes bits from the longword one at a time from most significant to least significant. In your case, some of the (upper) bits in the last longword are significant, otherwise it would not be there. If those bytes matter to you, you can try truncating some of the last few bytes (that is, replace them by FF to trigger the worst case if you don't know which data will be in memory right after the compressed data) and see if you get the correct result. This is of course not a sustainable solution, as you would need to redo the experiment every time you re-crunch your data.

Can you say a bit more about your use case?
Blueberry
Member
#18 - Posted: 28 Dec 2016 23:55
Reply Quote
krabob: You should really be using version 4.4. Version 4.0 is both buggy and slow. :)

If the problems persist in 4.4, please send me (uncrunched) examples of executables that trigger the bugs. Then I will take a look at what happens.
losso
Member
#19 - Posted: 31 Dec 2016 14:01
Reply Quote
Blueberry:
Can you say a bit more about your use case?


Came up when I was coding this at a point when the header (DOS header and deshrinkler routine) was not divisible by 4, so the size of the compressed data wasn't either.

The "try if it still works" route worked. :) Is it possible to determine the number of significant bytes from the best "After xth pass" number? For example, "755.375" meaning 756 in bytes, "756.992" meaning 757 etc.?

Blueberry
Member
#20 - Posted: 3 Jan 2017 12:42 - Edited
Reply Quote
losso:
Is it possible to determine the number of significant bytes from the best "After xth pass" number? For example, "755.375" meaning 756 in bytes, "756.992" meaning 757 etc.?

Not quite. The fractional numbers are based on the theoretical entropy for the compressed data. The actual, range encoded data need a few more bits to ensure correct rounding at the end.
 Page:  ««  1  2  3  4  5  

  Please register a new account or log in to comment

  

  

  

 

A.D.A. Amiga Demoscene Archive, Version 3.0