Discussion:
sparse files
(too old to reply)
I am Rob
2018-02-10 05:05:10 UTC
Permalink
I wrote a very short routine (26 bytes) to check if a block is all zeros (sparse) and is very fast. It can know if a block is non-sparse in as little as checking one byte or goes through the loop only 256 times. The way I figure it, within Prodos the routine needs to be called just before a WRITE call is sent to the firmware, or just before the volume bitmap is updated so that the block that is all zeroes, doesn't get protected.

or the alternate method from an ML program and takes a little more code is,

check the entire file in memory using this routine, and if a sparse block in memory is located, make a map of sparse blocks. Then all that has to be done is to set the write for data up to just before each sparse block, advance the SETMARK to just past the the sparse block then continue writing data up til the next sparse block and so on.

I have implemented this into a copy program that works good. Although, I don't see much use for sparse files on Prodos 8 files. Most files that are sparse that I come across are mostly IIGS SHR graphics files.

So I don't know if there would be a real need to implement sparse files under Prodos 8. Anyone else see how sparse files may be a benefit to Prodos 8?
qkumba
2018-02-10 06:51:54 UTC
Permalink
I'm using them for my ProDOS-based Infocom ports, to carry save-games without requiring 140kb of disk space if they're not used.

Some of my game ports like Ultima benefit from sparse files, too.

However, perhaps I've misunderstood something, because ProDOS 8 has support for sparse files since several versions already.
I am Rob
2018-02-10 09:45:55 UTC
Permalink
Post by qkumba
I'm using them for my ProDOS-based Infocom ports, to carry save-games without requiring 140kb of disk space if they're not used.
Some of my game ports like Ultima benefit from sparse files, too.
However, perhaps I've misunderstood something, because ProDOS 8 has support for sparse files since several versions already.
My understanding is that Bssic.system can create sparse files with the B parameter, but when a file is read that is sparse and written back to disk, the file is no longer sparse.
Michael J. Mahon
2018-02-10 18:40:18 UTC
Permalink
Post by I am Rob
Post by qkumba
I'm using them for my ProDOS-based Infocom ports, to carry save-games
without requiring 140kb of disk space if they're not used.
Some of my game ports like Ultima benefit from sparse files, too.
However, perhaps I've misunderstood something, because ProDOS 8 has
support for sparse files since several versions already.
My understanding is that Bssic.system can create sparse files with the B
parameter, but when a file is read that is sparse and written back to
disk, the file is no longer sparse.
This is true if a simple read-write copy loop is used. Sparse-aware copying
is also possible, but not as simple.

I use sparse binary files to provide dozens of "virtual files" in a single
sparse file. This is done using the B parameter, and allows many more
"streams" of indeterminate length than ProDOS would allow me to open
simultaneously.

I manage buffering in software with an LRU cache.
--
-michael - NadaNet 3.1 and AppleCrate II: http://michaeljmahon.com
I am Rob
2018-02-10 20:54:45 UTC
Permalink
Post by Michael J. Mahon
Post by I am Rob
Post by qkumba
I'm using them for my ProDOS-based Infocom ports, to carry save-games
without requiring 140kb of disk space if they're not used.
Some of my game ports like Ultima benefit from sparse files, too.
However, perhaps I've misunderstood something, because ProDOS 8 has
support for sparse files since several versions already.
My understanding is that Bssic.system can create sparse files with the B
parameter, but when a file is read that is sparse and written back to
disk, the file is no longer sparse.
This is true if a simple read-write copy loop is used. Sparse-aware copying
is also possible, but not as simple.
I use sparse binary files to provide dozens of "virtual files" in a single
sparse file. This is done using the B parameter, and allows many more
"streams" of indeterminate length than ProDOS would allow me to open
simultaneously.
I manage buffering in software with an LRU cache.
The only real advantage of doing this way is that all the streams are kept in one filename as opposed to having a separate file for each stream. The disadvantage is the B parameter would need to be kept track of for every stream that was BSAVE'd.

I was thinking more of targeting the BSAVE itself, that whenever a file is saved to disk, the file would be checked first if it has any sparse blocks.
qkumba
2018-02-11 02:48:48 UTC
Permalink
Post by I am Rob
My understanding is that Bssic.system can create sparse files with the B parameter, but when a file is read that is sparse and written back to disk, the file is no longer sparse.
Indeed, that is the case. An explicit write will convert from sparse to non-sparse.
Targeting BSAVE itself would be a fine extension!

I am Rob
2018-02-10 09:55:11 UTC
Permalink
Post by qkumba
I'm using them for my ProDOS-based Infocom ports, to carry save-games without requiring 140kb of disk space if they're not used.
Some of my game ports like Ultima benefit from sparse files, too.
However, perhaps I've misunderstood something, because ProDOS 8 has support for sparse files since several versions already.
For whole disk saves, I wrote a couple of my own that can read any volume of any size and store it in a file. Only used blocks are saved to the file and unused blocks are ignored. This still writes blocks though that are filled with zeroes but are marked used.
Loading...