Discussion:
Introducing DSKalyzer, a command-line disk tool written in Go
(too old to reply)
y***@gmail.com
2017-07-29 10:33:06 UTC
Permalink
Raw Message
DSKalyzer is a cross-platform command-line tool for manipulating and managing Apple II DSK (and other) images.

Download from: https://github.com/paleotronic/dskalyzer/releases
GitHub repository: https://github.com/paleotronic/dskalyzer

Features include:

- Read from ProDOS, DOS 3.X, RDOS and Pascal disk images;
- ProDOS or DOS ordered; 2MG and NIB; 113-800K
- Write to Prodos and DOS 3.3 disk images;
- Extract and convert binary, text and detokenize BASIC files (Integer and Applesoft);
- Write binary, text and retokenized BASIC (Applesoft) files back to disk images;
- Copy and move files between disk images; delete files, create new folders (ProDOS), etc;
- Generate disk reports that provide track and sector information, text extraction and more;
- Compare multiple disks to determine duplication, or search disks for text or filenames.
- Use command-line flags (allows for automation) or an interactive shell;
- Builds for MacOS, Windows, Linux, FreeBSD and Raspberry Pi.
- Open source; GPLv3 licensed.

DSKalyzer is a command line tool for analyzing and managing Apple II DSK images and their archives. Its features include not only the standard set of disk manipulation tools -- extract (with text conversion), import to disk (including tokenisation of Applesoft BASIC), delete, and so forth -- but also the ability to identify duplicates — complete, active sector, and subset; find file, sector and other commonalities between disks (including as a percentage of similarity or difference); search de-tokenized BASIC, text and binary / sector data; generate reports identifying and / or collating disk type, DOS, geometry, size, and so forth; allowing for easier, semi-automated DSK archival management and research.

DSKalyzer works by first “ingesting” your disk(s), creating an index containing various pieces of information (disk / sector / file hashes, catalogs, text data, binary data etc.) about each disk that is then searchable using the same tool. This way you can easily find information stored on disks without tediously searching manually or through time-consuming multiple image scans. You can also identify duplicates, quasi-duplicates (disks with only minor differences or extraneous data), or iterations, reducing redundancies.

Once you've identified a search you can also extract selected files. DSKalyzer can report to standard output (terminal), to a text file, or to a CSV file.

Shell commands (executing DSKalyzer without flags enters shell):

Note: You must mount a disk before performing tasks on it.

analyze Process disk using dskalyzer analytics
cat Display file information
cd Change local path
copy Copy files from one volume to another
delete Remove file from disk
disks List mounted volumes
extract extract file from disk image
help Shows this help
info Information about the current disk
ingest Ingest directory containing disks (or single disk) into system
lock Lock file on the disk
ls List local files
mkdir Create a directory on disk
mount Mount a disk image
move Move files from one volume to another
put Copy local file to disk
quarantine Like report, but allow moving dupes to a backup folder
quit Leave this place
rename Rename a file on the disk
report Run a report
target Select mounted volume as default
unlock Unlock file on the disk
unmount unmount disk image

Command-line flags:

(Note: You must ingest your disk library before you can run comparison or search operations on it)

-active-sector-partial
Run partial sector match (active only) against all disks
-active-sector-subset
Run subset (active) sector match against all disks
-adorned
Extract files named similar to CP (default true)
-all-file-partial
Run partial file match against all disks
-all-file-subset
Run subset file match against all disks
-all-sector-partial
Run partial sector match (all) against all disks
-all-sector-subset
Run subset (non-zero) sector match against all disks
-as-dupes
Run active sectors only disk dupe report
-as-partial
Run partial active sector match against single disk (-disk required)
-c Cache data to memory for quicker processing (default true)
-cat-dupes
Run duplicate catalog report
-catalog
List disk contents (-with-disk)
-csv
Output data to CSV format
-datastore string
Database of disk fingerprints for checking (default "/Users/melody/DSKalyzer/fingerprints")
-dir
Directory specified disk (needs -disk)
-dir-create string
Directory to create (-with-disk)
-dir-format string
Format of dir (default "{filename} {type} {size:kb} Checksum: {sha256}")
-extract string
Extract files/disks matched in searches ('#'=extract disk, '@'=extract files)
-file string
Search for other disks containing file
-file-delete string
File to delete (-with-disk)
-file-dupes
Run file dupe report
-file-extract string
File to delete from disk (-with-disk)
-file-partial
Run partial file match against single disk (-disk required)
-file-put string
File to put on disk (-with-disk)
-force
Force re-ingest disks that already exist
-ingest string
Disk file or path to ingest
-ingest-mode int
Ingest mode:
0=Fingerprints only
1=Fingerprints + text
2=Fingerprints + sector data
3=All (default 1)
-max-diff int
Maximum different # files for -all-file-partial
-min-same int
Minimum same # files for -all-file-partial
-out string
Output file (empty for stdout)
-quarantine
Run -as-dupes and -whole-disk in quarantine mode
-query string
Disk file to query or analyze
-search-filename string
Search database for file with name
-search-sha string
Search database for file with checksum
-search-text string
Search database for file containing text
-select
Select files for analysis or search based on file/dir/mask
-shell
Start interactive mode
-shell-batch string
Execute shell command(s) from file and exit
-similarity float
Object match threshold for -*-partial reports (default 0.9)
-verbose
Log to stderr
-whole-dupes
Run whole disk dupe report
-with-disk string
Perform disk operation (-file-extract,-file-put,-file-delete)

Getting Started

Ingest your disk collection, so dskalyzer can report on them:

dskalyzer -ingest "C:\Users\myname\LotsOfDisks"
Simple Reports

Find Whole Disk duplicates:

dskalyzer -whole-dupes
Find Active Sectors duplicates (inactive sectors can be different):

dskalyzer -as-dupes
Find Duplicate files across disks:

dskalyzer -file-dupes
Limiting reports to subdirectories

Find Active Sector duplicates but only under a folder:

dskalyzer -as-dupes -select "C:\Users\myname\LotsOfDisks\Operating Systems"
Antoine Vignau
2017-07-29 10:37:11 UTC
Permalink
Raw Message
Really nice. Congratulations!
6***@gmail.com
2017-07-29 12:10:14 UTC
Permalink
Raw Message
Great!
Steven Hirsch
2017-07-29 12:33:56 UTC
Permalink
Raw Message
Post by y***@gmail.com
DSKalyzer is a cross-platform command-line tool for manipulating and managing Apple II DSK (and other) images.
Download from: https://github.com/paleotronic/dskalyzer/releases
GitHub repository: https://github.com/paleotronic/dskalyzer
Looks good, but you should consider adding build directions. After installing
the 'go' compiler on my Ubuntu 16.04 system and running 'make.sh' from within
the source directory:

Building dskalyzer...
data.go:20:2: cannot find package "github.com/paleotronic/dskalyzer/disk" in
any of:
/usr/local/go/src/github.com/paleotronic/dskalyzer/disk (from $GOROOT)
/home/hirsch/go/src/github.com/paleotronic/dskalyzer/disk (from $GOPATH)
data.go:21:2: cannot find package "github.com/paleotronic/dskalyzer/loggy" in
any of:
/usr/local/go/src/github.com/paleotronic/dskalyzer/loggy (from $GOROOT)
/home/hirsch/go/src/github.com/paleotronic/dskalyzer/loggy (from $GOPATH)
ingestor.go:21:2: cannot find package "github.com/paleotronic/dskalyzer/panic"
in any of:
/usr/local/go/src/github.com/paleotronic/dskalyzer/panic (from $GOROOT)
/home/hirsch/go/src/github.com/paleotronic/dskalyzer/panic (from $GOPATH)

So, apparently the source directory must be here:

~/go/src/github.com/paleotronic/dskalyzer

in order to build correctly. A sentence or two on the subject will help others.
Antoine Vignau
2017-07-29 13:32:30 UTC
Permalink
Raw Message
Does DSKalyser handle extended files? Those with a data and resource forks?

Thank you,
Antoine
James Davis
2017-07-29 19:23:24 UTC
Permalink
Raw Message
"Introducing DSKalyzer, a command-line disk tool written in Go "

What is the "Go" language (and OS)? And/or, what Operating System does "DSKalyzer" run under?
Michael 'AppleWin Debugger Dev'
2017-07-29 19:27:58 UTC
Permalink
Raw Message
Post by James Davis
What is the "Go" language (and OS)?
The latest iHipster fad language from Google
https://en.wikipedia.org/wiki/Go_(programming_language)
https://golang.org/
Post by James Davis
And/or, what Operating System does "DSKalyzer" run under?
- Builds for MacOS, Windows, Linux, FreeBSD and Raspberry Pi.
Jorge
2017-07-29 20:33:38 UTC
Permalink
Raw Message
Post by Michael 'AppleWin Debugger Dev'
Post by James Davis
What is the "Go" language (and OS)?
The latest iHipster fad language from Google
"fad language"? created by Ken Thompson and Rob Pike no more no less? LOL.
Michael 'AppleWin Debugger Dev'
2017-07-29 22:27:42 UTC
Permalink
Raw Message
Post by Jorge
Post by Michael 'AppleWin Debugger Dev'
Post by James Davis
What is the "Go" language (and OS)?
The latest iHipster fad language from Google
"fad language"? created by Ken Thompson and Rob Pike no more no less? LOL.
By June 2015, its ranking had dropped to below 50th in the index, placing it lower than COBOL and Fortran.
Whether it has any long term staying power remains to be seen. (It was back up to the #13th spot in Jan. 2017.)

https://www.tiobe.com/tiobe-index/


Some of the problems with Go are:

1. It uses ASI (Automatic Semicolon Insertion).

Doug Crockford (Javascript, The Good parts) summarizes what we have learnt from Javascript:

@34:31 "Why am I betting my career on this piece of crap"



2. It uses the "One True Brace" style
https://en.wikipedia.org/wiki/Indent_style#Variant:_1TBS_.28OTBS.29

It is like we learnt _nothing_ from Python. Programming Languages should NOT dictate whitespace usage.

Their excuse is:

https://golang.org/doc/faq#semicolons

Q. Why are there braces but no semicolons? And why can't I put the opening brace on the next line?

A. Go uses brace brackets for statement grouping, a syntax familiar to programmers who have worked with any language in the C family. Semicolons, however, are for parsers, not for people, and we wanted to eliminate them as much as possible. To achieve this goal, Go borrows a trick from BCPL: the semicolons that separate statements are in the formal grammar but are injected automatically, without lookahead, by the lexer at the end of any line that could be the end of a statement. This works very well in practice but has the effect that it forces a brace style. For instance, the opening brace of a function cannot appear on a line by itself.


The _last_ thing I want in a programming language is telling me how to format my multi-column alignment. **I** should dictate where my braces go -- not some dumb compiler. I imagine I'm not alone.

The quickest way to spot a "fad language" is if statement terminators are optional.

My 0.03.
Michael 'AppleWin Debugger Dev'
2017-07-29 22:59:24 UTC
Permalink
Raw Message
Post by Jorge
Post by Michael 'AppleWin Debugger Dev'
The latest iHipster fad language from Google
"fad language"? created by Ken Thompson and Rob Pike no more no less? LOL.
More ways to spot a fad programming language:


3. It offers GC (Garbage Collector) with _zero_ way to manually control the time.

If the GC was made "soft realtime", where the user could pass in a *time limit* say in milliseconds then GC would be made more palatable to MANY people.
Post by Jorge
It may also block the entire program.
:-( The **last** thing I want is some stupid GC introducing "micro-stutter."

Ignoring the problem of memory management doesn't "magically" make it go away. To quote JWZ: "Now you have _two_ problems!"

And just because they have gotten then Mark-and-Sweep time down below 10 ms (2015) doesn't mean that is "good enough" for real-times apps that can't afford ANY stuttering.

At least you can turn it off via: debug.SetGCPercent(-1)


4. Go's excuse of "Just throw more RAM it it" is also a complete turn-off:

https://blog.golang.org/go15gc

Today 16 gigabytes of RAM costs $100 and CPUs come with many cores, each with multiple hardware threads. In a decade this hardware will seem quaint but the software being built in Go today will need to scale to meet expanding needs and the next big thing.



5. Go's multi-threaded performance/design has been criticized.
i.e.
Parts of the run-time were originally written in C, such as the Goroutine scheduler proc.c:
https://searchcode.com/codesearch/view/11771443/

It looks like the compiler is self-hosting now:
https://golang.org/src/runtime/proc.go


6. Go doesn't have the ? : operator
https://golang.org/doc/faq#Does_Go_have_a_ternary_form

7. Declarations are backwards, like Pascal
https://golang.org/doc/faq#declarations_backwards


Go is niche orientated, and like any tool does a few things well. As such, Go has numerous strikes against it that prevent it from ever becoming a popular language IMO.

At least they started with a clean slate and had core designers instead of being designed by committee.
Jorge
2017-07-30 08:09:02 UTC
Permalink
Raw Message
Post by Michael 'AppleWin Debugger Dev'
Post by Jorge
Post by Michael 'AppleWin Debugger Dev'
The latest iHipster fad language from Google
"fad language"? created by Ken Thompson and Rob Pike no more no less? LOL.
3. It offers GC (Garbage Collector) with _zero_ way to manually control the time.
If the GC was made "soft realtime", where the user could pass in a *time limit* say in milliseconds then GC would be made more palatable to MANY people.
Post by Jorge
It may also block the entire program.
:-( The **last** thing I want is some stupid GC introducing "micro-stutter."
Ignoring the problem of memory management doesn't "magically" make it go away. To quote JWZ: "Now you have _two_ problems!"
And just because they have gotten then Mark-and-Sweep time down below 10 ms (2015) doesn't mean that is "good enough" for real-times apps that can't afford ANY stuttering.
At least you can turn it off via: debug.SetGCPercent(-1)
https://blog.golang.org/go15gc
Today 16 gigabytes of RAM costs $100 and CPUs come with many cores, each with multiple hardware threads. In a decade this hardware will seem quaint but the software being built in Go today will need to scale to meet expanding needs and the next big thing.
5. Go's multi-threaded performance/design has been criticized.
i.e.
https://searchcode.com/codesearch/view/11771443/
https://golang.org/src/runtime/proc.go
6. Go doesn't have the ? : operator
https://golang.org/doc/faq#Does_Go_have_a_ternary_form
7. Declarations are backwards, like Pascal
https://golang.org/doc/faq#declarations_backwards
Go is niche orientated, and like any tool does a few things well. As such, Go has numerous strikes against it that prevent it from ever becoming a popular language IMO.
At least they started with a clean slate and had core designers instead of being designed by committee.
You and I may not see it, we may not inderstand the why, but rest assured that they have good reasons to be doing things this way rather than another, because they run circles around you and I and a thousand more like you and I together.

These are my 2 cents!
Zellyn
2017-08-01 14:20:40 UTC
Permalink
Raw Message
Post by Michael 'AppleWin Debugger Dev'
Go is niche orientated, and like any tool does a few things well. As such, Go has numerous strikes against it that prevent it from ever becoming a popular language IMO.
Hmmm. You seem decidedly negative on Go. I'm the opposite: I really enjoy programming in Go. I'm (slowly) working on a blog post introducing Go properly to the retrocomputing community: I think it's a good fit, actually. Go's a good language for grumpy old curmudgeons like me.

:-)

Zellyn
Nick Westgate
2017-08-01 22:25:11 UTC
Permalink
Raw Message
Post by Zellyn
I really enjoy programming in Go.
Yeah, I like it too. It's a well-designed ecosystem. Lots of good libraries, and native API interop is pretty clean too.

But I would prefer Allman style bracing. ; - )

Cheers,
Nick.
Zellyn
2017-08-02 13:31:07 UTC
Permalink
Raw Message
Post by Nick Westgate
But I would prefer Allman style bracing. ; - )
Heh. I don't have strong opinions on braces. But I do agree with the "gofmt proverb":

Gofmt's style is no one's favorite, yet gofmt is everyone's favorite.

:-)

Zellyn
fadden
2017-08-02 16:23:32 UTC
Permalink
Raw Message
"There are only two kinds of languages: the ones people complain about and the ones nobody uses." -- Bjarne Stroustrup
Michael J. Mahon
2017-08-02 17:03:37 UTC
Permalink
Raw Message
Post by fadden
"There are only two kinds of languages: the ones people complain about
and the ones nobody uses." -- Bjarne Stroustrup
Perfect!

Except academics, who complain about every language, sometimes even the one
they invented. ;-)
--
-michael - NadaNet 3.1 and AppleCrate II: http://michaeljmahon.com
Polymorph
2017-08-02 23:11:52 UTC
Permalink
Raw Message
Post by fadden
"There are only two kinds of languages: the ones people complain about and the ones nobody uses." -- Bjarne Stroustrup
And then there are those languages that everyone complains about *AND* nobody uses - I'm looking at you Lisp and Prolog. ;-)

And yes, I know there are probably some poor souls out there that actually do use these languages (I know they were popular in AI circles), but really <scientific_fact>too much recursion is bad for you!</scientific_fact> :-P

Cheers,
Mike

James Davis
2017-07-29 19:28:55 UTC
Permalink
Raw Message
Post by James Davis
"Introducing DSKalyzer, a command-line disk tool written in Go "
What is the "Go" language (and OS)? And/or, what Operating System does "DSKalyzer" run under?
Never mind. I found the answer at your Download webpage: https://github.com/paleotronic/dskalyzer/releases
Lifepillar
2017-07-30 11:06:36 UTC
Permalink
Raw Message
Post by y***@gmail.com
DSKalyzer is a cross-platform command-line tool for manipulating and managing Apple II DSK (and other) images.
Download from: https://github.com/paleotronic/dskalyzer/releases
GitHub repository: https://github.com/paleotronic/dskalyzer
Thanks, really nice.
I have made a Homebrew formula for people wanting to build from source
in macOS:

https://github.com/lifepillar/homebrew-appleii

Enjoy,
Life.
Loading...