Again, sorry for the month late reply, but others might find this useful.
After I wrote my last reply, I remembered something, see below.
On 3/23/2017 6:42 AM, Karen Lewellen wrote:
An example of a mass downloader included with a Linux shell?
If you mean for grabbing thousands of files, wget works well and runs almost
anywhere. Other good choices are aria2c and curl. I prefer aria2c,
especially for servers which use a lot of redirects and hide the final name
of the downloaded file. I'm seeing this a lot with podcasts. Most
downloaders give the downloaded mp3 file a useless name like audio.mp3 or a
bunch of hex numbers. I don't have this problem with aria2c. It also
supports torrents and I think parallel downloading. It's updated often, but
Debian testing should have a current version. There are Windows binaries but
I couldn't get them to run here. I'm not sure if it's in cygwin.
I want to test this, but am unsure of what tool to use.
I suggest wget, but see below.
The editing is not a problem, I am far from picky about it having, with the
work was smaller, used the m.edition of the site to secure things one at a
time.
I didn't know about the mobile site, or I forgot about it. I'll have to take
a look. Also, w3m and links the chain do a far better job of dumping the
HTML to text without as much junk. If you pass the -nolist option to Lynx
the cat, you don't get the often huge list of links at the end. Finally, for
getting a large number of pages, don't forget the -crawl option. I think
wget is more popular. I almost never see the -crawl option of Lynx the cat
mentioned.
That I might grab all the chapters at once would be a fine solution.
Certainly for the works I cannot get in epub.
I might have a solution. Archive Team did a huge fanfiction.net grab a few
years ago. They published a script to get huge blocks of stories en mass
(we're talking thousands) by range. It should work for a single story or a
group of any size as long as the range is sequential. It of course downloads
all chapters. I'm not sure if it gets .epub, but I seem to remember reading
them, so I think it does. I have to do some digging to see if it's still
around. My machine with the fanfiction archive crashed, but archive.org
should have it. Anyone interested in mass downloads of sites should look at
Archive Team, but be warned there is strong language. They preserve sites
which would otherwise be lost. http://www.archiveteam.org/
If you want a 2016-06-26 fanfiction repack (not sure of the original
download date), look here but be warned that most of the zip files are over
1 GB in size. I'm not sure about format.
https://archive.org/download/fanfiction_collection_repack_2016
Here is a general Internet Archive fanfiction search which is how I found
the above link:
https://archive.org/search.php?query=fanfiction
I can't seem to find the fanfiction download script. I think some grabs
might have been moved or deleted. You could try archiveteam.org. I did find
some Firefly fan fiction though. It looks like the 2016 repack link above is
the best bet for a huge, mass download.
_______________________________________________
Blinux-list mailing list
Blinux-list@xxxxxxxxxx
https://www.redhat.com/mailman/listinfo/blinux-list