TBF I face much more challenges saving data _from_ the WaybackMachine using the CDX API than most of the sites I've scraped:

Most tools for offline archiving simply don't work, and although I'm *really* slow with my requests I get throttled all the time :P

Oh, and I almost forgot that it's surprisingly hard to translate IA URL's to local file paths, esp. since the URL's retrieved from the API aren't properly encoded (https://web.archive.org/.../http://example.com/...)

@ciaranmak I'm not sure I follow. Are you doing this via the CDX API? If there is RSS what requires tweaking? The RSS feeds don't include the whole content so you have to scrape them for archiving?

@ciaranmak Got you! I'd say that hitting paywalls and even some JS-based UI monstrocity is the "normal" these days which I'd expect (and probably use Selenium or similar to grab it). But in case of the Wayback Machine I'd expect a friendlier API...

@buherator have you tried asciimoos omnom: https://github.com/asciimoo/omnom

i host an instance at https://links.ctrlc.hu/

@stf Thanks for the reminder, I never had the opportunity to use it! My goal is specifically to dump datasets from Wayback Machine for specific domains, so browser-based solutions are less useful for me now.