Skip to main content

View Post [edit]

Poster: aibek Date: Nov 22, 2013 7:33pm
Forum: forums Subject: Re: How to find only unique captures?

The official way to do it is probably this: https://github.com/internetarchive/wayback/tree/master/wayback-cdx-server#collapsing However, as the above only collapses adjacent entries, you would get better results (and also, would have Wayback Machine do less of the work), by using some Perl or Python script to do the job. https://duckduckgo.com/html?q=remove%20duplicate%20lines%20script
This post was modified by aibek on 2013-11-23 03:33:46

Reply [edit]

Poster: Zarkoff Date: Nov 23, 2013 9:55am
Forum: forums Subject: Re: How to find only unique captures?

This is brilliant, aibek, thank you.

Results collapsed only by adjacent identical captures is better for my purposes.