masterscraper/scraper.py

13 lines
209 B
Python

#!/usr/bin/python3
import masterscraper as ms
scrapelist = ms.scrapelist('conf/wikipedia.txt')
for url in scrapelist:
scrape = ms.scrape(url)
scrape.get_meta()
scrape.clean()
scrape.save()