masterscraper/scrape_list.py

10 lines
180 B
Python

#!/usr/bin/python3
scrapelist = ms.scrapelist('conf/wikipedia.txt')
for url in scrapelist:
scrape = ms.scrape(url)
scrape.get_meta()
scrape.clean()
scrape.save()