Skip to content

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

License

Notifications You must be signed in to change notification settings

voliveirajr/seleniumcrawler

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

8 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

seleniumcrawler

This is a Webcrawler based on Scrapy and Selenium frameworks

This spider crawls thru the directferries.com website in order to generate a json file with all tickets available for one of these directions Dublin-Liverpool / Liverpool-Dublin with departure tomorrow and returning in 3 days.

HOW TO EXECUTE:

Is required an environment with the following requirements installed: -Python 2.7 -Scrapy 0.18 -Selenium web-drivers

To execute the crawler the following command should be executed from the project path

scrapy crawl crawlermate_selenium -a category=[dublin or liverpool] -o [filename] -t json

for an example, to generate tickets for dublin to liverpool on items.json file you should execute scrapy crawl crawlermate_selenium -a category=dublin -o items.json -t json

References:

http://docs.seleniumhq.org/ http://http://scrapy.org/

About

An example using Selenium webdrivers for python and Scrapy framework to create a web scraper to crawl an ASP site

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages