x-spider

---

x-spider is a fully comprehensive data gathering suite of tools designed to collect all data from a 'target ur' .

Primary use : cybersecurity.

After running the spider/crawler on any given domain or url you will have a folder containing a complete data set of the 'attack surface' mapped.

• use in conjunction with 'Hummingbird' @ https://github.com/Mulc1b3R/hummingbird

---

• method : Add the target url to the example.env file and rename the file

•    ```.env```
  

• then run python data.py

• an output folder is created containing data.json file.

• the data in data.json can be visualized by running

• python data-viz.py creating a .png file for visual interpretation.

• whois-lookup.py provides the dns registration data as a .json file in

• the output folder.

• The vizualizations include a 'tree map' , 'word cloud' , 'website structure',

• url length 'bar chart' .

• comments can be extracted from comments sections (.json)

• text is extracted from the url (.txt)

• an 'output.json' is created from 'output.json' which then

• creates a 'output.html' of all external and internal links

• as clickable links .

• rss feeds can be extracted using rss.py

• xxx.py is for use in downloading potentially harmful files as txt files for examination.

• Data is gathered for analysis to the output folder...

---

Copyright © 2024 Psico Communications and Blockchain Development.

---