Package takeovers

[domwhewell-sage]

This may be a far-out idea that may not be feasible, but a module that extracts private pip, npm or gems packages and looks them up on the public registry's.

For example if a code repositories README.md mentions pip install nonexistentpackage that could be extracted and passed to this module which would look it up at https://pypi.org/ and if it does not exist it could be raised as a finding.

This is a good article which explains it better than me https://medium.com/@alex.birsan/dependency-confusion-4a5d60fec610

I don't believe README's are currently analyzed so may need that as a prerequisite. There are probably other things that need validating but would love to know your thoughts

[TheTechromancer]

Hmm interesting. I'm definitely on board with this. This will probably hinge on the family of modules we build around FILESYSTEM. I want to get to a point where FILESYSTEM events are consumed by all the usual modules like excavate, etc., essentially running grep -R on the files.

I'm currently working on optimizing our regex performance, and once that happens we can start leveraging all the CPU cores for this kind of extraction.

[domwhewell-sage]

I'm already looking at what data the RAW_DATA event contains when scanning git repository's... so... much... data...

[TheTechromancer]

holy lord

[TheTechromancer]

Excavate is already struggling lol. It's a good thing we're reworking it in 2.0. New excavate will use Yara and should be able to handle this kind of load.