Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[frontend/backend] In CSV Feed Ingester, take into account Default Marking definition options from CSV Mapper #7275

Merged
merged 29 commits into from
Jun 26, 2024

Conversation

Goumies
Copy link
Member

@Goumies Goumies commented Jun 8, 2024

Proposed changes

  • handle CSV mapper configuration on CSV feed ingester creation
  • handle CSV mapper configuration on CSV feed ingester edition
  • update Launch import dialogs

Related issues

Checklist

  • I consider the submitted work as finished
  • I tested the code for its functionality
  • I wrote test cases for the relevant uses case (coverage and e2e)
  • I added/update the relevant documentation (either on github or on notion)
  • Where necessary I refactored code to improve the overall quality

Further comments

To test this feature, create CSV mappers with both options:

  • Let the user choose
  • Use default marking definitions of the user

image

image

image

image

To set default markings:

  • you have to go to Customization > Entity types
    image

  • select an entity type

  • select the Marking attribute
    image

  • activate default values
    image

  • navigate to Security > Groups, select the current group (=> default)
    image

  • select allowed markings, so they can be added to the list of options for default markings

  • select default markings
    image

Single column CSV file to test your CVS mapper:
6149-ipv4-addr.csv

Single column CSV feed to ingest IPv4 addresses: https://lists.blocklist.de/lists/all.txt

To check for the successful ingestion:

  • make sure your worker is running
  • go to Observations > Observables
  • see the IPv4 addresses with corresponding markings

To check different use cases :

  • purge messages in push_sync queue in RabbitMQ
    image
  • delete ingested entities in elastic console
POST /opencti_stix_cyber_observables-000001/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
          { "term":{"entity_type.keyword": "IPv4-Addr" }}
        ]
    } // match_all
  }
}

image

@Goumies Goumies force-pushed the issue/6149 branch 3 times, most recently from d33f71c to d113e63 Compare June 10, 2024 13:22
@Kedae Kedae added the filigran team use to identify PR from the Filigran team label Jun 11, 2024
@Goumies Goumies force-pushed the issue/6149 branch 2 times, most recently from 903bcc5 to c8d1ae7 Compare June 11, 2024 21:32
@Goumies Goumies force-pushed the issue/6149 branch 2 times, most recently from f4684ab to c0ff113 Compare June 13, 2024 07:43
@Goumies Goumies marked this pull request as draft June 14, 2024 08:21
Copy link

codecov bot commented Jun 14, 2024

Codecov Report

Attention: Patch coverage is 85.18519% with 4 lines in your changes missing coverage. Please review.

Project coverage is 67.37%. Comparing base (06e8c93) to head (e7418dd).

Current head e7418dd differs from pull request most recent head d0ece65

Please upload reports for the commit d0ece65 to get more accurate results.

Files Patch % Lines
...rm/opencti-graphql/src/manager/ingestionManager.ts 25.00% 3 Missing ⚠️
...phql/src/modules/ingestion/ingestion-csv-domain.ts 0.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##           master    #7275      +/-   ##
==========================================
+ Coverage   67.34%   67.37%   +0.02%     
==========================================
  Files         563      563              
  Lines       69680    69661      -19     
  Branches     5891     5887       -4     
==========================================
+ Hits        46928    46935       +7     
+ Misses      22752    22726      -26     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@Goumies Goumies force-pushed the issue/6149 branch 6 times, most recently from 9698bbd to 566ba6d Compare June 18, 2024 14:46
@Goumies Goumies marked this pull request as ready for review June 18, 2024 15:09
@CelineSebe

This comment was marked as off-topic.

@Goumies

This comment was marked as resolved.

@Kedae Kedae merged commit 36e0eec into master Jun 26, 2024
3 checks passed
@Kedae Kedae deleted the issue/6149 branch June 26, 2024 11:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
filigran team use to identify PR from the Filigran team
Projects
None yet
Development

Successfully merging this pull request may close these issues.

In CSV Feed Ingester, take into account Default Marking definition options from CSV Mapper
6 participants