Cannot accept timestamps more precise than microseconds because of datetime module limitation

[zed-eiq]

👋 hello folks! While implementing support for an intel vendor's API, our application could not process their STIX 2.1 objects.

Turns out that the vendor is sending us timestamps with nanosecond precision (e.g.: 2023-11-12T14:16:48.1699798608Z), which then causes datetime.strptime to fail when we attempt to parse with stix2.

We checked the STIX 2.1 specification, and it only discourages timestamps more precise than nanoseconds, so the vendor is in this sense spec-compliant: https://docs.oasis-open.org/cti/stix/v2.1/os/stix-v2.1-os.html#_i7rrdtzgta8d

Relevant traceback:

File "/usr/lib/python3.8/_strptime.py", line 349, in _strptime\n    raise ValueError("time data %r does not match format %r" %\nValueError: time data \'2023-11-12T14:16:48.1699798608Z\' does not match format \'%Y-%m-%dT%H:%M:%S.%fZ\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/base.py", line 50, in _check_property\n    kwargs[prop_name], has_custom = prop.clean(\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/properties.py", line 383, in clean\n    return parse_into_datetime(\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/utils.py", line 252, in parse_into_datetime\n    raise ValueError(\nValueError: must be a datetime object, date object, or timestamp string in a recognizable format.\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  

We've implemented a workaround to round all timestamps to the nearest microsecond. But this mutates the original data, so it's no longer strictly the same STIX object.

What would the OASIS Open TC org recommend?

---

Full traceback for context/transparency:

WARNING  eiq.extensions.stix_taxii2.protocol:protocol.py:195 {'id': 'indicator--7032a43b-4263-4a26-935c-8aed935dc172', 'event': 'Skipping invalid STIX 2.1 object', 'logger': 'eiq.extensions.stix_taxii2.protocol', 'level': 'warning', 'timestamp': '2023-11-13T12:22:26.052552Z', 'exception': 'Traceback (most recent call last):\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/utils.py", line 249, in parse_into_datetime\n    parsed = dt.datetime.strptime(value, fmt)\n  
File "/usr/lib/python3.8/_strptime.py", line 568, in _strptime_datetime\n    tt, fraction, gmtoff_fraction = _strptime(data_string, format)\n  
File "/usr/lib/python3.8/_strptime.py", line 349, in _strptime\n    raise ValueError("time data %r does not match format %r" %\nValueError: time data \'2023-11-12T14:16:48.1699798608Z\' does not match format \'%Y-%m-%dT%H:%M:%S.%fZ\'\n\nDuring handling of the above exception, another exception occurred:\n\nTraceback (most recent call last):\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/base.py", line 50, in _check_property\n    kwargs[prop_name], has_custom = prop.clean(\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/properties.py", line 383, in clean\n    return parse_into_datetime(\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/utils.py", line 252, in parse_into_datetime\n    raise ValueError(\nValueError: must be a datetime object, date object, or timestamp string in a recognizable format.\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  
File "<REDACTED>/extensions/stix_taxii2/eiq/extensions/stix_taxii2/utility.py", line 59, in stix_obj_from_dict\n    obj = cls(allow_custom=True, **d)\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/v21/sdo.py", line 250, in __init__\n    super(Indicator, self).__init__(*args, **kwargs)\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/base.py", line 203, in __init__\n    temp_custom = self._check_property(\n  
File "<REDACTED>/env/lib/python3.8/site-packages/stix2/base.py", line 58, in _check_property\n    raise InvalidValueError(\nstix2.exceptions.InvalidValueError: Invalid value for Indicator \'modified\': must be a datetime object, date object, or timestamp string in a recognizable format.\n\nThe above exception was the direct cause of the following exception:\n\nTraceback (most recent call last):\n  
File "<REDACTED>/extensions/stix_taxii2/eiq/extensions/stix_taxii2/protocol.py", line 193, in transform_stix21\n    obj = stix_obj_from_dict(stix_obj_dict)\n  
File "<REDACTED>/extensions/stix_taxii2/eiq/extensions/stix_taxii2/utility.py", line 61, in stix_obj_from_dict\n    raise InvalidObjectError from ex\neiq.extensions.stix_taxii2.utility.InvalidObjectError'}