Handling constant subsequence

[oentaryorj]

In some occasion, a time series subsequence can be flat/constant for a while, which may lead to a zero standard deviation for that subsequence. However, the the current codes does not seem to handle this case yet. Is there a way to fix this?

[vanbenschoten]

@oentaryorj This is a great call-out. There are a couple of options:

1. We could remove flat/constant segments as a pre-processing routine in the library

2. We could report these segments back as motifs (though admittedly this is a "trivial" motif that may not be meaningful).

Since it sounds like you've encountered this situation in practice, what are your thoughts?

[aouyang1]

@oentaryorj Something that's worked for me was to add some very small noise to the signal. This has the added benefit that the flat regions can be detected later on as different segments of the time series.

[oentaryorj]

@vanbenschoten: Sorry for the late response - just back from vacation :) I would be more inclined toward option 2, although we could probably put an option to ignore/skip the flat segments.

@aouyang1: Thanks for the suggestion. Do you need to modify the current codes to make this work? Any other numerical issues encountered?

[ameya98]

If you read the original SAX paper (SAX was invented by the same people who invented the Matrix Profile) you will see that they handle subsequences with very low standard deviation (that includes the constant subsequence, as that has standard deviation zero) by not dividing by the standard deviation when normalizing. (The threshold for the standard deviation is a parameter here.)
I think incorporating that into the matrix profile makes the most sense.

[aouyang1]

Agreed. Found this from stack overflow as well https://github.com/scikit-learn/scikit-learn/blob/7389dbac82d362f296dc2746f10e43ffa1615660/sklearn/preprocessing/data.py#L70 treating the standard deviation as 1 keeping the existing subsequence values and not normalizing.

Since we're trying to compare to subsequences together such that one subsequence's spectral power does not bias over another, would it make sense to divide the signal such that we satisfy parseval's theorem?

[oentaryorj]

Thanks for the inputs. So I guess "ignoring" the standard deviation means option 2 in @vanbenschoten's reply (i.e., treating constant subsequence as "trivial" motif)? This makes sense to me, although it may depend on the application context.

[ameya98]

Performing the mean subtraction but not the division by the standard deviation in the distance profile computation should suffice as well?