Line tagger de-escapes escaped symbols when adding line tags

[desplesda]

What is the current behavior?

UntaggedLineListener deletes all escaping backslashes in a line when adding a tag to that line.

Please provide the steps to reproduce, and if possible a minimal demo of the problem:

Given a line of script that contains escaped characters and no line ID:

This is a line with an embedded \#hashtag in it.

If the line tagger is run on this line, the result will be something like:

This is a line with an embedded #hashtag in it. #line:abc123

This resulting text is a parse error, because unescaped hashtags have special meaning to the compiler and cannot be present in the middle of a line.

What is the expected behavior?

The line tagger should not delete the backslashes.

Please tell us about your environment:

• Yarn Spinner Version: 3.0.0-dev

Other information

This issue is present in the compiler, and affects all engines and targets.

The issue appears to be something to do with how UntaggedLineListener's TokenStreamRewriter is being queried for the line text, which does not include any hidden channels. When in BodyMode, the lexer shifts the escaping backslash to the HIDDEN channel (or, in TextMode, skips the backslash entirely.)

[desplesda]

Update to this: we're seeing user reports that the tagger is replacing \# with # across the entire project, rather than on single lines.

[dario-zubovic]

Noticed this bug on our end as well... \{size\}Oi!\{/size\} gets replaced with {size}Oi!{/size} #line:id, which then breaks the yarn script because {size} is unrecognized. We're using Text Animator plugin in Unity, so there are a lot of \{tags\} in our scripts.