Syntax Highlighting Strings (Custom Language)

[Zaven Alexander]

Hello, I am in the process of settings up a custom Language Tagger for Lua. Following the example provided in the documentation (https://docs.telerik.com/devtools/winforms/controls/syntax-editor/features/taggers/custom-language). Everything seems to be working correctly, however I am having a slight problem with setting a String Literal. A numerical Literal example is provided by overriding the TryGetClassificationType function, however in attempting to do a similar approach for Strings does not give the correct result.

The " characters are colored correctly but the words between the " " characters remain uncolored.

Here is my TryGetClassificationType function (Note I am using Telerik Winforms in C++ but the C# code is very similar) :

01.bool TryGetClassificationType(String^ word, ClassificationType^ %classificationType) override {
02.    int number;
03.    if (int::TryParse(word, number)) {
04.        classificationType = ClassificationTypes::NumberLiteral;
05.        return true;
06.    }
07.
08.    if (word->Contains("\"")) {
09.        classificationType = ClassificationTypes::StringLiteral;
10.        return true;
11.    }
12.
13.    if (word->Contains("\'")) {
14.        classificationType = ClassificationTypes::StringLiteral;
15.        return true;
16.    }
17.
18.    return WordTaggerBase::TryGetClassificationType(word, classificationType);
19.};

 

And my tagger registration as follows:

01.RadSyntaxEditor^ newEditor = gcnew RadSyntaxEditor();
02.//Register the Syntax Highlighting Taggers
03.LuaTagger^ luaTagger = gcnew LuaTagger(newEditor->SyntaxEditorElement);
04.newEditor->TaggersRegistry->RegisterTagger(luaTagger);
05.//Set the Syntax Colors
06.newEditor->TextFormatDefinitions->AddLast(ClassificationTypes::Keyword, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 216, 160, 223)))); // Keywords
07.newEditor->TextFormatDefinitions->AddLast(ClassificationTypes::Comment, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 87, 166, 74)))); // Comments
08.newEditor->TextFormatDefinitions->AddLast(ClassificationTypes::NumberLiteral, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 181, 206, 168)))); //Numbers
09.newEditor->TextFormatDefinitions->AddLast(ClassificationTypes::StringLiteral, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 214, 157, 133)))); // String
10.newEditor->TextFormatDefinitions->AddLast(ClassificationTypes::Operator, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 180, 180 ,180)))); // Operators
11.newEditor->TextFormatDefinitions->AddLast(LuaTagger::ObjectsClassificationType, gcnew Telerik::WinForms::Controls::SyntaxEditor::UI::TextFormatDefinition(gcnew SolidBrush(Color::FromArgb(255, 78, 201, 176)))); // Objects

 

 

This yields the following result: https://gyazo.com/6331b12a57135c478bd3dccf66037fe8

As you can see, the " symbols are being correctly colored but the text in between is not. How would I get the text inbetween the two " " to be registered as a String Literal?

Thanks!

[Dess | Tech Support Engineer, Principal]

Hello, Zaven,

It seems that the provided code snippet shows just a part of the custom tagger that you have. That is why I used the sample code snippet from the referred help article and just focused on the implementation you have for the TryGetClassificationType method.

In order to highlight the text wrapped in quotes: "some string", I have prepared a sample code snippet for your reference demonstrating how to split the words. I would like to note that this is just a sample approach and it may not cover all possible cases. Feel free to modify and extend it in a way which suits your requirements best. Please have in mind that such a parsing implementation isn't be easy and it is developer's responsibility to implement the exact logic and cover the possible cases. 

        public RadForm1()
        {
            InitializeComponent();

            LuaTagger pythonTagger = new LuaTagger(this.radSyntaxEditor1.SyntaxEditorElement);
            if (!this.radSyntaxEditor1.TaggersRegistry.IsTaggerRegistered(pythonTagger))
            {
                this.radSyntaxEditor1.TaggersRegistry.RegisterTagger(pythonTagger);
            }
            this.radSyntaxEditor1.TextFormatDefinitions.AddLast(ClassificationTypes.Keyword, new TextFormatDefinition(new SolidBrush(Color.FromArgb(255, 216, 160, 223))));
            this.radSyntaxEditor1.TextFormatDefinitions.AddLast(ClassificationTypes.Comment, new TextFormatDefinition(new SolidBrush(Color.FromArgb(255, 87, 166, 74))));
            this.radSyntaxEditor1.TextFormatDefinitions.AddLast(ClassificationTypes.NumberLiteral, new TextFormatDefinition(new SolidBrush(Color.FromArgb(255, 181, 206, 168))));
            this.radSyntaxEditor1.TextFormatDefinitions.AddLast(ClassificationTypes.StringLiteral, new TextFormatDefinition(new SolidBrush(Color.FromArgb(255, 214, 157, 133))));
            this.radSyntaxEditor1.TextFormatDefinitions.AddLast(ClassificationTypes.Operator, new TextFormatDefinition(new SolidBrush(Color.FromArgb(255, 180, 180, 180)))); 
        }

        public class LuaTagger : WordTaggerBase
        {
            private static readonly Dictionary<string, ClassificationType> WordsToClassificationType = new Dictionary<string, ClassificationType>();

            public LuaTagger(RadSyntaxEditorElement editor) : base(editor)
            {
            }

            protected override Dictionary<string, ClassificationType> GetWordsToClassificationTypes()
            {
                return LuaTagger.WordsToClassificationType;
            }

            internal static int GetCharType(char c)
            {
                if (c == '#')
                {
                    return 0;
                }

                if (char.IsWhiteSpace(c))
                {
                    return 1;
                }

                if (c == '\"')
                {
                    return 2;
                }

                if ((char.IsPunctuation(c) || char.IsSymbol(c)))
                {
                    return 3;
                }

                return 0;
            }

            protected override IList<string> SplitIntoWords(string value)
            {
                List<string> words = new List<string>();
                string word;
                int lastCharType = -1;
                int startIndex = 0;
                for (int i = 0; i < value.Length; i++)
                {
                    int charType = GetCharType(value[i]);
                    if (lastCharType == 2)
                    {
                        int start = Math.Max(0, i - 1);
                        StringBuilder res = new StringBuilder();
                        res.Append(value[start]);

                        start = i;
                        while (start < value.Length)
                        {
                            res.Append(value[start]);
                            if (value[start] == '\"')
                            {
                                i = start + 1;
                                break;
                            }

                            start++;
                        }

                        words.Add(res.ToString());
                        startIndex = i;
                        lastCharType = charType;

                        continue;
                    }

                    if (charType != lastCharType)
                    {
                        word = value.Substring(startIndex, i - startIndex);
                        words.Add(word);
                        startIndex = i;
                        lastCharType = charType;
                    }
                }

                word = value.Substring(startIndex, value.Length - startIndex);
                words.Add(word);

                return words;
            }

            protected override bool TryGetClassificationType(string word, out ClassificationType classificationType)
            {
                int number;
                if (int.TryParse(word, out number))
                {
                    classificationType = ClassificationTypes.NumberLiteral;
                    return true;
                }

                if (word.Contains("\""))
                {
                    classificationType = ClassificationTypes.StringLiteral;
                    return true;
                }
                return base.TryGetClassificationType(word, out classificationType);
            }
        }

I hope this information helps. If you need any further assistance please don't hesitate to contact me. 

Regards,
Dess | Tech Support Engineer, Sr.
Progress Telerik

[Zaven Alexander]

After further testing I have found a bug which lies somewhere within the SplitIntoWords function where if you type two braces 

()

And then attempt to type a quote character between them like so:

(")

the program will crash on an out of Index Bounds. I've tried locating which line in the function is causing the problem but have been so far unable to do so. Perhaps you would be able to investigate?

[Dess | Tech Support Engineer, Principal]

Hello, Zaven,

I would like to note that the previously provided code snippet just demonstrates a sample approach how you can split the words in your custom WordTaggerBase. It is not built-in functionality in RadSyntaxEditor. Hence, it may not cover all possible cases. Feel free to modify and extend it in a way which suits your custom requirements best.

In order to properly detect where exactly the exception occurs, you need to enable the exceptions in Visual Studio. I believe that it would help you to troubleshoot further:

I have tested inserting "(")" in my sample project and it seems to work as expected. Please refer to the attached gif file illustrating the behavior on my end. I have also attached my sample project for your reference.

I hope you will find it helpful.

Regards,
Dess | Tech Support Engineer, Sr.
Progress Telerik

[Zaven Alexander]

Hi Dess,

Thank you for your help with this.

After going through the sample code you kindly provided I found a really silly mistake in my code which has partially fixed the problem. I'm still getting a crash however it only occurs when editing the last line within the syntax editor. Every other line works fine, as shown in the giff: https://gyazo.com/acbb1a70b440df371ef78d734ca96002

The exception is being triggered somewhere inside the Telerik.dll library and is triggered after the SplitIntoWords override function. It is an "ArgumentOutOfRangeException: Specified argument was out of the range of valid values." exception relating to the parameter "span".

The last item in the call-stack was 

 Telerik.WinForms.SyntaxEditor.Core.Text.TextSnapshotSpan.TextSnapshotSpan(...)

 

What I find interesting is that in the demo you provided this issue does not exist, and beyond styling the code is identical for the syntax editor. I will continue digging to see if I can track down what piece of logic is triggering the problem. Thank you for all your help, I know this goes a bit beyond the scope, but any ideas you may have would be most appreciated :)

Thanks again!

[Dess | Tech Support Engineer, Principal]

Hello, Zaven,

I believe that it would be helpful to attach the Telerik source to your project and further investigate what happens on your end. The following KB article is quite useful about setting up the source code: https://docs.telerik.com/devtools/winforms/knowledge-base/attach-telerik-source-code-to-your-project 

This would facilitate the investigation of the exact work flow of the project in this specific situation.

Should you have further questions please let me know.

Regards,
Dess | Tech Support Engineer, Sr.
Progress Telerik

[Zaven Alexander]

Thank you Dess!

I shall give that a go and see if I can isolate where in my code I've created this nasty bug!

In case I am unsuccessful and someone else reading this runs into the same problem I have found a band aid solution which is a little dirty but does technically solve the problem by bypassing it (Please note this is only meant as a temporary solution and does not solve the underlying problem).

Inside the DocumentContentChanged Event of the RadSyntaxEditor, adding this little bit of code will force a blank line to always exist at the end of the document and thus removing the option of writing on the final line of the document, hence the exception which is triggered when editing the last line of the document, doesn't get triggered:

 

private: System::Void SyntaxEditor_DocumentContentChanged(System::Object^ sender, Telerik::WinForms::SyntaxEditor::Core::Text::TextContentChangedEventArgs^ e) {

    /*
        The following is a nasty fix to stop the String Literal Syntax highlighting from crashing
        when editing the last line of a document. This forces there to always be an empty line at
        the end of the file.
    */

    //Check whether the last line has any text - if so append a new line
    RadSyntaxEditor^ editor = this->MySyntaxEditor;
    String^ documentText = editor->Document->CurrentSnapshot->GetText(); // get text
    cli::array<String^>^ linesText = documentText->Split('\n'); // split into separate lines

   //Check if the last line is blank - if its not we need to add a new blank line
    if (linesText[linesText->Length - 1] != "") {
        CaretPosition^ start = gcnew CaretPosition(editor->SyntaxEditorElement->CaretPosition);
        int lineNumber = start->DisplayLineNumber;
        editor->Document->Insert(documentText->Length, Environment::NewLine);
        editor->SyntaxEditorElement->CaretPosition->MoveToLine(lineNumber - 1);
        editor->SyntaxEditorElement->CaretPosition->MoveToLineEnd();
    }
}

 

I will post with the results of my expedition into the source code of Telerik once I have found the cause of the problem!

Thanks again!

[Dess | Tech Support Engineer, Principal]

Hi, Zaven,

I am glad that you have found a suitable solution for your scenario.

However, without replicating the issue locally, it wouldn't be easy to determine whether it is an issue in the core implementation of RadSyntaxEditor or it is something specific in the exact setup. Then, we wouldn't be able to address it properly in our source code.

Feel free to use your solution if it fits the custom scenario. It would be greatly appreciated if you share with us the  results of your expedition into the source code of Telerik whenever you have any news. Thank you in advance for your cooperation.

I am looking forward to your reply.

Regards,
Dess | Tech Support Engineer, Sr.
Progress Telerik

[Ken]

I ran into the same error, and the problem is if there is no closing " on the line. In that case start becomes value.Length, i is never incremented, and additional words (that shouldn't be added) are, which apparently causes issues. Here is my fix.

protected override IList<string> SplitIntoWords(string value)
{
    List<string> words = new List<string>();
    string word;
    int lastCharType = -1;
    int startIndex = 0;

    for (int i = 0; i < value.Length; i++)
    {
        int charType = GetCharType(value[i]);

        // Special handling for "..."
        if (lastCharType == 2)
        {
            int start = Math.Max(0, i - 1);
            string res = string.Empty;
            res += value[start];

            start = i;
            while (start < value.Length)
            {
                res += value[start];
                if (value[start] == '\"')
                {
                    //i = start + 1;
                    break;
                }

                start++;
            }

            words.Add(res);
            // Now it is possible that we got here without finding a matching '\"'
            // if so, we have collected all the characters, and we should just return
            i = start + 1;
            if (i >= value.Length)
            {
                return words;
            }

            startIndex = i;
            lastCharType = charType;

            continue;
        }

        if (charType != lastCharType)
        {
            word = value.Substring(startIndex, i - startIndex);
            words.Add(word);
            startIndex = i;
            lastCharType = charType;
        }
    }

    word = value.Substring(startIndex, value.Length - startIndex);
    words.Add(word);

    return words;
}

[Dess | Tech Support Engineer, Principal]

Hi, Ken,

Thank you for sharing your solution with the community. I believe that they will find it helpful.

I have also updated your Telerik points for community efforts.

 

We always strive to encourage our customers to share different useful ideas and approaches with each other.

Regards,
Dess | Tech Support Engineer, Sr.
Progress Telerik