[Silverlight Only] How to load UTF-7 CSV data when not supported by Silverlight 4+

[Graeme]

I had a problem with loading & decoding UTF-7 data from a CSV file into Silverlight's RADSpreadProcessing. Special characters like 'é' were being flagged as invalid by the Microsoft UTF-8 decoder in Silverlight. WPF on the other hand supports UTF-7 decoding.

With the help of Nikolay Demirev (Telerik support) he pointed me in the right direction with this Microsoft information: Understanding Encodings for Silverlight

Below is the solution that I came up with to auto-identify the encoding and handle UTF-7 encoded CSV files. I hope that it helps others.

    Sub ProcessData()

        Dim FormatProvider = New CsvFormatProvider

        With FormatProvider
            With .Settings
                .Delimiter = ","
                .Quote = """"
                .HasHeaderRow = True
                .Encoding = New System.Text.UnicodeEncoding
            End With
        End With

        Dim Filename As String = Path.Combine(Environment.GetFolderPath(Environment.SpecialFolder.MyDocuments), "Telerik Test.csv")
        Dim WB As Workbook = LoadWB(Filename, FormatProvider)

        '-- do something with the loaded data...

    End Sub

    Function LoadWB(FileName As String, FormatProvider As CsvFormatProvider) As Workbook

        If Not File.Exists(FileName) Then
            Throw New FileNotFoundException(String.Format("{0} Not Found.", FileName))
        End If

        Dim WB As Workbook

        If GetFileEncoding(FileName) Is Encoding.UTF8 Then
            WB = FormatProvider.Import(GetUTF8(FileName))
        Else
            Using Data As FileStream = New FileStream(FileName, FileMode.Open)
                WB = FormatProvider.Import(Data)
            End Using
        End If

        Return WB

    End Function

    Function GetUTF8(Filename As String) As String

        Dim encodedBytes() As Byte = File.ReadAllBytes(Filename)
        Dim enc8 As Encoding = New UTF8Encoding(False, True)
        Dim decodedString As String = String.Empty

        Try
            decodedString += enc8.GetString(encodedBytes, 0, encodedBytes.Length)
        Catch e As DecoderFallbackException

            Dim sb As New StringBuilder

            For Each b As Byte In encodedBytes
                sb.Append(Microsoft.VisualBasic.ChrW(Convert.ToInt32(b)))
            Next

            decodedString = sb.ToString()

        End Try

        Return decodedString

    End Function

    Public Shared Function GetFileEncoding(srcFile As String) As Encoding

        ' Use Default of Encoding.Default (Ansi CodePage)
        Dim enc As Encoding = Encoding.UTF8

        ' Detect byte order mark if any - otherwise assume default
        Dim buffer As Byte() = New Byte(4) {}
        Dim file As New FileStream(srcFile, FileMode.Open)
        file.Read(buffer, 0, 5)
        file.Close()

        If buffer(0) = &HEF AndAlso buffer(1) = &HBB AndAlso buffer(2) = &HBF Then
            enc = Encoding.UTF8
        ElseIf buffer(0) = &HFF AndAlso buffer(1) = &HFE Then
            enc = Encoding.Unicode
        ElseIf (buffer(0) = &HFE AndAlso buffer(1) = &HFF) Then
            enc = Encoding.BigEndianUnicode
        ElseIf buffer(0) = 0 AndAlso buffer(1) = 0 AndAlso buffer(2) = &HFE AndAlso buffer(3) = &HFF Then
            enc = Encoding.Unicode
        ElseIf buffer(0) = &H2B AndAlso buffer(1) = &H2F AndAlso buffer(2) = &H76 Then
            enc = Encoding.UTF8
        End If

        Return enc

    End Function