Proxy bug with Transfer-Encoding 'chunked'

[Dan Humphrey]

Hi,

I use the proxy a lot to inject a javascript into the head of a page, which mostly works fine, but there are some troublesome scenarios, one of which is possibly a bug or my misunderstanding of how best to use the proxy.

1. When the response Transfer-Encoding is set to 'chunked' and the Content property is retrived from the response, the hex chunk sizes are found in the the Content - is this a bug?

An example can be seen when you Intercept the response from this URL:  http://www.bbc.co.uk/search/?q=Music

Here's the code I use to read and write the content (for a non compressed response):

public class ProxyContentHandlerPlain : IProxyContentHandler
    {
        public String readContent(byte[] content)
        {
            using (var stream = new System.IO.MemoryStream(content))
            {
                using (var reader = new StreamReader(stream,Encoding.Default))
                {
                    return reader.ReadToEnd();
                }
            }
           
        }

        public byte[] writeContent(string content)
        {
            return Encoding.Default.GetBytes(content);
        }
    }

When writing the Content back into the response, you will see the hex chunk sizes in the page.

I have been able overcome this when the encoding is not compressed, by a creative regex to find the hex chunk sizes and parse the source chunk by chunk - this should really be accomodated for within the proxy and/or respose class?

2. Related to abvove, when the Transfer-Encoding is 'chunked' and the Content-Encoding is 'gzip' or 'deflate' I have not bee able to parse the content from the response class. An example URL which has this scenario is: http://www.bing.com/search?q=Music&go=&form=QBLH&filt=all&qs=n&sk=

Any advice on how to overcome this?

Here's how I currently read and write gzip compressed content:

class ProxyContentHandlerGzip : IProxyContentHandler
    {
        public string readContent(byte[] content)
        {
            using (var stream = new System.IO.MemoryStream(content))
            {
                using (var zs = new GZipStream(stream, CompressionMode.Decompress))
                {
                    using (var reader = new StreamReader(zs,Encoding.Default))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
        }

        public byte[] writeContent(string content)
        {
            byte[] data = Encoding.Default.GetBytes(content);
            using (var stream = new MemoryStream())
            {
                using (var zs = new GZipStream(stream, CompressionMode.Compress))
                {
                    zs.Write(data, 0, data.Length);
                    return stream.ToArray();
                }
            }
            
        }
    }

In general, I'd love to see these sort of things handled better within the proxy and response class - I think it should be smarter and allow me to simply read and write the content from the response class - would you agree?

Any help or advice would be appreciated.

Thanks,

Dan

[Cody]

Hello Dan Humphrey,

To answer your questions:

When the response Transfer-Encoding is set to 'chunked' and the Content property is retrived from the response, the hex chunk sizes are found in the the Content - is this a bug? - Technically no, more like a feature request. Our proxy is not currently setup to dechunk/decompress the HTTP stream. Thus you are getting and seeing the chunk header.

this should really be accomodated for within the proxy and/or response class? Yes and we have a feature request logged to handle this very thing.

Related to abvove, when the Transfer-Encoding is 'chunked' and the Content-Encoding is 'gzip' or 'deflate' I have not bee able to parse the content from the response class. - Looking at your code it appears you must first dechunk the stream before you can decompress it. It isn't a valid GZIP/Deflate string prior to dechunking because you still have the chunk headers in place.

Greetings,
Cody
the Telerik team

[Dan Humphrey]

Cody,

Thanks for your reply.

The problem I have with the Transfer-Encoding is that this header doesn't appear in the response headers when using the WebAii proxy - I get all other headers except this one, so I don't really know if it's chunked or not - the response appears to be just gzipped.. When I view the net response in Firebug I can see that it does in fact have the Transfer-Encoding header.

I'm struggling with this whole scenario - my solution works 80% of the time, but the nuances of HTTP in cases like this one from the bing website (gzipped content encoding and chunked transfer encoding) aren't working.

I look forward to your official solutiion baked into the framework :) - are your feature requests visible, so that I can keep track of progress or do you have any idea on a timeline?

Thanks again.

Dan

[Cody]

Hello Dan Humphrey,

I spoke to one my of software developers about this issue. It appears this is a bug within the framework. I have files PITS ID: 5390 to track this issue. At this time I cannot provide an ETA when this problem will be addressed.

Regards,
Cody
the Telerik team

[Dan Humphrey]

Thanks Cody, I'll keep an eye on the release notes.

[Cody]

Hi Dan Humphrey,

I have an update for you on this. In our upcoming R2 release, due out next week, we've completely rewritten the HTTP proxy. Please give it a try and your convenience once released.

All the best,
Cody
the Telerik team

Check out Telerik Trainer, the state of the art learning tool for Telerik products.

[Dan Humphrey]

Hi Cody,

Thanks for letting me know.

I'm kinda hoping that the proxy now automatically handles the complexities of encoding and chunking, but would be happy if the bug is fixed too.

I look forward to trying the new release.

Thanks,

Dan