This is a migrated thread and some comments may be shown as answers.
Proxy bug with Transfer-Encoding 'chunked'
6 Answers 52 Views
This is a migrated thread and some comments may be shown as answers.
Dan Humphrey
Top achievements
Rank 1
Dan Humphrey asked on 16 Mar 2011, 04:40 AM
Hi,

I use the proxy a lot to inject a javascript into the head of a page, which mostly works fine, but there are some troublesome scenarios, one of which is possibly a bug or my misunderstanding of how best to use the proxy.

1. When the response Transfer-Encoding is set to 'chunked' and the Content property is retrived from the response, the hex chunk sizes are found in the the Content - is this a bug?

An example can be seen when you Intercept the response from this URL:  http://www.bbc.co.uk/search/?q=Music

Here's the code I use to read and write the content (for a non compressed response):

public class ProxyContentHandlerPlain : IProxyContentHandler
    {
        public String readContent(byte[] content)
        {
            using (var stream = new System.IO.MemoryStream(content))
            {
                using (var reader = new StreamReader(stream,Encoding.Default))
                {
                    return reader.ReadToEnd();
                }
            }
           
        }

        public byte[] writeContent(string content)
        {
            return Encoding.Default.GetBytes(content);
        }
    }

When writing the Content back into the response, you will see the hex chunk sizes in the page.

I have been able overcome this when the encoding is not compressed, by a creative regex to find the hex chunk sizes and parse the source chunk by chunk - this should really be accomodated for within the proxy and/or respose class?

2. Related to abvove, when the Transfer-Encoding is 'chunked' and the Content-Encoding is 'gzip' or 'deflate' I have not bee able to parse the content from the response class. An example URL which has this scenario is: http://www.bing.com/search?q=Music&go=&form=QBLH&filt=all&qs=n&sk=

Any advice on how to overcome this?

Here's how I currently read and write gzip compressed content:

class ProxyContentHandlerGzip : IProxyContentHandler
    {
        public string readContent(byte[] content)
        {
            using (var stream = new System.IO.MemoryStream(content))
            {
                using (var zs = new GZipStream(stream, CompressionMode.Decompress))
                {
                    using (var reader = new StreamReader(zs,Encoding.Default))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
        }

        public byte[] writeContent(string content)
        {
            byte[] data = Encoding.Default.GetBytes(content);
            using (var stream = new MemoryStream())
            {
                using (var zs = new GZipStream(stream, CompressionMode.Compress))
                {
                    zs.Write(data, 0, data.Length);
                    return stream.ToArray();
                }
            }
            
        }
    }

In general, I'd love to see these sort of things handled better within the proxy and response class - I think it should be smarter and allow me to simply read and write the content from the response class - would you agree?

Any help or advice would be appreciated.

Thanks,

Dan

6 Answers, 1 is accepted

Sort by
0
Cody
Telerik team
answered on 24 Mar 2011, 10:18 PM
Hello Dan Humphrey,

To answer your questions:

When the response Transfer-Encoding is set to 'chunked' and the Content property is retrived from the response, the hex chunk sizes are found in the the Content - is this a bug? - Technically no, more like a feature request. Our proxy is not currently setup to dechunk/decompress the HTTP stream. Thus you are getting and seeing the chunk header.

this should really be accomodated for within the proxy and/or response class? Yes and we have a feature request logged to handle this very thing.

Related to abvove, when the Transfer-Encoding is 'chunked' and the Content-Encoding is 'gzip' or 'deflate' I have not bee able to parse the content from the response class. - Looking at your code it appears you must first dechunk the stream before you can decompress it. It isn't a valid GZIP/Deflate string prior to dechunking because you still have the chunk headers in place.

Greetings,
Cody
the Telerik team
0
Dan Humphrey
Top achievements
Rank 1
answered on 31 Mar 2011, 10:46 PM
Cody,

Thanks for your reply.

The problem I have with the Transfer-Encoding is that this header doesn't appear in the response headers when using the WebAii proxy - I get all other headers except this one, so I don't really know if it's chunked or not - the response appears to be just gzipped.. When I view the net response in Firebug I can see that it does in fact have the Transfer-Encoding header.

I'm struggling with this whole scenario - my solution works 80% of the time, but the nuances of HTTP in cases like this one from the bing website (gzipped content encoding and chunked transfer encoding) aren't working.

I look forward to your official solutiion baked into the framework :) - are your feature requests visible, so that I can keep track of progress or do you have any idea on a timeline?

Thanks again.

Dan
0
Cody
Telerik team
answered on 05 Apr 2011, 12:22 AM
Hello Dan Humphrey,

I spoke to one my of software developers about this issue. It appears this is a bug within the framework. I have files PITS ID: 5390 to track this issue. At this time I cannot provide an ETA when this problem will be addressed.

Regards,
Cody
the Telerik team
Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items
0
Dan Humphrey
Top achievements
Rank 1
answered on 05 Apr 2011, 07:59 AM
Thanks Cody, I'll keep an eye on the release notes.
0
Cody
Telerik team
answered on 20 Sep 2011, 02:39 PM
Hi Dan Humphrey,

I have an update for you on this. In our upcoming R2 release, due out next week, we've completely rewritten the HTTP proxy. Please give it a try and your convenience once released.

All the best,
Cody
the Telerik team

Check out Telerik Trainer, the state of the art learning tool for Telerik products.
0
Dan Humphrey
Top achievements
Rank 1
answered on 21 Sep 2011, 02:42 AM
Hi Cody,

Thanks for letting me know.

I'm kinda hoping that the proxy now automatically handles the complexities of encoding and chunking, but would be happy if the bug is fixed too.

I look forward to trying the new release.

Thanks,

Dan
Asked by
Dan Humphrey
Top achievements
Rank 1
Answers by
Cody
Telerik team
Dan Humphrey
Top achievements
Rank 1
Share this question
or