Amazon S3 FileBrowserContentProvider

39 posts, 0 answers
  1. Telerik Admin
    Telerik Admin avatar
    1679 posts
    Member since:
    Oct 2004

    Posted 14 Oct 2009 Link to this post

    Requirements

    RadControls version

     

    RadControls for ASP.NET AJAX 
    Q3 2009 SP1
    .NET version

     

    .NET20 & .NET35
    Visual Studio version

     

    2005/2008
    programming language

     

    C# / VB.NET
    browser support

    all browsers supported by RadControls


    PROJECT DESCRIPTION
    Connecting the RadFileExplorer control to the Amazon S3 storage.

    HOW TO USE THE PROVIDER
    The bucket's name is hardcoded. The credentials for accessing Amazon service need to be set to the
     awsAccessKeyId and awsSecretAccessKey read-only properties.

    The paths set to the ViewPaths, DeletePaths and UploadPaths properties follow this logic :
    If we have this bucket : http://bucket.s3.amazonaws.com/, then in the provider, the BUCKET_NAME read-only property needs to be set to "bucket" value. The paths should point to the virtual folders in this bucket.
    Two folders for example :
    • http://bucket.s3.amazonaws.com/folder1
    • http://bucket.s3.amazonaws.com/folder2
    string[] paths = new string[] { "folder1/", "folder2/" };
    RadFileExplorer1.Configuration.ViewPaths = paths;
    RadFileExplorer1.Configuration.DeletePaths = paths;
    RadFileExplorer1.Configuration.UploadPaths = paths;
    RadFileExplorer1.Configuration.ContentProviderTypeName = typeof(S3ContentProvider).AssemblyQualifiedName;

    This C# Library for Amazon S3 is used in the attached project.

    NOTE
    In the earlier versions of the control the CheckDeletePermissions method does not exist and needs to be removed.

    UPDATE:
    As suggested by Bonnie, you can consider trying the TNTDrive tool, allowing you to map an S3 bucket as a network drive. Once this is done you can create a virtual directory in the IIS application root which can be accessed after that with a standard virtual path (with ~/).
  2. Daniel Frumkin
    Daniel Frumkin avatar
    19 posts
    Member since:
    Jun 2004

    Posted 02 Nov 2009 Link to this post

    wow!  just googled this looking for a solution-  way to go Telerik- always pushing it!.

    Question- is this solution pretty well tested?   any problems?
  3. Daniel Frumkin
    Daniel Frumkin avatar
    19 posts
    Member since:
    Jun 2004

    Posted 02 Nov 2009 Link to this post

    Any chance you could deploy a CustomFileSystemPRovider.vb file?

  4. ldc0618
    ldc0618 avatar
    8 posts
    Member since:
    Jun 2007

    Posted 30 Nov 2009 Link to this post

    This is a great start, but it appears to only support reading S3 files. What about uploading, copying, deleting, etc?

    Thanks!
  5. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 02 Dec 2009 Link to this post

    Hello Lane,

    The implemented provider supports all the features - delete, upload, move etc. You are the first that reports such behavior and I am not quite sure what is causing the problems on your case. Could you please provide me with a little bit more information about the behavior in your case? Do you experience any errors?

    I am looking forward to hearing from you.

    Kind regards,
    Fiko
    the Telerik team

    Instantly find answers to your questions on the new Telerik Support Portal.
    Watch a video on how to optimize your support resource searches and check out more tips on the blogs.
  6. Duane Faithen
    Duane Faithen avatar
    14 posts
    Member since:
    Jun 2005

    Posted 28 Dec 2009 Link to this post

    I want to thank you for putting this together. I think it is going to save me considerable time in my task at hand.

    3 things:
    1. Was one of the above download links supposed to be for a c# version?
    2. Is it possible to retrieve subdirectories on demand instead of on load? I would like to be able to use this against our s3 structure, except we currently have about 10000 documents nested in thousands of folders. It takes several minutes to load this entire structure up front and since our users are typically after just one document at a time it is unnecessary overhead. Just wondering if this is already a capability I am missing before I start modifying the provider. As well, it seemsd like even though it builds the whole structure up fromt, it rebuilds each nodes child structure on demand when you click that node. This seems to really make the up front structure load redundant and unnecessary.  - this only seems an issue with FileExplorerMode.Default
    3. There seems to be a bug in the directory listings when the directoryName has an & in it. Rather than listing that directories children under it, it relists that directories siblings as children underneath it.

    Thanks

  7. Duane Faithen
    Duane Faithen avatar
    14 posts
    Member since:
    Jun 2005

    Posted 28 Dec 2009 Link to this post

    The following seems to fix item 3 I mentioned above.
    In GetDirectories(), change

    string directoryFullPath = VirtualPathUtility.AppendTrailingSlash(realDirectoryPath) + GetDirectoryName(key); 
    to
    string directoryFullPath = VirtualPathUtility.AppendTrailingSlash(realDirectoryPath) + HttpUtility.UrlEncode(GetDirectoryName(key)); 

    I have not completely tested it yet but Im posting it for everyone anyways.
  8. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 29 Dec 2009 Link to this post

    Hi Duane,

    In reference to your questions:
    1. I have attached the C# version of the example as well. Thank you for bringing this to our attention.
    2. By design the folders in RadFileExplorer are loaded on demand (the control does not load all folders). When the page is loaded for the first time only the root folders and their first child folders are loaded (if the InitialPath property is not set). When you click on a child folder then the control initiates asynchronous requests and loads the sub-folders of the clicked folder. This approach is used in order to get the first child items of a folder from the S3 store.
      If you have more files in a folder, then the control will load slowly and this is a normal behavior. In order to optimize loading I recommend you apply one of these approach:
      • Set the ExplorerMode property of the RadFileExplorer control to FileTree.
        As you already know, this will improve the loading performance. The reason for this is that the data-binding is done on the server not the client. When the FileExplorerMode.Default is set the client side DataBinding feature of the grid is used, which is slower than the server-side binding.
      • Set AllowPaging="true" property in order to turn on paging in the grid. In this case only 100 (this is the default value and cannot be changed) items will be shown in the grid.
    3. The problem comes from the fact that the used library's converts the paths in a format that is URL encoded (you can check the result of the StreamResponseToXmlDocument method). Indeed, the solution is to use the approach from your last post.

    I hope this helps.


    Sincerely yours,
    Fiko
    the Telerik team

    Instantly find answers to your questions on the new Telerik Support Portal.
    Watch a video on how to optimize your support resource searches and check out more tips on the blogs.
  9. Duane Faithen
    Duane Faithen avatar
    14 posts
    Member since:
    Jun 2005

    Posted 29 Dec 2009 Link to this post

    Thanks for your response Fiko.
    In your # 2 comments you mentioned "if the initalpath property is not set." I cant seem to set this when accessing an s3 instance because it doesnt allow relative paths. How would I go about setting this when using the s3 provider?

    Also another problem I seem to be having. In the file explorer itself my credentials work and all seems well, but when I try to open a file I am getting the error "The request signature we calculated does not match the signature you provided. Check your key and signing method." I am using exactly the same credentials in the contentprovider and in the filesystemhandler, yet one works and one doesnt. Has anyone else encountered this?

    Thanks
  10. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 04 Jan 2010 Link to this post

    Hello Duane,

    I am not quite sure what is causing the problems on your side. For example this setting works and the value passed to the InitialPath property is applied correctly:
    string[] viewPaths = new string[] { "testFolder1/", "testFolder2/" };
    string[] deleteUploadPaths = new string[] { "testFolder1/NewFolder/Folder_2" };
    RadFileExplorer1.Configuration.ViewPaths = viewPaths;
    RadFileExplorer1.Configuration.DeletePaths = deleteUploadPaths;
    RadFileExplorer1.Configuration.UploadPaths = deleteUploadPaths;
    RadFileExplorer1.Configuration.ContentProviderTypeName = typeof(S3ContentProvider).AssemblyQualifiedName;
     
    RadFileExplorer1.InitialPath = "testFolder1/NewFolder/Flower333.jpg"

    After the page is loaded for the first time, the Flower333.jpg file becomes the selected file.
    The second issue can be caused by the fact that the BUCKET_NAME is not specified in the ASHX handler. Also I recommend you copy (again) the awsAccessKeyId and awsSecretAccessKey values used in the content provider to the ASHX handler as well.

    I hope this helps.

    Regards,
    Fiko
    the Telerik team

    Instantly find answers to your questions on the new Telerik Support Portal.
    Watch a video on how to optimize your support resource searches and check out more tips on the blogs.
  11. tomas
    tomas avatar
    15 posts
    Member since:
    Jan 2007

    Posted 31 Jan 2010 Link to this post

    This is a great solution however I seem to be having one small problem. The Delete Directory does not seem to be working from the Toolbar. There is no postback or events fired when a directory is selected and delete is clicked from the toolbar. I can however delete the directory from the context menu.

    Any suggestions?
  12. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 03 Feb 2010 Link to this post

    Hi Tomas,

    The described problem does not exists in the attached (to the code library) example projects. Could you please modify one of them (C# or VB.NET), open a new support ticket and send it to me? I will check it and do my best to provide a working solution as soon as possible.


    Kind regards,
    Fiko
    the Telerik team

    Watch a video on how to optimize your support resource searches and check out more tips on the blogs.
    Follow the status of features or bugs in PITS and vote for them to affect their priority.
  13. Joshua Holmes
    Joshua Holmes avatar
    2 posts
    Member since:
    Jan 2010

    Posted 24 Mar 2010 Link to this post

    The S3 Content Provider is great, however, I am using RackSpace CloudFiles (Mosso), and I understand there is no pre-made Provider for CloudFiles, so I am creating one.. However it would be REALLLY great to provide a blank provider solution.

    Basically all NonImplemented methods that we can implement our selves rather then having to download the S3 provider and tearing all kinds of junk out of it that we don't need or that is S3 specific, because there is a lot... A simple outlined file with the mandatory methods that RadFileExplorer will utilize and then we can decide how that should be implemented..

    Thanks!
  14. Lini
    Admin
    Lini avatar
    2144 posts

    Posted 25 Mar 2010 Link to this post

    Hello,

    We have an article on how to create a content provider in our documentation - http://www.telerik.com/help/aspnet-ajax/radfileexplorer-custom-filebrowsercontentprovider.html.

    The first article explains how to subclass an existing provider (if you need only a slightly different behavior), while the second part shows how to create an empty provider from scratch.

    All the best,
    Lini
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  15. John Hamman
    John Hamman avatar
    10 posts
    Member since:
    Feb 2006

    Posted 28 Apr 2010 Link to this post

    Great provider!

    One issue I can't seem to wrap my mind around yet.

    when I click on a file to download it, it trys to download the provider.ashx file.

    I did make one change to the coding which is I changed the file name and extension from S3FileSystemHandler.ashx
    to S3FileSystemHandler.jsx.
    I have the jsx extension mapped to run as .ashx and works fine with other hanlders that are renamed.
    Can someone help me understand why its trying to download the hanlder file and not the real file?
    John
  16. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 30 Apr 2010 Link to this post

    Hello John,

    I am not quite sure what is causing the problem on your side. By default the .ashx files are handled by the ASP.NET framework and these files are not opened for download. Could you please test the same project on a different machine? Also, for testing purposes, I recommend you to remove all Telerik controls from the page and check whether the problem persists. If the problem disappears, could you please open a new support ticket and send me the project which reproduces the problem (including web.config as well)? I will check it and do my best to provide a working solution as soon as possible.


    Sincerely yours,
    Fiko
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  17. Glenn
    Glenn avatar
    1 posts
    Member since:
    Oct 2008

    Posted 25 May 2010 Link to this post

    I was so glad to find this post. I tried the sample code with no changes (except ID's and bucket name). I don't see any subdirectories. I understand the concept that there are no folders in s3 but I was hoping that this control did that work for me. Should I be seeing subdirectories? Thanks in advance.
  18. Fiko
    Admin
    Fiko avatar
    1406 posts

    Posted 28 May 2010 Link to this post

    Hi Glenn,

    The code works and I am not sure what is causing the problem on your side. Could you please make sure that you set correct paths and credentials, as described in the description of the codelibrary?

    Regards,
    Fiko
    the Telerik team

    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Public Issue Tracking system and vote to affect the priority of the items.
  19. Alankar
    Alankar avatar
    1 posts
    Member since:
    Feb 2011

    Posted 11 Feb 2011 Link to this post

    Hi All,
    I am Alankar srivastava.
    I have currently used ASp.net with c#
    I need a sample code of follwoing featues:
    1. Upload
    2. Download
    3. Delete bucket
    4. Delete File

    Thanks in advance
    Alankar Srivastava
  20. Tony
    Tony avatar
    3 posts
    Member since:
    Jan 2013

    Posted 18 Mar 2013 Link to this post

    Hello,

    Are there any plans to update this Content Provider to work with the newest AWS SDK?
  21. Vessy
    Admin
    Vessy avatar
    1380 posts

    Posted 19 Mar 2013 Link to this post

    Hi Tony,

    I am afraid that we do not have plans to update this content provider to work with the newest AWS SDK.  Nevertheless, if you decide to update it by yourself, feel free to post the custom solution you have implemented in the code library section on our site. You will be rewarded with some Telerik points for the effort which can be used as a discount for future purchases/upgrades of our controls.

    Kind Regards,
    Vesi
    the Telerik team
    If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed now.
  22. Mike Bridge
    Mike Bridge avatar
    96 posts
    Member since:
    Oct 2005

    Posted 02 Apr 2013 Link to this post

    Has anyone made any progress on updating to the AWS SDK?

    I'm having a look at implementing this now, but thought I'd check to see if anyone had done it already.
  23. Roger
    Roger avatar
    5 posts
    Member since:
    Aug 2007

    Posted 22 Apr 2013 Link to this post

    This content provider is great!  Got me up and running pretty much "out of the box".  I do need advice on thing...is there anyway to use this provider to do a search of files on S3.  I know the S3 API does not offer a search method, but I wonder if there is anyway that I can leverage ListBucket or similar through your already powerful FileBrowserContentProvider?

    Thanks in advance!
  24. Vessy
    Admin
    Vessy avatar
    1380 posts

    Posted 25 Apr 2013 Link to this post

    Hi Roger,

    I am afraid that since the Search functionality is not supported by the S3 API itself we cannot manipulate it externally. I could suggest you to try the following approaches to achieve a similar functionality, so you could decide whether one of them would fit your scenario:
    • You could implement a Filter field in your FileExplorer, like it is described in this live demo: Filter files and folders. The main limitation here is that it would search only among the items in the current directory.
    • You could call recursively the ResolveDirectoryAsTree() and ResolveDirectory() methods from your content provider in order to take references to all listed files and folders. Once you gather all the needed data to one place you can make a search through the items in it. Despite of the accuracy of such an implementation, note that this information-gathering would be a slow process, and you  might meet performance issues.

    I hope this information is helpful for you.

    Kind regards,
    Veselina Raykova
    the Telerik team
    If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to their blog feed now.
  25. Roger
    Roger avatar
    1 posts
    Member since:
    Apr 2013

    Posted 05 Jun 2013 Link to this post

    I am assuming that this may be the same issue, but should I expect Configuration.SearchPatterns to work with the S3 FileBrowserContentProvider?  Or does this fall into the same thing as the previous question about searching?

    Thanks!
  26. Vessy
    Admin
    Vessy avatar
    1380 posts

    Posted 10 Jun 2013 Link to this post

    Hello Roger,

    The S3 FileBrowserContentProvider does not have a SearchPattern logic implemented, but you can extend the GetFiles() method of the provider in a similar way:
    private FileItem[] GetFiles(string realDirectoryPath)
    {
        List<FileItem> filesResult = new List<FileItem>();
        List<XmlNode> responseResult = ListBucket(realDirectoryPath);
        foreach (XmlNode content in responseResult)
        {
            string key = GetKeyNode(content).InnerText;
            if (!key.EndsWith("_$folder$"))
            {
                // This is a file, not a folder
                string size = GetSizeNode(content).InnerText;
                string url = (ItemHandlerPath + "?path=") + HttpUtility.UrlEncode(key);
                string fileName = VirtualPathUtility.GetFileName("/" + key);
                FileItem fileItem = new FileItem(fileName, VirtualPathUtility.GetExtension(key), int.Parse(size), string.Empty, url, string.Empty, GetPermissions(key));
                 
                foreach (var filter in SearchPatterns)
                {
                    if (filter.Substring(2) == fileName.Substring(fileName.LastIndexOf(".") + 1))
                    {
                        filesResult.Add(fileItem);
                    }
                }
            }
        }
        return filesResult.ToArray();
    }


    Kind regards,
    Veselina Raykova
    Telerik
    If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed now.
  27. Mike Bridge
    Mike Bridge avatar
    96 posts
    Member since:
    Oct 2005

    Posted 17 Jul 2013 Link to this post

    I'm having a problem with paging---the paging interface is functional (i.e. you can click on buttons, and drag the handle on the bar) but no matter what page I'm on, I only see the first 100 files.

    I verified that my GetFiles[] indeed returns the full list of several thousand files, but for some reason the list doesn't change when I page through.

    Any idea where I can look to solve this problem?  


            public override DirectoryItem ResolveDirectory(string path)
            {

                path = CanonicalPathHelper.NormalizeDirectoryPath(path);

                //Console.WriteLine("Looking at " + path);

                FileItem[] filesResult = GetFiles(path);

                // The directories will be added in the ResolveRootDirectoryAsTree method
                // this seems to return the directory requested too.
                return new DirectoryItem(GetDirectoryName(path),
                                         string.Empty,
                                         path,
                                         path,
                                         GetPermissions(path),
                                         filesResult,
                                         null);
            }

            private FileItem[] GetFiles(string path)
            {
                path = CanonicalPathHelper.NormalizeDirectoryPath(path);
                List<FileItem> filesResult = new List<FileItem>();
                try
                {
                    var client = CreateS3Client();
                    Console.WriteLine("Checking Path " + path);

                    var request = ListBucketRequest(path);
                    int i = 0;
                    do
                    {
                        ListObjectsResponse response = client.ListObjects(request);

                        // Files
                        filesResult.AddRange(response.S3Objects.Where(x => IsFile(x.Key)).Select(
                            s3Obj =>
                            new FileItem(
                                s3Obj.Key,
                                VirtualPathUtility.GetExtension(s3Obj.Key),
                                s3Obj.Size,
                                string.Empty,
                                GetUrl(s3Obj.Key, true),
                                //string.Empty,
                                i++.ToString(),
                                GetPermissions(s3Obj.Key)
                                )
                            )
                        );

                        if (response.IsTruncated)
                        {
                            // next page of results
                            request.Marker = response.NextMarker;
                        }
                        else
                        {
                            request = null;
                        }
                    } while (request != null);
                }
                catch (AmazonS3Exception amazonS3Exception)
                {
                    if (amazonS3Exception.ErrorCode != null &&
                        (amazonS3Exception.ErrorCode.Equals("InvalidAccessKeyId")
                         ||
                         amazonS3Exception.ErrorCode.Equals("InvalidSecurity")))
                    {
                        Console.WriteLine("Check the provided AWS Credentials.");
                    }
                    else
                    {
                        Console.WriteLine(
                            "Error occurred. Message:'{0}'",
                            amazonS3Exception.Message);
                    }
                    return null;
                }

                return filesResult.ToArray();
            }

  28. Vessy
    Admin
    Vessy avatar
    1380 posts

    Posted 18 Jul 2013 Link to this post

    Hi Mike,

    I have already answered your support ticket on the subject, for convenience I will write my answer here as well.

    Most probably the experienced behavior is connected with permission limitations. Could you test what is the result of the CheckReadPermissions() method of the provider with the passed by you path?
    public override bool CheckReadPermissions(string folderPath)
    {
        return base.CheckReadPermissions(folderPath);
    }

    Could you override it, by setting its return value to true, and see if the problem still occurs?
    public override bool CheckReadPermissions(string folderPath)
    {
        return true;
    }

    Regards,
    Veselina Raykova
    Telerik
    If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed now.
  29. Mike Bridge
    Mike Bridge avatar
    96 posts
    Member since:
    Oct 2005

    Posted 18 Jul 2013 Link to this post

    Hi Veselina,

    Yes, that seems to be the problem, thanks!

    A related question is: is there any way to get the currently requested page number?  Right now I'm retrieving the entire directory every time a paging event occurs---I was wondering if there's a way to make it more efficient.

    Thanks!

    -Mike
  30. Vessy
    Admin
    Vessy avatar
    1380 posts

    Posted 23 Jul 2013 Link to this post

    Hello Mike,

    I am afraid that with the current implementation of FileExplorer the desired functionality cannot be achieved out-of-he-box. The control nested in RadFileExplorer, which is responsible for the paging, is RadGrid and it needs to load all items from the current folder in order to calculate how many pages will be needed to list them.

    Nevertheless, if you decide implement your custom logic, implementing the wanted behavior, you could see a way to access the current page number and size in this KB article: Apply changes to the files on the current page

    Kind regards,
    Veselina Raykova
    Telerik
    If you want to get updates on new releases, tips and tricks and sneak peeks at our product labs directly from the developers working on the RadControls for ASP.NET AJAX, subscribe to the blog feed now.
Back to Top