Sequentially Crawl URLS WITH A DELAY.

2 posts, 0 answers
  1. What
    What avatar
    1 posts
    Member since:
    Sep 2015

    Posted 06 Sep 2015 Link to this post

    Not sure if these forums are active, but I tried the sequentially crawl URL script, however it dumps all the sequential requests at once as fast as the processor can handle it. Causing hundreds of requests every few seconds. How do I tell it to wait 1 second between requests?

     

    I've been trying to get this working with Thread.Sleep and Timers for the last 12 hours and I'm almost ready to throw my computer out the window.

     

     Here is the example script:

    public static ToolsAction("Crawl Sequential URLs")
        function doCrawl(){
          var sBase: String;
          var sInt: String;

          sBase = FiddlerObject.prompt("Enter base URL with ## in place of the start integer", "http://www.example.com/img##.jpg");
          sInt = FiddlerObject.prompt("Start At", "1");
          var iFirst = int.Parse(sInt);
          sInt = FiddlerObject.prompt("End At", "12");
          var iLast = int.Parse(sInt);

          for (var x=iFirst; x<=iLast; x++)
          {
            //Replace 's' with your HTTP Request. Note: \ is a special character in JScript
            // If you want to represent a backslash in a string constant, double it like \\
            var s = "GET " + sBase.Replace("##", x.ToString()) + " HTTP/1.0\r\n\r\n";
            var b=false;
            while(!b){
            try{
              FiddlerObject.utilIssueRequest(s);
              b=true;
            }
            catch(e){
              var iT = Environment.TickCount + 10000;
              FiddlerObject.StatusText = "Waiting 10 sec because we have too many requests outstanding...";
              while (iT > Environment.TickCount){ Application.DoEvents(); }
              }
            }
          }
        }

  2. Eric Lawrence
    Admin
    Eric Lawrence avatar
    833 posts

    Posted 08 Sep 2015 Link to this post

    How many requests are you sending? Is there some reason you don't want to send them as quickly as you can?

    You can simply put a System.Threading.Thread.Sleep(1000); on the line after utilIssueRequest and the script will pause one second after each request is issued.

    Regards,
    Eric Lawrence
    Telerik
    Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
Back to Top