This is a migrated thread and some comments may be shown as answers.

Sequentially Crawl URLS WITH A DELAY.

1 Answer 425 Views
Extensions and Customization
This is a migrated thread and some comments may be shown as answers.
What
Top achievements
Rank 1
What asked on 06 Sep 2015, 12:02 PM

Not sure if these forums are active, but I tried the sequentially crawl URL script, however it dumps all the sequential requests at once as fast as the processor can handle it. Causing hundreds of requests every few seconds. How do I tell it to wait 1 second between requests?

 

I've been trying to get this working with Thread.Sleep and Timers for the last 12 hours and I'm almost ready to throw my computer out the window.

 

 Here is the example script:

public static ToolsAction("Crawl Sequential URLs")
    function doCrawl(){
      var sBase: String;
      var sInt: String;

      sBase = FiddlerObject.prompt("Enter base URL with ## in place of the start integer", "http://www.example.com/img##.jpg");
      sInt = FiddlerObject.prompt("Start At", "1");
      var iFirst = int.Parse(sInt);
      sInt = FiddlerObject.prompt("End At", "12");
      var iLast = int.Parse(sInt);

      for (var x=iFirst; x<=iLast; x++)
      {
        //Replace 's' with your HTTP Request. Note: \ is a special character in JScript
        // If you want to represent a backslash in a string constant, double it like \\
        var s = "GET " + sBase.Replace("##", x.ToString()) + " HTTP/1.0\r\n\r\n";
        var b=false;
        while(!b){
        try{
          FiddlerObject.utilIssueRequest(s);
          b=true;
        }
        catch(e){
          var iT = Environment.TickCount + 10000;
          FiddlerObject.StatusText = "Waiting 10 sec because we have too many requests outstanding...";
          while (iT > Environment.TickCount){ Application.DoEvents(); }
          }
        }
      }
    }

1 Answer, 1 is accepted

Sort by
0
Eric Lawrence
Telerik team
answered on 08 Sep 2015, 09:23 PM
How many requests are you sending? Is there some reason you don't want to send them as quickly as you can?

You can simply put a System.Threading.Thread.Sleep(1000); on the line after utilIssueRequest and the script will pause one second after each request is issued.

Regards,
Eric Lawrence
Telerik
Do you want to have your say when we set our development plans? Do you want to know when a feature you care about is added or when a bug fixed? Explore the Telerik Feedback Portal and vote to affect the priority of the items
Tags
Extensions and Customization
Asked by
What
Top achievements
Rank 1
Answers by
Eric Lawrence
Telerik team
Share this question
or