fix: Prevent accidental request dropping with maxRequestsPerCrawl

[janbuchar]

• closes maxRequestsPerCrawl with RQ optimizations drops requests #3153

• core issue: maxRequestsPerCrawl and enqueueLinks({ limit }) pre-truncated request lists before sending them to the request queue. Duplicate URLs consumed budget slots, starving actual new URLs.

• Fix: Instead of guessing upfront which requests will be new, let the request queue tell us.

  • A new maxNewRequests option on addRequestsBatched feeds requests to the queue in budget-capped chunks, counts how many were actually new from the wasAlreadyPresent results, and stops pulling once the budget is exhausted.
  • chunkedAsyncIterable was extended to accept a dynamic size callback
  • Leftovers from the source iterator are returned as requestsOverLimit for callers to report.

[barjin]

Thank you @janbuchar !

Here are some late-night ideas I got (otherwise it looks pretty solid):

[l2ysho]

The generator instance problem highlighted by @barjin is valid point, check example test suite in thread. My other finding is probably just a nit, feel free to resolve as you like.

[l2ysho]

lgtm

[barjin]

lgtm, thank you!