Skip to content

Rate limits and concurrency

AzScraper uses credits for crawl capacity and applies selective HTTP throttling on abuse-prone public routes. There is no single global “requests per second” cap on all authenticated API traffic today; limits below reflect current product behavior and admin-configurable defaults.

Credit-based usage

Crawl work is metered in credits, not raw HTTP calls:

Job type Credits per target
Search (keyword SERP) 5 per keyword
Product details (ASIN/URL) 2 per ASIN or URL

Credits are reserved when you create or retry a job. Top-up rate: $1 = 10,000 credits (see marketing pricing). New accounts receive a signup bonus (currently 1,000 credits).

If the organization balance is too low, job creation returns 402 with INSUFFICIENT_CREDITS.

Queue and worker concurrency

  1. Create job — API validates the request, reserves credits, persists the job, and enqueues work on Redis (BullMQ).
  2. Worker — Background workers pull jobs from the queue and execute crawls. Multiple organizations share worker capacity fairly; large backlogs may increase queue wait time without failing the API request.
  3. Status — Poll GET /jobs/:id or use webhooks (when configured) for completion. Job states include queued, running, completed, failed, and cancelled.

There is no hard per-organization “N simultaneous running jobs” limit documented in the public API contract today. Practical concurrency is bounded by:

  • available credits
  • queue depth and worker capacity
  • upstream crawler rate limits (admin-configurable; default reference 60 requests/minute per crawler settings in admin)

Schedule cron jobs enqueue like manual jobs; they do not bypass the queue.

HTTP rate limits

Surface Limit (current) Notes
Authenticated private API Not globally throttled Use reasonable polling intervals (e.g. job status every few seconds, not sub-second loops)
POST /referral/click 5 requests / 10 seconds / IP Returns 429 RATE_LIMIT_EXCEEDED
Admin API Operator-configurable Defaults: API 600/min, crawler 60/min (admin settings)

If we introduce broader API throttling, it will be documented here and returned as 429 with RATE_LIMIT_EXCEEDED.

  • Batch targets in one job when possible instead of many tiny jobs.
  • Poll job status with backoff (2s → 5s → 10s) until terminal state.
  • Use export (GET /jobs/:id/export) after completion instead of paginating results aggressively during running.
  • Handle 402 by surfacing a top-up path; do not retry job creation in a tight loop.
  • Handle 429 with exponential backoff and jitter.

Usage limits endpoint

GET /usage returns monthly organization limits vs consumption (when enabled for your plan):

{
  "data": {
    "limits": { "jobsPerMonth": 10, "targetsPerMonth": 100, "exportsPerMonth": 20 },
    "used": { "jobsThisMonth": 3, "targetsThisMonth": 40, "exportsThisMonth": 2 },
    "remaining": { "jobsPerMonth": 7, "targetsPerMonth": 60, "exportsPerMonth": 18 }
  }
}

Exact limits depend on your organization configuration.

Tiếng Việt: Giới hạn tốc độ & đồng thời (VI)