Rate Limits and Quotas

The Eternity library acts as a wrapper for the Google Custom Search JSON API. Consequently, your usage is governed by the quotas and rate limits defined by Google Cloud. Understanding these limits is essential for ensuring reliable data retrieval in your applications.

Google Custom Search API Limits

By default, the Google Custom Search API provides a mix of free and paid usage tiers:

Once you exceed the 100-query daily limit on the free tier, the API will return an error, and the Eternity methods will fail to retrieve new data until the quota resets (typically at midnight Pacific Time) or billing is enabled.

Handling Quota Errors

When a rate limit is reached or a quota is exceeded, the search() and get_combined_text() methods will output the status code returned by Google.

Common status codes related to limits include:

429 Too Many Requests: You have sent too many requests in a given amount of time (Rate Limiting).
403 Forbidden: You have exhausted your daily quota or the API key is invalid.

To handle these gracefully, you should wrap your calls in logic that checks for empty results:

from eternity import Eternity

client = Eternity(api_key="YOUR_API_KEY", engine_id="YOUR_ENGINE_ID")
results = client.get_combined_text("Python automation")

if not results:
    # This may indicate a quota exhaustion or network error
    print("No data retrieved. Check your Google Cloud Console quota.")

Managing High-Volume Requests

If your application requires more than 100 searches per day, consider the following strategies:

1. Enable Billing

To increase your limit beyond 100 queries/day, you must enable billing in the Google Cloud Console. This allows for up to 10,000 queries per day per project.

2. Implement Caching

Since Eternity performs real-time web requests, repetitive queries for the same terms will consume your quota quickly. We recommend implementing a local caching layer (such as sqlite3 or redis) to store results for a specific duration.

3. Request Throttling

If you are running Eternity in a loop or a multi-threaded environment, implement a delay between requests to avoid triggering temporary rate blocks.

import time
from eternity import Eternity

client = Eternity(api_key="API_KEY", engine_id="ENGINE_ID")
queries = ["Query 1", "Query 2", "Query 3"]

for q in queries:
    data = client.search(q)
    print(f"Retrieved {len(data)} results for {q}")
    # Short delay to respect API stability
    time.sleep(1)

Monitoring Usage

You can monitor your real-time usage and configure alerts for quota thresholds through the APIs & Services > Dashboard section of your Google Cloud Console. This is the most accurate way to track how many requests your Eternity implementation is consuming.