Python Rate Limit Decorator Example

Copy Link LinkedIn Message Facebook Email X

Now that you've learned how to build a decorator with arguments, you might wonder what real-world applications for it are. Well, glad that you asked! Let's return to fetching data from the web and see how a decorator can be useful there.

With APIs

Imagine that you're using an API to retrieve some information from the Internet, and you want to avoid hitting it too often because the data provider has implemented some rate limiting. Or maybe you've built your own web scraper, and you don't want to put too much pressure on the pages that you're scraping.

Decorator for Max Requests

You could define a decorator that takes an argument for a maximum of requests and raises an exception if that maximum is exceeded:

from time import sleep

class RateLimitError(Exception):
    """Rate limit exceeded."""

def rate_limit(max_requests):
    def decorator(func):
        request_count = 0

        def wrapper(*args, **kwargs):
            nonlocal request_count
            if request_count >= max_requests:
                raise RateLimitError(func.__name__)
            else:
                request_count += 1
            return func(*args, **kwargs)

        return wrapper

    return decorator

You defined a custom exception, RateLimitError, that you'll raise when the wrapped function reaches the maximum number of requests.

Then you define a decorator that takes one argument, max_requests. The value for max_requests can be different for each function that you'll decorate! Most of the structure of the decorator might feel familiar by now.

One notable exception is that you declare the counter you use, request_count, in wrapper as a nonlocal variable using the nonlocal statement. This allows you to write to request_count that you defined in the enclosing scope. If you would define request_count directly in wrapper(), then you'd keep overriding the value with 0 every time you'd call a decorated function.

Benefits of a Decorator

By defining it in decorator() and as a nonlocal inside of wrapper(), you can use it as a counter for all executions of a decorated function.

To apply the rate-limiting decorator, you can now decorate any of your data fetching functions and limit the number of times that they can get called before the RateLimitError gets raised:

@rate_limit(4)
def get_user_data():
    # Your API code to get user data from a page
    print(f"Got user data")


@rate_limit(3)
def scrape_comment_page(url):
    # Your web scraping code to scrape comments from a page
    print(f"Scraped comments from {url}")

You defined that get_user_data() has a maximum of 4 retries before it should raise an exception, and scrape_comment_page() should already raise the exception after 3 retries.

If you attempt to call any of the functions more often than that, then Python will raise the custom RateLimitError you defined further up:

for _ in range(6):
    try:
        get_user_data()
        scrape_comment_page("www.example.com")
    except RateLimitError as re:
        print(f"(x) {re.__doc__} (function: {re})")

Here, you're calling both functions 6 times, and you're handling the exception that you're raising so that your script can continue to execute. If either of the functions raises the RateLimitError, then you'll print the error message and the function context to the console.

If you run this loop, then you'll see that the functions execute successfully as many times as the value you passed for max_requests when decorating each function:

Got user data
Scraped comments from www.example.com
Got user data
Scraped comments from www.example.com
Got user data
Scraped comments from www.example.com
Got user data
(x) Rate limit exceeded. (function: scrape_comment_page)
(x) Rate limit exceeded. (function: get_user_data)
(x) Rate limit exceeded. (function: get_user_data)

You can see that get_user_data() ran four times successfully, while scrape_comment_page() only ran three times successfully. After the third execution of scrape_comment_page(), Python raised the RateLimitError and printed a message to your terminal.

The for loop continued execution and bumped into the RateLimitError two more times, now both already when attempting to call get_user_data(), which had already exhausted its maximum of four calls.

Because get_user_data() already raises the exception, the code never even gets to scrape_comment_page() during these last two runs of the for loop. However, because the counter of that function has reached max_requests as well, it would raise the error as well.

Summary: Python Decorators with Arguments Examples

• The @rate_limit decorator was created to limit the number of times a function can be called
• The @rate_limit decorator takes the max_requests parameter, which represents the maximum number of function calls allowed.
• Any function decorating with @rate_limit can only be called max_requests number of times.
• When the limit is exceeded, Python raises the custom RateLimitError exception