DZone
Thanks for visiting DZone today,
Edit Profile
  • Manage Email Subscriptions
  • How to Post to DZone
  • Article Submission Guidelines
Sign Out View Profile
  • Post an Article
  • Manage My Drafts
Over 2 million developers have joined DZone.
Log In / Join
Please enter at least three characters to search
Refcards Trend Reports
Events Video Library
Refcards
Trend Reports

Events

View Events Video Library

Zones

Culture and Methodologies Agile Career Development Methodologies Team Management
Data Engineering AI/ML Big Data Data Databases IoT
Software Design and Architecture Cloud Architecture Containers Integration Microservices Performance Security
Coding Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks
Culture and Methodologies
Agile Career Development Methodologies Team Management
Data Engineering
AI/ML Big Data Data Databases IoT
Software Design and Architecture
Cloud Architecture Containers Integration Microservices Performance Security
Coding
Frameworks Java JavaScript Languages Tools
Testing, Deployment, and Maintenance
Deployment DevOps and CI/CD Maintenance Monitoring and Observability Testing, Tools, and Frameworks

The software you build is only as secure as the code that powers it. Learn how malicious code creeps into your software supply chain.

Apache Cassandra combines the benefits of major NoSQL databases to support data management needs not covered by traditional RDBMS vendors.

Generative AI has transformed nearly every industry. How can you leverage GenAI to improve your productivity and efficiency?

Modernize your data layer. Learn how to design cloud-native database architectures to meet the evolving demands of AI and GenAI workloads.

Related

  • Scaling in Practice: Caching and Rate-Limiting With Redis and Next.js
  • Integrating Redis With Message Brokers
  • Build a Data Analytics Platform With Flask, SQL, and Redis
  • Performance and Scalability Analysis of Redis and Memcached

Trending

  • Navigating and Modernizing Legacy Codebases: A Developer's Guide to AI-Assisted Code Understanding
  • Navigating Change Management: A Guide for Engineers
  • Dropwizard vs. Micronaut: Unpacking the Best Framework for Microservices
  • Building an AI/ML Data Lake With Apache Iceberg
  1. DZone
  2. Software Design and Architecture
  3. Microservices
  4. Rolling Time Window Counters with Redis and Mitigating Botnet-Driven Login Attacks

Rolling Time Window Counters with Redis and Mitigating Botnet-Driven Login Attacks

By 
Mikko Ohtamaa user avatar
Mikko Ohtamaa
·
Jul. 10, 14 · Interview
Likes (0)
Comment
Save
Tweet
Share
13.2K Views

Join the DZone community and get the full member experience.

Join For Free

this blog post presents rolling time window counting and rate limiting in redis. you can apply it to activate login captcha on your site only when it is needed. for the syntax highlighted python source code please see the original blog post .

table of contents

1. about redis

2. rollingwindow.py:

3. problematic captchas

4. captchas and different login situations

5. mitigating botnet-driven login attack with on-situation captcha

6. captchamode.py

1. about redis

redis-small

redis is a key-value store and persistent cache. besides normal get/set functionality it offers more complex data structures like lists, hashes and sorted sets. if you are familiar with memcached think redis as memcached with steroids.

often redis is used for rate limiting purposes . usually the rate limit recipes are count how many times something happens on a certain second or a certain minute. when the clock ticks to the next minute, rate limit counter is reset back to the zero. this might be problematic if you are looking to limit rates where hits per integration time window is very low. if you are looking to limit to the five hits per minute, in one time window you get just one hit and six in another, even though the average over two minutes is 3.5.

this posts presents an python example how to do a rolling time window based counting, so that rate counting does not reset itself back to the zero in any point, but counts hits over x seconds to the past. this is achieved using redis sorted sets .

2. rollingwindow.py:

if you know any better way to do this with redis – please let me know – i am no expert here. this is the first implementation i figured out.

"""

    redis rolling time window counter and rate limit.

    use redis sorted sets to do a rolling time window counters and limiters.

    http://redis.io/commands/zadd

"""

import time


def check(redis, key, window=60, limit=50):
    """ do a rolling time window counter hit.

    :param redis: redis client

    :param key: redis key name we use to keep counter

    :param window: rolling time window in seconds

    :param limit: allowed operations per time window

    :return: true is the maximum limit has been reached for the current time window
    """

    # expire old keys (hits)
    expires = time.time() - window
    redis.zremrangebyscore(key, '-inf', expires)

    # add a hit on the very moment
    now = time.time()
    redis.zadd(key, now, now)

    # if we currently have more keys than limit,
    # then limit the action
    if redis.zcard(key) > limit:
        return true

    return false


def get(redis, key):
    """ get the current hits per rolling time window.

    :param redis: redis client

    :param key: redis key name we use to keep counter

    :return: int, how many hits we have within the current rolling time window
    """
    return redis.zcard(key)

3. problematic captchas

everybody of us hates captchas . they are two-edged swords. on one hand, you need to keep bots out from your site. on the other, captchas are turn off for your site visitors and they drive away potential users.

even though the most popular captcha-as-a-service, google’s recaptcha, has made substantial progress to make captchas  for real visitors and hard for bots , captchas still present a usability problem. also in the case of recaptcha, javascript and image assets are loaded from google front end services and they tend to get blocked in china, disabling your site for chinese visitors .

4. captchas and different login situations

there are three cases where you want the user to complete captcha for login

  • somebody is bruteforcing a single username (targeted attack): you need to count logins per usename and not let the login proceed if this user is getting too many logins.
  • somebody is going through username/password combinations for a single ip: you count logins per ip.
  • somebody is going through username/password combinations and the attack comes from very large ip pool. usually these are botnet-driven attacks and the attacker can easily have tens of thousands of ip addresses to burn.

the botnet-driven login attack is tricky to block. there might be only one login attempt from each ip. the only way to effectively stop the attack is to present pre-login captcha i.e. the user needs to solve the captcha even before the login can be attempted. however pre-login captcha is very annoying usability wise – it prevents you to use browser password manager for quick logins and sometimes gives you extra headache of two minutes before you get in to your favorite site.

even services like cloudflare do not help you here. because there is only one request per single ip, they cannot know beforehand if the request is going to be legitimate or not (though they have some global heurestics and ip blacklists for sure). you can flip on the “challenge” on your site, so that every visitors must complete the captcha before they can access your site and this is usability let down again.

5. mitigating botnet-driven login attack with on-situation captcha

you can have the best of the both worlds: no login captcha and still mitigate botnet-driven login atttacks. this can be done by

  • monitoring your site login rate
  • in normal situation do not have pre-login captcha
  • when there is clearly an abnormal login rate, which means there might be an attack going on, enable the pre-login captcha for certain time

below is an pseudo-python example how this can be achieved with using rollingwindow python module from the above.

6. captchamode.py

from redis_cache import get_redis_connection

import rollingwindow


#: redis sorted set key counting login attempts
redis_login_attempts_counter = "login_attempts"

#: key telling that captcha become activated due to
#: high login attempts rate
redis_captcha_activated = "captcha_activated"

#: captcha mode expires in 120 minutes (attack cooldown)
captcha_timeout = 120 * 60

#: are you presented captcha when logging in first time
#: disabled in unit tests.
login_attempts_challenge_threshold = 500  # per minute


def clear():
    """ resets the challenge system state, per system or per ip. """
    redis = get_redis_connection("redis")
    redis.delete(redis_captcha_activated)
    redis.delete(redis_login_attempts_counter)


def get_login_rate():
    """
    :return: system global login rate per minute for metrics
    """
    redis = get_redis_connection("redis")
    return rollingwindow.get(redis, redis_login_attempts_counter)


def check_captcha_needed(redis):
    """ check if we need to enable login captcha globally.

    increase login page load/submit counter.

    :return: true if our threshold for login page loads per minute is exceeded
    """

    # count a hit towards login rate
    threshold_exceeded = rollingwindow.check(redis, redis_login_attempts_counter, limit=login_attempts_challenge_threshold)

    # are we in attack mode
    if not redis.get(redis_captcha_activated):

        if not threshold_exceeded:
            # no login rate threshold exceeded,
            # and currently captcha not activated ->
            # allow login without captcha
            return false

        # login attempt threshold exceeded,
        # we might be under attack,
        # activate captcha mode
        redis.setex(redis_captcha_activated, "true", captcha_timeout)

    return true


def login(request):

    redis = get_redis_connection("redis")

    if check_captcha_needed(request):
        # ... we need to captcha before this login can proceed ..
    else:
        # ... allow login to proceed without captcha ...
Redis (company)

Published at DZone with permission of Mikko Ohtamaa, DZone MVB. See the original article here.

Opinions expressed by DZone contributors are their own.

Related

  • Scaling in Practice: Caching and Rate-Limiting With Redis and Next.js
  • Integrating Redis With Message Brokers
  • Build a Data Analytics Platform With Flask, SQL, and Redis
  • Performance and Scalability Analysis of Redis and Memcached

Partner Resources

×

Comments
Oops! Something Went Wrong

The likes didn't load as expected. Please refresh the page and try again.

ABOUT US

  • About DZone
  • Support and feedback
  • Community research
  • Sitemap

ADVERTISE

  • Advertise with DZone

CONTRIBUTE ON DZONE

  • Article Submission Guidelines
  • Become a Contributor
  • Core Program
  • Visit the Writers' Zone

LEGAL

  • Terms of Service
  • Privacy Policy

CONTACT US

  • 3343 Perimeter Hill Drive
  • Suite 100
  • Nashville, TN 37211
  • support@dzone.com

Let's be friends:

Likes
There are no likes...yet! 👀
Be the first to like this post!
It looks like you're not logged in.
Sign in to see who liked this post!