Mikko Ohtamaa

Sometimes, moderately complex Python applications with several threads tend to hang on exit. The application refuses to quit and just idles there waiting for something. Often this is because if any of the Python threads are alive when the process tries to exit it will wait any alive thread to terminate, unless Thread.daemon is set to true. In the past, it used to be little painful to figure out which thread and function causes the application to hang, but no longer! Since Python 3.3 CPython interpreter comes with a faulthandler module. faulthandler is a mechanism to tell the Python interpreter to dump the stack trace of every thread upon receiving an external UNIX signal. Here is an example how to figure out why the unit test run, executed with pytest, does not exit cleanly. All tests finish, but the test suite refuses to quit. First we run the tests and set a special environment variable PYTHONFAULTHANDLER telling CPython interpreter to activate the fault handler. This environment variable works regardless how your Python application is started (you run python command, you run a script directly, etc.). PYTHONFAULTHANDLER=true py.test And then the test suite has finished, printing out the last dot… but nothing happens despite our ferocious sipping of coffee. dotdotdotmoredotsthenthenthedotsstopappearing .. How to proceed: Press CTRL-Z to suspend the current active process in UNIX shell. Use the following command to send SIGABRT signal to the suspended process. kill -SIGABRT %1 Voilá – you get the traceback. In this case, it instantly tells SQLAlchemy is waiting for something and most likely the database has deadlocked due to open conflicting transactions. atal Python error: Aborted Thread 0x0000000103538000 (most recent call first): File "/opt/local/Library/Fra% meworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 154 in _eintr_retry File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/socketserver.py", line 236 in serve_forever File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/functional.py", line 40 in run File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 921 in _bootstrap_inner File "/opt/local/Library/Frameworks/Python.framework/Versions/3.4/lib/python3.4/threading.py", line 889 in _bootstrap Current thread 0x00007fff75128310 (most recent call first): File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/engine/default.py", line 442 in do_execute ... File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/SQLAlchemy-1.0.0b5-py3.4-macosx-10.9-x86_64.egg/sqlalchemy/sql/schema.py", line 3638 in drop_all File "/Users/mikko/code/trees/pyramid_web20/pyramid_web20/tests/conftest.py", line 124 in teardown ... File "/Users/mikko/code/trees/venv/lib/python3.4/site-packages/_pytest/config.py", line 41 in main File "/Users/mikko/code/trees/venv/bin/py.test", line 9 in

this blog post presents rolling time window counting and rate limiting in redis. you can apply it to activate login captcha on your site only when it is needed. for the syntax highlighted python source code please see the original blog post . table of contents 1. about redis 2. rollingwindow.py: 3. problematic captchas 4. captchas and different login situations 5. mitigating botnet-driven login attack with on-situation captcha 6. captchamode.py 1. about redis redis is a key-value store and persistent cache. besides normal get/set functionality it offers more complex data structures like lists, hashes and sorted sets. if you are familiar with memcached think redis as memcached with steroids. often redis is used for rate limiting purposes . usually the rate limit recipes are count how many times something happens on a certain second or a certain minute. when the clock ticks to the next minute, rate limit counter is reset back to the zero. this might be problematic if you are looking to limit rates where hits per integration time window is very low. if you are looking to limit to the five hits per minute, in one time window you get just one hit and six in another, even though the average over two minutes is 3.5. this posts presents an python example how to do a rolling time window based counting, so that rate counting does not reset itself back to the zero in any point, but counts hits over x seconds to the past. this is achieved using redis sorted sets . 2. rollingwindow.py: if you know any better way to do this with redis – please let me know – i am no expert here. this is the first implementation i figured out. """ redis rolling time window counter and rate limit. use redis sorted sets to do a rolling time window counters and limiters. http://redis.io/commands/zadd """ import time def check(redis, key, window=60, limit=50): """ do a rolling time window counter hit. :param redis: redis client :param key: redis key name we use to keep counter :param window: rolling time window in seconds :param limit: allowed operations per time window :return: true is the maximum limit has been reached for the current time window """ # expire old keys (hits) expires = time.time() - window redis.zremrangebyscore(key, '-inf', expires) # add a hit on the very moment now = time.time() redis.zadd(key, now, now) # if we currently have more keys than limit, # then limit the action if redis.zcard(key) > limit: return true return false def get(redis, key): """ get the current hits per rolling time window. :param redis: redis client :param key: redis key name we use to keep counter :return: int, how many hits we have within the current rolling time window """ return redis.zcard(key) 3. problematic captchas everybody of us hates captchas . they are two-edged swords. on one hand, you need to keep bots out from your site. on the other, captchas are turn off for your site visitors and they drive away potential users. even though the most popular captcha-as-a-service, google’s recaptcha, has made substantial progress to make captchas for real visitors and hard for bots , captchas still present a usability problem. also in the case of recaptcha, javascript and image assets are loaded from google front end services and they tend to get blocked in china, disabling your site for chinese visitors . 4. captchas and different login situations there are three cases where you want the user to complete captcha for login somebody is bruteforcing a single username (targeted attack): you need to count logins per usename and not let the login proceed if this user is getting too many logins. somebody is going through username/password combinations for a single ip: you count logins per ip. somebody is going through username/password combinations and the attack comes from very large ip pool. usually these are botnet-driven attacks and the attacker can easily have tens of thousands of ip addresses to burn. the botnet-driven login attack is tricky to block. there might be only one login attempt from each ip. the only way to effectively stop the attack is to present pre-login captcha i.e. the user needs to solve the captcha even before the login can be attempted. however pre-login captcha is very annoying usability wise – it prevents you to use browser password manager for quick logins and sometimes gives you extra headache of two minutes before you get in to your favorite site. even services like cloudflare do not help you here. because there is only one request per single ip, they cannot know beforehand if the request is going to be legitimate or not (though they have some global heurestics and ip blacklists for sure). you can flip on the “challenge” on your site, so that every visitors must complete the captcha before they can access your site and this is usability let down again. 5. mitigating botnet-driven login attack with on-situation captcha you can have the best of the both worlds: no login captcha and still mitigate botnet-driven login atttacks. this can be done by monitoring your site login rate in normal situation do not have pre-login captcha when there is clearly an abnormal login rate, which means there might be an attack going on, enable the pre-login captcha for certain time below is an pseudo-python example how this can be achieved with using rollingwindow python module from the above. 6. captchamode.py from redis_cache import get_redis_connection import rollingwindow #: redis sorted set key counting login attempts redis_login_attempts_counter = "login_attempts" #: key telling that captcha become activated due to #: high login attempts rate redis_captcha_activated = "captcha_activated" #: captcha mode expires in 120 minutes (attack cooldown) captcha_timeout = 120 * 60 #: are you presented captcha when logging in first time #: disabled in unit tests. login_attempts_challenge_threshold = 500 # per minute def clear(): """ resets the challenge system state, per system or per ip. """ redis = get_redis_connection("redis") redis.delete(redis_captcha_activated) redis.delete(redis_login_attempts_counter) def get_login_rate(): """ :return: system global login rate per minute for metrics """ redis = get_redis_connection("redis") return rollingwindow.get(redis, redis_login_attempts_counter) def check_captcha_needed(redis): """ check if we need to enable login captcha globally. increase login page load/submit counter. :return: true if our threshold for login page loads per minute is exceeded """ # count a hit towards login rate threshold_exceeded = rollingwindow.check(redis, redis_login_attempts_counter, limit=login_attempts_challenge_threshold) # are we in attack mode if not redis.get(redis_captcha_activated): if not threshold_exceeded: # no login rate threshold exceeded, # and currently captcha not activated -> # allow login without captcha return false # login attempt threshold exceeded, # we might be under attack, # activate captcha mode redis.setex(redis_captcha_activated, "true", captcha_timeout) return true def login(request): redis = get_redis_connection("redis") if check_captcha_needed(request): # ... we need to captcha before this login can proceed .. else: # ... allow login to proceed without captcha ...

This blog post is a part of Mobilizing websites with responsive design and HTML5 tutorial. For all posts please see the Introduction post. iPhone might not be the market leader, but its users are known to be the most picky of phone folk. Thus, always make sure your mobile site works in iPhone. Here are tips for testing your site for iPhone. 1. iPhone / iPad Simulator iPhone/iPad/iOS emulator comes with iOS SDK. The SDK access costs 100 USD per year and the SDK only available on OSX platforms only. You basically must have a Mac to do any iOS development, there is no way around it. There exist “Hackintosh” attempts for running OSX in a virtual machine or non-Apple hardware, but these methods tend to be slow and cumbersome or have various software issues. Mac Mini is the cheapest Mac you can get for this work and is perfectly sufficient. The simulator is good for testing basic mobile sites. If you are relying on animations and Javascript then you need to test on actual device(s) too, because these two are subsystems which seem to differ between the simulator and the actual device mainly because of timing and racing conditions. To get iOS SDK access register yourself at http://developer.apple.com Pro tip: Alt+mouse press does the pinch and zoom gesture in the simulator, so you can test your web page zooming features. There is no fully simulated support for touch gestures, though. iPhone Simulator in action Though Apple assumes you run iPhone simulator through XCode you can also run it separately. Just keep the simulator icon in Dock. 2. iOS devices If you cannot afford iPhone you can always buy iPod Touch which comes with the same Mobile Safari software as iPhone. There practically exist no differences between iPhone and iPod Touch browser experience – only the lack of GPS might cause some issue if you are developing location based services. iPad also has the same WebKit browser engine, but it has different screen resolution and some other slightly differential behavior, so it is not suitable for your mobile site testing needs. But remember that most “mobile” tricks like CSS3 media queries or viewport tag apply on iPads too, so you might accidentally break your site on tables when building mobile version. On the right 2nd gen iPod Touch. It is no longer valid for testing as it did not receive iOS 5 update. If you are buying an used iPod Touch for testing make sure its the latest model. 3. Console Mobile Safari has a simple console for reading log messages on the simulator and iOS devices. You can enable it from the device Settings. 4. Easy screenshots from iOS simulator Below is a link to a screenshot application to take cropped screenshots from iOS simulator. The screenshots in this tutorial where taken using this application. http://www.curioustimes.de/iphonesimulatorcropper/

Amazon Route 53 offers DNS as a hosted service. They maintain robust DNS service for you with easy to user web interface and API. Name servers are part of any serious hosting infrastructure. They have been traditionally run on BIND or similar UNIX daemons. The usual hosting tasks including mapping domain names and subdomains (A, CNAME, et. al records) to servers and delivering email (MX and SPF records). Usually a basic hosting plan is only good for setting A records for few subdomains like www, but on the instant you need something more advanced like routing MX records to Google Apps email you need to start considering a real DNS service. GoDaddy and other big name registrants provide tools for this, but they are usually specific only for top level domains (TLDs) the registrant supports. If you come from a cold country like .fi you cannot centralize your DNS management to big popular domain registrants. The bad thing with running your own name servers is that when the service goes down the whole your infrastructure becomes virtually inaccessible – web sites, email, anything. Thus you usually run at least two BIND daemons on two physically separated servers just in case one blows up the other can still keep DNS records running. Also DNS, being core part of internet infrastructure, is subject for many kind of attacks and keeping your skills and knowledge up-to-date with BIND may be time consuming. Like with email servers, hosting name servers is a thing you really don’t want to do. Enter Amazon Route 53. (tha name Route 53 probably comes from the famous U.S. road Route 67 and DNS port 53?). Route 53 is ridiculous easy to use. Just go to the site and register one for yourself. However if you have legacy BIND servers running around there exist ways to import data automatically instead of manually reading through zone files and typing them in to Route 53 by hand. 1. Setting up cli53 cli53 is a Python based command line interface for Route 53. It uses buildout for installation (of Python package dependencies). These instructions have been tested with Ubuntu 8.04 server. Prerequisitements for using this stuff is knowing basic Ubuntu server management from the command line. Run everything as root. Install wget and unzip apt-get install wget unzip Download ZIP copy of cli53 from Github (we don’t do git clone here – no need to install git on the server): wget --no-check-certificate https://github.com/barnybug/cli53/zipball/master Extract unzip master Install as user local under the extracted folder (note that the folder varies across GitHub exports) cd barnybug-cli53-3b468b7 python bootstrap.py bin/buildout This will generate the local bin/cli53 command. Now you need to set up AWS credentials. Route 53, and other AWS service, are operated using access keys which you can get from Security Credentials page in AWS web console. Set the access keys for your shell/SSH session using enviroment variables: export AWS_ACCESS_KEY_ID=XXXXXXXXXXXXXX export AWS_SECRET_ACCESS_KEY=XXXXXXXXXXXXXXXXXXXXXXXXXXXXX Side note: as this point you need to check that your server clock is in correct time your AWS remote operations will fail (300 seconds thresold).. Use date command to see the server clock time and correct it with the same date command using this obscure date command syntax. Now you can run a test by listing your Amazon Route 53 Hosted Zones (Hosted Zone = zonefile = one per top level domain usually) using list command: bin/cli53 list The proper output for one domain which has been created throught-the-web in Route 53 control panel is something like: HostedZones: - CallerReference: 9265CCC3-9C41-98CE-8820-E26C0356C478 Config: Comment: test Id: /hostedzone/Z2L3NT0WCS8OWA Name: xxx.fi. 2. Importing zone files Now when cli53 is running and working it is time to rock’n'roll. On our server configuration BIND zonefiles lived in /srv/dns/zones though this is not the default location for Ubuntu BIND. There is one file per each zone and the filename is the domain name. [root@foobar][23:02][/srv/dns/zones]# ls -l total 176 -rw-r--r-- 1 root root 425 Dec 30 2008 abc.com -rw-r--r-- 1 root root 1713 Dec 30 2008 abcabc.com -rw-r--r-- 1 root root 1719 Dec 30 2008 abcabcabc.com We can simple create an one-liner shell script which will loop through all the files and import them to Amazon Route 53. However – there is a trick. You’ll get unknown origin exception if your zone files lack a certain line: Traceback (most recent call last): File "/root/barnybug-cli53-3b468b7/bin/cli53", line 22, in cli53.cli53.main() File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 495, in main args.func(args) File "/root/barnybug-cli53-3b468b7/src/cli53/cli53.py", line 268, in cmd_import raise Exception, 'Could not find origin' Exception: Could not find origin This is because the zone file doesn’t have $ORIGIN directive. it’s optional with BIND as BIND uses the filename as $ORIGIN if it’s not defined, but the directive is mandatory for cli53 (cli53 doesn’t yet synthetize $ORIGIN). You can simple add $ORIGIN to the beginning of each file and copy files to a working directory with the following shell one-liner (it’s always safe to make copies of processed files than try to fix them in place):./transfer/$i ; done mkdir ../transfer for i in * ; do echo "\$ORIGIN $i." > ../transfer/$i ; cat $i >> ../transfer/$i ; done Then do a test run for one import: ~/barnybug-cli53-3b468b7/bin/cli53 create xxx.fi ~/barnybug-cli53-3b468b7/bin/cli53 import xxx.fi --file xxx.fi --replace --wait Note: deleting zones in Route 53 control panel is painful difficult, so make sure you import zones you only really need. If everything looks good we are ok to upload everything to Route 53. Again, a shell one-liner does the trick for us: for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 create $i ; ~/barnybug-cli53-3b468b7/bin/cli53 import $i --file $i --wait ; done This will take abour 15-20 seconds per domain. Note: if you need to re-import add –replace flag. 3. The ugly part Each zone has Source of Authority record with authoritative nameserves (NS records). These will be change from your BIND server IP addresses to Amazon ones. Route 53 will re-assign its own name servers for each imported Hosted Zone. However, you cannot know beforehand what name servers the dice has chosen for your Hosted Zone, so automatizing this process is little bit difficult. NS records servers are overriden when the zone file is uploaded to Route 53. You can see new SOA and NS records when you choose the domain in Route 53 control panel and press Go to Record Sets. To make things easier later on here is a script which will dump all the name servers of all domains you have in Route 53 and collect are info to domain-info.txt file: for i in * ; do ~/barnybug-cli53-3b468b7/bin/cli53 info $i >> /tmp/domain-info.txt ; done 4. Testing Route 53 DNS Browser through imported domain records in Route 53 control panel and see if they look ok. You can also test the actual DNS functionality. When you use ping command, your web browser or any other “normal” mean which queries DNS records they use the Source of Authority name servers as set by in your domain registrant settings. When in the middle of migration to Route 53, the authorative name servers are still pointing to the old name servers in this point. dig command can be used to query a specific name server for DNS records. In Route 53 control panel pick any domain and any of its name servers Use dig command to query the records of this domain Example: dig -t ANY @ns-1340.awsdns-39.org yourdomainname.com Output should be something like this: ; <<>> DiG 9.4.2-P1 <<>> -t ANY @ns-1340.awsdns-39.org yourdomain.com ; (1 server found) ;; global options: printcmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 9352 ;; flags: qr aa rd; QUERY: 1, ANSWER: 10, AUTHORITY: 0, ADDITIONAL: 0 ;; WARNING: recursion requested but not available ;; QUESTION SECTION: ;yourdomain.com. IN ANY ;; ANSWER SECTION: yourdomain.com. 172800 IN NS ns-470.awsdns-58.com. yourdomain.com. 172800 IN NS ns-553.awsdns-05.net. yourdomain.com. 172800 IN NS ns-1340.awsdns-39.org. yourdomain.com. 172800 IN NS ns-1706.awsdns-21.co.uk. yourdomain.com. 900 IN SOA ns-1340.awsdns-39.org. awsdns-hostmaster.amazon.com. 1 7200 900 1209600 86400 yourdomain.com. 0 IN MX 10 aspmx.l.google.com. yourdomain.com. 0 IN MX 20 alt1.aspmx.l.google.com. yourdomain.com. 0 IN MX 20 alt2.aspmx.l.google.com. yourdomain.com. 0 IN TXT "v=spf1 a mx a:smtp.something.fi a:auth-smtp.something.fi include:aspmx.googlemail.com ~all" yourdomain.com. 0 IN A 84.20.128.49 ;; Query time: 32 msec ;; SERVER: 205.251.197.60#53(205.251.197.60) ;; WHEN: Wed Nov 23 00:04:53 2011 ;; MSG SIZE rcvd: 413 (subdomains are not listed) 5. Finalizing it Go to your domain name registrant and update the domain name servers point to your Route 53 servers. For each domain in domain-info.txt Go to the corresponding registrant from whom you obtained the domain (e.g. domain.ficora.fi) Update the authoritative name servers to be as stated in the file Wait 24-48 hours or whatever time-to-live time you have for your name servers – usually this is longish period See that your services still run Take down your old name servers 6. More info http://hathology.com/2011/02/how-to-import-bind-zone-files-into-amazon-route-53/ http://docs.amazonwebservices.com/Route53/latest/DeveloperGuide/index.html?MigratingDNS.html#Step_UpdateRegistrar