Military-grade bot detection: How to avoid bot detection Client Checks

How to avoid bot detection Client Checks

This section is all about the wonderful extensibility of javascript and the browser.
Client checks happen with a javascript library before any sensitive information is shown. Lot’s of metrics from browser performance, screen size and graphics rendering are all compiled into a nice little package and sent off to a validation server.

Bot detection script in the wild

So here’s a bot detection script out in the wild. Looks pretty neat right?

The code is obfuscated, but with a little reverse engineering, we can get something that looks a bit more like this. (function names added by me, implementation is just decoded)

After we dig through a few of these scripts, we can find a few categories of checks

  1. Browser property existance
  2. Math operation performance
  3. Font rendering
  4. System information (screen size, graphics card name etc)
  5. GPU rendering

How to masquerade browser properties

Detection scripts want to know if you’re a real browser. And if you don’t know better, there’s lots of obvious give-aways.

Browser automation tools tend to leave window properties to allow on-page scripts to reference the controller. Properties like window.webdriver are a clear indication that this is a scraping operation.

So how to do we “remove” properties. Well, a wonderful thing about javascript is the ability to create getters/setters.

Object.defineProperty(window, 'webdriver', {get: () => undefined})

tada! no more webdriver property.

How to hit Math Operation Performance checks

Detection scripts will often run things like the following

for(let i = 0; i < 1000000; i++) Math.cos(Math.rand()); 

The reason they do this for a few reasons:

  1. Scrapers are cost sensitive and want to use small VMs
  2. VMs typically don’t have GPUs, so browsers are forced into CPU rendering which creates a bottle neck that can be detected in javascript
  3. Regular consumers typically do have GPUs and sufficient CPU to run these math operations without much lag.

So, how do you get around this problem? Use a larger VM – with a GPU. In my experience, the GPU (even a small one) makes a HUGE difference on CPU load.

You can get fairly cheap GPU instances by using AWS workspaces (something around 30/mo at this writing).

How to bypass font rendering checks

If you were playing defence and wanted to check which fonts were present on a system how would you do it? Any ideas?

Well, I’ll give you a hint. Look at the width of these two W’s

Difference in width of two W’s

Yep, it’s that simple. Throw a bunch of W’s, or other wide character into a div, set the font-family and measure how wide it is. If the width matches what we’re expecting, the font must be installed. If the width matches the default or baseline measurement – no font installed. Here’s an some code I found in the wild that did this. (annotated for clarity)

Font detection script I found in the wild

So… how do we get around this? There’s an easy way and a hard way. Let’s see both:

First, the easy way to get around this is to have the fonts installed. Head over to your local package manager, find the fonts and get them fired up. On Linux there’s some fun things in /etc/fonts/conf.d, fc-cache and fc-list that will get you started on the right path.

But… the defensive team has thought of this. And they see a way to expose us.

They can test for fonts that SHOULDN’T be present too. For example, if we’re on a Windows system, Liberation and Dejavu fonts shouldn’t be installed. Those are Linux default fonts and there’s a 0.00001% of a Windows user having those installed.

So, that brings us to the second way to get around this. Did you see that detection script I posted above? Did you see how they’re measuring the size of the elements? It’s with HTMLElement.offsetWidth. Perfect – it’s something from javascript-land that we can override.

Object.defineProperty(window.HTMLElement.prototype, "offsetWidth", {
        get () { 
            // our font-detection-avoiding logic

Look familiuar?

Yep, we can override the offsetWidth method. When we see strange fonts on the HTMLElement, we can use a lookup table of character widths to calculate what the width would be if that font was installed – and more importantly, what the width would be if that font WAS NOT installed.

So… that’s font rendering

Avoiding system information detection (screen size, graphics card name etc)

Bot detection scripts also check for system properties that the browser exposes. You’re going to be surprised at the amount of data your browser exposes. Here’s the top 40 items (there are at least 200 individual parameters, I could go into more depth about all the fun tricks and turns in here)

  1. javascript version support (es version)
  2. browser extension list
  3. screen.availWidth
  4. screen.availHeight
  5. screen.width
  6. screen.height
  7. screen.availLeft
  8. screen.availTop
  9. screen.logicalXDPI
  10. screen.logicalYDPI
  11. screenLeft
  12. screenTop
  13. clientWidth
  14. clientHeight
  15. display-mode (fullscreen, standalone, minimal-ui, browser)
  16. navigator.language
  17. navigator.product
  18. function.prototype.bind
  19. DeviceOrientationEvent
  20. DeviceMotionEvent
  21. TouchEvent
  22. spawn
  23. chrome
  24. XMLHttpRequest
  25. XDomainRequest
  26. Buffer
  27. PointerEvent
  28. ActiveXObject
  29. screenY
  30. screenX
  31. navigator.vibrate
  32. getBattery
  33. bluetooth
  34. sendBeacon
  35. webkitTemporaryStorage
  36. screen.colorDepth
  37. navigator.platform
  38. navigator.buildID
  39. window.MediaKeyStatusMap
  40. GPU Version

So, how do we get around this?

It’s the same trick as before

Object.defineProperty(obj, "<prop>", {
        get () { 
            // our custom return value

Managing Browser Profiles

It can get difficult to manage all these properties. I ended up using a repository that gave had all the properties and reasonable combinations (e.g. common screen sizes that Macs support). Then I built a randomizer to build these properties into a profile and inject that into my sessions.

GPU Rendering

There’s actually a way that bot detection scripts can figure out what type of GPU you have, or at least what graphics libraries your application is using. SVG rasterizing.

When you make a nice drop shadow, some semi-transparent text or other complex rendering situation – your GPU/graphics library needs to make some decisions. It needs to decide how to merge layers, when to apply rounding, and at what resolution to perform the operation. Different GPUs and libraries do this differently.

So, the defense makes a very simple test. Render some semi-transparent object intersecting with another object (SVG/canvas) and export it to a rasterized image (JPG/PNG). Images rasterized on the same GPU/library will be identical. Ones on other systems will be ever so slightly different.

How can we get around this? Well we can’t really. Not without a whole lot of c++ programming and some custom browser code. I don’t have time for that.

But, what we can do is give it a really weird result and hopefully throw the detector off. If we randomize the image being rendered, the output will look nothing like what the detector expects and hopefully they’ll label us as “person worried about privacy”. There are user-land extensions that will do this for consumers – this is a known way to protect yourself from fingerprinting.

Fun fact: Apple’s iOS devices all expose the same fingerprint such that you cannot tell the difference between a late model iPhone and an old one. Privacy features – neat!


This has been client checks. The checks that a detection script runs on the browser to try and identify scrapers and block them. The detection script sends these results back to the server and either unlocks or blacklists the session.

Intro: How I bypassed military-grade bot detection software on popular ecommerce sites

Part 1: How to bypass the Server Bot Check

Part 2: How to avoid bot detection Client Checks

Part 3: How to bypass Interaction Checks