It sounds counter-intuitive but a new analysis from Imperva's Incapsula division has confirmed one of the Internet's most surprising secrets: a large proportion of website traffic isn't generated by human beings.
The firm's figures for 2015 (measured between July and October on websites using the firm's security) found that roughly half of all traffic was generated by automated bots, both good ones such as search engine spiders (19.5 percent) and bad ones such as spam engines and pricing scrapers as well as DDoS traffic (29 percent). Only a fraction over half was initiated by a person clicking a mouse.
Normally this traffic is invisible and nobody other than ISPs and website owners pay any attention or is even aware of it. Bad bot traffic is bad for reasons that are self-explanatory and it has been growning steadily as a proportion of all traffic in recent times. However, in 2015 Imperva's number-crunchers noticed something unexpected: the volume of human traffic is rising relative to bot traffic. It sounds innocuous but it means that automated traffic is still growing but less rapidly than human traffic which is being boosted by the Internet's spread in developing countries. This might or might not be temporary as the Internet adjusts to being global and not just something of importance in the US, Europe and bits of Asia.
The bad bot traffic is still a major issue with DDoS more than making up for a decline in spambots caused by Google's Penguin update from 2012 onwards that penalised sites linking to spam hosts. Meanwhile, the alarming growth of DDoS is contentious with some arguing (see our slideshow on Level 3) that service providers can only turn back this tide by intervening at network level. Fixing the problem using mitigration is at best just a way of deflecting it somewhere else.
We decided to ask report co-author and Imperva digital strategist Igal Zeifman for his views on the hidden world of bots, the good the bad and the frankly hopeless. It matters but if we're not aware of it, how?
Computerworld UK: Can you explain the botnet situation as you saw it in 2015 - what are the trends?
Ziefman: Bad bots originate from botnets, so the increase in their activity comes in parallel with the increase in botnet resources. We saw a steep increase in DDoS botnets in Q3, which led to double the amount of attacks on our clients when compared to Q2.
It was surprising to hear you make a distinction between 'good' bots and bad ones. Aren't bots always bad?
Ziefman: Not at all. When a Google crawler goes through your website to index its pages, or when a marketing too is used to establish how popular a piece of content was on social media, these are good bot visits. Any type of legal interaction between an automated piece of software and a website is considered a 'good bot' visit.
Sign up for CIO Asia eNewsletters.