How to Prevent Web Scraping

As the world economy continues to rapidly convert to online-based applications, cyber security has never been more important. Keeping data properly protected and secure in an age of accessibility is a very difficult task, and must be a top priority of all businesses and individuals alike. Web scraping in particular is one of the biggest obstacles to safe and effective cyber security, and knowing how to prevent a web scraping attack is essential for all businesses today.

What is Web Scraping?

In its most basic sense, web scraping is the process of collecting and compiling information from websites and can be accomplished by manually copying and pasting material. However, most web scraping endeavors utilize sophisticated software known as automated bots that are often custom-built for the job, and capable of copying massive amounts of information rapidly. It is important to note that not all forms of web scraping are illegal, specifically if the information being accessed is publicized information that is not being redistributed elsewhere. However, if the web scraper software is targeting hidden files containing proprietary business information or confidential customer data, the actions are illegal. Web scraping is very similar to web crawling, the main difference being that web scraping targets specific information, whereas web crawling targets all information on a website.

History of Web Scraping

Web scraping has its origins in 1993, soon after the invention of the internet, when Matthew Gray of MIT developed the World Wide Web Wanderer to measure the growth of the internet. Although technically an automated web crawler application, this software was the genesis of future developments in the web scraping process. It was specifically used to measure and record the early rapid growth of the internet by downloading and copying all the data it encountered from various websites.

Process of Web Scraping

The increasing use and popularity of web scraping by cybercriminals for illegal purposes and common individuals for legal purposes is a direct result of the fact that it is an easy task to successfully operate. Individuals simply target the websites and information that they need to extract and then develop an appropriate web scraper bot to accomplish this task. Once the data is collected in HTML format, often accomplished by utilizing a GET method, the data is then reorganized for easy analysis.

How to Prevent Web Scraping

The use of illegal web scraping has a significant financial cost every year, costing online businesses about 2% of their annual revenue. Thankfully, there are many different practices that businesses can implement to reduce and eliminate the effects of web scraping.

Data Masking/Site Design

The most effective way to prevent bots from accessing confidential or proprietary information is to have it encoded on the site server. Companies can also encapsulate data using a variety of techniques, forcing the automatic bot to employ complicated methods and processes to extract information. Although these techniques won’t always stop the most advanced cyber criminals, many cyber criminals have limited resources and will simply move on to another website instead of spending excessive amounts of time trying to develop sophisticated software capable of breaking through. Companies should also avoid having a single web page where most of their important information is stored, such as a company directory page.

Site Monitoring and Maintenance

E-commerce departments should aggressively monitor new accounts that generate large amounts of activity on their sites but don’t result in any sales, a common sign of a web scraping operation. Another indication of a web scraping event can be if a specific product has an unusually large amount of page views that far exceed the average web traffic for similar products. Finally, old customer accounts are a common source of all kinds of cyber attacks and should be routinely updated as well.

Competitor Activity

Companies should also be aware of sudden changes in competitor pricing and product availability, specifically if their competitors start to offer similar products at a lower price. Businesses such as airline or hotel corporations are particularly susceptible as they operate in a highly competitive environment offering similar services.

Bot Prevention Techniques

Many bot prevention techniques are becoming the industry standard for many e-commerce operations, and have had great success in warding off web scraping attacks and all types of bot activity in general. The most common bot prevention techniques include requiring users to specifically check a box to indicate that they aren’t a bot or to select pictures in a group of pictures that depict certain criteria such as pictures containing a bus, car, or crosswalk. Another common technique is CAPTCHA, a process where a user must copy a group of letters or numbers that are difficult to read.

Terms of Use Agreements

Routinely updating and developing terms of use agreements specifically prohibiting all forms of web scraping techniques can also be an effective deterrent against these types of attacks. While these agreements won’t intimidate hardened cyber criminals, they often will convince those who are new to the web scraper industry and not as effective in their use of automatic bot technology to stay away.

Bot Mitigation Software

Recent developments in bot mitigation software have also been used to great effectiveness by implementing a variety of techniques. This type of software can analyze the signatures for different web traffic sources to verify their legitimacy and/or closely monitor the behavioral patterns of users to identify any specific actions that closely mirror those of automated bots.

The rapid expansion of the e-commerce industry has resulted in an increased volume of cyber criminality, including many different web scraping techniques. All businesses should avail themselves of the many different techniques and methods available for guarding against these attacks.