How to Scrape AutoTrader Data Using a Web Scraping Tool?

Today, we will see how to utilize a web scraping tool for scraping websites like Autotrader!

This website Autotrader is an ideal website to get numerous car listing from dealerships as well as private sellers.

For any car dealership, to be in this car industry is extremely competitive as customers have higher switching costs. Getting a competitive benefit is vital to make sales.

For any consumer, getting the best car for fitting your requirements and price could be a tiresome procedure with different kinds of dealerships and cars.

For both the cases, you could use a free web scraping tool like Retailgators for scraping Autotrader data to assist you take the finest decision.

Here, we will extract Autotrader data like models, highlights, price, specs, etc.

Therefore, let's start.

Data Scraping from Autotrader

Here in this sample, we will scrape Autotrader data from Honda vehicles 2020 in Toronto. We will extract models, specs, highlights, Image URL, pricing comparison, etc. to market prices. We will export that as a CSV, Excel, or JSON files (here, we suggest JSON for the project because it’s easy to understand scraped data).

Let’s Start

Just download as well as install Retailgators Scraping Tool completely free. We would utilize this data scraping tool for the project here.

It’s easy to use the link given here:

2. When Retailgators gets open, just click on the “New Project” as well as utilize URL from an Autotrader results page. This page would now become rendered within the app.

Scrape Data Behind the Self-opening Windows

Whenever you initially load the site within the app, then you'll see the Autotrader come up to ask you to think about your search then you will require to tell Retailgators and click on that as it could mess up with the project.

2. For doing that, use the Select command and click on "got it" tab on a self-opening to choose it. You may rename that command to somewhat more expressive by just clicking on a command itself. Just name it as "closePopup".

3. Just click on plus button subsequent to "Choose & Scrape closePopup", as well as select Click command option from the toolbox.

4. The pop up would appear as well as ask if that is a following button, just click on the ‘No’ option and choose "continue performing the present template" option.

In case, you are unable to see any pop up straight away, then you can execute this step while testing the run. Just stop a test while seeing a come up.

Now, let’s start extracting car data!

Scrape Cars Data on Results Pages

Click on a PLUS (+) symbol next to the page selection as well as get the choose command

Begin through clicking on initial name of first model given on a list. This would be emphasized in green for specifying this has been chosen. At left sidebar, just rename the selection with “car_model”.

3. Now, the Retailgators will suggest other data that you wish to get scraped in yellow. Just click on second model given in the list for choosing them. All the models on first page would be highlighted within green. Now, you are gathering the name as well as URL lists of models.

4. Then, scrape data like location, prices, image URLs, and pricing comparison in the market. For doing that, you would need to perform a comparative select command. Begin by clicking PLUS(+) sign subsequent to the car model’s selection as well as select a command called Relative Select.

5. To use the command Relative Select, just click on a model of first car, which is highlighted within orange and on the price. One arrow would appear to display the connotation you’re making.

6. You could require to repeat the procedure for one more car to completely train a data scraper. At the left-hand sidebar, just rename the selection with “price”.

7. You need to repeat the steps 4 to 6 to get additional data for every Car. Here, we would also pull an image URL, description, location, as well as price comparison with market value (as not all the cars would have that collection). Your project needs to look something like this:

Scrape Additional Cars Data from All Car Listings

Retailgators Scraping Tool scrapes data from result pages like name or price, however, let's scrape specs and features of a vehicle as well as a dealership site in case, they have that.

We would require to tell Retailgators and click on every listing and scrape which data we need.

Just click on PLUS(+) symbol next to “car_model” collection and select Click command option.

2. The pop-up would appear asking if it is the link of “next page” option. Then click on the “no” option and enter the template name. We would call that “car_listing”. Now, you would be taken into a model of initial listing on a page.

3. A car page would reduce in the new browser’s tab as well as you would do your initial data selection to scrape. Here, we will choose the specs and highlights of a car as well as a dealership site.

4. Just click on PLUS(+) symbol given next to “page” selection, select the option Select command as well as you would make newer select commands as well as click on additional data to scrape. Scroll down till you get “specifications” as well as click one on specifications labeled like kilometers of a car. Then, Retailgators will indicate what you wish to scrape. Now, you can click on other specifications labelled to scrape all. Rename the selection to specification.

5. Then make the relative select commands and scrape data of specs data

6. Just repeat the last step to scrape financial payments, highlights as well as dealership website. The car_listing template will look like given here:

Adding Pagination

At the moment, Retailgators is scraping results pages as well as data from all listings on first page. Let's scrape data from various pages.

NOTE: Beforehand you start working on the pagination, you would require to call customer support for making your profile as well as admin user. It will help you to make the box "ignore disable elements" off that we would go over within some steps.

As Autotrader doesn't provide the next button, Retailgators can add the pagination. We have to perform some select as well as commands called relative select.

1. Just return to the main_template with left-hand-side tabs. Utilize a browser tabs and return to a search result page.

2. Then click on PLUS(+) symbol next to the “page” selection as well as pick Select command. Then scroll to bottom of this page as well as click on 1st page. Then rename the selection with “current_page”.

3. Just click on PLUS(+) symbol next to present current_page selection as well as click on command, Relative Select. Then click on present page -1 and click on second page - 2 for making the connection. Let's rename the Relative Select to the "next_page"

4. Then click on PLUS(+) symbol next to “next_page” selection as well as pick Click command option.

5. The pop-up window will come asking if it is the link of “next page”. Then click on the “Yes” as well as enter additional pages that you would love to extract. Here, we will extract 5 additional pages for the project. Make sure to choose "utilizes AJAX"

6. Then you would be automatically taken to next page for making sure that it is properly working, then scroll down at next selection as well as ensure the present as well as subsequent page are chosen. If not, just make some adjustments to facilitate that current_page collection is page -2, as well as the command ‘relative select’ is page 3. Now, Retailgators will understand what you want to get.

7. For making sure that pagination works well, just click on the browse button on the top as well as enter the browse mode. Just click on the page 3. Make the search-mode off as well as check the current_page as well as next_page selection, so pages 3 & 4 should be selected now.

8. Then click on Click command in the next_button selection as well as ensure that the box "ignore disabled elements" gets checked. In case, you don't observe it, make sure to call customer support as well as they could easily make the admin account for allowing you for checking the option off.

In case, you still want any assistance, you can call customer care or see this video to get more details.

In case, you don't wish to complete the step, then you could easily change display option for many vehicles you wish to extract.

The final project would look like that:

Run Your Data Scraping Project

Now, the fun part comes. Just click on green “Get Data” tab.

On the given screen, you’ll can test, run, and schedule the project. In case, you wish to continuously get the newest data extracted every day or weekly basis, a schedule option permits you to.

For greater projects, we suggest testing that to ensure it is scraping data correctly.

Here, we will run that right away.

Conclusion

If you are a car or a consumer dealership, you could use free data scraping tool for scraping sites like Autotrader for helping you make right decisions. You can find the finest price to sell vehicle or get the finest vehicle within the price and requirements.

What would you extract?

Happy scrapping!