I made and have continued to update a tool that scrapes numerous venue calendars and scrapes information needed by the client.
1. Python
Python is the programming/scripting language used for building this project.
2. Beautiful Soup 4
I used the Beautiful Soup library to parse the HTML downloaded by python requests.
3. Selenium
I used Selenium to load data that required JavaScript and user interaction.
4. Insomnia
I used Insomnia to build URLs that contained queries used by the site to fetch relevant data. This was the preferred method of retrieving data as it's the quickest to parse through.
This scraper was created for a client that needed data from numerous venue calendars. The data was scraped and formatted into a CSV file with the columns specified by the client.
I learned that some sites store data in different ways, and some even store them in variables inside script tags :).
The overall process of building and continuing to update this project has been fun. Writing readable code that's easy to debug has allowed for updates to be implemented quickly. The importance of proper error handling was reiterated to me in this project. Having the scraper continue and not crash after a failed attempt to get data due to a change in one of the site layouts is critical.
