Scraping the Data

I have started picking up traction since my last post. My initial challenges were that I needed to find a list of coaches, with relevant data, that was written in HTML. I initially wanted to use the NCAA website as they have the most extensive and accurate information, but their webpage is set up in a format that requires clicking on each coach to get to a seperate page. This would mean I would have to click 397 coaches over 15 years for a grand total of 5,955 pages. This obviously would not be efficient but then I found:

I then had two challenges, I needed to learn how to use the Beautiful Soup package to scrape the page, and subsequently I needed to learn how HTML pages are written. After a lot of research on html tags, classes, and divisions I succeeded in pulling in the information from the page.

Once I pulled in the information, I realized that in order to manipulate the data with the Pandas package that I needed to change the data type from strings to lists. I made some modifications to the code and was able to pull in the data as lists.

The next step is to import numpy and pandas and start dictating how the array’s and dataframes will look.

This process has really picked up steam over the last couple of weeks and i look forward to organizing the arrays in Numpy, and then I will be ready to pull in the rest of the years, combine them into pandas dataframe and then begin analyzing.

Related posts:

I recently spent 10 days in the bush with a group of 16-year-olds. On this expedition, my role, as the accompanying adult was really just to make sure they didn’t do something stupid! Otherwise, the…

Finding self love is the first thing you need to do before you can find a lasting, meaningful relationship. I truly believe, with all my heart, that a person will not find their person until they’ve…

For our group presentation, we had to come up with a creative presentation that had to elaborate on a certain topic. Eachgroup had to discuss amongst themselves and pick from a pool of choices. I did…