Website Scraping to Generate Keyword Frequency, Lahore
Job Posted: 05.03.2021 2:43:47
Website Scraping to Generate Keyword Frequency Objective: Scrape the data off competitor websites to give a weighting of the frequency of words used across the entire site 9or key pages. To answer the following questions: What is the positioning of the homepage? The most used terms on the website (Category Values / Differentiators will be positioning) What are the most used product words on the website? What are the most used needstates words of the website? What are the most used words to handle the biggest objections? What are the most used words to describe the results? What are the most used words to describe the Technology and Innovation? What are the most used words to target industries? What are the most used words to target job titles? What are the most used words for messaging concepts/creative thoughts? Step 1 Scrap the word count data from three competitors (https://www.appellon.com/, https://lattice.com/, https://www.cultureamp.com/ ) Neatly structure the data into the Example Output excel with the column completed for Competitor Name, Competitor Homepage, Page URL, Word, Word Occurrences, Frequency Score (you do not need to complete Word Category or Class of Competitor this will be done at a separate stage Deliver the file back for data validation (two processes/consultants will be used to ensure the date is accurate by matching the data) Here is an example process however you may choose to do it a better way that saves you time, if so please share that process with me so that I can ensure that it’s sound. Use Google sitemap Generator to give you a list of all the URLs of a site Paste each URL for each site into a too like https://www.online-utility.org/text/frequent_words.jsp to gain the ocurances and frequency score (occurrences is the only essential data from this tool but it’s good to get bother scores at once. Tools like https://www.spyfu.com/ can be helpful too in looking at a competitors We may need to limit the number of pages scraped to the top 10 most visited or main site (cut out blogs e.g Homepage, main product pages) as Culture Amp for example has over 1000 pages, but there may also be a way to automate this. Please advise. We will need to remove all the common useful words like “if, then, but, we, the” etc Step 2 Preliminary analysis of the data will be to ensure that the outputs are meaningful. There is an opportunity for you to do this but it is not required if you are not sure how you would do it. Step 3 The process used in step 1 would be repeated for Top 10 competitors.
