Web Scraping with AutoHotkey- Data Extraction & emailing to colleagues

web scraping with AutoHotkeyI regularly have to go to login to a website, navigate to a certain page, extract data and then place it into Excel, then email the files to my colleagues.

This video demonstrates how quickly and easily it is done when you program some web scraping with AutoHotKey.  Somehow I lost the audio track however you can still see what is going on.  Also, some of it happens so fast it looks like I sped up the video but it is real-time.  🙂

Demo video of Web Scraping with AutoHotkey and emailing to colleagues

Web Scraping- Data Extraction from a webpage & emailing to colleagues

If you’re new to scraping you can see my earlier post here which gives a good review as well as a program that helps write the AutoHotKey code.

Example of Web Scraping- Automating Exporting of contacts from Lexis Nexis

Example of Web Scraping

Example of Web Scraping– Automating the Exporting of Data from Lexis Nexis

A few years back Lexis Nexis had access to Jigsaw (a sales list provider).  They had built in restrictions which allowed for exporting up to 1,000 contacts at a time.

Using AutoHotKey I was able to automate the exporting of contacts from Lexis Nexis into a great format.   All-told I exported over 4 million business contacts including their Name, job title, job function, email address, phone number, company name, company revenue, company NAIC code and SIC codes and much more!  It was quite a score and I was in love with Web Scraping  from then on!

Example of Web Scraping with AutoHotkey

Automating Exporting of Data

When your IT department fails you Web Scraping with AutoHotkey can save the day!

web scraping with AutoHotkey

One of my clients has over 3,000 videos on their website.  Amazingly nobody in their IT department can provide a holistic file with metadata like: Title, # of views, Length, Format, Tags, etc.  How crazy is that?   After contacting their video hosting company, nobody could (would) share this information!

Web Scraping with AutoHotkey to the rescue!

I wrote web scraping with AutoHotKey script that first got a list of all the videos, then looped over them and extracted various attributes about each video.  Not an ideal way to get this, but the client was manually going to go through this by cutting & pasting from their browse.

Extracting Information about Videos

AutoHotKey GUI menu for API web service- Connecting to SilverPop via API

GUI menu for API web service

GUI menu for API web service

For years we have relied on using a browser for interacting with our client (SilverPop) servers.  I had heard that SilverPop had an API for doing much of the same thing but was always “too busy” to take the time to investigate.  At some point I started playing with it and was amazed at how much time could be saved.  After you get passed the “hand-shake” between your computer and the server, the rest is pretty easy.  You simply pass XML to tell the remote servers what action you want to achieve.

I then created an AutoHotKey GUI menu for API web service  which streamlined submitting the XML.

 

Here is the AutoHotkey script for GUI menu for API web service