Cross browser web scraping with AutoHotkey and Selenium


While AutoHotkey is an amazing tool for Web Scraping, many people complain about being limited to connecting with COM to IE.   In the below videos I walk through how you can use AutoHotkey and Selenium to automate web scraping in virtually any browser you wish.  🙂

What is Selenium & why should AutoHotkey users care?

Installing Selenium

In order to control Selenium with AutoHotkey you need to install the SeleniumBasic.  The current version is 2.09.0 and can be downloaded here.  Selenium is now on version 3 and there is a new SeleniumBasic version promised to be released soon which will connect to version 3 of Selenium.  Make sure you download the WebDrivers of choice for your browsers.

If all of this sounds confusing, don’t feel bad.  It is ridculous!  I found this post which documents/clarifies much of the confusion (although Selenium 3 is now out)

Please note several people reported when installing Selenium Basic it did not install in the program files location (i.e. here: C:\Program Files\SeleniumBasic or C:\Program Files (x86)\SeleniumBasic).  They also had problems getting Selenium to launch.   I recommend you make sure Selenium installs into one of the Program Files location and also make sure you get the Selenium drivers installed.  After install I had the following files on my computer:

  • C:\Program Files\SeleniumBasic\operadriver.exe
  • C:\Program Files\SeleniumBasic\chromedriver.exe
  • C:\Program Files\SeleniumBasic\edgedriver.exe
  • C:\Program Files\SeleniumBasic\iedriver.exe

After installing the Selenium Basic- Make sure you get the latest Chrome driver here and extract the “chromedriver.exe” into the folder where SeleniumBasic is installed. Also keeping your Chrome version up to date.  Currently I’m running 64.0.3282.39 64-bit version of Chrome and am using a 32-bit version of Selenium Webdriver (as there is no 64 bit version of Selenium webdriver).

Here are some links you might want to review (but you’ll need to adapt them for your purposes)

Installing Selenium for use with AutoHotkey


Using AutoHotkey and Selenium across various browsers

This video I show two ways I’ve learned how to start-up the Selenium Webdriver with AutoHotkey.

Tutorial showing how to start up and navigate with AutoHotkey and Selenium

Using AutoHotkey & Selenium- Starting the browsers & navigating to a page

Getting information from a page with Selenium and AutoHotkey

While there are a lot of similarities to data extraction in Selenium, there are quite a few differences as well. The below code is what I use in the following video.  It demonstrates some ways that you can extract data from a web page via Selenium and AutoHotkey.


using AutoHotkey and Selenium : Getting information from a page

Setting information on a page

Selenium and AutoHotkey are pretty different in how you set information.  Selenium has a “sendkeys” method which seems to be pretty reliable at triggering events on a page.

  • Make sure you review this:  Send keys to value:  ;Note: you need to add “driver” e.g. .SendKeys(driver.Keys.ENTER)

Setting text & clicking items on a page with Selenium and AutoHotkey

Selenium and AutoHotkey: Setting text and clicking elements


Using your Chrome Profile (Avoiding the need to re-login to a site)

In this video I demonstrate how you can leverage your Chrome profile so you do not need to keep logging into a website with Chrome


Setting Selenium Chrome profile with AutoHotkey

Iterating over Objects with Selenium and AutoHotkey

In the below video I demonstrate some of the important differences when iterating over objects with Selenium and AutoHotkey.    A COM based object does not have an enumerator thus you cannot simply use a for-loop to iterate over them.   Selenium does does have an enumerator however the objects are held in the Keys (not the values).


Demo video showing how to iterate over objects in Selenium

Various Selenium methods for getting & setting data on a page

In this tutorial I walk through various ways to get/set data on a page. With Selenium you can use both CSS and Xpath which are like QuerySelector.

I also shared these two resources from Michael Sorens which present the same data grouped by Method and grouped by Tool.


Selenium & AutoHotkey- Various methods for Getting/Setting data

Various methods from Selenium & by using JavaScript Execution

I went through and documented some of the additinol methods I used from Selenium & by injecting JavaScript. Check them out below as well as the video walking through the usage.

Selenium & AutoHotkey- Methods and JavaScript execution

Maneuvering Frames in Selenium with AutoHotkey

The below code provides some insights on how to navigate frames with AutoHotkey

Navigating Frames with Selenium and AutoHotkey

Downloading files with Selenium and AutoHotkey

In this tutorial I demonstrate how I used AutoHotkey and Selenium to download a PDF file. The same process will work for other files that are not, automatically, opened by Chrome.

Downloading a PDF file with Selenium and AutoHotkey

Connecting to a current instance of Chrome

Thankfully tmplinshi has come up with a solution on how to connect to an already launched version of Chrome.  Granted, you’ll need to launch Chrome with some command line parameters but this is an easy tweak to do by just adding them to your main shortcut to Chrome you’ll be able to connect to a current running Chrome window!

Here’s what you need to do for prep-work:

  1. Make sure all current versions of Chrome are closed
  2. Create a shortcut to chrome with this path: chrome.exe –remote-debugging-port=9222
  3. Launch Chrome from your new shortcut

Then you can use the below code to connect with it!  Check out the below video demonstrating how it works.


Connecting to a running instance of Chrome with Selenium and AutoHotkey

Update to Web Scraping syntax writer for AutoHotkey

I made a few updates to my Web Scraping syntax writer.  This new version adds in funWeb Scraping syntax writerctionality to Get/Set attributes of an element as well easily find all tables on a given page, then extract the text and dump it into a dynamic Listview.

Both can be pretty helpful when scraping data from the web.  For an extensive review of the tool, checkout the videos on my main Web Scraping page.

Demonstration of updated Web Scraping syntax writer

Updates to Web Scraping / AutoHotkey syntax writing tool

AutoHotkey Webinar- Intro to Web Scraping

In this AutoHotkey Webinar we cover an Intro to Web Scraping.Intro to Web Scraping

Video Hour 1: High-level:

  • What is an API call
  • Web Scraping vs. WebServices / API
  • Document Object Model (DOM)
  • Structure of a web page
  • Frequently used Methods & properties

Video Hour 2: Coding and Q&A

  • Building code
  • Using AutoHotkey syntax writer
  • IE verse Edge
  • Selenium for Chrome, Firefox, IE , etc.

Script Highlight: Long-pressing Hotkey to Exit a script

The below script is more of an example of how you can have more than one hotkey for a given key by tying it to how long you press the key.  In the below script, if you hold down Escape longer than 1/2 a second it will close the active program.


PowerPoint Deck presented in Intro to Web Scraping

Q) What is an API? (Application Programming Interface?)

A) Basically it is how devices (computers, tablets, phones, servers, etc) talk to each other.

Q) What are examples of common API calls

A) Below are a list of some examples (but the list is limitless)

  • Web Browser pulling up products on
  • Application / Software querying products for sale on
  • App on your phone getting latest Weather
  • Database pulling updated sales report
  • Using your Tablet to Select movies to watch on Netflix
  • DropBox application syncing files between your computer & cloud
  • Using Web browser to access files in DropBox

Q) What are the Main Differences between Webservice / API call & Web Browser

Q) What is the HTML DOM: Document Object Model

A) Think of the DOM as the “road map” for a webpage.  Depending on what you are trying to get/set, you’ll rely on the DOM to get the exact element you’re looking for.   Below is an image which stresses the more-frequently-used parts.

Q) How does a Family Tree similar to the DOM?

A) Both are “hierarchical” and, depending on what elements are populated, you can reliably “talk” to the correct node.  The first tree below shows how some nodes have all elements populated while some are missing some (like Name and ID)

Q) What happens if a new Node is inserted in the Tree?

A) It depends on which element you are looking at.  Names and IDs probably will not change however Tagnames often will.  This is why it is recommended to first use IDs, then Names, then ClassNames.  Use Tagname last (or go up the tree a bit and grab an ID/Name then use that “branch”)

Q) What are the frequently used methods for selecting an Element?

A) The methods below are sorted by the most reliable/stable (not presence).  When ID is present- use it!  If no ID, Name is another great one…  Classname is also becoming more reliable

  1. getElementByID– Great because if it is present it is should always unique
  2. getElementsByName– While not required to be unique, it often is so very reliable
  3. getElementsByClassName– Often unique and present
  4. getElementsByTagName– Check if there is an ID/name above it, then jump “down” to it first then use TagName)
  5. querySelector/querySelectorAll- Most flexible but is the most complicated- (If you know CSS, this is much not hard to adapt)

Q) What are Frequently used Get/Set/Trigger HTML Elements?

A) The below are the most frequently used however this is highly tied to what your goal is.

.click() – Clicks the element
.fireEvent(“onclick”) – Fires the click event (sometimes needed to trigger the event)
.focus() – Place the cursor in the element/edit field (set cursor)
.selectedIndex() – Dropdown box selection
.checked() – Radio / Checkbox selection

Properties (Get & Set):
innerText – the plain-text for the element
.value – the plain-text for the element
.outerHTML – the HTML for the element

Additional Resources

AHK specific:

Learning the DOM (If you spend time on this you’ll thank us later!):

Review of Web Scraping tools for identifying page elements

the below video is a review of web scraping tools I’ve used & still use for assisting my web scraping endeavors.

The three versions of iWB2 learner tool can be downloaded here:

  1. Original executable
  2. Larger & zoom level corrected
  3. Beta version which Includes className

In each of the modern browsers you can press the F12 key to bring-up the built-in Developer tool.

Also, don’t forget to download my web scraping syntax writer for AutoHotkey.

Review of Web Scraping tools

Review of various Web Scraping tools