AutoHotkey Webinar- Intro to Web Scraping

In this AutoHotkey Webinar we cover an Intro to Web Scraping.Intro to Web Scraping

Video Hour 1: High-level:

  • What is an API call
  • Web Scraping vs. WebServices / API
  • Document Object Model (DOM)
  • Structure of a web page
  • Frequently used Methods & properties

Video Hour 2: Coding and Q&A

  • Building code
  • Using AutoHotkey syntax writer
  • IE verse Edge
  • Selenium for Chrome, Firefox, IE , etc.

Script Highlight: Long-pressing Hotkey to Exit a script

The below script is more of an example of how you can have more than one hotkey for a given key by tying it to how long you press the key.  In the below script, if you hold down Escape longer than 1/2 a second it will close the active program.

 

PowerPoint Deck presented in Intro to Web Scraping

Q) What is an API? (Application Programming Interface?)

A) Basically it is how devices (computers, tablets, phones, servers, etc) talk to each other.

Q) What are examples of common API calls

A) Below are a list of some examples (but the list is limitless)

  • Web Browser pulling up products on Amazon.com
  • Application / Software querying products for sale on Amazon.com
  • App on your phone getting latest Weather
  • Database pulling updated sales report
  • Using your Tablet to Select movies to watch on Netflix
  • DropBox application syncing files between your computer & cloud
  • Using Web browser to access files in DropBox

Q) What are the Main Differences between Webservice / API call & Web Browser

Q) What is the HTML DOM: Document Object Model

A) Think of the DOM as the “road map” for a webpage.  Depending on what you are trying to get/set, you’ll rely on the DOM to get the exact element you’re looking for.   Below is an image which stresses the more-frequently-used parts.

Q) How does a Family Tree similar to the DOM?

A) Both are “hierarchical” and, depending on what elements are populated, you can reliably “talk” to the correct node.  The first tree below shows how some nodes have all elements populated while some are missing some (like Name and ID)

Q) What happens if a new Node is inserted in the Tree?

A) It depends on which element you are looking at.  Names and IDs probably will not change however Tagnames often will.  This is why it is recommended to first use IDs, then Names, then ClassNames.  Use Tagname last (or go up the tree a bit and grab an ID/Name then use that “branch”)

Q) What are the frequently used methods for selecting an Element?

A) The methods below are sorted by the most reliable/stable (not presence).  When ID is present- use it!  If no ID, Name is another great one…  Classname is also becoming more reliable

  1. getElementByID– Great because if it is present it is should always unique
  2. getElementsByName– While not required to be unique, it often is so very reliable
  3. getElementsByClassName– Often unique and present
  4. getElementsByTagName– Check if there is an ID/name above it, then jump “down” to it first then use TagName)
  5. querySelector/querySelectorAll- Most flexible but is the most complicated- (If you know CSS, this is much not hard to adapt)

Q) What are Frequently used Get/Set/Trigger HTML Elements?

A) The below are the most frequently used however this is highly tied to what your goal is.

Methods:
.click() – Clicks the element
.fireEvent(“onclick”) – Fires the click event (sometimes needed to trigger the event)
.focus() – Place the cursor in the element/edit field (set cursor)
.selectedIndex() – Dropdown box selection
.checked() – Radio / Checkbox selection

Properties (Get & Set):
innerText – the plain-text for the element
.value – the plain-text for the element
.outerHTML – the HTML for the element

Additional Resources

AHK specific:

Learning the DOM (If you spend time on this you’ll thank us later!):

AutoHotkey Webinar- Working session demonstrating ways to automate programs

AutoHotkey webinar– Working Session

This webinar was mostly working sessions and not a lot of resources.  It is a follow-up to our last webinar which discussed many different ways to automate windows programs using AutoHotkey.  Please the resources on this page, and watch the videos, if you want to review in depth.  We had links to the various tools that were used in the below webinar.

Script highlight: Use Google to Search for help

Thanks for attending the AutoHotkey webinar!  If you’re not currently registered for future ones, you can do so here

AutoHotkey Webinar- Various ways to use AHK to automate tasks in Windows

In the first hour of our March AutoHotkey webinar we covered core concepts of what to consider when writing a script.AutoHotkey webinar

The second hour we took a deeper dive into coding using some of the methods mentioned in hour one.

Content & Resources shared in AutoHotkey webinar

Script highlight:  Copy Explorer path to clipboard

Things to Consider when writing a NEW script

  1. Are you behind a time-crunch to get the job done?
  2. Are you looking for a 1-time solution?
    1. What are the odds that you could re-purpose your solution for something else at a later date?
  3. Will this be run on multiple:
    1. Computers
    2. Windows versions
    3. Users
    4. What software is also running on computers
    5. Bitness (32 & 64) of OS, AutoHotkey , program interacting with
  4. Is there existing code that you can borrow / steal / leverage?
    1. Previous code I’ve written
    2. Sites like:  AHK Forum  Stack Overflow  Reddit  Github  the-Automator  JSZApp  Jack Dunning Blog
    3. Ask colleagues / friends / Social channels
      1. AHK Forum   StackOverflow    Reddit
      2. LinkedIn 1  LinkedIn 2
      3. Facebook 1  Facebook2  Facebook 3
      4. Joe Glines  Jackie Sztuk  Hellbent  ahkTuts AboutScript  Chunjee  AHK vids
  5. Will someone other than you be using / maintain the code? Program to their “level”

Many ways to “Control / Interact” Window’s programs

Example Scripts used during Coding demonstration of AutoHotkey webinar

AutoHotkey Webinar: 01/17/2017: Troubleshooting and Debugging

AutoHotkeyAutoHotkey webinar Webinar Videos and Links

  • Here is a link to video of the First hour
  • Video of Second hour where we chit-chat about troubleshooting and other items.

Link to AutoHotkey Webinar resources:

Resources covered during AutoHotkey webinar

Script highlight Lintalist – Searchable interactive lists to copy & paste text with plugins.  GitHub

General Advice

1.Annotate -Document / Add Comments for later review (Very helpful in long-run)

2.Modularization – Compartmentalize – Design code smartly so each “section” does one thing

3.Test Frequently

  • Depending on your skill level, don’t make too many changes before re-running your script (the more experienced you are, the more changes you can make)
  • ; Comment out old lines so you have backups of what was close

4.Take a break – Similar to proof-reading, taking a break can help you spot your own errors

5.Bobble-Head doll / Teddy Bear / Rubber Duck

  • Clearly Explain the issue out loud to inanimate object
  • Ask friend for help (preferably some programming experience but not necessary)

6.Search the AHK forum & StackOverflow

7.Post question to forum:

  • Isolate issue & give example code / data / needed functions
  • Explain circumstances in graphic detail causes of issue & what you experience
  • Include Screenshots / Animated gif / Video if possible (this can help a lot)

Provide as much relevant data as possible (ver & bitness of AHK, Windows, other software, Hardware, etc.)

Troubleshooting Tools

Built-in Commands

  1. Message box / TooltipsDisplay values / pause script
  2. Error LevelSuccess / failure of a command
  3. Try / Catch / Throw
  4. ListVarsDisplays the script’s variables: their names and current contents.
  5. ListLinesView Lines most recently executed
  6. ListHotKeysDisplays the hotkeys in use by the current script
  7. KeyHistoryThis shows history of most recent keystrokes & mouse clicks
  8. IsObject()This function returns 1 if Object Value is an object; otherwise 0.
  9. OutPutDebugFor use in debugging- sends strings to display

Additional Tools

  1. dmp, DispFunctions for peaking inside objects
  2. Debug Clientstools you can tie to AHK & debugging
  3. AHK StudioBuilt-in debugging
  4. SciTE4AutoHotkey Videos & scripts demonstrating debugging with SciTE

Tips on General Approach

  1. Restart script & Computer (come on its windows…)
  2. Isolate the issue
    1. look immediately before and after where the issue is
      1. Use Message Box, ListVars, etc.
      2. Double-click icon & access ListVars, Key History, etc.
  3. Are you sure the error is where you think it is?
  4. Are you sure the variable is unique?
  5. Check scope issues (global vs. local)
  6. Data type / structure (String vs. Integer, Object, etc.)