AutoHotkey Webinar- Parsing plain text with AutoHotkey

Video Hour 1: High-level overview:AutoHotkey Webinar

In the first hour of our AutoHotkey webinar we demonstrated Quick Access Popup from our guest speaker Jean Lalonde.  It is an amazingly customizable tool that allows you to have what you want at your fingertips!

Here is an abbreviated list of some of the great features:

  1. Create shortcuts to folders, programs, files, links, Snippets, special folders, etc.
  2. Easily create Hotkeys for above shortcuts
  3. Works with most Open/Save dialog boxes
  4. Integrated with most file managers (Windows Explorer, Directory Opus, Total Commander, etc.)
  5. The tool itself is highly customizable
  6. Have “shared” sections as well as individual ones
  7. Organize your menu into folders & sub-folders
  8. Customize Menu/GUI shortcut with an icon (that makes sense to you)
  9. Available in 10 languages

You can watch a short overview video here or get a deeper dive here.  Jean was kind enough to walk me through his tool in this video.

We then talked about what a “text file” is and frequent sources of them (Export from programs, sever logs, emails, etc).  We also briefly discussed what is File / Character Encoding & why are there frequent problems with it (this is a very deep topic which we could have spent days discussing!)

Also discussed  what a “flat file” is and how they are different than:

  • Multi-row files
  • Relational files
  • XML and JSON files

We then shared some of the frequently used built-in commands and functions for parsing text in AutoHotkey.

Then we got into various custom-built in libraries and functions

Then our guest speaker, Jean, took over walking us through some examples of common problems with AutoHotkey LoopParse command

The library is pretty cool and here are some of the main functions available:

  • CSV2Collection() – Transfer the content of a CSV file to a collection of objects.
  • Collection2CSV() – Transfer the selected fields from a collection of objects to a CSV file.
  • Collection2Fixed() – Transfer the selected fields from a collection of objects to a fixed-width file.
  • Collection2HTML() – Builds an HTML file based on a template where variables are replaced with content.
  • Collection2XML() – Builds an XML file from the content of the collection.
  • Collection2ListView() – Transfer the selected fields from a collection of objects to ListView.
  • ListView2Collection() – Transfer the selected lines of the selected columns of a ListView to a collection of objects.

ObjCSV Examples:  Here are the files Jean worked through.

  • SIMPLE CSV FILE – COMA DELIMITED (TheBeatles-coma.txt)
  • SIMPLE CSV FILE – TAB DELIMITED – ISSUE #1 (TheBeatles-tab.txt)
  • MULTI-LINE CSV FILE – COMA DELIMITED – ISSUE #2 (TheBeatles-Lyrics.txt)

Jean then gave us a review of how to use his ObjCSV library

  • Example using ObjCSV_CSV2Collection
  • SIMPLE CSV FILE – COMA DELIMITED (TheBeatles-coma.txt)
  • SIMPLE CSV FILE – TAB DELIMITED – ISSUE #1 SOLVED (TheBeatles-tab.txt)
  • MULTI-LINE CSV FILE – COMA DELIMITED – ISSUE #2 SOLVED (TheBeatles-Lyrics.txt)
  • PROGRESS BAR WITH LARGE CSV FILE – SEMI-COLON DELIMITED (LargeFile-semicolon.txt)
  • CUSTOM HEADER – SIMPLE CSV FILE – COMA DELIMITED (TheBeatles-coma.txt)
  • GUI Example using Collection2ListView
  • SAVING WITH DIFFERENT DELIMITER using ObjCSV_Collection2CSV
  • EXPORT HTML Example using ObjCSV_Collection2HTML

Wrapping up the first section we then saw a peak at CSVBuddy

CSVBuddy Examples

  • Reading large file with auto-detection
  • Editing field names, re-order or select fields
  • Saving options
  • Saving file with multi-line content converted to single-line (Excel-ready)
  • Reverting the single-line file to multi-line (for example, after manipulation in Excel)

Video Hour 2: Q&A:

In the second hour we discussed various issues working with text data as well as took a deeper look into Quick Access Popup.

Near the end of the time Capn Odin shared a Trie script he’s been working on.  You can check out the script on the forum

Webinar- SciTE4AutoHotkey is an amazing editor!

SciTE4AutoHotKey

In this AutoHotkey Webinar we cover using/customizing  SciTE4AutoHotkey by FincsSciTE webinar

Video Hour 1High-level overview:

SciTE4AutoHotkey is a lightweight and easy to use SciTE-basedAutoHotkey script editor. It provides:

Demonstration of cool SciTE4AutoHotkey features

  1. Multi-line typing
  2. Dynamic Intellisense
  3. Move text up/down
  4. Comment / Un-comment
  5. Change applied Intellisense language
  6. Copy / Paste as RTF
  7. Custom Context Menu
  8. Search / Replace – with RegEx
  9. List of Extensions to “open”
  10. Highlight multiple instances of each word
  11. Change color of highlighting for found word
  12. Custom Status bar
  13. Configure hotkeys (w/o AutoHotkey)
  14. Custom your Status bar
  15. Setting default magnification level
  16. Import additional languages (here are an additional ~70 languages)
  17. Leverage the Output area as a GUI
  18. Customizing your context menu
  19. Running multiple instances of SciTE
  20. Debug your scripts
  21. Run selected text

Scripts / External Resources

  1. Main thread on SciTE4AutoHotkey
  2. 16 videos @ the-Automator on using / customizing SciTE
  3. 42 of built-in Hotkeys
  4. My SciTE User.properties file (backup yours before playing with it!)
  5. Adding Intellisense for an additional ~70 languages
  6. Scintilla / SciTE Documentation, Interface, Menu /Key commands
  7. UserGroup plus StackOverflow
  8. Second SciTE window by boiler (launch script, then do new window)
  9. Tweaking Intellisense to use your file / library from XeroByte
  10. Have Intellisense examine current file from lexikos

Video Hour 2Discussion and Q&A

Script Highlight: Button Clock

Why waste space for “Start” button.   Use Button_Clock_WINXP from Goyyah

 

AutoHotkey Webinar- Intro to Webservice / API calls

In this AutoHotkey Webinar we cover an Webservice / API calls.

Video Hour 1: High-level:

  • What is a Webservice API call
  • Web Scraping vs. WebServices / API
  • List of some “fun” WebService / APIs as well as links to resources to tens-of-thousands!
  • Differences between a Get vs. Post request
  • High-level look at oAuth1 vs. oAuth2
  • Resources
    • HTTP: Intro to HyperText Protocol, Types of Requests, oAuth, Parsing XML, XPath, JSON
    • AutoHotkey resource: Syntax writer, winHTTPrequest, Msxml2 vs. WinHTTPrequest, Parsing XML & JSON
  • Basic examples of API requests (compared using browser and winHTTP request)

Video Hour 2: Coding and Q&A

  • Played with several APIs
  • Discussed & demonstrated reverse-engineering API calls from a website
  • Passing key-value pairs (and using a function to keep it organized)
  • Reviewed additional APIs
  • Q&A

Script Highlight: Select text and “pretty” hyperlink text

The below script demonstrates how you can use AHK to automate highlighting text on a page and then, using the winClip library, constructing a “pretty” html link.  Here are links to the GetActiveBrowserURL and WinClipAPI / WinClip


What is a Webservice API? (Application Programming Interface?)


Examples of Webservice / API calls

  • Application / Software querying products for sale on Amazon.com
  • App on your phone getting latest Weather
  • Database pulling updated sales report
  • Using your Tablet to Select movies to watch on Netflix
  • DropBox application syncing files between your computer & cloud
  • Google places search

APIs are becoming increasingly available!

…Since 2005, we’ve seen APIs grow from a curiosity, to a trend, and now to the point where APIs are core to many businesses. APIs have provided tremendous value to countless organizations and developers, which is reflected in their continued growth.   Source: Programmable Web

growth in webservice APIs since 2005

 

 


Main Differences between WebService / API call & Web Browser

Webservice verse web browser calls


When to use API verse Web Scraping?

When to use webservice API call verse web scraping


Some Webservices / APIs

Some of the Amazing APIs out there…

Programmable Web – list of thousands of APIs as well as a great, six-part, article reviewing APIs

Short-list of APIs open to public on Wikipedia,   Google APIs– Over 100 APIs @ Google; Yahoo APIs

Social Sites : Twitter, Pinterest, Google+, YouTube, Facebook, LinkedIn, reddit, StumbleUpon, WordPress, Instagram

Contact / Business Lookup: ClearBit, Pipl, FullContact, Dunn & Bradstreet , Foursquare, Google Places, Yelp, GeoNames

Weather: Weather Underground, Yahoo Weather, World Air

Finance: Google Finance, Yahoo Finance, PayPal, Stripe, XigniteRealTime, Ally Invest

Government: A bunch of Government APIs, Cloud.Gov, US Census, 18F.gsa.gov

Additional: Wikipedia, Amazon Product, email to text, SmartSheet,SurveyGizmo, Bitly, SnapOCR, PasteBin, DropBox, Zoho, Zillow, imugr, Mailgun, MailChimp,  Microsoft: Graph, MS Office: Excel  Word  SharePoint  OneNote Outlook Yammer PowerPoint


Break-down of a REST API request via COM object

  • Create COM object
  • Open Endpoint (w/ parameters & Authentication if GET request)
  • Set RequestHeader(s)
  • Send (w/”payload & Authentication” for POST requests)
  • Get response (body or text)

Main differences of oAuth1 verse oAuth2

Unfortunately there is no “standard” implementation of oAuth1 or oAuth2 however, at a high-level, here are some of the main conceptual differences:

oAuth1:

  • Need a Key & Token from Webservice API (typically different than your username and password)
  • You use your Key & Token in your API call
  • oAuth1 is less secure and, generally, being phased out
  • While being phased out often the “developers” (us) can use oAuth1 for development of the “app”

oAuth2:

  • Need a Key and Token (same as oAuth1) however you use the Key & Token and some other parameters to perform a “handshake” which returns a secure token which typically times-out in seconds / minutes/hours
  • Your token is restricted to the level of your account (or what has been authorized)
  • The secure token is what is shared with your actual endpoint. (this allows other Social sites (like LinkedIn, Facebook, etc.) to assist your login but not have your username/password to the endpoint

HTTP and AutoHotkey Resources

HTTP Protocol & General Tutorials

AutoHotkey specific



Let’s make some Webservice / API calls!

Weather Underground

  1. Dallas forecast in XML
  2. Dallas forecast in JSON
  3. Using WinHttpRequest
  4. Example with browser

Yahoo business

  1. Pizza restaurants in zip code 75019 in XML
  2. Pizza restaurants in zip code 75019 in JSON
  3. Using WinHTTPRequest

AutoHotkey Webinar- Intro to Web Scraping

In this AutoHotkey Webinar we cover an Intro to Web Scraping.Intro to Web Scraping

Video Hour 1: High-level:

  • What is an API call
  • Web Scraping vs. WebServices / API
  • Document Object Model (DOM)
  • Structure of a web page
  • Frequently used Methods & properties

Video Hour 2: Coding and Q&A

  • Building code
  • Using AutoHotkey syntax writer
  • IE verse Edge
  • Selenium for Chrome, Firefox, IE , etc.

Script Highlight: Long-pressing Hotkey to Exit a script

The below script is more of an example of how you can have more than one hotkey for a given key by tying it to how long you press the key.  In the below script, if you hold down Escape longer than 1/2 a second it will close the active program.

 

PowerPoint Deck presented in Intro to Web Scraping

Q) What is an API? (Application Programming Interface?)

A) Basically it is how devices (computers, tablets, phones, servers, etc) talk to each other.

Q) What are examples of common API calls

A) Below are a list of some examples (but the list is limitless)

  • Web Browser pulling up products on Amazon.com
  • Application / Software querying products for sale on Amazon.com
  • App on your phone getting latest Weather
  • Database pulling updated sales report
  • Using your Tablet to Select movies to watch on Netflix
  • DropBox application syncing files between your computer & cloud
  • Using Web browser to access files in DropBox

Q) What are the Main Differences between Webservice / API call & Web Browser

Q) What is the HTML DOM: Document Object Model

A) Think of the DOM as the “road map” for a webpage.  Depending on what you are trying to get/set, you’ll rely on the DOM to get the exact element you’re looking for.   Below is an image which stresses the more-frequently-used parts.

Q) How does a Family Tree similar to the DOM?

A) Both are “hierarchical” and, depending on what elements are populated, you can reliably “talk” to the correct node.  The first tree below shows how some nodes have all elements populated while some are missing some (like Name and ID)

Q) What happens if a new Node is inserted in the Tree?

A) It depends on which element you are looking at.  Names and IDs probably will not change however Tagnames often will.  This is why it is recommended to first use IDs, then Names, then ClassNames.  Use Tagname last (or go up the tree a bit and grab an ID/Name then use that “branch”)

Q) What are the frequently used methods for selecting an Element?

A) The methods below are sorted by the most reliable/stable (not presence).  When ID is present- use it!  If no ID, Name is another great one…  Classname is also becoming more reliable

  1. getElementByID– Great because if it is present it is should always unique
  2. getElementsByName– While not required to be unique, it often is so very reliable
  3. getElementsByClassName– Often unique and present
  4. getElementsByTagName– Check if there is an ID/name above it, then jump “down” to it first then use TagName)
  5. querySelector/querySelectorAll- Most flexible but is the most complicated- (If you know CSS, this is much not hard to adapt)

Q) What are Frequently used Get/Set/Trigger HTML Elements?

A) The below are the most frequently used however this is highly tied to what your goal is.

Methods:
.click() – Clicks the element
.fireEvent(“onclick”) – Fires the click event (sometimes needed to trigger the event)
.focus() – Place the cursor in the element/edit field (set cursor)
.selectedIndex() – Dropdown box selection
.checked() – Radio / Checkbox selection

Properties (Get & Set):
innerText – the plain-text for the element
.value – the plain-text for the element
.outerHTML – the HTML for the element

Additional Resources

AHK specific:

Learning the DOM (If you spend time on this you’ll thank us later!):