Beta 0.9.0 (2024-05-02)

Here’s my new (beta) app for macOS, UI Actions. It’s bringing new Shortcuts actions that’ll allow you to control any app that has menus, buttons, and/or accepts keyboard input. (My initial, internal slogan for it is “scripting the unscriptable”, which is a bit much, so… I won’t use it in official communications.:sweat_smile:)

icon_128x128 UI Actions

In a nutshell: UI Actions adds a number of Shortcuts actions to macOS, for selecting/showing app menus or menu items, for clicking buttons, for simulating typing and key strokes.

Explainer video

I’ve made this 2-mins video, it’s definitely not super and only scratches the surface, but it should give you an idea about what the actions can do:

You mentioned a beta?

Yes! Go to https://actions.work/ui-actions and click the download button. Double-click the downloaded DMG, copy the app into your Applications folder, and launch it from there.

I’ve tried to make it as easy as joining a TestFlight (if not easier). A beta license is baked into each beta app, it’s valid for 2 months, every beta update comes with a new license key, so we get a rolling beta window for as long as the beta lasts, with outdated beta versions auto-expiring.

Why?

I like scripting things. Usually, I use Keyboard Maestro (:fire:!) but because I spend a lot of time in Shortcuts lately, I wanted something more Shortcuts’y. No other app had what I wanted, so here we are.

Will there be an iOS version?

No. Broadly speaking: The tech used to make this work is not available in iOS.

What happens to Browser Actions?

BA was merely put on hold for a few weeks. Work will continue shortly!

In fact, UI Actions was made possible by the foundational work I’ve put in Browser Actions. I love it when earlier planning pans out like this. :wink:


If you have any questions, let me know! The same goes for ideas, bug reports, and suggestions – I’m all ears.

This looks great! This will be very helpful. I agree there are so many things you can do with app automation using Keyboard Maestro, especially UI actions via Interface Control. However, it’s awkward to trigger a KM macro via URL (kmtrigger://macro=My%20Shortcut%20Macro), typed string, or hot key just so you can control an app via menus, buttons, or keyboard input. It will be nice to simply integrate such actions directly in my Shortcuts.

I can trigger a KM macro with URL and Open actions in Shortcuts, and KM macros can invoke Shortcuts with a KM action Execute Shortcut (or shell, JavaScript, AppleScript, or Swift scripts). However, then I need to deal with passing data as input/output between shortcuts and macros as using clipboard, files, or text, list, dictionary variables. Also, control flow is handled differently so I always have to remember what I can do in Shortcuts vs KM; for example, KM has a Switch/Case action whereas Shortcuts only has an if/otherwise action although a Choose from Menu action is similar but requires a menu prompt.

I will enjoy experimenting with UI Actions for app control within my shortcuts. Of course, KM can do more and perhaps some can accomplish a lot without even needing a KM license or figuring out appropriate data exchange.

Thanks, @doug78645! KM is the bomb, and I love it to bits. I don’t want to work w/o it, for sure. But for all the reasons you’ve just listed, UI Actions exists!

I think there’s a sweet spot for automating GUI stuff via Shortcuts – only a fragment of what you can do with KM, but tailor-made for Shortcuts. While KM is aimed at power users, I think; UIA is not.

BTW, it all started with a “huh” moment when I figured out how to control menus for Browser Actions… and simulate keyboard input. :sweat_smile:

I like your comment about “UI sweet spot” automation with Shortcuts vs KM.

There are many apps where I would like to automate menus, buttons, and keystrokes that don’t offer Shortcut actions, and UI Actions does that!

I might even be able to use it with the Arc browser since Browser Actions doesn’t support it (yet).

It’s certainly easier to provide a user interface using Shortcuts rather than KM, especially since it’s a nicer UI and supports Siri and voice, and works on Mac, iPhone, or Watch.

Actually, I’ve been exploring Swift scripts as well as SwiftUI interfaces for both Shortcuts and KM.

1 Like