It’s no secret that many of us in the blind community have embraced the rapid advances in Artificial Intelligence over the past two years. We've witnessed firsthand how these technologies can be a powerful force for good, especially within our community. AI-generated image descriptions have revolutionized how we navigate the online world, offering a perspective previously unimaginable. This impact is now undeniable, transforming how we interact with the world.”

I’ve declared the kingdom of the blind a republic—perhaps prematurely, but only by a small margin. With AI empowering us to perceive the digital world in new ways, we are no longer ruled by limitations, but actively shaping our future. Anthropic’s recent launch of ‘computer use’ marks the first steps into a new phase of AI evolution—one where AI agents begin to act independently on our behalf, initiating a shift in how we interact with technology.

As AI continues to evolve, so too will the Assistive Technology that many of us depend on. I envision a future where this intelligence becomes a true companion, guiding us seamlessly through both digital landscapes and real-world challenges. We may be just two years away from seeing JAWS, NVDA, or SuperNova transform into true Assistive Intelligence 1.0—or perhaps it will take a little longer. If AI has taught us anything, it’s that progress comes both more slowly than we expect and faster than we can possibly imagine.

What follows is my first attempt at describing how a screen reader of today could take the first steps towards becoming an Assistive Intelligence. If anyone wants to build it, I’d love to help if I can. Whatever you think, let me know what you think:

“Proposed AI-Powered Self-Scripting Feature for JAWS Screen Reader

Objective
The suggested feature seeks to integrate advanced AI-driven "computer use" capabilities, like those developed by Claude (Anthropic), into the JAWS screen reader. This functionality would enable JAWS to autonomously create and refine custom scripts in response to real-time user interactions and application environments. The aim is to enhance accessibility and productivity for visually impaired users, especially when navigating non-standard or otherwise inaccessible software interfaces.

Feature Description
The self-scripting capability would empower JAWS to analyse user interactions with applications, identify recurring actions or inaccessible elements, and generate scripts that optimize these processes. By enabling JAWS to perform this autonomously, users gain seamless and personalized access to applications without manual intervention, allowing for an enhanced, efficient experience.

The self-scripting feature will be powered by the following core functions:

1. Real-Time Autonomous Scripting: JAWS would use AI to observe user interactions with applications, especially non-accessible ones, and automatically generate scripts that improve navigation, label untagged elements, and streamline frequent tasks. For example, if a user frequently navigates to a particular form field, JAWS could create a shortcut to this area.

2. Adaptive Behaviour Learning: This feature would allow JAWS to recognize patterns in a user’s interactions, such as repeated actions or commonly accessed elements. JAWS would adapt its behaviour by creating custom macros, enabling faster navigation and interaction with complex workflows.

3. Dynamic Accessibility Adjustment: Leveraging Claude’s approach to visual recognition, JAWS could interpret visual elements (like buttons or icons) and provide instant labelling or feedback. This would be valuable in software with minimal accessibility features, as it enables JAWS to make live adjustments and effectively “teach itself” how to navigate new environments.

4. Community Script Sharing: Self-generated scripts, once verified, could be anonymized, and made available to other users via a shared repository. This would foster a collaborative environment, empowering users to contribute to a broader database of accessibility scripts for applications across various industries.

Value Proposition
This feature will address key challenges for visually impaired users, including the complexity of navigating inaccessible interfaces and the time-consuming nature of repetitive tasks. The ability for JAWS to generate its own scripts autonomously would mean:
1. Increased Accessibility: Improved interaction with non-accessible software interfaces.
2. Higher Productivity: Reduced need for external support or manual scripting, allowing users to accomplish tasks more independently.
3. Enhanced User Experience: Scripting and macro creation based on personal usage patterns -- leads to a more intuitive and personalized experience.

Technical Considerations
1. Performance: Processing real-time visual and user interaction data requires substantial computing power. A cloud-based model may be optimal, offloading some processing requirements and ensuring smooth, responsive performance.
2. Safety: Automated scripting must be closely monitored to prevent unintended interactions or conflicts within applications. Integration of safeguard protocols and user settings to enable/disable autonomous scripting will be essential.
3. Privacy: To ensure user data is protected, anonymization protocols and data privacy standards will be implemented. Data collected from user interactions would be handled in compliance with rigorous privacy standards, safeguarding user preferences and behaviour.

Conclusion
Integrating AI-powered self-scripting capabilities into JAWS would represent a significant leap in screen reader technology. By allowing JAWS to, when requested, autonomously learn, adapt, and script in response to user needs, this feature could provide visually impaired users with unprecedented control and flexibility in navigating digital environments, fostering both independence and productivity. The anticipated benefits underscore the feature’s potential to redefine accessible technology, turning screen reader into Assistive Intelligence.“

About the Author:

Lottie is a passionate advocate for the transformative potential of AI, especially within the blind and visually impaired community. She blends technical insights with a keen awareness of lived experiences, envisioning a future where AI doesn’t just assist but truly empowers. Her thoughtful reflections explore the shift from a "kingdom of the blind" to a republic, where emerging technologies like AI create new opportunities for autonomy and inclusion.

With a balance of optimism and critical realism, Lottie acknowledges the game-changing impact of AI tools like image descriptions while recognizing that more progress is needed. Her vision extends to the idea of "Assistive Intelligence," where screen readers like JAWS evolve into proactive companions, adapting to users' needs in real-time.

Known for turning complex ideas into actionable blueprints, Lottie is not just an observer of technological trends but a catalyst for innovation. Her proposals reflect a desire to elevate independence and productivity for blind users, pushing the boundaries of what's possible in assistive technology. Her insights continue to inspire conversations and shape the future of accessible tech.

I am the Blind AI, relying on AI every day to enrich my life. While my posts may occasionally benefit from AI assistance, the thoughts, perspectives, and final edits are entirely my own. AI is my tool, much like a calculator or spell-check, refining my expression but never replacing my voice.

#Accessibility #AI #AIsoftheBlind #Blind #ComputerVision #Disability #Innovation #JAWS #NVDA #ScreenReader #SuperNov

Hi everyone,

#ThreadsAndCoffee, a digital English speaking #FiberArts club based on FLOSS softwares, has had its 1st session today and will repeat next week!

Join us on saturday,
2d of november, at 8:30am UTC,

on JitsiMeet
(I expect you to have a microphone if you're able to talk, but camera is optionnal, even though it is useful to show your work!) :

meet.jit.si/moderated/1bfd60a6…

Please boost this post if you like the idea, like it if you plan to come!

#Knitting #Crochet #Sewing

@fiberarts

So apparently there are "AI" workshops about how to do better prompting, and they claim that offering chat gpt a tip makes it answer more accurately???

MY GOD, THIS IS JUST AN ASTROLOGY FOR TECH BROS

Welcome to the RB family, OwnDroid 🥳

apt.izzysoft.de/packages/com.b…

OwnDroid lets you use Device admin and Device owner privilege to fully manage your Android device. Thanks to joint efforts with its author, this app is now reproducible (starting with v6.1 which will go live with the next sync in less than 6 hours)

#reproducibleBuilds #IzzyOnDroid

„OwnDroid“ – IzzyOnDroid F-Droid Repository

Manage your device with Device owner privilege

^{IzzyOnDroid App Repo}

#curl is in the Mini

(via Alejandro Pablo Revilla)

Why don't you want to see this ad?

□ Sensitive topic
□ Not relevant to my interests
✅️ I don't want every second of my life to be monetized by blood sucking parasites

Holy shit. The LA Times didn't just squash a Kamala Harris endorsement. They buried an entire series of articles about Donald Trump.

"According to internal memos viewed by TheWrap, the series, tentatively called 'The Case Against Trump,' would have ran throughout this week. The endorsement of Kamala Harris would then have been published on Sunday."

thewrap.com/la-times-case-agai…

LA Times Planned 'Case Against Trump' Series Alongside Kamala Harris Endorsement Before Owner Quashed It | Exclusive - TheWrap

Patrick Soon-Shiong's interference with the paper's editorial freedom has sparked a crisis that includes canceled subscriptions and several high profile resignations

^{Ross A. Lincoln (TheWrap)}

We've been having great conversations at the Ubuntu Summit in The Hague this weekend - it's been fantastic to be with so many other open source projects, like @frameworkcomputer @ubports and of course, @ubuntu !

#UbuntuSummit #OpenSource

okay, i'm being put under pressure to allow my (12yo) daughter to have snapchat, so i created myself an account to check it out.

what

the

fuck

is

this

shit

#gnome got stuck on #Wayland on the menu again, I can't exit it and "Alt+F2 r" says not available on wayland.
Is there a way to recover from this without closing all apps?

Do you had problems like this with gnome before or am I one of the few people that has issues?

C is such a piece of shit language. Only language where "I need a dynamically growing array or a hashmap" is answered with "roll your own lol n_n"

EDIT: C (not C++, not Zig, not Rust) is the correct language for the job I'm doing. I am also very aware of how to implement dynamic arrays and hash tables. I have written my own stdlibs on freestanding C before. I do not actually need coaching in this matter, thank you. I also like C, sometimes, but sometimes I think it's a stupid piece of shit language, because it is

My family is a Linux family: both my parents and my wife use Linux, and so will the kids once they have their own computers. Apart from my Dad, they're all non-enthusiasts.

Yet, none of them use a beginner-friendly distribution. My Wife, who's by far the least technologically inclined of them all, is a very happy and satisfied NixOS user. A distribution that's the exact opposite of beginner-friendly.

How is that possible? None of them maintain their systems. I do that for them. In the past two decades, what I learned from helping friends and family with their computers is that the vast majority of non-enthusiasts have absolutely no desire to install and maintain an operating system, or even programs. They're much happier if someone else does that for them.

Therefore, while I see value in distributions aimed at beginner enthusiasts, I see very little value in distributions aimed at non-enthusiasts.

I wrote a few more words about this topic on my blog too.

#linux #desktop #gnome

Hey, hi there 👋 yeah, you 🫵

It's almost Follow Friday and you seem pretty awesome

So if I don't follow you or boost your stuff, it's probably because you don't regularly include alt-text on your pictures or videos.

You see, I have visually impaired followers, and I like the stuff I share to be accessible to them.

I also won't boost things without reasonable CWs for similar reasons—I have followers that don't want to see some of that shit, and I want to keep their Fedi feeds a safe space.

Thx! I look forward to boosting your stuff soon 💕

#AccessibilityIsHotAF #UseAltText #UseCWs #FollowFriday #Boost