Skip to main content

I recorded a quick demo of the current state of the Wayland-native accessibility stack I'm developing for @gnome (and eventually other free desktops), code-named Newton. In the demo, I run the GNOME Podcasts app on the Newton accessibility stack, using my GTK branch that integrates AccessKit. So in principle, the same app should also work on Windows, and soon, macOS. The demo was featured in the latest edition of This Week in GNOME: thisweek.gnome.org/posts/2024/… This work is funded by @sovtechfund.

#148 Variable Styles · This Week in GNOME

thisweek.gnome.org
@GNOME @Sovereign Tech Fund

Peter Vágner reshared this.

in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
Instructions for setting up the whole stack are in the README on my branch of Orca: gitlab.gnome.org/mwcampbell/or… The instructions assume some familiarity with building and installing system components like GTK and Mutter using Meson, in a way that doesn't break your distro installation or on a throwaway system. Let me know if you have any trouble, or if you get it working.

Files · newton · Matt Campbell / orca · GitLab

Welcome to GNOME GitLab
GitLab
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Woohoo, I have missed your instructions. I am just trying out if I can build it and play with it on my Arch linux install. Thanks for working on it.
@Matt Campbell
in reply to Peter Vágner

Peter Vágner
Peter Vágner
•

@Matt Campbell Yes, I made it work by rebuilding affected @Arch Linux :archlinux: packages from your branches. Flat review, key echo, mouse actions are working fine with GTK4 apps such as gnome-calculator, gnome settings and similar. Even at this stage it's perfectly useable. Another bonus, The accesskit based GTK4 is also going to provide accessibility on windows. Am I understanding that right?

Things I have noticed so far:
* With at-spi2 based GTK4 GTK labels are mapped to the label role. With accesskit based implementation it's mapped to static.
* In nautilus when typing into the address bar with at-spi based implementation auto completions are reported as they are inserted. So text insertions / deletions are not yet available with accesskit I guess.
* When restarting orca and nautilus is the active window this newton based orca causes nautilus to freeze I guess.

Once again thanks for your dedication and awesome work.

@Matt Campbell @Arch Linux :archlinux:
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •

@pvagner Hi, thanks a lot for building and testing.

Yes, the GTK AccessKit branch is already working on Windows. And it shouldn't be hard to make it work on macOS; I just have to write a bit of platform-specific code and figure out how to build on that platform.

I need to look into the first two bug reports. As for the third one, about stopping and restarting Orca, I think that's actually the same issue I'm working on right now.

@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Regarding windows that sounds fantastic. I imagine I might be able to recommend windows folks to run #fractal matrix client soon enough.
For building GTK with accesskit on windows do you use msis2 or gvsbuild?
#fractal @Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner I haven't used gvsbuild yet, but I have built GTK with the Visual Studio tools, just running Meson from a command prompt and letting it pull in dependencies via subprojects. I have also built it using msys2, which is where I've done most of my Windows testing lately, because that was the only way I knew that I could get it working with gnome-text-editor for some real-world GtkTextView testing.
@Peter Vágner
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@pvagner I think I heard the name Fractal before, but didn't know what it was. How accessible is Fractal with AT-SPI?
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Fractal is very accessible on linux. They have specifically addressed accessibility issues in few recent releases. I think there are things to improve e.g. autocomplete of mentions, but I think the app is generally accessible and nice to use.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •

@pvagner Would you please test it with your Newton installation and let me know how it compares?

By the way, with the GTK AccessKit branch, you can easily switch back to the AT-SPI backend by setting the GTK_A11Y environment variable to atspi

@Peter Vágner
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@pvagner Of course, if you already switched your installation back to the released packages so you could have a fully working system back, that's fine.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Yes, I've switched back for now as emulating mouse clicks on the web is holding me from switching fully to wayland and also it's a late night here. Still I'm extremelly happy about the outcome. It's going to be a huge success I am sure. I will sure post more as I notice it. Also I've tweaked all the needed pkgbuilds, so it's easy for me to rebuild updates as they appear. Huge thanks really!
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Thank the Sovereign Tech Fund for paying me to work on this. I look forward to hearing more tomorrow. Good night.
@Peter Vágner
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@pvagner Well, to be precise, STF isn't paying me directly; the GNOME Foundation is.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•

@Matt Campbell So I am back testing fractal and trying to compare how it performs with atspi and accesskit.

  • Fractal has tree of conversations on the main window. Root entries are room types such as explore, invited, favorite, low priority and similar. These items might have expanded / collapsed state. With accesskit this state is not yet reported.
  • When GTK4 label is focused, it's possible to use arrow keys to navigate with the caret on that label. With atspi orca is reporting caret navigation and text selection as it happens. With accesskit this is not yet working. I am not sure this might be an issue with the accesskit, newton prototype though, because with older versions of GTK4 this was not working either. Where can I put that environment variable to force atspi even with accesskit based build of GTK4? I am running this on the full gnome 46.
  • There is another reason why I think there might be difference with the GTK4 version I am running against atspi as compared to accesskit. Fractal has a list of room events called room history. Those are messages, files, recordings and other items found on the rooms timeline. With atspi this list is reported as room history and using up and down arrow key to navigate in this list does not cause orca to report items in this list as they are being focused. With accesskit this list is working perfectly.
  • With accesskit flat review is working in much more places including nautilus and fractal. With atspi orca freezes when trying to use flat review with these apps.
  • Flat review with accesskit feels damn snappy. Unfortunately I don't know how to prove it and make sure if it's for real or if this is a part of me being so much excited. Perhaps some control coordinates may need some refining or orca needs to be tweaked to the new way of doing things as the content now appears to be mostly presented on very long lines when using flat review. And also I think there might be some invisible content being presented as a part of flat review.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •

@pvagner Thanks for the feedback. I had no idea that a GtkLabel could have a caret. Should be easy to fix in my GTK AccessKit implementation.

As for the environment variable, the only way I know to set the variable would be to run the app from a terminal rather than starting it the normal way.

@Peter Vágner
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Also, can you please try the last push I did to GTK yesterday and let me know if that solves the problem you had with restarting Orca while Nautilus is running?
@Peter Vágner
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •

@pvagner Please rebuild the newton_atspi_compat Python package from the newton_consumer repository here: gitlab.gnome.org/mwcampbell/ne…

I moved the package so the low-level and high-level Newton consumer crates are consolidated in one repo. And since doing that move, I've pushed a couple of important fixes. Yes, I've updated my Orca README.


Matt Campbell / Newton consumer library · GitLab

Welcome to GNOME GitLab
GitLab
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Thanks for the updates. I'm rebuilding now. I'll report back soon.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner I know you just rebuilt, or are still rebuilding, but I just pushed support for selectable labels in GTK
@Peter Vágner
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Please try my latest Orca update and let me know if it improves flat review for you.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•

@Matt Campbell I apologize for the delay.

Good news first:

  • text review and text selection for GTK labels is now working on both atspi and accesskit respectivelly. Huge thanks!
  • Restarting orca while GTK4 app such as nautilus, gnome-calculator or fractal is in the foreground is now safe. I see no more exceptions when releasing keygrabs on exit.
  • This is true from the moment I have started testing this yesterday I have just forgotten to mention it. When key echo is enabled in orca with newton it correctly handles accented non ascii characters while typing. I have tested web content in firefox and chromium, gnome-terminal, libreoffice, gedit, nautilus and fractal.

And now the rest:

  • After todays orca update flat review now just displays window title for me for example name of the folder that is opened in nautilus. I think I have updated everything correctly but I can double check it later today.
  • Reporting expanded / collapsed state is really an accesskit integration issue I am afraid as switching to atspi on the same GTK4 version I am getting these reported fine.
  • Before starting this yesterday I was using GTK4 4.14.4 as packaged in archlinux. That version has some positive and some negative changes as well than the version you have started with while integrating accesskit. For example the thing with conversation history in fractal I have mentioned in my previous report is broken with GTK4 4.14.4 as packaged in arch linux but it's working fine with both atspi and accesskit when using your branch. Thus I think it's not related to your work. At the other hand in fractal there is a message composer which has some buttons that inwoke a popup menu such as more options. When that menu is dismissed fractal freezes with atspi and accesskit when using GTK4 built from your branch and with the arch linux packaged GTK4 version it's working fine.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Those differences between the Arch Linux GTK package and my version when using atspi are weird, because, as of a few days ago, my branch is based on GTK 4.14.4, with few modifications to GTK outside of the AccessKit backend itself.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Hmmm, I can't really think of what else to try. I will also rebuild the same version of GTK4 their are shipping on arch linux out of interest just in case.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner I think I got flat review working correctly. Please pull the latest update to the newton_consumer repo and rebuild and reinstall the newton_atspi_compat Python package from there.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•

@Matt Campbell Thank you again, I've great news this time as well. After updating the newton_atspi_compat package the flat review with newton is working fine. With nautilus it started to work immediatelly when I've restarted orca with the updated newton_atspi_compat package in place. With fractal it started to work after restarting fractal.

Also I must say I have found my-self reporting a strange thing yesterday. I have mentioned I have tried gedit as a part of this testing however now I have realized gedit is a GTK3 app not GTK4.
When looking at gedit main window I can use orca's flat review to explore its toolbar including moving the mouse pointer and synthesizing mouse clicks. When doing the same thing with GTK4 apps such as nautilus or fractal the flat review is working great but emulating the mouse is not.

Excuse me for this mistake. I got it wrong.

And just now I have noticed another thing. I wanted to verify the mouse emulation features by running gtk-demo app which is GTK3 app too. I have tried to run gtk-demo from the gnome-terminal. As it has started I have noticed orca briefly reporting GTK demo however I can't focus its window. When holding down the alt key and pressing the tab key to cycle through the open windows, I can hear GTK Demo is part of the window switcher but when releasing both the keys that window is never brought to the foreground. When expanding activities from the gnome top pannel I can't locate GTK demo either. The very same thing has happened to me with an electron based app called losslesscut but originally I have assumed that was a momentarily thing related to electron but now when seeing it for the second time with a different toolkit I am wondering if this might be related to the mutter changes.

I am sorry for mentioning too many things. For some of these I am really unsure so if it's not relevant please ignore those parts.

@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •

@pvagner The next time you update, it's crucial that you update newton_atspi_compat (again from the newton_consumer repo), the AccessKit C bindings, and GTK all at the same time, and restart any running GTK 4 apps using the AccessKit backend before you restart Orca. I made a protocol-breaking change. Also, labels now have the correct role.

I'll look at your other reports later. Thanks again.

@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell I managed to update. I can no longer reproduce freezes of fractal I mentioned yesterday and I can recognize the labels related change. Thank you so much.
@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Thanks. So then, if I remember correctly, the only issue you're having with Fractal now is the lack of expanded state on the tree items, right?
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•

@Matt Campbell Yes with fractal this is the only issue that has left.
Related to the other thing that is not related to fractal Now I managed to find out downgrading GTK4 to the 4.14.4 as packaged in arch linux it helps with electron and GTK2 based apps not being focusable through the alt+tab window chooser.

Also have you noticed my post regarding nautilus and accessible name calculation I have posted to the orca list? This is not related to your work in any way so far. I am just wondering if accesskit wouldn't be able to make this part better than what we currently have while using atspi directly. In other words while calculating label of an accessible node should it walk its entire subtree?

@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner I did see that post. In the accessible name calculation used by the GTK AT-SPI backend, the decision to walk the subtree for some roles (like buttons) but not others (list items) seems to be deliberate. I don't know why. The GTK AccessKit backend is using AccessKit's own name calculation, which currently doesn't walk the subtree for list items either, but I could easily change that.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Hmmm, so it's even more complicated than I was imagining. I think what is done within the nautilus files view is not entirely right either. At the other hand projects with good a11y reputation such as thunderbird are doing it too. Thunderbird has aria-label on its message list items too. When looking at nautilus window with flat review or object navigation all the textual information is there. If using it the way it has been designed just using standard navigation keys such as arrow keys it's looking very poor.
@Matt Campbell
in reply to Peter Vágner

Peter Vágner
Peter Vágner
•
@Matt Campbell I took a look at more apps that have lists as their integral part. Audio player #amberol has a playlist where its list items would benefit from calculated accessible names. gnome-podcasts looks as if it's already doing what we need the items have all the text from their children. I haven't looked how they are doing it though. Various lists within the gnome control center would also benefit from the accessible name calculation as an alternative to setting accessible manually. For example list of wifi networks is missing state information and signal strength. List of bluetooth devices is missing the state information. List of apps under notifications is missing a status information. List of keyboard shortcuts is missing assigned shortcuts, only action names are part of the accessible name. List of printers has all the details including status, location, description. Actually this might be the only list I am surprised looks about right to me. Hmm, perhaps I shal try to collect all these comments and submit it as a GTK feature request as calculating the labels might really be helpfull it seems.
#Amberol @Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner I'm curious if you've tested any of these apps with Newton yet.
@Peter Vágner
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@pvagner Also, if you update the Newton version of Orca, you should find that left-clicking the current flat review item works now in GTK 4 apps. It's not actually synthesizing a mouse event, but requesting the equivalent accessible action.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•

@Matt Campbell Yesterday I was running newton all day long. And all these tests were made with it. Today I am back to stable versions so far as I had to deal with inaccessible website about a hour ago.

I don't want to sound rude in any way but if I was able to do a little recap I can point out these issues related to newton prototype:

  • I am afraid there might be some invisible content present when using flat review.
  • Expanded or collapsed state is not passed through to orca (tested with fractal room list)
  • When running GTK4 from the accesskit branch, there are issues selecting GTK2 and electron based apps in the alt+tab window chooser.

These are the things I am aware of.
If you need me to test something more thoroughly, please do let me know and I will try to do my best.

@Matt Campbell
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner You're not being rude at all. Thanks for the recap.
@Peter Vágner
in reply to Sonny

Sonny
Sonny
• •
out of curiosity, is there a a11y benefit to running GNOME-GTK apps on macOS/Windows ? Besides using FLOSS apps we know and love
This entry was edited (2 months ago)
in reply to Sonny

Matt Campbell
Matt Campbell
• •

@sonny @pvagner Not sure if this is what you meant, but I asked about possible benefits of the Inkscape UI being accessible on Windows and macOS even if the canvas itself isn't accessible, and I got this reply: mas.to/@ChanceyFleet/112282476…

And yes, blind people (relatively few for now) do work with SVG. See blindsvg.com/ and this article: technologyreview.com/2023/06/1…


How tactile graphics can help end image poverty

Access to rich imagery shouldn’t be something that blind or low-vision people have to do without. These projects are helping.
Chancey Fleet (MIT Technology Review)
@Peter Vágner @Sonny
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@sonny @pvagner As far as I know though, Inkscape's transition to GTK 4 is still in progress.
@Peter Vágner @Sonny
in reply to Sonny

Matt Campbell
Matt Campbell
• •

@pvagner You mentioned that you'd soon be able to recommend Fractal to blind Windows users. Does this mean that you expect Fractal to be more accessible than other Matrix clients on Windows?

@sonny

@Peter Vágner @Sonny
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell @Sonny The most accessible @The Matrix.org Foundation client is now @Element. However this is either an electron app or a web app on windows and linux. Secont best matrix client when it comes to #a11y is fractal. At least I am seeing it that way. Since fractal is a native desktop app it will have its fans for sure. The fact it's very accessible is very good. Additionally I don't know but I haven't seen matrix client targetting windows that is up to date with matrix development. It's why I think fractal is closest to become one once accesskit is merged into GTK4 and fractal is built against it on windows.
#a11y @The Matrix.org Foundation @Matt Campbell @Sonny @Element
in reply to Peter Vágner

Matt Campbell
Matt Campbell
• •
@pvagner Hi. Just wondering if you continue to use Newton at all, or if you went back to released versions full-time so you could use X11.
@Peter Vágner
in reply to Matt Campbell

Peter Vágner
Peter Vágner
•
@Matt Campbell Hello, Huge thanks for asking. I am on holiday since monday and I had no chance to apply updates since that time. Earlier last week I was switching back and forth from newton to the old X based setup. I will definatelly continue to keep tracking the updates and running newton when returning back in some 10 days.
@Matt Campbell
in reply to Matt Campbell

Joëlle
Joëlle
• •
Looks good! I think you mentioned existing ATSPI solutions, do those work on Wayland as well or is this a first?
in reply to Joëlle

Matt Campbell
Matt Campbell
• •
@JoSuus AT-SPI kind of works on Wayland. There are problems though stemming from the fact that there's no connection between the AT-SPI accessibility tree and the actual Wayland surface. For example, Orca can't tell what's under the mouse pointer with AT-SPI on Wayland. That already works in the Newton prototype, though I didn't cover it in this demo. Also, AT-SPI is incompatible with strong sandboxing. I hope to demo a fully sandboxed Flatpak app with Newton soon.
@Joëlle
in reply to Matt Campbell

Matt Campbell
Matt Campbell
• •
@JoSuus Also, with AT-SPI, Orca can't observe keyboard input on Wayland unless you use a toolkit that supports the legacy AT-SPI method of key snooping, which GTK 4 does not, because that old method has various problems, like the requirement of an IPC round trip for every keystroke. I don't know if the Newton prototype's current solution to that problem will make it to production; there have been other efforts to solve that problem on Wayland.
@Joëlle
in reply to Matt Campbell

Joëlle
Joëlle
• •
Sounds amazing. If you ever publish how you worked around all these problems I'd be really interested.
in reply to Matt Campbell

Jason J.G. White
Jason J.G. White
• •
As a user who runs GNOME on Wayland with Orca every day, I need a solution to this, especially before the option of reverting to X11 goes away...
in reply to Jason J.G. White

Matt Campbell
Matt Campbell
• •
@jason Yes, solving the problem of screen reader keyboard handling on Wayland has been within the scope of the GNOME accessibility work funded by Sovereign Tech Fund since the beginning of the STF work last year. It was supposed to be done by someone else in parallel with my project. But that project has run into various delays. I hesitated to even do a partial proof-of-concept implementation in my prototype because I didn't want to step on someone else's turf.
@Jason J.G. White
⇧