Skip to main content


An idea I have been wrestling with for a while: a mobile app which allows you to create screens and simple screen sequences where you can place any kind of arbitrary elements known to the host system. Images? No problem! Fake buttons with accessibility properties such as fake rotor actions? Here you are! I short: any kind of native control that the system can recognize created in a step-by-step fashion that everyone can familiarize themselves with. Things this could be useful for: 1. Creating fantasy apps - hey, why not call it just that?, or more professionally: mockups in which a screen reader user demonstrates the ideal scenario of how a given interface should read. 2. Copying interface concepts from apps you would like to demo where sensitive data is stored that you wouldn't like to be there for everyone to read. 3. Setting up maps of touch screen interfaces not accessible to the blind for the purpose of visualising them so that you can still support a sighted third-party in handling (think vending machines, ticket machines etc. where someone is lost but you can help them out by looking at your visualisation). 4. Any kind of visual concept really with things laid out in a structural way. Bonus points if the controls can do simple things like navigating to other views, fading the current view out but fade another one in etc. #iOSDev #App #Accessibility #A11y #Blind #Idea
in reply to Paweł Masarczyk

If you were to do this for Android, you could even go as far as dumping the UI tree from an existing app and letting the user edit it.
in reply to Mikołaj Hołysz

@miki Thought of that. On iOS this could be a job for some kind of LLM give or take some hallucinations.
in reply to Paweł Masarczyk

Now this makes me wonder, if we had a few Android-using volunteers and an app that occasionally dumps the current app's accessibility tree and makes a screenshot, could we train a model for screen recognition? As far as I remember that paper, it's exactly the technique Apple used, so the answer is "probably yes".

This would also work on desktop, regardless of platform, basically everything but iOS.

in reply to Mikołaj Hołysz

@miki That's something that I wouldn't be surprised if the Jieshuo dev took up at some point seeing how he's introducing a basic LLM now.
in reply to Paweł Masarczyk

That would make sense.

Apparently you can get Chinese GPU instances at insanely low prices right now, so he'd have an easy time doing it.