OT: VScan: Turn your smartphone into any accessibility aid you can imagine with GPT4 vision

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

 



Hello everyone,
here comes my next idea & project. If you can think of a system & user 
prompt that would turn GPT 4 vision into an accessibility aid, i.e. by 
telling it what to look for in images and how to tell you the output, 
and then if you can simply do photos against these prompts using your 
smartphone, then you can basically turn your smartphone into a pretty 
wide range of accessibility tools (color detector, text reader, expiry 
date extractor, navigator, etc.).
I decided to try this in practice, and the results are pretty 
interesting! Well, you can try yourself:
https://github.com/RastislavKish/VScan

Note the app has been designed such that it can be easily used both for 
tools creation as well as standard image recognition you may be used to 
do with Be my AI or my Vision project, or you don't even need to be 
taking pictures at all, you can use the app to simply chat with GPT 4V 
(the model has the same textual capabilities as GPT4).
Though note there is currently no chat history review functionality nor 
conversation truncation when the 4k token limit of GPT 4V is exceeded, 
so the app is not optimized for this use-case.

There are few rough edges by now, probably the most annoying issue I'm 
facing on my device is that Talkback shows the braille keyboard in a 
reversed position i nthe session screen due to the display orientation, 
I need to figure out how to make CameraX adapt for the current device 
orientation.

But that's mostly a minor issue, the main functionality works as expected.
Any constructive thoughts and opinions on this project are very welcome, 
and, if you get to create some interesting accessibility tools you would 
like to share, I would love to hear about them!

Happy Visioning!

Best regards

Rastislav


-- 
You received this message because you are subscribed to the Google Groups "blinux-list@xxxxxxxxxx" group.
To unsubscribe from this group and stop receiving emails from it, send an email to blinux-list+unsubscribe@xxxxxxxxxx.




[Index of Archives]     [Linux Speakup]     [Fedora]     [Linux Kernel]     [Yosemite News]     [Big List of Linux Books]