How the S Pen for foldables evolved with AI technology – Samsung Global Newsroom
This year, Samsung Electronics’ portfolio of high-performance foldable devices has expanded with the release of the Galaxy Z Fold3 and Galaxy Z Flip3. In addition to the Galaxy Z Fold3 with a larger screen, the device has now evolved to include an S Pen for the first time with a foldable phone. The inclusion of this specially designed S Pen allows users to increase their productivity and ensure that they are getting the most out of every moment.
But Samsung’s new cutting-edge foldable solutions could not have been achieved without the hard work and collaborative efforts of Samsung’s global network of engineers. Among other achievements, these employees were able to incorporate advanced technologies such as AI to enable drawing and writing on the Galaxy Z Fold3’s big screen, and to enable the device to convert handwriting into a range. languages to text on screen with S Pen.
To find out more about what happened in the development of the innovative new S Pen, Newsroom spoke with Olga Radyvonenko from the visual intelligence team at Samsung Research Ukraine.
Q: Can you briefly introduce the Ukrainian Research Institute and the type of work that takes place there?
Artificial intelligence (AI) is one of the key technologies at our R&D center. Our lab is focused on providing Samsung users with human-centric AI technologies that make typing and interacting easy and enjoyable.
Q: The S Pen has now been applied to the Galaxy Z Fold3, making it the first of its kind among foldable smartphones. What role did you play in this process?
The main contributions of our lab to the Z Fold3 have been the development of in-device deep learning technologies for handwritten content recognition and S Pen input latency reduction technology that combines advanced graphics. and AI.
Q: What is S Pen SDK? What particular advantages does this bring?
The S Pen SDK is developed by Samsung researchers and developers from many countries around the world, and we are very proud to be part of this international team. The S Pen SDK provides ways to speed up rendering and reduce and optimize S Pen input latency. In addition, it includes intelligent AI-based features like recognition of handwritten text, shapes, diagrams, and handwriting straightening.
Q: Given the unique function of the S Pen on the Z Fold3, is there any function of the S Pen that has been developed or extended specifically for the foldable smartphone?
As the new Z Fold3 features a massive and unique foldable display, we focused on using our artificial intelligence technology to optimize the S Pen experience and improve latency. Using our deep learning-powered dot prediction technology, we’re able to predict the next movement of the S Pen to minimize the delay between entry and exit on the Z Fold3 screen. This, combined with the front stamp rendering technology and the unique large display, means our users can enjoy the most natural handwriting, note taking and drawing on their Z Fold3.
Q: How many languages does the S Pen now support? What were some of the difficulties encountered in increasing the number of recognized languages for the Z Fold3 S Pen?
Right now, the S Pen supports 88 languages in total, 14 of which were recently added last year.
Adding a new language for handwriting recognition takes a lot of effort. In order to achieve the best level of recognition quality and speed in the world, we have developed advanced AI technology. This technology is based on deep learning over millions of user samples and language modeling for natural language processing (NLP).
Q: Do you have any specific tips for getting the most out of the S Pen on Z Fold3 to share with users?
For those who like to write by hand, the S Pen To Text feature now supports writing with the S Pen directly into text fields. The handwriting is then automatically converted into text. Also, you can edit any text printed with your S Pen, for example, when correcting spacing or inserting words.
One feature that will be of interest to many users as well is the Neat Shape mode. It can help you not only to draw shapes, but also to create flowchart.
Make sure you check out the Tips app and don’t ignore the usage tips. Here you will learn about the latest S Pen and input features, and also learn how to use many other Samsung apps to their full potential.
Q: Looking to the future, what do you think the future of the S Pen looks like? How can it evolve?
We hope to introduce more ways to support better user experiences by providing even more personalization in the near future. Our mission is to make S Pen typing like writing in your notepad with a pen, but better and smarter.
Above all, I wish that as many people as possible could experience the power and convenience of the S Pen and its handwriting input capabilities. It has the potential to improve input on a wide range of devices. We see the S Pen not only as a cool gadget, but above all as an accessible and useful tool. I hope I see the S Pen and its features used casually in education, the workplace, and everyday life. I think we are going to see a revolution in input technology.
The full video of the interview can be viewed here: