AllYouCanTech

Apple Scientists Create AI System Able to Reason Regarding App Interfaces

**ILuvUI: An AI That Surpassed Its Foundational Model**

A recent study supported by Apple, in partnership with Aalto University in Finland, unveils ILuvUI: a vision-language model designed to comprehend mobile application interfaces through screenshots and natural language interactions. This research addresses a persistent issue in human-computer interaction (HCI): training AI models to interpret user interfaces as humans do, both visually and semantically.

Grasping and automating tasks on user interfaces (UIs) poses difficulties due to the intricate layers of information embedded in UI components, like list items, checkboxes, and text fields. Although large language models (LLMs) have demonstrated exceptional abilities to understand task instructions expressed in natural language, relying exclusively on text descriptions of UIs overlooks the abundant visual data available in the interface.

Most current vision-language models are developed using natural images, which restricts their efficacy in analyzing structured environments such as app UIs. The integration of visual and textual information is essential for understanding UIs, as it mirrors the way humans interact with their surroundings. Vision-Language Models (VLMs) usually process multimodal inputs of images and text but frequently produce only text outputs, which diminishes their effectiveness in UI applications due to a deficiency of UI examples in their training datasets.

To tackle this challenge, researchers fine-tuned the open-source VLM LLaVA and modified its training methodology for the UI context. They trained ILuvUI using synthetically created text-image pairs, encompassing Q&A-style dialogues, comprehensive screen descriptions, anticipated action results, and multi-step strategies. The resulting model surpassed the original LLaVA in both machine evaluations and human preference assessments.

ILuvUI eliminates the need for users to indicate a specific region of interest within the interface; it comprehends the entire screen contextually from a straightforward prompt. This proficiency allows it to deliver responses for scenarios such as visual question answering.

**In What Ways Will Users Gain from This?**

Apple’s researchers propose that ILuvUI may prove advantageous for accessibility and automated UI testing. Prospective enhancements could include larger image encoders, better resolution management, and output formats that integrate smoothly with current UI frameworks, like JSON. Furthermore, merging ILuvUI with ongoing investigations into AI models that forecast the repercussions of in-app actions could lead to major advancements, particularly for users depending on accessibility features or seeking more autonomous management of in-app processes.

New Enhancements for iPhone Screenshots in iOS 26

# Screenshots in iOS 26: Fresh Features and Functions

Screenshots in iOS 26 are evolving significantly, introducing new abilities that will give you even more motivation to capture a screenshot on your iPhone. Here’s what’s new.

## Visual Intelligence Brings Fresh Reasons to Capture Screenshots in iOS 26

Traditionally, there have been two primary motives for taking a screenshot on your iPhone:

1. To save an image for future reference
2. To share an image with others

However, iOS 26 is set to introduce three additional reasons.

This is because on [Apple Intelligence-compatible devices](https://9to5mac.com/every-device-that-supports-apple-intelligence/), screenshots will receive a visual intelligence boost in iOS 26, adding three exciting new functionalities.

Each visual intelligence feature enables you to interact with the contents of a screenshot in various ways. Here’s what’s on the horizon.

## #1: Add to Calendar

Scrolling through information for an event that’s coming up? In iOS 26, you can take a screenshot and see a new option at the bottom: Add to Calendar.

Instead of having to manually enter events into your calendar, this new button will pull the details of the captured event and automatically create a calendar entry.

What’s left for you? Just tap to confirm.

## #2: Image Search with Google, Etsy, and More

Image Search is yet another powerful new screenshot feature in iOS 26.

Capture a screenshot and you’ll notice an option in the bottom-right corner to search for similar images.

Typically, this feature will be powered by Google, but [Apple states](https://www.apple.com/newsroom/2025/06/apple-intelligence-gets-even-more-powerful-with-new-capabilities-across-apple-devices/) it’s even broader than that:

> Users can…search Google, Etsy, or other compatible apps to find similar images and products. If there’s an object a user is particularly interested in, such as a lamp, they can highlight it to look for that exact item or related objects online.

Shopping is definitely expected to be a popular application for the new Image Search feature.

## #3: Ask ChatGPT

Have a question regarding what’s in your screenshot? Now you can inquire with ChatGPT about it.

A new ‘Ask’ button located in the bottom-left corner of the screenshot interface activates OpenAI’s assistant. You can type any question you wish, and ChatGPT will process the image and reply accordingly.

There’s even an option to select a specific part of the screenshot that you want to ask ChatGPT about. No need to crop an image first for a more detailed response.

## Screenshot Features in iOS 26: Summary

Visual intelligence has become

watchOS 26 Debuts Capability to Minimize Notification Interruptions

# watchOS 26: Upgrades and New Features for Your Apple Watch

Apple continues to push the envelope with its watchOS releases, and watchOS 26 is no different. This new version brings forth an array of enhancements intended to improve the user experience, including fresh watch face tips, an exclusive Notes app, and a more robust Control Center. A highlight of watchOS 26 is the automatic volume adjustment, designed to minimize the disturbances caused by notifications.

## Automatic Volume Adjustment: A Remedy for Disruptive Alerts

For numerous Apple Watch users, handling notifications can prove challenging, particularly in loud environments. Since its inception in 2015, the Apple Watch has included a Silent Mode feature, which many users, myself included, have depended on to steer clear of disruptive alerts. Nevertheless, with watchOS 26’s automatic volume adjustment feature, users may find a strong reason to rethink their notification preferences.

As stated in Apple’s press release, this feature enables the Apple Watch to automatically modify the speaker volume in accordance with the ambient noise levels surrounding the user. This implies that notifications, timers, alarms, incoming calls, and even Siri will be presented at a suitable volume, assisting users in remaining connected without inconveniencing those nearby.

### Implications for Users

For those who appreciate audible notifications, this feature could greatly enhance the experience of utilizing the Apple Watch in public or tranquil settings. It seeks to harmonize the act of staying informed while being mindful of others, making it especially valuable in environments where noise sensitivity is paramount.

However, it is crucial to recognize that this capability is not accessible to all. Automatic volume adjustment will function solely on certain models, such as the Apple Watch Series 9, Apple Watch Series 10, and Apple Watch Ultra 2. This restriction may pose challenges for users with older versions who are eager to leverage this new capability.

### Conclusion

As watchOS 26 is rolled out, users can anticipate a host of enhancements that elevate the overall performance of the Apple Watch. Although automatic volume adjustment may not sway everyone to turn off Silent Mode, it symbolizes a considerate strategy for managing notifications in a manner that honors both the user’s needs and the social environment. Whether you are a devoted Apple Watch user or contemplating an upgrade, watchOS 26 offers features that accommodate a variety of preferences and lifestyles.

Thirteen Upcoming Vehicle Brands to Incorporate Car Keys into Wallet App Shortly

### Expansion of Apple Wallet’s Car Key Feature

Apple has made considerable progress in weaving technology into daily living, and one of its recent advancements is the broadening of the car key feature in the Apple Wallet app. Originally introduced in 2020, this capability enables users to unlock and start their cars using their iPhone or Apple Watch, effectively superseding conventional car keys.

#### Functionality of Car Keys

The operation of car keys differs among manufacturers, presenting various functionalities that enhance user ease. Here are some notable features:

– **Passive Entry**: As you near your vehicle with your device, it unlocks automatically. After entering, you can start the engine, and when you move away, it locks again.
– **Proximity**: Users have the option to lock, unlock, and start their vehicle by placing their device near the door handle or key reader.
– **Remote Access**: The device can be utilized to lock and unlock the car from a distance, granting extra control over vehicle entry.

For an exhaustive list of vehicles that are currently supported, users can visit the Apple Wallet car key support page.

#### New Vehicle Brands Accepting Car Key Support

During the recent WWDC event, Apple revealed that 13 new vehicle brands will soon adopt the car key feature, broadening the reach of this technology. The brands slated to incorporate this feature include:

– Acura
– Cadillac
– Chery
– Chevrolet
– GMC
– Hongqi
– Lucid
– Porsche
– Rivian
– Smart
– Tata
– Voyah
– WEY

While Apple has not offered a specific timeline for when these brands will introduce car key support, the announcement made during the iOS 26 presentation indicates that users can anticipate these updates by year’s end. Following the integration of these brands, Apple Wallet will then support a total of 33 vehicle brands.

#### User Feedback

As the integration of car keys in Apple Wallet grows, users are invited to share their experiences and insights. This feature is designed to simplify vehicle access and improve the overall driving experience, aligning with Apple’s vision of a more digital and interconnected lifestyle.