Getting code ready for Surface Neo and Surface Duo

Microsoft’s ambition to build a folding device has been the subject of much speculation over the past few years. A series of patent filings revealed what appeared to be a dual-screen tablet that could operate in several different ways. Then there were the codenames—Andromeda and Centaurus—and what appeared to be a planned new version of Windows, based on the Windows Core OS that uses a composable shell to power both HoloLens 2 and the second-generation Surface Hub wall screens.

Time passed. The Andromeda foldable phone appeared to be canceled, though the Centaurus tablet had at least one internal showing at Microsoft. But Microsoft’s veil of secrecy kept it all hidden. Then came 2019’s Surface hardware event and the curtain was drawn back. The Centaurus dual-screen folding tablet was the Surface Neo, with a magnetic keyboard that could be used on top of one of the screens. More surprisingly, Andromeda hadn’t been canceled after all, now it’s the Surface Duo, and it isn’t a Windows device and will run Android.

It’s going to be a while before the public get its hands on the new hardware; they won’t hit the shelves until well into 2020. But that gives us time to start thinking about how we build applications for these new form factors, and how we can build code that works across both Windows and Android.

Working with Windows 10X

Microsoft has yet to give details of Windows 10X, the Surface Neo’s implementation of Windows Core OS (WCOS). However, from what we know about HoloLens 2 and Surface Hub 2 and from what was said at the launch event, we can make some educated guesses. WCOS removes much of Windows’ legacy, with a focus on Universal Windows Platform (UWP) and “modern” applications. That likely indicates that the Surface Neo won’t run your old Win32 code without some work. What’s most likely is that it will allow you to use code that’s been optimized for the Windows store, either as a full UWP app or wrapped using the Desktop Bridge using XAML Islands to host modern controls.

Controls developer Progress Telerik expects to see a focus on WinUI for Surface Neo, building on Microsoft’s current refresh of the Windows UI framework. It’s certainly a logical step for Microsoft, with both new devices shipping in the same timeframe as the unified .Net 5.0, which should simplify migrating older .Net and Win32 code to newer runtimes.

Going back to the phone

After abandoning its own mobile operating system, it’s not surprising that Microsoft has switched to Android for its next generation of mobile hardware and the Surface Duo. We’re already seeing Android-based folding screen devices, so the hooks needed to build apps that support new form factors are already in the OS. It’s a platform decision that fits with Microsoft’s own mobile application strategy, using Xamarin for cross-platform applications building on familiar .Net technologies.

With Xamarin you still need to separate your UI from your core business logic, using Xamarin’s own Forms UI components for a cross-platform look and feel. There’s still some work to be done in Xamarin to support folding and multiscreen devices on Android, but with a year to go and APIs already in place in Android 10, there’s time enough for the relevant new features to ship.

Working with folding screens in Android 10

It’s worth looking at how those Android 10 features work, with a set of new APIs that handle what Google calls App Continuity. Using these APIs, an app that’s running on a folded device using a single screen can transition to another without losing context. There’s a lot to handle here, either completely resizing an app to fill the currently available screen or offering support for multiwindow operation.

Using these APIs, you set your app’s resizeableActivity property. When false, Android operates by putting the app into compatibility mode, displaying it in its original size and aspect ratio. When set to true, it’s automatically resized to fill the new screen resolution, handling size increases when the device is unfolded and decreases when it’s folded. As the device configuration changes when its form factor changes, your code should handle the change by saving its current UI state and reapplying it to the new device state when the change is complete.

Folding screens make a big change to app UIs; you now have to test them in a range of different display aspect ratios, for folded screens that are long and narrow, as well as for screens that match Microsoft’s preferred 3:2 aspect ratio, used across all its Surface devices. Android gives you the option to set maxAspectRatio and minAspectRatio for your code.

There’s a lot in Google’s foldable device Android specifications, and it’s going to be up to platforms like Xamarin to add the APIs to .Net. Until they’re fully supported, you can continue to use your current application user experiences, as they’ll at least run in compatibility mode or as multiwindowed apps on each of the Surface Duo’s separate screens. One useful option is the ability to use one of Surface Duo’s screens as a keyboard, with the other screen displaying a landscape view of an app.

Folding screens can be treated as multiple displays, another Android 10 feature. This approach should work well with the Surface Duo’s option of folding its displays back in a more phone-like configuration, where apps can be displayed on both front and back of the device. Android’s multiple display options allow use of external monitors, so there’s even the option of Surface Duo offering something like Windows 10 Mobile’s Continuum, where mobile apps could switch to a desktop version on larger screens.

Evolving the Web for folding screens

One of the more interesting ways that Microsoft can support cross-platform multiscreen development is using Web technologies, with PWAs that take advantage of the new Surfaces. A position paper on GitHub from the Edge development team describes a Windows Segments Enumeration API which allows browsers to work with windows that cross multiple displays and that support different folding models. Using this model, a browser can work with Window Segments across multiple displays or display areas.

With Window Segments measured in CSS pixels, a JavaScript API allows Web applications to build a map of the available segments, and to determine which are currently visible. Handling display occlusion, for example with a pop-up keyboard, is an important aspect of this approach, as it ensures your code isn’t updating unavailable screen real estate. It ensures that your application can scale screen elements appropriately, managing text boxes and forms so they don’t get hidden by an active keyboard.

Using this technique, you could treat Window Segments separately, for example showing a map on one Duo screen and a related list of places on the other. You can think of it as offering your apps the equivalent of a master/detail view in a single browser window. Building PWAs around this model gives you a flexible way to deliver content that would work well on a single display device, while giving dual-screen hardware access to more functionality and a different user experience from the same Web application.

With a JavaScript API to handle Window Segments there’s the possibility to use it with WebAssembly applications, built using Blazor or with Uno’s implementation of UWP controls. Using it in conjunction with other proposed browser APIs for working with folding devices should make it a lot easier to deliver cross-platform experiences using modern browser-based apps.

Talking to the Uno Platform team, it’s clear they’re confident that the combination of Uno and Xamarin, along with new browser technologies, should make it relatively easy to bring existing .Net applications that use WinUI to both new Surface devices. Whether they’re running on a .Net runtime or as WebAssembly code in PWAs won’t matter, if users get the same experience as they would running the app on a Windows PC.