We're shipping Emulsio 5 this week. You can find out more on its webpage, and on the App Store. The main new features are AI-powered slo-mo, upscaling, and an improved multi-frame stabilization technique.

Emulsio, née Movie Stiller, has been here for quite a long time. We first shipped it in 2011, fourteen years ago, when video capabilities of the iPhone were simple, sensors & compute limited, and video stabilization very much needed. Nowadays, video has become an essential part of communication between people, for media, marketing, you name it. We are bringing new innovations in Emulsio in this 5th major iteration that you can review below.

AI-powered slow motion

As its name suggests, this lets you insert slow motion short sequences into existing videos that are generated by on-device AI. But why do we need that in the first place? Nowadays, cameras used to create content often come from small devices like iPhone, iPad, vlogging camera, action cam or drone. They share a common limitation that is their limited-size sensor. This makes it imperfect in poorly lit situations, and high-speed video recording makes the problem even worse as a short exposure duration for a frame means less light, and thus more noise. Moreover, slo-mo by themselves are a great creative tool, but they require that you know in advance a high-speed action will take place, which you can't always predict. Finally, shooting everything in high-speed takes enormous space on your device, and possibly decreases image quality, as discussed above. For all these reasons, it absolutely makes sense to have access to a post-processing tool for adding slo-mos.

Generated slo-mos have progressed a lot lately with the arrival of AI. It is a form of frame interpolation that uses both motion and visual features from the video being processed and learning of a synthesis technique making use of these. Generation artefacts have been much reduced when compared to previous non-AI techniques (but can still occur). Although cloud-based operation could theoretically bring additional processing power, we wanted to bring these AI models on the device as we think this provides improved privacy and removes dependency on online resources. The challenge was to fit that into limited iPhone RAM / computing caps to still bring useful creative power to users.

A video processing platform

In its early days, the challenge for Emulsio was to enable processing and bring value to creators on fairly limited devices, in terms of compute and sensor quality. So it mostly was a technical achievement in an emerging field. Nowadays, iPhone, iPad, and Mac are extremely capable video recording and editing devices, and have been recently extended to handle HDR encoding and playback as well on EDR capable displays. Moreover, they've been receiving specific AI capabilities with Apple Silicon and Core ML, which enable new use cases for running neural networks locally for a variety of domains. 

Noticing all these enhancements, we opted to grow Emulsio into a platform for processing videos. Not standard editing that has been there for ages and for which many great apps exist, but more towards the video processing & enhancement aspect of it. To achieve this, we have transformed Emulsio's toolchain into a video processing pipeline that allows GPU accelerated frame-by-frame processing including previous/future full quality frame contexts that enable not only standard CPU and GPU image processing, but also AI image processing with custom hi-performance models. Slow motion, FPS increase, multi-frame stabilization, and upscaling are concrete examples of this, and this sets the ground for more future capabilities. Emulsio's new UI represents that well too, stacking the currently chosen capabilities with room to grow.

This new processing pipeline works identically on iPhone/iPad and Mac, even though we noticed not all devices are equal. Indeed, AI video processing requires a lot RAM, and we can deliver the best performance and resolution on devices with at least 6GB of RAM. This may be related to why Apple Intelligence is also limited to the most recent devices: it is not that much about compute power than it is linked to the availability of sufficient memory for storing intermediate inference data.

The challenges of iOS 26 / macOS 26

Latest Apple OSes bring important changes in terms of user interface. Emulsio has adopted a floating interface for a long time, as it always was our intention to put content first. As such, the Liquid Glass proposal by Apple is a great candidate for adoption in the short term (we're on it), and it will totally make sense in the way the app operates. Emulsio feels at home with Liquid Glass. More on this very soon.

But there's more to these new OSes, as they also bring new video editing capabilities for developers. We are currently working to integrate Apple's new video processing APIs directly in Emulsio, and so far what we've seen is that their slo-mo generation offers a faster frame generation process when compared to the one we currently provide in Emulsio (custom Core ML models). Image quality under fast motion is yet to be assessed for each technique though but our plan is to offer all possible options to Emulsio users for synthesizing slo-mos, and documenting pros/cons for each. More on this soon.

Development choices

As said earlier, Emulsio has been there for quite a long time, and provides a lot of legacy code. We nonetheless wanted to modernize its user interface with SwiftUI both to offer a refreshed and future-proof experience and to target all devices (iPhone, iPad, Mac) with a common code base. On the iOS side, it is a standard app and on the Mac side, it is a Mac Catalyst app. Older code (still needed) has been bridged to be easily usable from Swift/SwiftUI, and specific UI added for the Mac, which has "windows" to account for.

Beyond UI elements, video processing is pleasantly unified between Mac Catalyst and iOS, up to the display of HDR content to EDR screens, that Emulsio uses. This let us focus on other aspects of the app, as compatibility was mostly a non-issue. Not sure if that would have applied in the case of a standard Mac app though (non Catalyst), as the Mac has a long history of video processing that predates iOS with its own conventions.

Business model

About pricing of the app and the free app limitations, it is similar to what we generally offer. Emulsio can be fully tried for free with a watermark inserted in the generated outputs. This lets you fully experience the capabilities of the app and help you decide whether it provides value to your creative toolbox.

Purchase of the app can be made through either a one-time payment or a subscription, and the difference between these options is how future upgrades are handled. In the case of the subscription, all future upgrades are included, while they'll require an upgrade at some point for the one-off purchase. This gives you freedom whether or when you want to upgrade, and even let you review the new features before you commit. Depending on the next features we'll offer and the development cycle they require, it is well likely that a number of those will be included in both options initially but availability will diverge at a later stage.

It’s that exciting time of year again – new OS releases are here! We’ve been hard at work throughout the summer, fine-tuning our apps to make the most of the latest OS features, and today, we’re rolling them out! Here’s a quick look at what’s new. All of our apps have been rebuilt using the latest SDK and tools, fixed up, modernized where needed, and some even got shiny new features for a seamless experience with the new OS. Check out the list of updated apps below:

• Hydra: AI Camera (App Store, Website)
• Inko: Collaborative Whiteboard (App Store, Website)
• Emulsio: Video Stabilizer (App Store, Website)
• Prizmo: Document Scanner (App Store, Website)
• Prizmo Go: Text Grabber (App Store, Website)
• Carbo: Digital Notebook (App Store, Website)
• Golf Quartz: Golf for Apple Watch (App Store, Website)

All of our apps are getting the new icon support for iPhone/iPad homescreen, that is, custom standard, tinted, and dark icon variants that you can choose from.

Hydra

Beyond new icons, we optimized Hydra for working as a connected camera to the computer, taking advantage of screen mirroring to the Mac. Studio photography, with a controlled environment / lighting is popular with DSLR too. This allows to quickly try different setups / scene arrangements, photo settings, and lighting without relying on the variability of holding the camera by hand. Many Hydra options such as lens and mode switching are exposed, and the last picture can be easily accessed by drag and drop. You can also setup a shared folder for shooting, but that is a pre-existing feature in Hydra. This will make Hydra a great companion to studio photographers who want to use their iPhone capabilities to their max in the professional environment.

Hydra 2.4 also adds a number of buttons for the Lock Screen and for the Control Center, as well as for the Action Button of the latest iPhone. Photographic modes are exposed for easily accessing them from anywhere on iOS, as well as a button for just launching the app using the currently selected mode. Homescreen widget variants also gets an update to support the new look of iOS modes (tinted / dark).

We're also bringing Hydra into Siri and Apple Intelligence with the camera domain to allow better OS integration for shooting and changing mode. This might not be available just yet throughout iOS, so stay tuned for more.

Other improvements include snappier photo processing thanks to improved Core ML performance in iOS 18. (no timing available yet)

Inko / Emulsio

Inko 2.4 brings new icons optimized for iOS 18 styles. In addition to that, it also brings Siri and Apple Intelligence support by implementing a number of actions from the whiteboard domain. The goal is to make it easier to interact Inko to create new drawing boards and share with users. Finally, Inko widgets have been modernized to take advantage of the dark / tinted colors of iOS 18 alongside the new icons.

Emulsio 4.1 gets the new look & feel for iOS springboard with an updated set of icons for standard / tinted / dark modes.

Prizmo / Prizmo Go / Carbo

For Prizmo 5.9 and Prizmo Go 4.3, respectively our document scanning app and text extraction app, this seasonal update essentially brings new action buttons for the Lock Screen / Control Center and Action Button, to allow quick launch from anywhere on iOS. Button actions have been tuned to be easily recognizable and match default iOS style. Both Prizmo's also include updated OCR capabilities of iOS 18 for Arabic, with high-speed and high-accuracy on-device OCR. Updated application icons are also offered for the standard, tinted, and dark looks.

Carbo, our digital notebook for handwritten (Apple Pencil or photo based) offers updated icons as well.

Golf Quartz

Golf Quartz 1.1 brings a number of new things. On iOS, standard / dark / tinted icons are provided like for the others apps. On watchOS 11, we've tuned the behaviour of the Live Activity: because Golf Quartz exposes the current game (or practice session) as a Live Activity on the iPhone when you play, it also brings a matching interface on watchOS 11 as a new behaviour. But just like for the Apple workout app, because it stays front-most during the activity, we've disabled that overlay to not disturb the standard counting or distance golf interact on the watch.

Golf Quartz on watchOS also gets added support for Apple Watch Ultra action button. Players can now use the action button to initiate a new game or a new practice session, and also switch between to hazards target views from the stroke counting view while the game is active. This allows easy access to bunker distance with just a press. We've also update the swipe motion to display the green and hazard with the green above the counters, and hazard below them. This allows single swipe in all cases (instead of 2 for bunkers previously), which makes it quicker to change screen. Also, pausing and resuming the game is easily achieved on Apple Watch Ultra by simultaneously pressing the Action Button (left) and Home Button (right).

Finally, Golf Quartz complications on watchOS now let you decide to start the app either in Game or Practice mode.

A brand-new golfing app has just launched!

In a unique venture, we’re introducing a fresh sports app called Golf Quartz. My own golf journey began about six years ago when my father brought me along for a discovery session. The sport is challenging—it still challenges me—but I was instantly hooked by its rich blend of outdoor activity, the pursuit of perfection, ongoing humble learning, and the chance for social and generational exchange. It’s become clear to me that golf is more than just technique; it’s increasingly a lifestyle. But let’s pivot back to our app.

What distinguishes our app this time are several key features:

• It is built from the ground up for the Apple Watch, leveraging sensors, AI, and more. It’s not merely an adaptation from an iPhone app.
• It's actually not one app but three apps: Apple Watch, iPhone, and Mac + the cloud.
• Course mapping is brought back to those who know the course best, clubs and players. And trying to do in a decentralized way to benefit everyone.
• Open data is central: data formats are open for course maps and game data with free access and import/export.

Golf Quartz is available on the App Store, and more information can be found about it on our website.

The concept

It all started from my difficulty as a beginner to keep my own scores & those of other players. It can be a real challenge when you're under the game pressure, and not too familiar with counting, even after a couple years. From that starting point (score keeping), we evolved the concept to provide valuable information during the game like flag or hazard distance and location of the next tee for unknown courses, together with AI to let the watch know when (& from where) you're swinging. Finally, it became obvious we could do more by analyzing the swing motion when practicing & provide valuable feedback like amplitude, maximum speed, etc.

Many of my friends have an Apple Watch, but they typically switch to a specialized watch when playing golf. These can be good at golf, but I felt like we could do something to make a better use of their Apple devices which are actually packed with the latest GPS, processing power / AI, a refined user experience and a large choice of apps for everything, and pretty unique high-speed sensor technologies.

From a beginner-oriented initial idea (score keeping), we evolved it during development into something that could suit golfers of any level (customized course information, swing analysis). That came down to our skills at app making: offer a progressive complexity and a diversity of tasks for the variety of needs, yet never interfere with the game, keeping the distractions out and the mind free & focused on the game itself, which is what really matters for golfers.

A "watch-first" experience

Apple Watch is packed with sensors and has gained increased processing power years after years. watchOS 10, required by Golf Quartz, is now bringing the kind of capabilities that enable the complete experience.

Very early in the concept of this app, we quickly identified that the watch was the best place to keep the scores on the course because it is always with you. Indeed, if you happen to have your iPhone on the trolley and move it to the next tee before finishing with the green, it's not with you anymore, and you can't count the last strokes as they occur. The always-with-you thing is an absolute asset, but there's more to it.

In the early years of the Apple Watch, processing power and sensors were quite limited, as were the development capabilities to offer full-fledged user experiences. But as the time advanced, real APIs and highly accurate sensors appeared and created new possibilities. Health related tracking, AI swing detection & analysis, precise GPS-based positioning & distance determination have all become a reality.

Companion apps on iPhone and the Mac are offered respectively for reviewing past games/practices and for mapping courses. We tried to use the form factor that makes the most sense for each task, that is, reviewing past games make more sense on the iPhone than on the watch, and tagging courses with a precise mouse and a large display makes more sense than doing it on a phone.

A caddie on the course

As previously mentioned, score keeping was the first thing we were trying to solve. It had to work with multiple players, as this is how competitions make you do it: you have to count your own strokes and the ones of another player. We pushed it further to allow up to four players, although this is more for caddies as it requires many interactions while playing, but it's there.

Distances to greens, tee targets and hazards were all implemented on top of GPS positioning. We found it to be pretty accurate, especially on Apple Watch model feature double-frequency GPS. We complemented it with magnetometer data to provide direction in addition to distance, in the form of a compass-like arrow. Next tee, green, and tee target use that feature, and we find it particularly useful when you discover a new course: "Where should I aim? Where's the next tee? Etc.". It's also a pretty unique feature, as this requires not only the sensor but a live display for it to work that not many golf watches offer.

A pro at the range

Not telling that the app will replace a pro, of course not. But it can definitely complement it. A golf swing is a complex whole body motion, and it needs to be analyzed thoroughly. Yet, when working on a few detail points like the duration of transition between the backswing and the downswing, or the maximum rotational speed, or backswing angular amplitude, measurements can be made and provide valuable feedback. And this is a pretty unique feature that Golf Quartz provides. Unlike radar sensors that provide club head and ball related speeds and trajectories, Golf Quartz does not focus on the results but more on the motion itself that produces such results. Bound to the wrist, it can precisely determine rotational pace and related metrics to estimate motion information that can help training specific aspects. Based on high-speed sensors of the latest Apple Watch models, this is a pretty unique feature that most other watch brands can't replicate.

Golf Quartz will mirror these computed metrics both to the iPhone display when training (attached to the bag for instance) or use text-to-speech to announce the latest swing focused value. If your transition is too quick, select the transition value (milliseconds), and let the watch speak it to your AirPods after each swing as you try to increase its value by slowing down between backswing and downswing. Or validate that 1 month at the gym with specific exercises will indeed augment your maximal rotation speed in the downswing. Good practice is all about gathering useful feedback, and Golf Quartz will happily provide it for you.

Artificial intelligence

The watch app does make use of artificial intelligence. While playing or practicing, it continuously detects swings and putts to refine data of other subsystems. We initially thought that it wouldn't be possible to have an LSTM processing sensor data continuously for 5 hours+, but found out it actually was the case (good job CoreML team). Selection of sensors, recording and preparation of training data and handling of various watch configurations (left/right handed, watch orientation, etc.) were performed to train the model.

We expect to do more with AI. For instance, fully automatic counting based on AI is not yet possible (was too noisy), but we still use AI to filter out false scoring alerts in single-player score keeping (because in multiplayer situations, the watch doesn't know when others are swinging).

Collaborative course database

Even though multiplayer scorekeeping was the initial idea for the app, we quickly realized that providing distances on the course was too fundamental a feature for us to ignore. We explored existing databases, and determined that they were both expensive (cost that we'd have to pass to our users) and inaccurate in that they don't cope well with course changes, temporary like weather/season based, or definitive like mods or relabelling of holes. More fundamentally, it just makes sense to have a decentralized mapping task in the hands of those who really know the course. These were the starting thoughts about a mapping app.

This app was initially conceived as an app that players could use to map their own course, and share files together at the club. But sending files and managing versions could quickly become a friction. So we opted pretty late in the development cycle to go with a full-fledged cloud platform that not only players but also clubs could use to automatically distribute their maps to everyone. With all the advantages described on the site: same-day delivery of changes, get back control of the data, have happy members throughout seasons and course changes who will be able to use their watch at all time.

It was also an excuse for us to try new technologies like Swift on the server, which enabled us to quickly iterate by sharing code with the apps.

Developers' journey

We started out with the project under watchOS 9. It was doing pretty great, until watchOS 10 was announced last year at WWDC 2023, which was a major update for Apple Watch. We had to reconsider how to navigate within the app to provide an experience that is better aligned with the newest paradigms presented then. This was quite a challenge. Another difficulty that arose with the betas in the summer and fall, was that location access was mostly broken for watch/iOS companion apps, which in turn slowed us down massively as we couldn't test development builds for months, always trying to find workarounds. It finally returned to normal a couple months after the official OS launch, and we could finally iterate more effectively.

Similarly for the iPhone app and the mapping app on Mac, we decided to go with SwiftUI, which recently integrated a refined API for Apple Maps. Working with SwiftUI forces you to make radical design choices (as not everything can be achieved), but under those constraints, development can become quicker. Like for watchOS, we experienced a number of issues in the most recent OS APIs, but finally converged.

A note about open data & evolutions

It is important to me as a golfer (and even more generally as a user) to not be captive of some ecosystem, that is, if I stop paying all my data is gone. I actually think this is a hostile business model of the past, and strongly believe instead that offering a user experience around more open data is what makes sense in the end. That is why we opted right from the start for open formats (that others can use too) both for game data and for course mapping.

We also hope that bridges between systems, like having the watch directly send scores to an automated competition dashboard at the clubhouse, smart weather alerts, or a club-operated virtual marshal (for alerts, not electrical discharges) will become a reality at some point.

And above all, just keeping it simple and sound for a relaxing game. Thank you for reading.

Major update of our flagship camera app.

After multiple years in the making, we are releasing today Hydra 2, our next generation camera app for iPhone & iPad. You'll find here some additional background information about the making of this project but also about the wider perspective we're going after. Have a seat, a cup of coffee, and here we go.

Photography, reinvented.

Let's go to the point. Today we are announcing 3 products:

1. a unique camera experience as a fully-featured photo app with both automatic and manual control including normal & AI-based shooting modes.

2. a high-fidelity photo editor spanning from precise colorimetric rendering to boosted high-contrast & personalized photos to suit your liking or particular intent of the moment.

3. an evolutive AI framework for photographic processing encompassing platforms & brands, device aging and latest camera hardware innovations.

A camera, an editor, an AI photo framework. A camera, an editor, an AI photo framework. Are you getting it? 😇

Hydra brings a distinct and exciting new route for photographic innovations on iPhone & iPad (and more), with continuous and inexpensive (or even free) upgrades, with unique new features years after years, and best of all, with new software capabilities that add up with the latest camera hardware enhancements.

About portability, we think it would be awesome to have a reliable photographic software stack that is available throughout the diversity of devices you use, like DSLRs and other phone or tablet brands. Oftentimes, iPhone are said to make better photos than DSLRs and that's true when lighting conditions are good, and the reason is that the post-processing software (denoising, tone mapping…) is much more advanced than with regular cameras. Wouldn't it be great to edit ProRAW-style shooting out of your Canon or Nikon camera? We'd sure like over time to bring this type of advanced processing to all kinds of cameras, and take simultaneous advantage of larger sensors *and* photo enhancement.

That's the path we are proposing you to follow with us, and this new version of Hydra is the first step towards that broader goal of an independent software stack that goes from the sensor to the final picture.

Origins of the name "Hydra"

At the start, "Hydra" was just a pun about the HDR letters (HDR stands for high dynamic range, a photographic technique to capture both dark & bright areas of a scene), which Hydra was all about back in the 2000's on Mac: fusing multiple exposures shot from a DSLR camera. Then, with its arrival on iOS in January 2015, the name made even more sense with the photo fusion (hydra = many heads) technique required to grab more photos from small sensors that it pioneered back in the days and now the multi-faceted techniques it uses to push quality forward. Today, Hydra is taking a new turn as its mythological fighting character is here more than ever, although we hope it will remain gentle & useful to users.

Refined camera experience

When going through the design process of Hydra 2, we knew from the start we would expose more camera control than with the previous version. Indeed, manual control was a highly requested feature. Yet, we still wanted the app to be easy to use, approachable, that anyone could use without asking "what this button does", as it is often the case when you go with fancy ideas: they might look very good & cool and even be sound after you learn them, but we really wanted to remain true to our intuition of both keeping it simple and not augmenting the cognitive load for the sake of design ego.

We kept Hydra's signature: the big orange button that makes it obvious that this is the main action & is easy to hit. We created a dual interface (auto and manual shooting modes), just like a regular camera device. The automatic mode takes care of most things for you, but you still get some useful control like exposure control, while the manual mode will let you interact with all camera settings like ISO, Focus, white balance, etc. This subdivision made a lot of sense, as refined control can be exceptional depending on conditions, or in terms of user preference (pro might spend more time in there). With this dual mode, we felt very confident that everything was in its logical place even for first time users. We rationalized camera options by cleaning up the top bar, removing most of the visual burden to only keep A/M (auto/manual), exposure bias, and possibly flash. We moved other options in a specific panel, which also gives more room for description when needed.

Another goal for this design process was to make it not just look but also feel like a real camera. We had to build something more tactile than what we did previously, we had to feel light and be at home when shooting, turning an app into a trusty shooting hardware. The first step to achieve that was using the whole viewfinder surface as a trackpad for camera control. We call it the camera pad internally. Swiping up or down from anywhere in the preview will adjust exposure. In manual more, swiping horizontally changes the current setting (ISO, Focus…) and the vertical axis is used for secondary settings. We added vibration feedback for every stop (doubling/halving of gathered input light) just like real DSLR cameras have clicking crowns for those, and made sure that real-life tactile distance would match for all settings: ISO, Speed, EV Bias all have the same physical swipe distance & vibration for stops. 12 (visual) subdivisions are used to enable both third and half stops. Coherence of the various behaviors brings confidence when using the app.

The built-in camera shooting mode was also added to Hydra. At stake, and we discussed that a lot internally at the start, was whether it would make sense to try to bring the standard iPhone shooting capabilities into the app or just keep Hydra for niche situations. Because Hydra 2 would have manual camera control (that the default camera does not expose), and also because we wanted to somehow remove this modal thinking from the user (which app should I use), we went with integration of the built-in camera as the "normal" mode. It later made even more sense when ProRAW was introduced, as we could integrate that into our built-in editor combining the various imaging operators. It is also a path for future evolution, as camera hardware has so many intricate options that we could reveal for that particular mode.

iPad was also in our viewfinder (or is it the other way around?). Camera hardware is also evolving on iPad although possibly a bit behind iPhone, and we see people use it to shoot photos and videos. You shoot with what you currently have with you, that's how photography works, isn't it? Moreover, iPad is the most powerful device of the mobile line and certainly is a workhorse in terms of machine learning. We focused mainly on having a great shooting experience, moving buttons according to how it is held while shooting, and better using the large display. We know there is still some work for iPad (improved support for the left-handed, better photo editor layout), but we are nonetheless very happy with the result for the initial release of the new design and to offer a full blown version just for iPad users.

Technically, we moved to SwiftUI for many parts of the app, with some legacy parts remaining in UIKit. The viewfinder is mostly SwiftUI, as are the top bar, the camera settings, manual controls, and most panels displayed in the app. We can't say the transition was easy, especially with our constraint of supporting both iOS 14 and iOS 15, as lots of things break or require specific workarounds. We also had unexpected CPU consumption linked to the way information flows into the code, and we needed to experiment and learn how to handle these situations using Combine (it was interesting in the end). SwiftUI also brought a number of advantages like easy interruptible animations, absence of all "unsynced" UI bugs, and faster iteration when things are in place. We see this as an investment for the future to learn it and use it now, but it can be challenging at times when you want to design specific appearances on the edge of its current capabilities.

Hydra 2 shooting modes

From its initial release years ago, Hydra's goal was to enable shooting in unusual or difficult conditions. We've pushed that concept further with the help of AI in this new release, but we also added the "Normal" mode for shooting with integrated photo capabilities of the iPhone. This makes it convenient for both use cases (standard and unusual conditions) within a single app.

About the AI-powered modes, they currently cover 3 main categories: HDR (for High dynamic range, scenes that consist of simultaneous very bright and dark areas), denoising for poorly lit scenes (Lo-light), and resolution augmentation (Zoom, Macro, Hi-res). All modes perform integrated fusion, AI-enhancement, demosaicing of multiple RAW input photos.

HDR mode together with its tone mapper that compresses the photo back to display dynamics is best used for scenes with typically bright light and important contrast, like architecture, sceneries, indoor-outdoor combos. The new tone mapper of Hydra 2 improves or even suppresses halo banding versus previous versions, and it results from subtle contrast enhancement to a pretty extreme look. Other improvements versus the previous version are the much shorter shooting times (0.5s vs 10s) leading to fewer artifacts. HDR is a very creative mode, especially with the exposed tone mapper settings of the built-in editor.

Lo-light is as its name suggests a mode to enhance photo quality under dim lighting. This is clearly the Achilles heel of small sensors / mobile phones. Denoising while preserving details in photos is a complex thing, and AI brings some additional capabilities to the rescue. Quality has improved a lot when compared to previous version of Hydra, and also motion-related artifacts are better handled, especially when you consider that a single shot can span over 1 or 2 seconds.

Resolution augmentation modes like Hi-res, Zoom, and Macro share some common foundation and try to recreate higher resolutions both from motion & multiple samples and from prior knowledge. It works best for well lit scenes (as otherwise, noise gets in the way), and will typically enable substantial improvements when compared to traditional digital zooming. It won't replace a real sensor, but it's still very valuable. Contrasted edges and textures typically are well handled in these modes.

All these modes are driven with neural networks, and their results can be tuned with the "Neural style" and "Neural computation" options. The former lets you set the desired appearance and will tune network parameters accordingly including training style (adversarial or not), while the latter lets you choose the architecture that is used, simple or complex, to respectively favor speed or quality.

Final note on modes: this is the current proposal for specialty shooting in Hydra. We are experiencing other shooting modes and have a number of ideas for widening applications of the app, so let's discuss that again later.

A note on photo quality & custom look

Often on social networks, we see people discussing photo rendering comparing the latest iPhone 13 Pro vs the Samsung 13 Pro (or whatever). Let's be honest for a second here, camera are complex hardware & now software with so many parameters that can be tuned.

"Contrast is better on the iPhone"

"Colors are brighter on the Samsung"

"Sharpness is better on the Google"

Showing 2 photos and asking users can go like this: you see the labels under the photos (Apple, Samsung) and some would say I like Apple better. Swapping the labels, the same people would still say Apple (which tells more how much he likes the brand independently of photo quality). Or the other way around. Or, with no labels, you could prefer the first one, but when showing the actual scene conditions, you switch your judgment to the other, finding the former unnatural. It's all very subjective. Yet at the same time this does not mean there are no facts in there.

There is the inherent quality of the device / software stack, for sure. But there are also the default settings chosen by the manufacturer. If one photo is faithful to reality, and the other one is contrast and color boosted, many (as in most) will declare the latter much better than the former. This makes it that the industry is attracted towards boosted photos.

We have done testing about the defaults applied in iPhone 13 Pro, and we have noticed a pattern where both the tone mapper and contrast (boost) have been increased over the years when compared to older iPhone, comparing processed photos to RAW ones which are not that different tone-wise. This produces a nice image, but it is sometimes beyond reality depending on lighting conditions. This is certainly a trend followed by the entire industry: if one maker increases contrast, the other ones have to do it too to some extent otherwise their photos will look blemish. Same thing for TVs. Hydra's defaults try to be balanced, and they can be tested (editor) and changed if needed.

It really is a matter of taste, intent, and purpose among other things. This is why customizable defaults and editing capabilities are important, as taste and intent are not absolute. They vary from person to person, from moment to moment, for scene to scene, etc. You get it.

An evolutive AI framework for photography

One of our goals was to explore photography outside of the predefined photo sandbox offered by the Apple APIs. Apple's stack is remarkable in many ways, offering great photo quality and camera control, although all photo apps using them will end up with similar photos & the same pros and cons as Apple's built-in camera app.

Instead, what we were after here was going directly from raw sensor data and offering the whole processing chain. Think of an operating system for photography that brings it all: demosaicing, denoising, fusion, enhancement, post-processing, encoding. Take control of every step needed to create the final image, in order to give back user customization, algorithm experimentation, or in short, a diversity of processing capabilities.

Along the way of implementing this framework, we noticed that AI could serve to remove the usual artifacts of photos like quantization and limited dynamic range, which in turn, could be used to offer a more accurate 32-bit editing. This allows to re-expose photos, adjusting brightness and contrast, etc. to unusually broad ranges of values that would have destroyed images under standard editing conditions. Expressivity & creativity are increased with 32-bit instant editing that works directly on the hifi image before storage.

Note that it currently means editing must be done immediately. With HDR, we already enabled 32-bit lightmap exports to allow further editing after shooting, or even applying future tone mapper with better capabilities than the current ones. We'll also be exploring ways of delaying the processing for non-HDR photos at some point, as we understand immediate editing is not always practical.

About file formats, we went with the usual ones of course (JPEG, HEIC, DNG) and extended it with Apple ProRAW on compatible devices. As just said, HDR now allows 32-bit lightmap exports too in specific formats (OpenEXR, Radiance) for later/alternate tone mapping. We also added the experimental JPEG XL format, which exposes very interesting capabilities like 32-bit floats, masks, and a great compression, possibly beating HEIC and ProRAW. We wanted to onboard this early to support it because it's great, and also in case it emerges as a valuable tool for photographers.

By cross-platform, do you mean Android?

Yes we do. But not only. Think of all digital cameras. From DSLR to Raspberry-PI, through Android phones. Haven't you thought sometimes that the iPhone takes better photos than your DSLR? With good lighting, this is true indeed because of HDR and other post-processing. We'd like to bring that tooling more broadly. Bring back some new options like you have for music or series for instance.

Android is very popular, in particular here in Europe but also in many other places. And it's motivating to reach a great audience when making products. Android phones have a great diversity of camera capabilities which makes Hydra even more interesting there, and on high-end devices, the camera API is very interesting and enables rich use cases as it give low-level on the camera itself.

If you are interested in a version of Hydra for a particular hardware (camera, phone, other), feel free to reach out, as this helps us sense the need for it.

Artificial Intelligence: the why's.

The question that first comes to mind when talking about artificial intelligence is what is made up versus what is real. Examples of fake imagery are everywhere. Where previously, with good old JPEGs shot from digital camera, "everything was real".

As usual, this is a little more complicated. Traditionally, when shooting with a digital camera, sensors get a partial view of the scene. That is, after light goes through the optics machinery, it is sampled at regular spatial intervals named photo sites. Each photo site location corresponds to 1 output pixel, yet they each receive only one of the required colors (reg, green, or blue). In a typical bucket of 4, each photo site samples a single color (say R, G, G, B) through the color filter array (CFA), while the output pixels need to have all colors at each location (RGB, RGB, RGB, RGB). This means that 2/3 of the information is missing from sampled data, as a 12MP output photo has 12M photo sites but 36M color samples. The photo site samples are also corrupted by noise (photon/electron quantization, thermal sensor noise, optical point-spread function according to color wavelength, etc.), which makes the process of filling missing values even more challenging.

The missing signal is typically addressed by a process called demosaicing (or debayering) that fills the holes using either prior knowledge and/or correlation between color channels. Note that it can fail (differ from reality) if the missing values are unexpected from the used hypotheses. A number of demosaicing techniques exist, and this is often combined with denoising as you don't want to interpolate (too much) noise for missing values.

Related to this problem is super resolution (pixel dimension increase), where we want to estimate even more missing values between the (low-pass) sampled ones, which is even harder as we need to derive even more data (16 RGB pixels from a 4-photo-site RGGB quadruplet at 2x upscaling).

Traditional demosaicing and super resolution have gone at length over the years to provide good solutions to these problems, but not without flaws. Traditional image processing techniques need to make hypotheses that when met provide excellent results and when not met can generate artifacts and ruin images. The modeling of these hypotheses can be complex (yet incomplete), is hard to translate to memory/computationally efficient algorithms written by hand, hard to fine tune or improve without redesigning from scratch. Yet, these provide control, as we can precisely hand tune behaviors through thresholds for instance.

This is where artificial intelligence comes in. Filling up missing information from surrounding one and prior knowledge is a pretty natural task here. Assuming you can teach it through enough representative examples, AI techniques can be thought of as picking the best estimate from what it sees and what it knows. It is a bit like determining a tons of hypotheses (without naming them) and finding the optimum.

Artificial Intelligence in Hydra 2

2015's Hydra 1.0 had its challenges. It did not make use of AI, and its main purposes were image-fusion-based denoising, HDR tone mapping, and (classical) super resolution of 32MP from 8MP sensors. All this on 32-bit devices, using limited-accuracy OpenGL for GPU computing, and limited computing power and memory. The techniques were as described above, and I often discussed the super resolution one as thinking about the phone as a brush with straight hair that would sample adjacent color (1cm) let's say 10 meters away on photo 1, then again with some offset (think 0.32cm) on photo 2, etc. then combine all that info through subpixel motion estimation into a high-res photo.

We could have ported this technique to 64-bit and Metal to modernize it, but the room for improvement in traditional image processing was at best just marginal from existing implementation, yet highly demanding for development effort. This is why we started to explore AI possibilities about 4 years ago. The goal was mainly two-fold: 1. Increase the quality & capabilities when compared to traditional techniques, 2. Create an AI framework (training + runtime) that would allow future exploration of newer techniques.

So we started from scratch: on the training side, we had to learn Tensorflow 2, to define a physical model for photography, to figure out inputs / outputs for AI-based techniques, determine possible neural architectures and training data & schemes. See by ourselves what works and what doesn't, both through quantitative measures and visual quality assessment after training. It was not unlike terraforming a new planet for us (well, I guess). Many tools exist with their pro's and con's, but we as Apple developer maniacs had to come up with the tooling that would allow sufficient productivity to achieve our goals. Takes important research time & experimentation, scripting, remote commanding Linux computing boxes from a Mac, etc., but ends up being a valuable investment afterwards when things get rolling & quick iterations are needed. Isn't machine learning the future of all things, after all?

On the runtime side (iPhone), we had to carefully implement neural network execution on top of Core ML & Metal. We wanted to support all iPhones from 6s to the latest (13 Pro at the time of writing) which can be challenging (think of it, a 110MP floating-point image takes up 1.8GB of memory that most iPhone just don't have). We split the neural architectures into 2 versions, as older phones don't have the Apple Neural Engine (ANE) in their silicon (just standard GPU) and can't execute the more complex ones in a reasonable time. Also, because we are using specific tools in our network architectures, and because it's a fairly new domain as well, we had to write our custom Tensorflow > Core ML converter.

To be honest, with our limited resources, through the years, we just skimmed through the surface to realize a reasonable outcome, all the time focusing on shipping. There's so much yet to be done. It is far from perfect and we know very well where we should inject more work into it. Yet we're very proud of this first step as it opens so many doors. Feels very much like we can go anywhere from here. Quoting Robert A. Heinlein, “Once you get to Earth orbit, you’re halfway to anywhere in the solar system”.

What's Next?

It's been a long project for us during which we had a number of team achievements. We've been transitioning many things over the last few years: Obj-C to Swift, UIKit to SwiftUI, OpenGL to Metal, classical image processing to machine learning, office to home working, Apple-only mind to cross-platform plans. The SwiftUI work in particular was not easy because it is new technology, iOS 14/15 workarounds were needed, as well as bridges for existing Obj-C foundations. Tensorflow 2 and Core ML also needed a lot of investment to master.

We sure need to stabilize the app & fine-tune the photo rendering techniques & image quality to get robust foundations. There are a number of things we'd like to explore like a number of additional new modes, also, the capability of unifying high dynamic range & standard photo fusion together as a single scheme where they currently operate separately. As mentioned previously, an Android version too will be developed. Not forgetting DSLR & other cameras, just need to evaluate the best approach for them. Of course feedback is important, for instance, we discovered some kinds of scene lighting could cause some sort of artifact pretty late in the development stage, and that needed to retrain the whole thing with improved loss functions to address those. We need to iron out these cases through specific feedback to hopefully make the app better & more robust over time.

All in all, Hydra may sound like an ambitious project, and we sure have big hopes for it. At the same time, we're really exploring the new ways of computational photography & learning as we go. All we propose here is for you to join us in this new adventure & discover together where that brings us, capturing beautiful pictures & memories along the way.

Thank you for reading.

This has been a very unusual year for so many reasons that we all know. Frustration, sadness, but also change and hope.

About our developments, the team had to adapt and has been mostly working remotely. We tried to keep our focus as much as possible on bringing substantial improvements to our apps, implementing both user requests, adapting to new paradigms from Apple, as well as innovating on our main strengths & preparing the future.

Here's a quick recap of our recent releases in 2020.

Inko 2.0 & 2.1 (Collaborative Drawing)

In 2020, we brought Inko 2 out in May with the support of remote drawing (server backend) as well as a new macOS app through our first try at Mac Catalyst apps. Inko has been a pioneering app for us: our first app on Apple TV, our first app to make use of permanent cloud-based network connection for real-time collaboration, our first Mac Catalyst app. About Mac Catalyst, it was certainly not a simple move as we wanted to have a great experience on the Mac, not just a quick port, but at the same time Mac Catalyst (the first release) was so young & required a substantial number of hacks to make an acceptable app, which was both frustrating and exciting. Also, Inko (as well as Carbo) had a OpenGL rendering backend that we had to port to Metal, because Mac Catalyst wouldn't enable OpenGL rendering. We did our homework, and finally shipped it in May 2020. At the end of 2020, we shipped an important update as well to support iOS 14 (new widgets) and to cope with what we call Mac Catalyst v2 (Big Sur), and that was a relief as it now provides a much improved user experience from the previous one. Not yet a totally native Mac UI, but we now got speed (especially in Metal), and a much improved appearance with native controls (Optimized for Mac, as they say).

Emulsio 3.5 (Video Stabilization, Timeline Editing, Transcoding)

With Emulsio, we have a growing number of more advanced users for 4K and drone videos, among other things. Their number one request was to be able to control output quality. We were previously using the standard video export session of AVFoundation, which was easy to use, but was limited to just a few quality presets. So we implemented a fully featured video encoder relying on lower level AVFoundation (in turn accessing specific/optimized video encoding hardware of iPhone / iPad) and plugged it to our GPU frame rendering workflow. We can now provide output video at very high bitrate (>200 Mbps) for the most demanding use cases, including processing video obtained by higher-end cameras with larger sensors. This video encoding of Emulsio thus makes use of the native hardware encoder of iOS devices, which is both extremely fast and power efficient. We're not finished though, next steps are Metal & HDR stabilization and it's on our TODO list. Oh, and we also used this update to experiment with SwiftUI (the encoder settings are implemented that way) to get a first step into this new tech, and we also exposed the transcoding capability (no stabilization, just changing video format) as an iOS extension in this version. Combined with a gimbal, software stabilization gives exceptional quality & control, and we are thrilled to continue pushing things forward on that front.

Prizmo 5.3 (Pro Document Scanner)

Our pro scanning app Prizmo for iOS just got an important update to improve the iPad user experience, with keyboard and pointer (trackpad or mouse). OK that’s 2021, but development efforts mostly took place in 2020. We continue to further expose the new OCR capabilities of iOS 14, a.k.a. Apple OCR, now available in these languages: English, French, German, Italian, Spanish, Portuguese, and Simplified Chinese (was English only in iOS 13, other languages are still supported through our usual alternate OCR engine). Combined with advanced PDF caching, image cleanup options, and auto upload, custom compression (JBIG2) & small file size exports, Prizmo continues to offer advanced capabilities in the scanning field. Prizmo 5 was a multi-year project for us where we brought entirely new foundations, and we'll continue to build on that and bring support for latest OS features & new capabilities. So, scanning enthusiasts, stay tuned!

Carbo 2 (Digital Notebook for Handwriting Enthusiasts)

Carbo is a digital notebook app that was designed to bring handwriting into the digital realm. Both real handwriting (on paper) through a specific photo capture process that keeps all the finest details, as well as, more recently, using the Apple Pencil to draw directly on the tablet. Before 2020, we synced these notes on iPhone and iPad. Now, Carbo 2 brings note access and editing to the Mac, being our second Mac Catalyst app (the new one from Big Sur), fulfilling the vision of bridging handwriting & the digital ecosystem. You can now access your notes through the entire Apple ecosystem.

Carbo shares some technology foundation with Inko (the pixel-free rendering), and as such it had the same constraint of requiring Metal on the Mac (Catalyst does not allow OpenGL). Whereas Inko was released on Catalyst v1 (macOS 10.15 Catalina), Carbo skipped that release to focus directly on Catalyst v2 (macOS 11 Big Sur), which is a much richer environment. Carbo 2 thus features the Optimized for Mac technology, and feels snappier and better integrated for macOS users than it would have been otherwise.

Mac Apps on Apple Silicon

Finally, we were all very excited here about the WWDC 2020 with the announcement of Apple Silicon. Apple nails it here, because they know what really makes a difference in the final products, which CPU or GPU features make sense in the software and that should be optimized. Our company immediately took part in the transitioning program, and we have now released all of our Mac apps optimized for the new Apple M series of processors. And they scream!

To talk about the work that was involved here: it's not that much about the apps themselves, as this is mostly higher level code that ports easily to different CPU architectures, the hard work has been mostly to have our dependencies compile with the new tools (Xcode betas, with late changes up to November GM release), as this can be really involved with different build environments (Autotool, Makefile, CMake, third-party crash-reporting frameworks). It was also a great occasion to revisit our building tools and to automate many things along the way, and we're glad to be able to offer our apps to this amazing new platform. Another difficulty was Metal code, which had a number of different behaviors on Apple Silicon versus Intel/AMD GPU, and required stepping through code on both platforms to determine & hopefully fix any issue there.

More

There's also what's not visible as of now, the submarine stuff, as we are preparing for the future both technology-wise and platform-wise. This took important efforts in 2020 and we really can't wait to bring these things to the public. Also, we are very thankful to Apple about the Small Business Program, as this brings some fresh air and puts us in better shape to continue innovating.

In 2021, we’ve already been exploring new business models (subscription as a choice - more on that later), and we’ll try to revive some of our older heroes with phenomenal new powers, and who knows, maybe try new stuff.

So, stay tuned for more! And have a great new 2021 year everyone!

Inko 2, our collaborative whiteboard app, is not quite wrapped up yet, but we thought it could help to offer it through a public beta program considering the current lockdown situation. We’ve been working hard on it for the last 18 months, and the time has come to share it.

What is Inko?

Inko is a collaborative whiteboard that lets you draw together using multiple iPad, iPhone or Mac, and even interact on Apple TV. Ideal for team coworkers in a brainstorming session, for a creative classroom project, or for an interactive meeting between a graphic designer and their client.

What’s New in Version 2?

So, what does Inko 2 bring over the first release? A lot(*), but its core feature is collaborative drawing over the Internet so that you can interact with your grandchildren, students, friends, or colleagues from distant places. Combined with FaceTime audio or even video, Inko 2 provides rich drawing-based human interactions about teaching, brainstorming, creativity, or to share family love.

About the Beta Program

This beta program is invite-based, as we need to scale up our servers along the way, as well as fix any remaining issues. For this reason, we’ll add 500 new seats every Monday starting April 20. Also, we’ll let you know when new seats are being added through Facebook and Twitter — you’ll have to be quick to download the app through TestFlight when the update pops up. Finally, we’ll keep the beta app functional for 3 months (until the end of June) even if we release the app in the meantime, so that people in need can continue to use it for free. No signup will be asked for.

How to Install Inko 2

• Install TestFlight on your iOS device (running iOS 13.4)
• Open Invitation Link

You can now enjoy free collaborative drawing!

Do you have questions? Send us an email at contact@creaceed.com.

(*) Metal rendering engine, a Mac Catalyst version, support for Dark Mode, among other things!

Ten years after the initial introduction of Prizmo on the Mac, that came with the innovative idea, at the time, of using a digital camera as a document scanner, we are updating Prizmo on iOS. We’ve been busy with Prizmo 5 for over two years and we’re proud to finally release it.

Prizmo (App Store link, intro video, website) lets you scan documents on iPhone or iPad using the built-in camera or by loading input files, and then create PDFs with embedded text, read the recognized text aloud, or process business cards. Documents with text are OCRed (text is recognized) and the generated PDFs are searchable by contents.

Our hope along the way has been to offer the best document scanning experience on iPhone and iPad. Prizmo 5 for iOS is an entirely new app with a new user interface, improved foundations, and with many features that we’ve brought back from Prizmo Go after some successful experiments. We are so excited to finally be able to share this with our users, and we hope you’ll like it as well.

Version Highlights

Here’s a quick recap of the most important new features:

• Focus on speed: in just 3 taps, have your document scanned, cleaned up, cropped, and text-recognized to a multi-page PDF right into the cloud
• New best-in-class machine learning OCR options: reliable & accurate on-device OCR, and high-performance cloud-based OCR (same as Prizmo Go), including handwriting
• New shooting experience: detection & tracking, CoreML analysis, result preview, VoiceOver feedback
• New & much improved document editing workflow
• New background processing (OCR + PDF generation) and Auto Upload for PDFs
• Smart actions based on document contents (phone numbers, locations, etc.)
• Much improved Text Reader with text-to-speech
• New Messages extensions to scan & send a document without leaving the conversation
• And so much more

Simple or Capable App?

A design philosophy that we’ve followed all along has never been to avoid complexity and complex problems because simple solutions for sure can’t handle all cases. Nor was it to expose complexity up front. Instead, what we’ve tried hard to achieve is progressive complexity, that is, the most common cases should be handled simply, intuitively, with no compromise, and the less common cases, often the more complex ones, should get specific attention and options for when it’s needed. It is also good news business-wise as very often only the most common cases are well handled by the big names, this gives us room to explore.

Prizmo is a great example of this philosophy: internally it is the most complex app we’ve been working on, capable of so many kinds of manipulations, yet it proposes a fast path like we state on its main description: just 3 taps and you get your multi-page, cleaned up searchable PDF available in the cloud on all your devices. That is, one tap to initiate camera & autoshoot documents, one tap to terminate shooting, and one tap to close the document and go back to the app’s main screen.

What happens behind the scenes is this: on the first tap, camera restarts with its last cleanup mode, page size, and OCR language pre-configured. As soon as documents are presented in front of the camera, they will be shot after the “don’t move” progress bar completes. The user can dismiss an ongoing capture just by moving the device away (motion = cancel), or he can pause capture altogether by maintaining the device in oblique fashion (obliqueness = pause). When the user hits “Done” (second tap), the document is opened, with all pre-configured settings applied. When the document is closed (third tap), background OCR and PDF generation takes over, and after a few seconds, the cleaned and searchable PDF is sent to the cloud that has been pre-configured as well.

What is interesting though is that in any of these steps, you can dig in to override some settings like OCR language, cleanup mode, or export to a specific format like Microsoft Word DOCX instead of PDF. Or even automate the whole app from the outside.

New Capture Workflow

The overall workflow has been streamlined and simplified through a completely new user interface. Most options are chosen (and preserved for next use) at the time of capture, to avoid unnecessary downstream manipulations. Shots can be validated as they are being made, and you can immediately retake the picture if needed.

If activated, Prizmo can now autoshoot documents without user intervention when a document is presented in front of the camera, perform page cleanup (texture, lighting compensation) automatically from the user’s preferences, quickly perform OCR as it automatically runs in the background. Thanks to the new Auto Upload feature, the auto-generated PDF is uploaded to a user-defined location in the cloud.

Editing multi-page documents is easier than ever: no more back and forth navigation to the page list, swipe left for previous page, right for next one from anywhere. We tried to remove all previously required (and cumbersome) steps to offer a more natural experience. We did not drop the modal editing state from image processing tools, but instead augmented it.

Non-destructive Editing

Unlike most scanning apps, non-destructive editing is at the heart of Prizmo. After you’ve shot a document, you can apply a number of image enhancements like cropping, or image cleanup to remove uneven lighting, and of course, OCR to recognize text. Throughout this editing session, you can always go back and redo everything in case of error, as Prizmo preserves the source photo that was shot in its native document format.

Most editing in Prizmo is proposed in an orthogonal fashion, that is, most settings are independent of each other or exposed as if they were independent of each other. If the user wants to flatten a curved book page, he just enables that setting. It doesn’t matter whether this has to be done before or after cleanup or crop. Prizmo figures out the best path to get the best results in the shortest time, and sets the user’s mind free to achieve exactly what he really needs.

What is a Document?

Prizmo, as we see it, is a processing app. It lets you create PDFs, vCard, JPEG, TXT, or DOCX files, and that’s it. Once processed, outputs are yours. Like a camera app that shoots JPEG. Prizmo does use a native format to provide non-destructive editing, but we see these as temporary projects while working on the scan, and not as long term storage. We advise users to export files as PDF (or use auto upload), a widely available format, and then organize them the way they want, in a filesystem they can backup, and that they own.

On Prizmo’s start screen, under the document creation options & toolbar, a list of recent documents are shown for quick access. That’s it, recent documents for further editing or export. That recent document list can be expanded and it offers basic file management options, like searching, renaming, exporting. Native Prizmo documents can be stored on the device itself, or in iCloud, where they will be immediately available to the Mac version of Prizmo for further editing while staying in sync with Prizmo on iOS.

We did not want to expose native documents too much, that’s why you get a PDF when dragging from that list on iPad, as we think this is what users expect in the most common case (native documents are still available from Files app, or within the open document itself, in Prizmo). Additionally, Prizmo documents are now “actionable”, which means that based on their content, Prizmo will propose “smart actions” such as calling a phone number, navigating to an address, etc.

Deep-rooted Technologies

Technological advances have been specifically made for Prizmo 5. Our new automatic page detection technique in the camera viewfinder is more robust than before. We developed it with the specific (hard) case of a white document on white background in mind and with document borders that are not perfectly straight. Although this remains a challenging task, we’ve achieved unprecedented progress on that front. See by yourself how Prizmo compares to Apple’s Notes, the white-on-white top performer in our findings (most apps fail at this), in the video below.

Moreover, Prizmo now makes use of scene tracking (like augmented reality) to provide a more fluid experience & feedback. Again, we used our own scene tracking here (same as in our other apps Emulsio, Hydra, and Prizmo Go) as we found out it outperformed Apple’s Vision framework both in precision and energy consumption. As for capture, we’re using a custom-trained machine learning model (CoreML) for structural analysis to offer complementary capabilities such as automatic orientation determination (tap the compass icon to enable it).

Document cropping now features an edge repair tool to remove irregular borders or texture that typically occur around the cropped edges. Prizmo’s photo capture has been enhanced with an innovative stabilization technique that increases OCR accuracy. We found it to be more effective than hardware optical image stabilization (OIS) when shooting with Apple’s Camera app or other apps, that is, the image will be better suited to OCR if shot directly with Prizmo. Text polarity handling now lets Prizmo determine automatically lighter or darker text against the background, and a manual override is provided when needed, which makes it straightforward to OCR documents which have all kinds of contrasts and appearances. Document flattening has been specifically added to handle everyday documents like magazines or books, which are curved or not perfectly flat. Prizmo will just handle that and the generated digital copies will be made flat.

We hand-optimized neural network computations of the on-device OCR to get the most out of the hardware, and to have fast and accurate output at up to 6x the speed of the baseline. Advanced PDF caching is introduced together with background processing, and that means you can edit just a few pages in a 100-page document and re-generate the whole PDF in just a couple of seconds. About PDF generation, Prizmo now comes with efficient compression techniques (CCITT G4, JBIG2) in case you need to generate very small files of, say 50KB per page (usually 2MB, a 40x reduction), with high-resolution, 600DPI black & white scans. Generation of password-protected PDF documents is also available. New export formats are also proposed, like multi-image and DOCX with preserved document layout, and enable further editing in Microsoft Word, Apple’s Pages, or Google Docs.

Internally, we’ve now moved to a new mathematical foundation for OCR pre- and post-processing that is entirely made in Swift. We first initiated it for Prizmo Go in 2017 and further improved it to handle the more general cases of Prizmo. Swift is very well suited to handle geometrical structures and algorithms both in terms of speed and expressiveness, and this opens up new horizons for future developments.

Optical Character Recognition

Prizmo 5 now offers two distinct OCR options, just like Prizmo Go: a robust, always available, machine learning-based on-device OCR option that runs locally, as well as a high-performance cloud-based OCR option. We’ve been through our OCR modules previously in this blog post about Prizmo Go. Many Prizmo users wanted to have the latest advances that were introduced in Prizmo Go, and this is it.

Unlike Prizmo 4, both new OCRs are based on machine learning. They are quick, accurate, and robust. Among the very best in the industry. The on-device OCR can be finely tuned (binarisation) while the cloud one will typically offer enhanced accuracy, more language choices, and even experimental handwriting support (English only for now). Both OCR get our in-house pre- and post-processing techniques to enhance the readability of the text prior to the OCR and to perform document layout corrections after OCR has been executed. Also note that when the Cloud OCR feature is used, data is immediately removed after processing, as per Microsoft’s Privacy statement. Prizmo is “Privacy first”, and here’s our privacy policy.

Oh, and by the way, we’ll very soon update Prizmo to include a third (on-device) OCR: Apple’s new OCR available in iOS 13. It’s English only for now, and in our testing, it’s pretty good.

Accessibility

Accessibility has always been a high priority in Prizmo. Prizmo is not an app specifically made for blind or low vision users, nor for users coping with dyslexia, but its core capabilities of OCR and text reading make it useful in these situations. Not only does Prizmo offer full VoiceOver support, but it also comes with updated voice guidance during shooting as well as a new description feature (first introduced in Prizmo Go) that lets you know the quantity and position of text in front of the camera, as well as device horizontality.

Moreover, Prizmo’s Text Reader is customizable (text size & appearance) and it will highlight the words while they are being spoken by the device. You can read more about this in the next section.

Text Reader

Prizmo also comes with a redesigned voice-capable text reader. This is almost an app by itself actually. The new text reader uses built-in voices from iOS (a number of voices can be installed from iOS settings), and will read any text you send it, at your chosen rate. Recognized textual documents can be sent to the reader, but other apps can send text to it as well.

The reader has the qualities of an audio player on iOS, and text can be scrolled through with audio navigation buttons (e.g. go to next sentence). With special markers (##), you can define multiple pages that will help with navigation.

The reading experience itself can be fully customized with font, size, margins, etc. We also included the OpenDyslexic font as we are aware that Prizmo is also used by people with specific needs. That particular font combined with word highlighting as text is being read can really help.

Automation & Batch Editing

Previous versions of Prizmo were already pretty good at automation, as they could be invoked from other apps with x-callback-url scheme. It was a great feature first introduced in Prizmo 2 (yeah, a long time ago). Prizmo 5 further improves these automation capabilities: images can be imported into Prizmo with immediate text, image, PDF or DOCX conversions, or as native document (see documentation on GitHub). Cleanup options are exposed, as well as page format and most other options. You can also send texts to Prizmo for reading it aloud. All new OCR options are available as well. Prizmo is thus the perfect companion to the Shortcuts app for automated document processing. Note that it will also work from any other app with URL-based invocations, and the plan is to go beyond x-callback-url in future versions of the app.

Prizmo now also features batch editing for pages. The batch editor lets you propagate settings (like cleanup mode, OCR language, etc.) of one page to some or all the other pages of that document. This is a desktop-class feature and it is a snap for power users that need to process many documents as it avoids repetitive tasks.

Good iOS Citizen

As a good citizen on your iPhone or iPad, Prizmo brings many modern iOS features along the way. A new Messages extension is provided, in addition to Open In and Photos extensions, that enables quick scanning of a document or text without leaving the conversation. Full support for iPad multitasking and drag & drop is also offered.

Continuity and iCloud were previously introduced, and they are still there in this new release. In particular, the Mac version of Prizmo is updated to be able to handle the document format changes of Prizmo 5 for iOS. Even though the Mac does not yet support all the new features of Prizmo 5, the updated Mac version will still be able to interoperate with iOS on shared documents and preserve the new processing options.

Prizmo also provides new Siri Shortcuts to initiate documents or business card scanning through Siri. Handy. And, as mentioned previously, the callback scheme also makes it fully compatible with Apple’s Shortcuts app to automate document processing.

Finally, Prizmo also is a background audio player that will read scanned text on your stereo, smart speaker, AirPods, or in the car.

We’re constantly monitoring Apple’s new developments, and if there are OS features you’d like to have that we currently don’t offer, feel free to reach out (here).

Business Model(s)

Prizmo’s business model comes as 2 separate apps, the regular one, which is free with in-app purchases options, and the Volume Edition as a pre-paid app, targeted toward volume purchasers (enterprise and education).

The Volume Edition (link) is made specifically to address the issue that IAPs can’t be sold currently on the VPP store, and we’d advise against getting that app if you’re a standard user, as this version possibly won’t get all future upgrade offers due to limitations of paid apps. Also, at this time, we don’t propose the Cloud OCR feature in the Volume Edition.

About the regular version of Prizmo (link), as you’ve probably noticed by now, it comes as a free app. That is, Prizmo is free to download. We made that change for 3 reasons: it makes it possible to try the app before buying, it enables upgrade pricing for existing customers, and it is more flexible to deploy future features.

We believe it is important to be able to try out and compare apps before committing to one. This is especially true for deeper and possibly more expensive apps where you want to avoid bad surprises. The free download lets you do just that, try the various features of the app and its user experience, and determine if it suits your workflow. It comes with limitations (text access, watermark) that are removed when you get the in-app purchase.

Finally, Prizmo also offers a subscription option, specifically for the Cloud OCR feature. Some of our users do need an OCR that offers better accuracy, in particular with specific languages. We went with a subscription because this feature has ongoing costs on our side (server / pay-per-use), and this was the most fair and sustainable approach we could find, at the same time offering uncompromised results & accuracy. Icing on the cake: Cloud OCR is able to handle handwritten documents in English, which in turn makes it possible to make searchable PDF from manuscripts or annotated texts. Cool, isn’t it?

Thank you for reading!

We have now crossed the one-million download mark for Prizmo Go. In just over a year. That is our fastest user base growth so far.

It might seem small to apps that target audience growth. It’s never been the case for our apps, as we make productivity, accessibility, and creativity apps. Moreover, we aren’t used to publishing free apps. So, by our standards and the figures we have observed throughout the years, this is big.

Prizmo Go has been innovative in a number of ways. As we previously shared, we got the idea of Prizmo Go during the WWDC 2016 Keynote in San Francisco, at the exact moment when Apple introduced Universal Clipboard on stage (copy/pasting between iPhone and Mac). We already had Prizmo which excelled at PDF document scanning and OCR, and thus connecting the dots between what was just announced then and what we had was pretty obvious.

So let’s recap, simple idea, copy/pasting text from camera, building on what we already had. This was a quick three-month project, right? Well, it wasn’t. It never is.

1. Designing a good user experience takes time and energy. You start from a blank canvas, bring in some ideas, refine, bring additional ideas, refine again… From camera shooting UI to interactive text selection in perspective image, nothing was obvious. We started small, ideas came along and required experimentation, design updates were often necessary. It’s all but a linear process.

2. Quickly adapting to new technologies is vital. We have a long experience in the field of OCR, with Prizmo dating back to 2009. We have built pretty advanced toolkits for image preprocessing, multiple OCR engine abstractions & optimisations, as well as text/language/layout post-processing. However, during the development of Prizmo Go, we witnessed the rise of deep learning-based OCR. Results were so good that we were facing this question: do we release what we have despite the rise of that new technology? No. We wanted the best. We thus re-targeted Prizmo Go to include that (Cloud OCR) as of version 1.0. This change delayed the project by a few months and led us to rethink the business model as well. And about a year later (yeah, this year), starting with version 2.0, the built-in OCR also is deep-learning based.

3. Accessibility first. The standard Prizmo has had a long-time relationship with low-vision and blind users since 2009, bringing helpful OCR & VoiceOver capabilities to the Mac, and a bit later to iPhone. Because of its quick interaction process, we immediately understood the value Prizmo Go could bring in everyday use. We designed the app with that in mind, focusing on the speed of the interaction, as well as the control users have on the capture process. Although iOS comes with many tools to make it easier for developers, this still requires effort to do it right. For example, we added a button to verbally describe the scene prior to shooting with quantitative analysis of what sits before the camera. We also decided early on in the process that the scanning, built-in OCR, and text reading would remain available at all time, no in-app purchase required, when VoiceOver is enabled.

4. Staying ahead of its time. Prizmo Go introduced augmented reality-inspired text tracking when hovering over documents. This made the experience much more enjoyable, establishing the otherwise missing link between the app and the physical world. This was no small feat! Prizmo Go was released before Apple’s ARKit was even announced, and it was supporting devices back to iPhone 5 (have you tried running ARKit on a 5?). The great news is that when you optimize like mad for older devices, you get such a great boost when running on current generation hardware. Also, Microsoft & Google later released apps mimicking some of the capabilities of Prizmo Go. That’s OK, copies prove us right, don’t they? ?

Prizmo Go was also our first app to offer subscription. We did that in version 2.0 earlier this year, and it was proposed as an alternative to prepaid “units” for cloud services (Cloud OCR, adding the translation feature with that version). We probably lack some broader view about it but, to our surprise, it looks like people actually do prefer subscription over prepaid tokens.

On a technical standpoint, Prizmo Go is one of our first apps to be mostly developed with Swift for the new features (some pre-existing parts remain in Obj-C). The tools are much better now than they were at the end of 2016, but what can be said for sure beyond the code writing experience is that it sure feels like we have a much better mastery of unexpected crashes (talking about Hockey console) thanks to Swift, and the code feels solid and predictable.

Regarding Prizmo Go users, we believe the app is being used in different contexts, from business productivity (no more retyping) to language-related uses as it can conveniently translate text into many languages. Text-reading aid which has already been mentioned. Finally, a fourth use case that we currently observe is that Prizmo Go’s popularity is increasing in US schools, where it is being installed in large batches.

That was our mini recap about the Prizmo Go’s first million users. Thank you all who supported Prizmo Go, in particular, but not limited to, the AppleVis community for the warm welcome, the excellent in-depth reviewby Federico Viticci of MacStories which helped a lot, Florian Innocente’s “prise en main” on MacGeneration, Apple App Store editors for the App Today story they published, and obviously all enthusiastic Prizmo Go users worldwide! If you guys like it, feel free to leave a word or two on the App Store, or engage with us, we are always thrilled to hear feedback!

And stay tuned, some iOS 12 niceties might be coming pretty soon ?.

A couple of weeks ago, we released our new collaborative whiteboard app, Inko. Here’s the story of how it began, and some details of the making.

Before we start, let’s mention that we are releasing today a variant of the original app, Inko — Group Edition, in response to a number of requests we received regarding VPP compatibility. The original app was indeed distributed as a free app + in-app purchases, and was thus not available through Apple’s Volume Purchase Program. This should now make it easier to integrate Inko into schools and larger entreprises. We still hope for better distribution options (i.e., a single app) on the App Store in the future, but this is the best workaround for now.

Back at the end of 2015, we integrated Apple Pencil very early on into our digital notes app Carbo for iPad. We literally fell in love with the pencil. So accurate, and simple. Although it’s a bit slippery when compared to drawing on actual paper, it is otherwise a great digital alternative to a real pen. The end of 2015 was also when the first programmable Apple TV (with tvOS) was released. It was clear that we wanted to do more with the pen, building on our other app’s drawing capabilities. We had the idea of a collaborative whiteboard for quite some time and with the arrival of these devices, but also noticing that schools and entreprises were starting to move more towards digital solutions, it all made sense. And even more so since Apple has started deploying Apple Pencil and more affordable iPad in schools (see the March special event in Chicago).

We decided to start working on Inko in the summer of 2016. The project was at the time, and is still now, based on 3 key aspects:

• seamless collaboration,
• a great user experience focused on drawing,
• large screen mirroring.

Seamless Collaboration

We saw the app as if it were a shared piece of paper. Two people in the same room should be able to draw together. Lots of collaboration apps require devices to be connected to internet, that users manage some kind of network setup as well as invites, and they also typically don’t work if there’s no Wi-Fi.

Early on, we noticed Apple had built some great networking capability into iOS that enabled devices to “talk” to each other without requiring any setup. That first appeared in GameKit and then in MultipeerConnectivity(MPC) frameworks. These are Apple-specific extensions to Bonjour that allow NxM direct connectivity over Bluetooth/Wi-Fi radio (but to our understanding, they are not part of the standards).

We don’t have a long track record of networking apps, so we explored all options. It took months. We hesitated between using Apple’s MPC framework, using open source libraries such as CocoaAsyncSocket, or doing our own. We opted for the latter because of a number of limitations we were not able to overcome with the other approaches.

Network-based apps are a special kind of apps. You need to establish the connection between devices, define a protocol for passing messages around, encode/decode. But then the app should also handle the asynchronicity of messages as well as variability in the amount of data it has to process while remaining reactive at all times. Threads, queues, locks, throttling, compression. Fun stuff.

Collaboration is not just about networking. We needed a robust data model that would sync across users. There are a few possible options available like CRDT and OT. We have opted for a centralised repository-like approach that would store editing operations from different users. This is also the basis of the per-user undo system, and we combined that with a local database to store transient snapshots (canvas is bitmap based), so that we can redraw any point of the repo history with minimal number of operations. Repository also implements automatic pruning of older history to limit complexity for longer sessions. This works pretty well in the current star-shaped configuration, but we could well explore other models (mesh configuration).

A large part of the app has been written in Swift, which is very neat especially for representing data structure such as the ones described here. About Apple TV, it appears to be very similar overall and particularly in terms of networking, so that was pretty straightforward. During development, we noticed important changes in network behaviors between iOS 10 and iOS 11 (Inko only supports iOS 11) in the way devices establish a connection: under iOS 10, it would sometimes use Bluetooth instead of Wi-Fi and we had little control over that, but iOS 11 appears to achieve a form of unification of networking protocol to enable the speedier option each time. Finally, we also wanted to play with UDP as this would even enable smaller latencies, but it does not appear to be possible when using the peer-to-peer extensions at this time. So, we’ll try again later ?.

Drawing Experience & Design

We knew early on that Inko was not a “universal” app running on iPhone and iPad but rather an iPad app that also happens to run on iPhone. Inko is about contents, it is about drawing experience, and about live collaboration. We started the usual way with a standard navigation bar at the top, but this was covering useful drawing space on the screen. We finally opted for a fully transparent navigation bar (only buttons) that reveals the contents behind it, and that would automatically disappear when the pencil approaches while drawing.

For the tools, a rather small toolbar made a lot of sense. We wanted it to work both for the left and right-handed and to be relocatable because sometimes you just don’t want it in the way. We came up with the idea of a toolbar that would be “elastic” like rubber, with organic motion: it would morph into a ball when you grab it, and then you could throw it away where it would stick to some other place. This required quite a bit of force field tuning and other transitioning animations to get it right, but here it is, simple to grab and to reposition to any side. The captures here show the end result of repositioning the toolbar, how it interacts with split screen, as well as how we tune the force field at the core to achieve that.

About the actual drawing experience, and we already discussed that before with Carbo, Inko uses every bit of info that the Apple Pencil provides. Technically, it makes use of precise pen location, 240 Hz sampling, predictive motion/data that is corrected when actual values are available a few tens of milliseconds afterwards, and of course angles and pressure to construct the drawing shape. Inko is supposed to work great with Logitech Crayon as well, although some features like pressure are missing compared to Apple Pencil (feedback appreciated if you have one!). We iterated on the drawing method to further improve it, but also to make it work through the network, as these drawing shapes are propagated to other devices in real time as they occur, by using custom splitting and compression schemes to achieve that instantaneous look & feel. Inko proposes two (mostly) fixed pen sizes + one calligraphic variant that reacts more to pressure/speed/angles. We are already getting feedback to add more drawing capabilities like a “smart pen” ; these are are great suggestions, and we do listen!

Another important evolution from Carbo is the addition of colors to the canvas. Carbo has a monochrome black & white canvas that renders pixel-free contents, even though it’s backed by bitmap data. That monochrome aspect was its signature at the time, and also made it easier to implement on previous generation GPUs (remember iPad 3 with the first Retina display?). Colors were needed though in the context of a collaborative drawing app for obvious reasons. But adding color impacts how data is represented on GPU, how drawing is achieved, how contents are stored to disk (premultiplied alpha wouldn’t work, so we couldn’t use Apple’s Image I/O). In the last couple of months before release, we discovered we could actually use the P3 color gamut of iPad Pro’s but, because of the GPU rendering of Inko, there were interactions with True Tone and Night Shift that we could finally resolve with custom shader computations. The reds are particularly bright on P3 display, it looks unusual & fun, but we added an option for those who’d rather stay with sRGB instead.

Large Screen with Apple TV

Apps that run on a television are a great addition that is probably underused at this time (AppStories discussed it recently). Especially, in the context of a collaborative drawing app, TV apps open the door to classrooms, auditoriums, or meeting rooms in companies as dedicated hardware can be replaced by cheaper, off-the-shelf tablets and TVs, even beautiful 4K ones.

We wanted to go with a native app for Apple TV to offer a great experience. Indeed, had we limited ourselves to iPad display mirroring, we would have lost the ability of displaying full-screen 16:9 contents (iPad is 4:3 which would have showed margins on the TV), as well as faster live feedback, and contents interaction with the remote. The app, just like on iOS, had to be quick: launch the app and join a drawing group with just one press, no complex setup.

Thankfully, as we discovered, tvOS is an operating system that shares a large common foundation with iOS. The GPU rendering and networking parts of Inko were straightforward to port to tvOS, and only the user interface had to be recreated (and not even from scratch since some parts could be made cross platform with iOS). We kept the interface simple and focused: a screen to join nearby groups, the current drawing, the list of drawings, and the list of the group’s participants.

And here’s the quick story about Apple TV remote. Inko lets the user zoom in and out (double-press) and scroll by touch within the zoomed contents using the remote. However, we also wanted a way to interact collaboratively on the active drawing directly from the Apple TV itself. Drawing was obviously not accurate enough because the touch surface is a bit small, and we only added the highlighting tool. While we were at it, we noticed the white-circled remote had a gyroscopic sensor (that tracks rotational motion) and we gave it a test run to implement a laser pointer à la Wii remote. Clicking and wrist rotation can be used to highlight some contents. Unlike the Wii though, which has an absolute reference, we must set the zero position. We do that each time the user enters the highlighting mode by pressing the play button (think of a mini-calibration), so it’s important to point the remote towards the TV at that particular moment.

What’s Next?

With Inko, we’re tackling some new fields (education / companies) that we’re not necessarily familiar with. We’ve started discussions about school systems and modern teaching techniques here in Belgium, both with officials & teachers/professors in order to understand certain points: how things are organized, the needs that exist, and also how it compares to other regions of the world (there are important differences obviously).

At this time, Inko already received a warm welcome with an install base nearing 200,000 after just a month. It has been our fastest growing app so far (proud!). We’re in the process of gathering user feedback (there’s quite some of it), and tuning some behaviors of the app like making the evaluation duration longer + improving palm rejection (in 1.0.2), and also adding a VPP version as mentioned previously.

For the future, we already mentioned that we’d like to propose remote collaboration, as it could be very useful in certain use cases. We’ve also received a number of requests to make a better “One Note” app (from Microsoft) for collaborating together. To our understanding, this app is about documents, annotations and data of different types, which is not exactly the same thing as what Inko offers, namely “drawing together”, and that is what we’d like to keep focusing on at this time.

We are introducing this week version 2.0 of our Prizmo Go app, for instantly capturing text from the physical world. Prizmo Go got a great welcome last year when we released it, and was seen as a great and fresh execution of a number of technologies to simply achieve the goal of capturing text instantly. This year, we wanted to revisit its core feature, namely, text recognition.

As an anecdote, when we were working on version 1.0, in late 2016, we had only planned to have optical character recognition (OCR) performed on the device itself. We hadn’t thought a second about the cloud processing. At that moment, we heard of Azure’s offering and thought we’d give it a shot anyways, and we were pleasantly surprised by the accuracy of it. It was so good actually that we changed our plans to integrate it into 1.0, despite being late already. That’s the fun part in small companies, like, let’s build that major new feature 3 months prior to release ?, right?

Modern neural network techniques and deep learning had found their way into the text recognition world, giving it cognition capabilities that come closer and closer to humans. With previous generation OCR engines, we could achieve extremely good results when the image was of high quality and well framed. But this would degrade rapidly when the conditions were either non optimal or unexpected. Sometimes in odd ways, as we humans could essentially still read the text clearly. Well trained deep neural networks (DNN) come with that human-level capability of generalising to non-optimal conditions.

Prizmo Go 2.0 features a new neural network-based OCR that runs on the device, a new handwriting-capable OCR in the cloud, automatic cloud-based text translation, as well as the addition of a new subscription model. You can have a look at this video trailer to quickly review the new features:

Let’s walk through these new things in a bit more detail.

On-device Neural Network-based OCR

With the advent of cloud services that perform so well, we could have chosen to go only with those. Even though we are embracing them fully, we decided to double down on the integrated OCR offering as well. Why? Because it has complementary advantages, such as constant availability and better privacy.

Prizmo Go 2.0 comes with our CeedOCR text recognition library that is built on three things:

• smart image preprocessing & layout analysis
• a custom build of Tesseract (open source OCR library by Google)
• text results & layout post-processing

First, image preprocessing is important because shooting text image with a smartphone is so variable, going from a perfectly lit and framed image to one that even humans cannot read. Building over our 10-year experience in this field, we further improved it to generate such images that would specifically please neural network-based OCR, in turn improving the overall accuracy.

Second, as of last year, we noticed the Tesseract project was experimenting with LSTM (this is a form of recurrent neural network). We found that pretty exciting after witnessing similar cloud-based technologies and we gave it a test run. Despite some quirks, the gain in accuracy was immediately obvious. That got us thinking how this could fit in our apps, even though there were still important challenges. DNN-based OCR typically is more resource hungry than other OCRs, but specific optimizations and the power of recent devices are making it a reality.

Last, post-processing OCR results is typically needed, to filter OCR errors, or correct layout errors. We started this with Prizmo Go 1.0, where we built some mathematical foundations in Swift, and moving to the new OCR engine was the occasion to further build on that. We continued with Swift because it matches geometrical and math computations very well (thanks to generics, operators, and protocols), and we thought that would bring some more future-proofing to our libraries.

Photo-based Handwriting Recognition

Let’s start by a disclaimer: 1. it only works in English, 2. quality of results are highly dependent on writing style.

For the last 20–30 years, we thought that handwriting recognition were, in the best scenario, something that would be solved “later”. In fact, there are two cases: when writing directly on the device, or the photo-based one. The first case is somewhat easier, because not only the final image is available, but also the temporal sequence of inputs of “how it is made”. Some apps have been handling this very well is the last few years, just like the Nebo app, great stuff. In the latter case though, where just the final image is available, it’s been mostly impractical. Except for deep neural networks, of course.

Microsoft announced that they would be tackling this problem in the last few months, and we decided to have a test run in Prizmo Go. Why? Many people still need real paper to express their creativity. This is fine. And having tools to bridge the physical world with the digital one sure makes sense. It is very early and sure it is not perfect, but when it works, it is so impressive. This was not possible just 6 months ago, and now, look at this! We expect it to improve over time both in terms of accuracy and language support, so stay tuned!

Automatic Text Translation

In our reflection of what users would like to do with text, translation was high on the list. Typically useful when travelling, but also on other occasions.

Prizmo Go 2.0 supports this cloud-based automatic text translation feature which works in 59 languages. Note that text-to-speech and accessibility are still very well supported in this use case, bringing new opportunities in the app.

As this too is a cloud operated service, we can expect it to evolve over time. In our testing with French/English, we could witness that the modern neural-network translation variant is available, and it’s doing impressively well (available in 21 languages, as of writing). Just have a look at the English translation of this business magazine article written in French. Well written, same meaning, few mistakes. Added the German translation as well, please let us know if it’s good too.

Again, we expect the translation to gradually improve over time. We’ll also propagate the news when Microsoft updates the service with latest generation artificial intelligence on additional languages.

New Business Model

Some closing notes about Prizmo Go. The new neural-network based, on-device OCR replaces the old one, and is provided free of charge to existing users. It can be tried out for free, but text sharing requires the Export Pack one-time in-app purchase. The cloud-based handwriting recognition is available as part of the Cloud OCR option of Prizmo Go, under the same terms, that is, each invocation requires a unit (these can be purchased in 100 or 1000 packs).

In addition to existing options, we are introducing a new business model in Prizmo Go, the Premium Plan subscription. With Premium Plan, all features of the Export Pack (text sharing, copy/paste, smart interactions), as well as unlimited Cloud OCR operations are provided. This makes it easier to manage as there is no need to purchase Cloud OCR units and it makes it available at a fixed annual or monthly price, which is great for enterprise users. The cloud-based translation service is available to Premium Plan subscribers only. Furthermore, users who had already purchased the Export Pack are offered a special introductory pricing on Premium Plan (60% off first year, yearly plan only).

Since the first leaked pictures of Apple’s special new iPhone, it has all the internet’s population on edge.

It is indeed quite a change in design compared to the older models, which remained more or less the same in appearance since the introduction of the first iPhone in 2007.

Opting for an edge-to-edge device is becoming more and more the norm lately. It oozes indeed a sense of clarity as well as fluidity unmatched by classic smartphones, and that appeals to the public.

However, what mostly puzzled Apple aficionados at first was this clearly visible notch at the top of the screen. What was it for? And most importantly, why had the Cupertino-based company decided to disconnect itself from their previous seamless design that has been Apple’s iconic feature since the early years?

Answers to our multiple questions finally came during Apple’s mid-September keynote that took place in the brand-new Steve Jobs Theatre at Apple Park. The notch consists thus of advanced technologies that enable the use of the newly introduced Face ID.

More than a month later, the iPhone X with its very controversial notch still hits the headlines of every Apple news website. Several developers have therefore already started adapting their apps for the much awaited device.

We make no exception.

Discover here the new versions of our apps specifically upgraded for the iPhone X. And on day one, by the way. Wow!

Carbo — Notes & Sketches

After being recently optimized for iOS 11 as well as having the new Drag & Drop feature on iPad, Carbo has now been updated for the iPhone X’s iconic design. Your notes and sketches will appear more clearly than ever before. The iPhone X becomes the note, the note is the iPhone ?.

Emulsio — Video Stabilizer

Emulsio has also embraced the notch, which means that this one won’t hinder you from stabilizing & previewing your videos in landscape. The whole iPhone becomes the video. Already did this one ? ?

Prizmo Go — Instant Text OCR

Prizmo Go has also been customized to fit the needs of iPhone X owners. The user interface perfectly adapts itself to the edgeless screen, which gives you a better look of your recognized text. Notes, videos, texts. Notes, videos, texts. Are you getting it ?

Hydra — Amazing Photography

Hydra is the last (but not least) of our iOS apps to have been upgraded to perfectly fit the screen of the iPhone X in order to give you the greatest photographic experience. OK, time to stop with the jokes, for now…

These upgrades are free of charge for all existing users, so do not hesitate to grab them for your shiny iPhone X.

And feedback is more than welcome! iPhone X owners: please let us know what you think, each of our apps have their own Twitter account (@carboapp, @emulsioapp, @prizmo, @hydraapp)!

Busy summer! We just finished our second 12★apps promotion of the year (Back2School) and now iOS 11 is just around the corner. What a pace! Before that, WWDC’17 had revealed some great new stuff on the platform that sparked our interest.

From the beginning, we reflected on how we could (and would) include Drag & Drop in our apps in a way that would make most sense. Our apps are mostly productivity apps or do enable imports or exports of some forms, and as such, are good candidates for drag & drop.

Once used to the new APIs, we booked serious progress in just a few weeks. We knew then that it was just a matter of months before we would be able to launch these updates. And here we are ?

Note that dragging across apps is only available on iPad, as offered by iOS at this time. However, dragging within the app is fully supported on iPhone.

Let’s see what has changed, shall we?

Carbo

Our note-capturing/drawing app which uses a unique pixel-free technology to preserve all the expressiveness of hand-drawn sketches now goes even further.

With its new support of Drag & Drop, Carbo allows you, with just the flip of a finger, to move notes to different notebooks and tag them. On iPad, where cross-app dragging is possible, Carbo even enables to export them to other apps, be it from the home view (multi-drag supported) or the colored variant one, as well as import images (single image) from different locations.

Moreover, we have updated the app’s support of the Apple Pencil. This one brings more reactivity and accuracy, which greatly improves your note-taking experience. Improvements and fixes have also been made to the shapes generated by Apple Pencil.

All in all, it is a very visual and useful addition to the core features.

Check out our app here.

Prizmo Go

Our little newcomer did not escape this update either! Capturing text and recognising it are now made even easier thanks to the following new features. They include the import of images for processing and the export of recognized text to another app by dragging highlighted text directly from the picture, with this last feature being more impressive than we initially thought. Indeed, it does feel very natural and intuitive to select a few lines of text directly on the snapped picture, and then to directly drag and drop those to another app. We just love it.

This update also includes the Greek localization of Prizmo Go. Bear in mind though that Greek is only available if Cloud OCR is activated.

Go try it here!

Emulsio

Last but not least, our video stabilization app has also been updated with brand-new features related to iOS 11. This includes the import of compatible videos by Drag & Drop directly from the Files app (or any other app that supports dragging) as well as the import and export of HEVC, which is the new video format introduced in iOS 11 that enables more efficient storage. This makes it easier than ever to stabilize videos taken from third-party cameras (like GoPro or drone) directly on your iPad.

Try it out here!

These upgrades are free of charge for all existing users, so go grab them before we change our minds ?.

All things considered, we managed to book quite some progress these past months and we surely are not stopping there! We have quite some ideas up our sleeve so, stay tuned!

We are releasing today (June 29th 2017) an important upgrade of our Mac HDR editing app Hydra, now at version 4.1.

The main experience remains the same which is, enabling working in full resolution images and in real time thanks to the full GPU pipeline with Metal.

Creating a tone mapper is a tricky thing. It’s part technical, part based on taste. Ideally you want to propose many options so that users can create an image to their liking. But too many options also kill creativity, because users cannot recognize the ones they need to achieve a certain result.

We took this opportunity to revisit the color grading techniques used in Hydra with a focus on making default settings produce more drama. Indeed, part of our users requested more vivid colors, and while Hydra 4.0 did provide colors that were coherent with input photos, we got it that it also makes sense to go beyond reality for artistic needs.

Hydra 4.1 features a revised tone mapper with a new color gradingtechnique. You can see the difference between version 4.0 and version 4.1 in the images below with default settings.

The updated tone mapper gets a new tool to adjust color vibrancy. Vibrancy is similar to saturation in some ways, except that it focuses more on improving desaturated colors only and also treats the red differently (to preserve naturalness and skin tones).

Note that saturation can still be adjusted overall or per “scope” if you need that kind of fine control.

The updated tone mapper also features a local contrast improvement pass, that, combined with the new color grading, provides more drama to the resulting image.

Finally, this update also brings a number of bug fixes. For instance, we noticed differences in image loading between El Capitan and Sierra, where the former would typically crash when trying to load a RAW image using Metal (GL or software fallback had to be used), while Sierra was much better at it. Also, some older Macs with discreet GPUs (mid-2010 MacBook Pro) appear to have a hard time displaying mipmapped images (anti-aliasing technique that we use in Hydra), and we’ve built some workaround for that too that do not affect the final image.

nd with macOS High Sierra & Metal 2 just around the corner, we are keeping an eye on what can be done to further enhance Hydra. Note that in our testing, Hydra 4.1 has a couple issues with High Sierra on some Mac models when Metal is enabled (it does not affect all GPU types). It is likely that this is due to the major changes that High Sierra is bringing on that front with the new Window Server and new Metal version. We have reported these issues (rdar://32220731 & rdar://32999346 ) and we will keep you posted once the OS settles down. Metal can be disabled from Hydra preferences in the meantime if necessary.

You can download a trial version of Hydra 4.1 from here, and it even works from a single image (JPG or RAW), so give it a try!

Hi Everyone!

It’s been a while since you have heard from us, but rest assured we have not been twiddling our thumbs! Indeed, two weeks ago on 27th April, we released our brand new app, Prizmo Go.

If you haven’t seen Federico Viticci’s review of Prizmo Go on MacStories, please have a look, it really goes in depth about the app. We are all very proud here to “have gained a permanent spot” on Federico’s iPhone, such an achievement ?.

Also, we’ve already discovered a number of podcasts from the accessibility communities that we’d like to share here:

• AppleVis: https://www.applevis.com/podcast/episodes/get-ocr-results-go-prizmo-go
• MacAccessibility (M12y): http://maccessibility.net/tagged-podcast.html
• Blind Abilities: https://blindabilities.com/bateam/jeff-thompson/prizmo-go-ocr-app-with-voiceover-accessibility/

We’ll go through the main features of the app, as well as give a glimpse at some related technical bits.

Instant Text Capture

Prizmo Go allows you to take a picture of printed text, with real-time text highlighting in camera preview, which is then recognised in a split of a second. You can then interact with the text, copy it to other apps or to your Mac. Say goodbye to that tedious retyping. Feels like magic!

The real-time highlighting combines both detection of text lines and motion tracking that is pretty common in augmented reality. It makes those lines stick to text contents in the live video feed. Running this on the device consumes a fair amount of power because CPU, GPU, and camera all run at the same time. We noticed this during development, and decided to have a “Low Power” feature in case battery is too low or when you want to scan lots of text. This mode removes the visual feedback, but does not change OCR output.

Stabilized shooting mode also brings its share of innovation. Sharp image is really what brings the best text recognition results, and we really wanted to improve that part. What we did previously in our other apps was that we tracked accelerometer to determine when the device stopped moving. Not bad, but this was a very rough prediction of image quality because of delays, unpredictable and instantaneous shaking gesture, etc. In Prizmo Go, we came up with the idea of continuously tracking image sharpness of the video feed and pick the very best image. That came with a number of implementation complications, but in the end, it turned out to be effective.

Rich Interactions with the Captured Text

You can swipe your finger to select text parts directly from the picture, and that text can then be copied or read aloud. Other interactions include tapping to browse any printed website address, calling phone numbers, triggering Mail app from an email address and many more.

As you would expect, this is implemented using iOS’s 3D Touch Peek and Pop and Data Detectors. This is a great illustration of building upon OS features to provide complementary features that make a lot sense in this particular context.

Copy/paste to other devices is a feature that was introduced with iOS 10 and macOS Sierra. It’s a great feature, and, with Handoff and iCloud, it’s a feature that seamlessly bridges separate devices together. I remember when Universal Clipboard was announced at WWDC’16, I was there with my colleague @BunuelCuboSoto inside the Bill Graham Civic Auditorium, and this single feature is actually the one at the origin of Prizmo Go’s concept.

Accessibility & VoiceOver

Prizmo Go comes with enhancements specially built for VoiceOver users. We wanted to create an app that could become a trusted companion in case you need help reading printed documents. We hope it will improve the lives of visually impaired and blind users on a daily basis.

We had prior experience in developing a VoiceOver app with the standard version of Prizmo. However, this new concept of quickly and accurately capturing text had even more potential with vision accessibility, thanks to its focused use case. We took this opportunity to further improve the techniques.

For instance, Prizmo Go has some built-in capabilities of determining text configuration. But when used with VoiceOver, we go beyond the standard mechanism. Prizmo Go will test more text pose hypotheses which might be unconventional when used by sighted users, but that make sense otherwise.

Prizmo Go will also trigger automatic audio playback of the text when VoiceOver is active, and the entire text is automatically selected. The text panel is also maximized to give more space in that use case to better interact with the text. Finally, we also enabled the same energy optimization that we use in Low Power mode, because we expect VoiceOver users to use it more frequently in scanning sessions, and the experience is better if battery lasts longer.

Text Recognition

The app features an advanced image processing pipeline (cleanup, perspective correction, dynamic rescale). A new feature of Prizmo Go is that there is now a new Cloud OCR option, in addition to the built-in one.

This Cloud OCR option works from remote servers, as you would expect. It is implemented over Microsoft Cognition Services. The good news is that it’s very accurate and it offers 22 languages at this time including Chinese, Japanese, Korean and Arabic! This is the first time a ‘Prizmo’ app can handle non-Latin languages, and we took this opportunity to make sure the whole app UI would behave correctly in a right-to-left language. Another difficulty was to handle Japanese input, because it can be horizontal (left to right) or vertical (right to left), and we had to implement specific parsing here. Note that the OCR engine still cannot handle both directions on the same scan, but we sure hope a service update here that would allow that.

The reliable built-in OCR allows the use of 10 languages and does not require an internet connection. It’s the same OCR as Prizmo actually, and we tuned it some more to automatically handle unaligned text. Another improvement that we have worked on is the automatic contrast type selection: bright text on dark background or dark on bright is automatically detected throughout the document where it was previously necessary to manually pick the appropriate mode. Document language can also be automatically determined from text contents in this update.

Business Model

Most of our apps so far have been paid apps, except Emulsio which is a free app that offers a one-time in-app purchase to unlock the main feature. But Emulsio is more of a niche app, for specific use cases, where Prizmo Go addresses more situations. This is the first time we try free + IAP for this kind of app. The main motivation has been that users want to try before they buy, as it should be. Because App Store does not provide that feature, we implemented it through in-app purchase.

Users download the app for free, and try out capture and OCR on some text. Cloud OCR can also be tested after getting the free trial units. Text can be seen in the application interface, but to actually use the text, users need to purchase the text Export Pack.

After using the free Cloud OCR units, users will need to purchase additional ones by getting either the small pack (100 units) or the large one (1000 units), or they can stay with the built-in OCR if that’s enough. Each user is responsible for his Cloud OCR balance, and that way we can cover the service costs. We also discussed a subscription option for Cloud OCR, but did not retain it at this time.

Development Notes

We have been working on Prizmo Go for many months, as you would expect. It’s one of our first projects where we have committed to using Swiftinstead of Objective-C. It’s not 100% Swift because we have many reusable bits of code that do some real work and that we don’t want to rewrite now, but most of the new code is Swift. We are now using Swift both at the application architecture level (Model / View / Controller) and low-level algorithms. We also factor out some utility functions and classes that can be used in other apps.

As @benoitsan first noticed, a clear advantage of Swift was fewer crashes. Almost no crashes at all. It was so obvious that we clearly saw it throughout the first betas (we use HockeyApp to collect our crash reports). We have about 100 crashes for a 6-figure user count (in 2 weeks!), we’ve never achieved that before. Crash proofing is a remarkable feature of Swift, who played an important role here.

The code is more expressive, there are less files (no headers), but navigation inside classes can sometimes be difficult. The other difficulties we had were mostly due to the slow compiler, slow IDE completion, variable lookup in the debugger, and lack of static libs to organize code. Crossed fingers for Swift improvements at WWDC’17!

Between welcoming a new recruit and working on various absorbing projects for months, we have not had the opportunity to be bored! And we wanted to share with you a short summary of all the events that have marked the company and that have cheered us up throughout the year.

Get Productive!

We started off 2016 by taking part in Apple’s Productivity Bundle in January, which was a great success. Next to Moleskine’s Timepage or Cultured Code’s Things, 3 of ours apps were displayed among that great productivity round-up: both versions of Prizmo (iOS & Mac), our document scanning apps, and Carbo, our handwriting app. We were so proud of that recognition.

Carbo Optimized for Apple Pencil

Speaking of Carbo, we released a major update in May so that the app was further optimized for Apple Pencil and iPad Pro. Carbo 1.3 brought a much refined drawing algorithm and also proposed additional choices for the drawing style: calligraphic or fixed thickness with multiple width options. Moreover, Carbo now offers high-speed sampling for more accurate sketches and predictive sampling for reduced perceived lag while drawing, building on Apple Pencil greatness.

WWDC’16: Introducing iOS 10

We attended WWDC again last year in San Francisco, where @rsebbe & @BunuelCuboSoto had nice interactions in the labs, gather information about all the new APIs, and meet fellow developers.

Through the different announcements, we couldn’t wait to update our major apps with iOS 10 features. Therefore we added some cool new features to Hydra for iOS: 3D Touch for Peek and Pop on the last taken picture (which lets you execute custom actions), as well as other color fidelity improvement made possible by iOS 10 (True Tone shift decrease).

Brand New Hydra 4 for Mac

2016 was also the year we released a new major version of our high-dynamic-range (HDR) app for Mac, Hydra 4.

Hydra 4 offers a brand new interface and image processing architecture, which is now built on top of Metal and optimized for macOS 10.12 Sierra. Metal is Apple’s new API for modern GPU processing on Mac & iOS. We wanted to modernize our image processing architecture from previous OpenGL / CPU code to use pure Metal instead, with the goal of offering maximum rendering & display performance and also to make it future proof. We learned a lot from working with this great piece of technology!

It was the first time we developed a Metal-based app, so this wasn’t really effortless: it took time and patience, we made a lot of research. In the end, it paid off, the app is behaving great: fast interactions, fullscreen & full-resolution previewing. We were so glad to see that Hydra for Mac became “Editors’ Choice” on the Mac App Store. Our first time for a Mac app! Again, we were (and still are) so proud and thankful for this!

Embracing Swift

Still on the technical front, 2016 was also the year we moved to Swift for most new developments. We ran some experiments previously with Swift 1 and 2 for parts of our apps which actually shipped (Carbo’s CloudKit backend, Collectarium). But Swift 3 introduced some important changes in 2016, and it looked like it was starting to bring some real benefits. We think Swift can make our code more robust and more maintainable. The situation isn’t perfect yet (developer tools are so slow, static lib support still missing, making hard to build on modular concepts, etc.), but we felt like continuing with Objective-C would probably become a technical debt.

12★apps: Featuring European Indie Devs

At the very end of 2016, we launched a collaborative initiative to promote European developers: 12★apps. It was a great opportunity to bring indie developers together and to promote great apps from all over Europe. We’ll do that again throughout the years.

2016 in Numbers

• Thousands of emails answered within the 24 hours by our support team.
• Over 1,200 cups of coffee and 1000 Cokes drunk.
• Over 300 beta testers.
• Over 100 builds of our apps submitted to App Review.
• Over 60 press reviews of our apps.
• Over 15 Apple devices used for testing.
• 1 new MacBook Pro.

What’s Next for 2017?

2016 wasn’t straightforward but still full of great moments and amazing work. The tools and ecosystem are evolving so fast that it can be a challenge to catch up. Still, we are currently working on new concepts, as well as preparing substantial updates for our existing apps. We’ll continue to focus on apps that are both useful and enjoyable to use, providing great user experience and innovative technology.

And we really can’t wait to show you all the new stuff we’re working on.

Take care!