Blog posts related to Handhelds

WebAPI: The Mozilla Proposal For Smartphones (The Open App Interface)

WebAPI is basically an idea from Mozilla which promises to bridge the gap between native and web applications.

With the constant development of Mobile Applications for handheld devices there are quite a few solutions and frameworks out there for developers, but these solutions always refer to closed OS environments. On the other hand, there is a trend for a portion of developers to consider writing an application in JavaScript, HTML & CSS. By using a framework which converts it into a native app they can publish their application easily.

This can be done by using, for example Phonegap (iOS, Android, Blackberry OS, WebOS, WP7, Symbian & Bada) which is quite popular, or write an app and put it under the Cocoa Touch iPhone App Wrapper which will make it a native app.

Now… Mozilla’s idea on bridging the gap between native apps and web applications is quite different from frameworks like the aforementioned Phonegap because the purpose of WebAPI is not merely to make native applications but to reach low-level functionality of a device using HTML5.

Of course, if Mozilla wants this to be a success, it has to be adopted by companies like Google, Apple and Microsoft. So Mozilla has to, first and foremost, convince users it is the way forward. For the time being they are hiring developers for it.

Dual Boot Handsets

Many people, we are sure, remember the early days where everyone was struggling to setup their PC in a way that it had two Operating Systems (Windows & Linux) on a Dual Boot mode. People may still struggle for that matter! If it was correctly setup GRUB would display an option to choose the OS to boot. Those where the days…

And then came virtualization…

Using the new concept a user did not have to reboot all the time, switching from one OS to the other. They were running in parallel.

When smart phones came along the same idea was born: What if you wanted to run Android on your HTC HD2 even though it shipped with Windows Mobile 6.5? Well, the XDA Developers did an excellent job on this by making this possible for hand held devices. ROMs were ‘cooked’, OS were tweaked and so on, in order to reach a point where a user could choose the OS to load.

And then came virtualization ??? Deja Vu …


VMWare and LG presented a virtualized  Android phone in MWC 2011 and the potentials of this are very impressive. VMWare has started with Android and LG but i am pretty sure that there is a lot to come on this field.

Check out the video from Engadget along with some comments here.

The death of the ‘mechanical’ keyboard

During the Christmas break, we got a visit from some friends, along with their two kids (2 years and 5 months old respectively). What stood out during their visit, was how children discover, understand and use current technology. Here is how it all started…

During one of our endless discussions, next to the fireplace,  about nothing and everything, the father reached for his iPod Touch and activated the device, entered the password and gave it to the older boy. The boy surfed through the applications, found one that had a baby picture and pressed the touch screen to enter a menu of five categories. The father asked the boy to look for ‘cars’.  The boy selected the Automobile button and nine new pictures appeared. Then the father asked for the boy to look for trains. The boy navigated through the menu to the train picture, pressed it and started laughing when the train sound played through the device. Happy and excited he continued playing with his toys, leaving the iPod to his father.

After 5 minutes the device was locked and the kid wanted to play again. This time he reached the device by himself, entered the password that his father entered earlier – albeit an easy one – and launched the app again.

Child operating a Touchscreen (Image from University of Kent)

Child operating a Touchscreen (Image from University of Kent)

It was quite impressive how the 2-year old could handle the iPod, browsing through the pages and making the appropriate selections. However, we have seen that before, even with more complex interfaces. What was not so easily apparent is how young children are used to ‘touch-screens’, being fairly comfortable using them contrary to traditional keyboards.

As the father mentioned, a couple of months back he bought a cheap video camera which had the classic buttons for settings and usage. The boy wanted to play, so he opened the camera’s screen and started touching on it.

His disappointment was apparent and expressed vocally as the device did not respond when the boy touched the screen and he could not figure out another way of operating the device.

It occurred to me that when these kids start their own families it is possible that the technology that we use  currently may be obsolete … maybe we will not be in a position to understand new devices easily. Kind of like Robert Gu at the begining of Vernon Vinge’s Rainbow’s End. Or the scene from “Back to the future” where two kids try to play an old-school video game and shoot Indians. Michael J. Fox comes in order to show them how to play, grabs the gun which had a cable attached to it and starts pressing the trigger. Disappointed, one of the kids turns to Michael J. Fox and asks:  “You mean we have to use our hands ?”

Most of us have seen how things  began and evolved regarding mobile phones or PCs. I got my first mobile phone at the age of 21 and if I go back in time I remember I was one of the few kids that had a PC at the age of 10. It was a very powerful machine with a V20 Processor and a 20 MB HDD. By today’s standards, not enough even for a cheap mobile phone…

New generations may take technology for-granted, and probably will not be able to fully appreciate the steps that led us to today’s technology. Whether it is necessary to understand these steps is a different matter.

And it came to us…

Here we are today talking about AR & VR in small “smart” devices and a dozen other concepts that kids will probably use from an early age. How will that affect their creativity and technological progress. Will it hinder it, or will it open new possibilities and allow creators thing in a different way than we are. We are hopping for the latter.

2010 (A)R-evisited

Happy new year, may all your dreams come true!

After a brief break to catch our breath and have a much-needed holiday, Synthetic Toys is back up and running. This time we attempt a recap of 2010 in the area of Augmented Reality (AR).  AR has drawn much attention through the past year, almost a decade after we started working on the subject. Although the ‘hype’ surrounding the field is quite a lot we are still skeptical about recent ‘advancements’ and how these will shape the way AR evolves.  We will attempt to look into things aided from notable sources and offer our predictions and hopes for the field in 2011. One thing is for sure. AR is as interesting as it ever was, if not more, now that people begin understand its idiosyncrasies and pitfalls as well as its potential.

Handheld AR

Handheld AR is probably the sector that became more popular during 2010 with many companies offering AR-related applications, attracting lots of public attention [3,6]. Companies like Layar, Wikitude, Metaio and Qualcomm presented applications within this paradigm. The general idea is to use a smartphones modalities – GPS, compass, camera and wireless/GSM – in combination to superimpose 2D and 3D information in space using primarily the phones screen as a presentation medium. Examples of each company’s approach can be seen on their websites. In our minds we separate Handheld AR applications in two categories. Those that use a tangible reference for registration – like Qualcomm’s game using a camera tracked marker – and those that use a non-tangible one, like GPS and compass. Marker-less registration kind of falls between these categories but can be potentially more important.

Qualcomm & Mattel on Augmented collaborating on Reality

We at Synthetic Toys are thrilled with the current popularity of AR but can not help to remain skeptical on how current, marketing-driven, popularity can lead to the maturity of AR. But more of this further on…

3D Displays

3D Displays – HMD-less or not – have also gained lots off attention the last year, due to the appearance of blockbuster titles like Avatar in 3D. Big players like Sony, Panasonic, Toshiba, Mitsubishi and Philips introduced such displays in the market, in an effort to capitilize on the technology. Generally, HMD ones are not as easy to use due to various issues such as ocular and non-ocular symptoms, as commented on a previous post and require further research [10]. For this reason many researchers, including companies like Apple and Microsoft experiment with 3D displays that do not require a spectacles-type apparatus, but instead rely on different modalities to determine orientation of gaze [11]. Although the currently available solutions are at their infancy we feel this paradigm is quite promising and there is lots of potential and opportunities for research. Particularly, in conjunction with smartphones, where viewing distance is more or less easily determined, such displays can enhance handheld AR to another level, without requiring from the users to wear ‘dorky’ HMDs.

Kinect

Right from the moment of its release, Microsoft’s Kinect became a very popular gadget with hackers, with open-source drivers appearing immediately after its European release. Kinect can have a sizeable impact in the field, as many believe it is one of the best AR-related sensors in the market. Anything that can provide information on the user’s movements in real time has to be pretty useful in AR context. James McQuivey, Consumer Product Strategy analyst at Forrester, had written two weeks prior to the release of Kinect that it “is to multitouch user interfaces what the mouse was to DOS. It is a transformative change in the user experience, the interposition of a new and dramatically natural way to interact — not just with TV, not just with computers — but with every machine that we will conceive of in the future.” [4]. He further asked [5] “What will we call the new experiential medium that will result from natural user interfaces+3D+touch interfaces+augmented reality — technologies which are all conspiring in this decade to alter our lives?”

Augmented reality with hacked Kinect

Most researchers have used it to control software like word processors or games, like World of Warcraft depicted here but it is quite reasonable to foresee its use in AR applications. The popularity of Kinect is quite high and you can find some really nice AR-related experiments, such as the ones depicted in the video above, over the web. Moreover, the technology behind Kinect – or indeed part of it  – may soon be available to PCs enabling researchers and developers to use it in different scenarios with greater ease.  For us Kinect does one simple thing. Places the sensory modalities ‘away’ from the proximity of the user. Now, how about making a 3D display that is away from the user, independent of viewing distance and angle and… oh yes, we talked about that on the previous section…

Standardisation

One of the subjects often discussed in workshops and conferences is the need for stadardisation in AR. Many of the aforementioned AR players, researchers and institutions have debated on whether it is time for AR to have such standards. Granted standardisation does offer some benefits through unification, interoperability, reliability and predictability.

We feel that AR standardisation deserves a thorough examination and we will look into the subject in detail in the future. We are also keen on seeing the results of an AR Standards Meeting in Barcelona on February, expected to attract the interest of many important players in the field. However, we feel its a tad too early, mainly because we are reluctant to believe that AR has matured enough and that the current ‘call’ for standards is more market-driven than a technological necessity.

And that brings us to…

Our opinion

2010 was a year with a lot of publicity of AR, with many people getting involved and lots of new ideas flying around. But that is not necessarily a good thing. Augmented Reality has been around for quite some time – we’ve been researching it for over a decade – and approaching the field with the ‘naivety’ of modern marketing will not get far. In fact we are afraid that people are ‘over-hyped’ with AR and they will be disappointed soon, once they find the technology has limitations at the current state of things. Most of the market hype is driven from the, undoubtedly clever and successful, publicity methods of the big players. Handheld AR in particular seems a very easy way of getting people to know the concept and getting a hands-on experience. And that is the way it should be during the beginning. But lets face it, after a while is simply a gimmick. And that is attributed, in our eyes, to the limitations of current technologies, particularly when related to registration and the level of immersion of the presentation. Moreover, the next set of services and applications will not only need to exploit technological advancement to the fullest but will also some ‘out-of-the-box’ thinking to devise new paradigms. But lets get into the gist of things…

Handheld AR is probably the flavour of AR that will remain fairly popular due to the evolution in smartphones and tablets. However, there will soon be dire need for new types of services that enhance the user’s day-to-day routines and reality. Enhancing the latter does not necessarily mean placing high quality 3D graphics around. Enhancement also can come as any type of information, whether visual, audible or as tactile feedback.

Mechanics allowing tethering and location awareness are of course of paramount importance. However, increasing accuracy and performance while maintaining multi-modality will allow more complex – not complicated – frameworks to develop new services. We also feel that interaction between users, within the AR space much like a shared environment can have an important role is the sense of immersion of users.

Leaving the domain of Handheld AR behind and speaking in more generic terms, Interfaces in the context of AR are always a very challenging subject. Although some progress has been made on non-HMDs, very little improvement has been observed in HMDs – and to that we have to agree with Tom Carpenter [6]. Bearing in mind our own experience with HMDs and the sense of immersion they bring to the table there is still a lot to be done to investigate different technologies as well as the side effects of HMD usage. Naturally, we await with great interest the developments on the glass-less front.

If one thing has advanced considerably and will continue to do so, pushing AR development further, is processing power, particularly in small form factor platforms. Smartphones are capable these days to render complex graphics with very good frame rates. Consumption and battery longevity is indeed an issue but not so much as in the past. Although technically it belongs to 2009, Epic’s demonstration of the Unreal Engine 3 on IPhone, shown bellow, aptly demonstrates what these handheld beasts can do.

iPhone 4 Unreal Game – Project Sword

Concluding, we have to admit our review is nowhere near as extensive and detailed it could be. We gathered the things that had made an impression to us, those that we feel are the more important ones. We are certain we missed something and that our assessment may be erring a bit on skepticism. We are, nonetheless. very excited about the publicity AR is receiving and we are also planning on ‘playing ball’ within the current market with some ‘gimmicks’ of our own.

References

[1] The Year in Computing
[2] Eye-tracking for mobile control
[3] The Year in Enhancing Reality
[4] Get Ready for Kintetic to Completely Change our Lives
[5] Killing Me Softly With Kinect…And Leading Me To The Next Big Thing
[6] Augmented Reality – Year in Review – 2010
[7] Android 3.0 Honeycomb has Google-Built in Augmented Reality?
[8] Has Augmented Reality Peaked?
[9] Augmented Reality: Pure Hype or the Next Big Thing in mobile?
[10] Is 3D bad for you?
[11] 3D without four eyes

A Mobile Era

Technology Review has a nice post on the current state of the mobile phones industry, with emphasis on the social implications. You can find it here. While reading it I was thinking of the time when mobile phones were not as smart as today, PDAs were much simpler than the cheapest phones of the last five years, I was building my first wearable computer and was holding Open Day AR demonstrations in the University of Essex. I remember the groups of parents that came to see the work done in the Vision and Synthetic Environments (VASE) Laboratory and how impressed they seemed, whenever I told them that the computers of the future will be our mobile phones. Small, context-aware, pro-active, always tethered to some form of network and operational for large periods of time. Of course, this was not my prediction but more or less what all researchers in mobile, wearable and ubiquitous computing envisioned back in the previous couple of decades.

The article from TR summarizes the current state of mobile phones, mentioning how all of the aforementioned ‘features’ and how they are currently being encountered. Notably, as Jaffrey Rayport, the article’s author states, we are outsourcing our memory to our devices. More and more information, like phone numbers, addresses, location notes and task-lists are stored in those tiny gizmos instead of our head. Our decisions are often automated through criteria based searches where as our collaboration and communication patterns are much more immediate compared to the past.

This is indeed the era of mobility. An era where we are not anymore pinned to a location but mobile, unthethered and immersed in a cloud of information. Instead this time, the information that concerns us is pinned to our environment. Kind of like the notice board, back in the university, during those demonstration days.