Don’t worry, it’s only minor – Bug severity in Oculus Rift testing

Bug severity always raises different opinions. We’ve all submitted a bug and seen it edited down to a lower severity. Severity ratings become a loose guide to the nature of a bug. They can be useful, but seeing the 1-5 rating does not relate enough information on its own.


There are bugs that are a lower priority to fix, but there are no minor bugs when testing in VR.

*ANY* bug can break immersion.

Our aim is to give users the most immersive and seamless experience possible.

A bug may be minor in nature, but its knock-on effects are never minor. A user may recover immersion quicker from a less critical bug but that does not make it minor.

Immersion is totally possible, but only if we make it the smooth experience it needs to be.



Doesn’t look like that on my computer – The Oculus Rift version

So we’ve all been there…By we I mean developers and testers. The tester finds a bug and the developer says it doesn’t behave like that on their computer. So then you puzzle out what is different between the two machines; well that’s how it should go…

Now take that situation and increase the variables by at least a factor of 10. That is what happens when you bring Oculus Rift into the equation.

So I’m retesting a bug based around the position of the content. I briefly touched on depth of field issues in a previous post, but let’s get into it more. Where you position content in relation to the user is critical. If content is too close to the user; they will feel claustrophobic. If it’s too far; then they won’t be able to experience it as intended. If it’s a little too close; then it makes accessing content below the user’s resting eye level very uncomfortable.

In today’s situation the dev came around to watch me using Oculus Rift as I retested this bug. Not only did we realise it was still a bug, but we also realised the difference in how we experienced the same content. Between the way the headset was calibrated and the positioning and angle of the motion tracker; we realised there was a big difference between our ‘at rest’ eye level.

Through this bug we discovered the need to implement; not only a calibration process for users, but crucially a calibrated setup in the office. We need to be sure that both devs and tester are experiencing the same thing. Seeing the same thing is not enough!

We need to know that a turn of the head will react the same at either dev or tester’s workstation. Now obviously it’s unlikely that the separate workstations can be setup *exactly* the same, but realising the issue is key here!

The realisation gives us the opportunity to learn more, and how to give the user the highest quality VR experience they can get!

First thoughts – Testing with Oculus Rift

When I put on the headset for the first time; the immediate brightness instantly triggered my ‘design alarm’. Bad contrast and overly bright interfaces are one of my bugbears. It became apparent that it was going to be even more of an issue inside the headset.

It may seem obvious that overly bright interfaces would be worse in VR, but if it’s that obvious, why does it still happen on websites?

I noticed that line weights were dramatically reduced when viewed in Oculus Rift, rather than a monitor. This issue does connect to contrast. If your copy is rendering much thinner in a headset; it’s going to be very difficult to contrast between the copy and the surround.

I’ve been known to fuss a lot about contrast issues, but that’s because I believe it’s very important.

I fully support these people.

There are huge numbers of people with sight problems; both diagnosed and undiagnosed. If you present content that requires concerted effort of the user to read it; then you are isolating a big percentage of your possible audience.

Now extend this idea to VR.

If you create a product that alienates a fair percentage of your audience; they don’t decide to use another VR system. You’re not running a website where you may lose them to a competitor.

Alienating someone means they will most likely be lost to the world of VR. When you’re trying to present ‘the next big thing’ you need each and every person to go ‘WOW’.

If you make one person ‘WOW’ then they tell others, obviously the converse is true. You don’t lose one person to VR if they have a bad experience, you potentially lose more.

Good testing isn’t simply about pointing out issues with a user’s experience. It’s easy to say something has bad contrast and could be hard to read. It’s harder to see the knock-on effects that issue can cause. That’s where good testing comes in. The ability to see the problem and *potential* problems created by it.


Software testing with Oculus Rift

So recently I’ve had the opportunity to work with a company developing a media player for VR; most notably Oculus Rift.

As a user; putting on the headset does throw you into another world. As a tester that world is even more different. Not only are you coming to grips with what you’re seeing, but at the same time you’re questioning it all. Breaking down what you’re seeing into its respective aspects.

Questioning what you need to think about; that you never have before?

How do you take all your experience and apply it to this situation?

One of the first questions that came to me:

What’s more comfortable; to look up towards content or to look down?

This may seem like a facile question, but it’s deceptively important. One of the main differences with using VR; is the user’s ability to look around the virtual environment. The virtual environment needs to be purposed so the user can do this in a comfortable way.

So you’re in the headset, you move your head up and down. You use different rates of motions and different ranges. Then you look down far enough that your chin hits your chest. Now that is not a nice feeling! The sudden contact with your own body does disturb the immersivity.

But it’s not as straightforward as looking up is nicer than down.

If the content as you look up; is also too close to you. It can give a claustrophobic experience. Now that might be perfect within a specific section of a game, but it’s the last experience you want a user to have; when selecting from a wall of video content, for example.

I am going to write more articles on various aspects of testing with Oculus Rift. I also welcome any other testers, or developers with an interest in testing to contact me. Let’s talk about our experiences and build new heuristics that can form a good foundation to help testing with VR develop.

Going freelance

So no-one ever tells you all the stuff you have to do at the start when going freelance!

You always hear the cool stories and how someone’s life/work balance is much better now etc… etc…

So it’s quite a lot to take in but I’m making the move, so if anyone does want to hire me then get in touch and we’ll see how I can help!

Check the about tab for contact details.

From Emile Durkheim to Max Weber – When context became key

In this article I want to talk about some integral figures within Sociology. People who helped to shape the discipline.

When studying Sociology and traversing the classic studies; Suicide by Emile Durkheim is usually the first studied. His aim was to study suicide across Europe. At the time no sociologist had attempted something on this scale. In his own words from the preface,

The progress of a science is proven by the progress toward solution of the problems it treats. It is said to be advancing when laws hitherto unknown are discovered, or when at least new facts are acquired modifying the formulation of these problems even though not furnishing a final solution. Unfortunately, there is good reason why sociology does not appear in this light, and this is because the problems it proposes are not usually clear-cut. It is still in the stage of system-building and philosophical syntheses. Instead of attempting to cast light on a limited portion of the social field, it prefers brilliant generalities reflecting all sorts of questions to definite treatment of anyone. Such a method may indeed momentarily satisfy public curiosity by offering it so-called illumination on all sorts of subjects, but it can achieve nothing objective. Brief studies and hasty intuitions are not enough for the discovery of the laws of so complex a reality.
(Durkheim, 1897)

Here we can see his laudable aims, but in the tail of the quote we see an issue. Durkheim supposes reality a complex place, but still thinks there are laws which can be discovered by Sociology. He is imagining being able to apply formulae like a physicist and gain the answers to our world. However, he also recognises how the word suicide may not always mean the same thing,

Our first task then must be to determine the order of facts to be studied under the name of suicides. Accordingly, we must inquire whether, among the different varieties of death, some have common qualities objective enough to be recognizable by all honest observers, specific enough not to be found elsewhere and also sufficiently kin to those commonly called suicides for us to retain the same term without breaking with common usage. If such are found, we shall combine under that name absolutely all the facts presenting these distinctive characteristics, regardless of whether the resulting class fails to include all cases ordinarily included under the name or includes others usually otherwise classified.
Durkheim questions how an act is defined and the agency of the person committing the act,
…in general, an act cannot be defined by the end sought by the actor, for an identical system of behaviour may be adjustable to too many different ends without altering its nature. Indeed, if the intention of self-destruction alone constituted suicide, the name suicide could not be given to facts which, despite apparent differences, are fundamentally identical with those always called suicide and which could not be otherwise described without discarding the term. The soldier facing certain death to save his regiment does not wish to die, and yet is he not as much the author of his own death as the manufacturer or merchant who kills himself to avoid bankruptcy?
Durkheim used whatever official sources he could to acquire the data. He then went onto present it in tables such as these,


and decides that,
The suicide-rate is therefore a factual order, unified and definite, as is shown by both its permanence and its variability. For this permanence would be inexplicable if it were not the result of a group of distinct characteristics, solidarity one with another, and simultaneously effective in spite of different attendant circumstances; and this variability proves the concrete and individual quality of these same characteristics, since they vary with the individual character of society itself.
Now whilst Durkheim has questioned the nature of the word suicide. He takes the collected statistics as fact. He sees the variance in statistics reflecting the attitudes of the society. What he doesn’t do is ask whether the attitude of the country initially decides when a death is a suicide.

If someone in a Catholic country found their friend dead with a suicide note; would they report that as suicide? Would they doom their friend’s body to burial in unconsecrated ground? How many suicides were counted as accidental deaths to give Italy the least amount of “suicides” in the above table? How many still happen today?

We can see that Durkheim is certainly questioning and exploring the nature of reality, but has blind spots which give Suicide its flaws.

If we move forward to Max Weber, the founder of the Interpretive school of thought. Weber introduced the idea of ‘Verstehen’
Verstehen is a German term that means to understand, perceive, know, and comprehend the nature and significance of a phenomenon. To grasp or comprehend the meaning intended or expressed by another. Weber used the term to refer to the social scientist’s attempt to understand both the intention and the context of human action.
(Elwell, 2005)
Weber decides that all societies have their own ‘norms’ and ‘values’. How something such as suicide was seen; would be determined by those norms and values.

From our earlier example; Italy’s norms and values see suicide as a mortal sin. These norms and values will therefore determine not only an individual’s likelihood to commit suicide, but also how likely that suicide will be recorded as such.

No respected sociologist today would ever attempt to analyse statistics without evaluating how the statistics were collected. Every sociologist today recognises that context is key to every investigation, even in terms of why someone decides to research a certain topic!

Context-driven sociology doesn’t exist because it doesn’t need to. How long until testing is the same?

Who said that?


Whisper and the supposed sources of knowledge.

First off let me apologise for the vast length between posts.

I’ve recently been using the app Whisper. I don’t want to talk about anything to do with it’s levels of anonymity. Instead, something much simpler.

Whisper is buggy as hell and has some of the worst UX traits I’ve seen.

Now that’s out the way we can proceed.

I seem to always bring work home with me and commonly have bug reports flying back and forth. Whisper is the same. I really like the concept of the app but it is executed poorly.

I’ve sent emails detailing the issues I’ve discovered. The first reply I received was a standard;

  • have you tried clearing data?
  • have you tried to reinstall the app?

This made it clear that my email had not been read. I’d detailed all my actions in making sure these were firm, repeatable issues. After more back and forth I informed Whisper that I was a software tester and suddenly my emails were escalated to the dev team.

This annoyed me!

The issues I’d found were being seen by all users. The users were using the app to bitch about the problems. This email could have come from any of them and been as valid.

People generally don’t report bugs.

In fact it’s the bane of the OSS community that more people don’t actively report bugs. A user that has taken the effort to email through a bug is worth listening to IMO.

There is the assumption; that those who are not trained appropriately can have no valid input. This seems to be less and less true but it is questionable whether it ever really was.

Within Sociology there have been occasions where the supposed ignorant have been capable of insight greater than the Sociologists involved. My favourite example of this is,

Learning to Labour by Paul Willis

This is a landmark study in Sociology for various reasons and I would highly recommend reading it.

In very brief summary; Paul Willis uses the participant observation technique to spend time with “troublesome” kids of a class in 1970s Britiain. The premise is to investigate the way in which there is little social mobility out of the working class.

There is an assumption that the kids are unaware of the processes at work which dictate their life-chances. Yet Willis discovers that the boys are all too aware of their own abilities and situation,

  • Eddie – The teachers think they’re high and mighty because they’re teachers, but they’re nobody really, they’re just ordinary people ain’t they?
  • (…)
  • PW – I mean you say they’re higher. Do you accept at all that they know better about things?
  • Joey – Yes, but that doesn’t rank them above us, just because they are slightly more intelligent.
  • Bill – They ought to treat us how they’d like us to treat them

There is even lamentation from the group when pondering their time in school,

  • Joey – ….something should have been done with us, I mean there was so much talent there that it was all fuckin’ wasted. I mean X, he was as thick as pigshit really, but if someone took him and tutored him…he’d got so much imagination.

Before this study; working class kids expressing these concepts seemed unlikely. Yet here they are, expressing how all are capable given the correct time and effort. This is evidence that those seen as stupid, clearly aren’t.

A user doing something stupid doesn’t equal a stupid user. Ignore your users at your own peril.