android cardboard app using camera and OpenGL ES

Some time ago I started to program Android apps for my smartphone. This year I got my hands on a cardboard. My first idea was to use the smartphones camera. Stereo vision with a single camera is a fake, but still it was tempting.

I was searching for examples of cardboard apps and stumbled across this example. For a start this was very useful. The example app shows how to create a stereo view and how to display a live camera image. The app is using Open GL ES which won’t work on all smartphones.

After having played around a little with the app I realized that if I want to change the live image only the fragment shader has to be replaced. This can happen  while clicking on the display (difficult when the smartphone is in the cardboard) or when using the magnet switch.

cardboardView.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
        switchShader();
    }
});


@Override
public void onCardboardTrigger() {
    vibrator.vibrate(50);
    switchShader();
}

Excerpt of switchShader():

private void switchShader() {
    shader_selection++;
    if (shader_selection > 31)
        shader_selection = 0;

    if (shader_selection == ORIG)
        overlayView.show3DToast("Shader: original. " +     shader_selection);
    else if (shader_selection == NEG)
        overlayView.show3DToast("Shader: negative. " +     shader_selection);
    ...
}

To display a negative image the fragment shader looks like this:

#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 textureCoordinate;
uniform samplerExternalOES s_texture;
void main() {
    vec4 color = texture2D( s_texture, textureCoordinate );
    float inverted = 1.0 - color.r;
    vec4 inverted_vec = vec4( vec3(inverted), 1.0);
    gl_FragColor = clamp(inverted_vec, 0.0, 1.0);
}

To invert RGB images:

#extension GL_OES_EGL_image_external : require
precision mediump float;
varying vec2 textureCoordinate;
uniform samplerExternalOES s_texture;
void main() {
	vec4 color = texture2D( s_texture, textureCoordinate );
    float invertedr = 1.0 - color.r;
    float invertedg = 1.0 - color.g;
    float invertedb = 1.0 - color.b;
    vec4 inverted_vec = vec4( vec3(invertedr, invertedg, invertedb), 1.0);
    gl_FragColor = clamp(inverted_vec, 0.0, 1.0);
}


A nice sepia effect:

#extension GL_OES_EGL_image_external : require
#ifdef GL_ES
precision mediump float;
#endif

varying vec2 textureCoordinate;
uniform samplerExternalOES s_texture;

void main()
{
    vec4 color = texture2D( s_texture, textureCoordinate );
    vec4 sepia = texture2D( s_texture, textureCoordinate );
    sepia.r = (color.r * .393) + (color.g *.769) + (color.b * .189);
    sepia.g = (color.r * .349) + (color.g *.686) + (color.b * .168);
    sepia.b = (color.r * .272) + (color.g *.534) + (color.b * .131);
    
    gl_FragColor = vec4(sepia.rgb, 1);
}

Several good examples for shaders which can be adapted I found here:

http://littlecheesecake.me/blog1/2013/08/15/pretty-geeky-codes.html
http://littlecheesecake.me/blog1/2013/04/19/more-filters.html
http://littlecheesecake.me/blog1/2013/01/31/image-processing.html

It is quite important to stop the camera when the app is paused. Otherwise other apps could not use the camera in the meantime.

    @Override
    public void onPause() {
        super.onPause();
        stopCamera();
    }

    private void stopCamera() {
        if( camera!=null ) {
            camera.stopPreview();
        }
    }

The result looks like this (computer mouse in negative mode):

androidcardboardnegativemouse

The smartphone gets pretty warm while using the app…

Advertisements