Projector calibration

In order to project a 3d scene on a real life environment it is necessary to match the virtual camera position with the one of the projector.

To do this there are many existing techniques using cameras and computer vision methods.
However, we developed a method that doesn't require the use of a camera.

The goal of this method is to obtain the position and orientation of the projector using a center of coordinates relative to the scene.
This problem is similar to finding the position and orientation of the chosen center of coordinates relative to the projector.

This method uses three known perpendicular points at the corners of the calibration surface that are aligned with the target coordinate system. Hence the distances between those points are known.

The projector casts rays of light and three of these rays pass through the chosen points and also through the center of the original coordinates system (inside the projector).
To obtain the equation of these rays only one point different from the origin is needed.
This point can be obtained from the screen coordinates of the pixel that is projected on each target point, for example, by clicking on them.

Using the screen coordinates of the point and some known projector parameters like the resolution and the projection angle it is possible to find the 3D coordinates of the point and then the equation of the ray.

Now that the equations for the three rays are known, we need to find the parameter t where the ray meets the corner of the target area. This can be written as a system of equations using the ray formula and the dimensions of the target area.

a, b, c are the rays. j, k, l are the width, height and diagonal of the surface

The goal is to find the parameters t0, t1 and t2 that satisfy the non-linear system.
An approximation for this solution can be obtained using numerical methods. At the moment our implementation uses the trust-region-dogleg method from Matlab.

Once the values for t0, t1 and t2 are known, the coordinates for the corners of the target area are a(t0), b(t1) and c(t2). The base for the target coordinate system is

The location of the projector relative to this base of coordinates is obtained by projecting a(t0) on this base.

Stereo-Vision

Structure from Stereo- A Review

Stereopsis is a passive technique, the triangulation needs to be achived with the help of only the existing ambient illumination.
Hence a correspondence needs to be established between features from two images that correspond to some physical feature in space.
Then, provided the position of centers of projection, the effective focal length, the orientation of the optical axis, and the sampling interval of each camera are known, the depth can be reconstructed using triangulation.

The principal steps are: Preprocesing, establishing correspondence and recovering depth.

Preprocesing: in this step image locations satisfying certain well-defined feature characteristics are identified in each image.

Establishing correspondence - Matching: Given two or more views of a scene, correspondence needs to be established among homologous features(features that are projections of the same physical identity in each view).

Recovering depth: using trigonometric functions to obtain the depth.

They present diferents techniques for recovering the 3-D structure of a scene from analysis of stereo images.

A problem in Stereo is resolve the correspondence problem due to occlusion, most scenes contain regions which appear in only one of the two images. We call these regions half-occluded, or unmatched.

Aplication of stereo: cartography, aircraft navigation, autonomus land rovers, robotics, industrial automation and steromicroscopy.

VDMX

Realtime Video Studio for Professional VJs.

VDMX5 is a program that lets you assemble custom realtime video processing applications.

- Layers are the basic building block of VDMX. Movies, pictures and Quartz Composer documents could be displayed on layers.
The layers are composited with each other to obtain the resultant image.
- Modular Architecture, building by loading plugins from an ever-expanding list of available add-ons. Any setup can be saved and instantly restored, making it possible to switch between sets on the fly.

As Modul8 the mapping is realised manualy, see an example.

VDMX is a proprietary software.

Modul8

One of the most popular software for VJs

Modul8 is a MacOS X application designed for real time video mixing and compositing. It has been designed for VJs and live performers.

Features:
Real Time Video Mixer and a user interface for real time- Is thought for real time video.
Is based on a layer metaphor, each change can be seen immediately in the composition.

Each media is a layer that can be moved, resizaed an rotated.

Has 7 outputs plus one for the user interface, you can determine what region of the composition is sent to which projector or screen.

Is posible extend the system incorporating new modules, writing in Python script.
Is an accessible on-line library more of them free.
Modul8 is a proprietary software.

Differents structured light techniques

Structured light is the process of projecting a known pattern of pixels on to a scene, the displacement of the stripes allows for an exact retrieval of the 3D coordinates of any details on the object's surface (depth and surface information).

A structured-light 3D scanner is a device for measuring the three-dimensional shape of an object using projected light patterns and a camera system.

This paper presents a comprehensive survey on coded structured light techniques.
The patterns are specially designed so that codewords are assigned to a set of pixels. Every coded pixel has its own codeword, so there is a direct mapping from the codewords to the corresponding coordinates of the pixel in the pattern.The codewords are simply numbers, which are mapped in the pattern by using grey levels, color or geometrical representations.

Shows pattern projection techniques classified in three groups according to their coding strategy:

• time-multiplexing - generate the codewords by projecting a sequence of patterns along time, so the structure of every pattern can be very simple

• neighborhood codification - represents the codewords in a unique pattern

• direct codification - define a codeword for every pixel, which is equal to its grey level or color

Time-multiplexing strategy
A set of patterns are projected onto the measuring surface, the codeword for a given pixel is formed by the sequence of illuminance values for that pixel across the projected patterns. The codification is called temporal because the bits of the codewords are multiplexed in time.
This kind of pattern can achieve high accuracy in the measurements. This is due to two factors:
1- since multiple patterns are projected, the codeword basis tends to be small (usually binary) and therefore a small set of primitives is used, being easily distinguishable among each other;
2- a coarse-to-fine paradigm is followed, since the position of a pixel is being encoded more precisely while the patterns are successively projected.

They clasify the different techniques based on time-multiplexing as:
a) techniques based on binary codes: a sequence of binary patterns is used in order to generate binary codewords
b) techniques based on n-ary codes: a basis of n primitives is used to generate the codewords
c) Gray code combined with phase shifting: the same pattern is projected several times, shifting it in a certain direction in order to increase resolution
d) hybrid techniques: combination of time-multiplexing and neighborhood strategies

Spatial neighborhood
Tend to concentrate all the coding scheme in a unique pattern
clasification of techniques:
a) strategies based on non-formal codification: the neighborhoods are generated intuitively
b) strategies based on De Bruijn sequences: the neighborhoods are defined using pseudorandom sequences
c) strategies based on M-arrays: extension of the pseudorandom theory to the 2-D case

Direct codification
The entire codeword for a given point is contained in a unique pixel. In order to achieve this, it is necessary to use either a large range of color values or introduce periodicity.In theory, a high resolution of 3D information can be obtained.The sensitivity to noise is very high because the "distance" between "codewords", i.e. the colors used, is nearly zero.Moreover, the imaged colors depend not only on the projected colors, but also on the intrinsic color of the measuring surface.

They discuss of two groups of methods:
a) codification based on grey levels: a spectrum of grey levels is used to encode the points of the pattern
b) codification based on color: these techniques take advantage of a large spectrum of colors.

After clasification they implement some methods, compare them and present their resuls.

Structured light - Continuity vs Discontinuity

A third experiment with structured light was performed trying to scan a simple and very used scene in the video-mapping field: a group of primitive shapes.
We faced a lot of problems with this apparently simple scene and that made us think that structured light wasnt suitable for discontinuous shapes, so we built a complex but continuous geometry by wrapping the scene with a piece of cloth. The SL-applet had no trouble generating the 3d geometry.

After getting these results we asked the creator of the applet for confirmation on our early conclusion and we got a response. It was true that the experiment performed wasnt suitable for discontinuous shapes because it uses phase-shifting scanning based on the principle of propagating depth values across a surface. So if two surfaces are disconnected, it cannot determine how they are related depth-wise.
However, it wasn't true that SL is inappropriate for discontinuous shapes. Other pattern codifications and algorithms not so based on real-time can return better results for these type of scenes.
This is what we'll be working on the next weeks.

Photos of the scenes - discontinuous and continuous

Second scanning

On this second experiment with structured light we scanned a simple scene: two perpendicular walls.
A lot of video mapping applications like this one focus on objects with planar faces like boxes so the idea of this experiment was to test structured light with these type of objects.
The geometry was fairly well retrieved with minor tweakings. The pictures were only cropped around the target zone and the Z scale and Z skew were adjusted in the SL applet.
Here are the pictures taken and a snapshot of the retrieved cloud of points.