Transferred from: cocos official.

How does Cocos Creator 3.0 convert world coordinates to screen coordinates? How does Creator 3D convert 3D coordinates to Canvas? How can the screen coordinates of the touch be converted to world coordinates? How to convert node coordinates under Canvas into 3D scene coordinates?

Discussion posts related to "coordinate transformation" can be seen from time to time in cocos Chinese community. Whether it is the 3D version of Cocos creator or the 3.0 version after the integration of 2D and 3D products, a large number of developers are troubled by this operation. Today, we will use an article to make it easy for you to understand the principle of coordinate transformation.

Screen coordinates

When Cocos Creator 3.0 game is running, the displayed canvas size is the screen area, and the screen coordinates are calculated from the lower left corner of the canvas as the origin, which can be accessed through view Getcanvassize interface gets the screen size.

Knowing the screen starting point and screen size, you can roughly estimate the contact position value when clicking the screen. Touching an event can just help verify. Listen through systemEvent:

 // Monitor system touch movement events
    onEnable () {
        systemEvent.on(SystemEventType.TOUCH_MOVE, this._touchMove, this);
    }

    _touchMove(touch: Touch, event: EventTouch){
        // Print the acquired screen contact position
        console.log(touch.getLocation());
    }

Click and move to see the direction of the coordinate value. The more the contact goes to the right, the greater the x value, the higher the x value, and the greater the y value.

In Cocos Creator 3.0, the size of the 3D camera is always the same as the screen size without setting the viewport, which is very important because the following content is inseparable from the camera.

UI contact coordinates

The biggest feature of UI is adaptation and interaction. When viewing UI documents, you may get several contents that are most directly related to adaptation, such as design resolution, Layout component, alignment component, multi-resolution adaptation scheme, etc. Among them, the design resolution and multi-resolution adaptation scheme are directly related to the coordinates, which are mentioned in the official document multi-resolution adaptation scheme. Here, I will quickly take you to understand how the adaptation rules affect the presentation of UI content according to this knowledge.

UI multi-resolution adaptation scheme

During UI production, the size of UI design is usually determined first, which is called design resolution. The design resolution and adaptation mode determine the runtime UI content size (the size of the UI root node Canvas). The condition that must be clearly known here is that the design resolution and device resolution (screen resolution) must be fixed. Under this condition, learn about the two modes suitable for most game development, namely fit width and fit height.

Let's use an example to illustrate. Suppose the design resolution is 960x640, there is a background on the UI that is the same size as the design resolution (no Widget is added) and four sprites that are always close to the four corners of the UI content area (all four sprites have Widget components, which are up, down, left and right close to the four corners of the UI content area).

When the selected adaptation scheme is the adaptation height and the screen resolution is 1334x750, the engine will automatically support the height of the design resolution to the screen height, that is, enlarge the scene image to 750 / 640 = 1.1718 times, but the enlarged width 9601.1718 = 1125 is still less than the screen width 1334, so it is easy to appear black edges (as shown in the figure below, the background is not covered with the screen). This method is certainly something that most game developers don't want to see. Therefore, the engine will automatically "spread" the width of the screen according to the Widget component calculation (as shown in the four sprites in the figure below).

Of course, this method also has disadvantages, that is, if the background fully adapts to the design resolution (the adaptation component Widget is added, and the UI content area is adapted up, down, left and right at the same time), there will be a certain stretch. Here, because the stretching ratio is small, it is not obvious. Interested friends can verify it by themselves.

Conversely, if the screen resolution is 960x720, the scene image is enlarged to 720 / 640 = 1.125, and the enlarged width is 1.125960 = 1080, which is greater than the screen width 960, the phenomenon of content cutting will occur (as shown in the background below). Therefore, the engine will automatically "limit" the width in the screen according to the Widget component calculation (as shown in the four sprites in the figure below). The disadvantage of this method is that the background will be squeezed if it fully adapts to the design resolution.

The same principle applies to the adaptive width. After understanding the adaptation principle, you can select the appropriate adaptation scheme according to different platforms.

UI contact acquisition

On the UI, you should be most concerned about how to set UI elements according to the contact position.

For Creator 2D users, the method to obtain contact information is to use event.com in the event listening callback Getlocation () gets the contact information. The contact information here is calculated by the screen contact according to the UI content area; In Creator 3.0, the screen and UI are completely separated. Users can click the screen to obtain contact information without UI. Therefore, the screen contact is obtained through event getLocation(). If you want to obtain the "screen contact" coordinates in the same way as Creator 2D, you can use event Getuilocation(). event. The contact information obtained by getuilocation () can also be used to directly set the world coordinates of UI elements (because each pixel on the UI is equivalent to each unit on 3D, the world coordinates of UI elements can be directly set according to this coordinate point).

const location = touch.getLocation();
console.log(`screen touch pos: ${location}`);
const locationUI = touch.getUILocation();
console.log(`UI touch pos: ${locationUI}`);

const pos = new Vec3(locationUI.x, locationUI.y, 0);
// sprite here is a reference to the white picture on the screen
this.sprite.setWorldPosition(pos);

Of course, students who have a clear rendering principle can also deal with it directly through screen coordinates and UI camera. For the specific principle, please refer to the mutual rotation of screen coordinates and 3D Node world coordinates below.

// This script is mounted on the Canvas node
 // Get screen coordinates
 const location = touch.getLocation();
 const screenPos = new Vec3(location.x, location.y, 0);
 const pos = new Vec3();
 const canvas = this.getComponent(Canvas)!;
 // Gets the camera data of the Canvas associated camera
 const uiCamera= canvas.cameraComponent?.camera!;
// Use the camera data object to convert the screen points into values in world coordinates
uiCamera.screenToWorld(pos, screenPos);
this.sprite.setWorldPosition(pos);

Note: This is because UI There is no depth information, so you can set the world coordinates directly z The value is fixed to 0. If it is 3 D Node, also consider the depth.
In expected 3.4 And later versions will be converted by camera, so you don't need to understand UI The concept of contact will also provide corresponding conversion methods later. Please know the article.

Conversion between different coordinates

After knowing the screen coordinates, UI coordinates and camera, you can do more coordinate conversion. First, explain the relationship between local coordinates and world coordinates:

The center point of the world coordinate (also known as the origin) is created by the node in the scene. It is located at the outermost layer of the node tree, and the position of the node whose position attribute value is all 0 is the center point of the world coordinate. All nodes pass through node The values obtained by the getworldposition interface are all offset values for the position, which can also be understood as absolute difference. The local coordinate is the offset relative to the parent node. Assuming that the world coordinates of node A are (0, 10, 0), node B is the direct child node of node A, the local coordinates of node B are (0, 10, 0), and neither node A nor node B rotates and scales, what are the world coordinates of node B?

The answer is obvious (0, 20, 0). The coordinate of node B is the local coordinate, which is the offset relative to the parent node, that is, B is offset by 10 relative to A in the y value, and the y value of A's world coordinate is 10, that is, it is offset by 10 relative to the y value of the world origin, then it can be calculated that B is offset by 10 + 10 = 20 relative to the y value of the world origin.

Screen coordinates and 3D Node world coordinates rotate with each other

Before converting the position of 3D nodes to screen coordinates, we should first understand how 3D objects are finally rendered to the screen.

First of all, all objects need to be illuminated by the camera before they can be displayed. Therefore, the objects need to be in the sight distance frame of the camera. Here, we need to do an operation from world space to camera space. Secondly, depth detection and other operations need to be done in the camera space. Finally, the spatial content is projected onto the screen. How to understand this part? See the figure below:

Here, take the perspective camera as an example. This is the top view of the perspective camera. The near plane value of the perspective camera is 1 and the far plane value is 1000. The visual range of the camera is from the near plane to the far plane. It can be seen from the figure that the closer it is to the near plane, that is, the smaller the depth value, the smaller the cross-sectional area, and vice versa. When cross sections of different areas are finally covered and drawn on the same canvas, the effect of near large and far small will appear. At the same time, depth detection will also be carried out between different sections to realize that objects with near field of vision block objects with far field of vision. The final presentation is a bit similar to the process of taking photos with a camera, which "flattens" all three-dimensional things. Is it easy to know how objects are drawn on the screen and vice versa?

It's not that easy. Because flat things become three-dimensional and need to go through a lot of detailed processing. The most direct point to deal with here is deep restore. The original depth of the screen is determined according to the presentation of the screen. Therefore, the depth can be specified when changing the screen contact to the world coordinates of 3D nodes. As shown in the figure above, the range of depth is between the near plane and the far plane, i.e. 1 - 1000. The normalized value of 0 - 1 is used numerically. By default, if the depth is not specified, the converted position of the object is in the near plane position. If it is 0.5, it is about 500 away from the camera.

Next, through the following code, you can easily realize the mutual conversion between screen coordinates and 3D Node world coordinates:

 // 3D camera reference
 @property(Camera)
 camera: Camera = null!;
 
 @property(MeshRenderer)
 meshRenderer: MeshRenderer= null!;
 
 onEnable () {
     systemEvent.on(SystemEventType.TOUCH_START, this._touchStart, this);
}

_touchStart (touch: Touch, event: EventTouch) {
     // Get camera data in 3D camera
    const camera = this.camera.camera;
    const pos = new Vec3();

    // 1. 3D Node world coordinates to screen coordinates
    const wpos = this.meshRenderer.node.worldPosition;
    // Convert the world coordinates of 3D nodes to screen coordinates
    camera.worldToScreen(pos, wpos);

    // 2. Screen coordinates to 3D Node world coordinates
    // Gets the coordinates of the current contact on the screen
    const location = touch.getLocation();
    // Note the z value here. The distance from the near plane to the far plane is normalized to a value between 0-1.
    // If the value is 0.5, the converted world coordinate value is at the position of the central tangent plane from the near plane to the far plane of the camera
    const screenPos = new Vec3(location.x, location.y, 0.5);
    camera.screenToWorld(pos, screenPos);
}

For ease of observation, the far plane of the camera is adjusted to 20 here.

Coordinate conversion between 3D nodes

// Convert 3D Node nodeB local coordinates to 3D Node nodeA local coordinates
const out = new Vec3();
const tempMat4 = new Mat4();
const nodeAWorldMat4 = nodeA.getWorldMatrix();
Mat4.invert(tempMat4, nodeAWorldMat4);
Vec3.transformMat4(out, nodeB.worldPosition, tempMat4);

Screen coordinates to UI contact coordinates

It should be made clear here that the UI contact coordinates are the values calculated by the screen contact according to the UI content area, that is, the UI world coordinates. The concept itself is not easy to understand and complicated. Therefore, the engine group will improve this conversion method in later versions and deal with it directly through the camera. However, versions before 3.4 still need to be used, so here is a simple summary:

Execute event in the callback triggered by the listening node event The value obtained by getuilocation is the contact information. It is the value calculated by the screen contact according to the UI content area. It can be directly used to set the world coordinates of the UI node. If you need to click the screen to set the UI node for design requirements, you can directly use this method.

// An highlighted block
const locationUI = touch.getUILocation();
const pos = new Vec3(locationUI.x, locationUI.y, 0);
this.sprite.setWorldPosition(pos);

Coordinate conversion between different UI nodes

Unlike Creator 2D, the size and anchor information of UI nodes are no longer on the node, but each UI node will hold a UITransform component. Therefore, the transformation API s between UI nodes are on the component. After obtaining the world coordinates of the UI node after the screen contact conversion, you can convert to the local coordinates of different UI nodes according to the coordinates.

● screen contact coordinates are converted to UI node local coordinates

// Suppose there are two nodes here, nodeA and nodeB of its bytes. Click the screen to set the coordinates of nodeB

// Screen contacts are calculated from the UI content area
const locationUI = touch.getUILocation();
const uiSpaceWorldPos = new Vec3(locationUI.x, locationUI.y, 0);
const nodeAUITrans = nodeA.getComponent(UITransform)!;
// Convert to the local coordinates under the nodeA node (that is, the value relative to nodeA)
nodeAUITrans.convertToNodeSpaceAR(uiSpaceWorldPos, pos);
nodeB.position = pos;

● coordinate rotation between UI nodes

 // Suppose there are two nodes here, nodeA and nodeB, which are neither brothers nor parent-child relationship
 const nodeBUITrans = nodeB.getComponent(UITransform)!;
 // The final offset value is stored on pos
 const pos = new Vec3();
 // Gets the offset of nodeA relative to nodeB
 nodeBUITrans.convertToNodeSpaceAR(nodeA.worldPosition, pos);
 nodeA.parent = nodeB;
 nodeA.position = pos;
 
// If you want to take a point with a certain offset from itself for conversion, you can use the following methods:
// Offset 10 units from nodeA x axis
const offset = new Vec3(10, 0, 0);
const nodeAUITrans = nodeA.getComponent(UITransform)!;
// Converts the offset value to world coordinates
nodeAUITrans.convertToWorldSpaceAR(offset, pos);
// Obtain a point offset by 10 units on the x-axis relative to nodeA and convert to an offset relative to nodeB
nodeBUITrans.convertToNodeSpaceAR(pos, pos);
nodeA.parent = nodeB;
nodeA.position = pos;

3D Node world coordinates to UI node local coordinates

The above content has laid a lot of groundwork. Here, the code is directly applied. The principle is to convert the world coordinates of the 3D node to the screen coordinates, and then to the local coordinates of the UI node.

 // Suppose the 3D Node cube, UI node spriteA,
 const out = new Vec3();
 const wpos = cube.worldPosition;
 // Directly call the conversion interface of the camera to complete the whole operation
 // The value obtained here is the value offset from spriteA
 camera.convertToUINode(wpos, spriteA, out);
 const node = new Node();
 // The local coordinate is the offset relative to the parent node. Therefore, it is necessary to specify the converted parent node, otherwise the conversion will always be incorrect
 node.parent = spriteA;
 node.position = out;

Radiographic testing

Before we begin to talk about how to use radiographic testing, we should first talk about why there is radiographic testing. When many Creator 2D students develop games, they know that there is a "Size" attribute on the node, which represents the Size of the node. When clicking on the screen, in most cases, the engine calculates whether the node is clicked through the position and "Size" of the contact and node. The listening method uses node events to trigger a callback when the node is clicked. The code is as follows:

this.node.on(Node.EventType.TOUCH_MOVE, this._touchMove, this);

However, it is not feasible to use this method on Creator 3.0 3D nodes, because 3D nodes do not have the so-called "Size" attribute, and the Size of objects is different at different depths. Therefore, whether an object is clicked or not cannot be calculated simply by position and "Size". Here you need to use the ray detection function.

Ray detection simply means that you can specify A starting point A and an ending point B. the engine will emit A ray from A to B, collect and return the respective positions, normals and other information of all objects that collide with the ray, so that you can know who the clicked object is and how to deal with them respectively. Cocos Creator 3.0 provides three ways to create rays:

 import { geometry } from 'cc';
 const { ray } = geometry;
 
 // Method 1 By starting point + direction
 
 // Construct a ray starting from (0, - 1, 0) and pointing to the Y axis
 // The first three parameters are the starting point and the last three parameters are the direction
 const outRay = new ray(0, -1, 0, 0, 1, 0);
 // perhaps
const outRay2 = ray.create(0, -1, 0, 0, 1, 0);

// Method 2 Pass through the starting point + another point on the ray

// Construct a ray from the origin pointing to the Z axis
const outRay = new ray();
geometry.ray.fromPoints(outRay, Vec3.ZERO, Vec3.UNIT_Z);

// Method 3 Use the camera to construct a ray emitted from the camera origin to a point on the screen

// It is assumed that the previous camera has been associated here
const cameraCom: Camera;
const outRay = new ray();
// Obtain a ray emitted from a path screen coordinate (0, 0)
// The first two parameters are screen coordinates
cameraCom.screenPointToRay(0, 0, outRay);

From the above ray creation method, if you want to click the screen to determine whether an object is clicked, method 3 is used. Careful friends may want to ask, why do you need a camera to judge whether you click a 3D object on the screen? Next, I illuminate the same capsule with two cameras in the scene, and I can probably explain the relationship.

At this time, you will clearly find that even if there is only one capsule in the game scene, if you irradiate it with two cameras with different shooting angles, you will see two objects A on the canvas. At this time, click area 1, whether the capsule has been clicked? Click area 2, whether the capsule has been clicked? It's impossible to judge.

If you specify the camera to judge, it is much clearer. Judging from the perspective of camera A, click area 1 and no capsule is clicked; Judging from the perspective of camera B, click area 1 and click the capsule. It can be verified by the following code:

import { _decorator, Component, systemEvent, SystemEventType, Touch, EventTouch, Camera, geometry, MeshRenderer } from 'cc';
 onst { ccclass, property } = _decorator;
 onst { Ray, intersect } = geometry;
 onst { rayModel } = intersect;
 
 @ccclass('TestPrint')
 export class TestPrint extends Component {
     // Specify camera
     @property(Camera)
    camera: Camera = null!;

    // Specifies the rendering components of the model
    @property(MeshRenderer)
    meshRenderer: MeshRenderer= null!;

    onEnable () {
       systemEvent.on(SystemEventType.TOUCH_START, this._touchStart, this)
    }

    _touchStart(touch: Touch, event: EventTouch){
        const location = touch.getLocation();
        const ray = new Ray();
        // Creates a ray that connects the contact position to the camera position
        this.camera.screenPointToRay(location.x, location.y, ray);
        // Obtain the model used to store rendering data on the rendering component for ray detection
        const raycast = rayModel(ray, this.meshRenderer.model!);
        // The return value is only 0 and! 0, 0 is not detected
        if(raycast > 0){
            console.log('capsule clicked');
        }
    }    
}

Note: if the contents illuminated by two cameras need to be rendered on the canvas, the ClearFlags of one camera needs to be DEPTH_ONLY.