Establishing Communication Between MATLAB and Unity (TCP)

Objective: The goal is to perform trajectory planning in MATLAB and send the data back to Unity for visualization.To achieve this, we attempt to establish TCP communication, with MATLAB acting as the server and Unity as the client. For reference on the TCP protocol, see:

https://blog.csdn.net/crazymakercircle/article/details/114527369

1. Server Code:

.rtcContent { padding: 30px; } .lineNode {font-size: 10pt; font-family: Menlo, Monaco, Consolas, "Courier New", monospace; font-style: normal; font-weight: normal; }%% TCP Server function sendDebrisPositionsToUnity(positions)    % Create TCP server    tcpServer = tcpip('0.0.0.0', 12345, 'NetworkRole', 'server');    fopen(tcpServer);    fprintf('Waiting for Unity to connect...
');    % Wait for client connection    while ~tcpServer.Connected        pause(0.1);    end    fprintf('Unity has connected!
');    % Send number of debris    numDebris = size(positions, 1);    fprintf(tcpServer, '%d
', numDebris);    % Send coordinates (x,y,z) of each debris line by line    for i = 1:numDebris        fprintf(tcpServer, '%.6f,%.6f,%.6f
', positions(i,1), positions(i,2), positions(i,3));    end    % Close connection    fclose(tcpServer);    delete(tcpServer);    clear tcpServer;    fprintf('Coordinates sent successfully!
');end

Code Explanation:

tcpServer = tcpip('0.0.0.0', 12345, 'NetworkRole', 'server');

tcpip: A function in MATLAB used to create a TCP/IP communication object.

Parameter Explanation:

1. ‘0.0.0.0’: Indicates that the server listens on all available network interfaces on the local machine (allowing clients from any IP to connect).

2. 12345: The communication port number (must match the port used by the Unity client).

3. ‘NetworkRole’, ‘server’: Specifies that this object is a TCP server.

Function: Creates a TCP server object ready to listen for client connections.

while ~tcpServer.Connected

tcpServer.Connected: A property of the TCP object, true indicates connected, false indicates not connected.

Function: Blocks program execution until the Unity client connects to the server via port 12345.

2. Client Code

    // Establish connection to MATLAB    void ConnectToMatlab()    {        StartCoroutine(Connect());    }    IEnumerator Connect()    {        bool connectionSuccess = false;        float startTime = Time.time;        float timeout = 5f;        client = new TcpClient();        client.BeginConnect("127.0.0.1", 12345, ar => {            try            {                client.EndConnect(ar);                connectionSuccess = client.Connected;            }            catch (Exception ex)            {                Debug.LogError("Connection error: " + ex.Message);            }        }, null);        // Wait for connection to complete or timeout        while (!connectionSuccess && Time.time - startTime < timeout)        {            yield return null;        }        if (connectionSuccess)        {            stream = client.GetStream();            reader = new StreamReader(stream);            isReceiving = true;            StartCoroutine(ReceiveCoordinates());        }        else        {            Debug.LogError("Connection timed out");            yield return new WaitForSeconds(1f);            ConnectToMatlab();        }    }    IEnumerator ReceiveCoordinates()    {        // Read number of debris        string line = reader.ReadLine();        int numDebris = int.Parse(line);        Debug.Log($"Received {numDebris} debris coordinates");        // Loop to read each debris coordinate        for (int i = 0; i < numDebris; i++)        {            line = reader.ReadLine();            //Debug.Log(line);            if (line != null)            {                string[] coords = line.Split(',');                Debug.Log(coords[0]);                if (coords.Length == 3)                {                    float x = float.Parse(coords[0]);                    float y = float.Parse(coords[1]);                    float z = float.Parse(coords[2]);                    Debug.Log(x + ", " + y + ", " + z);                    // Generate debris                    //GenerateObjectAtPosition(x, y, z);                    Vector3 position = new Vector3(x, y, z);                    GameObject obj = null;                    obj = GameObject.CreatePrimitive(PrimitiveType.Cylinder);                    obj.transform.localScale = new Vector3(0.05f, 0.05f, 0.05f);                    if (obj != null)                    {                        obj.transform.position = position;                        obj.transform.SetParent(parentTransform);                        // Add random color                        MeshRenderer renderer = obj.GetComponent<MeshRenderer>();                        renderer.material.color = UnityEngine.Random.ColorHSV();                        Debug.Log($"Generated {shapeDropdown.options[shapeDropdown.value].text} at coordinates ({x}, {y}, {z})");                    }                }            }            yield return null;        }        // Receiving complete        isReceiving = false;        reader.Close();        stream.Close();        client.Close();        Debug.Log("Coordinates reception complete!");    }    //void CreateDebris(Vector3 position)    //{    //    // Instantiate debris prefab    //    GameObject debris = Instantiate(debrisPrefab, position, Quaternion.identity);    //    debris.transform.SetParent(debrisParent);    //    debris.name = $"Debris_{position.x:F2}_{position.y:F2}_{position.z:F2}";    //}    void OnDestroy()    {        if (isReceiving)        {            reader?.Close();            stream?.Close();            client?.Close();        }    }

Code Explanation:

void ConnectToMatlab(){    StartCoroutine(Connect()); // Start connection coroutine}

StartCoroutine(Connect()): A method in Unity to start a coroutine, where Connect() is a function returning IEnumerator, used for asynchronously executing connection logic (to avoid blocking the game frame).

        client.BeginConnect("127.0.0.1", 12345, ar => {            try            {                client.EndConnect(ar);                connectionSuccess = client.Connected;            }            catch (Exception ex)            {                Debug.LogError("Connection error: " + ex.Message);            }        }, null);

ar is a parameter of type IAsyncResult, which stands for “Asynchronous Result“. It is a key object in the .NET asynchronous programming model (APM) used to convey the status of asynchronous operations.

Specific Function:

When BeginConnect is called to initiate an asynchronous connection, the system performs the connection operation in the background, and upon completion, automatically calls the lambda expression you defined (i.e., ar => { … }). At this point, ar carries the result information of the asynchronous connection operation, mainly used for:

1. Obtaining the status of the asynchronous operation: For example, by checking ar.IsCompleted to determine if the operation is complete, ar.AsyncState to get additional state passed (here it is null, as the last parameter of BeginConnect is null).

2. Completing the asynchronous operation: You must call client.EndConnect(ar) to end the asynchronous operation; otherwise, it may lead to resource leaks. EndConnect confirms whether the connection was successful based on the information in ar and handles possible errors (such as throwing an exception if the connection fails).

Additionally:

1. 127.0.0.1: The local loopback IP, indicating a connection to the local MATLAB server (if MATLAB is on another device, replace it with the corresponding IP).

2. 12345: The port number, which must match the MATLAB server’s port (otherwise communication will fail).

The subsequent ReceiveCoordinates function frequently uses the Parse function to parse data:

1. First, you need to determine the data format from MATLAB, which is set up on the server side;

2. Then you can use the ReadLine function to read data, and Parse can convert various types of data, including int, float, etc.

Moreover, in this problem environment, if asynchronous operations are not used, what issues may arise:

1. Program freezing or hanging

The main thread of Unity is responsible for handling rendering, input response, UI updates, and other core logic. If synchronous network operations (such as TcpClient.Connect, StreamReader.ReadLine) are executed in the main thread, these operations will block the main thread:

For example, during a synchronous connection to the server, if there is network latency or the server does not respond, the main thread will wait for the connection to complete, causing the game screen to freeze and preventing input reception, leading users to feel that the program is “stuck”.

Similarly, during synchronous data reading, if data does not arrive in time, the main thread will block waiting, causing animations, UI, etc., to stop updating.

2. Difficulty in handling timeouts

In synchronous operations, if you want to implement timeout logic (such as retrying after a connection timeout), you need to manually poll the main thread, making the code complex and inefficient:

For example, without asynchronous operations, you might need to constantly check the connection status in the Update function, which occupies main thread resources and makes it difficult to control the timeout accurately.

Asynchronous operations (like the BeginConnect combined with coroutine waiting in the code) can naturally implement “waiting without blocking” timeout checks.

3. Impact on user experience

The smoothness of games or interactive programs relies on the efficient operation of the main thread. Synchronous blocking directly disrupts user experience:

For instance, if a player clicks “Connect to Server” and the program is unresponsive for several seconds, it may lead the user to mistakenly believe the program has crashed.

If a large number of coordinates are synchronously read during the data reception phase, blocking the main thread will cause the object generation process to stutter, visually resulting in a “jump” rather than a smooth generation.