Lesson 16: Resource Parsing

Resource Parsing: Importing the components of our world from a file

Now that we have a way for our program to render models easily and in multiple places in our OpenGL window, we can look at making this virtual world building process easier and more streamlined.

In this lesson, we'll look at how we can load an entire scene from a file, rather than have every model, texture, and entity hard-coded into our program.

This makes the process of rapidly prototyping different designs much easier, as once again we'll add the ability to hot-reload these scenes while our program is running. This also provides a convenient way in the future to have multiple scene objects in our program, should we wish to have more than one scenario or level that our program can run.

Designing a scene file

If we're going to parse all the objects in our world from a file, we should first take a moment to think about what the file will look like and what it needs to contain.

I'm going to concentrate on loading models, textures and entities from a file for now to keep things relatively straight-forward. Over the next few lessons we'll progressively add more things such as reading in the camera settings, but for now we'll just focus on these.

I'm also going to use a really simple text file format. While JSON or XML formatting might work quite nicely for this task and provide some level of protection against making obvious mistakes, I've opted not to use them here. There are a few reasons for this, including because we've already looked at some text file parsing before for model loading. I also don't want these lessons to push specific formats where possible, especially for things not related to graphics, and this means we avoid adding any non-core dependencies to our code. So, we'll just stick with a really trivial text file format for now.

There's absolutely no problem if you want to implement this lesson using one of those formats though, there's nothing in this lesson or future lessons which would make it problematic. I'd probably even encourage it, I just want that to be your decision!

The plan for storing all our texture, model, and entity data in a file therefore looks something like this:

#This is a comment line

Texture myTexture resources/myTexture.png

Model myModel resources/myModel.obj

Entity myEntity1 texture myTexture model myModel x 3 y 2 z 0 rx 0 ry 0 rz 0

Entity myEntity2 texture myTexture model myModel x 4 y 5 z 0 rx 0 ry 0 rz 0

Hopefully it's fairly intuitive. Each line of our scene file starts with the kind of object we're defining, followed it's name and it's parameters.

We can parse the file relatively easily by looping over each line and splitting each of them by spaces, just like we did to load Wavefront OBJ models. If the first token on the line matches "Model", "Texture" or "Entity", we'll create a new object of that type and then use the next token as the object's name. We can then attempt to parse more tokens from the line to read in the parameters of the object.

Any lines in the file where the first token isn't one of "Model", "Texture" or "Entity" we'll assume is just a comment, so we'll ignore the line entirely.

For handling entities, things are slightly more complicated. Currently in our code these use pointers to the relevant model and texture objects which they use for drawing. In our file format we instead reference the name of the texture or model to use for the entity. So in the example above, both entities will use the texture named myTexture and the model named myModel for drawing.

To implement this, we'll need to think about a few edge cases, but it shouldn't be too difficult. The main risks are that an entity references a model or texture which isn't defined in the file, or one which is defined later on in the file, but hasn't yet been parsed/created. A solution to both of these scenarios is that we can first parse the file and create all the textures and models, and then do a second pass this time just concentrating on the entities. Again this is similar to how we built our model loader, and means that when we're creating an entity, we can easily verify that the corresponding models and textures do exist and get a pointer to them.

That's basically it for our scene file. Not the most elegant file format ever invented, but it should be easy for us to work with, and quick for us to throw together a parser for. Again, I only use this to guide you on how it can be done, feel free to play with it and modify it as you like.

Scene class

To handle loading this file and managing it's contents, let's write a new "Scene" class. We'll declare it's structure in a new file called scene.h:

#pragma once

#include "texture.h"

#include "model.h"

#include "entity.h"

#include <map>

#include <string>

using namespace std;

class Scene

{

    public:

        void setFilename(string newSceneFilename);

        bool loadScene();

        void deleteScene();

        void draw();

        string getFilename();

        string getErrorMessage();

    private:

        map<string, Texture> textures;

        map<string, Model> models;

        map<string, Entity> entities;

        string filename;

        string errorMessage;

};

The class re-uses much of the same structure we've used before, so should be quite familiar.

The header file starts by including our texture, model and entity headers as we'll need access to those classes. We also include the map header from the standard library to give us an easy map/dictionary implementation to store each object we create from the file and to reference it by name. These are defined at the bottom of the class, the private map variables, which use a string as the key and the value is either a texture, model, or entity.

Once the scene has been loaded, the entities map will hold everything which needs to be drawn in the world, so the draw function is then simply a matter of iterating through the entire map and calling draw on each item.

Scene implementation

We can now write the code to define those functions in scene.cpp:

#include "scene.h"

#include <SDL3/SDL.h>

#include <fstream>

#include <sstream>

void Scene::setFilename(string newSceneFilename)

{

    filename = SDL_GetBasePath() + newSceneFilename;

}

We start by including the class's header file, the SDL header which we'll need for making file paths absolute, and the fstream and sstream libraries which we'll need for manipulating files and strings.

The first function we've written is setFilename. Like the rest of our classes, we'll accept a relative path from the executable to the file as a string. The function then prefixes the executable's location to make it an absolute path, and stores the result in the class's filename variable.

We can then move on to our loadScene function, which will do the parsing of the file:

    filename = SDL_GetBasePath() + newSceneFilename;

}

bool Scene::loadScene()

{

    errorMessage = "";

    deleteScene();

    if(filename.empty())

    {

        errorMessage = "Scene filename not set";

        return false;

    }

    ifstream fileStream(filename);

    if(!fileStream.is_open())

    {

        errorMessage = "Unable to open file";

        return false;

    }

    ...

The function returns a bool indicating success, where if we fail to parse and load the file, we'll return false and set the errorMessage variable to give an indication as to why loading failed.

The first thing we do is to reset the error message and call deleteScene, to make sure that if this function is called repeatedly (as happens when hot-reloading a scene), any old data is properly deleted before re-reading the file.

We also perform a check that a filename has been set before proceeding to try and open the file.

With the file open, we can start looping through it line-by-line:

        errorMessage = "Unable to open file";

        return false;

    }

    string line;

    while(getline(fileStream, line))

    {

        istringstream iss(line);

        string objectType, objectName, filename;

        iss >> objectType >> objectName >> filename;

        if(objectType == "Texture")

        {

            ...

        }

        else if(objectType == "Model")

        {

            ...

        }

    }

    ...

As we discussed, we'll parse the whole file twice, with the first run just focusing on just textures and models.

We use the getline function to read each line of the file into the line variable in a loop until we hit the end of the file. We start processing each line by first creating a stringstream from it to help us with the parsing.

The parsing starts by creating a set of string variables we'll use to hold the first three tokens of the line: the object type, it's name, and it's filename. We then use the stringstream to read the first three space-separated tokens of the line into these three variables. This is safe to do even if the line doesn't contain three tokens!

Then, we can test the value of objectType against the literal "Texture" or "Model" to decide if this is a line we're interested in, otherwise it gets ignored and we continue to the next line.

Let's now fill in the code block for handling a line defining a texture:

        if(objectType == "Texture")

        {

            textures[objectName].setFilename(filename);

            if(!textures[objectName].loadTexture())

            {

                errorMessage = "Failed to load texture ";

                errorMessage += objectName;

                errorMessage += " - ";

                errorMessage += textures[objectName].getError();

                deleteScene();

                fileStream.close();

                return false;

            }

        }

When we find a line starting with the literal "Texture", we make a call to the setFilename function of the texture in the map with that name. If the map doesn't contain an entry for the key objectName, which it won't at the start, the object will implicitly be created, and then the setting of the filename will be done.

With the texture created we make a call to the texture's loadTexture function to actually attempt to open the image and copy it to the GPU. If something failed while loading, we can use the texture's error message as the scene's error message. We also need to remember to delete anything from the half-loaded scene and close the file before returning in this case!

Otherwise that's all we need to do for parsing textures! Once this block of code has looped over each line in the file, our textures map will now contain all textures from the scene file, successfully loaded onto the GPU.

Next, we'll do the same for models. The code is largely the same:

        else if(objectType == "Model")

        {

            models[objectName].setFilename(filename);

            if(!models[objectName].loadOBJModel())

            {

                errorMessage = "Failed to load model ";

                errorMessage += objectName;

                errorMessage += " - ";

                errorMessage += models[objectName].getError();

                deleteScene();

                fileStream.close();

                return false;

            }

        }

    }

We again create a model in the corresponding map with the specified name, set it's filename, and then attempt to load it.

With this in place, our block of code looping through the file for the first pass is complete. When the code has finished running, we'll have all of our textures and models loaded on the GPU, ready to go.

We'll now go back to the start of the file and attempt to load any entities:

                return false;

            }

        }

    }

    fileStream.clear();

    fileStream.seekg(0);

    while(getline(fileStream, line))

    {

        istringstream iss(line);

        string objectType, objectName;

        iss >> objectType >> objectName;

        if(objectType == "Entity")

        {

            ...

        }

    }

    ...

We reset the file to start reading it from the beginning again, loop through it line-by-line, and this time look for lines beginning with "Entity".

Entities require a bit more effort to parse from our file than with the textures or models. Rather than the line simply being type, name, filename, they have a set of key/pair values on the line covering various parameters. As a result, we don't create the filename variable and parse three tokens as we did before.

Instead, we'll need something a bit more robust. Once we know we're looking at an entity definition line, we'll create a loop in our code which will continually attempt to parse two more tokens from the line, until there's nothing left on the line to read in. We can then treat of each of these pairs of tokens as a key/value pair for the entity's parameters. So if we see "x" as the first token, we'll expect a float for the second token of the pair which we can use as the entity's "x" coordinate.

To do this, we'll create a loop in our code like this:

        if(objectType == "Entity")

        {

            string key, value;

            iss >> key >> value;

            while(value.length())

            {

                ...

                key = "";

                value = "";

                iss >> key >> value;

            }

        }

    }

When we see an entity definition line in the file, we create a pair of strings and read the next two tokens from the line into them. We can then handle this key/value pair of parameters in the loop, and at the end blank the strings and attempt to read two more parameters from the line into them. This loop will keep running for each pair of tokens until we hit the end of the line.

Into this loop, we can then handle each kind of parameter we want to parse. First, textures:

            while(value.length())

            {

                if(key == "texture")

                {

                    if(!textures.count(value))

                    {

                        errorMessage = "Entity referenced texture which was not found: ";

                        errorMessage += objectName;

                        deleteScene();

                        fileStream.close();

                        return false;

                    }

                    entities[objectName].setTexture(&textures[value]);

                }

                ... 

If the first token of the pair is they key "texture", then the second token is the name of the texture object we should use.

As all textures in the file will have already been loaded into our textures map, configuring the texture for the new entity is quite easy. We first perform a check that a texture with that name exists somewhere in the map. Assuming it does, we can then set the entity's texture to the address of the texture in the map.

We can do the same for reading in the entity's model:

                    entities[objectName].setTexture(&textures[value]);

                }

                else if(key == "model")

                {

                    if(!models.count(value))

                    {

                        errorMessage = "Entity referenced model which was not found: ";

                        errorMessage += objectName;

                        deleteScene();

                        fileStream.close();

                        return false;

                    }

                    entities[objectName].setModel(&models[value]);

                }

                ...

Again checking a model with the specified name exists in the map, and setting it as the entity's model.

We can then finish the parameter parsing loop by adding some code to allow us to configure the position and orientation of the entity from the file:

                    entities[objectName].setModel(&models[value]);

                }

                else if(key == "x")

                {

                    entities[objectName].updatePosition(stof(value), 0, 0);

                }

                else if(key == "y")

                {

                    entities[objectName].updatePosition(0, stof(value), 0);

                }

                else if(key == "z")

                {

                    entities[objectName].updatePosition(0, 0, stof(value));

                }

                else if(key == "rx")

                {

                    entities[objectName].updateOrientation(stof(value), 0, 0);

                }

                else if(key == "ry")

                {

                    entities[objectName].updateOrientation(0, stof(value), 0);

                }

                else if(key == "rz")

                {

                    entities[objectName].updateOrientation(0, 0, stof(value));

                }

                key = "";

                value = "";

                iss >> key >> value;

            }

        }

    }

If we see a key/value pair where the key is "x", we make a call to updatePosition on the entity. The value is converted from a string to a float using the stof function. As the entity will default to a position of (0, 0, 0), updating it's position by this value will in effect set it's x coordinate to this value without changing the rest of the entity's coordinates.

The same is done if we see the keys "y" or "z", updating the entity's y or z value instead accordingly. Likewise for the orientation too, if we see a key value of "rx", we update the entity's rotation in the x axis, etc.

That completes the entity configuration loop, now giving us a way to create entities and configure their models, textures, positions and orientations from a file. This construction has the additional benefit of making us robust to the order of the parameters too, so it's perfectly fine for the file to contain the position of the entity first on the line, and then reference which model and texture it should use.

We can now add the finishing touches to our loadScene function:

                key = "";

                value = "";

                iss >> key >> value;

            }

        }

    }

    fileStream.close();

    return true;

}

After we've finished looping through the file looking for entities, we make sure the file gets closed before returning true to indicate success. All textures, models, and entities will now be properly configured and loaded, and stored in their corresponding maps.

With the parsing of the input file complete, we can move on to writing the rest of the class's functions. Next, let's write the deleteScene function to clear up the scene after we're finished with it:

    return true;

}

void Scene::deleteScene()

{

    for(auto& element : textures)

    {

        element.second.deleteTexture();

    }

    for(auto& element : models)

    {

        element.second.deleteModel();

    }

    models.clear();

    textures.clear();

    entities.clear();

}

For each texture, we need to call it's deleteTexture function to make sure that it's memory on the GPU is cleared up. We loop over the map, and for each element (a pair of it's name, and the texture object itself), we call the delete function on the second item, the texture object.

We then do exactly the same on the models, which will free up all the model's buffers on the GPU. Entity objects don't have any GPU or dynamically allocated memory themselves, so we don't need to do anything explicitly for them.

With all the memory freed, it's then just a matter of clearing/emptying each of the maps to return our scene object back to a default state.

Next, we can look at the draw function for our scene:

    entities.clear();

}

void Scene::draw()

{

    for(auto& element : entities)

    {

        element.second.draw();

    }

}

Drawing every entity in the scene is then simply a matter of looping over the entities map and calling draw on each object!

We can finish up the implementation of our scene class by defining the last few getter functions:

        element.second.draw();

    }

}

string Scene::getFilename()

{

    return filename;

}

string Scene::getErrorMessage()

{

    return errorMessage;

}

And with that, our scene object is complete!

Makefile

We've added a new source file, so let's now make sure it gets compiled:

CC = g++

OBJS = main.cpp shader.cpp texture.cpp model.cpp entity.cpp scene.cpp

INCLUDE_DIRS = -IC:\SDL3\include -IC:\SDL3_image\include -IC:\glm -IC:\glew\include

We need to make sure that scene.cpp is added to the list of files to be compiled in our Makefile.

Loading a scene

We can now integrate our scene class into our main.cpp. First, we need to include it's header:

#include "shader.h"

#include "texture.h"

#include "model.h"

#include "entity.h"

#include "scene.h"

#include <SDL3/SDL.h>

#include <SDL3/SDL_main.h>

#include <SDL3_image/SDL_image.h>

Then, we can replace all of our models, textures, and entities we were previously using with a single scene variable:

Shader mainShader;

Scene crateScene;

bool init()

{

All of those variables are removed entirely, we won't need them at all any more. We've replaced them instead with a scene object called crateScene.

We can then do exactly the same in our init function:

    mainShader.setFilenames("shaders/main_vertex.glsl", "shaders/main_fragment.glsl");

    if(!mainShader.loadShader())

    {

        printf("Unable to create shader from files: %s\n", mainShader.getFilenames().c_str());

        printf("Error message: %s\n", mainShader.getError().c_str());

        return false;

    }

    crateScene.setFilename("resources/crates.scene");

    if(!crateScene.loadScene())

    {

        printf("Unable to load scene from file: %s\n", crateScene.getFilename().c_str());

        printf("Error message: %s\n", crateScene.getErrorMessage().c_str());

        return false;

    }

    SDL_SetWindowRelativeMouseMode(window, true);

We rip out all the old code for manually creating textures, models, and entities, and instead replace it by just setting the path to our scene and attempting to load it.

The same is then done to our closing code for when the program has finished:

void close()

{

    crateScene.deleteScene();

    mainShader.deleteShader();

    SDL_GL_DestroyContext(context);

    SDL_DestroyWindow(window);

    SDL_Quit();

}

We strip out all the previous code for deleting our textures, models, and entities, and instead just call delete on the scene object.

Our draw function gets the same treatment:

    mainShader.bind();

    glUniformMatrix4fv(0, 1, GL_FALSE, glm::value_ptr(pMatrix));

    glUniformMatrix4fv(1, 1, GL_FALSE, glm::value_ptr(vMatrix));

    crateScene.draw();

    mainShader.unbind();

    SDL_GL_SwapWindow(window);

}

No individual draw calls are made any more. Instead, drawing involves binding the shader, setting the perspective and view matrices, and then calling draw on our scene, which will handle all the individual objects.

Finally, we'll update our handleEvents too. We strip out all the old variables and alter our code so that if the user presses "r" on their keyboard, we just hot-reload the scene rather than everything individually:

            else if(event.key.key == SDLK_R)

            {

                if(!mainShader.loadShader())

                {

                    printf("Unable to create shader from files: %s\n", mainShader.getFilenames().c_str());

                    printf("Error message: %s\n", mainShader.getError().c_str());

                    programRunning = false;

                }

                if(!crateScene.loadScene())

                {

                    printf("Unable to create scene from file: %s\n", crateScene.getFilename().c_str());

                    programRunning = false;

                }

            }

The call to loadScene will already have the scene's filename cached, and the first thing a call to loadScene will do is safely delete the old scene. So this call will throw away everything we had and then attempt to reload everything while the program's running.

Bear in mind that for complex scenes, as this is all done on the main thread currently, this may cause the program to stall for a moment until the loading process has finished. For me that's fine as I see this more as a useful debugging tool. If you wanted to have dynamic scene loading as an integral part of your program though then maybe you would show a loading screen to the user while it's happening, or push the call to another thread, rather than have the program appear to freeze momentarily.

With that, our code is now complete!

Recreating our crates

To test our new scene system is working, let's use it to recreate the crates from the previous lessons. To do that, we can write a scene file in the following way:

Texture CrateTexture resources/crate/diffuse.png

Model CrateModel resources/crate/crate.obj

Entity crate1 texture CrateTexture model CrateModel x 6 y 0.46 z 0 rx 0 ry 0 rz 5

Entity crate2 texture CrateTexture model CrateModel x 6 y -0.46 z 0 rx 0 ry 0 rz 83

Entity crate3 texture CrateTexture model CrateModel x 6.03 y 0 z 0.7 rx 0 ry 0 rz -2

We start by defining a texture and model for the crate, and the filename where each of these resources can be found.

We've then added 3 entities using these models and textures, and position them as they were before. If we run our program with this file, we should end up with exactly what we had in the previous lesson (just without all the hard-coding global variables!).

Summary

We now have a way to create a scene with tens, hundreds or even thousands of entities in our program without having to pollute our source code with pages of variables!

We also have a way to hot-reload those scenes if we need to, and even load several scenes into our program if want to switch between multiple scenes or levels.

Our virtual world is well on it's way!

In the next lesson we'll take this a step further, and see how we can build a fully-fledged virtual world into our program. See you there!