Precompiled binaries in biicode: a proof of concept with SFML

The problem: C and C++ compilation times

Biicode is a file-based dependencies manager for C and C++, focused on sharing and reusing source code, specifically, source (and header) files.

Biicode uses the CMake building system to configure and build blocks, its unit of source code sharing. The default way to develop blocks is to include the required sources and any required extra configuration for building such files on a CMakeLists.txt file at the root of the block. Also biicode provides other files for specific config such as dependencies.bii or paths.bii.

So writting our own biicode block is a process with three simple steps:

  1. Get the sources and copy them on the block directory.
  2. Configure the CMakeLists.txt file of the block for the specific build instructions for that sources.
  3. Upload the block to the biicode cloud via bii publish command.

So far so good. This approach works pretty well and the biicode community is growing everyday thanks to it. Whats exactly the problem with this approach? Its simple: Some C and C++ sources are hard to compile and it takes time. A lot of time.

SFML works fine as biicode example

Precompiled binaries

Since the early days of C the answer to this problem was to deploy the library sharing the output binaries instead of the source code itself. This has the advantage that the user doesn’t have to compile the library himself, but only to link it with the program which actually using the library.

But precompiled binaries have their own problems:

  • Binaries are OS/architecture specific, that means the output code of a C++ program is different on 32 bit linux than in 64 Windows for example. That makes sense, since the architecture is different. Never forget the final result of C and C++ compilation is binary code, not some kind of high-level assembly interpreted/compiled by a program to the final architecture at runtime, as many modern languages do. This means that if someone wants to develop a portable library (That is, capable of running in different architectures and operative systems), should compile and provide the binaries for each supported platform.
  • Binaries can have a huge size. That depends on the source code, the amount of variables with static storage duration, etc; but the fact is that a C/C++ binary can reach sizes measurable on megabytes, even hundreds of megabytes.

In this post we will learn how to develop a biicode block which contains precompiled binaries instead of the sources, and how to configure the block to be correctly compiled on different platforms.

The block

This tutorial will use the awesome SFML library as an example of a C++ library that can be deployed as a precompiled block with biicode.

For this example, we will make our precompiled block ready for 64bit Linux and Windows. That means we should compile and deploy two binaries, one for windows and other for linux. Both will target GCC on release mode, using MinGW in the case of Windows.

The block has the following structure:

Note the block is manu343627/sfml, since my biicode nick is “Manu343726” and I called the block “SFML”. You can choose the block name you want and use your own account.

As you can see, the sfml sources are located at the block root directory, and the precompiled binaries are located on a lib folder, with subfolders for each supported platform. That directory also contains the CMakeLists.txt file of the block and a bii subfolder with block configuration files.

Getting the binaries

Since this block contains the precompiled libraries of SFML, the first thing we should do is to get that binaries. SFML provides several binaries for each supported platform and toolchain, but the linkage of such binaries depends heavily on your setup (Something compiled with GCC 4.8.1 could not link with a library compiled with GCC 4.9.1 for example), I recommend you to download the sources and compile them yourself using the toolchain you will use with the biicode block. In my case, GCC 4.9.1 both on Linux and Windows, using the STL’s fork of MinGW in the latter.

The easiest way to achieve this is to fork the SFML libraries on GitHub, clone your fork, and then build the library;

Note we will be using static linking instead of dynamic, that’s why the BUILD_SHARED_LIBS variable is set to false.

These instructions are valid for our two supported platforms, both use the “Unix Makefiles” generator.  When you finish compiling SFML, copy the contents of the lib/ directory into its corresponding manu343726/sfml/lib/ directory. Also copy the extra libraries SFML depends on, located under the extlibs/ folder of the SFML repo. These are the OpenAL library and the libsndfile library.

Tuning SFML to be ready for biicode

Our SFML block is named manu343726/sfml, so a SFML hello world program using biicode could be:

Have you noticed any problem? Biicode uses lowercasing for block names, so the original SFML root becomes manu343726/sfml. The first thing we should do is to modify the SFML headers to take care of that change, that is, replace every #include <SFML/> with #include <sfml/> using your preferred text replacing tool.

Now our SFML headers are “biicodeized”, and biicode would find our block when an user refers to any of its headers via an #include "manu343726/sfml/whatever.hpp".

Tuning biicode to work with SFML

Sorry, that’s not all. There are some little things we should configure in our block. Be patient!

Biicode provides several .bii config files to specify explicit configurations that biicode could not infer properly trough its dependencies engine only.

Block include paths

Take a look at the #includes we have changed above: #include <SFML/> is #include <sfml/> on our biicode SFML block. If those includes begin with sfml/, our build should have manu343726/ as an include directory, isn’t? Then our compiler can find these files when preprocessing the code.

Biicode needs a similar setup too, since biicode analyzes dependencies via #include directives. For that purpose, biicode provides the paths.bii file, located under the bii/directory of a block:

The paths of paths.bii use the block root directory as root path, so in the example above we added manu343726/ as an include path (Its the parent of our SFML block directory). Exactly what we want.

Explicit block dependencies

Biicode is a smart file-based dependencies manager, that means biicode manages each file as a potential dependency, and each file dependency is managed separately.

Suppose we create a biicode project with a simple block just to try our awesome SFML block:

The manu343726/testsfml has a simple main.cpp file with the following code:

To build and run that program correctly, biicode should retrieve the required dependencies from the biicode cloud, manu343726/sfml/Graphics.hpp in this case. Remember what we said above: Biicode is a file-based dependencies manager. So the default behavior of biicode is to get the required dependencies only, the Graphics.hpp header in this example, instead of the whole content of the block where the dependency is located.

That means biicode will retrieve and store the manu343726/sfml/Graphics.hpp file only at the deps/manu343726/sfml folder instead of the all the block files, so our precompiled binaries are not downloaded!

This behavior is not the same if you open the dependency block with the bii open command. In that case, you open the dependency block to edit it, so biicode downloads the entire block and our binaries are available.

Of course the guys at biicode were aware of this, and gave us a way to solve our problem: The dependencies.bii file.

Each line specifies a dependency rule, with the following syntax:

meaning that the file at the left side depends on a file or set of files that match the pattern specified on the right. Biicode uses fnmatch for that patterns, refer to the biicode docs for more info.

So what rules we specified? For each SFML library, we said the header depends on a file matching the library name, also the two extra libraries the audio library needs. Our binaries match those names, so now biicode retrieves the binaries for each SFML header we are using.

Always remember that these are file-dependencies. One of the advantages of this approach is that our block will download and link the used SFML libraries only, instead of all the five libraries by default. In our hello world example at the begin of the post, the only header we used is Graphics.hpp, so biicode will download and link the graphics library binaries only.

Continue to the building configurations bellow to see how this exactly works.

Configure building

We have binaries, we have headers, now we have to specify how all those files should be used by the compiler to link the SFML block within the user block. Create a CMakeLists.txt file at the block root directory:

This is the default content of the CMakeLists.txt file of a biicode block. The first command includes all the biicode machinery (Macros, variables, etc), the second initializes the block variables, and then the last command adds the cmake targets corresponding to this block.

What we should do? Given an user program being builded using biicode, where our block is a dependency of it, we should specify cmake where our headers are and how our binaries should be linked within the user program. Here we go.

Include directories:

As we said before, the SFML headers are located at the root directory of the block, so when cmake reaches our block that directory becomes the cmake’s current source directory, which can be accessed via the CMAKE_CURRENT_SOURCE_DIR cmake variable. Also, SFML sources used the SFML (sfml in biicode version) folder as root in their includes.

That means the path to that include root, and the path that should be added to the set of include directories, is manu34326/. In other words, ${CMAKE_CURRENT_SOURCE_DIR}/... Its exactly the same process we have done for the paths.bii above:

Finding out the precompiled libraries:

Now we should configure cmake to be able to find the correct precompiled libraries within this block and then link them correctly. Lets take an overview of our block structure again:

Ok, so we have one folder for the Linux 64 binaries, and other for the Windows 64 binaries. Then our current task is to get that path correctly given the OS and architecture we are. Don’t worry, its a simple if:

The OS flags are self explanatory, but what about the void* trick? Its simple: If the size of a pointer in our architecture is 4 bytes, then the architecture has 32 bit word size, else suppose its 64 bit.

Now we know in what specific architecture/OS we are compiling to, so lets fill a SFML_LIBS_PATH variable containing the correct libraries location:

After the architecture/OS switching, we are able to find the libraries with cmake. SFML has five libraries (system,window,graphics,audio,network), and we will find all those libraries and store them in a SFML_LIBS variable:

The find_library() cmake command finds a library given a set of possible NAMES and possible PATHS. Stores it in a new cache entry with a given name (The first argumment).

Note we used the NO_DEFAULT_PATH flag. FIND_LIBRARY() first searches on the system libraries path by default, but thats not neccessary in our case since we know the libraries are on our block. So we pass our binaries path instead and specify no system searching with NO_DEFAULT_PATH.

Linking

We have searched the binaries with the find_library() cmake command, and the results were stored on several cmake variables, one for each SFML library.

Did you remember the dependencies.bii setup above? We have seen that biicode will download the binaries only for the headers we included. If you use Graphics.hpp, biicode will download libsfml-graphics.so, libsfml-graphics.dll, etc. Nothing more, nothing less.

So we should take care of binaries that were not downloaded, just checking the result of the find_library() calls:

We link the block output library within the SFML system binary only if it was used, downloaded, and found. Simple. Now do that for the five SFML libraries.

Windows .dlls

Even if we are using static linking, SFML depends on some .dlls on Windows, so these files should be placed besides the final user executable (Located at manu343726/bin/ after building). So we should copy those .dlls from our block to that location:

During build setup, biicode sets the cmake variable CMAKE_RUNTIME_OUTPUT_DIRECTORY to point to that bin/ directory, so the easiest way to achieve this is to copy our libraries directory content to that directory, using the copy_directory cmake portable command.

And thats all! Our SFML block is ready to be used on 64 bit Windows and Linux.

Publishing the block

At this time biicode has a maximum size limit of 12MB per block, which can be perfectly reached if we fill the block with binaries. There is a workaround to this problem: That max size is stored in a environment variable BII_MAX_BLOCK_SIZE, which measures that size on bytes. Just tooggle the variable to fit your block, and you are ready to publish!

The results

Here are some screenshots of the Pong SFML example. This is my Manjaro Linux x86_64:

SFML of precompiled binaries in biicode

And this is my Windows 7 x86_64:

SFML of precompiled binaries in biicode

This and more examples are available at the biicode cloud as a examples/sfml block.

Summary

In this post we learnt how to deploy libraries in biicode using precompiled binaries instead of providing and compiling the sources.

The process is not simple, but works as a proof of concept. Now biicode is working in a way to deploy precompiled blocks in a simpler way, to make our lives easier when doing C and C++ development. Stay tuned!


Related Posts
  • http://dimitros.be Dimitri

    Nice! I didn’t know that this was already possible. I was thinking of asking about it in the forums, because it can be useful in some cases (even though in general I have started to prefer linking to source code – makes for easier debugging too). So here’s the answer I guess :)

    • Manu343726

      My approach is based on the dependencies.bii file as a bridge between the headers and the libraries to link with. This works, but dependencies.bii is not aware of your compilation settings nor your system/architecture, so its hard to configure the setup to download the required binaries only (Windows x64 release, for example). Note how my example downloads all the binaries and then decides to what library it has to link to.

      We are working in a hook system to configure all those things in a simple way, I hope we will be able to upload big precompiled libraries to biicode in the near future.

      • http://dimitros.be Dimitri

        It’s already a pretty good method. And anyway, in the meantime, for me all’s fine though. I think that everything works great and not being able to use some library I am accustomed to can be an eye-opener: sometimes I end up realising that I never needed it in the first place :)

  • http://dev.my-gate.net/ Lukas Dürrenberger

    Sounds like a lot of work. Maybe if I was more into biicode, I’d take a closer look, but since I’m not, building from source is more efficient. :D

    • Manu343726

      Thanks for your comment. Is just a proof of concept to see if we can deploy binaries. And its true, it was a lot of work ;)
      We are planning to release a SFML block which builds from sources. Maybe you can try to build your own SFML from sources block? I challenge you :P Good luck!

      • http://dev.my-gate.net/ Lukas Dürrenberger

        Hehe, I first need to try biicode. You probably didn’t know, but I’m a member of the SFML Team and having to deal with people not being able to build and link libraries correctly, makes you wonder if there are alternatives/easier ways. As such biicode is certainly something, I’ll keep my eyes on. ;)