Developing at Google scale

Google has a huge source code tree and it is very interesting how it manages it. There are several links about it:

Build tools at Google

Problems with Perforce scaling

Ian’s linker notes

Ian Tailor, the author of the gold linker has a great series of posts in his webblog.

  1. Introduction, personal history, first half of what’s-a-linker
  2. What’s-a-linker: Dynamic linking, linker data types, linker operation
  3. Address spaces, Object file formats
  4. Shared Libraries
  5. More Shared Libraries — specifically, linker implementation; ELF Symbols
  6. Relocations, Position Dependent Shared Libraries
  7. Thread Local Storage (TLS) optimization
  8. ELF Segments and Sections
  9. Symbol Versions, Relaxation optimization,
  10. Parallel linking
  11. Archive format
  12. Symbol resolution
  13. Symbol resolution from the user’s point of view; Static Linking vs. Dynamic Linking
  14. Link time optimization, aka Whole Program optimization; Initialization Code
  15. COMDAT sections
  16. C++ Template Instantiation, Exception Frames
  17. Warning Symbols,
  18. Incremental Linking
  19. __start and __stop Symbols, Byte Swapping
  20. Last post; Update on gold’s status

Build tools list

At my day job I work on a large C++/Java project. The project is really huge ~700K files. We use GNU Make and as you might suspect it is pain to build the project. Especially it is pain to rebuild it incrementally. Even if I change a small C++/Java file it takes ~10 minutes to recompile the project. A lot of time Make spends in its own internals – it tries to parse the Makefiles hell, it tries to sats() all files, it constructs full dependency graph and then rebuild it.

I am not very satisfied with our build system. Compare to other Makefile based projects it is fine, but we need something better. And I think many project in the corporate word have the same issue.

I started researching this question some time ago and I was surprised how many build tools besides Make/Automake exists out there. Let me describe just few of them:

  • Autotools. It is de-facto standard for open-source projects. Old and portable across nearly all UNIX platforms. Many people hate it with passion because of its weird m4 language and unreadable output files.
  • Scons. Written in Python. It uses Python program for build description. Unfortunately it is not scalable. KDE tried scons but then gave up and moved to CMake. Real-world usages: MongoDB.
  • Waf. A fork of SCons optimized for speed. It also uses Python for build scripts. Who uses: Samba.
  • CMake. IMHO the most widely used build tool outside of Make/Autotools. It generates Makefiles that you need to run later. Large and active community. It uses ugly custom-made language for macroses, it looks like BASIC. Honestly, I am not very impressed with its internals. Who uses: KDE.
  • Premake. It tries to fix CMake script language by using Lua language instead. It also generates Makefiles that you need to run with Make. Activity is quite low.
  • Tup. A tiny build tool. Claimed to be fast for incremental build. When I say fast i mean really fast. It uses inotify daemon to find changed files. Instead of keeping dependency graph in memory the tool stores it to SQLite database. According to tup documentation build time depends on number of files changed, not the project size.

Main sell point of all tools (except the last one) is a new syntax for build files. But they do not solve my problem – slow incremental builds. So I played with Tup a little bit more. There is a Linux distributive called gittup that uses this build tool. I tried to build/rebuild this Linux distro and honestly I am impressed how well Tup handles it.

I spent about a hour and converted LLVM project to tup. This was not difficult at all. Tupfile syntax is dead-simple, very similar to Make. Incremental build are fast, inotify monitor also works great.

So I declare Tup is my new toy for the next couple of weeks. And maybe some day I’ll try to convert my project to tup. Who knows.

Vala + Autotools example

As a part of learning great Vala language I decided to implement the simplest possible example of Vala console application e.g. a simple calculator program that parses user input and evaluates the expression.

The project is going to have:

  • Unit tests that tests API level functions.
  • A usage example of libraries that can be used both for unit tests and for the program itself.
  • Small and readable Autotools code. The project should be reusable by anyone else as a skeleton for their own project.

The code of the project you can find at my github page.

I am not going to describe Autotools plumbing code. It is pretty much the same as any other Autotools based project has. I recommend to read following article if you are interested in the Autotools details.

Let’s look at the Vala-specific code instead.

  • The key file for autoconf tool.It has several interesting pieces:

AM_PROG_VALAC – Autotools already has Vala support and the given macros enables it. After you added it Automake knows what to do with *.vala source files – it invokes valac compiler.

pkg_modules="glib-2.0 >= 2.24.1
             gobject-2.0 >= 2.24.1"

This code checks in your system libraries needed to compile Vala C files. It also evaluates paths to header and shared library files – later these variables will be used for compilation.

  • src/calculator.vala is a core file that contains most of the calculator logic. Later we are going to test this functionality.
  • src/main.vala this is thin wrapper for the calculator library. It has main() method and this file is needed for creating end-used console application.
  • src/ This is the most interesting file in the project. Let’s look at it closely.
bin_PROGRAMS = calculator

Here we declare that this directory has 1 library and 1 binary file.

libcalculator_la_SOURCES = calculator.vala
libcalculator_la_VALAFLAGS = --library calculator -H calculator.h
Calculator library consists of 1 file – calculator.vala but you can as many *.vala files as you want. The second line declares that make should generate calculator.vapi file and *.h that is also needed for compilation.  *.vapi file is kinda ‘header’ file for Vala compier the *.vapi file contains Vala classes declaration, but does not have any implementation. So the library generates 3 files – *.la, *.vala, *.h.
calculator_SOURCES = main.vala
calculator_VALAFLAGS = calculator.vapi

This generates output binary file, our console application that we develop. Its source file is main.vala – wrapper around the library. _LDADD adds calculator library as a dependency, _VALAFLAGS  passes *.vapi file to valac compiler.

  • test/calculator_test.vala. This is a unit (aka API level) test
  • test/

Run ./ and then ‘make’ to build ./src/calculator console application. If you run ‘make check’ you’ll see test results

make  check-TESTS
/calculator/add: OK
/calculator/minus: OK
/calculator/multiply: OK
PASS: calculator-test

PS Things that I might add in the future:

  • Valadoc documentation generation. It is nice to have API in readable format so you can upload it to the project server.
  • CMake example. I don’t have a lot of experience in autotools so in this project I learned Autotools in addition to Vala language. As of my taste Autotools have a lot of black magic. I still do not understand how the build system works for my project. Some people say that CMake easier to learn that Autotools. So similar example for CMake could be a good lesson as well.

Краткий ликбез по SSD

На тематических форумах периодически задают вопросы о технических особенностях SSD. Я решил скомпилировать имеющуюся у меня информацию, благо команда в которой я работаю проводила тестирование на предмет использования SSD на серверах.
Практически все SSD производятся по технологии NAND. Все нижесказанное относится к этому типу флеш памяти.

Continue reading

Unit testing in Vala

Unittesting is a widely used technique. Unfortunately I haven’t seen any Vala projects with good and clean unittests.

This is surprising as GLib (and Vala) has unit testing framework that is similar to xUnit family. Here is the documentation

Sample application for Vala+Autotools

I started learning Vala language recently. The language looks very promising – fast execution speed, clean syntax, a lot of language features. Unfortunately Vala does not have a lot of documentation. One thing that makes learning process easier is an example. A good example can be used as documentation.

I tried to implement and share my sample application for Vala+Autotools, but then I found one implemented by someone else. Meet Sampala.

One thing I wish to see there is unittesting. Maybe this is a good opportunity for me to learn vala by adding tests to Sampala.