A short guide to compiling, linking, running and debugging issues in the viewer and its packaged libraries. Introduction A recent pass through some third-party libraries resulted in the collection of a lot of information about how things should and shouldn't be built in the viewer. Some of that is presented below with hints and rules about doing things well. What's presented is a guideline only. Not all suggestions are hard rules and you'll find exceptions all over. Some exceptions arise from solid reasoning, others may be legacy that hasn't been re-examined. Use good engineering judgement when applying this information. Compilation Windows Targets Significant compilation flags and defines follow: ---------------------------------------------------------------------------- Option Release RelWithDebInfo Debug ---------------------------------------------------------------------------- wchar_t /Zc:wchar_t- " " RTL type /MD /MD /MDd FLoating Point /fp:fast " " Debug Info /Zi (app/dll), /Z7 (lib) " " Optimizer /O2 /Ob2 /GR /Od /Ob0 /GR /Od /GR Incr. Link /INCREMENTAL:NO /INCREMENTAL /INCREMENTAL:NO Debug /DEBUG /DEBUG /DEBUG /OPT:REF Ignore Libs LIBCMT LIBCMT LIBCMT;LIBCMTD;MSVCRT Alignment Default " " Defines WIN32 " " _WINDOWS " " LL_RELEASE=1 LL_RELEASE=1 n/a LL_RELEASE_FOR_DOWNLOAD=1 n/a n/a NDEBUG NDEBUG _DEBUG n/a n/a LL_DEBUG=1 n/a LL_RELEASE_WITH_\ n/a DEBUG_INFO=1 n/a n/a _SCL_SECURE_NO_WARNINGS=1 _SECURE_STL=0 _SECURE_STL=0 _SECURE_STL=0 _HAS_ITERATOR_DEBUGGING=0 n/a n/a LL_WINDOWS=1 " " UNICODE " " _UNICODE " " WINVER=0x0501 " " _WIN32_WINNT=0x0501 " " LL_OS_DRAGDROP_ENABLED=1 " " CARES_STATICLIB " " LIB_NDOF=1 " " ---------------------------------------------------------------------------- Notes: 1. /Zc:wchar_t-. Not certain where this comes from. It may be due to a default set of compilation flags in Qt 4.X that then propagates outward. In Qt 5.X, this setting is flipped back to default (wchar_t is a built-in). Other options for dealing with this include: http://msdn.microsoft.com/en-us/library/dh8che7s%28v=vs.110%29.aspx Recommend trying to stay with /Zc:wchar_t (the default) when adding libraries. If incompatible, you'll typically get some missing ostream '<<' operators or something similar in the stream headers. 2. /Z7 (VC 7.0 compatibility symbols) gives us debug information in the static libraries we build. Otherwise builds generate vc100.pdb files all over the place which generally aren't useful. DLL's and .EXEs are to get /Zi or /ZI with separate .PDB files. These .PDB files can then be packaged up in symbol tarballs for the crash dump analyzer or used in debugging. There are issues here for VS 2013 (see below). Mac Targets Fairly straightforward, optimization level is easily changed (may be little or negative gain for -O3 and RelWithDebInfo should be kicked up to 1 or 2. Boost debug symbols to dwarf-2 with a goal of dwarf-2 in separate dSYM file when building .dylibs and executables. ---------------------------------------------------------------------------- Option Release RelWithDebInfo Debug ---------------------------------------------------------------------------- Strip Debug Symbols On " " During Copy Generate Debug Syms On " " Level Debug Syms -gdwarf-2 " " Optimization -O3 -O0 -O0 PIC -fPIC -DPIC " " Defines LL_RELEASE=1 LL_RELEASE=1 n/a LL_RELEASE_FOR_DOWNLOAD=1 n/a n/a NDEBUG NDEBUG _DEBUG n/a n/a LL_DEBUG=1 n/a LL_RELEASE_WITH_\ n/a DEBUG_INFO=1 LL_DARWIN=1 " " LL_OS_DRAGDROP_ENABLED=1 " " CARES_STATICLIB " " LIB_NDOF=1 " " ---------------------------------------------------------------------------- Notes: 1. We’re also building dylibs in a somewhat unusual way. They’re currently being generated with a link path of ‘@executable_path/../Resources/<library>’. If we were to follow the recommendations in dyld’s man page, we’d instead reference ‘@loader_path/<library>’, use -rpath on the executable link (pointing to the ‘Resources’ subdir of the main executable), and be able to avoid some symlinking in the .app tree. 2. Use the -headerpad_max_install_names link option on all .dylibs. Linux Targets Not much variety here. ---------------------------------------------------------------------------- Option Release RelWithDebInfo Debug ---------------------------------------------------------------------------- Debug Level -g (-g0/1 better?) -g -g During Copy Optimization -O2 -O0 -O0 PIC -fPIC " " ---------------------------------------------------------------------------- Notes: Linking The library update work has generally moved in the direction of preferring static libraries over dynamic (Qt4 being the notable exception). It also mostly eliminated the extremely bad practice of having multiple versions of a library built into an image. How bad was it? Very. Appalling. A nightmare. On Windows, at least four versions of zlib (1.2.3, 1.2.5, 1.2.6, unknown), three versions of Boost (1.45, 1.48, 1.52), two versions of OpenSSL (0.9.8q, 1.0.0g) and three different builds of libexpat 2.0.5/1.5.2 were used. Mac was worse with five builds or versions of zlib, two of PCRE, two of c-ares, and three of OpenSSL. Linux topped that by adding two builds of libpng. DO NOT ALLOW THIS TO HAPPEN AGAIN. It isn't enough to update a library and then stuff a new triplet of S3 URLs into the viewer's autobuild.xml. If you update a library you MUST: * Update the autobuild.xml of ALL consumers of the library. In the case of zlib, that meant updating freetype, libpng, openssl, libxml2, fontconfig, curl, Boost, SDL, llqtwebkit, google-mock and colladadom. * Confirm by test and observation that the consumers actually use your library rather than 'call home to mother' and find system-supplied versions of your library. This may consist of watching configuration scripts, probing with ldd/depends/otool, pulling text out of binaries with 'strings'. The previously- mentioned libraries all have a README.Linden file that gives examples specific to the consumer library. * DO NOT RE-EXPORT LIBRARIES. Colladadom was the worst offender of this rule. As a shared library, it was re-exporting part, but not all, of Boost filesystem and system, some zlib, some PCRE, some libxml2 and minizip. This meant that depending upon link- time and run-time symbol resolution, data constructed with one version of a library might be processed by a method built in a second, incompatible version of the library. Switching colladadom to a static library ended the re-export problem. * Preventing re-export is not sufficient. other libraries will still be shipped as shared and they can still have Singleton and Fragile Base Class issues. A DLL may be built with a static archive of a library that has global data. That same static archive might be linked into the application proper. An object created with a method in the DLL may pass into a method in the application where the archive's global data has a second instance and no knowledge of the object. This is a failure due to an assumption of Singleton global data which leads to some kind of failure. This is the same effect as when, in Windows, both MSVCRT and MSVCRTD get activated in a program. If you're lucky, some asserts fail in that case having to do with file handle global data. Running Windows Debug Build. Seems to have been rendered nearly useless by having the LL_CHECK_MEMORY define in llmemory.h calling _CrtCheckMemory(). Viewer is almost useful disabling this in llvoavatar code alone but not quite. Futures Static Versus Dynamic Libraries One solution to the above linking problems is the use of static libraries for everything. Single version, singleton instancing of data, etc. But it's not the 1950's and we're not running our applications on bare metal. Every platform comes with 100s of libraries waiting to interfere with operations by breaking the single-version and singleton-data assumption. Additionally, there are libraries that simply expect to be built into shared libraries. Qt4 is one such. The version we're using now, 4.7.1, is actually trying to disable both Webkit and plugin modules because we're building it statically on Mac and Linux. It's only because of configuration bugs that we're getting the functionality out of it that we want. With enough libraries and a single, global namespace, eventually there will be collisions and there may not be a warning. All it takes is two programmers who thought that 'FILE * open_file(const char *);' was a safe signature to use between compilation units in their libraries and glorious debugging sessions are in your future. Having debugged it, you will now become the proud owner of a one-off version of one of those libraries that uses a special symbol prefix which you will be maintaining forever. Lastly, we have some binary blobs that we must use as delivered. Executables can be isolated at run-time if necessary. Shared libraries are a different problem. They may bring their own library dependencies that affect link- and run-time symbol resolution and they'll impose that on us according to platform rules. So, what to do? My natural bias for large software is to use shared libraries for everything. It's a path to single-version and singleton data and isolates namespaces and prevents interactions. It also has some field serviceability benefits if you need to debug some bizarre problem a user has. But there's a local preference for static. Here, my rules-of-thumb are: * Static library used by default. * Shared library where the library must be built shared. * Shared library if that is the only means to enforce the single-version and singleton-data requirements. * Shared library *on a case-by-case basis* if the library is also provided by the platform and some benefit is plausible. (An example of this is freetype/fontconfig on Linux. The .so versions we build with, and incompletely ship, are inferior in behavior to the platform libraries. By being shared libraries, the platform-supplied option is available to all Linux users.) In all cases, beware of cmake which appears to collapse and move library references in links. This can drastically affect symbol resolution when there are multiple sources for a symbol. General VS 2013. The /Z7 flag is rumored to be somewhat broken in 2013. But it also sounds like there are explicit controls to name .PDB files associated with static archives. That would make this an ideal time to switch to /Zi or /ZI everywhere with explicit naming and bring all the .PDBs together. The embedded browser technology (e.g. Qt4 with Webkit) is the 800-pound gorilla in the viewer. When starting any major work, decide what changes you need here as those changes will propagate outwards forcing many other decisions (cf: /Zc:wchar_t- flag). The current package structure (./include, ./lib/release, ./lib/debug, etc.) really works against the conventions used by configure-like programs. I wasted a lot of time getting each library to work with our structure without having to go back to automake/autoconf. For Linux and Mac (and even for Windows), a structure like the following where each grouping is optional would probably save some work: ./debug/bin /include /lib ./debug/shared/bin /include /lib ./debug/static/bin /include /lib ./release/bin /include /lib ... In zlib and openssl and in a few of the libraries that consume them, I experimented with packaging both static and shared libraries and then having the consumer library move the unwanted pieces out of the way to use the library type of choice (see restore_dylibs() and restore_sos() functions). It was a bit fussy and simplicity and clarity are the keys to maintaining libraries in the future. But it did suggest another approach. The idea is that every build pre-stages inputs. Before anything is built, package pieces are copied or symlinked from the 'stage/packages' area to the 'stage/input' area. Builds then proceed with a single set of -I/-L options for the dependencies. And products are built and installed in a similar output staging structure for the next consumer: stage/packages/<package>/[above structure] stage/input/{bin,include,lib} stage/<package>/[above structure] Next library project. I'd recommend working on the related set of libexpat, apr, aprutil, xmlrpc-epi. We know libexpat has some updates that should improve stability. Libapr consumes it and it could use some /Z7 flag work to get rid of some 1000's of PDB warnings and improve our debug symbols. Miscellany to be sorted out: * The packaging of libfreetype and libfontconfig on Linux. Determine what the right thing is, do it. * Maybe do something with ICU4C. Qt5 will require it and a number of our packages can consume it typically replacing iconv or some other library. But it is a huge bolus of static data. It can be trimmed, but still. * Revisit openssl. Package as a shared library? Replace with LibreSSL when available? Start using platform-supplied crypto?