Why do the same thing over and over? Part 1: External dependencies, Swift Package Manager

January 05, 2023

Pre-canned food

Presentaiton version available here.

Video version available here.

If you are a Mobile Engineer like me, you’re likely extremely excited about Apple Silicon computers. They are performant and energy-efficient, but above all, they allow you to build your projects faster and without jet engine fan sounds. However, even with this improvement, the compilation times are very far from what you are getting in web development. Building locally, or on the CI, takes up momentum and can occasionally push you out of the flow, decreasing overall productivity.

Luckily, there are a couple of improvements for build time.

Why build the same things over and over again?

“Insanity is doing the same thing over and over and expecting different results.” Albert Einstein

If you process the same input, using the fixed process and configuration, you can usually expect the same output. This optimization idea is used in many places. The most basic would be caching: save the web page generated in the cache, to serve it again if requested a second time. This idea lays the foundation for something called hermetic build systems, like Bazel. So when building the same source files with the same compiler, we should get the same binaries.

I am not crazybrave enough to move from the vendor build system. Let's see what is possible to achieve in the world of Swift Package Manager with the existing tools.

Looking at the build logs and build timeline available in Xcode 14, I see that from the 10 minutes it takes to compile my project, around half of the time is spent building external dependencies. Taking into account that I don't change the versions of those dependencies that frequently, wouldn't it be great not to recompile them? One thing that is especially annoying here for me personally, is that Firebase rebuilds the protobuf library every time. Wait, I am sure I've seen this before. Where was it?

Reinventing Carthage

Now I remember clearly. Carthage allows this! Carthage is another package management tool that was created before SwiftPM. Carthage is building external dependencies and packaging them as binary XCFrameworks. Why not use it?

High-level overview

We'll move from using external dependencies compiled by SPM at build time, to using pre-compiled versions of the same dependencies:

  1. Define external dependencies in the Cartfile
  2. Build external dependencies to XCFramework
  3. Package all dependencies in your repo using git-lfs
  4. Add logic to unpack the dependencies if required

Let's get building

The first step would be to define the Cartfile, which is just a Carthage way of defining the list of dependencies.

You can see the list of the packages your project relies on in the Project preferences pane, inside Xcode.

You would end up with something like this in your Cartfile:

github "https://github.com/Alamofire/Alamofire.git" == 5.4.1
github "https://github.com/SwiftyJSON/SwiftyJSON.git" == 5.0.0
github "https://github.com/vadymmarkov/Fakery" == 5.0.0
github "https://github.com/SDWebImage/SDWebImage.git" == 5.8.2

But wait, would this mean we will be using two different package managers?

Yes, but wait to see the full picture, we will only need to use Carthage when we would need to change the libraries' versions.

Install and run Carthage

Carthage can be installed from Homebrew, using brew install carthage.

Then you'll need to run carthage bootstrap --use-xcframeworks --platform <Your platform, for exmaple iOS>. Carthage would then fetch your dependencies defined in Cartfile and sub-dependencies. Then it would find binaries for them (if provided) or would compile them from sources. In the end, you would get all of them compiled into XCFramework format.

Expose the frameworks to Swift Package Manager

XCFramework is nice, but if you are using Swift Package Manager for local modules, you would need to carry out one additional step: create a package that will expose those dependencies. This is necessary, since you cannot declare a dependency on a bare framework in your file system: it has to be declared as a Swift Package Manager package.

This can be achieved by defining one Package.swift for all of the XCFrameworks:

// swift-tools-version:5.3
import PackageDescription

let package = Package(
    name: "Contrib",
    platforms: [
        .iOS(.v11)
    ],
    products: [
        .library(
            name: "Alamofire",
            targets: ["Alamofire"]
        ),
        ...
    ],
    targets: [
        .binaryTarget(
            name: "Alamofire",
            path: "Alamofire.framework"
        ),
        ...
    ]
)

In the end, we'll have following file structure:

Root
├── Cartfile
├── Carthage
│   └── Build
│       ├── Alamofire.xcframework
│       ├── Fakery.xcframework
│       ├── Package.swift
│       ├── SDWebImage.xcframework
│       └── SwiftyJSON.xcframework
├── MyProject.xcodeproj
├── Sources
└── setup.sh

Package and store the binary frameworks

Carthage is going to put the frameworks to Carthage/Build. This is where we can take them and archive them. As an example, you can use the tar command:

tar cvzf Contrib.tar.gz Carthage/Build

Git LFS

Git LFS is a Git extension intended to make it easier to keep large-ish files in Git. Certainly, it's possible to just add such files to Git normally, but it is not the best idea, as it would make your repo bigger and bigger the more versions of a given file you add.

Installing and using Git LFS is extremely easy:

brew install git-lfs
git lfs install

Then, let's add our archive to Git LFS:

git lfs track Contrib.tar.gz
git add .gitattributes

Prepare the frameworks for use

We need to think of other engineers who would be interacting with the repository. It's a good idea to create a setup script that would make sure the Git LFS extension is installed and would pull the dependencies file. I will use tar as an example here again:

brew install git-lfs
git lfs install
git lfs pull
tar -xvf Contrib.tar.gz

Referring from Swift Package Manager

Since we have Package.swift now, we can refer from local Swift packages or Xcode projects to our binary frameworks. In your local package, it would look like this:

// swift-tools-version:5.3
import PackageDescription

let package = Package(
    name: "CommonUI",
    platforms: [
        .iOS(.v11)
    ],
    products: [
        ...
    ],
    dependencies: [
        ...
        .package(path: "../../Contrib")
    ],
    targets: [
        .target(
            name: "CommonUI",
            dependencies: [
              .product(name: "SDWebImage", package: "Contrib"),
            ],
            path: "Sources"
        )
    ]
)

Bottomline

As a result of this exercise, our team decreased the CI build and test time from 11 to 6 minutes. This has a significant effect on multiple verticals:

  • A much faster feedback loop on the pull requests
  • Saving money on the CI credit
  • Less CO2, since we need to spin the CPU less
  • Since it is faster to build the project locally, there is less chance for an engineer to lose focus
  • Satisfaction from not rebuilding Firebase over and over: priceless

Q&A

QQ: Should one consider switching to Carthage altogether?

In my opinion no, because Swift PM allows for modularization.

QQ: Why use XCFramework format, specifically?

This format, in comparison to a regular framework, allows the packaging of several binaries of the same SDK for different architectures. In practice, this means Apple Silicon simulator support.

QQ: Any additional effect from this?

One of the effects of this optimization is that the dependencies are now dynamic frameworks. This has pros and cons:

  • If you have multiple targets, every target would be able to link to the frameworks without growing in size. Static frameworks would be included in every target multiplying the size.

  • Dynamic frameworks negatively affect your app's startup time. iOS takes some time to load dynamic libraries on the app startup.