This summer, one of the development teams at the Ed-Fi Alliance has been hard at work building Project Buzz: “When school shutdowns went into effect across the country as a result of COVID-19, much of the information teachers need to support students in the new online-school model had become fragmented across multiple surveys and the Student Information System.” (Fix-It-Fridays Delivers Project Buzz, A Mobile App to Help Teachers Prepare for Back-to-School).
As project architect, my role has been one of support for the development team, guiding technology toolkit choices and supporting downstream build and deployment operations. The team agreed to develop the applications in TypeScript on both the front- and back-ends. My next challenge: rapidly create TeamCity build configurations for all components using Kotlin code.
Components
At this time, there are four components to the software stack: database, API, GUI, and ETL. The project is available under the Apache License, version 2, on GitHub. The build configurations for these four are generally very similar, although there are some key differences. This gave me a great opportunity to explore the power of creating abstract base classes in TeamCity for sharing baseline settings among several templates and build configurations.
Requirements
- Minimize duplication
- Drive configurations through scripts that also operate at the command line, so that developers can easily execute the same steps as TeamCity.
- The above item implies use of script tasks. When those scripts emit an error message, that message should trigger the entire build to fail.
- All build configurations should check for sufficient disk space before running.
- All build configurations should use the same Swabra settings.
- All build configurations will need access to the VCS root, and the Kotlin files will be in the same repository as the rest of the source code.
- All projects will need build steps for pull requests and for the default
branch.
- Pull requests should run build and test activities
- Default branch should run build, test, and package activities, and then trigger deployment.
- Both branch and pull request triggers should operate only when the given component is modified. For example, a pull request for the database project should not trigger the build configurations for the API, GUI, or ETL components.
- Pull requests should publish information back to GitHub so that the reviewer will know the status of the build operation.
Classes
BuildBaseClass
The most general settings are applied in class BuildBaseClass
, covering
requirements 3, 4, 5, 6, and the commonalities in the two branches of
requirement 7.
Structure of BuildBaseClass
Note that only the required imports are present. The class is made abstract via
the open class
keywords in the signature.
package _self.templates
import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.freeDiskSpace
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.swabra
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.powerShell
open class BuildBaseClass : Template({
// contents are split up and discussed below
})
Requirement 3: Fail on Error Message
It took me a surprisingly long time to discover this. PowerShell build steps in TeamCity behave a little differently than one might expect. You can set them to format StdErr as an error message, and it is natural to assume an error message will cause the build to fail. Not true. This setting helps, but as will be seen below, is not actually sufficient.
open class BuildBaseClass : Template({
// ...
option("shouldFailBuildOnAnyErrorMessage", "true")
// ...
})
Requirements 4 and 5: Free Disk Space and Swabra
Apply two build features: check for minimum available disk space, and use the Swabra build cleaner.
open class BuildBaseClass : Template({
// ...
features {
freeDiskSpace {
id = "jetbrains.agent.free.space"
requiredSpace = "%build.feature.freeDiskSpace%"
failBuild = true
}
// Default setting is to clean before next build
swabra {
}
}
// ...
})
Requirement 6: VCS Root
Use the special VCS root object, DslContext.settingsRoot
. Checkout rules are
applied via parameter so that each component’s build type will be able to
specify a rule for checking out only that component’s directory, thus preventing
triggering on updates to other components.
open class BuildBaseClass : Template({
// ...
vcs {
root(DslContext.settingsRoot, "%vcs.checkout.rules%")
}
// ...
})
Requirement 7: Shared Build Steps
The database project, which deploys tables into a PostgreSQL database, does not have any tests. Therefore this base class contains only the following build steps, without a testing step:
- Install and Use Correct Version of Node.js
- Install Packages
- Build
That first step supports TeamCity agents that need to use different versions of
Node.js for different projects, using nvm for
Windows. The second executes yarn
install
and the third executes yarn build
. Because the TeamCity build agents
are on Windows, all steps are executed using PowerShell.
open class BuildBaseClass : Template({
// ...
steps {
powerShell {
name = "Install and Use Correct Version of Node.js"
formatStderrAsError = true
scriptMode = script {
content = """
nvm install %node.version%
nvm use %node.version%
Start-Sleep -Seconds 1
""".trimIndent()
}
}
powerShell {
name = "Install Packages"
workingDir = "%project.directory%"
formatStderrAsError = true
scriptMode = script {
content = """
yarn install
""".trimIndent()
}
}
powerShell {
name = "Build"
workingDir = "%project.directory%"
formatStderrAsError = true
scriptMode = script {
content = """
yarn build
""".trimIndent()
}
}
}
// ...
})
BuildOnlyPullRequestTemplate
Structure of BuildOnlyPullRequestTemplate
Once again, the structure below contains only the required imports for this
class. Carefully note the brace style: in the abstract class, the class
“contents” were all inside braces as an argument to the Template
constructor.
In this concrete class, the “contents” are inside an init
method, which is in
turn inside a code block outside the BuildBaseClass
constructor. You can learn
more about this in the Kotlin: Classes and
Inheritance documentation.
This class inherits directly from BuildBaseClass
and does not need to apply
any additional build steps.
package _self.templates
import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.commitStatusPublisher
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.PullRequests
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.pullRequests
import jetbrains.buildServer.configs.kotlin.v2019_2.triggers.VcsTrigger
object BuildOnlyPullRequestTemplate : BuildBaseClass() {
init {
name = "Build Only Pull Request Node.js Template"
id = RelativeId("BuildOnlyPullRequestTemplate")
// Remainder of the contents are split up and discussed below
}
}
Requirement 8: Pull Request Triggering
Here I am attempting to use the Pull Request build feature. I have had trouble
getting it to work as advertised. This configuration needs further tweking, to
ensure that only repository members’ pull requests automatically trigger a build
(do not want random people submitting random code in a pull request, which might
execute malicious statements on my TeamCity agent). I need to try changing that
branch filter to +:pull/*
.
object BuildOnlyPullRequestTemplate : BuildBaseClass() {
init {
// ...
triggers {
vcs {
id ="vcsTrigger"
quietPeriodMode = VcsTrigger.QuietPeriodMode.USE_CUSTOM
quietPeriod = 120
// This allows triggering on "anything" and then removes
// triggering on the default branch and in feature branches,
// thus leaving only the pull requests.
branchFilter = """
+:*
-:<default>
-:refs/heads/*
""".trimIndent()
}
}
features {
pullRequests {
vcsRootExtId = "${DslContext.settingsRoot.id}"
provider = github {
authType = token {
token = "%github.accessToken%"
}
filterTargetBranch = "+:<default>"
filterAuthorRole = PullRequests.GitHubRoleFilter.MEMBER_OR_COLLABORATOR
}
}
}
// ...
}
}
Requirement 9: Publishing Build Status
This uses the Commit Status Publisher. Note that the authType
is
personalToken
here, whereas it was just token
above. I have no idea why this
is different ¯\(ツ)/¯.
object BuildOnlyPullRequestTemplate : BuildBaseClass() {
init {
// ...
features {
commitStatusPublisher {
publisher = github {
githubUrl = "https://api.github.com"
authType = personalToken {
token = "%github.accessToken%"
}
}
}
}
// ...
}
}
PullRequestTemplate
Unlike the class described above, this one needs to run automated tests.
Unfortunately, it demonstrates my (current) inability to avoid some degree of
duplication. Perhaps in a future iteration I’ll rethink the inheritance tree and
find a solution. For now, it duplicates features shown above, with the only
difference being the base class: it inherits from BuildAndTestBaseClass
, shown
next, instead of BuildBaseClass
.
BuildAndTestBaseClass
This simple class inherits from BuildBaseClass
and adds two steps: run tests
using the yarn test:ci
command and run quality inspections using command yarn
lint:ci
.
package _self.templates
import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.freeDiskSpace
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.swabra
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.powerShell
open class BuildAndTestBaseClass : BuildBaseClass() {
init {
steps {
powerShell {
name = "Test"
workingDir = "%project.directory%"
formatStderrAsError = true
scriptMode = script {
content = """
yarn test:ci
""".trimIndent()
}
}
powerShell {
name = "Style Check"
workingDir = "%project.directory%"
formatStderrAsError = true
scriptMode = script {
content = """
yarn lint:ci
""".trimIndent()
}
}
}
}
}
BuildAndTestTemplate
Based on BuildAndTestBaseClass
, this class adds a build step for packaging,
and artifact rule, and a trigger. Although these are TypeScript packages, the
build process is using NuGet packaging in order to take advantage of other tools
(NuGet package feed, Octopus Deploy). The packaging step is orchestrated with a
PowerShell script. The configuration can be used for any branch, but it is only
triggered by the default branch.
package _self.templates
import jetbrains.buildServer.configs.kotlin.v2019_2.*
import jetbrains.buildServer.configs.kotlin.v2019_2.buildFeatures.freeDiskSpace
import jetbrains.buildServer.configs.kotlin.v2019_2.buildSteps.powerShell
import jetbrains.buildServer.configs.kotlin.v2019_2.triggers.VcsTrigger
import jetbrains.buildServer.configs.kotlin.v2019_2.triggers.vcs
object BuildAndTestTemplate : BuildAndTestBaseClass() {
init {
name = "Build and Test Node.js Template"
id = RelativeId("BuildAndTestTemplate")
artifactRules = "+:%project.directory%/eng/*.nupkg"
steps {
// Additional packaging step to augment the template build
powerShell {
name = "Package"
workingDir = "%project.directory%/eng"
formatStderrAsError = true
scriptMode = script {
content = """
.\build-package.ps1 -BuildCounter %build.counter%
""".trimIndent()
}
}
}
triggers {
vcs {
id ="vcsTrigger"
quietPeriodMode = VcsTrigger.QuietPeriodMode.USE_CUSTOM
quietPeriod = 120
branchFilter = "+:<default>"
}
}
}
}
Component-Specific Projects
Bringing this all together, each components is a stand-alone project and contains at least two build types: Branch and Pull Request. These respectively utilize the appropriate template. The parameters are defined on the sub-project, making the build types extremely small:
BranchAPIBuild
package api.buildTypes
import jetbrains.buildServer.configs.kotlin.v2019_2.*
object BranchAPIBuild : BuildType ({
name = "Branch Build and Test"
templates(_self.templates.BuildAndTestTemplate)
})
PullRequestAPIBuild
package api.buildTypes
import jetbrains.buildServer.configs.kotlin.v2019_2.*
object PullRequestAPIBuild : BuildType ({
name = "Pull Request Build and Test"
templates(_self.templates.PullRequestTemplate)
})
API Project
Of the parameters shown below, only project.directory
and vcs.checkout.rules
will be familiar from the text above. The Octopus parameters are used in an
additional Octopus Deploy build configuration, which is not material to the
current demonstration.
package api
import jetbrains.buildServer.configs.kotlin.v2019_2.*
object APIProject : Project({
id("Buzz_API")
name = "API"
description = "Buzz API"
buildType(api.buildTypes.PullRequestAPIBuild)
buildType(api.buildTypes.BranchAPIBuild)
buildType(api.buildTypes.DeployAPIBuild)
params{
param("project.directory", "./EdFi.Buzz.Api");
param("octopus.release.version","<placeholder value>")
param("octopus.release.project", "Buzz API")
param("octopus.project.id", "Projects-111")
param("vcs.checkout.rules","""
+:.teamcity => .teamcity
+:%project.directory% => %project.directory%
""".trimIndent())
}
})
Summary
TeamCity templates have been developed in Kotlin that greatly reduce code duplication and ensure that certain important features are used by all templates. Unfortunately they did not completely eliminate duplication. Through use of class inheritance, merged-branch and pull request build configurations are able to share common settings. However, parallel templates with some duplication were still required.
In the future, perhaps I’ll explore handling this through an alternative approach using feature wrappers instead of or in addition to templates. My initial impression of these wrapper functions is that they obscure a build type’s action: in the examples above, a Template class reveals its base class, signaling immediately that there is more to the Template. In the feature wrapper approach, one only finds the additional functionality when reading the project file. It will be interesting one day to see if the two approaches can be combined, moving the wrapper inside the template or base class, insead of being applied to it externally.
License
All code samples above are Copyright © 2020, Ed-Fi Alliance, LLC and contributors. These samples are re-used under the terms of the Apache License, Version 2.
Previous Articles on TeamCity and Kotlin
- Getting Started with Infrastructure as Code in TeamCity
- Splitting TeamCity Kotlin Into Multiple Files
Posted with : DevOps Tools and Practices, Tech, Software Development Life Cycle