Skip to main content

Security

warning

WIP (Work in progress)

This page will help you learn some steps you can take to protect yourself and your users as you develop your code with ALKiln tests. It will also talk about the security practices ALKiln itself uses.

In software development, security can be an intimidating topic. By design, ALKiln tests are unable to do very much, but all software comes with risks. We can mitigate those risks with robust security measures. Over time, we can all build up our security tool chests to:

  • Protect our users' information
  • Protect our security-critical information
  • Protect our servers

We want to protect against anything that might expose or modify any important or sensitive data.

One of our strongest tools to increase security is the principle of least privilege.

Principle of least privilege​

Follow the principle of least privilege. That means we should give software (or people) the least access possible to your server and data while still letting them do what they need to do. For example, if someone just needs to fill out an interview, give them the role of "user", not "developer". If someone needs to develop interviews, give them the role of "developer", not "admin".

It is one of the most effective ways to protect yourself and your users. Only give ALKiln the access it needs and no more.

Personal data is information about real people, whether that is old information or new information. Security-critical data includes passwords, usernames, API keys, and other credentials.

If you are worried you have already shown sensitive information in your GitHub tests, you can delete your logs and artifacts.

Separate production and testing​

This is for: server admins

Using a separate testing server and separate testing data are some of the most effective ways to limit ALKiln's access to personal and security-critical data.

There are some kinds of tests that packages run on their production server, but ALKiln is in a different category of tests. Limiting ALKiln's access means that it:

  • Never has access to actual user information
  • Never has access to your production server's security-critical data like configuration tokens
  • Never has access to your current or old production databases
  • Can never affect the files on your production server

First and foremost, avoid installing or running GitHub+You™ and ALKilnInThePlayground™ tests on your production server1. Have a separate server for those testing methods. GitHub Sandbox™ tests already make their own server and can be the most secure if you use them properly. Use those if you are able to use them.

A separate server is only one part of separating production and testing sever access, though. You also need to:

  • Keep the production server's urls out of ALKiln test files, workflow files, and workflow environment variables.
  • Never share API keys or passwords between servers.
  • Never share real data or databases between servers, whether the data is current or old.

Use fake data in tests​

This is for: Everyone

Determined people can do a lot with very little data. For example, they can even use anonymized data to identify your users. Avoid giving your ALKiln tests access to real data, even if it is old data. If you need help creating fake data, sites or tools like https://cobbl.io/ might be useful.

If your test needs a logged-in user, avoid using a real person's account. Instead, use ALKiln's account or a completely separate fake account.

Never use a database with real information with your tests, even old information. For example:

  • Never share data between your production server and your testing server. For example, if your production server uses a Google spreadsheet to store some user or server data, create a separate Google spreadsheet with fake data for your testing server.
  • Never use old databases. Some projects realize that they should avoid using their current user database for testing. Instead, they decide to use an old database. Avoid that as well. Even if the data seems so old that it is useless, it can still expose you and your users.

Make accounts just for tests​

This is for: Everyone

Even though you are using a testing server, avoid giving ALKiln access to an actual developer's name, username, password, API keys, and other sensitive information by giving ALKiln its own accounts. This also lets you see its activity in the logs more clearly and lets you delete the account easily if you need to. There are non-security related reasons to make a separate account too2.

Only give ALKiln an account if necessary. Make sure no one else uses the account. Give that account a role with only the privileges it needs and no more. Also, avoid giving it other real data or access. For example, avoid giving it any real person's name and avoid giving it GitHub integration.

We usually call the ALKiln user "automated_tests" and give it the email "automated_tests@example.com" with a secure password.

The GitHub+You™ tests need to be connected to a docassemble account with the right role (permissions) to work properly. That is the "developer" role. Even if you are using another type of test, though, your tests may need an account. For example, you may want a test to sign in to an account before going to an interview.

If you do need an account to sign in before a test, think of that account as a separate account because you might be able to give that account even less access. If the sign-in account can just be a user, make a new account just for signing in that just has a "user" role.

If you stop needing an account, delete it.

ALKiln account role​

This is for: Everyone

ALKiln strongly limits what tests can do. Even so, you should limit ALKiln too by giving it the fewest permissions possible.

For GitHub+You™ tests, give the ALKiln docassemble account the "developer" role. "Developer" is the most limited role ALKiln can use with GitHub+You™ tests to do what it needs to do. When you make an API key for the account, ignore the extra permissions options.

If your ALKilnInThePlayground™ tests need an account on your server, give them the least privileges possible. In those cases, most authors only need to give ALKiln the "user" role.

The GitHub Sandbox™ tests run on their own server, create their own account, and manage their own permissions.

Still protect sensitive data​

This is for: Everyone

Even after keeping personal and security-critical data away from the account that runs ALKiln tests, there are still ways you can expose your test server's sensitive data in tests. Even if you are using a testing server, that server still has credentials, like API keys and passwords, and you should avoid exposing those to the ALKiln tests.

Avoid sensitive data in package files​

This is for: Everyone

Avoid putting personal or security-critical information in your package's files. It is possible that someone will see those files. For example, if you have a public repository on GitHub, anyone can see your package's files.

If you need to use sensitive information in your tests:

Avoid showing sensitive data on screen​

This is for: Everyone

Avoid visually showing personal or security-critical data on an interview page. For example, you can show a server config value on an interview page in the text of a subquestion. Even though you yourself can try to avoid taking pictures of those pages, an error can happen on that page. ALKiln takes pictures of pages that error and there is no way that ALKiln can know what text you are choosing to show to the user.

Using ALKiln to fill in answers in the interview is different. You can safely use sensitive information to fill in fields by using the "secret value" Step.

Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables, even when the test has an error on that page. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.

Avoid JSON Steps with sensitive data​

This is for: Everyone

The "get JSON" and "compare JSON" Steps show a page's JSON variables by logging them, saving them in a file, or both. ALKiln is currently unable to filter out sensitive values - personal or security-critical data - in that JSON. Only use those Steps when you are sure the JSON will be safe to show.

How to use sensitive data in GitHub tests​

This is for any packages that: run tests in GitHub

To do this you MUST be:

  • Either the GitHub repository admin or
  • The GitHub organization admin

These are the only people that can create GitHub secrets

It can help to know about:

Sometimes authors do need to pass personal or security-critical information to their tests. For example, some tests require the "user" to sign in, so the tests need to know the email and password for that user. In GitHub, you can do this securely by using GitHub secrets and adding environment variables to your repository's GitHub workflow file. You can also read about how to use sensitive information with ALKilnInThePlayground™ tests.

  1. Follow the GitHub instructions to set up one or more GitHub secrets. You can add these secrets to one secrets or you can add these to your organization3, whichever is right for you.
  2. Find your workflow file.
  3. It should include a line that looks similar to this:
jobs:
  alkiln-tests:
    runs-on: ubuntu-latest
    name: Run ALKiln tests

If this is the first environment variable you're adding, right below that last line add a new line like this:

    env:
  1. Whenever you want to add a secret, add a new line under env: indented from env::
      ALKILN_USER_PASSWORD: "${{ secrets.ALKILN_USER_PASSWORD }}"

ALKILN_USER_PASSWORD is just a placeholder in our example. You can name your secrets whatever you want to. Make sure the name here matches your GitHub secret name.

  1. Write your Step and use the names of these secrets as the values. For example:
  And I set the variable "very_secure_password" to "ALKILN_USER_PASSWORD"

Your workflow file code will end up including something like the below:

# Other workflow code

jobs:
  alkiln-tests:
    runs-on: ubuntu-latest
    name: Run ALKiln tests
    env:
      ALKILN_USER_PASSWORD: "${{ secrets.ALKILN_USER_PASSWORD }}"

# Other workflow code

We recommend that you start your environment variable names with "ALKILN_". That will help avoid conflicting with the environment variables of your operating system.

Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables, even when the test has an error on that page. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.

Right now, Story Tables are unable to use workflow environment variables to fill in fields.

How to use sensitive data in ALKilnInThePlayground™ tests​

This is for: ALKilnInThePlayground™ tests

To do this you MUST be able to:

  • Edit your docassemble server's config file

Sometimes authors do need to pass personal or security-critical information to their tests. For example, some tests require the "user" to log in, so the tests need to know the email and password for that user. You can do this securely in your ALKilnInThePlayground™ tests by adding this information to your docassemble server configuration file. ALKiln will use these as environment variables for your tests.

You can also read about how to use sensitive information in GitHub tests.

To do this, edit your docassemble server's configuration file to add your environment variables under the alkiln key. For example:

# Other config values
alkiln:
  ALKILN_USER_PASSWORD: 123password

Then use that environment variable name in your test. For example:

  And I set the variable "very_secure_password" to "ALKILN_USER_PASSWORD"

We recommend that you start your environment variable names with "ALKILN_". That will help avoid conflicting with the environment variables of your operating system.

Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables, even when the test has an error on that page. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.

Right now, Story Tables are unable to use workflow environment variables to fill in fields.

GitHub Sandbox™ tests​

This is for any packages that: have GitHub Sandbox™ tests

The GitHub Sandbox™ are very secure if you use them correctly. You still have to take some care.

In your tests, avoid using urls that lead to your server. That is, avoid using https://my-server.com or https://testing.my-server.com and interviews there or other pages there. That kind of url will send the tests to your server, which escapes the sandbox that makes the GitHub Sandbox™ method so useful. Using urls like that makes the tests less secure again.

You also still have to avoid saving personal or security-critical information in public files and follow other security precautions, like the ones described in this documentation.

Freeze your ALKiln version (advanced)​

This is for any packages that: run tests in GitHub

It can help to know about:

To control which version of ALKiln is running your tests on GitHub, you can manually set the exact version you want. That is called "freezing" the version. Then, when you are ready, you can look at what has changed and decide which version you want to update to.

By default, ALKiln runs your tests with its latest version. This means you have all the latest bug fixes and new features. Some organizations choose to use a version that has code and behavior that is familiar and comfortable for them.

ALKiln is actually made of 2 parts. One part is the ALKiln framework, which lets you control the robot that goes to your interview and fills in answers. Another part of ALKiln is its GitHub actions, which manage setting up, starting, and cleaning up the tests.

Because of factors out of our control, each of these parts has a different way to set the version. To freeze the framework, you need to use the semantic version number. For example, "5.8.0". To freeze the actions, you need to use the 40-character git commit id. Both of those need to point to the same5 code. For example, the semantic version "5.8.0" of the ALKiln framework should be paired with the git commit with the id e79eb61cb97603d41375134d505e4fe60de7780d that controls the actions. To freeze your version:

  1. Find the semantic version that you want.
    1. Look at the "git blame" for our package.json.
    2. Find the 3rd line. It should be the line that shows ALKiln's version.
    3. To see the previous version, you can tap on this symbol:
    4. Go back in history by tapping on that symbol until you find the version you are looking for. For example, 5.8.0.
  2. Find the matching commit id.
    1. Tap on the commit message for that line.
    2. In the address bar of your web browser, look for a 40-character long set of random letters and numbers. The whole address might look something like this: https://github.com/SuffolkLITLab/ALKiln/commit/e79eb61cb97603d41375134d505e4fe60de7780d. In that example url, the commit id (or SHA4) is e79eb61cb97603d41375134d505e4fe60de7780d.
  3. Set both versions.
    1. Find your workflow file.
    2. Tap to edit the file.
    3. Go to the lines that look similar to the ones below.
        uses: SuffolkLITLab/ALKiln@v5
        with:
          SERVER_URL: "${{ secrets.SERVER_URL }}"
          DOCASSEMBLE_DEVELOPER_API_KEY: "${{ secrets.DOCASSEMBLE_DEVELOPER_API_KEY }}"
  1. Replace v5 with the commit id you found. For example, a29519914ffc235802be9b9b143d78ffc3a06028.
  2. Add a line below DOCASSEMBLE_DEVELOPER_API_KEY. The indentation level should be the same as DOCASSEMBLE_DEVELOPER_API_KEY.
  3. Add the text ALKILN_VERSION: ""
  4. Between the quotes, put the semantic version that matches the commit id. For example, 5.8.0.
  5. Save the file.

Using the example above, the final code should look something like this:

        uses: SuffolkLITLab/ALKiln@a29519914ffc235802be9b9b143d78ffc3a06028
        with:
          SERVER_URL: "${{ secrets.SERVER_URL }}"
          DOCASSEMBLE_DEVELOPER_API_KEY: "${{ secrets.DOCASSEMBLE_DEVELOPER_API_KEY }}"
          ALKILN_VERSION: "5.8.0"

Update your tools​

This is for: Everyone

Update your tools, especially when that update includes a security fix. Security updates are meant to protect you from newly discovered risks.

Choosing when to update to the latest version of software can be tricky. New features and bug fixes can interfere with your working system. That said, when there is a security update you should seriously think about upgrading quickly.

Your tools should keep a record of their changes, often called a "changelog". This is ALKiln's changelog. This is docassemble's changelog. However, your software tracks changes, those records should clearly label security updates. Keep an eye out for those updates.

Update ALKiln when needed​

This is for: Everyone

Check ALKiln's changelog for updates and update regularly, especially when there is a security update.

Security updates will have the heading "Security". For example, this is one of ALKiln's security updates.

Update docassemble​

This is for: server admins

Upgrade your testing server's docassemble version quickly when there is a security update. Then run all your tests before upgrading your production server too.

With most updates, you can usually wait a little while to see if there are any bugs that come up and get fixed. You should deal with a security fix more quickly. If possible, you then want to update your production server too, to make sure your testing server and your production server are using the same version. That helps make your tests more accurate.

Start a new Docker container regularly​

This is for: server admins

Clear out the testing server and start a new docker container from scratch when there are security upgrades. Then run all your tests before upgrading your production server too.

Docassemble's Docker image gets updated occasionally. You can wait a little while to see if there are any bugs that come up, but generally keep up to date with those as much as you can, especially when there are security fixes.

Even if there is not a security-related update, it is good to stress test your server setup. Since you might have to suddenly start a new Docker container at unexpected times because of system failures, it is a good idea to do this every now and then anyway to make sure your system can handle recovering from that kind of event and see if complications come up.

If possible, you then want to update your production server too, to make sure your testing server and your production server are using the same version. That helps make your tests more accurate.

General GitHub practices​

This is for any packages that: run tests in GitHub

There are some ways to make your GitHub repository itself more secure that have nothing to do with ALKiln.

Protect pull requests​

This is for any packages that: run tests in GitHub

It is possible for you to add code to your workflow file to trigger tests when someone makes a pull request on GitHub. If you decide to do that, take extra steps to make sure malicious coders are unable to trigger your workflows.

If you configure your workflows to be triggered by a pull request, someone can change the code of the tests, workflows, or other package code to do something malicious, then trigger that code by just making a pull request.

By default, GitHub prevents first-time contributors from triggering pull request workflows without your permission. Unfortunately, some people make first-time contributions that are very reasonable, like fixing typos. Once you accept that contribution, they are then able to trigger workflows with their pull requests. Before making that pull request, they can edit the code of the tests, the workflow, or the rest of the interview to contain malicious code.

You can make those settings more strict for public repositories and for organizations. One setting makes sure no outside collaborator can trigger workflows with a pull request without your approval.

Private repositories can ignore this since they would first have to share their repository with an outside collaborator, presumably someone they trust. If they do want to collaborate with unfamiliar people, private repositories can also require approval from forks.

GitHub logs and artifacts are public​

This is for any packages that: run tests in GitHub

On GitHub, workflow job logs and artifacts of public repositories are visible to the public, just like the code. Also, any logged-in user can download those workflow artifacts.

That might be fine. Part of being transparent is letting the public understand how your packages are working. Also, there are some ways ALKiln lets you set variables that you want to keep secret. That said, if you are concerned about your logs and artifacts, there are some things you can do to make these less available:

Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables, even when the test has an error on that page. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.

Protect your default branch​

This is for any packages that: have a repository in GitHub

To do this you MUST be:

  • Either the GitHub repository admin or
  • The GitHub organization admin

You can protect your packages from unintended changes or changes by new collaborators by protecting your default branch.

Your default branch is usually called "main". That branch usually has the code that you run on your server and that other packages use. For that reason, many packages protect their default branch from unexpected changes.

For example, you can require that:

  • Contributors make new branches when they edit code
  • Contributors make a pull request before new code can be added to the main branch
  • Administrators have to approve the pull request before that code can be merged

You can allow administrators to skip these restrictions if you want.

Use GitHub's recommended workflow practices​

This is for any packages that: have a repository in GitHub

GitHub has documentation about keeping your GitHub workflows secure. We cover some of that content in here, but you can read that documentation for more.

You can specifically look at GitHub's recommendations for using third-party actions, like the ALKiln actions.

Things to worry about less​

There are some security protections that you have by default.

Worry less: Your production server​

This is for: Everyone

As long as you ensure you only test on your testing server, ALKiln cannot affect your production server. You will have new files in your package, but they are unable to do anything by themselves.

  • The .feature files that you have in your "Sources" folder do nothing by themselves. They are as safe as text files.
  • The GitHub workflow files can only run on GitHub.

Worry less: GitHub redacts secrets​

This is for any packages that: run tests in GitHub

In your GitHub workflow run console, GitHub redacts any text that matches the values of your GitHub secrets. It replaces that text with "***". Note that the text must be an exact match. GitHub is unable to recognize partial matches.

If you are worried about logs or artifacts that already exist, you can delete those.

Worry less: ALKilnInThePlayground™ tests already freeze your ALKiln version​

This is for any packages that: use ALKilnInThePlayground™ tests

The ALKiln framework lets you control the robot that goes through your interview answering questions. This documentation has already discussed how you can freeze your ALKiln versions on GitHub.

ALKiln itself already uses practices that protect you from malicious actors. If you want to take extra precautions, though, freezing the version you are using with ALKilnInThePlayground™ tests does give you more control over what code you are installing on your server and using to run your tests.

The first page of the ALKilnInThePlayground™ interview lets you choose the exact version of ALKiln that runs your tests. You can upgrade or downgrade at any time.

Worry less: You can disable tests​

This is for any packages that: run tests in GitHub

If you ever become uncomfortable or concerned about your ALKiln tests, there are multiple ways you can disable them.

ALKiln protects you too​

Our team takes special precautions with ALKiln's code to ensure you are as safe as we can make you. We are always looking for more ways to improve our security practices.

ALKlin is limited​

This is for: Everyone

By design, the ALKiln framework is very limited in what it lets a test do. A test can:

With GitHub+You™ tests, ALKiln must have developer privileges to set up the tests, but the tests themselves are just like an unregistered person online. For other types of test methods - ALKilnInThePlayground™ and GitHub Sandbox™ tests - ALKiln doesn't even need a an account on your docassemble server.

ALKiln hides environment variables​

This is for: Everyone

Workflow environment variables might contain sensitive information. Because of this, ALKiln takes extra care with them.

Since workflow environment variables might hold sensitive information, ALKiln avoids taking pictures or downloading the HTML of pages that use environment variables, even when the test has an error on that page. ALKiln avoids printing the value of an environment variable anywhere in the report or in the console log.

JSON variables

You do need to take care when you use JSON Steps.

ALKiln reports its security updates​

When ALKiln has a security update, we announce it on docassemble's Slack channel and post it in ALKiln's changelog. The Slack channel is called "#alkiln-automated-testing".

Security updates have the heading "Security". For example, this is one of ALKiln's security updates.

ALKiln freezes the versions of its dependencies​

This is for: Everyone

Just like you can freeze the versions of the ALKiln framework, ALKiln freezes the versions of its own dependencies - the libraries it uses. When we update our dependencies to their latest versions, we check the updates and do our best to make sure that we are comfortable with the changes.

ALKiln updates its dependencies​

This is for: Everyone

The ALKiln team checks on the security of ALKiln's dependencies. When a dependency reports a new security improvement, we update ALKiln to use that new improvement. Not all dependency fixes are relevant to ALKiln, but it is still good to keep up to date.

ALKiln protects its major branches​

This is for: Everyone

ALKiln protects itself from unintended changes or changes by new collaborators. ALKiln has gotten many volunteer contributions over time, and we want to make sure we check that code carefully before we release it to you.

As our documentation described, you can protect your default branch from changes. ALKiln has multiple branches that users depend on - one branch for every major version. For example, v3, v4, and v5. It protects all those branches in various ways. For example, we:

  • Require contributors to make new branches when they edit code
  • Require contributors to make a pull request before new code can be added to the main branch
  • Require administrators to approve a pull request before that code can be merged so that we have a chance to review it

Our administrators can get past these restrictions. We keep tight control of who our administrators are.

ALKiln uses good action file practices​

This is for any packages that: run tests in GitHub

ALKiln's action files are very similar to workflow files, like yours. As such, they use the same GitHub-recommended practices that your workflows can use.

You can report an ALKiln security vulnerability​

This is for: Everyone

If you find an ALKiln security vulnerability, you can report it to us privately.

Footnotes​

  1. Using a different server for testing is better in other ways, too. When developers pull packages that have python files onto your docassemble server, it reloads for a while. This can cause your clients' interviews to time out. GitHub+You™ tests do the same and can make for a slower user experience if they are used on a production server. ↩

  2. There are other good reasons to avoid using an existing developer's account that are not related to security. For one thing, the tests create Projects and run interviews. It may be confusing for an existing developer to find those in their account. For another thing, if you do use a real developer's account and that developer leaves your organization, tests will start failing. It may take you a while to track down the problem and fix it. ↩

  3. GitHub organization secrets can be useful if an organization wants to create a variable that all of its repositories will be able to use and avoid creating that secret in every individual repository. ↩

  4. SHA stands for "secure hash algorithm". Git has a secure hash algorithm of its own to make an "id" for each commit that is 40 characters long. Very broadly, for each commit a repository makes, git uses math to reliably turn the whole history of the package and all its versions, including the newest commit, into 40 letters and numbers. Unlike a branch name, no one can edit a commit SHA because it would break the math. A developer can only delete the commit. Because of this, you can pretty much rely on a commit SHA to always go to either the same code or no code at all. That is, no one can hide sneaky new code there. ↩ ↩2

  5. Technically, the same exact version is usually overkill. As long as the action code and the framework code are aligned, they can point to different versions. This simplification is a lot easier to explain, though. ↩