Wildland Core

Getting Started

welp, TODO

General Stability

Work In Progress Page!

Glossary

Stable API - breaking change requires MAJOR version bump, Experimental API - breaking change does not requires MAJOR

Motivation

After release process is introduced, it is very likely that some parts of its API may change over time and will break it or be incompatible. To prevent bumping MAJOR version every release (or slowing down development by stabilizing every feature) team decided to pick API parts that are mature and should not change very often and stabilize it. The rest is considered unstable/experimental - you can play with it but it might be prone to errors or major changes.

This document describes state after release 1.1.0

Legend

  • [!!!] - Experimental API
  • [???] - Partially stable API - used only for component
  • [—] - Stable API

Stability

components and their features states:

Component A ⚠

  • Feature xyz_1 ✅
  • Feature xyz 2 ⛔

General Versioning

Versioning scheme

  • Project is versioned with semver (MAJOR.MINOR.PATCH)
  • Major releases may break your product. Look for changelog/release notes to see if you need to update your code in order for it to work properly.
  • Minor releases will add new features without breaking any functionality.
  • Patch releases should be treated as hotfixes. They will not require any
  • changes to your deployment configuration other than a version number change. They may introduce new config settings which will be optional.
  • Major and minor releases might require changes to deployment configuration.

Release

  • On each release, repository state will be marked with a tag and artifacts will be built for all supported targets and platforms.
  • Standard(non-tagged) releases will be made only from main branch.
  • Tagged releases may be created from different branch, unless stated otherwise we do not guarantee theirs stability.

Breaking changes

By breaking change, we mean a change in public api, that is not backward-compatible. Public api defines how clients integrating with the libraries can call use those, and what are the requirements for those. Due to the project being specific, the internal API between the components is also considered public. To learn more, please review chapter below for reference.

Internal components changes

Each release will have a fixed version matrix i.e:

  • component A : 1.0
  • component B : 1.1
  • component C : 0.2

Tis will be recommended and stabilised versions of compoenents for the release. Release means that components are confirmed to work together and were extensively tested. However, usage of different versioning is possible taking above into considerations: some things can, but not must, be broken or incompatible. In that case, tT the very least, versioning matrix should be taken into consideration as a minimal supported versions of components.

Release schedule

There is no strict release schedule at this point. New versions will be released when there is a need for them(e.g. new feature implemented).

Architecture

Development

Introduction to the development procedures.

This documentation should be treated more as a help or guidance, rather thatn a procedure. It is unrelated to the processes described in /docs/processes), however, described here are tips and tricks to make the development easier and more automated.

Toml files formatting

.toml files are expected to be formatted with Taplo (https://taplo.tamasfe.dev/) tool (extension for VS code: Even Better TOML). Config file is placed in the root folder of the workspace which CLI Taplo should find automatically.

taplo fmt Cargo.toml

CI support for building SDK for Apple frameworks

Core project CI delivers SDK in form of multi-architecture framework published as binary Swift package.

This document describes the setup and requirements involved.

CI process

CI is configured to build Apple SDK according to figure below.

Apple SDK in CI process

Every merge request triggers building the SDK and each commit to main branch triggers both building and publishing the SDK.

Apple SDK builds are executed by CI workers tagged with macos tag. See Worker requirements section for detailed requirements that build machines must meet.

Build flow

SDK build is performed by ci/build_apple.sh script which implements the following flow.

Apple SDK build flow

Build is performed as shown on figure 2.

Involved infrastructure

Key elements of the involved build infrastructure are shown in the diagram at the beginning of this document. More details on their role is provided in this section.

Repositories

There are two git repositories involved in the process:

  • Wildland Core is where builds are triggered
  • Sdk Apple is a technical repository for storing Swift package manifests for SDK releases

When new build of SDK is uploaded, a corresponding package manifest is generated and pushed to Sdk Apple repository.

Binary artifact hosting service

The setup uses a Google Storage service (wildland-apple-dev-binaries) to host binary builds of the SDK. The service is expected to be publicly accessible through HTTP.

Note that Swift Package manager used as part of Apple development environment has currently an issue that, when accessing public HTTP sites it proactively sends HTTP basic authentication header with empty credentials. This does not work with Google Storage, which rejects such requests as unauthorized. To work this problem around, a HTTP proxy (xcode-proxy.wildland.dev) is deployed as part of the infrastructure. The sole task of the proxy is to remove Authorization header from HTTP requests relayed to Google Storage.

Worker requirements

The build step needs to be executed by worker capable with compiling artifacts for Apple targets (macOS/iOS). Additional requirements:

  • Rust compiler supporting aarch64-apple-darwin and x86_64-apple-darwin targets
  • Google Cloud SDK

Also, the account, from which builds are performed must be able to push to master branch of Sdk Apple repository.

Relevant configuration variables

Build script expects that APPLE_GCS_BUCKET_UPLODAER_ACCT secret contains base64 encoded authentication token for Google Storage uploads.

Other relevant variables are defined and documented in build_apple.sh.

Integrating Apple SDK into macOS project

To use Apple Wildland SDK in Xcode project one should enter https://gitlab.com/wildland/corex/sdk-apple.git as Swift package URL, use branch dependency resolution rule and set master as branch to be tracked.

Further work

The following issues are not handled yet and need to be addressed by future work:

  1. Separate debug/release builds.
  2. Version tagging.
  3. Add iOS SDK target.

About

Place to store developer processes related to the repository.

Commit message formalization and enforcement

Commit message formalization

In commit message, the line length must be limited to 80 characters per line. This applies to all lines in the commit, including, but not limited to the list of fixes, features, and commit message itself.

In the repository, all issues titles, pull request titles and commit messages must be prefixed with a tag.

Currently, following tags are allowed:

  • chore - dependency updates, CI updates, refactorings, other changes that don’t affect the project functionality
  • test - adding or removing tests; it does not include changes to deployment or CI. These go under chore.
  • fix - fixing a bug in code
  • feat - adding new feature to the project
  • docs - adding documentation (readmes, mdbook, plantuml etc.)

On merge, commits must be squashed and one of 5 tags must be used. Scope in the project does not matter (i.e. COMPONENT_XYZ) - most of the changes affect multiple components. The summary should describe which components were changed.

As for the long commit message:

tag: [TASK] summary

* tag: summary
* tag: summary

The parent tag is the main goal of the work, and the list contains all things we did during implementation. E.g.:

feat: [WILX-0123] add lazers

* chore: add deployment target on the moon
* docs: document usage of lazers (and how to keep them away from cats)

In case the commit is not connected to any task in jira, the [TASK] can be dropped.

Commit message enforcement

Commit message format is enforced via one of CI Actions/Workflows

Branching strategy

Acceptance in progress

=========================================================
shortened version is available at the end of the document
=========================================================

Glossary

  • Release Branch - A fork originating from the development branch, marking the point in time, in which a feature set is frozen, and an official release is made. example 1.3.1-123
  • Development Branch - The main, active branch to which all code that is currently worked on is merged. Only developer releases (a.k.a. release candidates) can be done from this main branch. example: develop
  • Develop or develop - a common name used in this document to refer to development branch (see above)
  • MAJOR,MINOR,PATCH - The standard notation for semantic versioning. The triplet represents three main categories of versions: major release, minor release, and patches. The importance of the version section decreases from left to right. Additionally, a bump in a specific category resets everything in categories to the right to 0.
  • Release - A fixed build of either the release branch or development branch that is then published for public usage.

Description

If we consider a release to be a collection of features added since a prior release, our goal in producing releases, is to provide a stable set of features that are accessible to end users, as well as maintaining a means by which we can conveniently and correctly provide patches for said releases.

The branching strategy is a a soft-restricted version of the gitflow.

When a release is planned, the next branch should be created (sprouted from the common development branch) with name based on semversion (e.g. v1.3.12).

First, all the features should be completed and then merged into develop. Develop should then be frozen and prepared for next release. A release branch should be then created from the tip of the freeze (if any last-minute fixes are applied).

If a feature is planned but not delivered in time before the freeze, then afterward the freeze not delivered changes should not be committed (only last-minute fixes to existing ones), until release branch is created, main branch is treated as one and it can not get new features. Additionally, a release branch’s origin cannot be moved after creation and features cannot be cherry-picked or merged afterward (no rebase or cherry-pick).

After the release branch is created (the stabilisation branch), the stabilisation effort will proceed on that branch; however the development branch can still be used for further development. That means, if the release deadline will be missed by a developers, stabilisation does not have to wait for the feature development to release. That also means that no new features can’t be committed to the released branch, only stabilisation patches and/or bug fixes (which also have to be mirrored to development branch if applies)

If Stabilisation branch is deemed stable, it can be merged to master. Squashing of the commits is required and release must be kept in proper format. Example format is shown below:

Release X.Y.Z

- feat: cat lasers
- feat: more lasers
- fix: lazers do not hurt other cats anymore
- ...

After the merge, stabilistaion branch must be deleted.

Versioning

Release branches are not supported long term. The reason releases are happening is to periodically provide user with stable and tested build artifacts, that are used by non-rust clients - i.e. c++, c#, or switft clients.

Master Branch

Tags are REQUIRED on this branch.

Versions and tags on master branch must be in semver format: For example 1.3.1

Development Branch

Tags are OPTIONAL on this branch.

tags on the develop branch must be in semver format followed with dev-<timestamp>. Timestamp format is <yyyymmdd-hhmm> For example 1.3.2-dev-20220911-0420

Stabilistaion Branches

This branch is not tagged.

Branch must be created using only semversion. For example 1.3.1.

This version is going to become next tag on master after merge. This branch after merge must be deleted.

Bugfixes

When bug fixes are required to fix incorrect behavior in a release. Code that fixes the incorrect behavior must be first added to develop, then, when the team decides its time introduce the fixes to the master branch, the code should be merged using the standard procedure (as in release procedure). For the rules regarding the release versioning, please check the Versioning chapter of this document.

TL;DR

This document is describing what is called, gitflow and many materials are freely available online in different sources. This document however introduces and specifies how tagging and release preparations should look like. Besides that, everything should be 1:1 with official documentation about the process.

Patch Versioning

  • Patches will result in a bump of the current version’s PATCH number on the affected release branch.
  • Patches will not result in bump in development (release candidates).
  • Patches will be collected for each release. One patch does not have to result in one new release.

Patch Application

  • If the development branch is affected, the patch should be fixed and changes should be pushed to the development branch.
  • If the supported release branch is also affected by the proposed patch… (to the developer’s discretion):
    • … and the fix is easily applicable, it should be cherry-picked from develop to release branch.
    • … but the resulting fix is not easy to forward to the release branch (i.e. missing refactor, changes in history, conflicting features, etc), the fix has to be crafted and applied manually for each branch.

Branch Tagging

  • The master branch will be tagged by its (MAJOR,MINOR,PATCH) tag for each release
  • On development branch, each patch may additionally introduce a build tagged with (MAJOR.MINOR.PATCH)

Documentation structure and RFC process

Acceptance in progress

Glossary

RFC - Request for Comments

Documentation structure

Architecture

  • path: docs/architecture
  • purpose: information about each component, schema registry, object builder, and other applications. It contains only our internal architecture description, the current state of each element. Information what it does, why and how, what algorithms (from a broad perspective without many implementation details) were used, the protocol between the systems, etc. It should be enough information to adapt a new developer to the project quickly.
  • used by: Dev Team

Configuration

  • path: docs/configuration
  • purpose: description of all ENV variables and configuration files
  • used by: Deployment/User

Deployment

  • path: docs/deployment
  • purpose: documentation of how to deploy the project or its artifacts
  • used by: Deployment/User

Examples

  • path: docs/examples
  • purpose: a place for tutorials, code examples and guides.
  • used by: Developer/User

Features

  • path: docs/features
  • purpose: a place for feature (behavior) description. Each feature should have at least one document describing it. Example: file_ordering.md, partial_delta_synchronization.md
  • used by: Developer/User/Management

Processes

  • path: docs/processes
  • purpose: a place for all our processes. For example: commit message formalization or this process.
  • used by: Dev Team

RFCs

  • path: docs/rfc
  • purpose: a place for all RFCs.
  • used by: Dev Team

RFC process

RFC describes only change to the project. The change might be:

  • New feature
  • Modification of existing feature
  • New process
  • Modification of existing process
  • Documentation about research done

It does not contain information about a single component (its current state, as this is the purpose of architecture).

RFC Category

Each RFC must contain a category informing whether it describes the feature, process or the research. Accepted values: Feature/Process/Research.

RFC Deadline

Each RFC must contain a deadline to speed up the process. Before the deadline is met, the author should remind all interested parties about it.

If the deadline is met without conclusion, the author of the RFC should create a meeting and invite all interested in these RFC parties. Then during this meeting resolution has to be made.

The deadline must be set at least 3 days from the announcement. RFC author should schedule the meeting on the nearest possible date that is acceptable for all parties.

Until the deadline, everyone is allowed to comment on the RFC. Participants should use the scheduled meeting only to decide when there is more than one strong opinion about the subject. All issues should be defined and discussed via the comments on git hosting site. The deadline meeting should be relatively short if every participant agrees on the subject.

There may be exceptions to this rule - for example, when during a deadline meeting, everyone agrees to wait for new information from superiors, etc.

RFC ordering number

Each RFC has an assigned ordering number that is an always-increasing integer. In rare situations, there might be gaps between two integers. RFC ordering number is encoded by 4 digit (fixed length) decimal integer (values from 0001 to 9999). In the future, it might be extended by prefixing additional zeroes (9999 becomes 09999).

  • RFC ordering number has to be unique.
  • RFC ordering number and feature ID are not the same concepts.

The process

Draft

The author of the RFC should create a new document in the rfc directory. This directory has a flat structure, and each RFC should have a filename in human-readable format with RFC ordering prefix number and version postfix number. For example: 1234_database_scheme.md.

Format: <RFC ordering number>_<Human readable name>.md

Each RFC should contain at least following sections in the format:

  • “Motivation” - explaining the reasoning behind the rfc (what does the rfc solves or why was it brought to the table)
  • “Impact Analysis” - containing, if applies:
    • work, packages, and tasks needed to implement RFC, rough estimates are also welcome,
    • breakage of contracts or features if any,
    • any other action that is requried to properly implement the RFC,

RFC Header Example:

# Front Matter

```
Title           : Schema-Registry-less deployment
Category        : Feature
Author(s)       : Yours Trully
Team            : Corex Team
Created         : 2021-06-24
Deadline        : 2021-06-26
Feature ID      : FEAT-00016-00
```
  • Feature ID is only required for feature descriptions. If the RFC describes a process, this field should be omitted.
  • Team field is optional.
  • RFC format allows other custom fields to be included if necessary, such as why RFC has been abandoned.
  • The author should commit the draft to a separate branch, for example, user/rfc/<filename>.

Pull Request

When the draft is ready for review, an author should create PR and inform reviewers.

Resolution

Each RFC must result in:

  • When accepted:
    • If the team decide to implement a feature described in the RFC:
      • Implementator should create tracking issue - he should include its id
    • in features/index.md table.
      • Update in architecture - all components that were changed should be updated, so this RFC won’t be needed to understand how, for example, Object Builder works. It should be done as a part of PR that has been merged.
      • Update in features - if RFCs category is Feature - Update existing or create new feature.md and write a new description of what this feature does. In some cases, the author can selectively copy-paste it from RFC; however, the author must be careful. It should be done as a part of PR that has been merged.
    • If RFC describes the process:
      • processes/index.md needs to be updated
      • Update in processes - if RFCs category is Process - Update existing or create a new process.md and write a further description of how the process looks. In some cases, the author can selectively copy-paste it from RFC; however, the author must be careful. Implementator should do it in PR that has been merged.
  • If the deadline has been met, but no conclusion has been reached - RFC should be updated with the new deadline (sub-issues).
  • When abandoned - RFC should be updated with Abandon reason in Front Matter (top of the document).

It also means PRs with implementations should not be accepted and merged before merging documentation.

Exception: It is allowed to split PR into smaller, by merging the first and then separate PR documentation. However, until the last PR implementation is merged, tracking issue has to be kept open.

Further changes

Apart from minor typo fixes, and formatting, RFC is immutable. Whenever there is a need to change the feature or the process, one should create another RFC describing new goals and changes.

Summary

RFC describes changes while features/processes/architecture keep up-to-date information about the current state of the project.