Wildland documentation (legacy)

⚠️ Warning: This documentation refers to the proof-of-concept implementation of Wildland client (opens new window) which is no longer maintained. We are currently working on a new Wildland client written in Rust. To learn more about Wildland and the current status of its development, please visit the Wildland.io webpage (opens new window).

# Quick Start Guide

Read the Practical Introduction to Wildland first to understand Wildland's fundamental concepts.

# Setup

# Downloading Wildland client

The source code for the Wildland client (opens new window) is maintained as a git repository and available for cloning from the GitLab project Wildland (opens new window):

$ git clone -b master https://gitlab.com/wildland/wildland-client.git

The client can be used on Linux platforms without any compilation and building process, but we nevertheless highly recommend to use the provided Docker image, which we discuss in the next section.

Client configuration is stored within the standard ~/.config/wildland/ directory. This directory is also used when using the Docker, as it is exposed to the Docker from the host by default. Read more about the configuration here (opens new window).

# Easy-to-use Docker image

The most convenient way to use the current Wildland client is through the provided Docker image. Please ensure you have Docker downloaded and properly configured for your platform. To build the Docker image:

$ cd wildland-client/docker/
$ docker-compose build

To run the Docker image:

$ docker-compose run --service-ports wildland-client

Docker has been successfully tested on recent Linux, macOS and Windows 10 platforms. Read more platform-specific HOWTOs here.

# Wildland command-line interface

All Wildland commands are implemented by the wl command. This command takes the sub-command, such as: container, forest, user, etc. We will be using and discussing some of these commands below. Full documentation about the command line interface can be found here (opens new window).

For convenience, popular commands such as container(s), user(s), etc., are available as one-letter shortcuts.

# Creating a new user

To create a new user:

$ user@wildland-client:~$ wl user create yatima
Generated key: 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072
No path specified, using: /users/yatima
Created: /home/user/.config/wildland/users/yatima.user.yaml
Using 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072 as @default
Using 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072 as @default-owner
Adding 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072 to local owners

This command will create two important artifacts:

  1. A cryptographic key pair representing this user (analogous to a blockchain wallet), together with a 32-byte hash of the public key, which is used as user ID (analogous to a wallet account address),

  2. A user manifest (a small yaml file), which can be further used for sharing access to our forest with others, as discussed at the end of this HOWTO.

The private key is used for both signing and decrypting the manifests either created by the user or those which were encrypted using that user's public key.

Since -- by default -- all manifests are encrypted to the manifest's owner (unless other users are explicitly indicated, as discussed in the Access Control and Sharing), it is very important not to lose access to the private key. We highly recommend backing up the keys and user manifests, which are stored in the users/ and keys/ sub-directories of the ~/.config/wildland/ directory.

To list all users known to the client:

$ wl user ls
/home/user/.config/wildland/users/yatima.user.yaml (@default) (@default-owner)
  owner: 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072
  private and public keys available
   path: /users/yatima

Wildland also supports so-called group users, which are discussed more in Multi User Forests HOWTO.

# Mounting the Wildland Filesystem

All Wildland containers are exposed as a filesystem, as discussed earlier. This Wildland filesystem, similarly to an external drive or Dropbox, is exposed under some defined mount point within the host filesystem. By default this is ~/wildland. If we're using Docker, then Docker exposes the filesystem through the WebDAV (opens new window) protocol and can be mounted wherever the WebDAV-accessing file viewer decides to map it. The default address to connect with Wildland content using WebDAV is 127.0.0.1:8080. Check out platform-specific HOWTOs for details.

To actually mount the Wildland filesystem you need to start the Wildland FUSE (opens new window) daemon:

$ wl start

Initially this filesystem will be empty, but as we perform subsequent steps below, we will see containers and even foreign forests exposed there.

# Accessing existing forests

We will now try to mount some existing forests[1], and show how to use Wildland in read-only mode.

Together with today's release of the client, we're also opening up for public read-only access two Wildland forests:

  1. The Pandora forest (a reference to the Avatar (opens new window) movie), which is a repository of memos, reports, documents, etc., which we've been using internally to help us with Wildland's development. It's a bit like a shared Dropbox folder, except that it uses abstracted storage and heavily utilizes multi-categorization.

  2. The Ariadne forest, which is a very small forest containing bridges to other forests, specifically to the Pandora forest, and some other, internal forests, which are encrypted and not exposed for public use.

# Importing bridges to other forests

To keep the Wildland client agnostic to any particular organization, such as Golem Foundation (opens new window), we have decided not to include bridges to any "default directory forests" together with the client sources. For this reason, you need to manually import a user manifest into some other forest, which you decided to use as a starting point in discovering other forests:

$ wl user import --path /mydirs/ariadne https://ariadne.wildland.io

Note how we explicitly specify --path to tell where -- in our namespace -- we want to place the bridge to this forest.

To check if a bridge was indeed created one can use the wl bridge ls command:

$ wl bridge ls
/home/user/.config/wildland/bridges/forest-owner.bridge.yaml
  owner: 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072 (/users/yatima)
  paths: /mydirs/ariadne

# Mounting forests

To mount any forest, for which we already have a bridge, we can use the following command:

$ wl forest mount :/mydirs/ariadne:

Note that we don't explicitly specify the user ID, leaving the first part of the Wildland address empty (so we start the address with :), telling the client to use the default user ID for resolution of this address. In this case, it instructs to use the user's default namespace, to which the bridge above got imported.

Also notice the trailing : at the end of a Wildland address -- this is because we don't point at any specific file within the Forest, but a container/bridge path. Those paths are the basis for how Wildland organizes the path traversal throughout Wildland forests. Note that every Wildland object which can be mounted (i.e. container, user, bridge) has a path key in its manifest.

Now we should be able to see the content of the Ariadne forest inside the directory represented by the bridge:

$ tree -F ~/wildland/mydirs/
/home/user/wildland/mydirs/
└── ariadne:/
    └── forests/
        └── pandora:/
            └── WILDLAND-FOREST.txt

We can mount the Pandora forest:

$ wl forest mount :/mydirs/ariadne:/forests/pandora:

We should now be able to see the content of the Pandora forest:

$ tree -F -L 1 ~/wildland/mydirs/ariadne:/forests/pandora:/
/home/user/wildland/mydirs/ariadne:/forests/pandora:/
├── WILDLAND-FOREST.txt
├── arch/
├── backends/
├── clients/
├── communication/
├── docs/
├── economy/
├── ecosystem/
├── events/
├── home/
├── hr/
├── maps/
├── ops/
├── persons/
├── places/
├── pubkeys/
├── reports/
├── resources/
├── similar_projects/
├── status/
├── teams/
├── timeline/
├── usecases/
└── ux/

# Creating containers and forests

# Creating storage template(s)

In order to simplify the process of creating containers, it is advisable to create a template for various storage backends. Wildland will use the indicated storage template when creating containers or setting up a new forest, as shown below.

Creating a storage template is easy. You need to specify the type of backend (WebDAV, S3, Dropbox, etc.) which is to be used. For example:

$ wl template create webdav mywebdav \
    --login <USERNAME> \
    --password <PASSWORD> \
    --url https://<URL>

Or, for a personal Dropbox:

$ wl template create dropbox mydropbox \
    --app-key <YOUR_DROPBOX_APP_KEY>

See Backends HOWTO for a list of currently supported plugins (a short list is also printed by wl template create).

In order to list all templates known to the client, you can use:

$ wl template ls
Available templates:
    mydropbox
    mywebdav

# Creating a forest

To create a new forest you need to: 1) have a user defined for which the forest is being created, and 2) define a storage template which will be used for storing the forest's catalog. Then, we can easily create a forest with:

$ wl forest create mydropbox

The command above will do two things:

  1. It will automatically create the catalog container (where the forest's container manifests will be stored later) from the given storage template and place it in the storage itself.

  2. It will update the user manifest file to include a link to the above-mentioned catalog container.

A newly-created forest like this is not very useful, unless we want to share it with others, which we'll discuss in the next step.

# Sharing a forest with others

There are essentially two ways how a forest can be shared with another person or computer:

  1. By sharing the user manifest (of the forest owner) with the recipient -- e.g. sending it via email.

  2. By placing a bridge to the forest in some other forest (e.g. on a forest like Ariadne, which effectively plays a role of a directory).

The Access Control and Sharing Tutorial discusses the details of forest sharing further.

# Creating a container

Once at least one storage template has been defined (as discussed above), and once the user's forest has been created, then creating any individual container is very straightforward:

$ wl container create \
    --title "My Container" \
    --category "/tests" \
    --template mydropbox

This command assumes the default user as the owner of the newly created container (and it will fail if there are no users defined in the system). If you want to create a container into another forest, it's important to use the --owner switch and specify explicitly the owner of the container. E.g., assuming the current user is part of the Pandora group user (see the Group users), which means they are able to sign manifests on behalf of such a group user, you would write:

$ wl container create knowledge-sharing \
    --owner pandora \
    --title "Knowledge Sharing in Wildland Project" \
    --category /docs \
    --category /ops \
    --category /persons/joanna \
    --template pandora-storage

Note how the owner was specified using the friendly name pandora instead of a full pubkey hash -- this is possible only if the client has a copy of the user manifest locally, which indeed is expected in this case. An explicit user ID in the form of a 0x... hash can be specified as well, of course.

# Mounting a container

In order to actually use the container (read and write files from and into it), we need to mount the container:

$ wl container mount @default:/tests/My\ Container:

Most Wildland commands operate on individual containers, rather than on individual files within the container, which is why in most cases we end the address with a trailing colon (:).

This time we used the user alias @default, indicating the default user as the namespace in which the container path should be resolved. But we could very well give the user ID explicitly, e.g. for Pandora:

$ wl container mount \
    0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe:\
    /docs/Container\ Addressing/:

This should mount the container of the given path (/docs/Container\ Addressing/) from the forest of the given ID. Here we assume that the client knows the user manifest for this user ID from a previous import by the user, as discussed earlier in this tutorial.

We can also use a bridge in place of the explicit user ID. For example:

$ wl container mount \
    0x30c9856e7d1903d80f201afb90f33741fafa870c9c5a656134bcf5a42312febd:\
    /forests/pandora:\
    /docs/Container\ Addressing/:

Here we used Ariadne's user ID first (0x30...), and then pointed to the bridge leading to Pandora, and only then did we point to the container address within Pandora.

Assuming Ariadne is at a known location within the user's default forest (e.g. /mydirs/ariadne:), we could write the simpler:

$ wl container mount :/mydirs/ariadne:/forests/pandora:/docs/Container\ Addressing/:

Assuming Ariadne is set as the default directory for this client, we could just write:

$ wl container mount :/forests/pandora:/docs/Container\ Addressing/:

The difference is the result of the fact, that Wildland is mounted in the filesystem starting with the forest of the default user. In other words, you always use the default user's forest as an entry point to Wildland. The default user can be changed while starting Wildland, as illustrated below, for Yatima (it is important to stop Wildland wl stop before attempting to start it again):

$ wl start --default-user yatima
$ tree ~/wildland
/home/user/wildland
└── tests
 └── My Container

and for Ariadne:

$ wl start --default-user ariadne.wildland.io
$ tree ~/wildland
/home/user/wildland
└── forests
    └── pandora:
        └── WILDLAND-FOREST.txt

This applies to mounting both containers and forests.

# Exploring a mounted container

Containers as represented by directories:

$ ls -a1  ~/wildland/mydirs/ariadne:/forests/pandora:/docs/W2H\ paper/
.
..
.manifest.wildland.yaml
wildland-w2h-0.3.pdf
wildland-w2h-0.4-pre3.pdf
wildland-w2h.pdf

Note the special file .manifest.wildland.yaml which is created dynamically and contains a copy of the container manifest.

This container can, of course, be accessed through any of the paths/categories which it claims, in this case also /arch/W2H\ paper/, /persons/joanna/W2H\ paper/, and /.uuid/72e13f99-3704-4d94-a9ac-08f96148a1da, as well as @-combinations of these.

# Reading and editing containers

Before reading and editing a container, the user needs to determine which container should be read or edited. For example, to determine to which container the addressing-v0.8.pdf belongs to:

$ wl container find ~/wildland/mydirs/ariadne:/forests/pandora:/docs/@arch/Container\ Addressing/addressing-v0.8.pdf
Container: wildland:0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe:/.uuid/1efd235c-6ab9-40e7-9db2-9dd5f08360cb:
  Backend id: 5570a6f5-857f-412b-8fa8-b5f6a83ea164

With the use of wl container dump the user can read the manifest of the container.

$ wl container dump wildland:0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe:/.uuid/1efd235c-6ab9-40e7-9db2-9dd5f08360cb:
object: container
owner: '0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe'
paths:
- /.uuid/1efd235c-6ab9-40e7-9db2-9dd5f08360cb
backends:
  storage:
  - reference-container: 'wildland:@default:/.infra:/home/joanna:'
    subdirectory: /addressing
    backend-id: 5570a6f5-857f-412b-8fa8-b5f6a83ea164
    type: delegate
title: Container Addressing
categories:
- /docs
- /arch
- /persons/joanna
version: '1'
access:
- user: '*'

Reading the manifest gives information about the locations of the container as the user can see all the categories and paths leading to the container. One of the possible paths is listed below:

$ tree -a ~/wildland/mydirs/ariadne:/forests/pandora:/docs/Container\ Addressing/
/home/user/wildland/mydirs/ariadne:/forests/pandora:/docs/Container Addressing/
├── .DS_Store
├── .manifest.wildland.yaml
├── addressing poster
│   ├── Addressing Poster.pdf
│   └── Poster.graffle
├── addressing-v0.8.pdf
├── addressing.git
└── addressing.md

Inside the container the user can find the .manifest.wildland.yaml file, which contains the most important information about the container, like the owner, paths and categories.

$ cat ~/wildland/mydirs/ariadne:/forests/pandora:/docs/Container\ Addressing/.manifest.wildland.yaml
object: container
owner: '0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe'
paths:
- /.uuid/1efd235c-6ab9-40e7-9db2-9dd5f08360cb
title: Container Addressing
categories:
- /docs
- /arch
- /persons/joanna
version: '1'
access: []

File .manifest.wildland.yaml is for information only. To actually edit the Container Addressing container's manifest, you need to use the path located with the use of wl container find above:

$ wl container edit wildland:0x1ea3909882be658d0ab69a822f7c923d12454ec024f4d8dd8f7113465167fcbe:/.uuid/1efd235c-6ab9-40e7-9db2-9dd5f08360cb:

Please note that the default editor in Docker image is set to vim. If it is not your cup of tea, you can change it with:

$ EDITOR=nano

Yatima is not the owner of the container and modification of the container above is impossible. If Yatima were a member of the group owner, modyfying the container would be possible. But Yatima is the owner of My Container and can modify it. To see all containers:

$ wl container ls
/home/user/.config/wildland/containers/8dc65644-c87e-4521-9719-6de4e1e211e6.container.yaml
  owner: 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072
  path: /.uuid/8dc65644-c87e-4521-9719-6de4e1e211e6
  path: /tests/My Container
  storage: type: webdav backend_id: 1189ea88-3ca6-43aa-a17b-936e77a1eac0

/home/user/.config/wildland/containers/yatima-forest-catalog.container.yaml
  owner: 0x0257ea023cfa9e88ed36f5f35265f2bce36e039579c832b3a3c6780418625072
  path: /.uuid/3f386d7b-b973-4047-93ec-20a6df13b641
  path: /.manifests
  storage: type: webdav backend_id: ef4e0e50-24fc-43d3-a140-b586a3690684

To edit My Container:

$ wl container edit 8dc65644-c87e-4521-9719-6de4e1e211e6

# Switching container storage backends

One of the killer features that Wildland aspires to bring to end users is the decoupling of data from the underlying infrastructure. This requires, in addition to abstracted addressing of data (as we just saw), to give the user an easy way to switch storage backends for each container.

This process involves a few aspects:

  1. Ability to easily obtain (buy/lease) new storage infrastructure for use,

  2. Ability to express access credentials to this infrastructure in a from of storage templates and/or manifests,

  3. Ability to attach/detach manifests to/from existing containers,

  4. Automation of the data migration between old and new storage backends.

The current implementation supports 2 and 3. 1 requires a marketplace integrated with Wildland and is not ready yet.

Regarding 4, the migration process is currently manual.

# Current limitations

  • Current implementation has not been optimized for performance in any major way, no caching is used for container content, mounting any larger forests is very slow. This has been intentional, as we want to be able to better test what is happening under the hood. Performance improvements are one of the major topics planned for the upcoming releases.

  • Currently only the Linux platform is supported directly, but the offered Docker image, which exposes the filesystem via a built-in WebDAV server, allows for reasonable use on other platforms, such as macOS and Windows. Right now the macOS version is in advanced development, and a Web-based client is in its initial phase. We will be releasing them separately in the coming months.

  • Current UI/UX is command-line only and very low-level. Some operations, especially those that apply to multi-user forests or batch-modifications of multiple containers, require many rudimentary steps to be performed manually. We plan significant UX improvements, especially with regards to bulk container management operations, in the upcoming versions.

  • There is no integrated marketplace for (automatic) leasing of storage backends. This is a major feature that we're working on.

  • There is a limited number of supported backends, although the ones we already have -- such as S3, WebDAV, Dropbox and Google Drive -- already let us embrace quite a sizable part of the infrastructure for use with Wildland.

# Further reading

We maintain a collection of tutorials with step-by-setup instructions for many different flows and features:

  1. A forest is a namespace built around one user identity. ↩︎

← Public Forests →