⚠️ Warning: This documentation refers to the proof-of-concept implementation of Wildland client (opens new window) which is no longer maintained. We are currently working on a new Wildland client written in Rust. To learn more about Wildland and the current status of its development, please visit the Wildland.io webpage (opens new window).

# Transpose Storage Backend

# What & Why

When handling large amounts of Wildland containers (whether automatically generated by some subcontainer-based plugin or imported from some source), one can find themselves struggling to maintain a clear directory structure that is intuitive and easy to navigate. Categories of those types of containers sometimes do not suit one's needs fully and often times it's impossible to change them manually. The transpose storage backend has been created with these issues in mind; thus, its main aim is to enable the user to better control the categorization of large amounts of containers so that one's data can be accessed easily and intuitively.

# How

The transpose storage backend is one of Wildland's proxy backends - it does not hold or manage any data itself, but instead helps adjust other containers to one's needs.

As with many other Wildland storage backends, there are a few parameters that the user will be prompted to provide upon creating a transpose-type storage:

--reference-container-url - a Wildland url (opens new window) pointing to the container whose children containers' categories are to be modified.
--rules - a number of adequately formatted rules to be used when modifying said categories.
--conflict - a keyword describing how the backend is supposed to behave whenever faced with conflicting rules. This parameter should take one of the three following values:
- first-apply - this indicates that when the backend is faced with some conflicting rules, it should follow the first of them
- last-apply - this indicates that when the backend is faced with some conflicting rules, it should follow the last of them
- all-apply - this is a special keyword to be used when one wants to perform multiple modifications on one category (e.g. in case of the match-category-regex rules)

# Creating the rule list

In order to create a storage of the transpose type, one needs to establish what rules are to be followed whe modifying the categories of the containers. Each rule passed to the backend should be expressed as a Python dictionary enclosed in single quotes and follow the syntax below:

'{"match-category": "/chosen/category", "{rule_keyword}": {expected_behaviour}}'

Or, when one wishes to modify a number of categories at once by utilizing regular expressions:

'{"match-category-regex": "/pattern", "replace-with": "/replacement"}'

When using the "match-category" rule type, the following rule keywords can be used:

"replace-with" - this will replace the category specified in the "match-category" field with an arbitrary category of choice. For example, to replace /category/one with /category/two, the following rule can be used: '{"match-with": "/category/one", "replace-with": "/category/two"}'
"include" - when this keyword is used, the backend will only include categories that are explicitly stated to be included. For example, to only include /category/one and /category/two in one's modified set of categories, the following rules can be used: '{"match-with": "/category/one", "include": True}' '{"match-with": "/category/two", "include": True}'
"exclude" - on the other hand, the exclude keyword can be used to explicitly specify which rules not to include in one's modified set of categories. For example, to ensure that /category/one will not appear in the modified set of categories, the following rule can be used: '{"match-with": "/category/one", "exclude": True}'

The "match-category-regex" rule type can be used to modify an existing category (e.g. by adding a prefix or only changing a part of it) by utilizing regular expressions.
- For example, if one wishes to add a prefix category of '/p' to every category, they can use the following rule which utilizes capture groups and general regular expression syntax: '{"match-category-regex": "/(.*)", "replace-with": r"/p/\1"}' Note: It is important to use the raw string indicator (r"string") in the "replace-with" part of the rule whenever backreferencing is a part of the rule.
- If one wishes to modify every occurence of /timeline with /t (for example to better incorporate a part of an imported forest into one's own timeline tree), the following rule can be used: '{"match-category-regex": "/timeline", "replace-with": "/t"}' To find out more about the syntax of regular expressions, refer to the documentation of the re module (opens new window).

# Transpose storage creation

The transpose storage backend cannot be used on it's own - instead, it is meant to be used with an existing set of containers. For this reason, it is recommended you follow the Forest Creation (opens new window) tutorial and set up your own personal forest prior to following this tutorial.

To begin the tutorial, mount the Wildland filesystem with the following command:

wl start

After the initial set up, you can create and publish a number of Wildland containers to the newly created forest - this will help showcase the backend. To create a Wildland container, use the following command:

wl container create container1 --title "Container1" \
                               --category /category/one \
                               --category /category/two \
                               --category /timeline/2021/09/02

Now you can add some storage to your new container. You can find out more about the types of storages that are currently available in the part of the guide dedicated to Storage backends (opens new window).In this tutorial, we choose to utilize the static storage type, as it will enable us to showcase the backend fully.

wl storage create static --container container1 \
                         --file "MyFile.md=Example text!"

Let us follow the commands one more time to add one more container - please note the change in categories granted to the container.

wl container create container2 --title "Container2" \
                               --category /category/one \
                               --category /category/three \
                               --category /timeline/2021/02/09
                               
wl storage create static --container container2 \
                         --file "AnotherFile.md=Here's another file!"

You can repeat these two commands as many times as you want, effectively adding more and more containers to your storage.

Let us go through the parameters necessary for the creation of the transpose storage container:

reference-container-url - when the transpose container is to be used with forests (whether personal or imported from some other user), the reference-container-url should be the Wildland url (opens new window) pointing to the forest's manifest catalog. This container is usually located under the /.manifests path in the forest root. Assuming that the owner of the forest above is the @default-owner, the parameter will take the wildland:@default-owner:/.manifests: .
conflict - in this tutorial, we will use the all-apply conflict resolution keyword.
rules - suppose we need to perform three different modifications on this forest. First, we want to replace the /category/one with /replaced/category. Next, we want to change the root of the timeline tree from /timeline to /calendar. Finally, we want to add a prefix of /forests/personal to all of the categories so that we can easily navigate between our personal and imported forests. Thus, we will need to utilize the following three rules:
- '{"match-with": "/category/one", "replace-with": "/replaced/category"}'
- '{"match-category-regex": "/timeline", "replace-with": "/calendar"}'
- '{"match-category-regex": "/(.*)", "replace-with": r"/forests/personal/\1"}'

To create a container for the transpose storage, the following command can be used:

wl container create transposeContainer

To add the transpose storage to this container, the following command (which utilizes the parameters above) can be used:

wl storage create transpose --container transposeContainer \
                            --reference-container-url wildland:@default-owner:/.manifests: \
                            --conflict all-apply \
                            --rules '{"match-with": "/category/one", "replace-with": "/replaced/category"}' \
                            --rules '{"match-category-regex": "/timeline", "replace-with": "/calendar"}' \
                            --rules '{"match-category-regex": "/(.*)", "replace-with": r"/forests/personal/\1"}'

The container can then be mounted using the following command:

wl container mount transposeContainer

We should now be able to access the content of the containers via the modified category paths.

Though the transpose storage backend can be used to change the categorization of the user's personal forest, it has other usecases as well:

Can be used with imported forests - in this usecase, the user should simply point the backend at the imported forest's manifest catalog (usually located under the /.manifests path in that forest's root)
Can be used with subcontainer-yielding backends - in this usecase, the backend can be pointed at the container utilizing the subcontainer-yielding backend via any of the container's paths. This usecase primarly concerns the IMAP and Gitlab storage backends.

← Caching Storages Wildland Paper (opens new window) →