ALERT!

*The Starlane crates and github repository are * NOT currently in a working state!

A new update is coming soon at which time the docs and tutorials will be updated.

DOCUMENTATION

GETTING STARTED

INSTALL RUST

To build Starlane you will need to have rust installed. Follow the official Rust instructions to install Rust.

NIGHTLY TOOLCHAIN

Starlane requires the nightly toolchain for compilation. To switch to the nightly toolchain:

rustup toolchain install nightly
rustup default nightly

BUILD AND INSTALL STARLANE

cargo install starlane

Congrats! You now have Starlane installed on your machine!

START A STARLANE SERVER INSTANCE

Open a terminal and run the following command to start a server instance of Starlane:

starlane

At this point starlane should be serving a Http Server on port 8080. Open a browser and point it to http://localhost:8080/. You should see a "404" page (since there isn't a localhost space or routing bind.)

COSMIC CLI

You need to have the cosmic-cli installed in order to access your starlane instance:

cargo install cosmic-cli

TERMS

The Cosmic Initiative uses terms with an Astro Physics them in order to avoid name collision with other domains in the architecture. For example a compute Node is called a Star and a Resource is called a Particle.

UNIVERSAL TERMS

These are terms that exist in the Universal Dimension which is not concerned with security or orchestration. Applications are typically built in this dimension in order to quicken development and centralize complexity.

  • Particle - A resource (User,File,Database, Database Row, Mechtron...etc.)
  • Kind - a hierarchtical type descriptor for a particle. A kind can be composed of a nested Base, Sub & a Specific to exactly describe a particle
  • Point - an address that in most cases is the location of a Particle
  • Wave - A message that is transmitted to and from Particles
  • Mechtron - A WebAssembly component that implements the cosmic-universe & mechtron packages giving it the ability to act as a Particle (send and receive Waves)
  • Cosmos - The entirety of the "World" that these Particles live in... The Entire Enterprise.
  • Cosmic Fabric The space between Particles, the space that Waves traverse
  • Universal Dimension - A simplified view and API of the Cosmos that is not concerned with security or orchestration.

HYPERVERSAL TERMS

The Hyperversal Dimension is the infrastructure view of the Cosmos which is responsible for security & orchestration and manages these concepts on behalf of the Universal Dimension.

  • Star - A node/container for managing state and execution of Particles. The Hyperverse distributes its provisioning of Particles amongst its Stars in order to spread computation load for storage, cpu & memory.
  • Lane - Stars are connected via Lanes which serves as the transit mechanism for Waves
    • Driver - Particles are supported through Drivers. Each Particle Kind has exactly one Driver
  • Registry The Registry holds important information on where Particles live and security rules
  • Machine - Although Stars are the 'node' component for managing Particles--the stars are more of 'virtual nodes' that live inside a Machine. The Machine will connect the internal Lanes between Stars, provide a service Stars within other Machines to connect & the Machine manages clients on behalf of its stars that are required to connect to other external Stars. This architecture facilitates the rearrangement of infrastructure without the Stars needing any special knowledge of the Hyperverse cluster that it resides in. For example in the standalone configuration ALL of the Stars execute on one Machine, and in yet another configuration each Star may have it's own Machine but in both cases the Stars see the Hyperverse the same without needing any special knowledge of how the Hyperverse cluster is composed.

PARTICLES POINTERS & KINDS

PARTICLES

A Particle is a resource which can send and receive Waves (messages).

KIND

Particles differ in functionality by their Kind. For example a User Particle will accept different types of Waves than a FileSystem Particle. They do different things.

DENOTATION

Kinds are denoted in angle brackets:

<Space>

Space is a single Base Kind without any Sub Kinds.

SUBS

Kinds can be hierarchical and are notated using delimited angle brackets like so:

<Base<Sub>>

Above you can see that a Base Kind and a Sub Kind are denoted.

For example to reference a Relational database you would denote it like this:

<Database<Rel>>

And You would reference a Graph database like this:

<Database<Graph>>

The more specific the Sub Kind the more custom functionality you can access in the Particle's API.

SPECIFIC

There is one more rung in the Kind hierarchy which is a Specific.

Here is an example of a Specific:

mechtronhub.io:postgres.org:postgres:gis:12.0.0

And it would go with the rest of our Kind definition like this:

<Database<Rel<mechtronhub.io:postgres.org:postgres:gis:12.0.0>>>

A Specific identifies an EXACT implementation of a Particle.

The various parts of the specific can be broken down as follows:

provider-domain:vendor-domain:product:variant:version

Provider Domain is the provider of the one supplying the Driver for the given Kind... which may not be the same as the vendor that actually supplies the product.

POINT

Every particle has a Point which acts as an address and unique identifier. The Particles Point is used in security, searching, message routing, logging and more.

Points are hierarchical and are usually delimited by a ':'. Here is an example of a Point:

my-domain.com:users:my-user

Notice above the first Point Segment my-domain.com is a domain name and in fact the first segment MUST be a lowercase valid hostname or domain name. The first segment is also required to be a Space Kind.

Also note that only the Space segment is allowed to have dots in it, the rest of the Base segments are lowercase kebab.

POINT + KIND

Point & Kind can be put together when referencing a Particle like this:

my-domain.com:users:my-user<User>

As you can see the Particle is identified by Point and Kind by immediately appending the Kind denotation after the Point.

FILESYSTEM POINTS

When referencing the children of a FileSystem Particle the Point Rules change a little.

my-domain.com:my-files:/some-file.txt

Notice that there is now a leading slash (which identifies the root of the FileSystem). Once the point segments pass a slash the rules change so that segments are delimited by slashes, and they can have any valid file characters.

Here is Another example showing directory structure:

my-domain.com:my-files:/some-dir/My-file.txt

Again, notice that My-file.txt has a capital letter in it, not allowed in the regular Base segments, but perfectly allowed once the transition is made to FileSystem segments.

ARTIFACT BUNDLES

Artifact Bundles are zip files that can be downloaded and cached by other services. They exist in a repo and are versioned. A Bundle must live underneath an ArtifactBundleSeries which in turn must be under a Repo.

Since a Bundle is versioned its Point Segment is a version number:

my-domain:repo:my-bundle-series:1.0.0

It is required that after a version is declared remaining segments will use FileSystem style which makes sense because an Artifact Bundle is really just a zip file. In order to reference an artifact:

my-domain:repo:my-bundle-series:1.0.0:/bind/my-particle.bind

SELECTING

The following describes how to select using the Command Line Interface. Cli can also be used within Mechtrons and Controls

To start with you can select a Particle by its exact name but that isn't very useful other than to test if it exists or not:

select localhost

And you can select a list of Particles you can use a very familiar wildcard:

select my-domain.com:*

If you want to get a recursive result use **:

select my-domain.com:**

What if you want to get a recursive result but you also want to include my-domain.com ? You can use the inclusive operator:

select my-domain.com:+**

If you want to select by Kind you can do that too:

select **<User>

The above will return ALL users in the Cosmos.

You can combine Point & Kind selects:

select my-domain.com:**<User>

Also you can wildcard a Sub Kind:

select my-domain.com:**<Database<*>>

For Artifact Bundles you can use SemVer matching rules:

select my-domain.com:repo:my-app:(^3.0.0);

The above will return all Bundles with a version 3.0.0 and greater

AND you can select for a Specific Version:

select my-domain.com:**<Database<Rel<mechtronhub.io:postgres.org:postgres:*:(^9.0.0)>>>

The above will return all Postgres databases above version 9.0.0 any variant.

CREATE

You can also create Particles from the Cli. To do so you must specify a Point and a Kind:

create my-domain.com<Space>

If you are creating a Particle with a Specific you can use Selection rules so that the Hyperverse will provision the best match for your needs. For example:

create my-domain:users<UserBase<OAuth<mechtronhub.io:keycloak.com:keycloak:*:(^10.0.0)>>>;

Above we are asking to create an instance of Keycloak, any variant version 10.0.0 or above. The Hyperverse should select the highest version that it has available that matches the given criteria.

PROPERTIES

Sometimes passing configuration properties is required when creating a Particle. For example an App MUST have a config:

create my-domain.com:app<App>{ +config=my-domain.com:repo:company-app:1.0.0:/config/my-app.config };

Above the realm of properties is entered using the curly brackets and adding new property definitions by using the + operator with assigned value.

ARGS

A particular Bind/Config may also require Args to be passed to it.

IMPORTANT: Properties are part of the Kind and Args are expected by the Bind and/or the Config. Try not to get them mixed up!

Pass args by using Parenthesis:

create my-domain.com:app<App>( company-name="ACME" )

ENVS

Finally, you can set Environment variables when creating a Particle by using square brackets:

create my-domain.com:app<App>[ users=my-domain.com:users ]

Environment variables Are also part of the Bind & Config however their values cascade down to all child references meaning that a Particle at point domain.com:users:my-user would have all the environment variables that were set from domain.com & domain.com:users

ALL TOGETHER NOW

And of course you can chain them together into one monster create statement:

create my-domain.com:app<App>{ +config=my-domain.com:repo:company-app:1.0.0:/config/my-app.config }( company-name="ACME" )[ users=my-domain.com:users ]

GET, SET & DELETE

GET

To get a property value:

get my-domain.com:users:uberscott{ email }

ENV

And if you want to grab an environment variable:

get my-domain.com:app:mechtrons:my-mechtron[ app ]

ARG

And for args:

get my-domain.com:app:mechtrons:my-mechtron( company-name )

SET

To set a property value:

set my-domain.com{ +bind=my-domain.com:repo:my-site:2.7.3:bind/routes.bind }

The Same goes for environment variables:

set my-domain.com[ slogan="We can do it!" ]

REMOVE

You can remove a Property if it is allowed:

set my-domain.com{ !bind }

ARGS NOT ACCESSIBLE

Args which are used in the process of creation of a particle cannot be modified by Set

DELETE

When you grow tired of your Particle you can delete it:

delete my-domain.com:app

The above will delete recursively, but you can also delete selectively:

delete my-domain.com:app:**<Mechtron>

SECURITY

AGENT

Every Wave that is sent from one Particle to another references an Agent which is just a Point that access rules can be applied to. Usually a Particle sends itself as the Agent, but there are cases where It can use other Agents if the Hyperverse allows it.

HYPERUSER

The HyperUser is the only user that access is not checked on. It can do anything (just like the Root user in unix OS.)

GRANT PERMISSION

to grant permission on a particle to a particle:

grant perm +csd-Rwx on my-domain.com:** to my-domain.com:**<User>

Above we are granting READ access to all users under my-domain.com on every particle under my-domain.com.

Let's examine this part: +csd-Rwx The letters stand for Create, Select, Delete, Read, Write and Execute. if the letter is capitalized then the permission is flagged as on, if lowercase it is off.

Also note the leading + operator. That is saying to OR these permissions with existing permissions. You can also choose to AND the permissions via the & operator which would have looked like this:

grant perm &csd-Rwx on my-domain.com:** to my-domain.com:**<User>

NOTE: The rules of how the Ands and the Ors are combined are consistent but a bit tricky and I will expand on it in a later iteration of this document -- uberscott

CSD

First let's break down CSD:

  • CREATE - Permission to create children under the Particle
  • SELECT - Permission to select this Particle and children
  • DELETE - Permission to delete this Particle and children

RWX

Let's break down the RWX:

  • READ - Permission to read the state of the Particle (not all particles have state)
  • WRITE - Permission to write the state of the Particle (not all particles have state)
  • EXECUTE - Permission to send Waves to the Particle

PRIVILEGES

Sometimes security has to be a little more fine-grained that broad permissions. For this we have Privileges.

One built in Privilege in the system is the User email property. Even if a Particle has full CSD-RWX access to a User Particle it cannot get the value of the email property unless it has been granted privilege to do so.

To grant a privilege:

grant priv prop:email<Read> on my-domain.com:users:**<User> to my-domain.com:my-app:mechtrons:emailer

As you can see above we are giving only one mechtron in our application, the emailer mecthtron the privilege to see the email addresses, all other mechtrons are untrusted.

LISTING GRANTS

You can list the grants on a particular Pattern:

grant list on my-domain.com:my-app+:**;

And that will result in a list with a numbered id with every grant. You can then revoke the grant using the number:

grant revoke 1

OWNERSHIP & CHOWN

When a Particle creates another Particle that Particle owns it and perform any action upon it full permissions and full privileges. To change ownership:

chown localhost:users:uberscott localhost:something;

CAPTURE MAPPING

There are times when you may want to capture and reuse a segment to make more complicated permission rules

grant perm +CSR-RWX on my-domain.com:profiles:$(user) to my-domain.com:users:${user}

Above the variable user is captured in profiles and filled in the UserBase giving each user complete access to his or her own profile.

SUPER

The HyperUser has complete access to the entire Cosmos, however it is also desirable to have SuperUsers which have complete access to only a portion of the overall cosmos. To accomplish this we can grant another user SuperUser access:

grant super on my-domain.com:** to my-domain.com:users:what-a-great-guy<User>

Above you can see that what-a-great-guy has been granted superuser access to everything under the my-domain, and he will have full access to anything that matches that pattern,

The use of SuperUser can be particularly useful for Apps which may need to control every aspect of their children:

grant super on my-domain.com:app:+** to my-domain.com:app<App>

above the App can now administer itself and its children with full access.

CONFIGURATION DOCS

Starlane features a rich DSL called Arrow Notation for configuration Docs.

Docs Bind & Config have very similar content and are therefore documented together:

BIND

Bind describes a contract between the particle and the Fabric... it enumerates the required args and environment variables that must be passed or set and requires other particles it should have access to. The Route shows the legal waves that other Particles can send to it but also describes the required pipeline those waves must travel. The Bind Cannot set Arg or Env Vars or Particles. Bind also restricts the permissions of the particle... A bind is sometimes set FOR a particle not BY the particle (and in some cases the bind is locked so the particle cannot change it.). this is for security and conformity ... it is saying "you can install any particle that is satisfied by and follows the rules of this bind"

CONFIG

Config is always under the control of the Particle. It has some overlap with Bind but is usually more powerful... for instance it can set/override Env vars whereas Bind can only set default values. Config also has the *Install selector which can create other particles and/or set permission ... basically it can access the Cli. Finally, a Config for a Mechtron or App will point to the actual Wasm guest binary.

CONTEXT

Points within the Bind & Config can be contextual, meaning they can be referenced relative to the Particle Point that the Bind or Config is describing.

For example within a Bind for my-domain:app instead of referencing a child Point like this:

my-domain:app:mechtron

It can instead be referenced using the working Point:

.:mechtron

Parent points can also be referenced using ..:

..:another-app

BUILTIN VARS

There are also some builtin Variables that can be used in a Document for example bundle:

${bundle}:/bind/special.bind

Above the ${bundle} var references the Artifact bundle that the document came from, this can be useful when you want to group related documents together in a release.

DOCUMENT DECL

The first part of a Doc is that it's Kind and Version must be declared:

Bind(version=1.0.0) { }

This tells the Doc parser what is being parsed and what version it must be translated from

Arg

The Arg Scope tells the Bind & Config which Arguments are expected to be passed to the Particle at creation time:

Bind(version=1.0.0) {
  Arg {
    # ensure company-name is set and is a <Text>
    company-name<Text>;

    # if num-employees is not set then throw the error message "Please set num-employees!"
    num-employees<Int> !! "Please set num-employees!";

    # if not set default to 'LCC'
    company-type<Text> ?= "LCC";
  }
}

And an example of a Config which can actually set vars:

App(version=1.0.0) {
  Arg {
    num-employees<Int> = 100;
  }
}

Env

Like Arg we can also ensure that environment variables are set. Environment variables are inherited from the parent.

Bind(version=1.0.0) {
  Env {
    slogan<Text> !! "everyone needs a slogan!"
  }
}

Environment Particles can also be required and set if needed:

Bind(version=1.0.0) {
  Env {
    # require that App has been set by a parent
    app<App>;

    users<UserBase> !! "must have a UserBase!";
  }
}

And the config for App:

App(version=1.0.0) {
  Env {
    # here the App is setting the environment variable app to itself so that the children of the app will have a reference to it
    app<App> = .;

    # if userbase does not exist, then create it
    users<UserBase> ?= create .:users<UserBase<OAuth<mechtronhub.io:keycloak.com:keycloak:community:(^10.0.0)>>>;
  }
}

The above example is interesting because the App can be provided with a UserBase (in the case where you may want to share the UserBase with several apps) but if not present it will create its own UserBase.

INSTALL

Config has a special document scope Install which allows for custom installation scripts:

App(version=1.0.0) {

  # Let's set some Env vars first
  Env {
    # so all child mechrons can reference this App
    app<App> = .;

    # let's ensure there is a UserBase for us (create one if not)
    users<UserBase> ?= create .:users<UserBase<OAuth<mechtronhub.io:keycloak.com:keycloak:community:(^10.0.0)>>>;
  }

  Install {
    # set the bind property
    set .{ bind=${bundle}/bind/my-app.bind };

    # let's create a place to hold our Mechtrons
    create .:mechs<Base>;

    # and a place for holding user profiles
    create .:profiles<Base>;

    # Create a specific mechtron, notice we can reference
    # The bundle that THIS document came from using ${bundle} variable
    create .:mechs:my-mechtron<Mechtron>{ config=${bundle}:/config/my-mechtron.config +bind=${bundle}/my-mechtron.bind };

    # create the emailer
    create .:mechs:emailer<Mechtron>{ config=${bundle}:/config/emailer.config +bind=${bundle}/emailer.bind };

    # you can see we are now referencing env.users variable:
    grant perm +cSd-RwX on ${env.users}:**<User> to .:mechs:**;

    # the emailer needs to have Read permission on users' email addresses
    grant priv +prop:email<Read> on ${env.users}:*<User> to .:mechs:emailer;

    # users should have complete access to their profiles
    # notice we capture username from porfiles so it will match the username in user account
    grant +CSD-RWX on .:profiles:$(username) to ${env.users}:${username};
  }
}

There's a lot going on there, but you can see that some important mechtrons are being created which are part of composing this application at install time.

MECHTRONS & APPS

Mechtrons and Apps have special configurations because they reference WebAssembly binaries:

Mechtron(version=1.0.0) {

  Wasm {
    bin = ${bundle}/wasm/my-wasm.wasm;

    # a wasm MechtronGuest can contain many Kinds of mechtrons, so we need to tell it which one we are talking about
    mechtron = my-mechtron;
  }

}

BIND ROUTES & PIPELINES

One of the most important interfaces that the Bind provides are the Routes & Pipelines.

ROUTE

Routes are selectors which take a Directed Wave's Method & Path and if matched forward that Wave through the selected Pipeline.

PIPELINE

The pipeline describes the series of filters, steps and stops to process the Directed Wave.

GET EXAMPLE

Here is a simple Route & Pipeline definition in arrow notation:

Bind(version=1.0.0) {
  Route<Http<Get>>> -> my-domain:app => &;
}

Let's break this down:

THE ROUTE SELECTOR

  • Route - This tells the Bind config that we are dealing with a Route
  • Http - this selects the Kind of wave we want to intercept... in this case an Http Wave
  • Get - this is a standard Http Get Method

THE PIPELINE

If the incoming Directed Wave matches the Route Selector then it enters the pipeline:

  • -> - This is a Pipeline Step segment. It will take any Directed Wave and pass it forward
  • my-domain:app - This is a Pipeline Stop segment. The Wave will be routed to the my-domain:app Particle
  • => - This Pipeline Step filters for a Reflected Wave (denoted by => intead of -> )
  • & - The final Pipeline Stop segment indicates that the Pipeline is complete and the result should be returned

EXPANDABLE OR COLLAPSABLE

To Make things as easy to read as needed, the Route declarations can be expanded or collapsed in these varieties:

  Route<Http<Get>>> -> my-domain:app => &;

  # is the same as:

  Route {
    Http<Get> -> my-domain:app => &;
  }

  # which is also the same as:

  Route {
    Http {
      Get -> my-domain:app => &;
    }
  }

WILDCARDS

You can apply Wildcards for the Method:

Route<Http<*>> -> my-domain.com:deal-with-all-traffic => &;

AUTH FILTERS

Filters can be applied for Authorization:

Route {
  Http(auth) {
    Get -> my-domain:members-only => &;
    Post -> my-domain:members-only => &;
  }

  Http {
    Get -> my-domain:anonymous => &;
    Post -> my-domain:anonymous => &;
  }
}

PATH FILTERS

Path Filters can also be applied:

Route<Http<Get>>/some-path -> my-domain.com:app:process-some-path => &;

CORE STOP

Use the (()) operator to denote that the directed should now go into the core of the Mechtron for processing


Route {
   Http<Post> -> (()) => &;
}

CHAINING PIPELINES

You can chain your pipeline however you like to transform the incomming request or filter input:

Route<Http<Post>>/set-password -> my-domain:password-validator -> (()) => &;

Above the password validator will throw an error if it isn't valid. Also note that my-domain:password-validator returns a new Directed Wave in the body of its Reflected Wave which is grabbed when the -> Pipeline step is encoutered.

FILTERING BODY

The Body of a Directed and Reflected Wave can be filtered by Type:

Route<Http<Post>>/set-password -[ Text ]-> my-domain:password-validator -[ Text ]-> (()) =[ Json ]=> &;

If the submitter tries to send an image it will be rejected. And this bind enforces that the underlying Mechtron return Json.

VAR SUBSTITUTION

Some examples of Var substitution have already been shown, but it's good to know you can use Vars throughout the pipeline:

# redirect to app
<Http<Get>> -> ${env.app} => &;

PATH CAPTURING

And Path fragments can be captured using Regex and captured variables can be used later in the pipeline:

Route {
  Http(auth) {
     Post/users/(?P<user>.*)/profile-pic -> my-domain:app:profiles:${user} => &;
  }
}