Creating and Managing Storage Providers using the SDK or CLI

This page provides instructions on how to create and manage storage providers using the SDK or the CLI. Each storage provider is associated with a backend storage provider, such as S3 or GCS, and a set of credentials that allow access to the backend. The credentials are encrypted and stored in the Rig database. The providers additionally contain a set of buckets that are associated with the provider.

---

Creating Providers

To create storage providers, you can utilize the CreateProvider endpoint. When making the request it is necessary to provide a unique name for the provider, and a config object for a particular backend. What this config object contains is different for each backend:

• An S3 Config object contains a default region and a set of credentials.
• A Minio config object contains a set of credentials, and an endpoint to connect to.
• For GCS, the config should contain a service account key in JSON format.

The Create Provider endpoint also takes a boolean LinkBucketsparameter. When true, the provider will be linked to all existing buckets in the backend for that configuration. Below is an example of creating a provider for GCS using the SDK and the CLI.

Golang SDK

// Create a provider for Google Cloud Storage
resp, err := client.Storage().CreateProvider(ctx, connect.NewRequest(&storage.CreateProviderRequest{
  Name: "my-provider",
  Config: &storage.Config_Gcs{
    Gcs: &storage.GcsConfig{
        Config: buf,                // The service account key in JSON format.
    },
  },
  LinkBuckets: true,
}))
if err != nil {
  log.Fatal(err)
}
log.Printf("successfully created provider \n")

Typescript SDK

// Create a provider for Google Cloud Storage
const resp = await client.storage.createProvider({
  name: "my-provider",
  config: {
    config: {
      case: "gcs",
      value: {
        config: buffer, // The service account key in JSON format.
      },
    },
  },
  linkBuckets: true,
});
console.log("successfully created provider");

CLI

rig storage create-provider --name --link-buckets --gcs --s3 --minio --credsfile --access-key --secret-key --region --endpoint

It is also possible to provide the configs as flags. Here the type of provider is specified using one of the three binary flags --gcs, --s3, or --minio. The flags --credsfile, --access-key, --secret-key, --region, and --endpoint are then used to provide the config object. The --link-buckets flag is used to specify whether the provider should be linked to all existing buckets in the backend.

Example:

rig storage create-provider -n my-gcs-provider -l --gcs -c ./service-account-key.json

If the fields for the provider are not specified, they will be prompted interactively.

---

Deleting Providers

To delete a provider, simply provide the ID of the provider to the DeleteProvider endpoint. This will delete the provider and all the associated credentials from the Rig database. Using the CLI, you can also provide a provider name instead of an ID, as these are also unique. It is important to note, that this will not delete buckets or files in the backend, but simply remove the link to Rig.

Golang SDK

// Delete a provider for Google Cloud Storage
providerID := "" // NOTE: insert providerID here
resp, err := client.Storage().DeleteProvider(ctx, connect.NewRequest(&storage.DeleteProviderRequest{
  ProviderID: providerID
}))
if err != nil {
  log.Fatal(err)
}
log.Printf("successfully deleted provider \n")

Typescript SDK

// Delete a provider for Google Cloud Storage
const providerID = ""; // NOTE: insert providerID here
const resp = await client.storagesClient.deleteProvider({
  providerID: providerID,
});
console.log("successfully deleted provider");

CLI

rig storage delete-provider [id | name]

Example:

rig storage delete-provider my-gcs-provider

---

Fetching Providers

Getting a provider

Getting a single provider can be done using the GetProviderendpoint, which takes a provider ID as input, or using the LookupProvider endpoint which takes a provider name as input. In the CLI these are gathered under a single command.

Golang SDK

// Get a provider
providerID := "" // NOTE: insert providerID here
resp, err := client.Storage().GetProvider(ctx, connect.NewRequest(&storage.GetProviderRequest{
  ProviderID: providerID
}))
if err != nil {
  log.Fatal(err)
}
log.Printf("Succesfully retrieved provider %s \n", resp.Msg.GetProvider().GetName())

// Lookup a provider
resp, err := client.Storage().LookupProvider(ctx, connect.NewRequest(&storage.LookupProviderRequest{
  Name: "my-provider"
}))
if err != nil {
  log.Fatal(err)
}
log.Println("Succesfully retrieved provider %s", resp.Msg.GetProvider().GetName())

Typescript SDK

// Get a provider
providerID = ""; // NOTE: insert providerID here
const resp = await client.storagesClient.getProvider({
  providerID: providerID,
});
console.log("Succesfully retrieved provider", resp.provider);

// Lookup a provider
const resp = await client.storage.lookupProvider({
  name: "my-provider",
});
console.log("Succesfully retrieved provider", resp.provider);

CLI

rig storage get-provider [id | name] --json

Example:

rig storage get-provider my-gcs-provider --json

Listing providers

Listing providers can be done using the ListProviders endpoint. Pagination can be implemented using the Offset and Limit fields. The Offset field determines the starting point of the list, while the Limit field specifies the maximum number of providers to retrieve in each request.

Golang SDK

resp, err := client.Storage().ListProviders(ctx, connect.NewRequest(&storage.ListProvidersRequest{
    Pagination: &model.Pagination{
        Offset: 10,
        Limit:  10,
    },
}))
if err != nil {
    log.Fatal(err)
}
log.Printf("successfully fetched %d storage providers. Total amount is: %d \n", len(resp.Msg.GetProviders()), resp.Msg.GetTotal())

Typescript SDK

const resp = await client.storagesClient.listProviders({
  pagination: {
    offset: 10,
    limit: 10,
  },
});
console.log(
  `successfully fetched ${resp.providers.length} storage providers. Total amount is: ${resp.total}`,
);

CLI

rig storage list-providers --offset --limit --json

Example:

rig storage list-providers -o 10 -l 10

---

Was this page helpful?

---