Keep artifacts local in a cloud instance¶

If you want to default to keeping artifacts local in a cloud instance, enable keep_artifacts_local.

import lamindb as ln

ln.settings.transform.stem_uid = "l9lFf83aPwRc"
ln.settings.transform.version = "1"
ln.track()

💡 connected lamindb: testuser1/keep-artifacts-local

💡 notebook imports: lamindb==0.72.0

💡 saved: Transform(version='1', uid='l9lFf83aPwRc5zKv', name='Keep artifacts local in a cloud instance', key='keep-artifacts-local', type='notebook', updated_at=2024-05-20 08:58:50 UTC, created_by_id=1)

💡 saved: Run(uid='zK2UjFG5IJ0rOTA6PKj1', transform_id=1, created_by_id=1)

You can register a managed local storage location as follows:

ln.settings.storage_local = "./my_storage_local"

💡 defaulting to local storage: /home/runner/work/lamindb/lamindb/docs/faq/my_storage_local

Now, you have two storage locations: one in the S3 bucket, and the other locally.

ln.Storage.df()

Show code cell output Hide code cell output

	created_at	created_by_id	run_id	updated_at	uid	root	description	type	region	instance_uid
id
2	2024-05-20 08:58:51.856155+00:00	1	None	2024-05-20 08:58:51.856236+00:00	qObmUkhYU2wo	/home/runner/work/lamindb/lamindb/docs/faq/my_...	None	local	None	6uGWmLpZlNoJ
1	2024-05-20 08:58:46.241792+00:00	1	None	2024-05-20 08:58:46.241858+00:00	dY6koACZwpEo	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ

Update storage description¶

You can add a description to the storage by using the description parameter:

storage_record = ln.Storage.filter(root=ln.settings.storage_local).one()
storage_record.description = "Files stored locally in site X on server Y for reason ABC"
storage_record.save()
ln.Storage.df()

	created_at	created_by_id	run_id	updated_at	uid	root	description	type	region	instance_uid
id
2	2024-05-20 08:58:51.856155+00:00	1	None	2024-05-20 08:58:51.894804+00:00	qObmUkhYU2wo	/home/runner/work/lamindb/lamindb/docs/faq/my_...	Files stored locally in site X on server Y for...	local	None	6uGWmLpZlNoJ
1	2024-05-20 08:58:46.241792+00:00	1	None	2024-05-20 08:58:46.241858+00:00	dY6koACZwpEo	s3://lamindb-ci/keep-artifacts-local	None	s3	us-west-1	6uGWmLpZlNoJ

Use local storage¶

If you save an artifact, by default, it’s stored in local storage.

original_filepath = ln.core.datasets.file_fcs()
artifact = ln.Artifact(original_filepath, description="My fcs file").save()
local_path = artifact.path
local_path

You’ll see the .fcs file named by the uid in your .lamindb/ directory under ./my_storage_local/:

ln.settings.storage_local.view_tree()

If you’d like to upload an artifact, you pass upload=True to the save() method.

artifact.save(upload=True)

You now see the artifact in the S3 bucket:

ln.setup.settings.storage.root.view_tree()

And it’s no longer present in local storage:

ln.setup.settings.instance.storage_local.root.view_tree()

Direct upload¶

You can also directly upload a file by passing upload=True:

filepath = ln.core.datasets.file_mini_csv()
artifact2 = ln.Artifact(filepath, description="My csv file").save(upload=True)
artifact2.path

Now we have two files on S3:

ln.Artifact.df(include="storage__root")

Show code cell output Hide code cell output

	storage__root	version	created_at	created_by_id	updated_at	uid	storage_id	key	suffix	accessor	description	size	hash	hash_type	n_objects	n_observations	transform_id	run_id	visibility	key_is_virtual
id
2	s3://lamindb-ci/keep-artifacts-local	None	2024-05-20 08:58:53.059914+00:00	1	2024-05-20 08:58:53.059969+00:00	OLtZPqdQn44xgSfQhcIQ	1	None	.csv	None	My csv file	11	z1LdF2qN4cN0M2sXrcW8aw	md5	None	None	1	1	1	True
1	s3://lamindb-ci/keep-artifacts-local	None	2024-05-20 08:58:52.759403+00:00	1	2024-05-20 08:58:52.797539+00:00	frNkuOvJ6I19NOcmsWI3	1	None	.fcs	None	My fcs file	6785467	KCEXRahJ-Ui9Y6nksQ8z1A	md5	None	None	1	1	1	True

Pre-existing artifacts¶

Assume we already have a file in our registered local storage location:

1 sub-directory & 2 files with suffixes '', '.bam'
/home/runner/work/lamindb/lamindb/docs/faq/my_storage_local
├── .lamindb
│   └── _is_initialized
└── output.bam

If we create an artifact from it, it remains where it is during saving:

my_existing_file = ln.Artifact("./my_storage_local/output.bam", description="my existing file").save()
ln.UPath("my_storage_local/").view_tree()

The storage path of the artifact is constructed using key because key_is_virtual=False:

my_existing_file

However, if we decide to upload the artifact, we’ll use the uid for constructing the storage path and switch key_is_virtual=True:

my_existing_file.save(upload=True)

Here is the remote path of the artifact:

my_existing_file.path

And here are the contents of the storage locations:

# the path on S3
ln.setup.settings.storage.root.view_tree()
# the local path
ln.setup.settings.instance.storage_local.root.view_tree()

Delete the test instance¶

Delete the artifacts:

artifact.delete(permanent=True)
artifact2.delete(permanent=True)
my_existing_file.delete(permanent=True)

Delete the instance:

ln.setup.delete("keep-artifacts-local", force=True)