Integrate scRNA-seq datasets#
!lamin load test-scrna
Show code cell output
💡 found cached instance metadata: /home/runner/.lamin/instance--testuser1--test-scrna.env
✅ loaded instance: testuser1/test-scrna
import lamindb as ln
import lnschema_bionty as lb
import pandas as pd
import anndata as ad
✅ loaded instance: testuser1/test-scrna (lamindb 0.51.0)
ln.track()
💡 notebook imports: anndata==0.9.2 lamindb==0.51.0 lnschema_bionty==0.30.0 pandas==1.5.3
✅ saved: Transform(id='agayZTonayqAz8', name='Integrate scRNA-seq datasets', short_name='scrna2', version='0', type=notebook, updated_at=2023-08-28 14:18:38, created_by_id='DzTjkKse')
✅ saved: Run(id='XLtF43RASfz8MnvMWglS', run_at=2023-08-28 14:18:38, transform_id='agayZTonayqAz8', created_by_id='DzTjkKse')
Query files based on metadata#
# lookup objects for auto-complete
assays = lb.ExperimentalFactor.lookup()
species = lb.Species.lookup()
query = ln.File.filter(
experimental_factors=assays.single_cell_rna_sequencing, # scRNA-seq
species=species.human, # human
cell_types__name__contains="monocyte", # monocyte
).distinct()
query.df()
| storage_id | key | suffix | accessor | description | version | initial_version_id | size | hash | hash_type | transform_id | run_id | updated_at | created_by_id | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| id | ||||||||||||||
| RAdyFo8MWTzVkqkFCK8T | ljWPEsjj | None | .h5ad | AnnData | 10x reference pbmc68k | None | None | 589484 | eKVXV5okt5YRYjySMTKGEw | md5 | Nv48yAceNSh8z8 | p3vdxLrlEIijgVQLdUd0 | 2023-08-28 14:18:31 | DzTjkKse |
| WtEvWQ5KVML36kWeCyJt | ljWPEsjj | None | .h5ad | AnnData | Conde22 | None | None | 28049505 | WEFcMZxJNmMiUOFrcSTaig | md5 | Nv48yAceNSh8z8 | p3vdxLrlEIijgVQLdUd0 | 2023-08-28 14:18:13 | DzTjkKse |
Intersect measured genes between two datasets#
# get file objects
file1, file2 = query.list()
file1.describe()
💡 File(id='RAdyFo8MWTzVkqkFCK8T', key=None, suffix='.h5ad', accessor='AnnData', description='10x reference pbmc68k', version=None, size=589484, hash='eKVXV5okt5YRYjySMTKGEw', hash_type='md5', created_at=2023-08-28 14:18:31, updated_at=2023-08-28 14:18:31)
Provenance:
🗃️ storage: Storage(id='ljWPEsjj', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-28 14:18:36, created_by_id='DzTjkKse')
📔 transform: Transform(id='Nv48yAceNSh8z8', name='Validate & register scRNA-seq datasets', short_name='scrna', version='0', type='notebook', updated_at=2023-08-28 14:18:31, created_by_id='DzTjkKse')
👣 run: Run(id='p3vdxLrlEIijgVQLdUd0', run_at=2023-08-28 14:17:25, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-28 14:18:36)
Features:
var (X):
🔗 index (695, bionty.Gene.id): ['asa6P3SWGqBF', 'sOu1hW4id709', 'mLZxpATriwGh', 'yo4j3UPxzM21', 'z4HRihQZPQ11'...]
external:
🔗 assay (1, bionty.ExperimentalFactor): ['single-cell RNA sequencing']
🔗 species (1, bionty.Species): ['human']
obs (metadata):
🔗 cell_type (9, bionty.CellType): ['cytotoxic T cell', 'CD38-negative naive B cell', 'effector memory CD4-positive, alpha-beta T cell, terminally differentiated', 'CD16-positive, CD56-dim natural killer cell, human', 'B cell, CD19-positive']
file1.view_lineage()
file2.describe()
💡 File(id='WtEvWQ5KVML36kWeCyJt', key=None, suffix='.h5ad', accessor='AnnData', description='Conde22', version=None, size=28049505, hash='WEFcMZxJNmMiUOFrcSTaig', hash_type='md5', created_at=2023-08-28 14:18:13, updated_at=2023-08-28 14:18:13)
Provenance:
🗃️ storage: Storage(id='ljWPEsjj', root='/home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna', type='local', updated_at=2023-08-28 14:18:36, created_by_id='DzTjkKse')
📔 transform: Transform(id='Nv48yAceNSh8z8', name='Validate & register scRNA-seq datasets', short_name='scrna', version='0', type='notebook', updated_at=2023-08-28 14:18:31, created_by_id='DzTjkKse')
👣 run: Run(id='p3vdxLrlEIijgVQLdUd0', run_at=2023-08-28 14:17:25, transform_id='Nv48yAceNSh8z8', created_by_id='DzTjkKse')
👤 created_by: User(id='DzTjkKse', handle='testuser1', email='testuser1@lamin.ai', name='Test User1', updated_at=2023-08-28 14:18:36)
Features:
var (X):
🔗 index (36503, bionty.Gene.id): ['hX0qP176evu9', 'XTNci8QqmQdO', 'dawqiy9gRXpa', 'sDa10RYPhTE4', '9dwaEdtkGoBj'...]
obs (metadata):
🔗 cell_type (32, bionty.CellType): ['lymphocyte', 'mast cell', 'megakaryocyte', 'plasma cell', 'dendritic cell, human']
🔗 assay (4, bionty.ExperimentalFactor): ["10x 5' v1", "10x 3' v3", "10x 5' v2", 'single-cell RNA sequencing']
🔗 tissue (17, bionty.Tissue): ['spleen', 'skeletal muscle tissue', 'transverse colon', 'jejunal epithelium', 'lamina propria']
🔗 donor (12, core.Label): ['621B', '582C', 'A36', 'D496', 'A31']
file2.view_lineage()
Load files into memory:
file1_adata = file1.load()
file2_adata = file2.load()
💡 adding file RAdyFo8MWTzVkqkFCK8T as input for run XLtF43RASfz8MnvMWglS, adding parent transform Nv48yAceNSh8z8
💡 adding file WtEvWQ5KVML36kWeCyJt as input for run XLtF43RASfz8MnvMWglS, adding parent transform Nv48yAceNSh8z8
Here we compute shared genes without loading files:
file1_genes = file1.features["var"]
file2_genes = file2.features["var"]
shared_genes = file1_genes & file2_genes
len(shared_genes)
695
shared_genes.list("symbol")[:10]
['S1PR4',
'GLRX',
'NUDCD2',
'TMEM69',
'CD82',
'HP1BP3',
'HIGD2A',
'IL7R',
'GATA2',
'FLT3LG']
We also need to convert the ensembl_gene_id to symbol for file2 so that they can be concatenated:
mapper = pd.DataFrame(shared_genes.values_list("ensembl_gene_id", "symbol")).set_index(
0
)[1]
mapper.head()
0
ENSG00000125910 S1PR4
ENSG00000173221 GLRX
ENSG00000170584 NUDCD2
ENSG00000159596 TMEM69
ENSG00000085117 CD82
Name: 1, dtype: object
file2_adata.var.rename(index=mapper, inplace=True)
Intersect cell types#
file1_celltypes = file1.cell_types.all()
file2_celltypes = file2.cell_types.all()
shared_celltypes = file1_celltypes & file2_celltypes
shared_celltypes_names = shared_celltypes.list("name")
shared_celltypes_names
['CD16-positive, CD56-dim natural killer cell, human',
'conventional dendritic cell']
We can now subset the two datasets by shared cell types:
file1_adata_subset = file1_adata[
file1_adata.obs["cell_type"].isin(shared_celltypes_names)
]
file2_adata_subset = file2_adata[
file2_adata.obs["cell_type"].isin(shared_celltypes_names)
]
Concatenate subseted datasets:
adata_concat = ad.concat(
[file1_adata_subset, file2_adata_subset],
label="file",
keys=[file1.description, file2.description],
)
adata_concat
AnnData object with n_obs × n_vars = 126 × 695
obs: 'cell_type', 'file'
obsm: 'X_umap'
adata_concat.obs.value_counts()
cell_type file
CD16-positive, CD56-dim natural killer cell, human Conde22 114
conventional dendritic cell Conde22 7
CD16-positive, CD56-dim natural killer cell, human 10x reference pbmc68k 3
conventional dendritic cell 10x reference pbmc68k 2
dtype: int64
Show code cell content
# clean up test instance
!lamin delete --force test-scrna
!rm -r ./test-scrna
💡 deleting instance testuser1/test-scrna
✅ deleted instance settings file: /home/runner/.lamin/instance--testuser1--test-scrna.env
✅ instance cache deleted
✅ deleted '.lndb' sqlite file
❗ consider manually deleting your stored data: /home/runner/work/lamin-usecases/lamin-usecases/docs/test-scrna