Working with local and remote data

The filesystem module datamol.fs is not strictly related to molecule but it makes it very convenient to work with files both locally and remotely (AWS S3, GCS, HTTP, FTP, Git, etc) in a smooth and transparent manner. Under the hood the Datamol fs module is built on top of the library fsspec.

In [1]:

import tempfile

import datamol as dm
import tempfile

import datamol as dm

Destructive path manipulation¶

The below examples are made locally for the purpose of the demo but all the function supports remote path (S3, GCS, etc).

First let's get temp directory

In [29]:

temp_dir = tempfile.mkdtemp()
dm.fs.exists(temp_dir)
temp_dir = tempfile.mkdtemp()
dm.fs.exists(temp_dir)

Out[29]:

True

Create a directory and check if it has correctly been created.

In [30]:

subdir_path = dm.fs.join(temp_dir, "subdir1", "subsubdir293")
dm.fs.mkdir(subdir_path, exist_ok=True)

dm.fs.exists(subdir_path)
subdir_path = dm.fs.join(temp_dir, "subdir1", "subsubdir293")
dm.fs.mkdir(subdir_path, exist_ok=True)

dm.fs.exists(subdir_path)

Out[30]:

True

Copy a file from a source path to a destination path

In [32]:

destination_path = dm.fs.join(subdir_path, "Compound_000000001_000500000.sdf.gz")
dm.fs.copy_file(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz",
    destination=destination_path,
    progress=True,
    force=True,
)
destination_path = dm.fs.join(subdir_path, "Compound_000000001_000500000.sdf.gz")
dm.fs.copy_file(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz",
    destination=destination_path,
    progress=True,
    force=True,
)

  0%|          | 0.00/321M [00:00<?, ?B/s]

Now, we would like to copy an full directory tree to a given destination.

In [47]:

subdir2_path = dm.fs.join(temp_dir, "subdir2")
dm.fs.copy_dir(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/specifications/",
    destination=subdir2_path,
    progress=True,
)
subdir2_path = dm.fs.join(temp_dir, "subdir2")
dm.fs.copy_dir(
    source="https://ftp.ncbi.nlm.nih.gov/pubchem/specifications/",
    destination=subdir2_path,
    progress=True,
)

  0%|          | 0/15 [00:00<?, ?it/s]

Let's check the files have been copied correctly.

In [48]:

dm.fs.glob(dm.fs.join(subdir2_path, "**"))
dm.fs.glob(dm.fs.join(subdir2_path, "**"))

Out[48]:

['file:///tmp/tmphlww_l88/subdir2/README',
 'file:///tmp/tmphlww_l88/subdir2/pubchem.asn',
 'file:///tmp/tmphlww_l88/subdir2/pubchem.xjb',
 'file:///tmp/tmphlww_l88/subdir2/pubchem.xsd',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_deposit.pdf',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_deposit.txt',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_fingerprints.pdf',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_fingerprints.txt',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_pug.pdf',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_pug.txt',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_sdtags.pdf',
 'file:///tmp/tmphlww_l88/subdir2/pubchem_sdtags.txt',
 'file:///tmp/tmphlww_l88/subdir2/pug.dtd',
 'file:///tmp/tmphlww_l88/subdir2/pug.xsd',
 'file:///tmp/tmphlww_l88/subdir2/pug_soap.readme.txt']

Non destructive path manipulation¶

Retrieve the paths matching a path pattern

In [34]:

dm.fs.glob("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/**")[:5]
dm.fs.glob("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/**")[:5]

Out[34]:

['https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-11-25/',
 'https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-11-25/ASN/',
 'https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-11-25/ASN/Compound_021500001_022000000.asn.gz',
 'https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-11-25/ASN/Compound_021500001_022000000.asn.gz.md5',
 'https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-11-25/ASN/Compound_157000001_157500000.asn.gz']

Get the name of the file or directory for a given path

In [35]:

dm.fs.get_basename(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)
dm.fs.get_basename(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)

Out[35]:

'Compound_013500001_014000000.sdf.gz'

In [36]:

dm.fs.get_basename("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03")
dm.fs.get_basename("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03")

Out[36]:

'2021-12-03'

Get the extension of a given path

In [37]:

dm.fs.get_extension("s3://an-s3-bucket-random/subdir1/subdir2/hello.txt")
dm.fs.get_extension("s3://an-s3-bucket-random/subdir1/subdir2/hello.txt")

Out[37]:

'txt'

Check whether a file or a directory exists

In [38]:

dm.fs.exists("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/")
dm.fs.exists("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/")

Out[38]:

True

Check whether a file or a directory exists

In [39]:

dm.fs.exists("gs://a-gcs-bucket-random/subdir1/subdir2/hello.txt")
dm.fs.exists("gs://a-gcs-bucket-random/subdir1/subdir2/hello.txt")

Out[39]:

False

Check whether a path is a file and exists

In [40]:

dm.fs.is_file(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)
dm.fs.is_file(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Daily/2021-12-03/SDF/Compound_013500001_014000000.sdf.gz"
)

Out[40]:

True

Check whether a path is a directory and exists

In [41]:

dm.fs.is_dir("gs://a-gcs-bucket-random/subdir1/subdir2/")
dm.fs.is_dir("gs://a-gcs-bucket-random/subdir1/subdir2/")

Out[41]:

False

Check whether a path is local or remote

In [42]:

dm.fs.is_local_path("/home/hello/a_subdir")
dm.fs.is_local_path("/home/hello/a_subdir")

Out[42]:

True

Join paths together

In [43]:

data_dir = "gs://awesome-data-bucket/data_dir"
filename = "molecules.sdf"
dm.fs.join(data_dir, filename)
data_dir = "gs://awesome-data-bucket/data_dir"
filename = "molecules.sdf"
dm.fs.join(data_dir, filename)

Out[43]:

'gs://awesome-data-bucket/data_dir/molecules.sdf'

Get the size of a file (in byte)

In [44]:

dm.fs.get_size(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz"
)
dm.fs.get_size(
    "https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/SDF/Compound_000000001_000500000.sdf.gz"
)

Out[44]:

336817141

Get the MD5 checksum of a file

In [45]:

dm.fs.md5("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/README-Compound")
dm.fs.md5("https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/CURRENT-Full/README-Compound")

Out[45]:

'b2d7b30c1466ab9582df47b2664d04b5'

Cache directory¶

It's often convenient to get the path of a persistent cache folder. Unfortunately this path will change depending on the OS you're working on. Datamol offers a function to easily retrieve the path of the "official" cache directory on which it's running.

In [3]:

dm.fs.get_cache_dir(app_name="datamol-demo", suffix="subdir1", create=False)
dm.fs.get_cache_dir(app_name="datamol-demo", suffix="subdir1", create=False)

Out[3]:

PosixPath('/home/hadim/.cache/datamol-demo/subdir1')