{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": [
     "hide-output"
    ]
   },
   "outputs": [],
   "source": [
    "!pip install aiohttp requests s5cmd \"xarray[io]\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import xarray as xr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "MBTnYN9sjM8N"
   },
   "source": [
    "# Bulk Data Download\n",
    "\n",
    "This notebook shows how to perform bulk downloads with a S3 command line tool. This is useful if you want to have local access to a big subset of the data or event download the whole archive!\n",
    "\n",
    "We can download data in bulk using any command line that supports the S3 protocol. We recommend using the [s5cmd](https://github.com/peak/s5cmd) tool, which can be simply installed by running:\n",
    "\n",
    "`pip install s5cmd`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NSKjNdNljqhl"
   },
   "source": [
    "Now we can download data using the `cp` command.\n",
    "\n",
    "In this example, we are going to transfer the thompson scattering data for shot `30420` locally.\n",
    "\n",
    "We need to set the endpoint of where the bucket is hosted (for now: `https://s3.echo.stfc.ac.uk`) and we need to set `--no-sign-request` for annonymous access."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "_CqfL5dfiFh0",
    "outputId": "290bc0a1-fdc2-490b-ec00-32e146f98e9f"
   },
   "outputs": [],
   "source": [
    "%%capture --no-display\n",
    "%%bash\n",
    "s5cmd --no-sign-request --endpoint-url https://s3.echo.stfc.ac.uk cp s3://mast/level2/shots/30420.zarr/thomson_scattering/* ./30420.zarr/thomson_scattering;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NhghzZ99jzip"
   },
   "source": [
    "Finally, we can open the file locally:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 457
    },
    "id": "udo56w_3jFp_",
    "outputId": "911b525e-40eb-497c-b6ee-caf26b4e4f09"
   },
   "outputs": [],
   "source": [
    "xr.open_zarr('30420.zarr/thomson_scattering', consolidated=False)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.13.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 0
}