Overview and Access Points
BDSP data is stored in Amazon S3 in us-east-1 (N. Virginia). You reach it through S3 access points: once your AWS account is authorized, you read the data from your own AWS account, using the access point wherever a tool asks for a bucket name.
[YOUR-ACCESS-POINT-ALIAS]; replace them with the values from your dashboard.
How to work with the data, in order of preference
- Recommended: run your code next to the data. Use a machine in us-east-1 in your own AWS account (the BDSP notebook, your own EC2 instance or a SageMaker notebook) and stream the files you need straight from your access point.
- If you need a working copy: copy it to your own S3 bucket in us-east-1. The copy stays inside S3.
- Please don't download BDSP data to a laptop, a workstation or another cloud.
Access levels
| Access Level | Contains | How to get authorized |
|---|---|---|
| Open | Publicly available datasets | Add your AWS Account ID at Cloud Credentials; click "Request Access" on a project page |
| Restricted | Restricted-access publication datasets | Same as above, plus sign each project's Data Use Agreement |
| Credentialed | Large research repositories (EEG, ECG, PSG, EHR, Imaging) and credentialed publication datasets | Same as above, plus complete CITI Data or Specimens-Only Research training and submit your certificate |
The projects at each access level are listed further down this page.
Recommended: Run Your Code Next to the Data
Run your code on a machine in us-east-1 in your own AWS account, and read BDSP files straight from your access point. Programs stream what they need, so you don't wait for downloads, the data never leaves AWS, and reading it from us-east-1 carries no data-transfer charge.
Option 1: the BDSP notebook (one click)
On Settings → Cloud, click Launch a notebook in your AWS account. The button is shown once you are credentialed and have saved your AWS account ID. It opens the AWS CloudFormation console in us-east-1 with BDSP's template filled in. The stack creates, in your account:
- a SageMaker notebook instance (JupyterLab) of the size you choose. The default
ml.t3.medium, about $0.05 an hour, is enough for the starter notebook; - the BDSP Python environment and the starter notebook
bdsp-notebooks/starter.ipynb; - an IAM role that can only list and read through BDSP's access points.
- Sign in to your AWS account, then click the button.
- Keep the region as US East (N. Virginia). The template refuses to run anywhere else.
- Choose a bigger instance type if you want one, tick "I acknowledge that AWS CloudFormation might create IAM resources", and click Create stack.
- After about 5–10 minutes the stack shows
CREATE_COMPLETE. On its Outputs tab, open NotebookUrl. - In JupyterLab, open
bdsp-notebooks/starter.ipynb, choose the BDSP (Python 3.12) kernel and run all cells. It lists the datasets your account can read and opens tables, sleep, EEG, ECG and imaging files.
Stop the instance when you are not using it (SageMaker console → Notebook instances → Stop): your files are kept and you pay only for the disk. Delete the stack when you are done (CloudFormation → Stacks → Delete); download any notebooks you wrote first. To launch it you need permission in your AWS account to create CloudFormation stacks, IAM roles and SageMaker notebook instances.
Option 2: your own EC2 instance or SageMaker notebook
- Region: us-east-1 only. Create the instance (and any bucket you copy to) in US East (N. Virginia).
- Give it read access through BDSP's access points. Create an IAM role with this policy and attach it to the instance (as its instance profile) or to the notebook (as its execution role). It covers the credentialed access points; for other levels, use the ARNs on your dashboard.
IAM policies must name access points by ARN, and object permissions use the{ "Version": "2012-10-17", "Statement": [ { "Sid": "ListThroughBdspAccessPoints", "Effect": "Allow", "Action": "s3:ListBucket", "Resource": [ "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point-1", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap-1" ] }, { "Sid": "ReadObjectsThroughBdspAccessPoints", "Effect": "Allow", "Action": "s3:GetObject", "Resource": [ "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point/object/*", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point-1/object/*", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap/object/*", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap-1/object/*" ] }, { "Sid": "BucketsOnlyThroughBdspAccessPoints", "Effect": "Allow", "Action": [ "s3:ListBucket", "s3:GetObject" ], "Resource": [ "arn:aws:s3:::bdsp-opendata-repository", "arn:aws:s3:::bdsp-opendata-repository/*", "arn:aws:s3:::bdsp-opendata-credentialed", "arn:aws:s3:::bdsp-opendata-credentialed/*" ], "Condition": { "StringEquals": { "s3:DataAccessPointArn": [ "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point-1", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap", "arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-projects-ap-1" ] } } } ] }…/object/*form. That form belongs only in policies: in the AWS CLI and boto3, use the alias or the plain access-point ARN (see "Access Point Alias or ARN" below). The third statement is needed too: a read through an access point is also checked against the bucket in your policy, and it allows BDSP's buckets only through BDSP's access points, never directly. - EC2: add an S3 gateway endpoint to the instance's VPC. It is free. Without it, an instance in a private subnet reaches S3 through a NAT gateway, and AWS bills you about $0.045 per GB processed:
aws ec2 create-vpc-endpoint --region us-east-1 --vpc-endpoint-type Gateway \ --vpc-id vpc-XXXXXXXX --service-name com.amazonaws.us-east-1.s3 \ --route-table-ids rtb-XXXXXXXX - Read the data where it is. For example:
In Python (boto3), pass the alias or ARN asaws s3 ls s3://[YOUR-ACCESS-POINT-ALIAS]/[FOLDER-NAME]/ --region us-east-1Bucket=. The BDSP starter notebook and itsbdsp_stream.pyreader also run on your own instance:for f in starter.ipynb bdsp_stream.py requirements.txt GUIDE.md; do curl -fsSLO "https://bdsp.io/static/notebooks/v1/$f" done
A Working Copy in Your Own Bucket
Prefer streaming. If you do need a working copy, copy it to an S3 bucket in us-east-1 in your own AWS account. The copy happens inside S3 and never passes through your machine:
aws s3 mb s3://[YOUR-BUCKET] --region us-east-1
aws s3 sync s3://[YOUR-ACCESS-POINT-ALIAS]/[FOLDER-NAME]/ \
s3://[YOUR-BUCKET]/[FOLDER-NAME]/ \
--region us-east-1 --copy-props metadata-directive
- Keep
--copy-props metadata-directive. Without it, files over 8 MB fail withAccessDeniedonGetObjectTagging: the AWS CLI then also tries to copy the files' tags, which BDSP's access points don't allow (BDSP files carry no tags you need). - Access points copy only within a region, so a bucket in another region won't work.
- You pay for the storage (about $0.023 per GB-month for S3 Standard) and the requests. Same-region copies carry no data-transfer charge.
- Your copy is still covered by the data use agreement. Keep the bucket private, share it with no one, and delete it when the project ends.
Please Don't Download
Please don't aws s3 cp or aws s3 sync BDSP data to a laptop, a workstation or another cloud. It is slow, downloads out of AWS cost BDSP about $0.09 per GB, and copies outside AWS are harder to keep secure. Run your code next to the data instead, and bring back only results, figures and code.
Who pays what
| Item | Who pays |
|---|---|
| Notebook or EC2 instance, per hour while it runs | You (ml.t3.medium is about $0.05 an hour; see SageMaker pricing) |
| Its disk, while the stack or instance exists (stopped or not) | You (a few dollars a month for the default 20 GB) |
| Your own S3 bucket: storage and requests | You |
| A NAT gateway, if your EC2 instance uses one for S3 | You (avoid it with the free S3 gateway endpoint) |
| Reading BDSP data from us-east-1 | No data-transfer charge; BDSP pays the small S3 request fees |
| Reading BDSP data from another AWS region | BDSP, about $0.02 per GB. Please don't |
| Downloading BDSP data out of AWS (laptop, other cloud) | BDSP, about $0.09 per GB. Please don't |
BDSP doesn't bill you; AWS bills you directly for your own resources. Set an AWS budget alert if you want a warning.
Projects by Access Level
Use the access point for each level from your dashboard, and replace [FOLDER-NAME] in the examples above with the project or dataset folder.
Open access
Open-access publication datasets are available to anyone with an AWS account; no DUA or training is required. To get authorized, register your AWS Account ID at Cloud Credentials and click "Request Access" on the project page.
- Deep Hypothermia and EEG (folder:
hypothermia/) - Other open-access publication datasets — see the project listings on Browse Projects
Restricted access
Restricted-access publication datasets require a signed Data Use Agreement (DUA) per project, in addition to a registered AWS Account ID. Once approved, the project's data is reachable through the restricted access point.
- I-CARE (folder:
i-care/) - Other restricted-access publication datasets — see the project listings on Browse Projects
Credentialed access: large repositories
BDSP's large credentialed repositories cover hundreds of thousands of records of physiological signals and clinical data. Access requires a registered AWS Account ID, a signed credentialed Data Use Agreement, and proof of completion of CITI Data or Specimens-Only Research training.
EEG/— ElectroencephalographyECG/— ElectrocardiographyPSG/— Polysomnography (sleep)EHR/— Electronic health recordsImaging/— Imaging data
Credentialed access: publication-specific projects
Publication-specific credentialed datasets are tied to individual papers. Authorization requires the same prerequisites as the large repositories (registered AWS Account ID, signed DUA, CITI training), plus per-project DUA review where required.
- Philosopher's Stone (folder:
philosoph/) - ICU SLEEP
- Brain Age Index / Dementia
- Other credentialed publication datasets — see the project listings on Browse Projects
Access Point Alias or ARN
Verify your AWS identity
aws sts get-caller-identity
The Account field must match the 12-digit AWS Account ID you registered with BDSP.
Alias or ARN
Use an access point wherever a tool asks for a bucket name. Both forms from your dashboard work in the AWS CLI and in boto3 (Bucket=):
- Alias (shorter):
s3://bdsp-credentialed-ac-…-s3alias/PSG/bids/ - ARN:
s3://arn:aws:s3:us-east-1:184438910517:accesspoint/bdsp-credentialed-access-point-1/PSG/bids/. The path in the access point follows its name directly.
aws s3 ls s3://[YOUR-ACCESS-POINT-ALIAS]/PSG/bids/ --region us-east-1
aws s3 ls s3://arn:aws:s3:us-east-1:184438910517:accesspoint/[YOUR-ACCESS-POINT-NAME]/PSG/bids/ --region us-east-1
The …:accesspoint/NAME/object/… form is only for the Resource of an IAM policy. It does not work in the AWS CLI.
Each BDSP access point has a twin (for example bdsp-credentialed-access-point and bdsp-credentialed-access-point-1), and your account is on one of each pair. Use the names on your dashboard: AccessDenied on the other twin is normal.
Troubleshooting "Access Denied"
- Check the account: run
aws sts get-caller-identity; theAccountmust match the ID on Cloud Credentials. - Use the alias or ARN from your dashboard, not access-point names from older documentation.
- Wait for access to propagate: after approval, wait 5–10 minutes, then try Request Access on the project page again.
- Use an IAM role or user, not the account's root credentials, and make sure it has the read permissions above.
- Copying to your bucket and only big files fail with
GetObjectTagging: add--copy-props metadata-directive(see "A Working Copy in Your Own Bucket"). - Organisations: if your organisation uses service control policies or VPC endpoint policies, ask your administrator to allow the access-point ARNs above.
AWS S3 Permissions Setup (keep existing)
Step 1: Sign in to AWS
- Go to the AWS Management Console: https://console.aws.amazon.com/
- Log in with your AWS account.
Step 2: Open IAM
- In the search bar at the top of the AWS console, type IAM.
- Click IAM (Identity and Access Management).
Step 3: Select Your User or Role
- If you are using an IAM User: click Users from the left menu, then select your username.
- If you are using an IAM Role: click Roles from the left menu, then select the role you use.
Step 4: Attach S3 Permissions
- Go to the Permissions tab.
- Click Add permissions → Attach policies directly.
- Search for and select AmazonS3FullAccess.
- If your organization requires limited access, create a custom policy with at least the following permissions:
{ "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "s3:GetObject", "s3:ListBucket" ], "Resource": "*" } ] } - Click Add permissions.