Skip to content
This repository has been archived by the owner on Sep 25, 2023. It is now read-only.

Commit

Permalink
Add option for smaller dataset in IO notebook (#473)
Browse files Browse the repository at this point in the history 
Adds the option to use a smaller ~300MB SigMF dataset for the IO examples, as well as code to automatically download / extract the dataset of choice.

Note that there are some failures in the notebook - raised #474 to cover this.

Closes #472

Authors:
  - Charles Blackmon-Luca (https://github.com/charlesbluca)

Approvers:
  - Adam Thompson (https://github.com/awthomp)

URL: #473
  • Loading branch information
@charlesbluca
charlesbluca authored Nov 21, 2022
1 parent 1dacf7d commit 4de4081
Showing 1 changed file with 136 additions and 31 deletions.
167 changes: 136 additions & 31 deletions notebooks/api_guide/io_examples.ipynb
View file
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"We are using the Northeastern University Oracle SigMF recordings found [here](http://www.genesys-lab.org/oracle) (Dataset #2)"
"We are using the [Northeastern University Oracle RF Fingerprinting Dataset](http://www.genesys-lab.org/oracle):"
]
},
{
Expand All @@ -39,8 +39,106 @@
"metadata": {},
"outputs": [],
"source": [
"meta_file = '/data/oracle/KRI-16Devices-RawData/2ft/WiFi_air_X310_3123D7B_2ft_run1.sigmf-meta'\n",
"data_file = '/data/oracle/KRI-16Devices-RawData/2ft/WiFi_air_X310_3123D7B_2ft_run1.sigmf-data'"
"# set input SigMF dataset - note that this is ~28gb\n",
"base_url = 'https://repository.library.northeastern.edu/downloads/neu:m044q5210?datastream_id=content'\n",
"path = 'KRI-16Devices-RawData'\n",
"file = '2ft/WiFi_air_X310_3123D7B_2ft_run1'\n",
"\n",
"# uncomment to use smaller ~300mb dataset\n",
"# base_url = 'https://repository.library.northeastern.edu/downloads/neu:m044q523j?datastream_id=content'\n",
"# path = 'KRI-16IQImbalances-DemodulatedData'\n",
"# file = 'Demod_WiFi_cable_X310_3123D76_IQ#1_run1'\n",
"\n",
"comp = '.zip'"
]
},
{
"cell_type": "code",
"execution_count": 3,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Downloading https://repository.library.northeastern.edu/downloads/neu:m044q523j?datastream_id=content to ../data/KRI-16IQImbalances-DemodulatedData.zip\n"
]
},
{
"name": "stderr",
"output_type": "stream",
"text": [
" % Total % Received % Xferd Average Speed Time Time Time Current\n",
" Dload Upload Total Spent Left Speed\n",
"100 295M 100 295M 0 0 798k 0 0:06:18 0:06:18 --:--:-- 822k\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Decompressing ../data/KRI-16IQImbalances-DemodulatedData.zip...\n",
"Archive: ../data/KRI-16IQImbalances-DemodulatedData.zip\n",
" creating: ../data/KRI-16IQImbalances-DemodulatedData/\n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#13_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#13_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#17_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#17_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#5_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#5_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#15_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#15_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#14_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#14_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#26_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#26_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#25_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#25_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#3_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#3_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#2_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#2_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#4_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#4_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#9_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#9_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#1_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#1_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#19_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#19_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#18_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#18_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#7_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#7_run1.sigmf-meta \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#32_run1.sigmf-data \n",
" inflating: ../data/KRI-16IQImbalances-DemodulatedData/Demod_WiFi_cable_X310_3123D76_IQ#32_run1.sigmf-meta \n",
"../data/KRI-16IQImbalances-DemodulatedData.zip decompressed\n"
]
}
],
"source": [
"import os\n",
"\n",
"# create data directory\n",
"data_dir = '../data/'\n",
"if not os.path.exists(data_dir):\n",
" print('creating data directory')\n",
" os.system('mkdir ../data')\n",
"\n",
"# download dataset\n",
"if not os.path.isdir(data_dir + path):\n",
" if not os.path.isfile(data_dir + path + comp):\n",
" print(f'Downloading {base_url} to {data_dir + path + comp}')\n",
" os.system(f'curl -o {data_dir + path + comp} {base_url}')\n",
" print(f'Decompressing {data_dir + path + comp}...')\n",
" os.system(f'unzip {data_dir + path + comp} -d {data_dir}')\n",
" print(f'{data_dir + path + comp} decompressed')\n",
"else:\n",
" print(f'Dataset already exists under {data_dir + path}')\n",
" \n",
"# set input SigMF files\n",
"meta_file = os.path.join(data_dir, path, file + '.sigmf-meta')\n",
"data_file = os.path.join(data_dir, path, file + '.sigmf-data')"
]
},
{
Expand All @@ -54,7 +152,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
"For our purposes here, [SigMF](https://github.com/gnuradio/SigMF) data is treated as a JSON header and processed on CPU, while the *binary* payload file is mmaped to GPU and cuSignal uses a CUDA kernel to parse the file. While we've focused on SigMF here, you can use the underlying `cusignal.read_bin` and `cusignal.parse_bin` (and corresponding write functions) for your own datasets."
"For our purposes here, [SigMF](https://github.com/gnuradio/SigMF) data is treated as a JSON header and processed on CPU, while the *binary* payload file is mapped to GPU and cuSignal uses a CUDA kernel to parse the file. While we've focused on SigMF here, you can use the underlying `cusignal.read_bin` and `cusignal.parse_bin` (and corresponding write functions) for your own datasets."
]
},
{
Expand All @@ -66,7 +164,7 @@
},
{
"cell_type": "code",
"execution_count": 3,
"execution_count": 4,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -79,14 +177,14 @@
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"128 ms ± 147 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
"13.5 ms ± 12.4 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
]
}
],
Expand All @@ -104,14 +202,14 @@
},
{
"cell_type": "code",
"execution_count": 5,
"execution_count": 6,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"188 ms ± 341 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
"15.5 ms ± 430 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
]
}
],
Expand All @@ -137,14 +235,14 @@
},
{
"cell_type": "code",
"execution_count": 6,
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"82.2 ms ± 172 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
"7.38 ms ± 28.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
]
}
],
Expand All @@ -170,7 +268,7 @@
},
{
"cell_type": "code",
"execution_count": 7,
"execution_count": 8,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -180,14 +278,14 @@
},
{
"cell_type": "code",
"execution_count": 8,
"execution_count": 9,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"82.2 ms ± 50.1 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
"7.37 ms ± 32.2 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
]
}
],
Expand All @@ -213,7 +311,7 @@
},
{
"cell_type": "code",
"execution_count": 9,
"execution_count": 10,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -223,14 +321,14 @@
},
{
"cell_type": "code",
"execution_count": 10,
"execution_count": 11,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"82.8 ms ± 866 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
"7.39 ms ± 22.9 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)\n"
]
}
],
Expand All @@ -249,7 +347,7 @@
},
{
"cell_type": "code",
"execution_count": 11,
"execution_count": 12,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -271,14 +369,14 @@
},
{
"cell_type": "code",
"execution_count": 12,
"execution_count": 13,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"338 ms ± 837 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
"41.8 ms ± 576 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
Expand All @@ -297,14 +395,14 @@
},
{
"cell_type": "code",
"execution_count": 13,
"execution_count": 14,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"341 ms ± 830 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
"43 ms ± 1.4 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
Expand All @@ -323,7 +421,7 @@
},
{
"cell_type": "code",
"execution_count": 14,
"execution_count": 15,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -333,14 +431,14 @@
},
{
"cell_type": "code",
"execution_count": 15,
"execution_count": 16,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"253 ms ± 1.22 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
"36 ms ± 2.24 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
Expand All @@ -359,7 +457,7 @@
},
{
"cell_type": "code",
"execution_count": 16,
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
Expand All @@ -369,14 +467,14 @@
},
{
"cell_type": "code",
"execution_count": 17,
"execution_count": 18,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"253 ms ± 950 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)\n"
"36.3 ms ± 419 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)\n"
]
}
],
Expand All @@ -385,11 +483,18 @@
"cusignal.write_sigmf(test_file_ext, sigmf, buffer, append=False)\n",
"cp.cuda.runtime.deviceSynchronize()"
]
},
{
"cell_type": "code",
"execution_count": null,
"metadata": {},
"outputs": [],
"source": []
}
],
"metadata": {
"kernelspec": {
"display_name": "Python 3",
"display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
Expand All @@ -403,9 +508,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.8.2"
"version": "3.9.7"
}
},
"nbformat": 4,
"nbformat_minor": 4
}
}

0 comments on commit 4de4081

Please sign in to comment.