Link Search Menu Expand Document

A Guide to Data Manipulation and Visualization

Convenience

This technical guide is aimed at researchers running experiments with Byteflies Kits. It discusses relevant Byteflies Cloud and API functionality, as well as basic data science instructions to visualize and review the recorded data. It is not a replacement for the main user manual which discusses how to safely operate Sensor Dots and their accessories, and is strictly provided for your convenience.

Applicable Firmware version: 1.15+
LAST UPDATE: October 28th, 2020

Table of contents
  1. Glossary
  2. Setting up a New Experiment in the Byteflies Cloud
    1. Before You Start
    2. Patient ID
    3. Ready to Start?
  3. Managing Completed Recordings
    1. Default Signal Settings
    2. Basic Data Plotting
      1. Plot an EEG Recording (Single Sensor Dot)
      2. Plot an ECG and EMG Recording (Dual Sensor Dot)
    3. Advanced Options
    4. Accessing a Recording via the API
      1. Log-in to your Byteflies Account
      2. Get all the Recordings available in a Group
      3. Get the metadata for a specific Recording
      4. Get the CSV files for a Recording
      5. Plot a Recording Programmatically

Glossary

Signal
A physiologic or motion signal, as recorded by a Sensor Dot (e.g. ECG).
Channel
A Sensor Dot can record two biopotential Signals simultaneously and has two pairs of electrode connectors that each form a Channel.
Recording
The combination of all data recorded by a single Sensor Dot is grouped in a Recording.
Group
A Group combines one or more Docking Stations that are assigned to a specific study.
Patient ID
A unique string used to identify a study subject.
Patient Record
A list of pre-configured Patient IDs and the hardware they are associated with.
User Account
A Byteflies Cloud account that can contain one or more Groups and can only be accessed with a specific username and password.

Setting up a New Experiment in the Byteflies Cloud

Byteflies Kits are set up in the factory for your study. However, for more advanced use cases, it may be necessary to change certain configuration settings. These can be accessed from your Byteflies Cloud account. Review the Byteflies Cloud Instructions for more details.

Before You Start

Sensor Dots can be configured to measure two biopotential (ExG) Channels, specifically EEG, ECG, EMG, and EOG. It is possible to select a different Signal for each Channel. The amplifiers in the Sensor Dot work differently depending on the selected Signal, so it is important to set them correctly.

If the configuration loaded on your Sensor Dots needs to be changed, follow the configuration instructions. It is currently not possible to check the active configuration of a Sensor Dot through the Byteflies Cloud web interface. For complex experiments with multiple Sensor Dots, it is a good idea to label them with a specific identifier/tag. In case the configuration information is lost, you can always reconfigure your Sensor Dots. This will not result in any loss of already recorded data.

Patient ID

You can use the Byteflies Cloud to configure a Patient ID to associate one or more Recordings with a specific study subject, as epxlained in the Byteflies Cloud instructions.

Ready to Start?

Start each Sensor Dot you plan to use by lifting it from the Docking Station. Within 1 sec the Recording will start. To collect as much useful data as possible, it is good practice to make sure the cradles, patches and/or electrodes are already applied so you can pop the Sensor Dot immediately into the cradle. Detailed instructions are in the user manual and any specific information you may have received from your Byteflies support contact.

Each Sensor Dot will store an accurate timestamp when it starts a recording based on the internet connected clock of the Docking Station. In other words, it is not necessary to try to time the start of a Recording of multiple Sensor Dots with respect to each other.

Managing Completed Recordings

Once you have completed a Recording and placed the Sensor Dot(s) back in the Docking Station, the data will upload to your Byteflies Cloud account.

These files are quite large and this may take time depending on the speed of your internet connection. Please be patient for as long as the LED on the Sensor Dot(s) is/are orange.

To access a recording directly via the Byteflies Cloud web interface, refer to the Byteflies Cloud instructions.

Default Signal Settings

Signal Conversion Factor* Units Sampling Rate
EEG/EOG 0.4808 nanoVolts (nV) 250 Hz
ECG 1.4424 nanoVolts (nV) 250 Hz
EMG 0.4808 nanoVolts (nV) 250 Hz
ACC 0.0002 g-force (g) 25, 50, 100, or 200 Hz
GYR 0.0305 degrees/sec (deg/sec) 25, 50, 100, or 200 Hz

Multiple the recorded signal with this factor to obtain the correct units.

For more information on the files that are generated per Recording, refer to the Byteflies Cloud and Clinical Insights documents.

Pro Tip: You can get the UNIX timestamp in millisecond resolution for a Recording by clicking on the Recording Start value from the Recording Detail view .

Basic Data Plotting

There are a lot of possible ways to visualize the recorded data. For your information, we’ve included a number of examples using Python and Jupyter Notebooks below.

Make sure you are familiar with the structure of the files associated with a Recording before proceeding!

These examples use the time index in seconds which is stored for each sample, relative to the start time of the Recording. Note that it is possible to assume a fixed sampling frequency (e.g. 250 Hz for EEG) but when comparing Signals from more than one Sensor Dot, the time values will be slightly more accurate.

The drift in a Sensor Dot for a 24 h Recording is < 3 seconds when using the time values.

Plot an EEG Recording (Single Sensor Dot)

This example assumes a dual Channel EEG recording was made with the accelerometer activated at 25 Hz.

import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime

# Load data (assuming the CSV file formating is exactly as it was downloaded from 
# Byteflies Cloud and the files are sitting in the same directory as your script or notebook)
eeg_ch1 = pd.read_csv("filename1.csv", names=["time", "raw"], skiprows=1)
eeg_ch2 = pd.read_csv("filename2.csv", names=["time", "raw"], skiprows=1)
eeg = pd.merge(eeg_ch1, eeg_ch2, on="time", suffixes=("_ch1", "_ch2"))
acc = pd.read_csv("filename3.csv", names=["time", "x", "y", "z"], skiprows=1)
eeg.head()

# Get the start timestamp for the Recording from the Recording detail page on the 
# Byteflies Cloud and fill it out below (YYYY,MM,DD,HH,MM,SS,uS) [uS = microseconds, 
# i.e. Cloud milliseconds * 1000] 
start_dt = datetime(2020,6,19,7,26,13,805000)
start_ts = start_dt.timestamp()

# Convert Signals to correct unit (note that this is a linear transformation and thus
# not strictly necessary)
eeg_list = eeg.keys()
acc_list = acc.keys()

for i in range(len(eeg_list)-1):
	eeg[eeg_list[i+1]] *= 0.4808  # To nV

for i in range(len(acc_list)-1):
	acc[acc_list[i+1]] *= 0.0002  # To g
	
# Convert the time values in seconds to a datetime index for easier plotting and Signal comparison
t_ns_eeg = (start_ts + eeg["time"])*1e9
eeg.index = t_ns_eeg.astype("datetime64[ns]")
eeg.drop(["time"], axis=1, inplace=True)

t_ns_acc = (start_ts + acc["time"])*1e9 
acc.index = t_ns_acc.astype("datetime64[ns]")
acc.drop(["time"], axis=1, inplace=True)

# Plot the data
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, tight_layout=True)
ax1.plot(eeg, lw=1)
ax1.grid()
ax1.set_title("EEG")
ax1.set_ylabel(r"$\mu$V")
ax1.legend(["EEG Ch1", "EEG Ch2"], loc="best")

ax2.plot(acc, lw=1)
ax2.grid()
ax2.set_title("ACC")
ax2.set_ylabel("g")
ax2.legend(loc="best")
ax2.legend(["X-axis", "Y-axis", "Z-axis"], loc="best")

fig.autofmt_xdate()
plt.show()

Plot an ECG and EMG Recording (Dual Sensor Dot)

This examples assumes two Sensor Dots were used with only one ExG Channel in use each, and accelerometer activated at 25 Hz for both Sensor Dots.

import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime

# Load data (assuming the CSV file formating is exactly as it was downloaded from 
# Byteflies Cloud and the files are sitting in the same directory as your script or notebook)
ecg_ch1 = pd.read_csv("sensordot1_channel1.csv", names=["time", "raw"], skiprows=1)
emg_ch1 = pd.read_csv("sensordot2_channel1.csv", names=["time", "raw"], skiprows=1)
acc_sd1 = pd.read_csv("sensordot1_acc.csv", names=["time", "x", "y", "z"], skiprows=1)
acc_sd2 = pd.read_csv("sensordot2_acc.csv", names=["time", "x", "y", "z"], skiprows=1)
ecg_ch1.head()

# Get the start timestamp for the Recording from the Recording detail page on the 
# Byteflies Cloud and fill it out below (YYYY,MM,DD,HH,MM,SS,uS) [uS = microseconds, 
# i.e. Cloud milliseconds * 1000] 
start_dt = datetime(2020,6,19,7,26,13,805000)
start_ts = start_dt.timestamp()

# Convert Signals to correct unit (note that this is a linear transformation and thus 
# not strictly necessary)
ecg_ch1["raw"] *= 1.4424  # To nV
emg_ch1["raw"] *= 0.4808  # To nV
acc_list = acc_sd1.keys()

for i in range(len(acc_list)-1):
	acc_sd1[acc_list[i+1]] *= 0.0002  # To g
	acc_sd2[acc_list[i+1]] *= 0.0002  # To g
	
# Convert the time values in seconds to a datetime index for easier plotting and Signal comparison
sigList = [ecg_ch1, emg_ch1, acc_sd1, acc_sd2]

for i in range(len(sigList)):
	t_ns = (start_ts + sigList[i]["time"])*1e9
	sigList[i].index = t_ns.astype("datetime64[ns]")
	sigList[i].drop(["time"], axis=1, inplace=True)

# Plot the data
fig, (ax1, ax2, ax3) = plt.subplots(3, 1, sharex=True, tight_layout=True)
ax1.plot(ecg_ch1, lw=1)
ax1.plot(emg_ch1, lw=1)
ax1.grid()
ax1.set_title("ExG")
ax1.set_ylabel(r"$\mu$V")
ax1.legend(["ECG Ch1", "EMG Ch1"], loc="best")

ax2.plot(acc_sd1, lw=1)
ax2.grid()
ax2.set_title("ACC Sensor Dot 1")
ax2.set_ylabel("g")
ax2.legend(["X-axis", "Y-axis", "Z-axis"], loc="best")

ax3.plot(acc_sd2, lw=1)
ax3.grid()
ax3.set_title("ACC Sensor Dot 2")
ax3.set_ylabel("g")
ax3.legend(["X-axis", "Y-axis", "Z-axis"], loc="best")

fig.autofmt_xdate()
plt.show()

Advanced Options

If you cannot use the time column and absolutely require a completely fixed sampling frequency, you can calculate the closest approximation as follows:

# Assuming you already loaded a CSV file as demonstrated above
import numpy as np

fs = np.mean(1 / np.diff(ecg_ch1["time"]))

The uncertainty on the start timestamp for a specific Recording is less than 3 milliseconds. The drift of a single Sensor Dot is maximally 26 ppm (~2.25 seconds per 24 hours).

Accessing a Recording via the API

Downloading individual data files from the Byteflies Cloud interface works well for a relatively low number of recordings but it can get tedious for larger studies. In those cases, it is easier to use the API directly from whatever programming environment you’re used to.

The full API spec is available here. What follows provides a practical example implemented in Python.

Log-in to your Byteflies Account

import requests

# Provide your Byteflies Cloud username and password in these variables
username = "<USERNAME>"
password = "<PASSWORD>" # NOTE: please retrieve the password from a safe place

# Request a temporary API token with your user credentials, to use in future requests
url = "https://cognito-idp.eu-west-1.amazonaws.com"
headers = {"X-Amz-Target":
       "AWSCognitoIdentityProviderService.InitiateAuth",
       "Content-Type": "application/x-amz-json-1.1"}
data = {"ClientId": "5kgqk7n83evq9cbqnam1biqnbr",
        "AuthFlow": "USER_PASSWORD_AUTH",
        "AuthParameters": {"USERNAME": username,
                           "PASSWORD": password
                            }
        }
response = requests.post(url, headers=headers, json=data)
idToken = response.json()["AuthenticationResult"]["IdToken"]

# Url of the Byteflies Cloud
url = "https://api.cloud.byteflies.net/"

# Request all the group this user has access to
headers = {"Authorization": idToken}
response = requests.get(url + "groups/", headers=headers)
groups = response.json()
print(groups)

If authentication is successful, you should get an answer from the API (list of JSON) that includes the name of all the Groups you have access to and their unique IDs:

[
    {
        "groupId": "1234",
        "groupName": "TestGroup",
        "groupDescription": "Description of group"
    }
    , ...
]

Get all the Recordings available in a Group

# Extract the Group Name of the group you'd like to request a recording from 
groupName = "<GROUPNAME>"  # NOTE: Should be filled in or requested from the user
for group in response.json():
    if group["groupName"] == groupName:
        groupID = group["groupId"]
        break
else:
    print("No group available with that name!")
    exit(1)

# List all recordings available in a Group
headers = {"Authorization": idToken}
response = requests.get(url + "groups/" + groupID + "/recordings", headers=headers)
recordings = response.json()
print(recordings)    

If the Group exists, you should get an answer from the API (list of JSON) that specifies a unique ID and other relevant information for your available Recordings:

[
    {
        "id": "33d8951c-999f-11e9-a2a3-2a2ae2dbcce4",
        "dockName": "DemoDock-8e6bde5ac7e7191caaf7e3c9",
        "dotId": "24:de:b6:f2:e3:60",
        "patient": "patient-ID",
        "startDate": 1526919822.1422584,
        "uploadDate": 1526939822.453,
        "signals": [...]
        "duration": 99.384483839
    }
    , ... 
]

Get the metadata for a specific Recording

# Retrieve the metadata of a specific Recording
recordingID = "<RECORDING ID>"  # NOTE: Should be filled in or requested from the user
headers = {"Authorization": idToken}
response = requests.get(url + "groups/" + groupID + "/recordings/" + recordingID, headers=headers)

if response.status_code != 200:  #if response is invalid, return an error
    print("Recording ID not found")

recordingMetadata = response.json()
print(recordingMetadata)

If the Recording exists, you should get an answer from the API (JSON) that contains the full metadata for that Recording and a list of available Signals in the Recording:

{
    "id": "33d8951c-999f-11e9-a2a3-2a2ae2dbcce4",
    "dockName": "DemoDock-8e6bde5ac7e7191caaf7e3c9",
    "dotId": "24:de:b6:f2:e3:60",
    "patient": "patient-ID",
    "startDate": 1526919822.1422584,
    "uploadDate": 1526939822.453,
    "signals": [
      {
            "type": "ACC",
            "rawData": "https://...",
            "samplingRate": "25",
            "conversionFactor":0.000244140625
        },
        {
            "type": "EEG",
            "channel": 1,
            "rawData": "https://...",
            "samplingRate": "250",
            "conversionFactor":0.4808108585052719,
            "algorithms":[
                {"type":"EEG_ARTFCTS",
                    "col":1,
                    "id":"09cdf800-7268-11ea-a8d6-a7682dc8f48c"}
                ]
        },
        {
            "type": "ECG",
            "channel": 2,
            "rawData": "https://...",
            "samplingRate": "250",
            "conversionFactor":1.4424325755158156
        },
        {
            "type": "GYR",
            "rawData": "https://...",
            "samplingRate": "200",
            "conversionFactor":0.030517578125
        }
    ],
    "duration": 99.384483839
}
, ... 

Get the CSV files for a Recording

# To download all CSV files for all Signals in a Recording, use the rawData property as download link
for signal in recordingMetadata["signals"]:
    response = requests.get(signal["rawData"], allow_redirects=True)
    open(recordingID+"_"+signal["type"]+"_"+str(signal["channel"])+".csv", 'wb').write(response.content)

If successful, this will download a CSV file for each Signal in the Recording to your current working directory.

Plot a Recording Programmatically

You can use the same code as specified above but you already have the timestamp and conversion factors available in your API payloads.

# Imports
...
# Load CSV files
...
# Get the start timestamp
start_ts = recordingMetadata["startDate"]

# Convert Signals to correct units
idx = 0
# Where [idx] is the index of the Signal as specified in `recordingMetadata`
cF_EMG = recordingMetadata["signals"][idx]["conversionFactor"]  
data["channel"] *= cF_EMG

# Plot the data
...

Copyright © 2020 Byteflies