Movesense Sensor Data Storage

This document describes the data storage features of the Movesense sensor.

Technical capabilities

The Movesense sensor contains either 386 kB EEPROM memory for storing data. To assist software developer to utilize this memory the Movesense Core Library contains two services DataLogger and Logbook that make it effortless to store and retrieve the data from the sensor. The Logbook EEPROM memory does not have a filesystem, but is instead organized as a circular buffer. This means that as the memory gets full, the old data gets overwritten. To distinguish between recordings the DataLogger uses a LogID that is incremented every time that the sensor is restarted or a new log is created by POST to /Mem/Logbook/Entries.

Alternatively the EEPROM can be used via the low level /Component/EEPROM -API (requires enabling the EepromService module).

DataLogger service

To help logging data from Whiteboard ResourceProviders the Movesense Core Library contains a DataLogger service. This is a simple service that can be configured to subscribe to any whiteboard resources (even the ones you create in the application!) and record all the data (notifications) they provide. DataLogger has two resources: /Mem/DataLogger/Config and /Mem/DataLogger/State. Config contains the current configuration of the logger (which data resources to log), and state describes the state of the logger (see: DataLoggerState in datalogger.yaml)

The DataLogger service stores the data in the EEPROM as a circular buffer. The data is stored in binary format using code that is generated automatically from the yaml files as a part of the application build process. The generated code has some limitations, such as:

if there are multiple arrays, they must be of equal length (like in /Meas/IMU6)
no string support
optional data still takes space even if it has null value

The DataLogger stores data one entry at the time until the internal buffer (one for each configured path) is full and then synchronizes the data to the EEPROM memory. If the memory gets full the DataLogger starts over from the beginning of the designated area in EEPROM and begins overwriting the oldest data.

NOTE: The storage format can change when the firmware changes, so it is a good idea fetch data before and empty the Logbook memory after firmware update.

DataLogger and Logbook memory area

To specify which area of EEPROM is reserved for the DataLogger/Logbook, use the LOGBOOK_EEPROM_MEMORY_AREA macro in App.cpp of your firmware project:

LOGBOOK_EEPROM_MEMORY_AREA(offset, size_in_bytes);

The first value is the start address of the memory area (offset from the beginning of the memory) and second one is the size in bytes or MEMORY_SIZE_FILL_REST. In current implementation, both values must be divisable by 256 and size less than 2048 will probably not work. Maximum size in current sensor implementation is 384*1024. To use the rest of EEPROM from start address to the end, specify size as MEMORY_SIZE_FILL_REST.

NOTE: pay special attention that you do not overlap the LOGBOOK_MEMORY_AREA and DEBUG_EEPROM_MEMORY_AREA (DebugService's EEPROM storage) or accidentally overwrite Logbook with /Component/EEPROM -API.

DataLogger usage

To use datalogger to log data, one needs to

Configure the DataLogger. This is done by PUT:ing DataLoggerConfig -object to the /Mem/DataLogger/Config -resource. For example to log accelerometer and gyro data with different samplerates, you could use following configuration:

{
    "dataEntries" : {
        "dataEntry" : [{
                "path" : "/Meas/Acc/104"
            }, {
                "path" : "/Meas/Gyro/13"
            }
        ]
    }
}

To start logging, PUT value DATALOGGER_LOGGING (=3) to /Mem/DataLogger/State resource
To stop logging, PUT value DATALOGGER_READY (=2) to /Mem/DataLogger/State resource

It is possible to start and stop datalogger multiple times for one log as well as change configuration.

DataLogger usage on sensor firmware

Using DataLogger on sensor firmware can be studied in firmware samples

Logbook service

Logbook service is used to access the data stored in the EEPROM memory by the DataLogger. The yaml file contains lots of different things, but Movesense sensor only supports a small subset of that API (see below). The Logbook service can enumerate the contents of the EEPROM memory (/Mem/Logbook/Entries resource), provide the data of the made recordings, create a new log file and wipe the data memory.

The recordings are returned in Suunto Oy's proprietary SBEM-format, which can be converted to JSON using the Movesense Mobile Library's (MDS) Logbook helper proxy service. A good description of the binary format can be found in StackOverflow. To find out more about how to use the Logbook from MDS, see the DataLoggerSample in MDS Android samples.

Getting data from Logbook (sensor API)

To get recording from the Movesense sensors EEPROM storage (without the help of MDS helper service), you need to:

Do GET on /Mem/Logbook/Entries. This returns a list of LogEntry objects. If the status was HTTP_OK, the list is complete. If the result code is HTTP_CONTINUE, you must GET again with the parameter StartAfterId set to the Id of the last entry you received and you'll get the next batch of entries.

In sensor firmware code, you can get the object out in the onGetResult() with code: c++ const auto &logEntries = rResultData.convertTo<const WB_RES::LogEntries &>();
Choose the Log that you are interested in and notice the Id of it.
Fetch the descriptors with GET to /Logbook/byId/{LogId}/Descriptors. This returns a bytestream with the similar HTTP_CONTINUE handling as above. However you must keep re-requesting the GET until you get error or HTTP_OK, or the Logbook service will stay "in the middle of the stream" (we hope to remove this limitation in the future).

c++ const auto &stream = rResultData.convertTo<const wb::ByteStream &>();
Fetch the data with GET to /Logbook/byId/{LogId}/Data. This returns also a bytestream (just like the /Logbook/Descriptors above).

OR (in versions >= 2.1.x)

Fetch the data with SUBSCRIBE to /Logbook/byId/{LogId}/Data. This will send the whole log as notifications of format: json { "offset": 12345, "bytes":[1,2,3,4,5,.....,99] } and when the whole log is sent, a few copies of empty notification are sent to mark end-of-file, where offset is the final size of the file: json { "offset": 98765, "bytes":[] } (This method enables verifying the integrity of the received file by checking that no data is missing.)
Convert the data using the converter tools or your own code.

Getting data from Logbook (using MDS's proxy service)

To get recording from the Movesense sensors Logbook service using the MDS:

Call get() with URL "suunto://MDS/Logbook/{sensor_serial}/Entries". This calls the sensors internal /Mem/Logbook/Entries repeatedly and returns the full list of LogEntry objects from the sensor.
Choose the Log that you are interested in and notice the Id of it.
Fetch the whole data of the log as JSON by doing get() with URL "suunto://MDS/Logbook/{sensor_serial}/ById/{LogId}/Data". This GET's both Descriptors and Data from the sensor (or subscribes to the Data-notifications if available) and performs the sbem2json conversion to it and returns the full data as a JSON string.

NOTE: The resource GET supports optional parameter "ToFile", which allows direct writing of the resulting JSON into a file in the applications private data directory: {"ToFile":"my_json_filename"}

Creating a New Log

To start a new log, send POST to /Mem/Logbook/Entries. The response will be the Id of the new log.

Finding out if Logbook is full

If the Logbook EEPROM is full of data, the datalogger will overwrite the oldest data. To avoid this and app can check the Logbook fullness by either GET:ing or SUBSCRIBING a path /Mem/Logbook/IsFull. Once the Logbook has less than small abount (about 1kB) of free memory, the ../IsFull will return true and it will remain so until the logbook is emptied.

Emptying Logbook

To empty the whole logbook (and return IsFull -status to "false"), send DELETE request to path /Mem/Logbook/Entries. Note: This will reset the whole Logbook area.

Low level EEPROM Access

The Movesense module EepromService implements the low level access API for the EEPROM memory. The EEPROM -API exposes the inner structure of the EEPROM memory so it is a bit more complicated to use.

Memory layout

Finding EEPROM memory layout is done by issuing GET to /Component/Eeprom/{EepromIndex}/Info -path incrementing EepromIndex starting from 0 until error is returned. The returned information contains the size of the EEPROM chip in bytes.

The current Movesense sensors have the follwing layout:

Chip #0: 128kB (=131072 B)
Chip #1: 256kB (=262144 B)

Accessing Memory Content

To access the EEPROM content one must know which chip the area is. To read, do a GET and give as a parameter the Addr and Len. To write, do a PUT with parameters Addr and Data (="data to write").

When using EEPROM -API care must be taken to avoid sections of the EEPROM memory that contain the Logbook (if it is enabled) and DEBUG_MEMORY_AREA.

Note: The EEPROM chips in use have MTBF (mean time before failure) of one million writes on same address. Care must be taken to avoid writing so much in same addresses that this becomes a problem during lifetime of the product.