Redo retrieval of measurement list, dataTimeSeries and mlAct to fix major issues with incorrect sorting of data. (#141) (#143)

* v1.39.0

-- Fix matlab generated error displaying measurement list visible channels. Take MeasListVis field out of processing stream input (ProcInputClass) since it does not actually contribute to proc stream processing and move it to TreeNodeClass field chVis.

* v1.40.0

-- Fix major issue with incorrect sorting of data and measurement list.
-- Change unit tests to include wavelength when scrambling channel order. Also make scrambling for unit tests an option

* v1.40.1

-- Add back display of standard error bars, and x and y-axis labels to MainGUI data display.
-- Fix current element condition name display being cut off in PlotProbeGUI.m

* -- Fix bug in shared lib synching code which incorrectly ignores differences in files with .cfg extensions thus ignoring differences in AppSettings.cfg

* v1.41.0

-- Add ability to plot selected measurements in exported HRF file from ExportDataGUI
-- In ConfigFileClass exclude AppSettings.cfg in submodules folder under Homer3 root folder

* v1.42.0

-- Fix issue with probe.snirf or any other snirf files with no data or intentionally corrupt data being accepted as valid.
-- Fix Nirs2Snirf not working for some some .nirs datasets which do not have CondNames. Add default CondNames to NirsClass.ErrorCheck()
-- Fix display of mlAct in MainGUI broken after the measurement list order issue.
-- For diagnostic purposes save error code in err field of all the SnirfClass sub-classes
-- Add back to MainGUI making channels invisible by mouse right-clicking on them.

* v1.42.1

-- Fix bug removing channels from display by mouse right-clicking in MainGUI axesSDG. Fixed incorrect distance criteria for source and detector from end points on clicked line.

* v1.42.2

-- Fix error when loading snirf with corrupt stim data. Added error checking to flag bad stim data. Then add file with bad stim data and discard bad stims while displaying warning.
-- Improve warning reporting for files that were allowed to load but with warnings.
-- Fix matlab error in backward compatability code when derived output folder is same as root group folder

* -- Log the current processing stream function call chain every time RUN button is clicked in MainGUI to generate derived data.
-- Minor code formatting in SubjClass.m

* v1.42.4

-- Fix bugs found by Meryem in fNIRS 2018 course SS_DEMO_1, Finger_Tapping_*.snirf.

Errors when running processing stream in
1. hmrR_MotionArtifact fixed by getting rid of 'reshape' option to DataTimeSeries()
2. hmrR_OD2Conc.m fixed by getting rid of 'reshape' option to DataTimeSeries()
3. hmrR_GLM.m fixed in DataClass.m by NOT assuming source and detector indices do not skip numbers

-- Just like for hmrR_MotionArtifact.m no need to reshape for hmrR_MotionArtifactByChannel.m since the

-- Better error/warning logging in DataFilesClass.m and FileLoadSaveClass.m

* v1.43.0

-- Set inactive channel data to NaN. That way they don't show up in MainGUI display. Implement this in DataClass: inactive channel is a channel for which all data is exactly zero (not approximately zero).

-- Fix an issue in ProbeClass which incorrectly upload sourcePos3D or detectorPos3D as a column vector IF AND ONLY IF it is the ONLY optode in the probe. This leads to incorrect result in at least one user function processing hmrR_OD2Conc in which the calculation of rho is sensitive to row vs column vector.

-- Fix several issues in incorrect unit tests results for Example9_SessRuns because of file naming in Simple_Probe_run* which contains '_' between Simple and Probe which are interpreted as a subject and session name. This is not how the original .nirs benchmark was processed based on the Simple_Probe*_. In order to match the original subject/session breakdown need to get rid of '_'.

-- Fix another unit test bug which has to load saved derived data in compareDcAvg.m before comparing to benchmark.

-- Fix another unit test bug in unitTest_BandpassFilt_LPF.m lpf wasn't being changed no matter what the value was because hmrR_BandpassFilter() changed to hmrR_BandpassFilter_old()

-- Add to unit tests simulation of occasional random errors (simulateDataError.m and generateErrorOddsConstant.m) to do a check of whether the unit test is really checking data against the benchmark correctly.

* -- Fix bug in SnirfClass .snirf file fomat conversion to NirsClas .nirs.
-- Set default simulation of errors to none in UnitTests_Init.m
-- Add namespace DataTreeClass to shared library Utils function points_on_line.m

* v1.44.0

-- Fix ordering issue with active/inactive (mlAct) channels in user functions and MainGUI display. Use sources, detectors, wavelength matrix rather than a logical vector.
-- When manually pruning channel only prune at the current wavelength.
-- Implement missing application of active/inactive (mlAct) channels at the higher averaging user functions: hmrE_RunAvg.m, hmrG_SubjAvg.m, hmrS_SessAvg.m
-- Fix incorrect nTrials calculation at the group level
-- Fix PlotProbeGUI/plotProbe.m display issue as a result of fixing the MeasList ordering issue.

* v1.45.0

-- Fix bug in channel exclusion in hmrR_PruneChannels.m
-- Fix display of manually pruned channels in DisplayAxesSDG.m

* v1.46.0

-- Fix plotProbe.m by rewriting the plotting to be based on SNIRF style 2D measurement list rather than .nirs era wavelength based measurement.

* v1.47.0

-- Fix mlAct in hmrR_tCCA.m
-- In PlotProbeGUI get condition dynamically from parent gui, that is MainGUI
-- In PlotProbeGUI.m add back ability to display OD HRF and also any data type that is currently selected in MainGUI

* v1.48.0

-- Fix very slow processing at the group, subject and session levels and slow display. All were caused by the same issue: very inefficient DataClass.GetDataTimeSeries('reshape') method. Changed it's code to be much more efficient.
-- Fix typo in plotProbeAndSetProperties.m causing matlab-generated error
-- Minor improvement to display of location of current processing in MainGUI listboxGroupTree when calculating provcessing stream

* v1.48.1

-- Fix typo in hmrR_MotionCorrectWavelet.m causing matlab-generated exception
-- More fixes in user functions hmrR_MotionArtifactByChannel.m and hmrR_MotionCorrectCbsi.m of inactive channels mlAct
-- Added another performance enhancer in DataClass.GetDataTimeSeries() to speed up retrieval of measurement list and thus data time series. Added cache for measurement list matrix to avoid recalculating it.

* v1.48.2

-- Capture and log to log file the function call stack when exception error occurs while running processing stream.

* v1.48.3

-- Fix some child gui repositioning issues when MainGUI is upodated especially ProcStreamOptionsGUI
-- Don't redisplay ProcStreamOptionsGUI if function call list is exactly same as last time.

* v1.48.4

-- Fix time exclusion and motion correction by channel having incorrect channel order. Modify tIncCh to include measurement list info about sources, detectors  and data type.
-- Fix display of time exclusion and motion correction by channel showing incorrect channels. Clean up and rewrite DisplayExcludedTime() function.
-- Improve speed of display function GetMeasurementList to use ml matrices instead of structures. Change definitions of GetMeasurementList() functions to be uniform and compatible with each other up and down the call stack, as well as having separate arguments for matrixOption ('matrix') and dataType ('dod', 'dc', dcAvg', etc).
-- Minor graphics adjustment to MainGUI datatype listboxes for wavelength and HbType to be exactly overlapping

* v1.50.0

-- Fix another channel order bug in DataClass.GetDataTimeSeries. order need to match linear order when squeezing dimensions > 1 into 1D. This caused incorrect results to be displayed for concentration HbT after running  hmrR_MotionCorrectCbsi
-- Fix minor display issue in MainGUI.m icorrectly enabling OD and concentration radio buttons even though no results exists.
-- Add diagnostic/debug functions to unit test to generate simulated data time series whose data identifies the channel it originated in.

* v1.51.0

-- Fix channel order issues excluded time by channel in hmrR_MotionCorrectSplineSG. Change GetDataTimeSeries to return measurement list associated with the data time series as well as adding other options to control channel order.
-- Fix matlab exception bug in plot probe when clicking non-run, non-hrf data in MainGUI. Fix in plotProbeAndSetProperties.m, updateData() function which was cleaned up in addition to fixing problem.
-- Fix incomplete isequal method in ProcStreamClass comparing 2 objects. This caused active ProcStreamOptionsGUI not to update correctly when removing function from function call chain in ProcStreamEditGUI.
-- Fix issue in SubjClass when processing multiple sessions incorrectly adding trials from all the sessions.

* v1.51.1

-- Fix matlab exception in hmrR_MotionCorrectCbsi.m because call to GetDataTimeSeries method was not updated to match new function definition.
-- MainGUI and PlotProbeGUI/plotProbe.m display improvements: thinner data plot lines, exclude time patches incorrectly changing y lim.

* v1.51.2

-- Fix matlab exception in hmrR_MotionCorrectSplineSG because of typo missing 'matrix' argument in GetMeasurementList(). hmrR_MotionCorrectSplineSG expects measurement list in matrix form rather than structure.

* Diagnostic code which does not effect main Homer3 operation (so version number stays the same):

-- Change generateSimData.m to include all data files in a group. This function replaces the files real data with fake data that has the channel ID (source idx, detector index, data type idx, condition idx) embedded in the data itself for diagnosing/uncovering channel order issues.
-- Add more sim data user functions

Author: jayd1860

DataTree/AcquiredData/DataFiles/DataFilesClass.m

@@ -20,7 +20,7 @@
     methods
 
         % ----------------------------------------------------
-        function obj = DataFilesClass(varargin)            
+        function obj = DataFilesClass(varargin)
             obj.files = FileClass.empty();
             obj.filesErr = FileClass.empty();
             obj.filetype = '';
@@ -472,25 +472,34 @@ function ErrorCheck(obj)
                 if obj.files(ii).isdir
                     continue;
                 end
-                filename = [obj.files(ii).rootdir, obj.files(ii).name];
+                filename = [obj.rootdir, obj.files(ii).name];
                 eval( sprintf('o = %s(filename);', constructor) );
                 if  o.GetError() < 0
-                    obj.logger.Write('DataFilesClass.ErrorCheck:   FAILED error check - %s will not be added to data set\n', filename);
+                    obj.logger.Write('DataFilesClass.ErrorCheck - ERROR: In file "%s" %s. File will not be added to data set\n', obj.files(ii).name, o.GetErrorMsg());
                     errorIdxs = [errorIdxs, ii]; %#ok<AGROW>
-                elseif  contains(o.GetErrorMsg(), 'WARNING: ''data'' corrupt and unusable')                    
-                    obj.logger.Write('DataFilesClass.ErrorCheck:   WARNING data is unusable - %s will not be added to data set\n', filename);
+                elseif  contains(o.GetErrorMsg(), '''data'' field corrupt and unusable')                    
+                    obj.logger.Write('DataFilesClass.ErrorCheck - WARNING: In file "%s" %s. File will not be added to data set\n', obj.files(ii).name, o.GetErrorMsg());
                     errorIdxs = [errorIdxs, ii]; %#ok<AGROW>
+                elseif  contains(o.GetErrorMsg(), '''data'' field is invalid')                    
+                    obj.logger.Write('DataFilesClass.ErrorCheck - WARNING: In file "%s" %s. File will not be added to data set\n', obj.files(ii).name, o.GetErrorMsg());
+                    errorIdxs = [errorIdxs, ii]; %#ok<AGROW>
+                elseif ~isempty(o.GetErrorMsg())
+                    obj.logger.Write('DataFilesClass.ErrorCheck - WARNING: In file  "%s"  %s. File will be added anyway.\n', obj.files(ii).name, o.GetErrorMsg());
+                    dataflag = true;
                 else
                     dataflag = true;
                 end
+                if ~isempty(o.GetErrorMsg())
+                    obj.files(ii).SetError(o.GetErrorMsg());
+                end
                 hwait = waitbar_improved(ii/length(obj.files), hwait, msg);
             end
+            
             if dataflag==false
                 obj.files = FileClass.empty();
             else
                 for jj = 1:length(errorIdxs)
                     obj.filesErr(end+1) = obj.files(errorIdxs(jj)).copy;
-                    obj.filesErr(end).SetError('Invalid Data Format');
                 end
             	obj.files(errorIdxs) = [];
                 obj.nfiles = obj.nfiles - length(errorIdxs);

DataTree/AcquiredData/DataFiles/DeleteDataFiles.m

@@ -70,7 +70,7 @@ function DeleteDataFiles(varargin)
 % Get final list of data files
 datafiles = mydir(dirname);
 if iscell(datafiles0)
-    for ii=1:length(datafiles0)
+    for ii = 1:length(datafiles0)
         datafiles(ii) = mydir([dirname, datafiles0{ii}]);
     end
 elseif ischar(datafiles0)
@@ -81,7 +81,7 @@ function DeleteDataFiles(varargin)
 
 
 % Delete data files
-for ii=1:length(datafiles)
+for ii = 1:length(datafiles)
     if datafiles(ii).isdir
         continue;
     end

DataTree/AcquiredData/DataFiles/FileClass.m

@@ -19,7 +19,7 @@
         rootdir
         err
         logger
-        errmsgs
+        errmsg
     end
 
     methods
@@ -38,9 +38,9 @@
             obj.filename   = '';
             obj.map2group  = struct('iGroup',0, 'iSubj',0, 'iSess',0, 'iRun',0);
             obj.rootdir    = '';
-            obj.err        = -1;          % Assume file is not loadable
+            obj.errmsg     = '';          % Assume file is not loadable
             obj.logger     = InitLogger(logger);            
-            obj.errmsgs    = {'Invalid Data Format','Invalid File Name'};
+            obj.errmsg    = '';
 
 
             if nargin==0
@@ -538,22 +538,19 @@ function NameConflictFixed(obj)
 
         % -----------------------------------------------------
         function err = GetError(obj)
-            err = obj.err;
+            err = ~isempty(obj.errmsg);
         end
 
 
         % -----------------------------------------------------
-        function err = SetError(obj, err)
-            if ischar(err)
-                err = find(strcmp(obj.errmsgs, err));
-            end
-            obj.err = err;
+        function SetError(obj, errmsg)
+            obj.errmsg = errmsg;
         end
 
 
         % -----------------------------------------------------
         function msg = GetErrorMsg(obj)
-            msg = obj.errmsgs{abs(obj.err)};
+            msg = obj.errmsg;
         end
 
     end

DataTree/AcquiredData/DataFiles/FileLoadSaveClass.m

@@ -142,6 +142,10 @@ function SetError(obj, err)
 
         % -------------------------------------------------------
         function err = GetError(obj)
+            err = [];
+            if isempty(obj)
+                return
+            end
             err = obj.err;
         end
 

DataTree/AcquiredData/DataFiles/Hdf5/HDF5_Transpose.m

@@ -8,8 +8,9 @@
 % HDF5 stores them in row-major order. We want to transpose the data to agree
 % with the file format's storage order.
 if (~isrow(val) && ~iscolumn(val))              || ...      % Matrices
-     ~isempty(findstr('multidim', options))      || ...      % Force multi-dimensional even if vector
-     ~isempty(findstr('2D', options))                        % Force 2D even if vector
+     ~isempty(findstr('multidim', options))     || ...      % Force multi-dimensional even if vector
+     ~isempty(findstr('2D', options))           || ...      % Force 2D even if vector
+     ~isempty(findstr('3D', options))                       % Force 3D even if vector
 
     val = permute(val, ndims(val):-1:1);
 

DataTree/AcquiredData/DataFiles/Snirf2Nirs.m

@@ -0,0 +1,27 @@
+function Snirf2Nirs(rootdir, options)
+if ~exist('rootdir','var')
+    rootdir = pwd;
+end
+if ~exist('options','var')
+    options = 'sort';
+end
+rootdir = filesepStandard(rootdir);
+files = DataFilesClass(rootdir, 'snirf');
+files = files.files;
+for ii = 1:length(files)
+    if files(ii).IsDir()
+        continue
+    end
+    fname = filesepStandard([files(ii).rootdir, files(ii).name]);    
+    s = SnirfClass(fname);
+    n = NirsClass(s);
+    if optionExists(options, 'sort')
+        n.SortData();
+    end
+    [pname, fname] = fileparts(fname);
+    fnameNew = [filesepStandard(pname), fname, '.nirs'];
+    fprintf('Converting %s to %s\n', fname, fnameNew)
+    n.Save(fnameNew);
+end
+
+

DataTree/AcquiredData/Nirs/NirsClass.m

@@ -132,6 +132,15 @@ function SortStims(obj)
         end
 
 
+        % ---------------------------------------------------------
+        function SortData(obj)
+            [obj.SD.MeasList, order] = sortrows(obj.SD.MeasList);
+            obj.d = obj.d(:,order);
+            [obj.SD.MeasList, order] = sortrows(obj.SD.MeasList,4);
+            obj.d = obj.d(:,order);
+        end
+        
+        
         % ---------------------------------------------------------
         function err = LoadMat(obj, fname, ~)
             err = 0;
@@ -468,10 +477,9 @@ function SaveMat(obj, fname, options)
 
             % Always sort stimulus conditions and associated stims
             % to have a predictable order for display
-            objnew.s          = obj.s;
-            obj.SortStims();
-            
+            objnew.s          = obj.s;            
             objnew.CondNames  = obj.CondNames;
+            objnew.SortStims();
         end
 
 
@@ -650,25 +658,36 @@ function SetCondNames(obj, val)
 
 
         % ---------------------------------------------------------
-        function SD = GetSDG(obj, option)
+        function SD = GetSDG(obj, ~)
             SD = obj.SD;
         end
 
 
         % ---------------------------------------------------------
-        function ml = GetMeasList(obj, iBlk)
+        function ml = GetMeasList(obj, ~)
             ml = obj.SD.MeasList;
         end
 
 
+        % ---------------------------------------------------------
+        function ml = GetMeasurementList(obj, ~, ~)
+            ml = MeasListClass();
+            for ii = 1:size(obj.SD.MeasList,1)
+                ml(ii).sourceIndex = obj.SD.MeasList(ii,1);
+                ml(ii).detectorIndex = obj.SD.MeasList(ii,2);
+                ml(ii).wavelengthIndex = obj.SD.MeasList(ii,4);
+            end
+        end
+        
+        
         % ---------------------------------------------------------
         function wls = GetWls(obj)
             wls = obj.SD.Lambda;
         end
 
 
         % ---------------------------------------------------------
-        function SetStims_MatInput(obj,s,t,CondNames)
+        function SetStims_MatInput(obj, s, ~, ~)
             obj.s = s;
         end
 
@@ -703,13 +722,13 @@ function SetConditions(obj, CondNames)
 
 
         % ---------------------------------------------------------
-        function srcpos = GetSrcPos(obj,option)
+        function srcpos = GetSrcPos(obj, ~)
             srcpos = obj.SD.SrcPos;
         end
 
 
         % ---------------------------------------------------------
-        function detpos = GetDetPos(obj,option)
+        function detpos = GetDetPos(obj, ~)
             detpos = obj.SD.DetPos;
         end
 
@@ -728,13 +747,13 @@ function SetConditions(obj, CondNames)
 
 
         % ----------------------------------------------------------------------------------
-        function n = GetDataBlocksNum(obj)
+        function n = GetDataBlocksNum(~)
             n = 1;
         end
 
 
         % ----------------------------------------------------------------------------------
-        function [iDataBlks, ich] = GetDataBlocksIdxs(obj, ich)
+        function [iDataBlks, ich] = GetDataBlocksIdxs(~, ich)
             iDataBlks = 1;
             ich={ich};
         end
@@ -882,30 +901,30 @@ function MoveStims(obj, tPts, condition)
 
 
         % ----------------------------------------------------------------------------------
-        function SetStimTpts(obj, icond, tpts)
+        function SetStimTpts(~, ~, ~)
             return;
         end
 
 
         % ----------------------------------------------------------------------------------
-        function tpts = GetStimTpts(obj, icond)
+        function tpts = GetStimTpts(~, ~)
             tpts = [];
         end
 
 
         % ----------------------------------------------------------------------------------
-        function SetStimDuration(obj, icond, duration)
+        function SetStimDuration(~, ~, ~)
             return;
         end
 
         % ----------------------------------------------------------------------------------
-        function duration = GetStimDuration(obj, icond)
+        function duration = GetStimDuration(~, ~)
             duration = [];
         end
 
 
         % ----------------------------------------------------------------------------------
-        function SetStimAmplitudes(obj, icond, vals)
+        function SetStimAmplitudes(~, ~, ~)
             return;
         end
 
@@ -1027,7 +1046,7 @@ function RenameCondition(obj, oldname, newname)
 
 
         % ----------------------------------------------------------------------------------
-        function b = CopyProbe(obj, SD)            
+        function CopyProbe(obj, SD)            
             fields = propnames(obj.SD);
             for ii = 1:length(fields)
                 if eval( sprintf('isfield(SD, ''%s'')', fields{ii}) )
@@ -1082,15 +1101,19 @@ function ConvertSnirfData(obj, snirf)
         function ConvertSnirfStim(obj, snirf)
             obj.s = zeros(length(obj.t), length(snirf.stim));
             for ii = 1:length(snirf.stim)
-                k = round(nearest_point(obj.t, snirf.stim(ii).data(:,1)));
-                for jj = 1:length(k)
-                    if k(jj) == 0
-                        k(jj) = 1;
+                if isempty(snirf.stim(ii).data)
+                    ik = [];
+                else
+                    [~,ik] = nearest_point(obj.t, snirf.stim(ii).data(:,1));
+                end
+                for jj = 1:length(ik)
+                    if ik(jj) == 0
+                        ik(jj) = 1;
                     end
-                    if k(jj) > length(obj.t)
-                        k(jj) = length(obj.t);
+                    if ik(jj) > length(obj.t)
+                        ik(jj) = length(obj.t);
                     end
-                    obj.s(k(jj),ii) = 1;
+                    obj.s(ik(jj),ii) = 1;
                 end
                 obj.CondNames{ii} = snirf.stim(ii).name;
             end
@@ -1127,36 +1150,38 @@ function ErrorCheck(obj)
                     obj.SD.SrcGrommetType{ii} = 'none';
                 end
             end
-            
             if isempty(obj.SD.DetGrommetType)
                 for ii = 1:size(obj.SD.DetPos,1)
                     obj.SD.DetGrommetType{ii} = 'none';
                 end
             end
-            
             if isempty(obj.SD.DummyGrommetType)
                 for ii = 1:size(obj.SD.DummyPos,1)
                     obj.SD.DummyGrommetType{ii} = 'none';
                 end
             end
-            
             if isempty(obj.SD.SrcGrommetRot)
                 for ii = 1:size(obj.SD.SrcPos,1)
                     obj.SD.SrcGrommetRot(ii) = 0;
                 end
             end
-            
             if isempty(obj.SD.DetGrommetRot)
                 for ii = 1:size(obj.SD.DetPos,1)
                     obj.SD.DetGrommetRot(ii) = 0;
                 end
             end
-            
             if isempty(obj.SD.DummyGrommetRot)
                 for ii = 1:size(obj.SD.DummyPos,1)
                     obj.SD.DummyGrommetRot(ii) = 0;
                 end
             end
+            if isempty(obj.CondNames)
+                for ii = 1:size(obj.s,2)
+                    if length(obj.s(:,ii)) == length(obj.t)
+                        obj.CondNames{ii} = num2str(ii);
+                    end
+                end
+            end
 
         end
 

DataTree/AcquiredData/Snirf/AuxClass.m

@@ -110,6 +110,8 @@
                 end
 
             end
+            
+            obj.SetError(err); 
         end