Merge Tool¶

Merge Tool is a program designed to effectively merge SimObserver data in the event that a participant's drive is interrupted. This often occurs due to study interruptions such as SimCreator shut downs. Merge Tools is run on these separate data files from the same participant's drives and merges them into one file to ensure a uniform presentation of the data.

For example, if during a study, called 'ExampleProject', Participant 3's original drive 1 was interrupted
prematurely two times, there would be 3 separate data files created, with different drive IDs. These drive files could be merged to create a single drive file for further processing.

Merge Types¶

Merge Tool has two different kinds of merge options. This section will highlight each of them.

Sequential¶

Sequential Merge is a file merge based on time. Once all the files in the merge directory (-d) have been processed by the program, Merge Tool will use the SimCreator's recorded SimTime metric to concatenate the files. This is accomplished by iterating through every SimTime column, reading the last SimTime value of the current data file, and adding that value as a constant through the next data file's SimTime column.

Spatial¶

Spatial Merge is a file merge based on X & Y coordinate positions. Much like the Sequential merge, Spatial Merge will perform the same SimTime corrections as multiple drives for the same participant are merged. However, along with the time correction, Spatial Merge also analyzes SimCreator's recorded XPos and YPos metrics. out_frame is the variable name in which all the merged data will end up, before a csv is created. For every next_frame (files that comes after the first for the participant), the row with the minimum distance from the last position in out_frame will be the starting index of the data that is appended to out_frame.

Usage¶

This section covers how Merge Tool is executed as well as the command line parameters that the user is expected to supply. - Command Line execution - Ran from where pydre is ran from - Ex: python pydre_merge.py -d 'exampleMergeDirectory' -t 'spatial or sequential' - Two Args. 1. -d (DataFile directory). The directory to be merged, is occupied by multiple drives for at least one participant. 2. -t (Merge Type). Chosen as either "Spatial" or "Sequential", determines which type of merge will be performed.

Testing¶

Within the pydre folder, there will be a test directory with a sub-directory called MergeTools. This sub-directory contains all the materials for testing pydre_merge.py. To run the tests, navigate to the pydre module directory (top level) and use the command: python -m unittest tests\MergeTools\test_merge.py.

Testing Explanation¶

• There are two, primary functions to test within merge_tool.py, one for each merge type (Sequential and Spatial). These two functions are found at (considering pydre.merge_tool as p_merge) p_merge.MergeTool.sequential_merge() and p_merge.MergeTool.spatial_merge()(see Overview above for merge-type explanations).

• To properly test each of these, the testing functions must supply the two, required parameters without using the command line (refer to Usage above). Briefly, below will address how each argument is supplied in the testing.
  

• Merge Directory (-d) - Within MergeTools\test_dats_to_merge\ there are a host of directories. Each of these directories is unique per test case and will contain particular data files to simulate the intended use of the program.

• Merge Type (-t) - Supplied as a string within the test case, unique to the target MergeTool() function of the current test.

Results and Expected¶

This section will cover the results from running the program, the expected results, and how the two will be compared for the sake of testing.

Results:

After calling the target function (either Sequential or Spatial) with the two parameters from the test case, the program, after running successfully, will create a .csv file per participant found in the Merge Directory (-d). This .csv file(s) can always be found in MergeTools\test_dats_to_merge\current-dir\MergedData\. With that, using the pandas module for Python, it is easy to recover the contents of the .csv.

Expected:

Within MergeTools\expected_csv there will be a sub-directory for every test. Each of these sub-directories will be named with the prefix expected_ followed by the Merge Directory (-d) name. (Ex: with "-d test_dats_to_merge/ref_dir" the expected csv will be found at MergeTools\expected_csv\expected_ref_dir\). These directories and their contents must be prepared prior to testing, as these .csv files' are recovered using pandas as well.

With both the results and expected results in the testing program, the contents of both .csv files will be compared. In order for the MergeTool() function to pass the test, it must produce a .csv file that is exactly the same as the corresponding file found in expected_csv.

Note: For streamlined testing, MergedData is completely removed after each test case. This ensures test_merge.py will have easy access to only the pertinent results of the current, running test case.