Data access and processing
Data from DanMAX is stored centrally at MAX IV and is available for further processing using MAX IV resources – or for download using either the Globus connect service or sftp. Detailed information about downloading the data can be found here: Data access
It is possible to access MAX IV computing resources by setting up a VPN connection to MAX IV. Instructions can be found here: VPN connection to MAX IV
File location and structure:
All scans at DanMAX results in a number of files written in the HDF5 format (*.h5). The files are written to be NeXuS compatible.
The raw files are stored in /data/visitors/danmax/#Proposal/#Visit/raw/ while reduced files are stored in /data/visitors/danmax/#Proposal/#Visit/process/. Separate experiments/samples may be stored in their own folder.
Each scan will be saved with the name scan-####.h5 where #### is a sequential number. It is important to keep track of this number and the corresponding sample/experiment, e.g. by using the elogy logbook (https://elogy.maxiv.lu.se). The scan-####.h5 file is a master file that contains the command issued to perform the scan, the scanned parameters (e.g. motor positions), experimental channels (e.g. beam intensity, position, temperature) a snapshot of the instrument at the beginning of the scan and links to the detector datafiles. The data from the detectors are located in separate files, e.g. scan-####_pilatus.h5 and scan-####_falconx.h5. The link from the master file to the detector data is relative, and thus all files must be located in the same folder.
To visualize the data and the file structure we recommend to use silx view (http://www.silx.org/doc/silx/latest/install.html) which is available on the control computers. As the area detector data is compressed using a bit-shuffle algorithm, it is necessary to install hdf5plugin (https://pypi.org/project/hdf5plugin/).
A description of the experimental snapshot parameters is given below.
Data processing and analysis (PXRD2D):
An automated azimuthal integration pipeline is available based on the MATFRAIA algorithm developed by the Birkedal group at Aarhus University (more details will be available soon). To prepare the pipeline a .poni file and a detector mask (.npy) is needed. pyFAI-calib2 is installed on the control computers and is commonly used to generate these files. More information about the pipeline can be found on the beamline wiki (https://wiki.maxiv.lu.se/index.php/Pilatus_2M_CdTe). The integrated data will be available as saved in the /data/visitors/danmax/#Proposal/#Visit/process/azint/ folder mirroring the folder name and scan number in the /raw/ folder.
It is also possible to process the data in Jupyter notebooks using the MAX IV Jupyterhub; http://jupyterhub.maxiv.lu.se (access requires VPN access), where you need to log in using your DUO credentials. To have access to the regular environment with HDF5, silx and pyFAI tools you should select the ‘HDF5 / Simple Analysis /GPU’ kernel. You can find more information on using the Jupyterhub on the DanMAX Wiki page (access requires VPN access).
The experimental snapshot is a recording of the essential instrument parameters at the beginning of the scan. The snap shot is contained in the master file for each scan, and is located under /entry/instrument/start_positioners/. Some names are self-explanatory and not listed here.
ea_ppm3_1 Sample to detector distance (Pilatus 2M) – may not reflect the actual distance if calibration has not been performed after changing configuration.
ea_xrf_z Sample to detector distance (Fluorescence detector) – may not reflect the actual distance if calibration has not been performed after changing configuration.
ea_slit1_* This is the slit located on the BCU
ea_slit2_* This is the slit located on the PXRD2D instrument
ea_mm_* Is the location of the collimator/pinhole
ea_bst_* Is the location of the beamstop
hex_* Is the positions and angles of the hexapod
Spinner moving Is indicating wheter or not the spinner is spinning at the beginning of the scan.
IVU_R3_304_GAP Is the undulator gap
hdcm_energy Energy of the Si111 hDCM based on encoder position
mlm_energy Energy of the hMLM based on encoder position
Pilatus energy Is the energy setting on the Pilatus responsible for setting the flatfield of the detector
Pilatus threshold Is the threshold energy of the Pilatus responsible for energy discrimination