Metadata-Version: 2.1
Name: sdm
Version: 1.8.1
Summary: A collection of API to manage the Scientific Data from the User Office to the Central storage
Home-page: https://gitlab.maxiv.lu.se/kits-maxiv/lib-maxiv-sdm
Author: KITS
Author-email: kits@maxiv.lu.se
License: GPLv3
Platform: UNKNOWN
License-File: LICENSE

# SDM library

This library is composed by several python modules:
  - sdm: Library to manage the AD operations.
  - duo: Library to handle the communications with the Digital User Office.

# DUO Communication Library

## Introduction

This is a library developed at MAXIV Laboratory used for the communications with
the Digital User Office (a.k.a DUO).

This communication use a REST API.

## Monitor Data Path Creation

This library also manages the creation of the path where acquisition data will be stored.

Extracted from [here](https://alfresco.maxiv.lu.se/share/page/site/datamanagement/wiki-page?title=Beamline_Prototype&listViewLinkBack=true) we can read:
```
Data path or central storage

The data path for each data collection should have a structure meeting the following requirements:

    Splitting data by user types.
    Grouping by date, proposal and beamline
    Make the mounting of data folders for a specific user not too complicated

The proposed structure is:

/data/<user type>/<proposal>/<beamline>/<visit>/raw

The <user type> includes visitor (bread and butter academic users), staff (MAX IV staff) and proprietary (mostly industrial users with a different need for data security).

The <visit> should begin with the date when the first shift starts.

In addition to /raw there will be a /process folder at the same level. This is where the user has permission to read and write files.
```

## Test Coverage
To get the test coverage you need to have installed: coverage, pytest, pytest-cov, and if you want distributed
testing, pytest-xdist.

```
pytest --cov=duo tests/
```
Resulting in the following report:

```
	Name              Stmts   Miss  Cover
	-------------------------------------
	duo/UO.py           208     19    91%
	duo/__init__.py       5      0   100%
	duo/commons.py        8      0   100%
	-------------------------------------
	TOTAL               221     19    91%
```

and the same for sdm, i.e. sxchange duo for sdm in the coverage test call

```
	Name               Stmts   Miss  Cover
	--------------------------------------
	sdm/SDM.py           485    134    72%
	sdm/__init__.py        7      0   100%
	sdm/__main__.py        2      2     0%
	sdm/cli.py            52     48     8%
	sdm/config.py         16      0   100%
	sdm/directory.py     434     66    85%
	sdm/storage.py        90     69    23%
	sdm/user.py          142     66    54%
	sdm/visitor.py       189    146    23%
	--------------------------------------
	TOTAL               1417    531    63%
```

# Active Directory Communication Library

## Introduction
The sdm/directory.py module handles the communication with the Active Directory through LDAPS protocol. In order to login to the Active Directory you need to have a file in your home directory which give the user and password. The file should located in ~/.sdm/activedirectory.yml and should contains:

```shell
cat ~/.sdm/activedirectory.yml

LDAPUSER: user@mydomain.se
LDAPPASSWORD: my-secret-password
TESTLDAPUSER: test@test.se
TESTLDAPPASSWORD: my-secret-test
```

# NOTE on merging
When merging to master all tests in the tests/ will be run. If they don't pass the merge will not be carried out and the coder will need to fix the code.


# User and Proposal Report
The SDM library provides a command line interface for reporting the status of users and proposals.

### Report a user
Suply a comma separated list of users user (or a single one), and make sure the production flag is present.
```
python sdm -r --production --user mikegu,carlcarl
```

The output would be:

```
+----------------+
Report from DUO (Digital User Office)
+----------------+----------+----------+----------+------------+---------------------+---------------------+
|   User name    | account  | Proposal | Beamline | Session ID |      Start Date     |       End Date      |
+----------------+----------+----------+----------+------------+---------------------+---------------------+
| Mikel Eguiraun |  mikegu  | 20170251 |  BioMAX  |     3      | 2018-04-27 00:00:00 | 2018-04-27 04:00:00 |
|                |          | 20170251 |  BioMAX  |     3      | 2018-04-27 04:00:00 | 2018-04-27 08:00:00 |
|                |          | 20200318 |  CoSAXS  |    900     | 2020-10-21 16:00:00 | 2020-10-21 20:00:00 |
|                |          | 20200318 |  CoSAXS  |    900     | 2020-10-22 16:00:00 | 2020-10-22 20:00:00 |
|                |          | 20200318 |  CoSAXS  |    900     | 2020-10-23 16:00:00 | 2020-10-23 20:00:00 |
|                |          | 20200318 |  CoSAXS  |    900     | 2020-10-25 08:00:00 | 2020-10-25 12:00:00 |
|                |          | 20200318 |  CoSAXS  |    991     | 2020-12-07 04:00:00 | 2020-12-07 08:00:00 |
|                |          | 20180163 | Veritas  |     74     | 2018-05-16 08:00:00 | 2018-05-16 12:00:00 |
|                |          | 20210589 |  DanMAX  |    1048    | 2021-03-23 08:00:00 | 2021-03-23 12:00:00 |
| Carla Carlsson | carlcarl | 20190586 |  BioMAX  |    305     | 2019-05-27 08:00:00 | 2019-05-27 12:00:00 |
|                |          | 20190586 |  BioMAX  |    336     | 2019-06-24 08:00:00 | 2019-06-24 12:00:00 |
|                |          | 20190586 |  BioMAX  |    336     | 2019-06-24 12:00:00 | 2019-06-24 16:00:00 |
|                |          | 20190586 |  BioMAX  |    378     | 2019-09-17 08:00:00 | 2019-09-17 12:00:00 |
|                |          | 20170251 |  BioMAX  |     1      | 2017-10-24 00:00:00 | 2017-10-24 04:00:00 |
|                |          | 20170251 |  BioMAX  |     1      | 2017-10-25 00:00:00 | 2017-10-25 04:00:00 |
|                |          | 20170251 |  BioMAX  |     1      | 2017-10-28 00:00:00 | 2017-10-28 04:00:00 |
|                |          | 20170251 |  BioMAX  |     1      | 2017-11-25 00:00:00 | 2017-11-25 04:00:00 |
+----------------+----------+----------+----------+------------+---------------------+---------------------+
Report from AD (Active Directory)
+----------+--------+---------------+------+---------------------------------------+------------------------------+
| account  |  uid   | Primary Group | gid  |               Member of               | VPN granted (only for users) |
+----------+--------+---------------+------+---------------------------------------+------------------------------+
|  mikegu  |  1414  |    MAX-Lab    | 1300 | vmware control systems administrators |                              |
|          |        |               |      |               vpn-white               |                              |
|          |        |               |      |               vpn-staff               |                              |
|          |        |               |      |               vpn-green               |                              |
|          |        |               |      |                vpn-blue               |                              |
|          |        |               |      |            vpn-blue-biomax            |                              |
|          |        |               |      |            acc-biomax-staff           |                              |
|          |        |               |      |                 Staff                 |                              |
|          |        |               |      |                  KITS                 |                              |
|          |        |               |      |                 biomax                |                              |
|          |        |               |      |              Splunk-User              |                              |
|          |        |               |      |                KITS-SW                |                              |
|          |        |               |      |             20180163-group            |                              |
|          |        |               |      |             20170257-group            |                              |
|          |        |               |      |             20190588-group            |                              |
|          |        |               |      |             20170251-group            |                              |
|          |        |               |      |             20200318-group            |                              |
|          |        |               |      |             20210589-group            |                              |
|          |        |               |      |             20211218-group            |                              |
|          |        |               |      |             20211219-group            |                              |
| carlcarl | 400462 |    Visitors   | 1332 |                Visitors               |              na              |
|          |        |               |      |             20170257-group            |              --              |
|          |        |               |      |             20170254-group            |  vpn-blue-biomax-remote-usr  |
|          |        |               |      |             20170251-group            |  vpn-blue-biomax-remote-usr  |
|          |        |               |      |             20190586-group            |              --              |
+----------+--------+---------------+------+---------------------------------------+------------------------------+
```

### Report a Proposal


For the case of reporting a proposal:

```
python sdm -r --production --proposal 20170251
```

In this case the output would be:

```
+----------+----------+----------+--------------+----------+
| Proposal | Beamline |   Type   |    Users     | Sessions |
+----------+----------+----------+--------------+----------+
| 20170251 |  BioMAX  | in-house |   dapvan1    |    1     |
|          |          |          |    vinhar    |    3     |
|          |          |          |    maglar    |   779    |
|          |          |          | rocconumber1 |          |
|          |          |          |    elmjag    |          |
|          |          |          |   sdmuser2   |          |
|          |          |          |    abdamj    |          |
|          |          |          |    jienan    |          |
|          |          |          |    thours    |          |
|          |          |          |    aleceh    |          |
|          |          |          |    isalin    |          |
|          |          |          |    jonsch    |          |
+----------+----------+----------+--------------+----------+
+--------------------------------+--------------------+----------------+----------------+-----------------------+
|              Path              |    Description     |     Owner      |     Group      |         Status        |
+--------------------------------+--------------------+----------------+----------------+-----------------------+
|     /data/visitors/biomax      | Beamline data path | biomax-service |     biomax     | Beamline path correct |
| /data/visitors/biomax/20170251 | Proposal data path | biomax-service | 20170251-group |     Group correct     |
|                                |                    |                |                |      User correct     |
|                                |                    |                |                |                       |
+--------------------------------+--------------------+----------------+----------------+-----------------------+
Proposal group 20170251-group is part of beamline group BioMAX. All good!
```

# SDM Synchronization scripts

## Sync duoactive
Runs daily, nowadays as a fallback to DUO on demand calls. It does the following:
		
       for every accepted proposal:
       	* check and create proposal group in AD
       	* for all the proposal users in DUO:
       	    * Sync with the AD proposal group members (DUO is the truth)
       	    * Move the user to DUOACTIVE if not there alredy
       	    * If it was moved (or force home flag)
                    * only for visitor: add to Visitors AD group, add extra_attributes in AD, enable, create_home
                    * if staff: nothing
              * Add the user to the proposal group
	
	And then:

        for every accepted proposal:
            * add the beamline group to the proposal group
            * associate extra resources to the proposal group (e.g. presto)

Runs at 20:30 and 21:30 everyday. 

We run once at the end of day to ensure next day's user access rigths. It runs twice because there could be a delay between what the script does about the user operations in Active Directory and the following home folder creation (it needs the user account information propagated into all MAX IV AD Clients). We run twice so that if some user's home folder was missed in the first run, the second one will correct it.

These issues are nowadays rare. Although we still see occasional failures due to old un-merged DUO accounts, of mixed staff-visitors people. In those cases manual intervention is needed, the script most likely will not fix.

## Sync proposal force home

Same as sync but only for the supplied proposal, and forces the addition of the visitors to the Visitor AD group and creates their home folder (these have no effect if already done).

This job is triggered automatically from DUO when the users do changes on the session schedule and participants. Also, if the `sync` run fails on some users, the sync proposal force will fix, tyically run as part of some support intervention.

Runs on demand, via gitlab api calls on triggering the job [here](https://gitlab.maxiv.lu.se/kits-maxiv/lib-maxiv-sdm/-/pipeline_schedules). Remember to edit the job and change the proposal.

## Sync offsite

This is the job that adds proposal groups to the remote VPN access group. It only applies to BioMAX but it can be easily extended to others.

It checks if the proposal has a session with a shift in the coming hours (from when the script runs). If so, it will add this proposal group to the vpn access group.

If the proposal has no valid shift, it ensures it is not part of the vpn.

Runs at 0:00, 4:00, 8:00, 12:00, 16:00, 20:00 everyday
	


