Versions Compared

Old Version 108

changes.mady.by.user Sam Tsoi

Saved on

compared with

New Version Current

changes.mady.by.user Sam Tsoi

Saved on

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.

...

Code Block
languagesql
themeRDark
create table `previousQueries` (`networkId` BIGINT NOT NULL,`userName` TEXT NOT NULL,`domain` TEXT NOT NULL,`queryName` TEXT NOT NULL,`expression` TEXT,`dateCreated` BIGINT NOT NULL,`deleted` BOOLEAN NOT NULL,`queryXml` TEXT NOT NULL,`changeDate` BIGINT NOT NULL);
create table `queryFlags` (`networkId` BIGINT NOT NULL,`flagged` BOOLEAN NOT NULL,`flagMessage` TEXT NOT NULL,`changeDate` BIGINT NOT NULL);
create table `queryResults` (`resultId` BIGINT NOT NULL,`networkQueryId` BIGINT NOT NULL,`instanceId` BIGINT NOT NULL,`adapterNode` TEXT NOT NULL,`resultType` TEXT,`size` BIGINT NOT NULL,`startDate` BIGINT,`endDate` BIGINT,`status` TEXT NOT NULL,`statusMessage` TEXT,`changeDate` BIGINT NOT NULL);
create table `queryBreakdownResults` (`networkQueryId` BIGINT NOT NULL,`adapterNode` TEXT NOT NULL,`resultId` BIGINT NOT NULL,`resultType` TEXT NOT NULL,`dataKey` TEXT NOT NULL,`value` BIGINT NOT NULL,`changeDate` BIGINT NOT NULL);
create table `queryResultProblemDigests` (`networkQueryId` BIGINT NOT NULL,`adapterNode` TEXT NOT NULL,`codec` TEXT NOT NULL,`stamp` TEXT NOT NULL,`summary` TEXT NOT NULL,`description` TEXT NOT NULL,`details` TEXT NOT NULL,`changeDate` BIGINT NOT NULL);

shrine.conf changes

.conf Files Consolidated

The dashboard.conf, steward.conf, and shrine.conf files have been consolidated into a single shrine.conf file. (The three files had several sections that were duplicated and had to be identical. This should make managing SHRINE's configuration simpler.) All services expect to find all the configuration values they need inside shrine.conf. steward.conf and dashboard.conf will be ignored.

Combine the shrine.conf, steward.conf, and dashboard.conf files into shrine.conf. 

Configuration Changes

See the canonical heavily annotated shrine.conf file in source code control.

Missing Default Properties

Add these section within the shrine block to properly log problems

Configure the SHRINE Data Steward

Database - i2b2

The SHRINE Data Steward is typically backed by the i2b2 PM cell used by SHRINE. From the steward application's point of view, all users on the SHRINE project are considered Researchers. However, there is some additional work that has to be done to the i2b2 user list to accommodate the SHRINE Data Steward.

QEP User

The Steward application requires set of user credentials that it will use to submit queries through to SHRINE. It is recommended that this be a dedicated user separate from any other account. Additionally, it will need to have the parameter "qep" defined (name: qep, value: true, type: text), which can be set in the Manage Users section of the i2b2 Admin Panel.

In shrine.conf, make sure there is a shrineSteward block in the queryEntryPoint section, and that the qepUserName and qepPassword properties match the user with the qep parameter.

Steward User

In Steward application deployments that require manual topic approval, a trusted user will have to be given permission to review proposed research topics and approve/reject them. To mark a user as such, add the "DataSteward" parameter (name: DataSteward, value: true, type: text) to that user in the Manage Users section of the i2b2 Admin Panel.

Changes to shrine.conf

Consolidation of .conf files

The dashboard.conf, steward.conf, and shrine.conf files have been consolidated into a single shrine.conf file. (The three files had several sections that were duplicated and had to be identical. This should make managing SHRINE's configuration simpler.) All services expect to find all the configuration values they need inside shrine.conf. steward.conf and dashboard.conf will be ignored.

Combine the shrine.conf, steward.conf, and dashboard.conf files into shrine.conf. 

Configuration Changes

See the canonical heavily annotated shrine.conf file in source code control.

Missing Default Properties

Add these section within the shrine block to properly log problems

Code Block
languagejs
themeRDark
  problem {
    problemHandler = "net.shrine.problem.LogAndDatabaseProblemHandler$"
  }
  dashboard {
    gruntWatch = false //false for production, true for mvn tomcat7:run . Allows the client javascript and html files to be loaded via gruntWatch .
    happyBaseUrl = "https://localhost:6443/shrine/rest/happy"
    statusBaseUrl = "https://localhost:6443/shrine/rest/internalstatus"

    database {
      dataSourceFrom = "JNDI" //Can be JNDI or testDataSource . Use testDataSource for tests, JNDI everywhere else
      jndiDataSourceName = "java:comp/env/jdbc/problemDB" //or leave out for tests
      slickProfileClassName = "slick.driver.MySQLDriver$" // Can be
Code Block
languagejs
themeRDark
  problem {
    problemHandler = "net.shrine.problem.LogAndDatabaseProblemHandler$"
  }
  dashboard {
    gruntWatch = false //false for production, true for mvn tomcat7:run . Allows the client javascript and html files to be loaded via gruntWatch .
    happyBaseUrl = "https://localhost:6443/shrine/rest/happy"
    statusBaseUrl = "https://localhost:6443/shrine/rest/internalstatus"

    database {
      dataSourceFrom = "JNDI" //Can be JNDI or testDataSource . Use testDataSource for tests, JNDI everywhere else
      jndiDataSourceName = "java:comp/env/jdbc/problemDB" //or leave out for tests
      slickProfileClassName = "slick.driver.MySQLDriver$" // Can be
      //        slick.driver.H2Driver$
      //        slick.driver.MySQLDriver$
      //        slick.driver.PostgresDriver$
      //        slick.driver.SQLServerDriver$
      //        slick.driver.JdbcDriver$
      //        freeslickslick.driver.OracleProfile$H2Driver$
      //        freeslickslick.driver.MSSQLServerProfile$MySQLDriver$
      //
      //   slick.driver.PostgresDriver$
     (Yes, with// the $ on the end)

   slick.driver.SQLServerDriver$
   createTablesOnStart = false //for testing with H2 in memory, when not running unit tests. Set to false normally
slick.driver.JdbcDriver$
      //        freeslick.OracleProfile$
      //      }  freeslick.MSSQLServerProfile$
  }

If you are using hub-and-spoke SHRINE architecture add this setting to shrine.queryEntryPoint

Code Block
languagejs
themeRDark
    //
      trustModelIsHub// = true

Node Meta Data Service

SHRINE networks now include a JSON api at every node that reports about the contents of the metaData section. What belongs in the metaData section is completely configurable, and we recommend that it be used as a means for serving relevant contact information. Bear in mind that this information is publicly available. For more information on interacting with the Node Data Service, please see the wiki page.

 

...

metaData {
  siteAdminsContactInfo = ["admin1@example.com", "admin2@example.com"]
  dataStewardContactInfo = "data.steward@example.com"
}

 

       (Yes, with the $ on the end)

      createTablesOnStart = false //for testing with H2 in memory, when not running unit tests. Set to false normally
    }
  }

If you are using hub-and-spoke SHRINE architecture add this setting to shrine.queryEntryPoint

Code Block
languagejs
themeRDark
    trustModelIsHub = true

	attachSigningCert = true

Node Meta Data Service

SHRINE networks now include a JSON api at every node that reports about the contents of the metaData section. What belongs in the metaData section is completely configurable, and we recommend that it be used as a means for serving relevant contact information. Bear in mind that this information is publicly available. For more information on interacting with the Node Data Service, please see the wiki page.

 

metaData {
  siteAdminsContactInfo = ["admin1@example.com", "admin2@example.com"]
  dataStewardContactInfo = "data.steward@example.com"
}

 

We strongly encourage networks to agree on key-value pairs for contacting remote SHRINE node system admins, data stewards, and data admins. This new service can share that information across the SHRINE network.

Configurable Obfuscation

Obfuscation parameters are now configurable. Results are rounded to the nearest 5. The default values force a nefarious researcher to run about 30 queries to identify an individual patient, and an additional 30 queries per fact they wish to verify. If you change these values, be sure to change the javascript property that controls the "+- 10" clamp value in SHRINE's web client to match. See Configuring SHRINE Webclient Obfuscation . 

Code Block
languagexml
themeRDark
shrine {
...
  adapter {
..

We strongly encourage networks to agree on key-value pairs for contacting remote SHRINE node system admins, data stewards, and data admins. This new service can share that information across the SHRINE network.

Configurable Obfuscation

Obfuscation parameters are now configurable. Results are rounded to the nearest 5. The default values force a nefarious researcher to run about 30 queries to identify an individual patient, and an additional 30 queries per fact they wish to verify. If you change these values, be sure to change the javascript property that controls the "+- 10" clamp value in SHRINE's web client to match. See Configuring SHRINE Webclient Obfuscation . 

Code Block
languagexml
themeRDark
shrine {
...
  adapter {
...
//    obfuscation {
//      binSize = 5 by default //Round to the nearest binSize. Use 1 for no effect (to match SHRINE 1.21 and earlier).
//      obfuscation {
//      binSize = 5 //by default. Round to the nearest binSize. Use 1 for no effect (to match SHRINE 1.21 and earlier).
//      sigma = 6.5 //by default. //Noise to inject. Use 0 for no effect. (Use 1.33 to match SHRINE 1.21 and earlier).
//      clamp = 10 //by default. //Maximum ammount of noise to inject. (Use 3 to match SHRINE 1.21 and earlier).
//    }
...

...

This article contains links to alternative database drivers, as well as links to alternative database scripts for creating tables for the SHRINE Data Steward. It also includes information on the necessary changes to make to shrine.xml, steward.xml, and shrine.conf.

shrine.conf includes several instances of slickProfileClassName. By default, these are configured to use MySQL. Users of Oracle or SQL Server for SHRINE must add them. See the below for an example:includes several instances of slickProfileClassName. By default, these are configured to use MySQL. Users of Oracle or SQL Server for SHRINE must add them. See the below for an example:

Code Block
languagexml
themeRDark
nopaneltrue
shrine {
[...]
  queryEntryPoint {
    audit {
      database {
        slickProfileClassName="freeslick.driverNameHere$"
      }
    }
    [...]
  }
[...]
 
  adapter {
    audit {
      database {
        slickProfileClassName="freeslick.driverNameHere$"
      }
    }
    [...]
  }
  dashboard {
    database {
      slickProfileClassName="freeslick.driverNameHere$"
    }
    [...]
  }

  steward {
    database {
      slickProfileClassName="freeslick.driverNameHere$"
    }
    [...]
  }
[...]
}

Replace "freeslick.driverNameHere$" with the name of the driver you are using ("freeslick.OracleProfile$" for Oracle, and "freeslick.MSSQLServerProfile$" for SQL Server).

Tomcat context.xml

To support shrine's improved error message feature, add a context.xml file at /opt/shrine/tomcat/conf/context.xml with contents appropriate for your database:

Code Block
languagexml
themeRDark
titlecontext.xml
<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->

<Context>
	<WatchedResource>WEB-INF/web.xml</WatchedResource>
    <Resource name="jdbc/problemDB" auth="Container" type="javax.sql.DataSource"
              maxActive="100" maxIdle="30" maxWait="10000"
              username="shrine" password="demouser" driverClassName="com.mysql.jdbc.Driver"
              url="jdbc:mysql://localhost:3306/shrine_query_history"
              testOnBorrow="true" validationQuery="SELECT 1"/>

    <Resource name="jdbc/shrineDB" auth="Container" type="javax.sql.DataSource"
               maxActive="100" maxIdle="30" maxWait="10000"
               username="shrine" password="demouser" driverClassName="com.mysql.jdbc.Driver"
               url="jdbc:mysql://localhost:3306/shrine_query_history"
               testOnBorrow="true" validationQuery="SELECT 1"/>

    <Resource name="jdbc/adapterAuditDB" auth="Container" type="javax.sql.DataSource
Code Block
languagexml
themeRDark
nopaneltrue
shrine {
[...]
  queryEntryPoint {
    audit {
      database {
        slickProfileClassName="freeslick.driverNameHere$"
      }
    }
    [...]
  }
[...]
 
  adapter {
    audit {maxActive="100" maxIdle="30" maxWait="10000"
      database {
        slickProfileClassName="freeslick.driverNameHere$username="shrine" password="demouser" driverClassName="com.mysql.jdbc.Driver"
      }
    }
    [...]url="jdbc:mysql://localhost:3306/adapterAuditDB"
  }
  dashboard {
    database {
      slickProfileClassName="freeslick.driverNameHere$"
testOnBorrow="true" validationQuery="SELECT 1"/>

     }
    [...]
  }

  steward {
<Resource name="jdbc/qepAuditDB" auth="Container" type="javax.sql.DataSource"
         database {
    maxActive="100" maxIdle="30" slickProfileClassNamemaxWait="freeslick.driverNameHere$10000"
    }
        [...]
  }
[...]
}

Replace "freeslick.driverNameHere$" with the name of the driver you are using ("freeslick.OracleProfile$" for Oracle, and "freeslick.MSSQLServerProfile$" for SQL Server).

Tomcat context.xml

To support shrine's improved error message feature, add a context.xml file at /opt/shrine/tomcat/conf/context.xml with contents appropriate for your database:

Code Block
languagexml
themeRDark
titlecontext.xml
<?xml version='1.0' encoding='utf-8'?>
<!-- The contents of this file will be loaded for each web application -->

<Context>
    <!-- Default set of monitored resources -->
    <WatchedResource>WEB-INF/web.xml</WatchedResource>  username="shrine" password="demouser" driverClassName="com.mysql.jdbc.Driver"
              url="jdbc:mysql://localhost:3306/qepAuditDB"
              testOnBorrow="true" validationQuery="SELECT 1"/>

    <Resource name="jdbc/problemDBstewardDB" auth="Container" type="javax.sql.DataSource"
              maxActive="100" maxIdle="30" maxWait="10000"
              username="yourDbUserNameshrine" password="yourDbPassworddemouser" driverClassName="com.mysql.jdbc.Driver"
              url="jdbc:mysql://yourMysqlDbHostlocalhost:3306/shrine_query_historystewardDB" 
              testOnBorrow="true" validationQuery="SELECT 1"/>
</Context>

...