...

Next, configure the initial network structures and queues for the hub.

Download the shrine network lifecycle tool from https://repo.open.catalyst.harvard.edu/nexus/content/groups/public/net/shrine/shrine-network-lifecycle-tool/4.1.0/shrine-network-lifecycle-tool-4.1.0-dist.zip into /opt/shrine:

cd /opt/shrine


wget https://repo.open.catalyst.harvard.edu/nexus/content/groups/public/net/shrine/shrine-network-lifecycle-tool/4.0.0/shrine-network-lifecycle-tool-4.0.0-dist.zip -O shrine-network-lifecycle.zip unzip shrine-network-lifecycle.zip

cd shrine-network-lifecycle

...

shrine {
  queryEntryPoint{
    audit {
      database {
        dataSourceFrom = "testDataSource" //Can be JNDI or testDataSource . Use testDataSource for tests and command line tools, JNDI everywhere else
        testDataSource {
          driverClassName = "com.mysql.cj.jdbc.Driver" //JDBC driver class name
          url = "jdbc:mysql://localhost:3306/qepAuditDB?serverTimezone=UTC" //URL to your database
          credentials {
            username = "yourUserName"
            password = "yourDatabasePassword"

TODO configure for MOM systems here or in subchapters.

          }
       }
     }
  }
}

Next, create a file named network.conf to meet your needs. At a minimum include the network section and a section for the hub's QEP:

shrine {
  network {
    network {
      name = "Network Name"
      hubQueueName = "hub"
      adminEmail = "yourEmail@yourhospital.edu"
      momId = "HubQueue"
    }
    nodes = [
      {
        name = "Hub's QEP"
        key = "hub-qep"
        userDomainName = "network-hub"
        queueName = "shrinehub"
        sendQueries = "false"
        adminEmail = "yourEmail@yourhospital.edu"
        momId = "HubQepQueue"
      }
    ]
  }
}

This will use in-tomcat messaging. See below to use AWS SQS or Kafka message-oriented middleware.

In-Tomcat or External Messaging Systems

SHRINE uses a message-oriented-middleware system to process queries and results asynchronously. Researchers can watch progress of their queries at different nodes, and see results as soon as any results are available. 

Starting in SHRINE 4.1 the hub admin can select from SHRINE's own in-tomcat message system, AWS SQS, or Kafka. It is possible to switch from one MOM system to another in shrine 4.1, but the process is not gentle: Stop all nodes on the network. Configure all nodes for the new messaging system. Restart all nodes in the network.

In-Tomcat

For most shrine networks we recommend using shrine's in-tomcat messaging system: Networks that want to minimize administration effort, small networks, isolated networks, networks set up for demonstrations, and networks with few researchers. The main drawback to shrine's in-tomcat messaging is that messages in-flight and in-process are stored in memory; if a message is in transit when the hub admin stops tomcat it will be lost. (The 45-node ACT network has used this messaging system since SHRINE 1.25 with few incidents.)

There is no additional configuration or set-up to use this messaging system.

AWS SQS

For large networks that desire telecom-like reliability we use AWS SQS. AWS SQS stores messages that are in-flight or in-process, not the hub's tomcat. Messages stored in AWS SQS remain available if the hub stops. The hub processes those messages after it recovers. SHRINE 4.1 supports exactly one hub tomcat server; a shrine network can be no more reliable than the hub's tomcat. We hope to make the hub replicable in the future. Moving to AWS SQS now will make that transition more gentle.

To configure for AWS SQS see SHRINE 4.1.0 Chapter 8.2.1 - Configuring a Hub for AWS SQS .

Kafka

Shrine can also be configured to use Kafka. Using Kafka requires setting up, securing, and maintaining a cluster of Kafka servers - more admin expertise, attention, and expense than AWS SQS without the benefits of AWS SQS' reliability. 

To configure for Kafka see SHRINE 4.1.0 Chapter 8.2.2 - Configuring a Hub for Kafka .

Create the Network Record and Queues 

Finally use the shrineLifecycle tool to set up the network:

...