Page History

Applications with an Endorsement Cycle

Qualtrics documentation is also helpful.

Creating a New Unique OC

(See also: Endorsements - Copying From a Previous OC)

1. Create two surveys, the application form and the endorser form. Title them accordingly, eg, ECR Application, ECR Endorsement. 

You may consider initially only including the survey questions that are displayed on the endorsement survey or in emails. This will allow you to submit the forms quickly to test the workflow without having to complete the entire survey. Once the survey is functioning properly, you can add the rest of the survey questions.

Applicant survey

Add all questions, logic, etc. (be sure to include questions for Endorser Name and Email. Leave Endorser first and last name in one field if you don't need to split them up.)

Click Survey Options > select Open Access (review all settings) 

Click Survey Flow

Click Add a New Element Here

Click Embedded Data

Type "LoginID" (be careful; case matters)

Click Set a Value Now

From the dropdown, select Insert Piped Text > Random Number > Integer 100000...999999

This method does run the remote possibility of duplication.

Drag this new block above the Question Block (using Move)

Click Save Flow

Add a Contact List Trigger

You will be using this contact list as a database to store name/value pairs that you can pull into the endorser survey

Tools > Triggers > Contact List Trigger

Click "Please Select"

Select My Library:...

Click New Contact List unless you're working from an existing OC

Type: endorseList

Click Save (not Save Triggers, which closes the pop-up)

Under External Data Reference select Embedded Data and enter "LoginID"

This is the key field that will be used to ensure that the endorser is accessing the correct record. Be sure the case matches everywhere.

Click "Add additional fields"

Enter each field from the application survey that you want to display on the endorsement survey (Click + to add more fields) (first name, last name?)

Set the value of each field with the "Select Question" and "Select Choice" drop downs

Click SaveTriggers

Note: to edit/view existing trigger details, click the name of the trigger, eg, "Add respondent to the panel..."

Add email action for endorser (you can also add an email confirmation to the applicant)

Click Actions > Add Action

Event: Survey Response

Conditions: N/A

Tasks: Send Email

To: from the dropdown, select Survey Question > Endorser Email > Endorser Email

You can copy your departmental email address on messages. Separate 'To' emails with a comma. Each recipient will only see their own email address, so it acts as a BCC.

Add a From Address, From Name, and Reply-To Address

In the Subject field

Type "Endorsement request from "

From the {a} dropdown select "Survey Question" > [first_name]

Type a space

From the {a} dropdown select "Survey Question" > [last_name]

Add instructions to the Message field

If you would like to collect the endorser's response using email you are done with the Applicant survey

OR

If you would like to collect the endorser's response using a Qualtrics survey you will need to come back to this email once the endorser survey is done.

Add email action for applicant (use piped text {a} to customize your To, Subject, and Message fields)

should we add a separate section about customizing email messages?

Endorser survey (optional if you accept endorsements via email)

Click "Create Project"

Create questions and add all questions, logic, etc. 

Add a question of type Text/Graphic to display applicant data

Add instructions to endorser

Add a placeholder for each field that you want to display (Name, project title etc.)

Click Survey Flow

Click Add a New Element Here and select Authenticator

Authentication Type: Contact

Authenticate Using Contact:  endorseList

Authentication Fields: External Data Reference; Check Pre-fill; Type LoginID

Move the Authenticator Block above Question Block

Click Add a New Data Element Here and click Embedded Data

Type LoginID

Move above all other blocks

Click Add a New Data Element Here and click Embedded Data

Add each element that you want to reference from the Applicant survey. 

Type the field name, i.e. Applicant First Name (Make sure the field names match)

Move this block below Authenticator and above the question block

Click Save Flow

Add Applicant data to the placeholder questions

Click the question text to edit

Select Piped Text > Embedded Data Field > from dropdown, select Applicant First Name, Applicant Last Name, etc.

Customize the Endorser email

Copy the Anonymous Link of the Endorser Survey

Click Distributions

Click Get Reusable Link (if visible)

Click "Anonymous Link"

Copy the Anonymous Survey Link

Paste in the Anonymous Link of the Endorser Survey into the initial Endorser email (within the application survey)

Append with ?LoginID=${e://Field/LoginID}

Be sure to take a look at the Survey Options to make sure it is set up properly

You are ready to test. Fill out the applicant survey and include your own email as the endorser. Once you receive the endorsement request email, complete that survey

Qualtrics workflow - Application cycles with Endorsements

...

Sharable documentation on Google Drive (no login necessary). Bill and Genoa are owners. Anyone with the link can view. Link: 

Outstanding Questions

• Long-term, will Informatics have a support role for Qualtrics? We can create a how-to document and share that, and HMS IT offers Qualtrics support (Tamara Will).
• As we roll this out to program admins (PAs), we should be establishing best practices, like having all content on one screen, having minimal intro text, having OC name at top, testing the surveys, etc.
• We should have a sample course survey and a sample pilot grant survey.
• Figure out what is reported in Salesforce and make Qualtrics questions match (eg institutions, degrees, etc).
• For non-PGE folks, create previous OCs in Qualtrics and give them that example. That way they start out building their next forms and we're only here for support.

• "This application form cannot be saved. Please prepare your responses and save elsewhere before starting this form." Qualtrics allows users to save and come back. How? Do they need credentials?
• Make sure we test time-out of forms. As a regular not-logged-in user, the timeout is beyond 2 hours 15 minutes. 

• Survey Options > Survey Expiration

• Co-Investigators: We should look at Apply Hub to see the highest number of Co-I's. We have to create the questions per co-i, so co-i 7 phone number, co-i 7 department, etc. Where should we cap the number? 
• Talk with the PAs about what's actually required (eg, online course Qualtrics form doesn't have final checkbox)
• Lisa Riva sent Bill and me her best practices document, which has lots of helpful hints we should incorporate.
• Change navigational buttons from arrows to "Submit": Look and Feel -> Next Button Text - type "Submit." This makes it clear that there are no additional pages past this one.
• AppID/LoginID is random and non-sequential. This is by design to hinder access to others' applications.
• Link the endorsement and/or review to the application by including LoginID in the endorsement email, endorsement form, applicant email, reviewer email, and reviewer form.

Sharing Library messages etc.

As a Brand Administrator, you can use Groups to allow two or more users to share projects, graphics, files, and messages in Libraries. Users within Groups can also share contact lists in their Contacts page. Groups can include everyone in your organization, everyone in a specific Division, or only specific users that you manually select. Groups administration can only be accessed by Brand and Division Administrators in the Admin page.

KL2

KL2: In 2019, Linda Bard has three reviewers fill out the review form. Then she wants one of those reviewers to fill out the interviewer form. Apply Hub won't let you assign the same person to multiple reviews, but you can reopen the individual review form, and on the assign page, change the reviewer's review type from reviewer to interviewer. The interview form contains all the fields as the review form, plus additional questions. By changing the reviewer type from reviewer to interviewer, the answers to the review questions carry over to the interviewer form. I'm not sure how Apply Hub is doing this. I copied most of the yml forms over from last year, but Linda said that last year the interviewers were interviewers from the beginning; they didn't first fill out the review forms. Sophia Cheng created the forms last year; maybe she magically met this future need. Anyway, we should figure out how to replicate this in Qualtrics.

Components for OC with endorse phase, using Qualtrics for endorsement form

1. Application form
2. Endorsement form
3. Application confirmation email
4. Endorser assignment email
5. Endorser confirmation email?
6. Triggers...

Accessing Qualtrics

Log into HMS eCommons, https://ecommons.med.harvard.edu/default.asp, click on Applications in upper left, click on Qualtrics Survey Tool from list.

How to's

Create Project

Blank Survey Project > Enter Project Name > Click Create Project

Email Address

Content Validation > Content Type: Email Address

Phone Number

Content Validation > Content Type: US Phone (Does NOT allow for extensions.) 

Institution List

Where to maintain list of institutions that is kept up to date and centrally controlled? Possibly share a "master" survey from which questions could be copied. The problem would be getting people to actually do it.

T Domain question (or any question where one selection clears the others)

Select "Make Answer Exclusive" (next to Undecided, in dropdown) to clear all other selections
 - can't put OR before Undecided

Adding Co-Investigators

This should be created as a "starter survey" and saved to the Library. This is the only way to retain the display logic. Other questions can then be imported as needed.

• Create the "How many Co-Investigators are you working with?" question
  • Multiple Choice
  • 5 choices (1 more than max desired)
  • Single Answer
  • Column, same as number of choices
  • Force Response
  • Enter answer values as 0,1,2,3,4
• Create the Co-Investigator 1 questions
  • Be sure to prepend each question with "Co-Investigator 1", so "Co-Investigator 1 First Name"
  • Add Display Logic to every question as follows

  • If How many Co-Investigators are you working with? 1 Is Selected
    Or How many Co-Investigators are you working with? 2 Is Selected
    Or How many Co-Investigators are you working with? 3 Is Selected
    Or How many Co-Investigators are you working with? 4 Is Selected

• Repeat for each Co-Investigator

  • Display Logic should start with Co-I number, so 2 is selected, or 3, or 4.

Agree to program requirements question

• Multiple Choice, Choices = 1, Multiple Answers (allows unchecking the box), Force Response (prevents form submission without agreement)
• Question: "To apply you must agree to the program requirements"
• Answer: "By checking this box, I certify that I understand the program requirements and agree to abide by them if I am accepted into this course."
  • Use Rich Content Editor to add program requirements (RFA) link. Change target to "_blank" so it opens in a new window.

Survey Options

Be sure to review all of the survey's options before testing. Unless you are sending the survey to a specific list of people, your survey should be set to "Open Access." Other options include:

• Choosing custom messaging
• End of survey redirect
• Password protection
• Prevent users from skipping landing page (HTTP Referrer Validation)
• Set dates that survey is open

...

Required Fields

Qualtrics doesn't automatically add asterisks to required fields.

• If you want to indicate required fields, add the asterisk manually to each question and include "* = Required Field" near the top of the survey.
• To require fields, select "Force Response" in the question settings in the right column.
• You can select multiple questions at once using the checkbox at the top left of each question then select "Force Response" in the right column

...

Requesting or Requiring a response for "other"

Here's how to collect a text response when someone chooses "other" from a Multiple Choice question.

• Question Type: Multiple Choice
• Answers: Single Answer OR Multiple Answer
• Click "Click to write choice..." (if you are editing an existing answer, click on the answer text)
• Type "Other"
• Click the arrow to expand the dropdown
• Click "Allow Text Entry"
• To require this field...
  
  • Click the arrow to expand the dropdown
  • Click Text Entry Validation > Force Response
• If the answers are in multiple columns and there are an uneven number of answers, don't put Other as last option. If you do it will push down other options.

...

Section Headings

Add a question as Descriptive text, then use Rich Content Editor to make it bold.

...

Help/Subheading Text

To add help text or a subheading under a question;

• Click on the question
• Click Rich Content Editor
  
  • Add a line return after the question
  • Type your help text
  • Highlight/select the help text
  • Select a smaller font size (i.e. 10px)
  • Click the italics icon  

Keeping the Survey to One Page

It is recommended that you keep your surveys to one page whenever possible. This will allow people to see all of the questions in order to prepare their responses, and it will help to reduce survey fatigue.

• Each block is on a different page. Avoid blocks to keep all on one page.
• Conditional questions are validated via the next arrow button using "display logic." This can be avoided for "If Other, Please Specify": Multiple Choice > Single or Multiple Answer > Add Other and on dropdown next to text box, select Allow Text Entry.
• If keeping the survey on one page isn't possible, add this language to the first page: "Please read the RFA (link) to ensure you have the required materials before beginning this form." Also add text to last page to tell user it's the last page. Qualtrics arrows are all the same; there's no indication when it's the last arrow and you're going to submit the form. You could also say "Page 1 of 4," "Page 2 of 4," etc. You can edit the arrows to add text, but each page's button will have the same text.
• To prevent multi-page, we display all institutions as a picklist as opposed to a dropdown list. Qualtrics does not allow in-page display logic based on drop-down list (for Other Institution).

...

Updating Published Surveys

Changes will only be presented to respondents after you click the Publish button. Then, only respondents who didn’t start the survey until after you published will see new changes you’ve made to the survey.

If you want respondents who started before you published to experience the changes, then they will need to start their survey over.

• If you distributed the survey through the Anonymous Link, ask them to clear their browser cache, open an incognito window, or try to take the survey in a different browser.
• If you distributed the survey through email or personal links, send them a new Individual Link.

Creating a hidden question that can be used to document status/endorsed

• Add a text entry question
  • For question text type "Is Endorsed?" or "Endorsed by Primary Endorser?"
  • Question Type: Multiple Choice
  • Choices: 2
  • Automatic Choices: Yes/No
  • Answers: Single Answer
  • Actions: Add Display Logic
    • Choose any existing question that is always displayed and select the condition "is not displayed"
      This will ensure that the "Is Endorsed?" question is never displayed
• Marking applications as endorsed
  As you receive endorsements (or in a batch) you can use this field to mark each application as endorsed
  • Open the project
  • Click Data & Analysis
  • If you do not see the "Is Endorsed" question
    • Click Tools > Choose Columns
    • Click the "Is Endorsed" question
    • Click Tools to close the menu
  • To Edit the "Is Endorsed" question
    • Click "Edit"
    • Click the + icon
    • Select yes or no
    • Click "Edit" when done

Authentication

The HTTP Referer Verification option under Survey Options/Survey Protection prevents people from taking the survey unless they come from a specific URL. We should investigate how this might be used to provide authentication.

Qualtrics supports authentication via LDAP (and others). See https://www.qualtrics.com/support/survey-platform/survey-module/survey-flow/advanced-elements/authenticator/sso-authenticator/

Security

Select - Prevent Ballot Box Stuffing

Select - Prevent Indexing

Select - Secure Participant's Files

Select - Survey Expiration > enter a date

End of Survey

We can create a custom end of survey message

We can also redirect to a specific URL

File Uploads

• Can be required.
• Can limit to PDF file type.
• Can include multiple file types.
• Recommend limiting file size. Apply Hub's is 10mb.

Harvard Authentication

Since the application forms are no longer behind login, program admins can ensure applicants are at Harvard by adding a required field for eCommons username or Harvard email.

Endorsement Cycle:

In Apply Hub:

On the application form, applicant specifies endorser(s)' information (eg name, institution, and email address). Upon submit of the application form, endorsers are sent an email with a tokenized URL (no login required for endorsers) and they complete the endorsement form. The application is then marked Endorsed.

There can be multiple endorser types, with different endorser forms per endorser type.

As the endorser deadline approaches, applicants have the ability to send reminders to their endorsers, based on their application status (not endorsed) in Apply Hub.
    - This will need to be done manually via email. They have no way of knowing if it was done anyway (no dashboard).

There are two options for the Endorsement cycle: 1) application form triggers email to endorser, endorser replies to email to endorse (reply-to email would be departmental email address). PA can mark applications as endorsed in Qualtrics as responses come in. Or 2) application form triggers email to endorser with link to endorsement form. PA can export application and endorsement data to Excel to match the endorsements with the correct, associated applications.

Endorsements in Qualtrics

Qualtrics documentation is also helpful.

Creating a New Unique OC

(See also: Endorsements - Copying From a Previous OC)

...

Create two surveys, one for applicant questions and one for endorser question(s). Title them accordingly, eg, ECR Application, ECR Endorsement. 

...

You may consider initially only including the survey questions that are displayed on the endorsement survey or in emails. This will allow you to submit the forms quickly to test the workflow without having to complete the entire survey. Once the survey is functioning properly, you can add the rest of the survey questions.

Applicant survey

...

Add all questions, logic, etc. (be sure to include questions for Endorser Name and Email. Leave Endorser first and last name in one field if you don't need to split them up.)

...

Click Survey Options > select Open Access (review all settings) 

...

Click Survey Flow

• Click Add a New Element Here

• Click Embedded Data

• Type "LoginID" (be careful; case matters)

• Click Set a Value Now

• From the dropdown, select Insert Piped Text > Random Number > Integer 100000...999999
  This method does run the remote possibility of duplication.

• Drag this new block above the Question Block (using Move)

• Click Save Flow

...

Add a Contact List Trigger

You will be using this contact list as a database to store name/value pairs that you can pull into the endorser survey

• Tools > Triggers > Contact List Trigger

• Click "Please Select"

• Select My Library:...

• Click New Contact List unless you're working from an existing OC

• Type: endorseList

• Click Save (not Save Triggers, which closes the pop-up)

• Under External Data Reference select Embedded Data and enter "LoginID"

  This is the key field that will be used to ensure that the endorser is accessing the correct record. Be sure the case matches everywhere.

• Click "Add additional fields"

• Enter each field from the application survey that you want to display on the endorsement survey (Click + to add more fields) (first name, last name?)

  • Set the value of each field with the "Select Question" and "Select Choice" drop downs

• Click SaveTriggers

• Note: to edit/view existing trigger details, click the name of the trigger, eg, "Add respondent to the panel..."

...

Add email action for endorser (you can also add an email confirmation to the applicant)

• Click Actions > Add Action

• Event: Survey Response

• Conditions: N/A

• Tasks: Send Email

  • To: from the dropdown, select Survey Question > Endorser Email > Endorser Email

    • You can copy your departmental email address on messages. Separate 'To' emails with a comma. Each recipient will only see their own email address, so it acts as a BCC.

  • Add a From Address, From Name, and Reply-To Address

  • In the Subject field

    • Type "Endorsement request from "

    • From the {a} dropdown select "Survey Question" > [first_name]

    • Type a space

    • From the {a} dropdown select "Survey Question" > [last_name]
      

  • Add instructions to the Message field

  • If you would like to collect the endorser's response using email you are done with the Applicant survey
    OR

  • If you would like to collect the endorser's response using a Qualtrics survey you will need to come back to this email once the endorser survey is done.

...

• Click "Create Project"
• Create questions and add all questions, logic, etc. 
  • Add a question of type Text/Graphic to display applicant data
    • Add instructions to endorser
    • Add a placeholder for each field that you want to display (Name, project title etc.)
• Click Survey Flow
  • Click Add a New Element Here and select Authenticator
    • Authentication Type: Contact
    • Authenticate Using Contact:  endorseList
    • Authentication Fields: External Data Reference; Check Pre-fill; Type LoginID
    • Move the Authenticator Block above Question Block
  • Click Add a New Data Element Here and click Embedded Data
    • Type LoginID
    • Move above all other blocks
  • Click Add a New Data Element Here and click Embedded Data
    • Add each element that you want to reference from the Applicant survey. 
    • Type the field name, i.e. Applicant First Name (Make sure the field names match)
    • Move this block below Authenticator and above the question block
  • Click Save Flow
• Add Applicant data to the placeholder questions
  • Click the question text to edit
  • Select Piped Text > Embedded Data Field > from dropdown, select Applicant First Name, Applicant Last Name, etc.
• Customize the Endorser email
  • Copy the Anonymous Link of the Endorser Survey
    • Click Distributions
    • Click Get Reusable Link (if visible)
      
    • Click "Anonymous Link"
    • Copy the Anonymous Survey Link
  • Paste in the Anonymous Link of the Endorser Survey into the initial Endorser email (within the application survey)
  • Append with ?LoginID=${e://Field/LoginID}

...