Versions Compared

Key

  • This line was added.
  • This line was removed.
  • Formatting was changed.
Warning

The ETL toolkit recently underwent a major re-architecture that will be part of our upcoming 2.0MS4 release. As a result, templates generated with versions prior to 2.0MS3 will no longer supported. Grab a snapshot of the 2.0MS3 code if you'd like to start experimenting.There is a problem with released vesion 3.1.1 which will be addressed in our upcoming release (1Q15)
In the meantime, please use ETL toolkit version 3.0.4

Info

The toolkit described herein is currently not user-friendly (though it works well – we use it routinely to bulk-upload data for the Consortium members). If you encounter issues, please do not hesitate to contact us.

...

To generate etl templates and maps, navigate to the dedicated directory (above) and run the script:

No Format

./generate-inputs.sh -t typeURI

...

Transformation maps will be contained in a subdirectory of ./maps named after the type and ontology version, e.g:

No Format

./maps/instrument_ont_v1.1.0

...

  1. Place your input files (i.e. the completed templates) in a directory of your choice, e.g. dataDirectory. All files contained in this directory will be processed by the ETLer.
  2. To run an ETL, execute the following command (note that all records will be uploaded in the requested workflow state):

    No Format
    
    ./ETLer.sh -d dataDirectory [-p DRAFT|CURATION|PUBLISH] -c username:password -r repositoryURL
    
    Info

    If you are practicing the ETL process, you may wish to upload your data to the common eagle-i training node. In this case, if your directory is named dataDirectory, the script would be executed as follows (default workflow state is DRAFT):

    No Format
    
    ./ETLer.sh -d dataDirectory  -c L4:Level4 -r https://training.eagle-i.net
    

    Note that the data that is uploaded to the training node CAN be viewed and modified by others even in a draft state (even if you subsequently lock the records). Note also that the information in the training node is not persistent as the node is refreshed periodically.

  3. To verify the data upload, log on to the SWEET application and select the lab to which the ETLd resources belong.

...

Resources that are uploaded to an eagle-i repository via ETL are tagged with the name of the file from which they were extracted. It is therefore relatively simple to de-ETL an entire file. To do so, execute the following command:

No Format

 ./deETLer -f filename -c username:password -r repositoryURL

...