Page History
| Table of Contents | ||
|---|---|---|
|
Introduction
The eagle-i search application is packaged and deployed in two configurations: institution and central. The institution search application deployment supports the search over of resource data in a single institution's eagle-i repository. The central search application deployment supports the search over of data in a set of eagle-i institution repositories.
...
The bulk of the search application implementation lives in the library project dependencies of institution and central WAR projects.
...
Search Back-End Components
Key components of the back-end and client implementation are described below:
...
Notes:
- => Indicates a singleton (Spring Bean) dependency
- Italic comments in parenthesis indicates potential changes to the current implementation
org.eaglei.ui.gwt.search.server.SearchServlet
SearchServlet handles the core set of search operations requested by the the client. In the SearchServlet init() method, references to its singleton dependencies are obtained, and harvesting of eagle-i repository data is initiated.
...
- Compute "root" type of search result result (have
SearchProvidercompute; part ofSearchResult)
...
- DB logging of search request, counts, performance
org.eaglei.services.InstitutionRegistry
- Static list of institution information (should be dynamically initialized, for example, determine which institutions are currently accessible)
- Provides name, id, and
RepositoryHttpConfigfor each institution (migrate from legacy URIs.) - Used to lookup institution EIEntity (label) from id
/common/services/src/main/resources/services-config.xmlis the primary configuration file forInstitutionRegistry. It is packaged in JAR (Should be loaded from classpath location). It has a dependency on finding an institution-registry.properties file in the classpath which contain propertiesorg.eaglei.tier- used to compute URIs
- (Need need to confirm that things work if prop is omitted, prod implicit)
org.eaglei.subdomain- defines what node app is running as: "search" for central, or institution subdomain/id
- used to compute URIs
- (TODO: bad prop name, fairly hacky overloading...)
org.eaglei.repository.usernameorg.eaglei.repository.password_ (TODO: central: can't assume same for every institution)
...
org.eaglei.ui.gwt.suggest.server.
...
DataSuggestServlet
...
DataSuggestServlet is the server-side provider of the list of suggestions displayed as the user types in a search string. Note that client-server communication for this servlet is JSON-based rather than GWT RPC.
=> >DataHarvestero
- On suggest operation, first check if initial data indexing has
...
- completed (make SuggestionProvider api?)
=> > SuggestionProvidero
- Process EnityMatchRequest
...
- Get suggestions:
...
-
List<EntityMatch>
org.eaglei.search.harvest.PollingDataHarvester extends DataHarvester
...
addChangeListener(ResourceChangeListener)
...
(EIEntity institution)
...
onChangeEvent(ResourceChangeEvent event)
...
onChangeStreamEnd(EIEntity institution, Date lastModifiedDate)
...
...
- (no rollback support.
...
- Needed?)
...
optimize()
...
harvest()
...
hasInitalData()
...
- ; (move to SearchProvider?)
...
startPolling()
org.eaglei.search.harvest.RepositoryStreamHarvester implements PollingDataHarvester
=> >InstitutionRegistryo
- Get
RepositoryHttpConfigfor a given institution, to call harvest
=> > EIOntModelo
- Lookup
EIEntityfrom URIs (label)
...
- Lookup valid property list for type
...
- Get list of resource provider property URIs
...
- Harvests from a single eagle-i repository
...
- List of
ResourceChangeListeners
...
- Parses harvest response body in to
ResourceChangeEvent
...
- Two event types: Delete,
...
- Changed
...
...
- (Withdrawn, Return to Curation/Draft vs. Delete?)
...
- (Create vs Changed?)
...
- Drops all resource change notifications that don't have a label property, or type property
...
- Ignores all properties that don't conform to current
EIClass
...
- Ignores all types that aren't in the current data model
...
- Exception in a notification is isolated to that
...
- listener (no rollback support.
...
- Needed?)
...
- (add support for computing inverse properties, if not done in the
...
- repository; here or in indexer)
...
- on call to
startPolling(), creates a thread that callsharvest()
...
- 10 sec
...
- sleep (make configurable)
...
- (Timeout if
ResourceChangeListenertakes too long?)
...
- (Drop out on out of memory exception?)
...
- (Would be nice to get change notification from repo for referencing resources, on delete especially)
org.eaglei.search.harvest.MultiDataSourceStreamHarvester implements PollingDataHarvester
...
- multiple institution repository DataHarvester
...
- holds a list of DataHarvester
...
- on call to startPolling(), creates thread that calls each DataHarvester.harvest() sequentially
...
- 2 sec sleep (make configurable)
...
- change events go each ResourceChangeListener from a single thread
org.eaglei.solr.search.SolrSearchIndexer implements ResourceChangeListener
...
- *added to a DataHarvester as a ResourceChangeListener
=> EIOntModel
=> Analyzer
=> Directory
=> ClassUsageCacheo
- Used to avoid getting counts for unused classes
=> ProviderUsageCacheo
- Used to avoid getting counts for unused provider types
=> SearchExcludeConfigurationo
/common/runtime/solr/src/main/resources/search-config.xml
...
- Exclude all "Generic Lab" provided resources
...
- Exclude all non-resource provider Organization instances
...
- handle resource data received in any order
...
- unresolved object property values
...
- compute and store inferred supertypes
...
- store resource provider info
...
- assumption: embedded class data always received after containing class
...
- rolls up properties of "flatten" annotated instances (embedded & search_flatten) into containing instance
...
- exclude all flatten instances from search results (add support for some flatten instances that can be search results)
...
- (Index-time boosting of doc types and fields)
org.eaglei.solr.suggest.SolrDataSuggestIndexer implements ResourceChangeListener
...
- added to a
DataHarvesteras aResourceChangeListener
=> EIOntModel
=> Analyzero Different analyzer from
- This is a different analyzer than the search analyzer
=> Directoryo
- Separate index from search
...
- index (same index?)_
...
- Stores "terms" used in all resource documents
...
- Resource instance label
...
- Ontology class label
...
- Ontology class synonyms
...
- (Resource instance Alternative Name)
...
- (Select free text property values:
...
- _ gene name)
...
- (Find term usage in free text)
...
- Stores the "categories" in which the term is utilized
...
- The top-level search sidebar resource types
...
- UnknownCategory (People, Orgs)
...
org.eaglei.search.provider.MultiDataSetSearchProvider implements SearchProvider
...
- Maps a dataset id (eagle-i, PubMed, Entrez Gene, NIF, etc.) in SearchRequest to a SearchProvider implementation for that source of data.
org.eaglei.solr.search.SolrSearchProvider implements SearchProvider
SolrSearchProvider is class that executes a search query on the eagle-i data stored in a Solr indices. The data available to be searched may be single- or multi-institution based on application configuration. The SolrSearchProvider ensures that UI dependencies such as the match highlighting snippit are set in the search response.
=> InstitutionRegistryo
- For lookup of institution EIEntity (label) to put in response
=> SolrServero
- The SolrServer to be queried -- implementation described below.
=> ClassUsageCacheo
- Get supertypes and subtypes
...
- to perform count queries on
=> ProviderUsageCacheo
- Get resource provider types to perform count queries on
=> SolrDataSuggestProvidero
- Used to extract entity URI(s) from search string.
...
- For example, recognize that the query string "thermal cycler" is the label of a type of instrument in the eagle-i ontology.
...
- By determining a URI for a search string, matches on the URI can be prioritized.
=> SolrSearchQueryBuildero
- Generates
...
- a Lucene query from a
SearchRequest
Solr
The eagle-i system supports three solr Apache.Solr index configurations: ModelSuggest-core, Search-core, and DataSuggest-core.
An instance of a SolrServer utilized by eagle-i web application (e.g. Search, SweetSWEET, Ontology Browser) will contain one or more of these three indices. Currently, each each web application instantiates its own embedded SolrServer with the cores that the particular web app need. It is anticipated that in the future, web applications will utilize a shared SolrServer.
ModelSuggest-coreis an index of all terms in the eagle-i ontology.
...
- This core is used to provide autosuggest of ontology class labels and class synonyms.
...
- The suggestion list returned can be filtered by type (class uri):
...
- for example, only return suggestions of type Instrument.
...
- Clients of
ModelSuggest-coreinclude the
...
- SWEET form fields, and the Ontology Browser (Glossary) searchbox.
- Search-core is an index of eagle-i resource data.
...
- Querying this index produces the relevance ranked list of search results presented to the user in the search application. The central search application maintains an index of data from all institutions in the network.
...
- The search application on each institutional node maintains an index of the data of that particular institution.
- DataSuggest-core supports the suggestions offered by the search application search box. It is an index of all resource instance labels.
...
- It also indexes ontology terms and synonyms, however, unlike
ModelSuggest-core,DataSuggest-corewill only include ontology terms *currently used in the dataset*.
...
- For example, if the ontology term "Lepidosauria" is not currently used by any resource in the eagle-i dataset, it will not be offered in autosuggest.
Another feature of the DataSuggest-core is that it keeps track of the resource categories in which any given term (resource label or class label) is used. This support supports the "in Instruments" and "in Reagents" feature of the search autosuggest.
org.eaglei.solr.EagleISolrConfig
This class is provides api for reading in solr configuration files, and determining the location that solr data files will be written.
By default, the configuration files for these solr cores are read from the solrhome directory in the eagle-i-common-solr.jarja.
The location of data files created for the these core indices is the concatenation of the system property "org.eaglei.home", a string passed to the constructor of EagleISolrConfig (normally the "application name" -- search, sweet, glossary), plus the name of the core.
Overview
Content Tools