Page History
| Table of Contents | ||
|---|---|---|
|
Introduction
The
...
eagle-i
...
search
...
application
...
is
...
packaged
...
and
...
deployed
...
in
...
two
...
configurations:
...
institution
...
and
...
central
...
.
...
The
...
institution
...
search
...
application
...
deployment
...
supports
...
the search
...
of resource
...
data
...
in
...
a
...
single
...
institution's
...
eagle-i
...
repository.
...
The
...
central
...
search
...
application
...
deployment
...
supports
...
the search
...
of data
...
in
...
a
...
set
...
of
...
eagle-i
...
institution
...
repositories.
...
In
...
the
...
eagle-i
...
source
...
tree,
...
the
...
webapps/institution
...
project
...
is
...
used
...
to
...
build
...
the
...
single
...
institution
...
search
...
app.
...
The
...
webapps/central
...
project
...
is
...
used
...
to
...
build
...
the
...
multi-institution
...
search
...
application.
...
The
...
institution
...
and
...
central
...
webapp
...
projects
...
contain
...
very
...
little
...
code
...
consisting
...
of
...
two
...
java-based
...
configuration
...
files
...
that
...
determine
...
in
...
which
...
mode
...
the
...
client
...
and
...
server
...
search
...
implementation
...
will
...
be
...
executing.
...
InstitutionApp.java
...
and
...
CentralApp.java
...
are
...
the
...
...
Web
...
Toolkit
...
(GWT)
...
EntryPoint
...
implementations
...
that
...
determine
...
in
...
which
...
mode
...
the
...
GWT-based
...
browser
...
UI
...
will
...
execute.
...
They
...
instantiate
...
a
...
SearchApplicationContext
...
with
...
the
...
parameter
...
true
...
or
...
false
...
that
...
tells
...
the
...
UI
...
whether
...
it
...
should
...
display
...
single
...
or
...
multi-institution
...
search
...
components.
...
For
...
example,
...
in
...
single
...
institution
...
mode,
...
the
...
Institution
...
filter
...
will
...
not
...
be
...
displayed
...
in
...
the
...
Location
...
section
...
of
...
the
...
sidebar,
...
and
...
the
...
institution
...
name
...
is
...
not
...
displayed
...
in
...
the
...
search
...
result
...
summary.
...
InstitutionSearchConfig.java
...
and
...
CentralSearchConfig.java
...
are
...
Spring
...
configuration
...
files
...
that
...
determine
...
the
...
back-end
...
Spring
...
beans
...
that
...
will
...
be
...
loaded
...
for
...
the
...
two
...
search
...
app
...
modes.
...
In
...
particular,
...
it
...
defines
...
the
...
rootSearchProvider
...
singleton
...
bean
...
which
...
is
...
the
...
core
...
handler
...
for
...
search
...
requests
...
from
...
the
...
client.
...
The
...
bulk
...
of
...
the
...
search
...
application
...
implementation
...
lives
...
in
...
the
...
library
...
project
...
dependencies
...
of
...
institution
...
and
...
central
...
WAR
...
projects.
...
Search Back-End Components
Key components of the back-end
...
and
...
client
...
implementation
...
are
...
described
...
below:
Notes:
- => Indicates a singleton (Spring Bean) dependency
- Italic comments in parenthesis indicates potential changes to the current implementation
org.eaglei.ui.gwt.search.server.
...
SearchServlet
SearchServlet handles the core set of search operations requested by the the client. In the SearchServlet init() method, references to its singleton dependencies are obtained, and harvesting of eagle-i repository data is initiated.
=> ConnectionManager
- Called to validate user session ids (Move into SearchProvider?)
=> InstitutionRegistry
- API for getting Institution list when running in central mode (Should be moved to some other servlet.)
=> PollingDataHarvester
- Initiate search index population (Make SearchProvider API?)
- On search operation, first check if initial data indexing has completed (Make SearchProvider API?)
=>SearchProvider
- Process
SearchRequests - Get search result list:
SearchResultSet,SearchResult - Get supertype & subtype counts:
ClassCountResult - Get resource provider type counts:
ClassCountResult - Standard implementation is a
MultiDataSetSearchProvider, described below
=>*EIOntModel
- Compute "root" type of search result (have
SearchProvidercompute; part ofSearchResult)
=> AsynchronousLoggerSearch, AsynchronousLoggerCount
- DB logging of search request, counts, performance
org.eaglei.services.InstitutionRegistry
- Static list of institution information (should be dynamically initialized, for example, determine which institutions are currently accessible)
- Provides name, id, and
RepositoryHttpConfigfor each institution (migrate from legacy URIs.) - Used to lookup institution EIEntity (label) from id
/common/services/src/main/resources/services-config.xml
...
- is
...
- the
...
- primary
...
- configuration
...
- file
...
- for
...
InstitutionRegistry
...
- .
...
- It
...
- is
...
- packaged
...
- in
...
- JAR
...
- (Should
...
- be
...
- loaded
...
- from
...
- classpath
...
- location)
...
- .
...
- It
...
- has
...
- a
...
- dependency
...
- on
...
- finding
...
- an
...
- institution-registry.properties
...
- file
...
- in
...
- the
...
- classpath
...
- which
...
- contain
...
- properties
org.eaglei.tier
...
- used to compute URIs
- (need to confirm that things work if prop is omitted, prod implicit)
org.eaglei.subdomain
...
- defines what node app is running as: "search"
...
- for
...
- central,
...
- or
...
- institution
...
- subdomain/id
...
- used
...
- to
...
- compute
...
- URIs
- (TODO:
...
- bad
...
- prop
...
- name,
...
- fairly
...
- hacky
...
- overloading...)
...
org.eaglei.repository.username
...
org.eaglei.repository.password
...
- (TODO:
...
- central:
...
- can't
...
- assume
...
- same
...
- for
...
- every
...
- institution)
...
org.eaglei.ui.gwt.suggest.server.
...
DataSuggestServlet
DataSuggestServlet is the server-side
...
provider
...
of
...
the
...
list
...
of
...
suggestions
...
displayed
...
as
...
the
...
user
...
types
...
in
...
a
...
search
...
string.
...
Note
...
that
...
client-server
...
communication
...
for
...
this
...
servlet
...
is
...
JSON-based
...
rather
...
than
...
GWT
...
RPC.
...
=>
...
DataHarvester
...
- On
...
- suggest
...
- operation,
...
- first
...
- check
...
- if
...
- initial
...
- data
...
- indexing
...
- has
...
- completed
...
- (make
...
- SuggestionProvider
...
- api?)
...
=>
...
SuggestionProvider
- Process EnityMatchRequest
- Get suggestions:
List<EntityMatch>
org.eaglei.search.harvest.
...
PollingDataHarvester
...
extends
...
DataHarvester
addChangeListener(ResourceChangeListener)
...
(EIEntity
...
institution)
...
onChangeEvent(ResourceChangeEvent
...
event)
...
onChangeStreamEnd(EIEntity
...
institution,
...
Date
...
lastModifiedDate)
...
- (no
...
- rollback
...
- support.
...
- Needed?)
...
optimize()
...
harvest()
...
hasInitalData()
...
- ;
...
- (move
...
- to
...
- SearchProvider?)
...
startPolling()
...
org.eaglei.search.harvest.
...
RepositoryStreamHarvester
...
implements
...
PollingDataHarvester
...
=>
...
InstitutionRegistry
- Get
RepositoryHttpConfigfor a given institution, to call harvest
=> EIOntModel
- Lookup
EIEntityfrom URIs (label) - Lookup valid property list for type
- Get list of resource provider property URIs
- Harvests from a single eagle-i repository
- List of
ResourceChangeListeners - Parses harvest response body in to
ResourceChangeEvent - Two event types: Delete, Changed
- (Withdrawn, Return to Curation/Draft vs. Delete?)
- (Create vs Changed?)
- Drops all resource change notifications that don't have a label property, or type property
- Ignores all properties that don't conform to current
EIClass - Ignores all types that aren't in the current data model
- Exception in a notification is isolated to that listener (no rollback support. Needed?)
- (add support for computing inverse properties, if not done in the repository; here or in indexer)
- on call to
startPolling(), creates a thread that callsharvest()- 10 sec sleep (make configurable)
- (Timeout if
ResourceChangeListenertakes too long?) - (Drop out on out of memory exception?)
- (Would be nice to get change notification from repo for referencing resources, on delete especially)
org.eaglei.search.harvest.
...
MultiDataSourceStreamHarvester
...
implements
...
PollingDataHarvester
...
- multiple
...
- institution
...
- repository
...
- DataHarvester
...
- holds
...
- a
...
- list
...
- of
...
- DataHarvester
...
- on
...
- call
...
- to
...
- startPolling(),
...
- creates
...
- thread
...
- that
...
- calls
...
- each
...
- DataHarvester.harvest()
...
- sequentially
...
- 2
...
- sec sleep (make
...
- configurable)
...
- change
...
- events
...
- go
...
- each
...
- ResourceChangeListener
...
- from
...
- a
...
- single
...
- thread
...
org.eaglei.solr.search.
...
- SolrSearchIndexer
...
- implements
...
- ResourceChangeListener
*
...
- added
...
- to
...
- a
...
- DataHarvester
...
- as
...
- a
...
- ResourceChangeListener
...
=>
...
EIOntModel
...
=>
...
Analyzer
...
=>
...
Directory
...
=>
...
ClassUsageCache
- Used to avoid getting counts for unused classes
=> ProviderUsageCache
- Used to avoid getting counts for unused provider types
=> SearchExcludeConfiguration
/common/runtime/solr/src/main/resources/search-config.xml
...
- Exclude all "Generic
...
- Lab"
...
- provided
...
- resources
...
- Exclude
...
- all
...
- non-resource
...
- provider
...
- Organization
...
- instances
...
- handle
...
- resource
...
- data
...
- received
...
- in
...
- any
...
- order
...
- unresolved
...
- object
...
- property
...
- values
...
- compute
...
- and
...
- store
...
- inferred
...
- supertypes
...
- store
...
- resource
...
- provider
...
- info
...
- assumption:
...
- embedded
...
- class
...
- data
...
- always
...
- received
...
- after
...
- containing
...
- class
...
- rolls
...
- up
...
- properties
...
- of
...
- "flatten"
...
- annotated
...
- instances
...
- (embedded
...
- &
...
- search_flatten)
...
- into
...
- containing
...
- instance
...
- exclude
...
- all
...
- flatten
...
- instances
...
- from
...
- search
...
- results
...
- (add
...
- support
...
- for
...
- some
...
- flatten
...
- instances
...
- that
...
- can
...
- be
...
- search
...
- results)
...
- (Index-time
...
- boosting
...
- of
...
- doc
...
- types
...
- and
...
- fields)
...
org.eaglei.solr.suggest.
...
SolrDataSuggestIndexer
...
implements
...
ResourceChangeListener
- added to a
DataHarvesteras aResourceChangeListener
=> EIOntModel
=> Analyzer
- This is a different analyzer than the search analyzer
=> Directory
- Separate index from search index (same index?)_
- Stores "terms" used in all resource documents
- Resource instance label
- Ontology class label
- Ontology class synonyms
- (Resource instance Alternative Name)
- (Select free text property values:_ gene name)
- (Find term usage in free text)
- Stores the "categories" in which the term is utilized
- The top-level search sidebar resource types
- UnknownCategory (People, Orgs)
org.eaglei.search.provider.
...
MultiDataSetSearchProvider
...
implements
...
SearchProvider
...
- Maps a dataset id (eagle-i,
...
- PubMed,
...
- Entrez
...
- Gene,
...
- NIF,
...
- etc.)
...
- in
...
- SearchRequest
...
- to
...
- a
...
- SearchProvider
...
- implementation
...
- for
...
- that
...
- source
...
- of
...
- data.
...
org.eaglei.solr.search.
...
SolrSearchProvider
...
implements
...
SearchProvider
...
SolrSearchProvider
...
is
...
class
...
that
...
executes
...
a
...
search
...
query
...
on
...
the
...
eagle-i
...
data
...
stored
...
in
...
a
...
Solr
...
indices.
...
The
...
data
...
available
...
to
...
be
...
searched
...
may
...
be
...
single
...
-
...
or
...
multi-institution
...
based
...
on
...
application
...
configuration.
...
The
...
SolrSearchProvider
...
ensures
...
that
...
UI
...
dependencies
...
such
...
as
...
the
...
match
...
highlighting
...
snippit
...
are
...
set
...
in
...
the
...
search
...
response.
...
=>
...
InstitutionRegistry
- For lookup of institution EIEntity (label)
...
- to
...
- put
...
- in
...
- response
...
=>
...
SolrServer
- The SolrServer to be queried -- implementation described below.
=> ClassUsageCache
- Get supertypes and subtypes to perform count queries on
=> ProviderUsageCache
- Get resource provider types to perform count queries on
=> SolrDataSuggestProvider
- Used to extract entity URI(s) from search string. For example, recognize that the query string "thermal cycler" is the label of a type of instrument in the eagle-i ontology. By determining a URI for a search string, matches on the URI can be prioritized.
=> SolrSearchQueryBuilder
- Generates a Lucene query from a
SearchRequest
Solr
The eagle-i system supports three Apache.Solr index configurations: ModelSuggest-core, Search-core, and DataSuggest-core.
An instance of a SolrServer utilized by eagle-i web application (e.g. Search, SWEET, Ontology Browser) will contain one or more of these three indices. Currently, each web application instantiates its own embedded SolrServer with the cores that the particular web app need.It is anticipated that in the future, web applications will utilize a shared SolrServer.
ModelSuggest-coreis an index of all terms in the eagle-i ontology. This core is used to provide autosuggest of ontology class labels and class synonyms. The suggestion list returned can be filtered by type (class uri): for example, only return suggestions of type Instrument. Clients ofModelSuggest-coreinclude the SWEET form fields, and the Ontology Browser (Glossary) searchbox.
- Search-core is an index of eagle-i resource data. Querying this index produces the relevance ranked list of search results presented to the user in the search application. The central search application maintains an index of data from all institutions in the network. The search application on each institutional node maintains an index of the data of that particular institution.
- DataSuggest-core supports the suggestions offered by the search application search box. It is an index of all resource instance labels. It also indexes ontology terms and synonyms, however, unlike
ModelSuggest-core,DataSuggest-corewill only include ontology terms *currently used in the dataset*. For example, if the ontology term "Lepidosauria" is not currently used by any resource in the eagle-i dataset, it will not be offered in autosuggest.
Another feature of the DataSuggest-core is that it keeps track of the resource categories in which any given term (resource label or class label) is used. This supports the "in Instruments" and "in Reagents" feature of the search autosuggest.
org.eaglei.solr.EagleISolrConfig
This class is provides api for reading in solr configuration files, and determining the location that solr data files will be written.
By default, the configuration files for these solr cores are read from the solrhome directory in the eagle-i-common-solr.ja.
The location of data files created for the these core indices is the concatenation of the system property org.eaglei.home, a string passed to the constructor of EagleISolrConfig (normally the application name -- search, sweet, glossary), plus the name of the core.
Overview
Content Tools