Enterprise Search

TIVITY's Enterprise Search is a powerful, centralized search solution that enables users to find information across multiple applications and data sources within their organization.

Enterprise Search revolutionizes information retrieval across your organization. This guide offers comprehensive instructions for administrators, app creators, and end-users to harness its advanced capabilities, including unified search, semantic query expansion, and customizable personas.

Search

In addition to the default Global Search, the TIVITY platform provides an optional advanced Search feature. This feature is accessible via the Explore App. The default installation will have it enabled, providing a powerful tool for finding specific information within the platform.

General

Unlike the default Global Search, the Explore App doesn't use Adapters to perform search operations. Instead, it creates and maintains a dedicated search index. This approach enables advanced features such as:

Filtering
Facets
Customized search ranking
Search personas

Users

The advanced search feature is accessible through the App Explore which can be found within the list of installed applications within the workspace.

If the App is not available, your Workspace is not allowed to use it. Please contact the Platform Administrator in this case.

First Time Setup

When the Explore App is opened for the first time within a Workspace, a prompt to initialize it will appear. This prompt will only show up once and only as long as it hasn't been completed yet. If you are an Administrator within the Workspace, you can complete the setup by executing the Initialize Now action. If you are not a Workspace Administrator, please contact one of your Administrators to do that. For more information on this process, refer to the Workspace Administrator section.]

Remarks: If your Workspace does not contain any App which supports the Search feature, you will not be able to set up the Workspace. If this is the case, either install an App which supports the Search or ask your App manufacturers to support the feature.

General Usage

The Explore App allows you to search within all supported Apps within a Workspace. You can enter free text into the search box to search for your specified search terms and/or open the filter menu to drill down until you get to the item you are looking for.

Search Syntax

Normal Search

You can enter multiple words within the search box. For an item to be included within the search result, the item must contain all terms you've entered.

Searching for

abc jkl

will only contain items which matches the terms abc and jkl. To match a term, an item must have a word which

Exactly matches. For abc, this means that test abc def would match.
Start with the given term. For abc, this means that test abcd def would match, but test abdc def would not.
End with the given term. For abc, this means that test zabc def would match, but test zabck def would not.

All terms are case-insensitive.

Each term has a little bit of fault tolerance built in. If you make a typo in your search phrase and enter Affacted, items which contain the correctly spelled word Affected will be included. This includes both directions. If the word within an item is misspelt, it would still match a correctly spelt search phrase. For performance reasons, the fault tolerance is not infinite. The first two characters must match and the rest of the characters can be a little bit off. It supports substation, insertion and removal of single characters.

Each term will also bring additional terms with it. These synonyms are picked automatically. For example, symbol will also search for figure. These synonym matches are ranked lower compared to the originally entered term.

Exact Matches

To search for an exact match, you can surround single or multiple search terms within quotes. An exact match means that a word must exactly match with the entered term. If an exact match contains multiple terms, the order of the terms and the exact spelling of the terms must match. Additionally, these terms must be close to each other, there can only be one word between them.

process will match processing or preprocess.
"process step" will not match processing step or step process or process to be executed in one step, but it will match process a step

When using exact searches, the term will not try to correct spelling mistakes. Additionally, no synonyms will be added.

Like normal searches, exact searches are case-insensitive.

Excluding

You can specify one or more terms to exclude items to get into the search result when they include one of those excluded words. To exclude a term, prepend an - in front of it. If the - is directly between two terms without any white space, it is not detected as an exclude filter.

abc -def will match test abc jkl but will not match test abc def
abc-def will match test abc jkl and test abc def

You can add the exclude filter to normal terms or to exact matches.

-abc will exclude abc, abcd and zabc
"abc" will exclude abc but not abcd
"abc def" will exclude abc def but not abc

Semantic Search

Additionally to the Normal Search, the Explore App offers a Semantic Search mode. This mode will not only search for items which matches your search phrase, but also items which might be relevant for the matching items.

As an example, imagine you have a set of tasks which contain a category and a title. If you enter the category name as your search phrase, all tasks with that category will get returned. If you switch to the Semantic Search mode, not only these tasks will be returned but for example also E-Mails which contain the title of the task.

This interconnection is generated automatically based on word combination statistics. The best way to think about it: you do not specify the terms in your search phrase which must match, you provide key words describing a context with your search words and you will find the items which belong to this context.

To work properly, your total number of searchable items needs to have a sufficient size.

The previous methods described the basic syntax of a search phrase. The syntax is valid for both, the normal search and the semantic search.

Filters and Facets

Additionally to the search phrase, you can drill down into your search result by adding filters or by using facets. To do that, open the filter pane. This can be done by clicking on the filter icon on the left-hand side.

Facets

With the filter pane, you will get a list of all available facets.

By clicking on the checkbox within a facet, the result will be reduced to include items which match the selected facet. If you select multiple values from a single facet, only items which match one of the selected facet items will be within the search result. If you select items from multiple facets, only items which have a match on all facets will be within the search result.

All facets which are currently active will appear on top of the search result as well.

By clicking on an item, you can modify the selection of facet items.

By clicking on the cross on the right hand side of each item, you can remove that facet from the list of active filters.

By clicking on the Add Filter button, you will get prompted to select the facet you want to filter by first. After selecting the facet, a popup will appear in which you can select all facet items for that facet. Closing the popup will apply the filter to the search result.

Custom Filter

At the top of the filter pane, you can add a custom filter. To do that, click on the Add custom filter button. A dropdown will appear below it. In that dropdown, you can select the field for which you want to filter by.

Remarks: The list will only contain fields of all items of the current search result.

After selecting the field, press the Add button to add the filter. Depending on the data type of that field, different filter input forms will appear.

Numeric filter

For numerical fields, two input bars will appear.

Within the left one, you can specify the minimum value (inclusive) for that field for an item to be included within the search result.

Within the right one, you can specify the maximum value (inclusive) for that field for an item to be included within the search result.

If you enter a value in both fields, the value must be within the range of both of your specified numbers.

Date Time Filter

For the Date Time fields, a date range input will appear. The default value will be the date range for the last seven days.

You can customize the range by clicking on the input bar. From the dropdown, you can select from one of the predefined templates or you can specify a custom range. For that, first click on the day in the left calendar first to specify the oldest date. After that, click on a day in in the right calendar to specify the newest date. The filter will get applied after pressing Apply

Sorting

The sorting of the search results can be changed. The default sorting is Relevance. If Relevance is selected and no search phrase is entered, the sorting will be Recent + Relevance

By clicking on the Sort by name, a dropdown will appear where you can select the field to sort by.

Relevance sorting will sort the result purely on the relevance calculated based on the match of the search phrase and the matched items.
Time + Relevance sorting will sort the result on the relevance calculated based on the match of the search phrase and the matched items. Additionally, it will move items which where created or modified recently more to the top. Items which were created or edited more recently will be further up within the search result.

Customization

The Explore App can be customized to better adjust for individual needs. To open the configuration, click on the cog icon in the header of the main Explore App page.

Sources

The Sources tab allows to add pre-configured user sources to the search index. When opening the tab, a list of possible user sources will be displayed. The list of user sources depend on the Apps installed within the workspace and if the Apps are providing user sources. If no user sources are visible, no currently installed application provides any user source.

By clicking on a User Source, you can add it to your personal search index. To do that, click on the Add this source action.

Important: make sure you've authenticated yourself for the source if necessary. You can do that in the Team App.

After you've added a user source, it will get indexed in the background after a while. Depending on the source itself, it can take a few minutes until you will see search results from that source. The source will be added for all searches automatically if the type is not excluded by your selected persona.

Important: by adding a User Source, the content of that source will be persisted on the TIVITY server for indexing. Only you can search in this source. If you do not want to use one of your User Sources anymore, you can remove it on the same screen. Removing a user source will delete all persisted data from that source from the TIVITY server.

Personas

Personas allow you to customize the priority order of each type in the search result. Each user will get the first matching persona as its default one. As soon as one user can select between multiple personas, he can switch between them either by selecting one on the start screen with the Search As selection

or by opening the now-visible Persona Pane on the left-hand side.

Hint: You can switch between Personas even if you already entered a search phrase or added a filter. The search result will update as soon as a different persona gets selected.

The persona selection does not persisted.

To create a Persona, click on the + sign in the Persona tab. A new Persona will be created.

Groups

Not all Personas are relevant for all users within a single workspace. For that, Personas can be filtered by Groups. Select the Groups, a user must be a member in, in the Common tab. A user can only select a Persona if his or her group memberships matches one of the selected groups for this persona.

Remarks: if a user can not use any persona, the Explore App will fall back to the default values for this user.

Remarks: the first matching Persona will be the default selected persona for each user. So applying an order for the Personas is important. You can change the order of the Personas by dragging the items up or down with your mouse.

Excluded Types

By default, all types which are prepared to use the Explore App will be searchable. In some cases, this might be too much for certain Personas. In the Excluded Types tab, you can exclude certain types from the search result entirely.

This is not the recommended way, especially for Personas sorted more on the top. The user will not get any information about excluded types. If he searches for something but doesn't find it, it might be confusing for this user first. Try to adjust the Relevance instead in the next tab. This allows you to move all items of one or more types up or down in the search result without excluding them entirely.

Relevance

In this tab, you can change the priority of types within the search result. The priority is a numeric factor. All values are relative to each other. If you set the priority of a value to 2, it will be more on the top compared to types having a lower value. These relevance settings are not absolute. The match relevance of a search phrase with each item is still calculated and will influence the order in the search result. It is not guaranteed that items with a higher value will be always in front of items with a lower value.

The value supports fractional inputs. You can boost an item by a factor of 1.5 if you want.

It is recommended to do small steps, like incrementing the value by 1 or decrementing it by 0.2 and then test the result with various search phrases.

Custom Facets

Custom facets allow you to create a set of pre-defined filters which will be displayed above the build-in facets in the filter pane.

To create a facet, click on the + sign above the facet list. A popup will appear where you must select the field you want to create a facet for. For that, you first select the type of the field and you will get a list of field names which have this type.

Hint: if you hover over a type, a hint will appear which will display all types which have this field.

After creating the facet, you have to define the facet items. You can create these from scratch or start with one of the templates.

Hint: you can modify all items from each added template, add new ones or even delete them. You can even combine multiple templates. Even if no template is a perfect fit for your data, it might help you getting started.

You can always preview the result with your data by clicking on the Refresh Icon in the Live Preview box.

Numerical values work exactly like adding a filter in the Explore App. Date and DateTime values can be entered in absolute or by using the built-in expression. When the field supports Date Expressions, an inline help will be visible. It describes the syntax and has a test field where you can enter an expression and see the result immediately.

After saving your custom facets, you need to refresh the Explore App main window to load them.

Workspace Administrator

Workspace Administrators play a crucial role in managing and customizing the search experience for their specific workspace. Their primary responsibilities include:]

Initializing the Explore App for their workspace
Managing workspace-specific search settings
Customizing search experiences for users within their workspace

Initialize a Workspace

After the Platform Administrator has enabled the Explore App for a workspace, the Workspace Administrator needs to perform a one-time initialization:

Navigate to the Workspaces tab in the Search Administration section.
Locate the workspace requiring initialization in the list of pending initializations.
Click on the workspace to display the initialization form.
Review the initialization information.
Click on the Initialize Now action to complete the setup.

After the initialization is done, the workspace details will display two additional tabs, Index Statistics and Timing Statistics.

Note: The initialization will not be available if the workspace does not contain any App having a Search Configuration.

The initialization will not be available if the workspace does not contain any App having a Search Configuration.

After the Platform Administrator has enabled the Explore App for a workspace, the Workspace Administrator needs to perform a one-time initialization:

Workspaces

This tab allows performing the one-time initialization of a workspace and viewing various statistics for that workspace.

Within the tab, a list of workspaces with pending initialization is displayed. Clicking on a workspace will display the initialization form. After reviewing the information, clicking on Initialize Now will set it up.

Managing Search Settings

Workspace Administrators can:

Monitor Index Statistics for their workspace
View Timing Statistics for search operations
Manage user sources within the workspace

Customizing Search Experience

Workspace Administrators have the ability to:

Create and manage search personas for their workspace
Configure custom facets
Adjust relevance settings for different content types within their workspace

App Creator

The responsibility of the App Creator is to define which classes should be searchable and which fields should be included within the search index. Additionally, the App Creator should add proper cleanup operations for each field.

The Explore App search is designed to work as independently as possible. Because of that, only a limited set of options within the class designer affects the advanced search.

Note: The configuration for the search is only available if the search feature is not disabled globally. A workspace is not required to be eligible to use the Explore App to configure the Search.

General Behavior

Before starting the configuration, some general information about the behaviour is important to know.

Mapping

In general, the search works with its own class structure compared to the one defined within the class designer. The translation between the class designer structure and the search structure is called mapping. This allows introducing search-only related configurations to each class and adjusting the configuration independently from the structure within the class designer.

Source Types

The search has two operation modes per class: Shared Source and User Source. The crawling of shared sources and user sources doesn't differ.

A shared source is the common case. Shared sources are classes where items can be seen by multiple users (not considering rights). A database table is a common case for a shared source.

A user source defines a source where only one user can see a given set of items. A mailbox is a common case for a user source where each user has its own mailbox even within a single workspace. Each user can manually enable the user sources he or she wants to index.

Mapping Key

All items of a class are translated to a configurable structure. The structure is a flat list of key-value pairs. Because the search works outside of any application, the key is not the same as the key of a field within the class designer because this could lead to unwanted conflicts. Instead, a field is mapped to a different independent key. This independence helps avoid unwanted conflicts and allows merging fields from different classes across multiple applications if desired.

The mapping key must have the format of <Prefix>.<Key>. Both can be customized. When a search is executed, all values with the same mapping key are grouped into a single search field.

Value Processors

Because the raw field values might not always be suitable for a good search experience, each field can have one or more value processors. These will transform the raw value from the source. A good example of this is HTML fields. Without a value processor, the search index will not only contain the text content of the HTML value but also all HTML tags. To exclude these unwanted words from the search index, you can add value processors that will remove these tags.

Strategies and Crawling Settings

Not all sources behave equally across the board, and not all sources are updated from within the platform itself only. To be able to also include these sources properly, the App Creator can configure specific strategies and crawling settings.

Configuration

Within the Configuration Mode for an App, a dedicated Search section is available within the Modules category. Click on this section to open the designer.

The designer has two tabs: Sources and Searching.

Within Sources, you configure which classes should be indexed and how the indexing should operate.

Within Searching, you configure how items of that class are displayed in the search result.

Add a Class to be Searchable

Click on the Sources tab. To add a class to the list of mapped classes, click on the + above the list of mapped classes.

A popup will appear where you can select one or more classes.

The prefix should be the general domain of the application. By default, it is the name of the source. The entered value can be changed for each field later on.

For each source, a list of available classes is displayed. Clicking on a class will select it; clicking on it again will unselect it.

After clicking on Execute, the class will be added to the list of mapped classes and can now be configured in detail.

Fields

First, the fields that should be stored within the index should be selected here. The list of fields is grouped by their data type.

Type

NaturalText indicates that the content of the field is some kind of free text without any restriction to a single simple format. The body text of an email is an example of that.
String indicates that the content of the field contains text but follows a specific pattern or is not even modifiable directly by the user, like a combo box value.
DateTime, Date, and Time fields represent a single point in time.
Numeric fields contain numbers only. All numeric fields have a 64-bit precision.
Float and Double fields contain numbers with decimals.
Duration fields represent a time duration without any reference to a point in time. Compared to a Time field, a duration field can be longer than one day.

For certain types of fields, the type can be changed. The pre-selected type is calculated depending on the configuration for that field from within the class designer.

Fields with a data source are marked as String.
String fields are marked as NaturalText.
DateTime fields are marked as DateTime. Depending on the control type, it might get changed to Date or Time.
Numeric fields like Integer, Byte, etc., are all marked as Numeric. A long field with the control type Duration will be changed to Duration.
Float fields are marked as Float.
Double and Decimal fields are marked as Double.
Boolean fields are not supported and are not available.
All other fields are marked as NaturalText fields.

Search Mode

The search mode indicates the relevancy of this field when the user enters a search text. An item matching a search term in a high-relevancy field will be more at the top compared to a match in a lower-relevancy field.

The order of relevancy is:

Title
SubTitle
Content

Fields set to NoSearch are saved within the index, but a user-entered search term will not search within this field. The user can only filter for fields saved within the index, so this option is useful to allow the user to narrow down the search result with more filters later on.

DateTime fields should be set to NoSearch normally. The search mode operates on words, and the text representation of DateTime fields is not how the user expects it to be.

Multi-Value

Selecting this option indicates that this field might have more than one value simultaneously. While the platform does not support this feature by itself, it can be enabled for search. This can be because a value processor might convert a single text value into multiple distinct values.

Enabling this setting for values that do not have multiple values at the same time has a performance penalty and should be avoided.

Not enabling this setting for values that do have multiple values at the same time might result in incomplete search results.

Facet

When enabling this option, a facet will be created for this field at indexing time. A facet allows the user to drill down the search result. A facet is a string representation of the field value, and each value is case-sensitive.

Keeping the Fields Up to Date

The fields will not automatically get updated after changes are made within the class designer. To update an existing mapping, click on the Sync button. This will compare the mapping with the latest configuration from the class designer. A popup will appear listing:

All new fields
Fields that are not equal to the initial mapping configuration
All deleted fields

In this popup, select the modifications you want to apply and press Execute.

Fields that got modified will be updated. New fields will be added to the list but in a deactivated state. You need to activate them manually by clicking on the checkbox in front of the field.

Important: Newly added fields will have the default prefix for the mapping key. Please review this generated mapping key and adjust it to match the expected value.

Crawling Settings

These options change how items are retrieved from the source.

Paginated Loading

This is the default loading operation. This operation will fetch individual pages with a fixed number of items per page from the source and process them. This option can be configured by providing a JSON configuration like this:

{ 
    "pageSize": 100, "retryCount": 5 
}

The pageSize property defines how many items each page should include. The default value is 100 but it can be adjusted depending on the behavior of the source. If the source itself does not support pagination and pagination is done in memory within the adapter, the page size can be increased significantly (e.g., to 10000) to improve performance and reduce the load on the source itself. This value should also be increased to a proper value if the source itself has high latency, but the latency is not proportional to the number of items requested.
The retryCount specifies how many times a single page should be fetched from the source in case a fetch has failed. This allows indexing sources that are overloaded and produce timeouts regularly. The default value of 5 should be okay for almost all cases.

Item Receiving

This option allows adjusting for the behaviour of a given source.

The From Page behaviour is the default one. It means that each requested page will contain all relevant fields from the source itself. This option doesn't have any further configuration.
The Fetch Individually behaviour is for sources that only return a subset of fieldswhen multiple are requested. Microsoft Exchange is an example where the body of an email will not be returned if a list of emails is requested. This behaviour will fetch the items first and then fetch each returned item from the source again. This option produces a lot more load on the source and should only be used if the source shows this behaviour.

This option has the following further configuration:

{ 
    "parallelRequests": 1 
}

The option parallelRequests specifies how many individual item requests should be performed in parallel for each item returned from a single page. Higher values will decrease the time needed to iterate over a list of items but increase the load on the source itself. The default value for parallelRequests is 1.

Strategies

This section configures how this class should behave in general and how updates within the class should get detected.

Source Type

This specifies the general type of the source itself.

The type Shared Source is the default type. This means that there will be one index for this class per workspace. Search queries for all users will be executed within that search index.

This option has an optional advanced configuration:

{ 
    "ignoreConditionalRights": false 
}

When ignoreConditionalRights is set to true:

All configured conditional rights of the platform will be skipped during crawling.
The index will still apply the configured rights.
Items that the crawling user cannot see will be included in the index.

This option should be set to true for:

Classes that use ACL rights
Bigger tables with many relations or poor indices (only after detailed analysis)

Otherwise, only items visible to the crawl user will be indexed. The default value for this option is false.

Updates

This section specifies how updates are detected and propagated to the search index. Note that multiple strategies can be active simultaneously.

With Changes within the Platform enabled, the crawler will periodically check the Activity Stream of the TIVITY platform to detect changes.

With Periodically Reindex enabled, the crawler will reindex all items of that class at the configured interval. This option should only be used if really needed because it can create a high load on the source. This option requires an additional configuration:

{
    "reindexInterval": "00:10:00"
}

The option reindexInterval specifies the time between each reindex. The time must be specified in the format <days>.<hours>:<minutes>:<seconds>. If no days a specified, the format is <hours>:<minutes>:<seconds>. The smallest allowed interval is five minutes.

With an incrementing field enabled, the crawler will fetch the highest value of each of the specified fields from the index and asks for the source if there are any new items where the field value is greater than the last known value. This option requires an additional configuration:

{
  "fields": [
    "SortingTime"
  ],
  "checkInterval": "01:00:00"
}

The option fields is a list of field keys (not the mapping key) which should be considered for evaluation. These fields must be saved within the index itself and they must be mapped to one of the following index data types:

DateTime
Double
Float
Numeric

If multiple fields are specified, each known value is combined with OR.

Example: given that the fields Foo and Bar are specified. The highest known value for Foo is 12 and the highest known value for Bar is 42. This would generate a selection like

SELECT FROM source
WHERE
    Foo > 12
OR  Bar > 42

The option checkInterval specifies the time between each check. The time must be specified in the format <days>.<hours>:<minutes>:<seconds>. If no days a specified, the format is <hours>:<minutes>:<seconds>. The smallest allowed interval is five minutes.

With Reindex old items enabled, the crawler will reindex all items of that class where the last time they were crawled is more than the configured amount of time ago. This option will not perform any operation if the index itself is empty. This option requires an additional configuration:

{
    "itemAge": "30.00:00:00"
}

The option itemAge specifies the age an item within the index must have before it gets scheduled for a reindex. The time must be specified in the format <days>.<hours>:<minutes>:<seconds>. If no days a specified, the format is <hours>:<minutes>:<seconds>. The smallest allowed age is five minutes.

Value Processors

Value processors will modify the content of a given field before it will get send to the search index. Value Processors are predictively pre-generated when a class is added to the mapping for the first time. The prediction is based on the data type, configured data source and the configured control type from within the class designer.

Value Processors are executed for each field from top to bottom. The output of a value processor will be the input of the next value processor. If the chain of value processors returns null for a field, it will not be added to the index.

Remark: a field with an empty string will be added to the index which increases the size of the index.

Convert to lowercase will convert any input to lowercase characters only
Convert to upper case will convert any input to upper case characters only
Remove LegacyExchangeDN from Email Address will remove any ActiveDirectory-only related address from an E-Mail address field text. These values are not valid E-Mail addresses and further E-Mail address-related processors might fail if these values are not removed. If the field can contain multiple E-Mail addresses, do not use the Split Email Address List processor before using this one.
Ensure not default date: Some adapters might have a default Date value for a field which has no value. This date value refers to the first of January of the year 0, which is not a common date for most business cases. This Processor will replace this value with null meaning that the value will not saved within the index.
Ensure not empty: String fields where the value is an empty string will get replaced by null meaning that it will not get saved within the index.
Ensure resolved not null: For fields which have a data source, but the data source does not have any item for the value saved within the field, the foreign key of that field will get saved within the index. With this processor, these values will get replaced with null which will not add them to the index. If you use the EditComboBox control type, this processor should not be added.
HTML Decode: this processor will replace common HTML substitutions like   or & with their corresponding normal characters. This processor should come after the HTML strip processor.
HTML Strip: this processor will remove all HTML control characters from the string (e.g. <div> or <span>). This processor tries to represent the layout structure of the HTML itself with its plain text counterpart. If an element has inline styles which will hide its content, this processor will not include any text found in that element in its output. It is recommended to use this processor for all HTML fields.
Normalize Email Address: This processor will remove the optional display name from a single E-Mail address and convert the address itself to lowercase. This is highly recommended if the field is used as a facet. If the field can contain multiple E-Mail addresses, use the Split Email Address List processor before using this one.
Normalize Line Break will convert Windows Style line breaks or macOS style line breaks into the one character UNIX style line break.
Reduce Whitespace will remove unnecessary white space characters from a text. Unlike Trim, the evaluation is performed per line. It is recommended to put the Normalize Line Break processor before this one.
Replace Value will replace values with a set of predefined values. It can be used to exclude fields when having a given value.
Split will create splits of the value on the specified split character into a list of values. This value processor will produce a multi-value list, it is recommended to enable the option Multi Value for this field if this processor is used.
Split Email Address List will convert a single recipient address list string into a list of individual E-Mail addresses. The string must be RFC5322 compatible and individual addresses must be separated by a ,. This value processor will produce a multi-value list, it is recommended to enable the option Multi Value for this field if this processor is used. It is highly recommended to use the processor before adding further processors for E-Mails like for example for using the Normalize Email Address.
Trim will remove all leading and tailing whitespace characters from a value.

Split

The split processor expects the following configuration:

{
    "splitOn": "<value>"
}

The option splitOn specifies the text on which the value should be split on.

Example: Given a field contains multiple values separated by a ; like

"A;B;C"

the configuration would look like this:

{
    "splitOn": ";"
}

The processor would output the value as

["A", "B", "C"]

Replace Value

The replace value processor expects the following configuration

{
    "values": {
       "<field value>": "<replace with>",
       "<other value>": "<replacement>"
    }
}

The option values contains key-value pairs where the key is the value which should get replaced and the value is the replacement.

If the value is set to null, the field will not get added to the index.

{
    "values": {
       "Do Not Include Me": null
    }
}

If the value does not match any configured replacement, the value will not get replaced.

The keys are case-sensitive.

Measure and Analyze

This tab allows to get a look into the data which gets produced by the provided configuration. This allows to quickly iterate over the configuration without having to save it first and waiting for the crawler to catch up to the changes.

Performing the Measure Now will fetch the first 100 elements from the source and perform all mapping operations. As a result, the operation produces a JSON for each mapped item. At the bottom of the page, these mapped results are displayed. Additionally, the duration for the operations is measured and displayed.

Keeping the duration small ensures that the load on the source is minimized and updates are published quickly to the search index. The durations are not proportional to normal search operations performed by the end user.

Items appearing within the search result

After having the Sources configured, the appearance of all configured classes can be changed in the Searching tab.

Summary field

By default, an item will display the Name and the Description field within the search result. For items which contain long texts, a summary text can be generated. The summary will detect important sentences based on word heuristics and word frequencies and will take the most relevant sentences to generate the summary itself.

By adding one or more fields to the Summary Field, the summary will be based on these fields combined. You should only add the main text field from the mapping and multiple fields containing lookup values etc.

The summary quality is highly dependent on the quality of the text to summarize. The summary generation performs optimally when the text consists primarily of paragraphs and sentences, without excessive boilerplate content or complex visual structures. Additionally, the text should be free of HTML tags for best results.

Note: The text used will be the content from within the search index. Therefore, any value processors previously applied to these fields may affect the results.

If no field is specified or all specified fields are empty, the configured description field from within the Loading and Display tab will be used instead.

Loading and Display

Item Loading

This option specifies where the data for displaying an item will come from.

From Source is the default selection. For every item to display, the platform will fetch the latest information from the source itself before sending the results to the user. This ensures that the latest information is always displayed for the user. This option has a performance penalty because each displayed item will get loaded first.
When using From Search Index, each matched item will be sent to the user directly without consolidating the source first. This option is highly recommended for slow sources. A slow source might block an available search result for the end user for a long time which reduces the user experience. The downside of this option is that it is not guaranteed that the information displayed is the latest one. For most cases however, this tradeoff might be worth for the performance gain.

Display

This section allows to specify which field value should be displayed where within the search result. When using the From Source option, the setting Automatic will be available. This option will fill the fields depending on the specified internal keys of a field from within the class designer. For each placeholder, you can select one field as the source for this information. If the option From Source is selected, each field is added twice to the list of possible fields - one for the field value from the data source and one for the field value from the search index.

Remarks: If a field from the search index is selected, the displayed value will be the value which comes out of the value processor for that field.

Additional Attributes

In addition to the placeholder areas, one or more attributes can be added. These attributes will be displayed between the Title and the Description placeholder.

Platform Administrator

General

The advanced search feature requires a dedicated search index. This index is used to perform search operations and is maintained within a dedicated search service.

The size of each index depends on two main factors:

The size of the original data source
The number of fields saved within it In many cases, the index size can be approximately equal to the size of the underlying data source.

To ensure the index remains current, the platform continuously indexes items in the background.

Disabling

The search feature can be disabled globally without affecting other parts of the platform. To do that, open the appsettings.json within the installation folder and remove the configuration with the key:

"Modules:ExploreApp:ServiceUri": "http://<some host>:<some port>/,"

After a platform restart, the advanced search will not be available anymore.

Note: If the feature is disabled this way, all UI elements for the search will not be available anymore.

General Administration

The platform administrator has a dedicated page within the Admin Center which allows monitoring and managing the search feature.

To open this page, go to the Admin Center, navigate to the Misc tab, and click on the Search Administration action.

Service Health and Service Logs

These tabs allow monitoring of the current health state of the dedicated search service.

Crawler Health

This tab provides an overview of the current state of the crawler itself. The crawler is responsible for sending updates to the search service to keep the search index up to date.

Crawler Logs

The crawler records specific events in the standard log file location. This tab also offers a quick view of the most recent log entries.

Note: Only the latest log statements are displayed here. A platform restart will clear the list of available log entries.

Availability

The Explore App operates on an opt-in basis. By default, it is not available for any workspace. Workspaces must be explicitly marked to use the Explore App, which can be done in this section.

When a workspace is marked as eligible:

The Explore App is enabled for all users in that workspace.
Users will see the Explore App after refreshing the web app or logging in.

Note: The first time a workspace is marked eligible, it requires initial setup. Please refer to the Workspace Administrator section for details on this process.

If a workspace is removed from the list of eligible workspaces:

The Explore App will no longer be visible to users of that workspace.
The workspace will automatically remove itself from the crawler.

Important: Removing a workspace from the eligible list does not delete its search index. To remove the index data, you must either:

Manually invoke the corresponding API of the search service, or
Delete the workspace entirely.

Index Statistics

An initialized workspace will display the Index Statistics tab. This tab will display various information about each created search index for that workspace. The information will include the last time the index was updated, the total size, and the number of items within the index.

Timing Statistics

An initialized workspace will display the Timing Statistics tab. This tab will display various timing information about each performed search operation for that workspace. This includes the number of executions and min, max, and average time of various stages for a search operation.

Troubleshooting

If, for some reason, the search index is not filled correctly or is out of sync, the index can be forcefully re-created. A complete re-index will iterate over all available items and send them to the search service. The reindexing is an iterative operation, meaning that the index will be re-created while keeping the current content available for new searches. This allows this operation to be performed while keeping the search feature available for the users.

Like the normal crawling operations, the re-indexing happens in the background. The number of parallel re-indexing operations is limited by the number of available CPU cores. The number of parallel executions is 0.33 * [CPU core count]. Depending on the current amount of work to do for the crawler, it can take a while until the actual operation starts.

To start a reindex, go to Search Administration -> Workspaces -> [The workspace the reindex should happen in] -> Index Statistics. On the right-hand side of every index is a small reindex action.

Note: You cannot schedule a reindex for a user source. In this case, the reindex action will not be visible.

Note: The list of pending reindex operations is not persisted. If a long list of reindex operations is requested and the platform gets restarted, the reindex operation will not resume automatically.

Note: For information on initializing a workspace, please refer to the Workspace Administrator section

PreviousMongoDB Adapter NextIntegration into an App

Last updated 5 months ago