Muse Proxy End User Interfaces

The Muse Proxy applications offer rich and full responsive interfaces for end users, implemented with new technologies like Bootstrap, jQuery and AngularJS.
The template applications shipped with Muse Proxy - identifier MKPF, has built-in 3 interface layouts. The main difference between them is the way the sources information is displayed.

  1. The Default, List Layout

  2. The Flip Cards Layout

  3. The Drop-Down Cards Layout

All three interface layouts offer the same features:

  • Sources display by different methods/criteria: alphabetical, flat, by subject or by type criteria. When selecting one of Flat View, Subject View or Type View displays, which involve groups, another display option is available, that shows the items of the same group in a bordered zone/cell.
  • Selection of the number of items to display per page, available only for the Flat View.
  • Sort options: by source names ascending/descending.
  • Filter/search box to display only the sources names containing the entered text criteria.
  • Tools page with the following functionalities, very useful for handling publisher resources URLs: URL Encode and Prefix, Un-Rewrite URL and Prefix and MuseKnowledge Proxy Target URL.

 

Tools Page (click on image to enlarge)

All application's features cand be enabled/disabled. Also default values cand be set, like the number of items to display per page and the sorting.

The interface layouts are detailed below:

1. The default, list type, which is the out of the box layout. Below are images with the layout display for each available view: Flat View, Alphabetical View, Subject View, Type View.

 

Flat View (click on image to enlarge)
Type View (click on image to enlarge)
Alphabetical View (click on image to enlarge)
Subject View (click on image to enlarge)

2. A second display layout in which the sources are displayed as cards/tiles. A featured image is displayed at the beginning of the page, followed by the sources listing. When clicking a card/tile, it flips revealing the source description.

Below is a full image with this Muse Proxy Application sources display layout.

 

Flip Cards/Tiles Layout (click on image to enlarge)

The same views are available for this layout as well: Flat View, Alphabetical View, Subject View, Type View.

3. A third display layout in which the sources are displayed as cards/tiles, but without the fliping effect. The source description is expanding below the tile. A featured image is displayed at the beginning of the page, followed by the sources listing. A full image is available below.

 

Drop Down Cards/Tiles Layout (click on image to enlarge)

In Muse Proxy version 5.5, the application layout can be set by editing the applicationLAF property from configuration file ${WEB_CONTEXT_HOME}/www/application.inc.

Starting with the next Muse Proxy version, this configuration cand be made from the Muse Proxy Administrator Console.

Overview

 

eLearning is widely regarded as mainstream nowadays, it is a service offered by most colleges and universities and the core of many business plans. The main tool for eLearning is the Learning Management System (LMS), which is a software application or web-based technology used to plan, implement and assess a specific learning process.

Some popular LMSes used by educational institutions include Moodle, Blackboard, Atutor, Sakai and Canvas.

Learning Tools Interoperability (LTI) is a standard developed by the IMS Global Learning Consortium and it specifies a method for a learning system to invoke and to communicate with external systems. A LMS may use LTI to host course content and tools provided by external systems on a web site, without requiring a learner to log in separately on the external systems, with information about the learner and the learning context shared by the LMS with the external systems.

LTI launch requests can be used to include a Muse Proxy Application or proxified source directly within the Learning Management System (LMS) as an External Tool. This mechanism enables teachers to enhence the lessons by adding online content from the institution's electronic subscriptions.

Once a user is authenticated to the LMS (s)he can access a Muse Proxy Application or source being defined as an External Tool. Muse Proxy acts as a Tool Provider.

This article describes the integration of publisher subscribed content into LMS via Muse Proxy, based on LTI. For exemplification we used the Canvas LMS, a self hosted demonstration instance.

Muse Proxy supports LTI 1.3 Authentication and the legacy LTI 1.0, as a Tool. The configuration of LTI in Muse Proxy is not covered here, the complete instructions are available in the Muse Proxy Administrator Console, Configuration -> LTI 1.3 Authentication menu.

Settings to be done in Canvas

 

After finalizing the LTI configuration in Muse Proxy, an URL of the below form must be saved, to be used in Canvas:

https://proxy.domain.com/ssoRWP3/config/{ApplicationID}/instance/0/canvas.json

where {ApplicationID} must be replaced with the actual identifier of the Muse Proxy Application.

Using this JSON URL, the administrator of the institution must create a new LTI key in Canvas. This is done from Admin > Site Admin > Developer Keys > + Developer Key > + LTI Key.

Under Configure > Method, choose Enter URL, and paste the Canvas Config URL from the previous step into the JSON URL field. Then click Save.

After clicking Save, on success the key is created and the interface returns to the key table.

Go to another page and return to Developer Keys, otherwise the Edit section might be empty. Then you should chose the recently added key and edit it to add a Key Name, Owner email and eventually Notes - these fields are there for future admin reference - there are no rules on how to fill in these. If, for some reason, such as SSL certificates errors the connection to the JSON URL cannot be realised, then choose Paste JSON, obtain the above JSON in another browser tab and copy and paste it there.

Then, back to the Developer Key table, change the state of the Tool to from OFF to ON. Also, make it visible by pressing the eye icon. After ticking the state label and the eye icon to change the state and the visibility do not use the Edit this key section directly.

Get the value for the client-id from the Developers Keys table from the Details column. It is the value directly visible in the table (not the one under Show Key). Modify the lti.properties from Muse Proxy to have the correct client-id value instead of the placeholder value, for example:
lti.clients.MuseProxyFoundation.definitions[0].client-id=10000000000043

To make available the Muse Proxy tool in Canvas go to the Account/Sub-account/Course, and click Settings > Apps > + App. Choose By Client ID as configuration type and enter the Client ID of the Developer Key created above.

Canvas Placements

 

The Placements available for Muse Proxy tools require new window, due to all the browser security. However, for selecting and embedding a link (Deep Linking), the Muse Proxy Application is still launched inside an iframe so make sure the corresponding iframe related security HTTP headers are correctly set/unset. Also, make sure the browser used for embedding links is configured to support third-party cookies (Safari does not have an explicit option for this, rather deselect "Prevent cross-site tracking").

  • Account Navigation - this will automatically open the Muse Proxy Application in a new window from the navigation menu of the account.
  • Course Navigation - this will automatically open the application in a new window, from the navigation menu of the course.
  • Editor Button - This allows for the Muse Proxy Application tool to be invoked in Rich Content Editors in order to insert a proxified source. This requires the Muse Proxy Application to be embedded as an iframe for selecting the resource. Then, the end-user, when clicking in the content of the page will have the proxy source always opened in a new page. Do not alter the Muse Proxy URL after adding it. Not all the parameters are visible in Canvas.

    After clicking the button:
    After selecting a Muse Proxy source from the Muse Proxy Application:
  • Link Selection - This makes possible to add direct proxified sources in modules. After selecting the Muse Proxy source, always check the Load in a new tab. Do not alter the Muse Proxy URL after adding it, because it is based on Deep Linking and parameters are managed behind by the platform. Embedd a Link Resource from the Muse Proxy External Tool:


    And the link listing in the module:

Conclusion

 

The LTI integration in Muse Proxy offers a powerfull extension for LMSes and enables the content creators (e.g. teachers) to add unlimited resources from the institutional subscriptions to their learning content (courses, modules, etc.).

The MuseKnowledge Application embeds speech technology for the following features:

  1. Speech to text, which allows users to insert text into the inputs of the application, like the main search form, just by speaking to a microphone. The spoken text is recognized and it is automatically written in the selected input.
  2. Text to speech, which allows the results metadata to be read into natural-sounding speech.

The speech functions in Muse are currently implemented using the Microsoft Cognitive Services, for which subscription is required. To get started, a new resource must be created into the Microsoft Azure Portal, of type Cognitive Services - Speech. All API requests to Azure services require an endpoint URL and a read-only key for authenticating access.

Documentation about the Azure Speech Service with information for creating the service is available here:

https://learn.microsoft.com/en-us/azure/cognitive-services/speech-service/

When done creating the Speech resource, copy the API Keys and region details to be configured in Muse. The API Keys and URLs are available in the Keys and Endpoint section in the Azure Portal:

Speech to Text

Muse Speech to Text is available starting with MuseKnowledge Application version 7.4, release date October 2019.

Configurations

The Speech to Text functionality should be already enabled in the MuseKnowledge Application, a microphone icon should be visible in all application search inputs. If the microphone symbol is not displayed in the search inputs, the following configurations must be checked:

  1. Access the Muse Console for Applications Administration (MCAA), select the Muse Application and access the Application General Settings left menu item, Interface OptionsFunctionality tab and make sure Enable Speech to Text is enabled and the selection integration method is Microsoft Azure.
  2. Edit the ${MUSE_HOME}/web/www/logon/${APPLICATION_ID}/lib/microsoft/microsoft.cognitiveservices.speech.sdk.jsp file on disk, and fill in the API key and region values in the subscriptionKey and region variables.

Text to Speech

Muse Text to Speech is available starting with MuseKnowledge Application version 8.6, release date February 2023.

More details about Muse Text to Speech are available in the MuseKnowledge™ Application 8.6 Release article.

Configurations

The Text to Speech functionality should be already enabled in the MuseKnowledge Application, a Play button should be visible in the results page, for each result, in footer section of the result.  If not, check in the Search Options panel that the Show Speech Controls flag is set to Yes. If there is no Show Speech Controls flag, the following items must be verified, enabled and configured:

  1. Access the Muse Console for Applications Administration (MCAA), select the Muse Application and access the Application General Settings left menu item, Interface OptionsFunctionality tab and make sure Enable Text to Speech is enabled and the selection integration method is Microsoft Azure.
  2. Edit the ${MUSE_HOME}/web/www/logon/${APPLICATION_ID}/lib/microsoft/microsoft.cognitiveservices.speech.sdk.jsp file on disk, and fill in the API key and region values in the subscriptionKey and region variables.
  3. Check that the DetectLanguage Muse Source Package (SP) is installed in the application and configured with the MS Azure Translator API key. Edit its configuration in MCAA, (select the Source Package and access Edit Source Advanced Configuration) and in the Custom Parameters place the API key as obtained from the Azure portal as value for the API_KEY metavariable.

Access the below links to view recordings of this functionality, one taken while in the English interface of the MuseKnowledge Application and for an English result, and the second taken while in the Arabic translated interface, for an Arabic result.

English Result Speech

Arabic Result Speech

The following elements that use text translations into a specific language are available in the MuseKnowledge Application:

  1. Query Translation
  2. Search Result Translation

All text translation functions in Muse are currently implemented using the Microsoft Cognitive Services. Thus the requirement is to have a Microsoft Azure Subscription in which to create a Translator service which will be used by the Muse translation features. (see below more details for setting up a Microsoft Translator Service)

Azure Translator is a cloud-based machine translation service that is part of the Azure Cognitive Services family of REST APIs. Azure resources are instances of services that you create. All API requests to Azure services require an endpoint URL and a read-only key for authenticating access.

A tutorial for creating a Translator resource in the Azure portal is available here:

https://learn.microsoft.com/en-us/azure/cognitive-services/translator/create-translator-resource

When done creating the Translator resource, copy the API Keys and Endpoint values to be configured in Muse. The API Keys and URLs are available in the Keys and Endpoint section in the Azure Portal:

Query Translation

Muse Query Translation is available starting with MuseKnowledge Application version 7.2, release date June 2019.

More details about Muse Query Translation are available in the MuseKnowledge™ Application 8.0 article.

Configurations

The Query Translation functionality should be already enabled in the MuseKnowledge Application. To ensure the proper configuration, the following items must be verified, enabled and configured if not already:

  1. Access the Muse Console for Applications Administration (MCAA), select the Muse Application and access the Application General Settings left menu item, Interface Options, Functionality tab and make sure the Enable Query Translation flag is set to yes.
  2. Check that the TranslatedQueriesMS Muse Source Package (SP) is installed in the application and configured with the MS Azure Translator API key. Edit its configuration in MCAA, (select the Source Package and access Edit Source Advanced Configuration) and in the Custom Parameters place the API key as obtained from the Azure portal as value for the API_KEY metavariable.
  3. Check that the TranslatedQueriesMS Muse Source Package is part of a Sources Group, by accessing in MCAA the Organize Sources -> Source Grouping section. If not, create a new group in which add the TranslatedQueriesMS SP.
  4. Check that the Source Group from the previous point #3 is of type Query Translation. Access in MCAA the Organize Sources -> Update Interface section and in the groups listing select as Group Type the Query Translation option.
  5. Test the TranslatedQueriesMS SP in MCAA in the Test Source(s) section with a query in a different language, other than English which is the default language.

Search Result Translation

The Search Result Translation functionality is available starting with MuseKnowledge Application version 8.4, release date June 2022.

More details about Search Result Translation are available in the Muse Release 2.8.0.0 news article.

Configurations

Muse Search Result Translation should be already enabled in the MuseKnowledge Application. To ensure the proper configuration, the following items must be verified, enabled and configured if not already:

  1. Access the Muse Console for Applications Administration (MCAA), select the Muse Application and access the Application General Settings left menu item, Interface Options, Functionality tab and make sure the Enable Record Translation flag is set to yes.
  2. In MCAA, select the Muse Search Application and access the Application Modules -> Translator Keys left menu. Make sure the MicrosoftTranslator key is installed, edit its configuration file (
    Translator Module: MicrosoftTranslator Properties: Configuration File) and in the Connection Parameters section, API Key input add the API key as obtained from the Azure portal.

A Live Chat is available in the MuseKnowledge Search Application starting with the upcoming version 8.1. It allows end-users to connect in real time via chat with a Library representative (operator) for online support about how to use MuseKnowledge Search Application and its search resources.

End User’s Perspective

The chat is available for end users in the MuseKnowledge Search Application in the lower right corner and it can have different appearances, depending on the Live Chat Platform integrated. Below is a screenshot with the initial form of the chat box and the next forms after filling in the required details and starting the conversation.

Administrator’s Perspective

Many Live Chat Platforms can be integrated into the MuseKnowledge Search Application. The following platforms were tested successfully: Live Helper ChatLiveChat and Pure Chat.

1. Customers without an existing live chat solution

For customers without an existing Live Chat Solution, either self hosted or subscribed, we can offer this service in our hosted Live Chat Platform. We will provide you operator account(s) in our hosted chat platform and take care of the necessary configurations in your MuseKnowledge Search Application for enabling the Live Chat functionality.

2. Customers with an existing live chat solution

The customers having a subscription for a commercial chat solution, or hosting one themselves, can enable the Live Chat functionality by following the below instructions. Minimal knowledge on JavaScript, HTML is needed; if the integration does not succeed please contact the MuseGlobal Support team for help.

  1. Obtain the JavaScript code from your Live Chat Platform which needs to be integrated into the application. Usually this is obtained from an administration console of your chat platform or FAQ item. For more details you should address to the support team of your Live Chat Platform.
  2. Access the MuseKnowledge Console for Applications Administration of your installation, e.g. https://yourdomain/mmc . Select the application for which to enable Live Chat functionality from the applications list and access the Application General Settings -> Interface Options left menu item.
  3. In the new page access the Functionality tab and set to yes the Enable Chat: option.
  4. Replace the existing code from the revealed text area with the below code in which replace the placeholders (e.g. {js_chat_object}{html_chat_object}, etc.) with the appropriate values corresponding to your chat JS or HTML code content. Note that it is mandatory to have the "init" and "destroy" functions for chat object initialization and removal, as these are wired from the application code:
    app.constant("chat", {
        "init": function(language){
    
            // If the chat object is already initialized, then exit, nothing else to do.
            // Example:
            if (window.{js_Chat_Object}){
                return;
            }   
       
         // Paste here the JS code specific to your chat platform, obtained as explained above.
        },
        "destroy": function(languageChanged){
            // If to apply a new chat language destroy is not needed, just return when languageChanged == true.
    
            // If the chat object does not exist, then exit, nothing else to do.
            // Example:
            if (!window.{js_Chat_Object}){
                return;
            }
    
            // Remove all JS/HTML chat objects/elements unless your chat has its own unloading method (which must be used in this case).
            // Examples:
            window.{js_Chat_Object}" = undefined;
            angular.element("{html_chat_object}").remove();
        }
    });

 

Following its development trail, the MuseKnowledge™ Application reached version 8.0, which brings all improvements and new functionalities that were implemented over the past 3 years.

Here are some of the new features of our flagship search application, grouped by their location.

Search Form:

  • Query Suggestion with type-ahead features. The suggestions are presented to the end-user as he is typing, at any moment the user can select any of them from the list and launch the search. The suggestions are driven from the set of queries used by all users of the Muse system.
    Query Suggestions
  • Spellcheck / "Did you mean" functionality for the end-user's query. After the end-user launches the search, if there are misspellings in the query, a corrected one is displayed below the search form with a "Did you mean:" label. Clicking on the suggested query will launch the search with the new terms. This feature is based on indexed dictionaries per language, with extended capabilities such as words family, e.g. singular, plural, infinitive, participles, past tense, and checks queries against common spellings of each word. In addition, it is possible to use as data source besides dictionaries, the entire set of queries used in the Muse system. This is most useful when specialized query terms are being used which are not found in dictionaries.
    Spellcheck from a single data source, e.g. dictionary
    Spellcheck from single data source
    Spellcheck from a two data sources, e.g. dictionary and user queries
    Spellcheck with multiple data sources
  • Translation of the query. When the language of the query entered into the search form is detected as being different than application's active interface language (or a different configured language), the query is translated at the end-user's disposal into the referred language so that it can be available for launching a new search. For example, while in the MuseKnowledge Search Application English interface and entering an Arabic query, the end-user is offered with the possibility of translating the Arabic query into English and to use the resulted translation for a new search.
    This is most useful for non-English end-user speakers while searching with non-English terms against resources that have content only in English and which usually retrieve no results. With this new feature they can search using the translated query, thus increasing the possibility of retrieving results.
    For example, if the current language of the Muse Search application interface is English, and the user enters an Arabic query (or a query in any other language than English), then the Query Translation functionality is offered for the English language:
    Query Translate
    and upon end-user action of clicking on the link, the translation appears:
    Query Translate
  • Speech to Text functionality feature allows users to insert text into the inputs of the application, like the main search form, just by speaking to a microphone. The spoken text is recognized and it is automatically written in the selected input. This feature is only available on desktop computers (not phones or tablets, which already have this native functionality) and for a small set of inputs, only the ones where users can enter free text, such as search, filtering or help terms.
  • Virtual Keyboard functionality which allows to input free text without the need of physical keys. It can be used to enter login credentials or search terms and has a layout for each language of the application interface.
    Virtual Keyboard

Home and Search Results Page:

  • Multiple widgets are available for the application's Home page to offer increased functionality for the end-users. All widgets are configurable, they can be enabled/disabled through the Muse Console for Applications Administration.
    • Titles A to Z. The lists with the subscribed titles (books and journals) are ingested into the Muse Central Index and made available for browsing and searches.
    • Publishers. The list of subscribed publishers and their products are nicely displayed in tiles with publisher logos and descriptions. Direct links are available to navigate to publisher platforms using the institutional customer subscription.
      Publishers
    • RSS feeds. A RSS feed can be configured to display items in a widget in the application's Home page.
    • Important Links. External links with accompanying descriptions can be configured into this widget.
    • New: Books and Journals. Static metadata displaying a list of subscribed journals/books to promote for the end-users.
    • Top 10 Most Accessed Titles. This widgets presents the top 10 most accessed titles from the Titles A to Z widget.
  • Implemented the possibility to switch between a Simpler interface view with light functionality and a Richer one with the most complete set of application features that were enabled. A default view (Simpler or Richer) can be set per each application, switching between the views can be done from the "Search Options" panel, "Display Options" section, "View Display" option.
  • "Refine your Results" extended its visual representation with vertical bars for the "Dates" faceting option. The filtering action can be triggered by clicking a specific vertical bar corresponding to the desired date value. Also a slider is available for narrowing the dates for filtering the current result set based on the selected interval. Furthermore, the "Dates" filtering options were enhanced with new values: "Last Month", "Last 6 Months", "Last Year.
    Date Facet
  • The records images are also displayed in a separate widget called Images. It is available in the main area, just after the results and pagination display. The new Images widget presents in an images carousel all images from all retrieved records, clicking on any image will open the image viewer where various actions can be performed, like zoom in/out, move, rotate and share on social networks.
    Images Widget
  • "My List" functionality, which allows to save selected results from multiple searches into a temporarily list for later handling. The results saved into "My List" are available for the current session and they can be used with the existing actions, such as email, saving them into various formats, etc.

Muse Console for Applications Administration (MCAA):

  • All application's new features are configurable through the Muse Administration Console. The application's look and feel can also be customized by writing custom HTML code for header, footer and login page, adding custom CSS. Images such as customer logos can be uploaded on the server and referred into the custom code.
  • The application's widgets can be arranged by drag and drop.
  • The main functionalities can be enabled/disabled per each display type, e.g Extra Small Devices, Small Devices, Medium Devices, Large Devices.

As usual we offer a Free 30 Day Trial period to fully evaluate the new version of MuseKnowledge™ Application. You can request a trial access by simply filling in the trial form.

Configure the Muse Proxy service with NGINX front-end

NGINX is an HTTP and reverse proxy server that can be placed in front of web applications to achieve custom workflows and provide solutions where multiple web applications cannot co-exist with similar specifications.

Specifically the NGINX server can be configured as front-end for Muse Proxy when Muse Proxy cannot be configured to use the standard ports – 80 for HTTP and 443 for HTTPS access. Because there are other applications needing the standard ports as well. Only for this situation, when multiple web applications are hosted on the same server it is recommended to use NGINX, otherwise if Muse Proxy is the only hosted service, it must be accessed directly, without a front-end reverse proxy solution.

Hosting multiple web applications with NGINX as front-end reverse proxy requires assigning a dedicated fully qualified domain name (FQDN) for each application.

In this article we are describing the necessary configurations for NGINX and Muse Proxy only. It is assumed that NGINX and Muse Proxy are already installed, this article does not cover their installations.

Further assumptions:

  1. To exemplify, we consider the Muse Proxy assigned FQDN to be proxy.mydomain.com. And the server IP – 192.168.1.100. The server OS is a Linux distribution (Debian, Ubuntu, RedHat, CentOS, etc).
  2. Muse Proxy is configured with its default ports – 9797 for HTTP and 9443 for HTTPS
  3. The wildcard DNS record is enabled for the Muse Proxy assigned FQDN – *.proxy.mydomain.com.
  4. A wildcard SSL certificate is available, thus the certificate and the private key exist on the server. The certificate must contain as SAN entries both *.proxy.mydomain.com and proxy.mydomain.com.

Configure the Muse Proxy service with NGINX front-end

In the NGINX configuration file for the Muse Proxy FQDN (e.g. /etc/nginx/sites-available/proxy.mydomain.com) define the backend service:
upstream backend_proxy {
        server 192.168.1.100:9797;
}

For each port (80 and 443) contexts make sure the server_name has both values, e.g. *.proxy.mydomain.com and proxy.mydomain.com. Sample entries:

server {
        listen 80;
        listen [::]:80;
        server_name *.proxy.mydomain.com proxy.mydomain.com;
        location / {
                include proxy_params;
                proxy_pass http://backend_proxy;
                proxy_buffer_size 128k;
                proxy_buffers 4 256k;
                proxy_busy_buffers_size 256k;
        }
}
server {
        server_name *.proxy.mydomain.com proxy.mydomain.com;
        listen 443 ssl;
        ssl_certificate /etc/ssl/proxy.mydomain.com/fullchain.pem;
        ssl_certificate_key /etc/ssl/proxy.mydomain.com/privkey.pem;
        location / {
                include proxy_params;
                proxy_pass http://backend_proxy;
                proxy_buffer_size 128k;
                proxy_buffers 4 256k;
                proxy_busy_buffers_size 256k;
        }
}

The content of /etc/nginx/proxy_params refered in the above samples:

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Increase the timeout value for NGINX in /etc/nginx/nginx.conf, e.g.:

proxy_read_timeout 600s;

The NGINX service must be reloaded to pick up the configuration changes.

Configurations for Muse Proxy

Some configurations are needed for Muse Proxy as well, in its main configuration file ${MUSE_HOME}/proxy/MuseProxy.xml:
Define the external ports of NGINX as follows:

<PORT enabled="true" external="80">9797</PORT>
<SSL_PORT enabled="true" external="443">9443</SSL_PORT>

Specify the allowed IPs (of the machine itself) that allowed to provide the X-FORWARDED-FOR header value:

<ALLOW_X_FORWARDED_FOR>192.168.1.100</ALLOW_X_FORWARDED_FOR>

Make sure the FQDN value is configured in the SERVER_NAMES field, e.g.:

<SERVER_NAMES>proxy.mydomain.com</SERVER_NAMES>

In this article we are revealing some of the new features introduced in the MuseSearch™ Application (version 4.0) that will be available in the next release of the Muse™ Platform.

The latest MuseSearch™ Application (version 4.0) can be also deployed and run in the current version of the Muse™ Platform (2.7.0.0), however this can be done only with the help of Muse™ Technical Support since it implies applying system patches.

MuseSearch™ is the current production template application used by end-users for accessing the Muse™ Federated Search Platform.

Security Related Improvements

We updated the Freemarker library to the latest version available – 2.3.23. The latest Freemarker version brings many security improvements and bug fixes. The most important ones are related to the mechanism of loading the templates, which now enforces more security checks and prevents malicious attempts to get unauthorized file content or run arbitrary JavaScript code.

A major improvement in the MuseSearch™ Application structure brought by the upgrade to the latest Freemarker version is the decrease of the number of files on disk. Because the latest Freemarker permits serving the template files from multiple repositories we removed the duplicate files from all languages folders.

New User Interface Features

  • Expert search page. An expert search page was introduced to allow the end-user construct complex queries:

    The query can be visually built step by step using the query builder tool or it can be manually written in the Raw Query section:
  • Faceting on the searched sources. The filtering on the searched sources was moved as a faceting feature into a separate side panel:
  • New facets were introduced: Authors and Years:
  • New side panels were added to display results from specially content sources, such as dictionary, Wikipedia, news, whether, etc. Only a configurable number of records is displayed, default is one record:
  • Related Queries were added into the MuseSearch™ Application to display in a side panel all queries that are related to the one entered by the end user. The check is done against the queries that were submitted in time by the end-users of that Muse™ system.
    Behind the scene there is a complex mechanism implemented with Muse™ Control Center dedicated tasks that mine the ICE Statistical log files for queries and index them in the Muse™ Central Index component. At search time the index is being checked and the matched related queries are displayed:
  • Query Highlight feature means that the search term is now highlighted in the retrieved records in the Muse™ application interface anywhere it is found:

In this current article we are describing a new feature introduced with Muse Proxy 4.1: the ability to perform actions transparently, behind the scene. For example change the default language in the remote website, get the end-user in different sections of the website like an advanced search form, etc.

As example we considered the Ted website (http://ted.europa.eu) which offers a Username/Password authentication type. The implementation request for profiling this source in Muse Proxy is to perform the Username/Password authentication, change the language to “English”, then select “UK – United Kingdom” , then select “Contract notice” to display all contract notices available for “United Kingdom”.

The Username/Password authentication profiling is not presented in this current article, please refer to the “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario” article for a profiling example.

The native webpage after the authentication steps is as below:

Select “English” to change the language:

Select “UK – United Kingdom”, then select “Contract notice” to display all contract notices available for “United Kingdom”.

The steps for profiling the above scenario are the following:

  1. Create a new Muse Proxy Source profile named EuropaTED_eu_UP.xml.
  2. Create the rules for the authentication Username/Password process. Because the authentication process for the Ted website is complex as it is based on dynamic parameters and values, we are making use of the Extract and Navigate functionality. For more details on this please refer to this article “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario”.

    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&submit=Login%21]]>
    </POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in
    to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
  3. Select the English language:
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
  4. Click on the UK – United Kingdom link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
  5. Click on the Contract notice link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
  6. The complete XML profile is given below:
    <ICE-CONFIG>
    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&
    submit=Login%21]]></POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
    <SHOW_GET_PARAMETERS>true</SHOW_GET_PARAMETERS>
    <USER_AGENT>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36</USER_AGENT>
    <CONNECT_TIMEOUT>60000</CONNECT_TIMEOUT>
    <READ_TIMEOUT>60000</READ_TIMEOUT>
    <NAME>TED Tenders Electronic Daily</NAME>
    <DESCRIPTION>TED provides free access to business opportunities. It is updated 5 times a week with some 1,500 public
    procurement notices from the European Union, the European Economic Area and beyond. You can browse, search and sort procurement
    notices by country, region, business sector and more. Information about every procurement document is published in the 24 official
    EU languages. All notices from the EU's institutions are published in full in these languages.</DESCRIPTION>
    <AUTHENTICATION_TYPE>User/Password</AUTHENTICATION_TYPE>
    <HTTP_AUTHORIZATION_USER_NAME/>
    <HTTP_AUTHORIZATION_USER_PASSWORD/>
    <HTTP_AUTHORIZATION_SCHEME/>
    <PROXY_USED>SOURCE_LEVEL</PROXY_USED>
    <PROXY_HOST></PROXY_HOST>
    <PROXY_PORT></PROXY_PORT>
    <PROXY_PAC></PROXY_PAC>
    <PROXY_AUTHORIZATION_USER_NAME/>
    <PROXY_AUTHORIZATION_USER_PASSWORD/>
    <PROXY_AUTHORIZATION_SCHEME/>
    <REWRITING_PATTERNS>*europa.eu;*/*.js*;</REWRITING_PATTERNS>
    <REWRITE_BY_HOST>true</REWRITE_BY_HOST>
    <ENCODING>UTF-8</ENCODING>
    <REFERER></REFERER>
    <COOKIES></COOKIES>
    <REPLACE_HOST/>
    <REPLACE_PATH></REPLACE_PATH>
    <SSL_CERTIFICATES/>
    <SSL_ALIASES/>
    <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS>
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>
    <LAST_UPDATED>2015-07-14</LAST_UPDATED>
    </ICE-CONFIG>

Username/Password authentication is the second most popular authentication type after IP authentication. There are many cases when content providers offer only Username/Password for the institutional subscriptions. The subscribers are starting to prefer Username/Password type of authentication too over IP due to IPs shortage, as it is becoming an issue to obtain new IPs.

In this article we describe a new feature of Muse Proxy 4.1: Extract and Navigate. This new feature is useful for implementing a Username/Password authentication flow for complex websites that use dynamic state information in the authentication process.

In the simplest case it suffices to provide in a single request the Username/Password values along with other CGI parameters requested by the native website.

But there are websites which store and make use of state information, a good example are the websites created using Microsoft’s ASP.NET technologies. In these cases it is not working to submit directly the authentication form with the static values, hidden fields values must be dynamically parsed from the current page and passed along with the static parameter values. Moreover, the dynamically parsed values must be URL encoded.

The Extract and Navigate feature available from Muse Proxy version 4.1 is based on sequences of regular expression EXTRACTORs, URLs and POST_PARAMETERSelements which must be specified in the order of execution. URL and POST_PARAMETERS can make use of parameters extracted by the previous EXTRACTORs as well as static parameters defined in the Source Profile via PARAMETERS.
The parameters used are specified through the construct of ${parameterName}.

The EXTRACTOR elements are containing JDK regular expression including capturing groups as described in the JDK documentation, for example, in “Summary of regular-expression constructs” [http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html]. Capturing groups can be created via parenthesis ( “(groupPattern)”) are normally referred through numbers via $1$2. But because we will be saving these captured groups in the global source parameters we need to differentiate between multiple extractors. The differentiation is done via the ref attribute of the EXTRACTOR element and a variable will become accessible as ${ref_group}, that is the ref attribute value concatenated with underline (_), concatenated with the desired number. If a variable that does not exist is referenced than it will be left literally as the variable, hence make sure a number greater than the number of groups is not used. Also make sure that each EXTRACTOR has a ref attribute and its value is unique at the source profile level.

To exemplify we considered the https://legal.com.tr/ website which has Username/Password authentication.
The page https://legal.com.tr/uye-girisi contains the logon form. We consider YYY and ZZZ the Username/Password values.

In the page source we see the logon HTML form with parameters (hidden fields, view state, control state):

We start the profiling process by creating a new Muse Proxy source XML from an existing one and changing the values for the following nodes with the appropriate values:

  • URL
  • NAME
  • DESCRIPTION
  • REWRITING_PATTERNS

E.g.

<URL>https://legal.com.tr/uye-girisi</URL>
<NAME>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri</NAME>
<DESCRIPTION>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri - E-Kitap / E-Dergi / Online Kitap /
Online Dergi / Bas&#305;l&#305; Yay&#305;n - Legal Yay&#305;nevi</DESCRIPTION>
<REWRITING_PATTERNS>*legal.com.tr;</REWRITING_PATTERNS>

We name the new Muse Proxy Source: Legal_tr_UP.xml . UP is the convention we are using for labeling a Username/Password authentication type profile, tr is the 2 letter language code of the website. It is not specified when the website content is English.
Next we define and profile the steps made in the authentication process:

  1. Get the page with the logon form (https://legal.com.tr/uye-girisi) and parse from it the dynamic parameters: __EVENTTARGET, __EVENTARGUMENT, __VIEWSTATE, __VIEWSTATEGENERATOR, __EVENTVALIDATION.
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS></POST_PARAMETERS>
    <EXTRACTOR ref="eventtarget" refProcess="urlEncode">type="hidden"\sname="__EVENTTARGET"\sid="__EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="eventargument" refProcess="urlEncode">type="hidden"\sname="__EVENTARGUMENT"\sid="__EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstate" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATE"\sid="__VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstategenerator" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATEGENERATOR"\sid="__VIEWSTATEGENERATOR"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="eventvalidation" refProcess="urlEncode">type="hidden"\sname="__EVENTVALIDATION"\sid="__EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR>

    Description: For each value to be parsed we define an EXTRACTOR field. In the ref attribute we specify the name of the extractor rule, name with which we will further refer the value. In the refProcess attribute we specify a processing method, in this case we want to URL encode it.
    The value of the EXTRACTOR field is the regular expression for matching the desired value.
  2. Use the values extracted above and along with the Username/Password and other static CGI parameters and values submit to achieve the authentication.
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS><![CDATA[__EVENTTARGET=${eventtarget_1}&__EVENTARGUMENT=${eventargument_1}&__VIEWSTATE=${viewstate_1}&
    __VIEWSTATEGENERATOR=${viewstategenerator_1}&__EVENTVALIDATION=${eventvalidation_1}&
    ctl00%24MainContent%24userlogin%24UserName=${userName}&ctl00%24MainContent%24userlogin%24Password=${userPassword}&
    ctl00%24MainContent%24userlogin%24Button1=Giri%C5%9F]]></POST_PARAMETERS>

    The ${userName} and ${userPassword} metavariables are declared as following:
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>

    The full content of the xml configuration file is below, it can be downloaded from the EduLib website too by clicking here
    <ICE-CONFIG>
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS></POST_PARAMETERS>
    <EXTRACTOR ref="eventtarget" refProcess="urlEncode">type="hidden"\sname="__EVENTTARGET"\sid="__EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="eventargument" refProcess="urlEncode">type="hidden"\sname="__EVENTARGUMENT"\sid="__EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstate" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATE"\sid="__VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstategenerator" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATEGENERATOR"\sid="__VIEWSTATEGENERATOR"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="eventvalidation" refProcess="urlEncode">type="hidden"\sname="__EVENTVALIDATION"\sid="__EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR><URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS><![CDATA[__EVENTTARGET=${eventtarget_1}&__EVENTARGUMENT=${eventargument_1}&__VIEWSTATE=${viewstate_1}&amp
    ;__VIEWSTATEGENERATOR=${viewstategenerator_1}&__EVENTVALIDATION=${eventvalidation_1}&
    ctl00%24MainContent%24userlogin%24UserName=${userName}&ctl00%24MainContent%24userlogin%24Password=${userPassword}&
    ctl00%24MainContent%24userlogin%24Button1=Giri%C5%9F]]></POST_PARAMETERS>
    <USER_AGENT>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36</USER_AGENT>
    <CONNECT_TIMEOUT>180000</CONNECT_TIMEOUT>
    <READ_TIMEOUT>180000</READ_TIMEOUT>
    <NAME>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri</NAME>
    <DESCRIPTION>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri - E-Kitap / E-Dergi /
    Online Kitap / Online Dergi / Bas&#305;l&#305; Yay&#305;n - Legal Yay&#305;nevi</DESCRIPTION>
    <AUTHENTICATION_TYPE>User/Password</AUTHENTICATION_TYPE>
    <HTTP_AUTHORIZATION_USER_NAME/>
    <HTTP_AUTHORIZATION_USER_PASSWORD/>
    <HTTP_AUTHORIZATION_SCHEME/>
    <PROXY_USED>SOURCE_LEVEL</PROXY_USED>
    <PROXY_HOST></PROXY_HOST>
    <PROXY_PORT></PROXY_PORT>
    <PROXY_PAC></PROXY_PAC>
    <PROXY_AUTHORIZATION_USER_NAME/>
    <PROXY_AUTHORIZATION_USER_PASSWORD/>
    <PROXY_AUTHORIZATION_SCHEME/>
    <REWRITING_PATTERNS>*legal.com.tr;</REWRITING_PATTERNS>
    <TRANSPARENT_CONTENT_PATTERNS>*/*.js*;</TRANSPARENT_CONTENT_PATTERNS>
    <ENCODING>UTF-8</ENCODING>
    <REFERER></REFERER>
    <COOKIES></COOKIES>
    <CUSTOM_HTTP_HEADERS/>
    <REPLACE_HOST/>
    <REPLACE_PATH></REPLACE_PATH>
    <SSL_CERTIFICATES/>
    <SSL_ALIASES/>
    <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS>
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>
    <LAST_UPDATED>2015-09-08</LAST_UPDATED>
    </ICE-CONFIG>

General observations:

  • There are many online tools for testing the regular expressions, use them to make sure that the defined extractor rules behave correctly;
  • If a defined extractor rule does not match, the flow stops and the last page is displayed;
  • At any time make sure that the source profile is XML well formed.