MuseKnowledge Live Chat

A Live Chat is available in the MuseKnowledge Search Application starting with the upcoming version 8.1. It allows end-users to connect in real time via chat with a Library representative (operator) for online support about how to use MuseKnowledge Search Application and its search resources.

End User’s Perspective

The chat is available for end users in the MuseKnowledge Search Application in the lower right corner and it can have different appearances, depending on the Live Chat Platform integrated. Below is a screenshot with the initial form of the chat box and the next forms after filling in the required details and starting the conversation.

Administrator’s Perspective

Many Live Chat Platforms can be integrated into the MuseKnowledge Search Application. The following platforms were tested successfully: Live Helper ChatLiveChat and Pure Chat.

1. Customers without an existing live chat solution

For customers without an existing Live Chat Solution, either self hosted or subscribed, we can offer this service in our hosted Live Chat Platform. We will provide you operator account(s) in our hosted chat platform and take care of the necessary configurations in your MuseKnowledge Search Application for enabling the Live Chat functionality.

2. Customers with an existing live chat solution

The customers having a subscription for a commercial chat solution, or hosting one themselves, can enable the Live Chat functionality by following the below instructions. Minimal knowledge on JavaScript, HTML is needed; if the integration does not succeed please contact the MuseGlobal Support team for help.

  1. Obtain the JavaScript code from your Live Chat Platform which needs to be integrated into the application. Usually this is obtained from an administration console of your chat platform or FAQ item. For more details you should address to the support team of your Live Chat Platform.
  2. Access the MuseKnowledge Console for Applications Administration of your installation, e.g. https://yourdomain/mmc . Select the application for which to enable Live Chat functionality from the applications list and access the Application General Settings -> Interface Options left menu item.
  3. In the new page access the Functionality tab and set to yes the Enable Chat: option.
  4. Replace the existing code from the revealed text area with the below code in which replace the placeholders (e.g. {js_chat_object}{html_chat_object}, etc.) with the appropriate values corresponding to your chat JS or HTML code content. Note that it is mandatory to have the "init" and "destroy" functions for chat object initialization and removal, as these are wired from the application code:
    app.constant("chat", {
        "init": function(language){
    
            // If the chat object is already initialized, then exit, nothing else to do.
            // Example:
            if (window.{js_Chat_Object}){
                return;
            }   
       
         // Paste here the JS code specific to your chat platform, obtained as explained above.
        },
        "destroy": function(languageChanged){
            // If to apply a new chat language destroy is not needed, just return when languageChanged == true.
    
            // If the chat object does not exist, then exit, nothing else to do.
            // Example:
            if (!window.{js_Chat_Object}){
                return;
            }
    
            // Remove all JS/HTML chat objects/elements unless your chat has its own unloading method (which must be used in this case).
            // Examples:
            window.{js_Chat_Object}" = undefined;
            angular.element("{html_chat_object}").remove();
        }
    });

 

Following its development trail, the MuseKnowledge™ Application reached version 8.0, which brings all improvements and new functionalities that were implemented over the past 3 years.

Here are some of the new features of our flagship search application, grouped by their location.

Search Form:

  • Query Suggestion with type-ahead features. The suggestions are presented to the end-user as he is typing, at any moment the user can select any of them from the list and launch the search. The suggestions are driven from the set of queries used by all users of the Muse system.
    Query Suggestions
  • Spellcheck / "Did you mean" functionality for the end-user's query. After the end-user launches the search, if there are misspellings in the query, a corrected one is displayed below the search form with a "Did you mean:" label. Clicking on the suggested query will launch the search with the new terms. This feature is based on indexed dictionaries per language, with extended capabilities such as words family, e.g. singular, plural, infinitive, participles, past tense, and checks queries against common spellings of each word. In addition, it is possible to use as data source besides dictionaries, the entire set of queries used in the Muse system. This is most useful when specialized query terms are being used which are not found in dictionaries.
    Spellcheck from a single data source, e.g. dictionary
    Spellcheck from single data source
    Spellcheck from a two data sources, e.g. dictionary and user queries
    Spellcheck with multiple data sources
  • Translation of the query. When the language of the query entered into the search form is detected as being different than application's active interface language (or a different configured language), the query is translated at the end-user's disposal into the referred language so that it can be available for launching a new search. For example, while in the MuseKnowledge Search Application English interface and entering an Arabic query, the end-user is offered with the possibility of translating the Arabic query into English and to use the resulted translation for a new search.
    This is most useful for non-English end-user speakers while searching with non-English terms against resources that have content only in English and which usually retrieve no results. With this new feature they can search using the translated query, thus increasing the possibility of retrieving results.
    For example, if the current language of the Muse Search application interface is English, and the user enters an Arabic query (or a query in any other language than English), then the Query Translation functionality is offered for the English language:
    Query Translate
    and upon end-user action of clicking on the link, the translation appears:
    Query Translate
  • Speech to Text functionality feature allows users to insert text into the inputs of the application, like the main search form, just by speaking to a microphone. The spoken text is recognized and it is automatically written in the selected input. This feature is only available on desktop computers (not phones or tablets, which already have this native functionality) and for a small set of inputs, only the ones where users can enter free text, such as search, filtering or help terms.
  • Virtual Keyboard functionality which allows to input free text without the need of physical keys. It can be used to enter login credentials or search terms and has a layout for each language of the application interface.
    Virtual Keyboard

Home and Search Results Page:

  • Multiple widgets are available for the application's Home page to offer increased functionality for the end-users. All widgets are configurable, they can be enabled/disabled through the Muse Console for Applications Administration.
    • Titles A to Z. The lists with the subscribed titles (books and journals) are ingested into the Muse Central Index and made available for browsing and searches.
    • Publishers. The list of subscribed publishers and their products are nicely displayed in tiles with publisher logos and descriptions. Direct links are available to navigate to publisher platforms using the institutional customer subscription.
      Publishers
    • RSS feeds. A RSS feed can be configured to display items in a widget in the application's Home page.
    • Important Links. External links with accompanying descriptions can be configured into this widget.
    • New: Books and Journals. Static metadata displaying a list of subscribed journals/books to promote for the end-users.
    • Top 10 Most Accessed Titles. This widgets presents the top 10 most accessed titles from the Titles A to Z widget.
  • Implemented the possibility to switch between a Simpler interface view with light functionality and a Richer one with the most complete set of application features that were enabled. A default view (Simpler or Richer) can be set per each application, switching between the views can be done from the "Search Options" panel, "Display Options" section, "View Display" option.
  • "Refine your Results" extended its visual representation with vertical bars for the "Dates" faceting option. The filtering action can be triggered by clicking a specific vertical bar corresponding to the desired date value. Also a slider is available for narrowing the dates for filtering the current result set based on the selected interval. Furthermore, the "Dates" filtering options were enhanced with new values: "Last Month", "Last 6 Months", "Last Year.
    Date Facet
  • The records images are also displayed in a separate widget called Images. It is available in the main area, just after the results and pagination display. The new Images widget presents in an images carousel all images from all retrieved records, clicking on any image will open the image viewer where various actions can be performed, like zoom in/out, move, rotate and share on social networks.
    Images Widget
  • "My List" functionality, which allows to save selected results from multiple searches into a temporarily list for later handling. The results saved into "My List" are available for the current session and they can be used with the existing actions, such as email, saving them into various formats, etc.

Muse Console for Applications Administration (MCAA):

  • All application's new features are configurable through the Muse Administration Console. The application's look and feel can also be customized by writing custom HTML code for header, footer and login page, adding custom CSS. Images such as customer logos can be uploaded on the server and referred into the custom code.
  • The application's widgets can be arranged by drag and drop.
  • The main functionalities can be enabled/disabled per each display type, e.g Extra Small Devices, Small Devices, Medium Devices, Large Devices.

As usual we offer a Free 30 Day Trial period to fully evaluate the new version of MuseKnowledge™ Application. You can request a trial access by simply filling in the trial form.

Configure the Muse Proxy service with NGINX front-end

NGINX is an HTTP and reverse proxy server that can be placed in front of web applications to achieve custom workflows and provide solutions where multiple web applications cannot co-exist with similar specifications.

Specifically the NGINX server can be configured as front-end for Muse Proxy when Muse Proxy cannot be configured to use the standard ports – 80 for HTTP and 443 for HTTPS access. Because there are other applications needing the standard ports as well. Only for this situation, when multiple web applications are hosted on the same server it is recommended to use NGINX, otherwise if Muse Proxy is the only hosted service, it must be accessed directly, without a front-end reverse proxy solution.

Hosting multiple web applications with NGINX as front-end reverse proxy requires assigning a dedicated fully qualified domain name (FQDN) for each application.

In this article we are describing the necessary configurations for NGINX and Muse Proxy only. It is assumed that NGINX and Muse Proxy are already installed, this article does not cover their installations.

Further assumptions:

  1. To exemplify, we consider the Muse Proxy assigned FQDN to be proxy.mydomain.com. And the server IP – 192.168.1.100. The server OS is a Linux distribution (Debian, Ubuntu, RedHat, CentOS, etc).
  2. Muse Proxy is configured with its default ports – 9797 for HTTP and 9443 for HTTPS
  3. The wildcard DNS record is enabled for the Muse Proxy assigned FQDN – *.proxy.mydomain.com.
  4. A wildcard SSL certificate is available, thus the certificate and the private key exist on the server. The certificate must contain as SAN entries both *.proxy.mydomain.com and proxy.mydomain.com.

Configure the Muse Proxy service with NGINX front-end

In the NGINX configuration file for the Muse Proxy FQDN (e.g. /etc/nginx/sites-available/proxy.mydomain.com) define the backend service:
upstream backend_proxy {
        server 192.168.1.100:9797;
}

For each port (80 and 443) contexts make sure the server_name has both values, e.g. *.proxy.mydomain.com and proxy.mydomain.com. Sample entries:

server {
        listen 80;
        listen [::]:80;
        server_name *.proxy.mydomain.com proxy.mydomain.com;
        location / {
                include proxy_params;
                proxy_pass http://backend_proxy;
                proxy_buffer_size 128k;
                proxy_buffers 4 256k;
                proxy_busy_buffers_size 256k;
        }
}
server {
        server_name *.proxy.mydomain.com proxy.mydomain.com;
        listen 443 ssl;
        ssl_certificate /etc/ssl/proxy.mydomain.com/fullchain.pem;
        ssl_certificate_key /etc/ssl/proxy.mydomain.com/privkey.pem;
        location / {
                include proxy_params;
                proxy_pass http://backend_proxy;
                proxy_buffer_size 128k;
                proxy_buffers 4 256k;
                proxy_busy_buffers_size 256k;
        }
}

The content of /etc/nginx/proxy_params refered in the above samples:

proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;

Increase the timeout value for NGINX in /etc/nginx/nginx.conf, e.g.:

proxy_read_timeout 600s;

The NGINX service must be reloaded to pick up the configuration changes.

Configurations for Muse Proxy

Some configurations are needed for Muse Proxy as well, in its main configuration file ${MUSE_HOME}/proxy/MuseProxy.xml:
Define the external ports of NGINX as follows:

<PORT enabled="true" external="80">9797</PORT>
<SSL_PORT enabled="true" external="443">9443</SSL_PORT>

Specify the allowed IPs (of the machine itself) that allowed to provide the X-FORWARDED-FOR header value:

<ALLOW_X_FORWARDED_FOR>192.168.1.100</ALLOW_X_FORWARDED_FOR>

Make sure the FQDN value is configured in the SERVER_NAMES field, e.g.:

<SERVER_NAMES>proxy.mydomain.com</SERVER_NAMES>

In this article we are revealing some of the new features introduced in the MuseSearch™ Application (version 4.0) that will be available in the next release of the Muse™ Platform.

The latest MuseSearch™ Application (version 4.0) can be also deployed and run in the current version of the Muse™ Platform (2.7.0.0), however this can be done only with the help of Muse™ Technical Support since it implies applying system patches.

MuseSearch™ is the current production template application used by end-users for accessing the Muse™ Federated Search Platform.

Security Related Improvements

We updated the Freemarker library to the latest version available – 2.3.23. The latest Freemarker version brings many security improvements and bug fixes. The most important ones are related to the mechanism of loading the templates, which now enforces more security checks and prevents malicious attempts to get unauthorized file content or run arbitrary JavaScript code.

A major improvement in the MuseSearch™ Application structure brought by the upgrade to the latest Freemarker version is the decrease of the number of files on disk. Because the latest Freemarker permits serving the template files from multiple repositories we removed the duplicate files from all languages folders.

New User Interface Features

  • Expert search page. An expert search page was introduced to allow the end-user construct complex queries:

    The query can be visually built step by step using the query builder tool or it can be manually written in the Raw Query section:
  • Faceting on the searched sources. The filtering on the searched sources was moved as a faceting feature into a separate side panel:
  • New facets were introduced: Authors and Years:
  • New side panels were added to display results from specially content sources, such as dictionary, Wikipedia, news, whether, etc. Only a configurable number of records is displayed, default is one record:
  • Related Queries were added into the MuseSearch™ Application to display in a side panel all queries that are related to the one entered by the end user. The check is done against the queries that were submitted in time by the end-users of that Muse™ system.
    Behind the scene there is a complex mechanism implemented with Muse™ Control Center dedicated tasks that mine the ICE Statistical log files for queries and index them in the Muse™ Central Index component. At search time the index is being checked and the matched related queries are displayed:
  • Query Highlight feature means that the search term is now highlighted in the retrieved records in the Muse™ application interface anywhere it is found:

In this current article we are describing a new feature introduced with Muse Proxy 4.1: the ability to perform actions transparently, behind the scene. For example change the default language in the remote website, get the end-user in different sections of the website like an advanced search form, etc.

As example we considered the Ted website (http://ted.europa.eu) which offers a Username/Password authentication type. The implementation request for profiling this source in Muse Proxy is to perform the Username/Password authentication, change the language to “English”, then select “UK – United Kingdom” , then select “Contract notice” to display all contract notices available for “United Kingdom”.

The Username/Password authentication profiling is not presented in this current article, please refer to the “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario” article for a profiling example.

The native webpage after the authentication steps is as below:

Select “English” to change the language:

Select “UK – United Kingdom”, then select “Contract notice” to display all contract notices available for “United Kingdom”.

The steps for profiling the above scenario are the following:

  1. Create a new Muse Proxy Source profile named EuropaTED_eu_UP.xml.
  2. Create the rules for the authentication Username/Password process. Because the authentication process for the Ted website is complex as it is based on dynamic parameters and values, we are making use of the Extract and Navigate functionality. For more details on this please refer to this article “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario”.

    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&submit=Login%21]]>
    </POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in
    to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
  3. Select the English language:
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
  4. Click on the UK – United Kingdom link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
  5. Click on the Contract notice link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
  6. The complete XML profile is given below:
    <ICE-CONFIG>
    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&
    submit=Login%21]]></POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
    <SHOW_GET_PARAMETERS>true</SHOW_GET_PARAMETERS>
    <USER_AGENT>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36</USER_AGENT>
    <CONNECT_TIMEOUT>60000</CONNECT_TIMEOUT>
    <READ_TIMEOUT>60000</READ_TIMEOUT>
    <NAME>TED Tenders Electronic Daily</NAME>
    <DESCRIPTION>TED provides free access to business opportunities. It is updated 5 times a week with some 1,500 public
    procurement notices from the European Union, the European Economic Area and beyond. You can browse, search and sort procurement
    notices by country, region, business sector and more. Information about every procurement document is published in the 24 official
    EU languages. All notices from the EU's institutions are published in full in these languages.</DESCRIPTION>
    <AUTHENTICATION_TYPE>User/Password</AUTHENTICATION_TYPE>
    <HTTP_AUTHORIZATION_USER_NAME/>
    <HTTP_AUTHORIZATION_USER_PASSWORD/>
    <HTTP_AUTHORIZATION_SCHEME/>
    <PROXY_USED>SOURCE_LEVEL</PROXY_USED>
    <PROXY_HOST></PROXY_HOST>
    <PROXY_PORT></PROXY_PORT>
    <PROXY_PAC></PROXY_PAC>
    <PROXY_AUTHORIZATION_USER_NAME/>
    <PROXY_AUTHORIZATION_USER_PASSWORD/>
    <PROXY_AUTHORIZATION_SCHEME/>
    <REWRITING_PATTERNS>*europa.eu;*/*.js*;</REWRITING_PATTERNS>
    <REWRITE_BY_HOST>true</REWRITE_BY_HOST>
    <ENCODING>UTF-8</ENCODING>
    <REFERER></REFERER>
    <COOKIES></COOKIES>
    <REPLACE_HOST/>
    <REPLACE_PATH></REPLACE_PATH>
    <SSL_CERTIFICATES/>
    <SSL_ALIASES/>
    <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS>
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>
    <LAST_UPDATED>2015-07-14</LAST_UPDATED>
    </ICE-CONFIG>

Username/Password authentication is the second most popular authentication type after IP authentication. There are many cases when content providers offer only Username/Password for the institutional subscriptions. The subscribers are starting to prefer Username/Password type of authentication too over IP due to IPs shortage, as it is becoming an issue to obtain new IPs.

In this article we describe a new feature of Muse Proxy 4.1: Extract and Navigate. This new feature is useful for implementing a Username/Password authentication flow for complex websites that use dynamic state information in the authentication process.

In the simplest case it suffices to provide in a single request the Username/Password values along with other CGI parameters requested by the native website.

But there are websites which store and make use of state information, a good example are the websites created using Microsoft’s ASP.NET technologies. In these cases it is not working to submit directly the authentication form with the static values, hidden fields values must be dynamically parsed from the current page and passed along with the static parameter values. Moreover, the dynamically parsed values must be URL encoded.

The Extract and Navigate feature available from Muse Proxy version 4.1 is based on sequences of regular expression EXTRACTORs, URLs and POST_PARAMETERSelements which must be specified in the order of execution. URL and POST_PARAMETERS can make use of parameters extracted by the previous EXTRACTORs as well as static parameters defined in the Source Profile via PARAMETERS.
The parameters used are specified through the construct of ${parameterName}.

The EXTRACTOR elements are containing JDK regular expression including capturing groups as described in the JDK documentation, for example, in “Summary of regular-expression constructs” [http://docs.oracle.com/javase/6/docs/api/java/util/regex/Pattern.html]. Capturing groups can be created via parenthesis ( “(groupPattern)”) are normally referred through numbers via $1$2. But because we will be saving these captured groups in the global source parameters we need to differentiate between multiple extractors. The differentiation is done via the ref attribute of the EXTRACTOR element and a variable will become accessible as ${ref_group}, that is the ref attribute value concatenated with underline (_), concatenated with the desired number. If a variable that does not exist is referenced than it will be left literally as the variable, hence make sure a number greater than the number of groups is not used. Also make sure that each EXTRACTOR has a ref attribute and its value is unique at the source profile level.

To exemplify we considered the https://legal.com.tr/ website which has Username/Password authentication.
The page https://legal.com.tr/uye-girisi contains the logon form. We consider YYY and ZZZ the Username/Password values.

In the page source we see the logon HTML form with parameters (hidden fields, view state, control state):

We start the profiling process by creating a new Muse Proxy source XML from an existing one and changing the values for the following nodes with the appropriate values:

  • URL
  • NAME
  • DESCRIPTION
  • REWRITING_PATTERNS

E.g.

<URL>https://legal.com.tr/uye-girisi</URL>
<NAME>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri</NAME>
<DESCRIPTION>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri - E-Kitap / E-Dergi / Online Kitap /
Online Dergi / Bas&#305;l&#305; Yay&#305;n - Legal Yay&#305;nevi</DESCRIPTION>
<REWRITING_PATTERNS>*legal.com.tr;</REWRITING_PATTERNS>

We name the new Muse Proxy Source: Legal_tr_UP.xml . UP is the convention we are using for labeling a Username/Password authentication type profile, tr is the 2 letter language code of the website. It is not specified when the website content is English.
Next we define and profile the steps made in the authentication process:

  1. Get the page with the logon form (https://legal.com.tr/uye-girisi) and parse from it the dynamic parameters: __EVENTTARGET, __EVENTARGUMENT, __VIEWSTATE, __VIEWSTATEGENERATOR, __EVENTVALIDATION.
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS></POST_PARAMETERS>
    <EXTRACTOR ref="eventtarget" refProcess="urlEncode">type="hidden"\sname="__EVENTTARGET"\sid="__EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="eventargument" refProcess="urlEncode">type="hidden"\sname="__EVENTARGUMENT"\sid="__EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstate" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATE"\sid="__VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstategenerator" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATEGENERATOR"\sid="__VIEWSTATEGENERATOR"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="eventvalidation" refProcess="urlEncode">type="hidden"\sname="__EVENTVALIDATION"\sid="__EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR>

    Description: For each value to be parsed we define an EXTRACTOR field. In the ref attribute we specify the name of the extractor rule, name with which we will further refer the value. In the refProcess attribute we specify a processing method, in this case we want to URL encode it.
    The value of the EXTRACTOR field is the regular expression for matching the desired value.
  2. Use the values extracted above and along with the Username/Password and other static CGI parameters and values submit to achieve the authentication.
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS><![CDATA[__EVENTTARGET=${eventtarget_1}&__EVENTARGUMENT=${eventargument_1}&__VIEWSTATE=${viewstate_1}&
    __VIEWSTATEGENERATOR=${viewstategenerator_1}&__EVENTVALIDATION=${eventvalidation_1}&
    ctl00%24MainContent%24userlogin%24UserName=${userName}&ctl00%24MainContent%24userlogin%24Password=${userPassword}&
    ctl00%24MainContent%24userlogin%24Button1=Giri%C5%9F]]></POST_PARAMETERS>

    The ${userName} and ${userPassword} metavariables are declared as following:
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>

    The full content of the xml configuration file is below, it can be downloaded from the EduLib website too by clicking here
    <ICE-CONFIG>
    <URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS></POST_PARAMETERS>
    <EXTRACTOR ref="eventtarget" refProcess="urlEncode">type="hidden"\sname="__EVENTTARGET"\sid="__EVENTTARGET"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="eventargument" refProcess="urlEncode">type="hidden"\sname="__EVENTARGUMENT"\sid="__EVENTARGUMENT"\svalue="([^"]*?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstate" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATE"\sid="__VIEWSTATE"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="viewstategenerator" refProcess="urlEncode">type="hidden"\sname="__VIEWSTATEGENERATOR"\sid="__VIEWSTATEGENERATOR"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="eventvalidation" refProcess="urlEncode">type="hidden"\sname="__EVENTVALIDATION"\sid="__EVENTVALIDATION"\svalue="([^"]+?)"</EXTRACTOR><URL>https://legal.com.tr/uye-girisi</URL>
    <POST_PARAMETERS><![CDATA[__EVENTTARGET=${eventtarget_1}&__EVENTARGUMENT=${eventargument_1}&__VIEWSTATE=${viewstate_1}&amp
    ;__VIEWSTATEGENERATOR=${viewstategenerator_1}&__EVENTVALIDATION=${eventvalidation_1}&
    ctl00%24MainContent%24userlogin%24UserName=${userName}&ctl00%24MainContent%24userlogin%24Password=${userPassword}&
    ctl00%24MainContent%24userlogin%24Button1=Giri%C5%9F]]></POST_PARAMETERS>
    <USER_AGENT>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.125 Safari/537.36</USER_AGENT>
    <CONNECT_TIMEOUT>180000</CONNECT_TIMEOUT>
    <READ_TIMEOUT>180000</READ_TIMEOUT>
    <NAME>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri</NAME>
    <DESCRIPTION>Legal Yay&#305;nevi - Hukuk Kitaplar&#305; ve Dergileri - E-Kitap / E-Dergi /
    Online Kitap / Online Dergi / Bas&#305;l&#305; Yay&#305;n - Legal Yay&#305;nevi</DESCRIPTION>
    <AUTHENTICATION_TYPE>User/Password</AUTHENTICATION_TYPE>
    <HTTP_AUTHORIZATION_USER_NAME/>
    <HTTP_AUTHORIZATION_USER_PASSWORD/>
    <HTTP_AUTHORIZATION_SCHEME/>
    <PROXY_USED>SOURCE_LEVEL</PROXY_USED>
    <PROXY_HOST></PROXY_HOST>
    <PROXY_PORT></PROXY_PORT>
    <PROXY_PAC></PROXY_PAC>
    <PROXY_AUTHORIZATION_USER_NAME/>
    <PROXY_AUTHORIZATION_USER_PASSWORD/>
    <PROXY_AUTHORIZATION_SCHEME/>
    <REWRITING_PATTERNS>*legal.com.tr;</REWRITING_PATTERNS>
    <TRANSPARENT_CONTENT_PATTERNS>*/*.js*;</TRANSPARENT_CONTENT_PATTERNS>
    <ENCODING>UTF-8</ENCODING>
    <REFERER></REFERER>
    <COOKIES></COOKIES>
    <CUSTOM_HTTP_HEADERS/>
    <REPLACE_HOST/>
    <REPLACE_PATH></REPLACE_PATH>
    <SSL_CERTIFICATES/>
    <SSL_ALIASES/>
    <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS>
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>
    <LAST_UPDATED>2015-09-08</LAST_UPDATED>
    </ICE-CONFIG>

General observations:

  • There are many online tools for testing the regular expressions, use them to make sure that the defined extractor rules behave correctly;
  • If a defined extractor rule does not match, the flow stops and the last page is displayed;
  • At any time make sure that the source profile is XML well formed.

A new MuseSearch™ widget based on the Responsive CSS Bootstrap technology and jQuery is available. The widget works with Muse version 2.7.0.0 and with the MuseSearch™ Application version 3.9.

In the below example the widget was included in an iFrame element because the EduLib website uses the jQuery UI library which may conflict with the Bootstrap technology.

A simple Bootstrap page that has included the new MSWiget version is available at: https://demo.museglobal.ro/muse/MSWidget/MSWidgetBS.html.

Install files

Installing the MuseSearch™ widget is actually pretty simple. You just need to add the following code to the <head> of your web page:

<meta name="viewport" content="width=device-width, initial-scale=1">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/bootstrap/3.3.0/css/bootstrap.min.css">
<link rel="stylesheet" href="//maxcdn.bootstrapcdn.com/font-awesome/4.2.0/css/font-awesome.min.css">
<link rel="stylesheet" href="//demo.museglobal.ro/muse/MSWidget/MSWidgetBS.css">

At the bottom of the file include the following files:

<script src="//ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>
<script src="//maxcdn.bootstrapcdn.com/bootstrap/3.3.0/js/bootstrap.min.js"></script>
<script src="//demo.museglobal.ro/muse/MSWidget/MSWidgetBS.js"></script>

This will make sure all the required files are loaded properly.

Add markup

The HTML markup for the MuseSearch™ widget is also very simple. You simply need to create a <div> with an id (EmbededMSWidget in this case). In this demo we added the MuseSearch™ widget in a jQueryUI "dialog" object. For that we created a <div> with EmbededMSWidgetPosition id in witch the MuseSearch™ will be opened.

<div id="EmbededMSWidget" class="container-fluid" style="height: 600px"></div>

Hook up the widget

Because the access to Muse Web Bridge is CORS protected, if you want to test this widget against our Muse installation on http://demo.museglobal.ro/muse let us know which is your container web page address (protocol://FQDN:port) so that we add it to our CORS ACLs. If you want to test it against your local Muse 2.7.0.0 Q&A system let us know to provide you with an application patch. Finally you need to hook up the widget by adding the following code after the EmbededMSWidget div we created.

<script>
mSWidget.init({
URL: "http://demo.museglobal.ro/muse/servlet/MusePeer", // Server URL
USER_ID: "MSWidget", // User ID
USER_PWD: "lzOoUwTu7E/UvHCw9KpOsmoEbl4=", // User Password
USER_PWD_ENCRIPTION: "SHA1", // User Pasword Encription. Values: "" | "SHA1"

WIDGET_PLACEDOLDER: "#EmbededMSWidget", // The Placeholder where the Widget will be displayed. Values: body | #ID (The ID of a HTML element such as DIV, SPAN, TD, P)

RESULTS_PER_PAGE: 10, // Results per page
RESULTS_PER_SOURCE: 10, // Results per source
USE_PAGINATION: "true", // Use Pagination. Values: "false" | "true". The "false" value means that an infinite scroll is used for the result set list. The "true" value means that the result set list is displayed in multiple pages.
SHOW_PROGRESS: "true" // Show Search Progress in another tab.
})
</script>

Besides the dedicated SharePoint MuseSearch™ WebPart, available since Muse 2.3.0.0, starting with Muse version 2.7.0.0, MuseSearch™ Application 3.9 version, in this post we describe two other simple ways to integrate the MuseSearch™ Application into a SharePoint site.

1. Insert a MuseSearch™ Application into a SharePoint 2013 Site Using the “Page Viewer Web Part”

Based on the tutorial http://community.bamboosolutions.com/blogs/sharepoint-2013/archive/2013/07/30/how-to-insert-a-web-page-onto-a-site-using-the-page-viewer-web-part-for-sharepoint-2013.aspx the actions to include the MuseSearch™ Application are:

1. Edit an existing SharePoint site page or Add a new page and Edit it;
2. On the “Edit Page”, select the “Insert Web Part” tab. From there, select “Media and Content” from “Categories”, and then “Page Viewer” from the “Parts” list;
3. Go to “Edit Web Part” to open the “Tool Pane” and input the Muse address (e.g. http://demo.museglobal.ro/muse/) you wish to display on your site into the “textbox” on the “Page Viewer”;
4. Click to “Apply” and “OK” to finish inserting the MuseSearch™ Application login page.

An example of a SharePoint page with MuseSearch™ Application interface included is depicted below.

2. Insert a MuseSearch™ Widget into a SharePoint 2013 Site Using the “Script Editor Web Part”

Based on the tutorial http://blog.cloudshare.com/2012/10/29/how-to-insert-custom-javascript-code-in-sharepoint-2013-pages-part-i the actions to include the MuseSearch™ Widget are:

1. Edit an existing SharePoint site page or Add a new page end Edit it;
2. On the “Edit Page”, select the “Insert Web Part” tab. From there, select “Media and Content” from “Categories”, and then “Script Editor Web Part”;
3. Click on the “EDIT SNIPPET” link and insert the HTML and JavaScript code for MuseSearch™ Widget just as we have in the https://www.edulib.com/blog/musesearch-widget-demo-2-using-pagination-for-the-result-set-list/ page:

<html>
<head>
<link rel="stylesheet" href="http://demo.museglobal.ro/muse/logon/MuseSearch/skins/redmond/jquery-ui.css"/>
<script src="http://demo.museglobal.ro/muse/logon/MuseSearch/javascripts/jquery.js"></script>
<script src="http://demo.museglobal.ro/muse/logon/MuseSearch/javascripts/jquery-ui.js"></script>
<link rel="stylesheet" href="http://demo.museglobal.ro/muse/MSWidget/MSWidget.css"/>
<script src="http://demo.museglobal.ro/muse/MSWidget/MSWidget.js"></script>
</head>
<body>
<div id="EmbededMSWidget" style="width: 600px; height: 400px;"></div>
<script>
mSWidget.init({
URL: "http://demo.museglobal.ro/muse/servlet/MusePeer", // Server URL
USER_ID: "MSWidget", // User ID
USER_PWD: "lzOoUwTu7E/UvHCw9KpOsmoEbl4=", // User Password
USER_PWD_ENCRIPTION: "SHA1", // User Pasword Encription. Values: "" | "SHA1"
WIDGET_PLACEDOLDER: "#EmbededMSWidget", // The Placeholder where the Widget will be displayed. Values: body | #ID (The ID of a HTML element such as DIV, SPAN, TD, P)
RESULTS_PER_PAGE: 10, // Results per page
RESULTS_PER_SOURCE: 10, // Results per source
USE_PAGINATION: "true" // Use Pagination. Values: "false" | "true". The "false" value means that an infinite scroll is used for the result set list. The "true" value means that the result set list is displayed in multiple pages.
})
</script>
</body>
</html>


An example of a SharePoint page with MuseSearch™ Widget included is depicted below.

We have made a comparison of the features between CERTivity® KeyStores Manager and the most relevant similar products. The features are organized in categories, each category initially showing all features.

Although this comparison was made by EduLib, the creator of CERTivity, we tried to be as objective and fair as possible. If you have any comments or suggestions, do not hesitate to contact us.

You can download this comparison in PDF format also.

Feature NameCERTivity
2.0
Keystore
Explorer
5.0.1
Portecle
1.7
KeyTool
IUI
2.4.1
Released Date2014-01-232013-11-242011-01-232008-10-18
Maintained
PlatformsOn any Platform That Can Run JavaOn any Platform That Can Run JavaOn any Platform That Can Run JavaOn any Platform That Can Run Java
Has bundled JRE
Has installer
KeyStore
Management
Supported Java KeyStore TypesJKS, JCEKS, PKCS12, BKS, BKS-V1, UBERJKS, JCEKS, PKCS12, BKS, UBERJKS, PKCS#12, JCEKS, JKS (case sensitive), BKS, UBER, GKR
(but option is inactive)
JKS, JCEKS, PKCS#12, BKS, UBER
Create a New KeyStore
Open an Existent KeyStore
Open Windows Root CA KeyStore
Open Windows User KeyStore
Discover JREs CA TrustStores
Open JREs CA TrustStoresOnly main JREOnly main JREOnly main JRE
Save a KeyStoreIt's done automatically after some operations
Defining a Default KeyStore(planned for future releases)
Convert KeyStore Type
Change KeyStore Password
Delete Entry
Change Entry Password
Change Entry Alias
Cut/Copy - Paste Single KeyStore EntryAllows only cloning a certificate into the same
KeyStore.
Allows only copying a certificate into the same
KeyStore
Cut/Copy - Paste Multiple Entries
TrustStore
Management
Set/Remove CA Certs TrustStore at runtime without
restarting the application
Set Multiple TrustStores for Trust Path Validation
Availability to use JRE CA Certs TrustStores (from
discovered JREs) for Trust Path Validation
Availability to use Windows KeyStores (for Microsoft
Windows Systems) for Trust Path Validation
(only Windows Root CA)
Availability to use Custom KeyStores for Trust Path
Validation
(only if the CA Certs is changed to a custom one)(only if the CA Certs is changed to a custom one)
Availability to use current opened (and selected) KeyStore
for Trust Path Validation
Display Trust Status for Certificate Entries in
KeyStores
Display Trust Status for Opened Certificates
Customizable Trust Path Validation Options Without
Restarting the Application
Available Trust Path Validation OptionsInhibit any policy, Explicit policy required, Inhibit
policy mapping, Use revocation checking, Use policy qualifier
processing, Use path length constraint (with customizable path
length size), Use custom validation date, Provider selection
(default provider or Bouncy Castle provider)
Interface
Usability
MDI Interface for KeyStores
MDI Interface for Certificates/CRL/CSR
KeyStore RepresentationTree List (Entries are displayed as a list of expandable
nodes) Available SubItems for KeyPairs : Private/Public Keys,
Certificate Chains, Certificates, Extensions Available Subitems
for Certificates: Public Key, Extensions
Simple List (entries are not expandable)Simple List (entries are not expandable)Simple List (entries are not expandable)
Available Entries Direct InformationAlgorithm and Size, Expiry Date, Last Modified, Validity
Status, Trust Status
Algorithm and Size, Expiry Date, Last Modified, Validity
Status
Alias Name, Last ModifiedFor Key Pairs and Certificates: Alias, Entry Type, Valid
Date, Self-Signed, Trusted C. A., Key Size, Cert. Type, Cert. Sig.
Algorithm, Modified Date For Secret Keys: Alias, Entry, Modified
Date
Mark Locked Keys
Mark Expired Key Pairs/Certificates
Mark Certificate Trust Status
Mark Key Pairs with Key sizes smaller than a configurable
value
Undo/Redo for KeyStore Operations and Imports
Prompting to re-enter password in case of wrong password
for unlocking Private/Secret Keys
Prompting to re-enter password in case of wrong password
when converting a KeyStore to a different type (operation does not
fail)
Informing when a Key Store which contains Secret Keys can
not be converted to a Key Store type that does not support Secret
Keys before entering all the passwords
Converts with removing secret keys (it gives a slight
warning first)
Prompting for passwords when converting from a KeyStore
type which does not support passwords to a KeyStore type which
supports entry passwords
Displaying Entry Information ModeBottom Panel (And few details in the KeyStore View)New Dialog (And few details in the KeyStore View)New Dialog (and few details in the KeyStore View)New Dialog (text based content)
Allows rearranging Key Store/Certificate tabs
Configurable Arrangement and Positioning of Tabs
Configurable Tabs Position by Dragging
Window Configuration OptionsMaximize, Float, Float Group, Minimize, Minimize Group,
Dock, Dock Group, New Document Tab Group, Collapse Document Tab
Group
Multiple KeyStore Entries Selection
Multiple KeyStore Entries Copy - Paste between
KeyStores
Copy a Certificate From a Certificate Chain and Paste It
Into Another KeyStore
Configurable Key Shortcuts (Keymap)
Displaying Providers List(planned for future releases)
"Close All Documents" Option
Opened Tabs Manager
Opened Tabs Manager OptionsSwitch to Document, Close Document(s)
Easy Tab Selector Drop list
Available Actions/Options Tree Like Structure(planned for future releases)
Quick Search (with text box)
Change Look And Feel(planned for future releases)
Password Strength Indicator(planned for future releases)
Show tips at startup(planned for future releases)
Key Pair
Operations
Generate Key Pair (RSA/DSA)
Regenerate Key Pair
Sign With Selected KeyPair at Generation Time
Key Pair Generation - Signature Algorithms (for DSA
Keys)
SHA1 With DSA, SHA224 With DSA, SHA 256 With DSA, SHA 384
With DSA, SHA 512 With DSA
SHA.1 with DSA, SHA-224 with DSA, SHA-256 With DSA, SHA-384
with DSA, SHA-512 with DSA
SHA1withDSA, SHA224withDSA, SHA256withDSASHA1withDSA
Key Pair Generation - Signature Algorithms (for RSA
Keys)
MD2 with RSA, MD5 with RSA, SHA1 with RSA, SHA1 With RSA
and MGF1, SHA224 With RSA, SHA224 With RSA and MGF1, SHA256 With
RSA, SHA256 With RSA and MGF1, SHA384 With RSA, SHA384 With RSA
and MGF1, SHA512 With RSA, SHA512 With RSA and MGF1, RIPEMD128
With RSA, RIPEMD160 With RSA, RIPEMD256 With RSA
MD2 with RSA, MD5 with RSA, RIPEMD-128 with RSA, RIPEMD-160
with RSA, RIPEMD-256 with RSA, SHA.1 with RSA, SHA-224 with RSA,
SHA-256 With RSA, SHA-384 with RSA, SHA-512 with RSA
MD2withRSA, MD5withRSA, SHA1withRSA, SHA224withRSA,
SHA256withRSA, SHA384withRSA, SHA512withRSA, RIPEMD128withRSA,
RIPEMD160withRSA, RIPEMD256withRSA
MD5withRSA, SHA256withRSA, SHA384withRSA, SHA512withRSA,
RIPEMD128withRSA, RIPEMD160withRSA, RIPEMD256withRSA
Generate Key Pair (EC)
Key Pair Generation - EC AlgorithmsEC(ECDSA), ECGOST3410EC(ECDSA)
Key Pair Generation - EC Parameters Specification (for
ECDSA Algorithm)
c2pnb272w1, c2tnb191v3, c2pnb208w1, c2tnb191v2, c2tnb191v1,
c2tnb359v1, prime192v1, prime192v2, prime192v3, c2tnb239v3,
c2pnb163v3, c2tnb239v2, c2pnb163v2, c2tnb239v1,, c2pnb163v1,
c2pnb176w1, prime256v1, c2pnb304w1, c2pnb368w1, c2tnb431r1,
prime239v3, prime239v2, prime239v1, sect233r1, secp112r2,
secp112r1, secp256k1, sect113r2, secp521r1, sect113r1, sect409r1,
secp192r1, sect193r2, sect131r2, sect193r1, sect131r1, secp160k1,
sect571r1, sect283k1, secp384r1, sect163k1, secp256r1, secp128r2,
secp128r1, secp224k1, sect233k1, secp160r2, secp160r1, sect409k1,
sect283r1, sect163r2, sect163r1, secp192k1, secp224r1, sect239k1,
sect571k1, B-163, P-521, P-256, B-233, P-224, B-409, P-384, B-283,
B-571, P-192, brainpoolp512r1, brainpoolp384t1, brainpoolp256r1,
brainpoolp192r1, brainpoolp512t1, brainpoolp256t1,
brainpoolp224r1, brainpoolp320r1, brainpoolp192t1,
brainpoolp160r1, brainpoolp224t1, brainpoolp384r1,
brainpoolp320t1, brainpoolp160t1
prime192v1, prime239v1, prime256v1
Key Pair Generation - EC Parameters Specification (for
ECGOST3410 Algorithm)
GostR3410-2001-CryptoPro-A, GostR3410-2001-CryptoPro-XchB,
GostR3410-2001-CryptoPro-XchA, GostR3410-2001-CryptoPro-C,
GostR3410-2001-CryptoPro-B
Key Pair Generation - Signature Algorithms (for ECDSA EC
Keys)
SHA1withECDSA, SHA224withECDSA, SHA256withECDSA,
SHA384withECDSA, SHA512withECDSA
SHA1withECDSA,, SHA224withECDSA, SHA256withECDSA,
SHA384withECDSA, SHA512withECDSA
Key Pair Generation - Signature Algorithms (for ECGOST3410
EC Keys)
GOST3411 with ECGOST3410
Key Pair Generation CERT X.500 DN FieldsCommon Name (CN), Organization Unit (OU), Organization (O),
Locality (L), State (ST), Country (C), Email (E)
Common Name (CN), Organization Unit (OU), Organization (O),
Locality (L), State (ST), Country (C), Email (E)
Common Name (CN), Organization Unit (OU), Organization (O),
Locality (L), State (ST), Country (C), Email (E)
Common Name (CN), Organization Unit (OU), Organization (O),
Locality (L), State (ST), Country (C), Email (E)
Standardized DN Country Codes (2 letter code)
support
Key Pair Generation CERT X.500 DN Fields (extended)(planned for future releases)Title, Device serial number name, Business category, DN
qualifier, Pseudonym, 1-letter gender, Name at birth, Date of
birth, Place of birth, Street, Postal code, Postal address,
2-letter country of residence, 2-letter country of
citizenship
Key Pair Generation CERT X.520 Name(planned for future releases)Surname, Given name, Initials, Generation, Unique
Identifier
Import Key Pair into KeyStore (from PKCS#12 Files)
Import Key Pair into KeyStore (from PKCS#8 private key and
Certificate)
Import Key Pair into KeyStore from OpenSSL private key and
certificate)
Import Key Pair into KeyStore (from PVK private key and
Certificate)
(planned for future releases)
Import Key Pair into KeyStore (from PEM private key and
Certificate Chain)
Import Key Pair into KeyStore (from other KeyStore)(planned for future releases)
Import Key Pair into KeyStore from a Private Key and More
Certificate Files (which can create a chain)
Export Key Pair (PKCS#12)
Export Key Pair (PEM Encoded)(planned for future releases)
Extend Validity of Self-Signed KeyPairs
Enter New Serial Number When Extending Validity of
Self-Signed Certificates
Certificates
Operations
Open a standalone certificate/Examine standalone
certificate
Open a Certificate Chain/Examine Certificate Chain
View Certificate Details
View Certificate Details From Signature(only Certificate Type and Subject DN, for each signed
entry, for JAR files)
Available Certificate DetailsFormat, Version, Serial Number, Valid From/To, Public Key,
Extensions, Signature Algorithm, Multiple Fingerprints,
Subject/Issuer Information (CN, OU, O, L, ST, C, E), PEM,
ASN.1
Version, Serial Number, Valid From/Until, Public Key,
Signature Algorithm, Multiple Fingerprints, Subject/Issuer
Information (CN, OU, O, L, ST, C, E), Extensions, PEM,
ASN.1
Chain position and total number of certificates in the
chain, Version, Serial Number, Valid From/Until, Public Key,
Signature Algorithm, Fingerprints, Subject/Issuer DN String,
Extensions, PEM Encoding
Owner (Subject DN String), Issuer (Issuer DN String),
Version, Serial Number, Valid From/Until, Signature Algorithm,
Fingerprints, Extensions
Available FingerprintsMD2, MD4, MD5, SHA1, RIPEMD-128, RIPEMD-160, RIPEMD-256,
SHA-224, SHA-256, SHA-384, SHA-512
MD2, MD4, MD5, RIPEMD-128, RIPEMD-160, RIPEMD-256, SHA-1,
SHA-224, SHA-256, SHA-384, SHA-512
SHA1, MD5MD5, SHA.1
View PEM Representation for a Certificate
View ASN.1 for a Certificate
Import Certificate from files into KeyStore
Import Root CA Certificate (directly into the Root CA certs
KeyStore)
(planned for future releases)
Import Certificate into a KeyStore directly from
Certificate Details Dialog
(planned for future releases)
Import Certificate into KeyStore with trust path
validation
(manual validation)
Import Certificate from Server into KeyStore
Import Certificate from Signature into KeyStore
Export Certificate
Export Certificate From Signature to file (JAR, APK, PDF,
XML)
Export Certificate Supported FormatsX.509, X.509 PEM Encoded, PKCS#7, PKCS#7 PEM Encoded, PKI
Path
X.509, X.509 PEM Encoded, PKCS#7, PKCS#7 PEM Encoded, PKI
Path, SPC
DER Encoded, PEM Encoded, PKCS#7, PkiPathDER, PKCS#7, PEM
Export Certificate Chain(only when exporting with private key also)
Export Certificate Chain Supported FormatsPKCS#7, PKCS#7 PEM Encoded, PKI PathPKCS#7, PKCS#7 PEM Encoded, PKI PathPKCS#7, PkiPathDER, PEM
Obtain the Revocation Status
Retrieve Certificate From SSL ServerTLSv1, TLS v1.1, TLS v1.2 and default algorithmTLSv1, TLS v1.1, TLS v1.2TLSv1 (SSLv 3.1)
Retrieve Certificate From SSL Server (additional connection
info)
(planned for future releases)Connection Protocol, Connection Cipher Suite
Retrieve Certificate From SSL Server using HTTPS URL (not
host and port specifically)
Test Certificates on Given Protocol
View Associated CRL
Append signer certificate to key pair certificate
chains
Remove signer certificate from key pair certificate
chains
Rename Certificate
Delete Certificate
Renewal of CertificateOnly when the certificate is within a Key Pair
Certificate
Extensions
View Certificate Extensions
View ASN.1 for a Certificate Extension
Add Certificate Extensions when generating a new
KeyPair
Add Certificate Extensions to CA Replies
Save Certificate Extensions Template
Save Certificate Extensions Template as XML
Available Certificate ExtensionsAuthority Information Access, Authority Key Identifier,
Basic Constraints, Certificate Policies, CRL Distribution Points,
Extended Key Usage, Freshest CRL, Inhibit Any Policy, Issuer
Alternative Name, Key Usage, Name Constraints, Netscape Cert Type,
Private Key Usage Period, Policy Constraints, Policy Mappings,
Subject Alternative Name, Subject Information Access, Subject
Directory Attributes, Subject Key Identifier.
Authority Information Access, Authority Key Identifier,
Basic Constraints, Certificate Policies, Extended Key Usage,
Inhibit Any Policy, Issuer Alternate Name, Key Usage, Name
Constraints, Netscape Base URL, Netscape CA Policy URL, Netscape
CA Revocation CRL, Netscape Certificate Renewal URL, Netscape
Certificate Type, Netscape Comment, Netscape Revocation URL,
Netscape SSL Server Name, Policy Constraints, Policy Mappings,
Private Key Usage Period, Subject Alternative Name, Subject
Information Access, Subject Key Identifier
(many only for display, but not specified anywhere)(many for display) For Key Pair Creation: Key Usage,
Extended Key Usage
Extensions display at creation time (GUI Point of
view)
Tree - like Structure where all extensions, properties and
sub-items are visible in a single dialog
List of extensions, each one opening in a different dialog
for setting properties, and each sub-item opens also in a
different dialog
Certificate Authority
Functions
Check PKI file type
Certificate Signing made easier using “Select as CA Issuer”
and “Sign Certificate by ” actions
Certificate chain management: append and remove signer
certificate (with Copy/Paste/Delete/Undo/Redo functionality
included)
(supported only from menu without Copy/Paste)
Generate Certificate Signing Request (CSR) files
Sign Certificate Signing Request (CSR) files
Import CA Reply
Trust verification when Importing CA Reply
Trust verification when Importing CA Reply (with user
confirmation when trust is not established)
Act as a testing purposes CA (by generating CSR files,
signing CSRs and importing CA Replies
CSR
View CSR Details/Examine CSR(only PEM display)
Available CSR DetailsFormat, Version, Public Key (with details available),
Signature Algorithm, Subject (CN, OU, O, L, ST, C, E), Challenge,
CSR Dump (PEM)
Format, Public Key (with details available), Signature
Algorithm, Subject (CN, OU, O, L, ST, C, E), Challenge, CSR Dump
(PEM, ASN.1)
Version, Subject DN String, Public Key (Algorithm and
size), Signature Algorithm, PEM
PEM
Generate CSR Files
Generate CSR Files Supported FormatsPKCS#10, SPKACPKCS#10, SPKACPKCS#10 (probably)PKCS#10
CRL
View CRL Details/Examine CRL
View Remote CRLs
Protocols Supported for Opening Remote CRLsHTTP, HTTPS, FTP, LDAP
Available CRL DetailsType, Version, This Update, Next Update, Signature
Algorithm, Issuer (CN, OU, O, L, ST, C, E), Extensions, ASN.1,
Revoked Certificates (+Extensions)
Version, Issuer (CN, OU, O, L, ST, C, E), Effective Date,
Next Update, Signature Algorithm, Extensions, ASN.1, Revoked
Certificates (+Extensions)
Version, Issuer DN String, Effective Date, Next Update,
Signature Algorithm, Extensions, Revoked Certificates
(+Extensions)
View CRL Extensions
Next Update Exeeded Verification
CA Reply
Import CA Reply With Trust Path Validation
View CA Reply Details(Only if opened as a certificate and browse through the
chain)
(Only if opened as a certificate, and browse through the
chain)
(Only if opened as a certificate, and you can browse
through the chain)
Create CA Reply
Secret Key
Operations
Available Secret Keys InformationAlgorithm, Last ModifiedAlgorithm, Key Size, Last ModifiedLast ModifiedModified date
View Secret Key Details(planned for future releases)Algorithm, Format, Size, Value in hexa
Generate Secret Key
Secret Key Supported AlgorithmsAES, AESWrap, ARCFOUR, BlowFish, Camellia, Cast5, Cast6,
DES, DESede, DESedeWrap, GOST28147, Grainv1, Grain128, HC128,
HC256, Noekeon, RC2, RC4, RC5, RC5-64, RC6, Rijndael, Salsa20,
Seed, Serpent, Skipjack, TEA, Twofish, VMPC, VMPC-KSA3, XTEA,
HmacMD2, HmacMD4, HmacMD5, HmacRIPEMD128, HmacRIPEMD160, HmacSHA1,
HmacSHA224, HmacSHA256, HmacSHA384, HmacSHA512, HmacTIGER
AES, ARC4, Blowfish, Camellia, CAST-128, CAST-256, DES,
DESEDE, GOST 28147-89, Grain v1, Grain-128, HC-128, HC-256,
HMac-MD2, HMac-MD4, HMac-MD5, HMac-RipeMD128, HMac-RipeMD160,
HMac-SHA1, HMac-SHA224, HMac-SHA256, HMac-SHA384, HMac-SHA512,
HMac-Tiger, NOKEON, RC2, RC5, RC6, Rijndael, Salsa20, Serpent,
SEED, Skipjack, TEA, Twofish, XTEA
AES, ARCFOUR, Blowfish, DES, DESede, HmacMD5, HmacSHA1,
HmacSHA256, HmacSHA384, HmacSHA512, RC2
Provider Selection for Generation Available
Offers Supported Key Sizes for Each Algorithm
Import Secret Key From File(planned for future releases)
Export Secret Key To File(planned for future releases)
Export Secret Key To File Format(planned for future releases)DER, PEM
Private Key
Operations
View Private Key Details
Available Private Key Details (for DSA)Algorithm, Key Size, Fields (Basic Generator G, Prime
Modulus P, SubPrime Q, Private Key Value; ), ASN.1
Algorithm, Key Size, Fields (Prime Modulus P, Prime Q,
Generator G, Secret Exponent X), ASN.1
Key Size
Available Private Key Details (for RSA)Algorithm, Key Size, Fields (Modulus, Private Exponent,
Public Exponent, CRT Coefficient, Prime Exponent P, Prime Exponent
Q, Prime Modulus P, Prime Q), ASN.1
Algorithm, Key Size, Format, Encoded, Fields (Public
Exponent, Modulus, Prime P, Prime Q, Prime Exponent P, Prime
Exponent Q, CRT Coefficient, Private Exponent), ASN.1
Key Size
Available Private Key Details (for ECDSA /
ECGOST3410)
Algorithm, Key Size, Parameters Specification, Fields
(Private Value S, Cofactor, First Coefficient A, Second
Coefficient B, Field Size, Seed, Generator Affine X-Coordinate,
Generator Affine Y-Coordinate, Generator Order), ASN.1
Algorithm, Key Size (for ECDSA only), Format, Encoded,
ASN.1
Key Size (for ECDSA only)
Export Private Key(but only together with certificate file)
Export Private Key Supported FormatsPKCS#8, PKCS#8 PEM Encoded, Open SSL PEM EncodedPKCS#8, PKCS#8 PEM Encoded, PVK, OpenSSL PEM
Encoded
DER, PEM
Export Private Key Encryption Algorithms (PKCS#8)PBE_SHA1_2DES, PBE_SHA1_3DES, PBE_SHA1_RC2_40,
PBE_SHA1_RC2_128, PBE_SHA1_RC4_40, PBE_SHA1_RC4_128
PBE with SHA.1 and 2 key DESede, PBE with SHA.1 and 3 key
DESede, PBE with SHA.1 and 40 bit RC2, PBE with SHA.1 and 128 bit
RC2, PBE with SHA.1 and 40 bit RC4, PBE with SHA.1 and 128 bit
RC4
Export Private Key Encryption Algorithms (OpenSSL)AES-128-CBC, AES-128-CFB, AES-128-ECB, AES-128-OFB, BF-CBC,
BF-CFB, BF-ECB, BF-OFB, DES-CBC, DES-CFB, DES-ECB, DES-EDE-CBC,
DES-EDE-CFB, DES-EDE-ECB, DES-EDE-OFB, DES-EDE, DES-EDE3-CBC,
DES-EDE3-CFB, DES-EDE3-ECB, DES-EDE3-OFB, DES-EDE3, DES-OFB,
RC2-40-CBC, RC2-64-CBC, RC2-CBC, RC2-CFB, RC2-ECB, RC2-OFB
PBE with DES CBC, PBE with DESede CBC, PBE with 128 but AES
CBC, PBE with 192 bit AES CBC, PBE with 256 bit AES CBC
Public Key
Operations
View Public Key Details
Available Public Key Details (for DSA Keys)Algorithm, Key Size, Fields (Basic Generator G, Prime
Modulus P, SubPrime Q, Public Key), ASN.1
Algorithm, Key Size, Format, Encoded, Fields (Prime Modulus
P, Prime Q, Generator G, Public Key Y), ASN.1
Available Public Key Details (for RSA Keys)Algorithm, Key Size, Fields (Modulus, Public Exponent),
ASN.1
Algorithm, Key Size, Format, Encoded, Fields (Public
Exponent, Modulus), ASN.1
Available Public Key Details (for ECDSA / ECGOST3410
Keys)
Algorithm, Key Size, Fields (Basic Generator G, Prime
Modulus P, SubPrime Q, Public Key), ASN.1
Algorithm, Key Size, Format, Encoded, ASN.1
Export Public Key
Export Public Key Supported FormatsOpen SSL, Open SSL PEM EncodedOpen SSL, Open SSL PEM Encoded
Sign and Verify
Verify Signatures for JAR Files
Verify Signatures for APK Files
Verify Signatures for PDF Files
Verify Signatures for XML Files
Verify XML Signature - allow using external cert.
validation
Verify XML Signature - set use external cert. validation
and embedded cert. validation order
Verify XML Signature - allow selecting the external cert.
from file or from a given KeyStore entry (from KeyStore
file)
Sign JAR Files
JAR Signing - Signature AlgorithmsSHA.1 with DSA, MD2 with RSA, MD5 with RSA, SHA.1 with RSA,
SHA.1 with ECDSA
SHA.1 with DSA, MD2 with RSA, MD5 with RSA, SHA.1 with
RSA
SHA.1 With DSA, SHA.1 With RSA
JAR Signing - Digest AlgorithmsMD2, MD5, SHA.1, SHA224, SHA256, SHA384, SHA512MD2, MD5, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512SHA.1
JAR Signing - Add Full Manifest Digest Attribute
Configurable
Sign APK Files
APK Signing - Signature AlgorithmsSHA.1 with DSA, MD2 with RSA, MD5 with RSA, SHA.1 with
RSA
SHA.1 with DSA, MD2 with RSA, MD5 with RSA, SHA.1 with
RSA
APK Signing - Digest AlgorithmsMD2, MD5, SHA.1, SHA224, SHA256, SHA384, SHA512MD2, MD5, SHA-1, SHA-224, SHA-256, SHA-384, SHA-512
APK Signing - Add Full Manifest Digest Attribute
Configurable
Sign XML Files
XML Signing - Signature TypesEnveloped, Enveloping, DetachedEnveloped
XML Signing - Digest AlgorithmsSHA1, SHA256, SHA512
XML Signing - Canonicalization AlgorithmsInclusive, Inclusive With Comments, Exclusive, Exclusive
With Comments
XML Signing - Allow Attaching Key To Signature
XML Signing - Allow Attaching Certificate To
Signature
Sign PDF Files
PDF Signing - Signature Subfiltersadbe.pkcs7.sha1, adbe.x509.rsa_sha1,
adbe.pkcs7.detached
Sign CSR Files/Create Certificate from CSR
Prevention for Signing CSR Files by the Same Key Pair That
Created Them
CSR Signing - Signature AlgorithmsSHA.1 with DSA, SHA224 with DSA, SHA256 with DSA, SHA384
with DSA, SHA512 with DSA, MD2 with RSA, MD5 with RSA, SHA.1 with
RSA, SHA.1 with RSA and MGF1, SHA224 with RSA, SHA224 with RSA and
MGF1, SHA256 with RSA, SHA 256 with RSA and MGF1, SHA384 with RSA,
SHA 384 with RSA and MGF1, SHA512 with RSA, SHA512 with RSA and
MGF1, RIPEMD128 with RSA, RIPEMD160 with RSA, RIPEMD256 with
RSA
SHA.1 with DSA, SHA-224 with DSA, SHA-256 With DSA, SHA-384
with DSA, SHA-512 with DSA, MD2 with RSA, MD5 with RSA, RIPEMD-128
with RSA, RIPEMD-160 with RSA, RIPEMD-256 with RSA, SHA.1 with
RSA, SHA-224 with RSA, SHA-256 With RSA, SHA-384 with RSA, SHA-512
with RSA
Sign J2ME MIDlet Applications Files
Verify Detached Signature - CMS(planned for future releases)
Sign With Detached Signature - CMS(planned for future releases)
Detached Signature - CMS Formats - CMS Signature
File
(planned for future releases)P7M, P7S
Detached Signature - CMS Formats - CMS Certs-only
file
(planned for future releases)P7C
Detached Signature - CMS Formats - digest
algorithms
(planned for future releases)SHA1, SHA224, SHA256, SHA384, SHA512, MD5, RIPEMD128,
RIPEMD160, RIPEMD256
Verify Detached Signature - Other(planned for future releases)
Sign Detached Signature - Other(planned for future releases)
Detached Signature - Other Formats - Signature File(planned for future releases)DER, PKCS#7, PEM
Detached Signature - Other Formats - Certificate
File
(planned for future releases)DER, PKCS#7, PEM
Allow signing using any Key Pair irrespective of
Certificate extension
Suggest candidate KeyPairs for signing (the ones that have
the right extensions for their certificates)
(planned for future releases)
Encrypting Files
Encrypt file using Secret Key
Encrypt file using RSA trusted certificate
Encrypt file using private key
RSA Encryption AlgorithmsRSA/ECB/PKCS1Padding, RSA/NONE/PKCS1Padding,
RSA/NONE/OAEPWithSHA1 AndMGF1Padding
Other
KeyStore Persistence between sessions
KeyStore Persistence typeFully persist (name and passowrd), Only KeyStore names, No
persistence
Open Files Using Drag & Drop
File Types Supported For Drag & DropKeyStore, Certificate, CSR, CRL irrespective of the file
extension
KeyStoreOnly based on extension: KeyStore, Certificate, CSR,
CRL
Supported KeyStore file extensionscacerts, ks, jks, jce, p12, pfx, bks, ubr, keystoreks, keystore, jks, jceks, bks, uber, pfx, p12ks, jks, jceks, p12, pfx, bks, cacertsubr, jks, ks, jce, bks, pfx, p12
Open Recent Files(maximum 4 files)
Remember last file directory between sessions
Remember last file directory for each specific action
(Opening a Key Store, a Certificate, etc.)
KeyStore Properties (Tree - like entries
structure)/KeyStore Report
(planned for future releases)
KeyStore Properties - Export structure in text and XML
formats)
(planned for future releases)(copy in memory)
Set Password Quality(planned for future releases)
Configure/Set Internet Proxy(planned for future releases)
View Cryptography Strength/Policy Details
Detection of Cryptography Strength Policy Limitation when
Launching the Application
(planned for future releases)
GUI Support for Upgrading Cryptography Strength
Support for Manual Upgrading Cryptography Strength in case
automatic upgrade fails
Customizable PropertiesCertificate expiry notification interval, RSA Key Pair
minimum allowed size, RSA Key Pair maximum allowed size, RSA Key
Pair default size, Autogenerated certificate serial number maximum
bit length, Undo level, Log level, Memory usage maximum threshold
level, Keystore persistence type, Recent file list maximum size,
JRE CA KeyStore list max size, Certificates Retriever connection
type, Inspected and draggable file size limit
Set CA Certificates Key Store, Minimum Password Quality,
Look And Feel, Internet Proxy, Trust Checks
Import/Export Configuration Properties
Add extension to file name on export, if the name does not
contain an extension from the selected file filter
Password Manager (remember passwords after
unlocking)
Archiving directories into JAR/APK files(planned for future releases)
OS File Associations(planned for future releases)(only for KeyStores)

The Muse manuals are written in DocBook and a build process generates the PDF files which are actually the documents for the end users. We have improved the building process of Muse manuals so that it does not hang up forever. We used a workaround, more details about this and what the issues are for reference, after explaining the steps involved.

Basically each document is built through a call to the Ant script ${MUSE_HOME}/doc/tools/build.xml where the FOP is launched in Java task using fork = true(i.e. another process). The FOP was blocking here sometimes when many manuals are run in a row – this happens when invoking ${MUSE_HOME}/buildall.xml doc which calls through all the other ants and finally reaches this FOP.

We have ended up adding a timeout to the Java process. We have added a big enough timeout value (15 minutes) that a single manual cannot take that long. Upon killing the Java process the return value is -1. Previously the return value was tested to see if it is 0 (OK) or non-zero (wrong). Upon non-zero it was considered a FOP error and the whole process stopped. We have added a test against -1, and when -1 is detected we output this:


Warning: FOP probably timed out (15 minutes of execution), due to the JAI infinite looping bug.
Although it should be unaffected inspect the PDF document ${docName}.pdf.

FOP should not be returning the value -1 upon various errors. We have checked the exact FOP source code for this and it can only return 0 (on success), 1 (a File not found) or 2 (a real FOP issue). But for covering everything, the Release Manager should inspect the manuals, as well as actually run through all the build log for any other operation. Note that every time ${MUSE_HOME}/buildall.xml is called, a log file ${MUSE_HOME}/buildall.log is created (overwritten). So, this file should be inspected on all the steps for unusual errors.

This timeout scenario will be recognized as below:

[echo] Converting Muse Designer Console.xml
DocBook2PDFSingle:
[INFO] Using org.apache.xerces.jaxp.SAXParserImpl$JAXPSAXParser as SAX2 Parser

[WARNING] Zero-width table column!
[WARNING] Zero-width table column!
[INFO] Parsing of document complete, stopping renderer
Timeout: killed the sub-process
Java Result: -1
[echo] Warning: FOP probably timed out (15 minutes of execution), due to the JAI infinite looping bug.
Although it should be unaffected inspect the PDF document Muse Designer Console.pdf.

FOP (actually Java Advanced Images – JAI) hangs upon JVM termination so we will always see that FOP has done its job:

Parsing of document complete, stopping renderer.

This is the state the Release Manager was founding the building process, and here he stopped (^C) the ant process. Now it will take up to 15 minutes (or what was left of them) and the Java process is killed by timeout and the above messages are an indication for this.

If one is following live the building process and sees it stalled in:

Parsing of document complete, stopping renderer.

he can do the usual stopping to avoid waiting up to 15 minutes. If the process is in the background it will do its job. We have tested this and it actually blocked 2 times and the total amount of time spent for building all the manuals was 71 minutes. For a background process this is fine. Usually it happened that manuals were stalled for hours there because the Release Manager was doing other tasks in parallel so this would be an improvement. As well for the Muse Control Center task that weekly builds the latest manuals. It should not hang up as before.

——–

Now about how a set of bugs and “features” of open source components could chain each other making almost impossible to achieve a reliable environment.

So, we had this bug of Muse Manuals hanging. Based on the observation CPU was 50% (as there are two units). A stack trace revealed that the JAI codec was doing some clean-up and intermittently filling up a stack trace…and JVM could not stop…


Thread-0? prio=2 tid=0×03088000 nid=0xee0 runnable [0x03fbf000]
java.lang.Thread.State: RUNNABLE
at java.lang.Throwable.fillInStackTrace(Native Method)
- locked <0×06cf3958> (a java.util.ConcurrentModificationException)
at java.lang.Throwable.<init>(Throwable.java:181)
at java.lang.Exception.<init>(Exception.java:29)
at java.lang.RuntimeException.<init>(RuntimeException.java:32)
at java.util.ConcurrentModificationException.<init>(ConcurrentModificationException.java:57)
at java.util.HashMap$HashIterator.nextEntry(HashMap.java:793)
at java.util.HashMap$KeyIterator.next(HashMap.java:828)
at com.sun.media.jai.codec.TempFileCleanupThread.run(FileCacheSeekableStream.java:323)
Locked ownable synchronizers:
- None
“DestroyJavaVM” prio=6 tid=0×002b7400 nid=0×8a0 in Object.wait() [0x0090f000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
– waiting on <0×0b71ad30> (a com.sun.media.jai.codec.TempFileCleanupThread)
at java.lang.Thread.join(Thread.java:1143)
- locked <0×0b71ad30> (a com.sun.media.jai.codec.TempFileCleanupThread)
at java.lang.Thread.join(Thread.java:1196)
at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:79)
at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:24)
at java.lang.Shutdown.runHooks(Shutdown.java:79)
at java.lang.Shutdown.sequence(Shutdown.java:123)
at java.lang.Shutdown.shutdown(Shutdown.java:190)
- locked <0×2b79e3e0> (a java.lang.Class for java.lang.Shutdown)
Locked ownable synchronizers:
- None

Now we have jumped into to try to see the whole DocBook related process. There is a hierarchy of Ant builds from the global buildall.xml to buildall.xml of each project, to build of each sub-project and finally calling ${MUSE_HOME}/doc/tools/build.xml where this line was invoked


<java classname=”org.apache.fop.apps.Fop” fork=”true” failonerror=”false” maxmemory=”512m” resultproperty=”errorCode” timeout=”900000?>
<sysproperty key=”MUSE_HOME” value=”${MUSE_HOME}”/>
<!--...-->
</java>

We were trying to read the FOP documentation. We do not use the recommended way (http://xmlgraphics.apache.org/fop/0.95/anttask.html), that is, using the Ant Task and we were calling the command line from an external process. We considered doing this (using the FOP Ant task), although we had a very very old FOP (fop-0.20.5). However we needed to split what the command line was able to do and the Ant task was not into two operations. But that is the recommended way. Namely first create the *fo file, through an XSLT and then run the FOP converter. The command line was running the XSLT inside and without the need of a temporary file.

We considered doing this because we were thinking in the direction of switching to FOP 1.0 in the future, which actually accepts the XML dockbook as parameter and not the FO. Also we were thinking there could be improvements to speed because we could be running in the future multiple docs through a file set, without the need to step into each document. Considering this we tried using the documented way of using FOP in Ant and…we ran into a PermGen issue. After increasing it to 256 MB we still get OOM PermGen. We considered then that this is due to the recommended way of doing a task def…


<taskdef classname="org.apache.fop.tools.anttasks.Fop">
<classpath>
<fileset dir="${fop.home}/lib">
<include/>
</fileset>
<fileset dir="${fop.home}/build">
<include/>
<include />
</fileset>
</classpath>
</taskdef>

We had many classpath elements not few as the example above, and although this task def was in the context of the build script for the manual finished with each manual, still persist in Ant. This is … an ANT bug:

“Sub-builds (antcall, subant) load a task each time the task is defined, but do not release it when the sub-build project completes.[...]“
[https://issues.apache.org/bugzilla/show_bug.cgi?id=49021]

Following the workarounds mentioned we ended up trying to define the FOP task at the upper level and at the lowest level we tested if it is defined and only then we were defining it. To make it simpler we used the recommended Antlib approach. We ended up with this in the ${MUSE_HOME}/doc/tools/build.xml


<condition property=”alreadyDefined” value=”true” else=”false”>
<typefound name=”antlib:org.apache.fop.tools.anttasks.Fop:fop”/>
</condition>
<echo message=”alreadyDefined:${alreadyDefined}”/>
<if value=”false”>
<echo message=”redifining”/>
<taskdef uri=”antlib:org.apache.fop.tools.anttasks.Fop” resource=”antlib.xml” classpath=”${MUSE_HOME}/doc/tools/fop/lib/”/>
</if>
<!--...-->
<fop:fop format=”application/pdf” userConfig=”${MUSE_HOME}/doc/tools/fop/conf/userconfig.xml”
basedir=”${docBaseDir}/${docName}” fofile=”${temp.fopFile}” outfile=”${docBaseDir}/${docName}.pdf”/>

“basedir Base directory to resolve relative references (e.g., graphics files) within the FO document. No, for single FO File entry, default is to use the location of that FO file.”
[http://xmlgraphics.apache.org/fop/0.95/anttask.html]

Hence we used basedir and were happy that it worked. Another run, forgot a document was opened, the whole process needed to be revert, run again…finished in about 60 minutes…forgot to mention that we left the ANT_OPTS on the build machine to use more memory (-Xmx896M -XX:MaxPermSize=128M). This should make all the building process take a little less.

Then we were happy that the images appeared in the docs. But we said we should be comparing various new rendered PDFs with old ones. Many of them had the same dimension, but there were others with differences. Muse Testing.pdf (one containing many images) was smaller than the one from CVS with 1 MB.
By looking into it we saw images from Muse Designer Console.pdf and from ICE MARC to XML Converter.pdf. Well we said it has to do with some cache, because now there is a single instance of FOP to the entire run…and, indeed that was it, actually a feature of FOP:

“FOP caches images between runs. There is one cache per FopFactory instance. The URI is used as a key to identify images which means that when a particular URI appears again, the image is taken from the cache. If you have a servlet that generates a different image each time it is called with the same URI you need to use a constantly changing dummy parameter on the URI to avoid caching.”
[http://xmlgraphics.apache.org/fop/1.0/graphics.html#caching]

At this point we said stop to this whole chain as it wasn’t worth. Although we are keeping, for reference, the work done, so that when these issues may be resolved in the future we use the recommended way, adding a dummy parameter (hadn’t done this anyway) or trying to use the name of the file in the image or even mentioning the directory (which is the document name) means many modifications, and also in case a document name change is necessary then there are too many troubles in modifying it. So, using different file names for images should never be considered.

Meanwhile, while doing this by following the right documentation, we came across the timeout parameter of the Java fork, and had always had this workaround in mind. But we were curios why that JAI (Java Advanced Imaging) blocks there in the stack trace from above, and actually found that this is a JAI bug, not resolved in the latest version either. The bug is detailed here:

“The shutdownHook TempFileCleanupThread throws a ConcurrentModificationException in fileIter.next() sometimes. This exception is ignored and because .next() doesn’t succeed the loop never ends. It is about this code in com/sun/media/jai/codec/FileCacheSeekableStream.java:”

/**
* Deletes all <code>File</code>s in the internal cache.
*/
public void run() {
if(tempFiles != null && tempFiles.size() > 0) {
Iterator fileIter = tempFiles.iterator();
while(fileIter.hasNext()) {
try {
File file = (File)fileIter.next();
file.delete();
} catch(Exception e) {
// Ignore
}
}
}
}

[http://java.net/jira/browse/JAI_CORE-121]

According to the comments on Java Net Jira this isn’t resolved so far. We have also a peek in the latest JAI code from trunk and it is not yet resolved. It is strange something can block upon termination. So, we ended up doing the timeout trick…nothing else came to mind, and there was already too much time spent on this.

We wrote all this in case someone else confronts with something similar, and if not to see how not to design things. Recently we have come across many Swing bugs, even filled one to Oracle.

The Core team was fighting a similar stopping bug with RMI from Sun (in the context of Tomcat and Jackrabit)…all these are not nice, in case that there is not even a workaround for some bugs.