Muse® Proxy Source Profiling: Transparent Navigation Scenario

In this current article we are describing a new feature introduced with Muse Proxy 4.1: the ability to perform actions transparently, behind the scene. For example change the default language in the remote website, get the end-user in different sections of the website like an advanced search form, etc.

As example we considered the Ted website (http://ted.europa.eu) which offers a Username/Password authentication type. The implementation request for profiling this source in Muse Proxy is to perform the Username/Password authentication, change the language to “English”, then select “UK – United Kingdom” , then select “Contract notice” to display all contract notices available for “United Kingdom”.

The Username/Password authentication profiling is not presented in this current article, please refer to the “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario” article for a profiling example.

The native webpage after the authentication steps is as below:

Select “English” to change the language:

Select “UK – United Kingdom”, then select “Contract notice” to display all contract notices available for “United Kingdom”.

The steps for profiling the above scenario are the following:

  1. Create a new Muse Proxy Source profile named EuropaTED_eu_UP.xml.
  2. Create the rules for the authentication Username/Password process. Because the authentication process for the Ted website is complex as it is based on dynamic parameters and values, we are making use of the Extract and Navigate functionality. For more details on this please refer to this article “Muse Proxy Source Profiling: A Complex Username/Password Authentication Scenario”.

    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&submit=Login%21]]>
    </POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in
    to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
  3. Select the English language:
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
  4. Click on the UK – United Kingdom link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
  5. Click on the Contract notice link:
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
  6. The complete XML profile is given below:
    <ICE-CONFIG>
    <URL>https://webgate.ec.europa.eu/cas/login</URL>
    <EXTRACTOR ref="step1" refProcess="xmlUnescape"><![CDATA[<a href="(https://webgate.ec.europa.eu/cas/wayf.cgi\?
    loginRequestId=[^"]+?)"\sclass="main_domain external_hover"\stitle="External"><span class="external"></span>]]></EXTRACTOR>
    <URL>${step1_1}</URL>
    <EXTRACTOR ref="postURL">id="loginForm"\sname="strongLoginForm"\saccept-charset="UTF-8"\saction="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="TxId">type="hidden"\sname="TxId"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="lt">type="hidden"\sname="lt"\svalue="([^"]+?)"</EXTRACTOR>
    <EXTRACTOR ref="loginRequestId">type="hidden"\sname="loginRequestId"\svalue="([^"]+?)"</EXTRACTOR>
    <URL>${postURL_1}</URL>
    <POST_PARAMETERS><![CDATA[TxId=${TxId_1}&lt=${lt_1}&domain=external&loginRequestId=${loginRequestId_1}&useDefaultStrength=true&
    timeZone=GMT%2B03%3A00&selfHost=webgate.ec.europa.eu&username=${userName}&zpassword=&password=${userPassword}&
    submit=Login%21]]></POST_PARAMETERS>
    <EXTRACTOR ref="lastURL"><![CDATA[<a\sid="fallbackLink"\shref="(https://webgate.ec.europa.eu/cas/login\?[^"]+?)"]]></EXTRACTOR>
    <URL>${lastURL_1}</URL>
    <URL>http://ted.europa.eu</URL>
    <EXTRACTOR ref="authRedir"><![CDATA[url=(https://webgate.ec.europa.eu/cas/redirecting-to/[^"]+?)"\s/>\s+<p>You are now logged in to ECAS.</p>]]></EXTRACTOR>
    <URL>${authRedir_1}</URL>
    <URL>http://ted.europa.eu/TED/browse/browseByBO.do</URL>
    <POST_PARAMETERS><![CDATA[action=cl&lgId=en]]></POST_PARAMETERS>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?action=loadRightTree&amp;refDataId=COUNTRY&amp;key=UK</URL>
    <URL>http://ted.europa.eu/TED/browse/ajaxBrowse.do?refDataId=DOCUMENT_TYPE&amp;key=3</URL>
    <POST_PARAMETERS><![CDATA[action=selectFacet&quickSearchCriteria=&searchScope=LAST_EDITION]]></POST_PARAMETERS>
    <SHOW_GET_PARAMETERS>true</SHOW_GET_PARAMETERS>
    <USER_AGENT>Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/43.0.2357.132 Safari/537.36</USER_AGENT>
    <CONNECT_TIMEOUT>60000</CONNECT_TIMEOUT>
    <READ_TIMEOUT>60000</READ_TIMEOUT>
    <NAME>TED Tenders Electronic Daily</NAME>
    <DESCRIPTION>TED provides free access to business opportunities. It is updated 5 times a week with some 1,500 public
    procurement notices from the European Union, the European Economic Area and beyond. You can browse, search and sort procurement
    notices by country, region, business sector and more. Information about every procurement document is published in the 24 official
    EU languages. All notices from the EU's institutions are published in full in these languages.</DESCRIPTION>
    <AUTHENTICATION_TYPE>User/Password</AUTHENTICATION_TYPE>
    <HTTP_AUTHORIZATION_USER_NAME/>
    <HTTP_AUTHORIZATION_USER_PASSWORD/>
    <HTTP_AUTHORIZATION_SCHEME/>
    <PROXY_USED>SOURCE_LEVEL</PROXY_USED>
    <PROXY_HOST></PROXY_HOST>
    <PROXY_PORT></PROXY_PORT>
    <PROXY_PAC></PROXY_PAC>
    <PROXY_AUTHORIZATION_USER_NAME/>
    <PROXY_AUTHORIZATION_USER_PASSWORD/>
    <PROXY_AUTHORIZATION_SCHEME/>
    <REWRITING_PATTERNS>*europa.eu;*/*.js*;</REWRITING_PATTERNS>
    <REWRITE_BY_HOST>true</REWRITE_BY_HOST>
    <ENCODING>UTF-8</ENCODING>
    <REFERER></REFERER>
    <COOKIES></COOKIES>
    <REPLACE_HOST/>
    <REPLACE_PATH></REPLACE_PATH>
    <SSL_CERTIFICATES/>
    <SSL_ALIASES/>
    <FOLLOW_REDIRECTS>true</FOLLOW_REDIRECTS>
    <PARAMETERS>
    <PARAMETER>
    <NAME>userName</NAME>
    <VALUE>YYY</VALUE>
    </PARAMETER>
    <PARAMETER>
    <NAME>userPassword</NAME>
    <VALUE>ZZZ</VALUE>
    </PARAMETER>
    </PARAMETERS>
    <LAST_UPDATED>2015-07-14</LAST_UPDATED>
    </ICE-CONFIG>