[BUG] “Unexpected Response” Access Gateway Enterprise in NetScaler build 9.3-53.5

If you are using Citrix Receiver on iOS or Android to access pulished applications on Citrix Access Gateway Enterprise (NetScaler) do not upgrade to the latest firmware which is 9.3-53.5.

Users will get an “Unexpected Response” on an iOS device when they try to login to a Citrix Access Gateway environment. On Android the error statement is “The Citrix Access Gateway you are connecting to is not configured for this device. Please contact your administrator."

On the Citrix Forum there is a post where more users are reporting this issue: http://forums.citrix.com/thread.jspa?threadID=297625

Citrix is aware of the issue and are working on it.

Update: Problem is resolved as of build 9.3-54.4 or newer.

Unexpected_Error

Configure RSA RADIUS monitoring on NetScaler

Ok, so this one is pretty easy and speaks for itself for the most part but can have some pitfalls while configuring.

In this example we are going to assume that the RSA backend is already in place and functioning properly.

Why

A good question is why would you want to. When you leave the monitoring to default which would be tcp-default it marks the server up as it response to a tcp connection. That would simply not be enough, we need to make sure that the RSA RADIUS Service is actually up and ready to accept connections.

Prerequisites

Make sure you have an account configured in RSA Authentication Manager that is authorized to send the response that you want to retrieve from the RSA Radius service. More on the RADIUS Response codes later on and why you would like to use a valid account.

Next to the account we are assuming that authentication is working properly and the Citrix NetScaler IP is added in RSA as a host that is allowed the use RSA Radius Authentication.

The Basics (load balancing)

Create a load balanced Virtual Server where the (in our case) two RSA Servers reside (The IP addresses mentioned is off course to be adjusted accordingly).

add server rsa01.domain.local 192.168.0.1
add server rsa02.domain.local
192.168.0.2

add serviceGroup sg_radius_rsa_001 RADIUS -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport NO -cltTimeout 120 -svrTimeout 120 -CKA NO -TCPB NO -CMP NO -appflowLog DISABLED

add lb vserver lb_vs_rsa_001 RADIUS 192.168.0.3 1812 -persistenceType NONE -cltTimeout 120

bind serviceGroup sg_radius_rsa_001 rsa01.domain.local 1812
bind serviceGroup sg_radius_rsa_001 rsa02.domain.local 1812

Creating the Monitor (Standard Parameters)

Create a monitor as displayed below. Important things to configure is the Type (RADIUS) and Destination Port (1812) on the Standard Parameters tab.

Note: I had to change the Response Time-out setting to 4 (default is 2) since the response took longer then 2 seconds to be received by Citrix NetScaler (This something to check).

image

In the screenshot below you can see that the response takes a bit more then 2 seconds to be received meaning we had to adjust the Response time-out within Monitor => Standard Parameters.

image

Creating the Monitor (Special Parameters)

Under the Special Parameters put in the account credentials that is created within RSA Authentication Manager that will pass the proper response. As a Response Code we have chosen to go with 2 which is Access-Accept.

image

add lb monitor radius-rsa RADIUS -respCode 2 -userName netscaler_monitoring -password dc0b445466e821 -encrypted -radKey e93d11543846hf963ef -encrypted -LRTM ENABLED -resptimeout 4 -destPort 1812

RADIUS Response Codes

There are different types of RADIUS Response Codes, see the table below. When you cannot or don’t want to create a user account within RSA Authentication Manager then you will not be able to receive a Code 2 (Access-Accept) response code and are not able to verify if the entire Authentication chain is actually working. That would mean Citrix NetScaler will just send Authentication Requests to the RSA Service even though the database is down but RADIUS still receives Access-Reject so according to Citrix NetScaler the server is up and running and ready to accept Authentication requests.

image

More information on RADIUS: http://en.wikipedia.org/wiki/RADIUS

Publish RSA Self-Service Console through NetScaler

This week I was at a customer which would like to publish the RSA Self Service Console so that users can self-service their RSA tokens, passwords and accounts and create some sort of redundancy with multiple RSA Authentication Servers. RSA has limited documentation on publishing the RSA Self-Service Console using a reverse proxy, especially Citrix NetScaler.

First of all, what you need to be aware of is that the RSA Servers works in a Primary/Replica model in which only the Primary can be written to by users, all other RSA Servers are read-only replica’s. So you can not use the replica servers for changing tokens, resetting passwords or enabling accounts. Replica’s can only be used for authenticating purposes.

image

What does publishing RSA Self Service Console through Citrix NetScaler involve:

- HTTPS RSA Self Service servers (in my example I use two: rsa01.domain.local and rsa02.domain.local):
- NetScaler Load Balancing (Active-Passive);
- Redirect using Rewrite rules;
- Rewrite Header content;
- Rewrite Body conent;
- Monitoring (this an is optional).

Prerequisites

We are going to assume that the HTTPS RSA Self-Service servers are already configured in the backend and that these are reachable by Citrix NetScaler. The RSA Self-Service Console uses port 7004 and have to be approached by this port on the outside. As far as I can see this cannot be changed due to hard redirection by the RSA software.

We are going to assume an SSL certificate is installed for binding to the external DNS name that is being used for publishing the RSA Self-Service Console to the public network. We’ll be using the fake name rsa.domain.public in this post.

The Basics (building RSA redundancy)

Create the Servers for the RSA Self Service Console Servers under Load Balancing => Servers:

add server rsa01.domain.local 192.168.0.1
add server rsa02.domain.local 192.168.0.2

Now, remember that RSA cannot be active-active load balanced due to the read-only replica’s so that means we will have to create a Service for each RSA Server, do not create a ServiceGroup.

add service svc_selfservice_rsa_001 rsa01.domain.local SSL 7004 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport NO -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

add service svc_selfservice_rsa_002 rsa02.domain.local SSL 7004 -gslb NONE -maxClient 0 -maxReq 0 -cip DISABLED -usip NO -useproxyport NO -sp OFF -cltTimeout 180 -svrTimeout 360 -CKA NO -TCPB NO -CMP NO

Create a load balanced Virtual Server and bind the above first created Service svc_selfservice_rsa_001 to this Virtual Server (Remember to change the IP-addresses accordingly).

add lb vserver lb_vs_rsa_ss SSL 192.168.0.3 7004 -persistenceType COOKIEINSERT -timeout 0 -cltTimeout 180

Create a second load balanced Virtual Server and bind the above second created Service svc_selfservice_rsa_002 to this Virtual Server (Do not assign an IP-Address to this vServer by de-selecting "Directly Addressable”).
Note the -persistenceType COOKIEINSERT -timeout 0, this is very important or you could get immediate Authentication error statements. Bind an SSL certificate to this vServer.

add lb vserver lb_vs_rsa_ss_failover SSL 0.0.0.0 0 -persistenceType COOKIEINSERT -timeout 0 -cltTimeout 180

Note the -persistenceType COOKIEINSERT -timeout 0, this is very important or you could get immediate Authentication error statements. Bind an SSL certificate to this vServer.

Open the properties of the lb_vs_rsa_ss vServer, go to the Advanced and select lb_vs_rsa_ss_failover under Backup Virtual Server.

set vserver lb_vs_rsa_ss –backupVServer lb_vs_rsa_ss_failover

Rewrite (Redirect)

We would like users to be able to just type rsa.domain.public without using explicit port numbers or sub paths, this is why we redirect the incoming connection of http://rsa.domain.public to https://rsa.domain.public:7004/console-selfservice/.

What we basically want is a two step procedure:
1. Redirect from http to https and the correct port number which is 7004;
2. Redirect to the sub path /console-selfservice.

First step is to create a load balanced Virtual Server for the RSA publication on http with no services or serviceGroups bound to it. Go to the Advanced page and enter https://rsa.domain.public:7004 under “Redirect URL”.
(This Service will be marked DOWN, but that’s OK).

Second step is creating a Rewrite action and policy that will attach the subpath to the URL. This will look like this in the GUI.

image

image

The above screenshots will look like this on the CLI.

add rewrite action rw_ac_rsa_ss replace HTTP.REQ.URL "\"/console-selfservice\""
add rewrite policy rw_pol_rsa_ss "HTTP.REQ.URL.EQ(\"/\")" rw_ac_rsa_ss

Bind the Rewrite Policies to the newly created vServer (The one where we made the URL Redirect under Advanced and is marked DOWN).

bind lb vserver lb_vs_rsa_ss_failover -policyName rw_pol_rsa_ss -priority 100 -gotoPriorityExpression NEXT -type REQUEST

Rewrite (REQ Host Header)

add rewrite action act_replace-host-header-rsa01 replace_all "HTTP.REQ.HEADER(\"Host\")" "\"rsa.domain.public\"" -pattern sa01.domain.local
add rewrite policy pol_replace-host-header-rsa01 true act_replace-host-header-rsa01

add rewrite action act_replace-host-header-rsa02 replace_all "HTTP.REQ.HEADER(\"Host\")" "\"rsa.domain.local\"" -pattern rsa02.domain.local
add rewrite policy pol_replace-host-header-rsa02 true act_replace-host-header-rsa02

Rewrite (REQ BODY)

add rewrite action rw_act_req_rsa01 replace_all "HTTP.REQ.BODY(200000)" "\"rsa.domain.local\"" -pattern rsa01.domain.local

add rewrite policy rw_pol_req_rsa01 "http.req.body(200000).contains(\"rsa01.domain.local\")" rw_act_req_rsa01

Rewrite (RES Host Header)

add rewrite action act_replace-location-header-rsa01 replace_all "HTTP.RES.HEADER(\"Location\")" "\"rsa.domain.public\"" -pattern rsa01.domain.local
add rewrite policy pol_replace-location-header-rsa01 true act_replace-location-header-rsa01

add rewrite action act_replace-location-header-rsa02 replace_all "HTTP.RES.HEADER(\"Location\")" "\"rsa.domain.local\"" -pattern rsa02.domain.local
add rewrite policy pol_replace-location-header-rsa02 true act_replace-location-header-rsa02

Rewrite (RES BODY)

add rewrite action rw_act_res_rsa01 replace_all "HTTP.RES.BODY(200000)" "\"rsa.domain.public\"" -pattern rsa01.domain.local
add rewrite policy rw_pol_res_rsa01 "http.res.body(200000).contains(\"rsa01.domain.local\")" rw_act_res_rsa01

add rewrite action rw_act_res_rsa02 replace_all "HTTP.RES.BODY(200000)" "\"rsa.domain.public\"" -pattern rsa02.domain.local
add rewrite policy rw_pol_res_rsa02 "http.res.body(200000).contains(\"rsa02.domain.local\")" rw_act_res_rsa02

Monitoring (Optional)

Ok, this is optional. Select https and click Add under Load Balancing => Monitor. Fill in the fields like below.
(Type: HTTP, Destination Port: 7004, Secure (checked).
I’ve have increased the Response Time-out and Interval on purpose, because I really want to make sure the primary RSA Server is down before doing a fail-over because of the Primary/Replica model.

image

Under tab Special Parameters make sure you have filled in “"HEAD /console-selfservice/” and as a response code 302 because of a hard redirect that is being made by RSA Software.

image

add lb monitor https-self-service-rsa HTTP -respCode 302 -httpRequest "HEAD /console-selfservice/" -LRTM ENABLED -interval 20 -resptimeout 10 -destPort 7004 -secure YES

Bind the newly created monitor to both of the created RSA Services.

bind lb monitor https-self-service-rsa svc_selfservice_rsa_002
bind lb monitor https-self-service-rsa svc_selfservice_rsa_001

Access Gateway: “Corrupted Content Error” when using FF7 or higher

“Corrupted Content Error” when logging on to a Citrix Access Gateway Enterprise/Web Interface for NetScaler with Mozilla Firefox 7 or Higher

I’m not yet sure if this is a Citrix issue or a Firefox issue since this problem occurs with a lot more websites with the new Firefox versions. On Citrix Access Gateway Enterprise it seems to occur when using Citrix Access Gateway Enterprise (NetScaler) and Citrix Web Interface for NetScaler (have not yet tested with Web Interface for Windows). Apparently Firefox does not like the fact that either the header Content-Length and/or Content-Disposition values change when passing from the Access Gateway to the Web Interface.

Firefox says web developers need to change there code, I have submitted a support case with Citrix Tech Support. Will let you know what the outcome will be for those interested.

Update: Citrix has confirmed this issue and is working on a solution which will be resolved in the next Citrix NetScaler firmware release. This is not a bug but a “security enhancement” from Firefox.

Update: Citrix Tech Support provided a temporary solution for this problem, copy and paste the rewrite action and policy into an SSH session and bind it globaly.

add rewrite action ReplaceContentLengthZeroHeader_act replace “HTTP.RES.FULL_HEADER.REGEX_SELECT(re!Content-Length: 0!)” “\”Cotnent-Length: 0\”"

add rewrite policy ReplaceContentLengthZero_pol “HTTP.RES.STATUS.EQ(302) && HTTP.RES.FULL_HEADER.REGEX_MATCH(re!Content-Length: 0!) && HTTP.RES.FULL_HEADER.REGEX_MATCH(re!Location: .*/login.jsp!)” ReplaceContentLengthZeroHeader_act

bind rewrite global ReplaceContentLengthZero_pol 10 END -type RES_DEFAULT

image

[BUG] Citrix NetScaler Build 51.5: GUI Secure Only not working through SSL/VPN

Citrix NetScaler GUI console reachable through http while “Secure Only” is enabled when using an SSL/VPN connection.

To make sure your NetScaler implementation is compliant with PCI-DSS standards you have to make sure you cannot approach the Management Console in a non-encrypted manner. During my most recent implementations using Build 51.5nc I noticed that although having enabled this option I can still access the GUI through an unencrypted http connection instead of https when connected by an Access Gateway SSL/VPN connection.

Citrix Support has confirmed this bug and is working on a solution.

1

How to Configure Client Detection and Deployment on Citrix Web Interface for NetScaler

Edit: Updated this post for deployment using the new Citrix Receiver for Windows 3.0 and Citrix Receiver for Mac 11.4

When you are here after searching the web you have probably found out that the Web interface for NetScaler doesn’t nicely deploy (push) the Citrix client as you are used to from Citrix Web Interface for Windows. Here I am going to explain how you can deploy the Citrix Client from Web interface for NetScaler. At least for Windows and Mac. This blog post was based on a document I got from Andrew Sandford, tech support Citrix Systems Ireland. They are working on an official KB article to address this problem. In the mean time I hope this can help some people out.

1. Use WinSCP (or puTTY if you feel comfortable using commandline and tools like vi) to enter the Citrix NetScaler;

2. Go to you Citrix Web Interface directory, in my example I use /Citrix/XenApp, the default first site on Citrix NetScaler: /var/wi/tomcat/webapps/Citrix/XenApp;

3. Create a Clients directory within the XenApp folder ( #mkdir Clients within the Linux shell)
    Under the newly created Clients directory create the following directories:
    Windows (#mkdir Windows within the Linux shell)
    Mac ( #mkdir Mac within the Linux shell)
    Under Windows create the following directories:
    Online plug-in (#mkdir “Online plug-in” within the Linux shell, quote or you will get 2 directories)
    Offline plug-in (#mkdir “Offline plug-in” within the Linux shell, quote or you will get 2 directories)

4. Download the latest Citrix Web clients from the Citrix website.

5. Copy the downloaded CitrixReceiver.exe file with WinSCP to: /var/wi/tomcat/webapps/Citrix/XenApp/Clients/Windows/Online plug-in/
Copy the downloaded CitrixOfflinePlugin.exe file with WinSCP to: /var/wi/tomcat/webapps/Citrix/XenApp/Clients/Windows/Offline plug-in/
Copy the downloaded CitrixReceiver_11.4.dmg file with WinSCP to: /var/wi/tomcat/webapps/Citrix/XenApp/Clients/Mac/

6. Uncomment and edit the following line in /var/wi/tomcat/webapps/Citrix/XenApp/WEB-INF/WebInterface.conf:
# ClientIcaLinuxX86=Filename:linuxx86-11.0.140395.tar.gz,Directory:Linux,Mui:Yes,Version:11.0
ClientIcaMac=Filename: CitrixReceiver_11.4.dmg,Directory:Mac,Mui:Yes
# ClientIcaSolarisSparc=Filename:solaris.tar.Z,Directory:Unix,Mui:No,Version:8.59
# ClientIcaSolarisX86=Filename:solx86.tar.Z,Directory:Unix,Mui:No,Version:8.5
ClientIcaWin32=Filename:CitrixReceiver.exe,Directory:Windows,Mui:Yes,ClassID:238f6f83-b8b4-11cf-8771-00a024541ee3
ClientStreamingWin32=Filename:CitrixOfflinePlugin.exe,Directory:Windows,Mui:Yes,ClassID:4384F3C5-4A9E-4E81-9AAE-4251C2813861

7. Enable autodetect by uncommenting and editing the following /var/wi/tomcat/webapps/Citrix/XenApp/WEB-INF/WebInterface.conf line:
EnableWizardAutoMode=On

8. Restart the Java Webserver by entering the following Linux command:
#killall java
It will take about 30 seconds for the webserver to be up and running again, if you are impatient (like me Smile) you can monitor if the java webserver has started again by entering the following command:
#ps aux | grep java

Depending on the browser being used and existing policies it should be possible that the url has to be added to the Trusted Sites within Internet Explorer. With Internet Explorer 9 you will have to add the url anyway or users will get the download ICA file notifications when trying to start Citrix published applications.

image

The new Citrix Receiver for Windows 3.0:

Citrix Receiver 3.0-1

image

Citrix Allowing more Web Interface Sites on NetScaler

As stated in one of my previous blogs I mentioned that it wasn’t possible to install more then 3 Citrix Web Interface sites on for instance a Citrix NetScaler MPX 5500. If you wanted more then you would have to move to the 7500 series minimum where the limit was 25 Citrix Web Interface sites. For a lot of customers the price difference between the MPX 5500 and the MPX 7500 is not worth just a single Citrix Web interface site.

Appearently Citrix has changed his mind about these hard limits! There was ofcourse a huge gap between 3 and 25 Citrix Web Interface sites and Citrix Support explanation on why this gap existed was that the webserver used for publishing the Citrix web Interface sites could be a huge resource hog and the Citrix NetScaler MPX 7500 is equipped with better/more hardware to facilitate these number of Citrix Web Interface websites. Could be a valid reason ofcourse but you can’t tell me Citrix didn’t limit to 3 Citrix Web interface sites on purpose (marketing).

Ok, so as of now when you reach the limit of you appliance (virtual or physical) you will get the following message:

3-25

Now, happy as I am I clicking Yes, getting the following message:

MemNoEnough

This is my (for testing purposes) Citrix NetScaler VPX 1000 (9.3-48.6) with 2 Gb of RAM assigned to it. Now, let’s do what the message says and look in the Command Log:

Command_Log

Hm, nothing about memory shortage or how much of it. Ok, shutdown VM and assign 4 Gb to the VM and boot it up. Let’s create the 4th Citrix Web Interface, bingo:

4_Sites_Created

So giving your VM enough memory allows you to create more Citrix Web Interface sites. I had already tested this on multiple Citrix NetScaler MPX 5500 and so I know it worked on the hardware models.

Now, how about creating 26 websites (25 being the limit) and changing it to 500 on my Citrix NetScaler VPX 1000 with 4 Gb of RAM Smile. That did not work:

25-500-MemNoEnough

It seems Citrix has decided to change the hard limit per appliance to the amount of memory present in the Citrix NetScaler, I was able to change 25 to 50 on my testing machine though.

I’ve downloaded the Citrix NetScaler 9.3 Admin Guide and there it still mentions the (old) hard limits per appliance model. So I do not yet know what Citrix’s official statement is on this but I think they changed there minds about seeing the performance results of running multiple Citrix Web Interface sites on the more entry or virtual level models and decided the amount of memory should be leading.

Good job Citrix!

Update 6/27/2011: I’ve tested the above setting on the new 9.3 Build 49.5 and it still works, after altering the memory from 2 GB to 4 GB I was able to set the limit to 50 sites. Although the upgrade removed my Citrix Web interface so remember to backup you configs and customizations just in case.

Maximum Access Gateway Users Allowed remains at 0 after changing hostname on a Citrix Access Gateway Enterprise

First off I’m not a big fan of changing the default hostname ‘ns’ of a Citrix NetScaler/AGEE, this creates potential problems on moments when you cannot use them. For instance with a situation where you have to rebuild a Citrix Access Gateway Enterprise as soon as possible to restore user productivity as soon as possible.

Citrix NetScaler and Citrix Access Gateway Enterprise are pre-configured with the hostname ns and there really is no reason to change this. This hostname is not being broadcasted to other machines in any way. So it will not lead to conflicts with other systems on your network.

But offcourse there will always be customers who want the hostnames altered no matter what! Today I had run into a little problem with a customer who has changed the default ns name to an alternate hostname.

Whilst setting the correct hostname and installing the license files the count remains at 0 for the Maximum Access Gateway Users Allowed.

1

After a little investigation it became clear that the /etc/hosts file still has the “127.0.0.1 ns” mapping and not the new hostname which offcourse is present in the license files. Now, on a Citrix NetScaler you cannot edit the /etc/hosts file directly. Any change you make in this file will be lost when rebooted.

The solution is as follows, create an rc.conf file in the /nsconfig directory with an editor of choice (vi) and add the following entry hostname=<hostname>

This will add the correct hostname to the /etc/hosts file and from there on license files will be read correctly and applied.

OWA 2010 freezes in Citrix Access Gateway portal page

Within Citrix Access Gateway you can configure the E-mail tab to point to your Microsoft Exchange Outlook Web Access page so that your users can go to their webbased mail from within the Citrix Access Gateway portal page.

Officially Citrix does not support the integration of OWA 2010 within the Citrix portal page, and if I’m not mistaken OWA 2007 isn’t officialy supported either. Although you can get OWA 2007 to work within the Access Gateway portal page without much effort.

OWA 2010 is a little bit different, Microsoft changed the OWA backend to a heavier interface (Premium) in which there are more possibilities. OWA 2010 also still has the Light version (formerly known as Basic) for the older Web browsers like Internet Explorer 6.

Now when you configure OWA 2010 within the Citrix Access Gateway portal page and try to open it with IE7 or IE8 you will (probably) see that it freezes and you will have to close the browser because it is not responding.

What it does is, it tries to open the premium OWA 2010 interface because it sees the browser version as IE7 or IE8 capable. You can see your mailbox just for a brief moment if you look real fast, now, isn’t that frustrating Smile. There seems to be some kind of Javascript refresh somewhere in the code which Citrix Access Gateway doesn’t like. Funny detail is that the Premium version does work with Firefox (all versions) no problem! When you use IE6 it opens up the Light version and that works correct also, but you will have a tough time finding users that still actually use IE6.

The only resolution I have found this far is to force the OWA 2010 to Light View from within Exchange. You could make a separate OWA Virtual Directory for use with the Access Gateway. I think this is a very acceptable solution for a lot of customers since having the premium version within Citrix Access Gateway does not add that much value.

You can force the OWA Virtual Directory to Light View by executing the following PowerShell command:
Set-owavirtualdirectory –id:<id of virtual directory> -PremiumClientEnabled:$false

It is also possible to set the Light View per specific user:
Set-CasMailbox –id:<id of mailbox> -OWAPremiumClientEnabled:$false