Google Search Appliance (GSA) Crawl Proxy

Readme

Google Search Appliance is able to natively crawl secure content coming from multiple sources using for instance the following main methods:

• HTTP Basic/NTLM authentication: if your backend content systems either support HTTP Basic or NTLM (Windows native authentication), you can create crawling rules to let the appliance access securely the documents.
• Cookie sites: if the access to your content servers is protected by cookie, you can define cookie rules that authenticate the crawler against some URL patterns. If any URL matches with any of these rules, the appliance will send cookies back to the server where documents are.
• Forms based authentication: if you have a Single Sign-On system like CA Siteminder (aka Netegrity) or Oracle Access Manager (aka Oblix) protecting applications, a Forms Based crawling rule can be used. The appliance will recover the content if the right credentials are provided exactly the same way as a user is accessing those SSO protected applications. As these systems are driven by a cookie, the appliance can manage that SSO authentication cookie to know if the user is already authenticated or not. As of today the appliance only supports one Forms Based authentication crawling rule.

The above offering is useful in most cases but in some others we would need to use an extended functionality. This is usually occurring when we have some content sources that are not able to manage any of the above security mechanisms or we have more than one Forms based authentication solution, not being protected by a central SSO server.

GSA Crawl Proxy, this open source project, is meant to extend the crawling possibilities taking the advantage of the connectivity provided by the AuthN/AuthZ modules for the GSA Valve Security Framework . These modules implement the integration complexity to securely access to the content sources where documents are. The GSA Crawl Proxy is able to authenticate the crawler user coming from the search appliance using HTTP Basic, and send those credentials to any of these modules that can convert them to any security mechanism the content servers would understand.

The integration between the GSA and this crawling tool is done configuring a proxy access in the appliance. This application is acting as a proxy that gets the crawling requests from the appliance and returns back the result, either the document's content if the request was successful or an HTTP error code if does not.

Latest Changes

Release 1.0 - June 18 2008

• Initial version

More Information

Checks the documentation available at the open source project site at code.google.com to get more information and the instructions on how to deploy this application.