Technology related articles from The Plan Collection

Search Using ASP.Net and Lucene

2008-03-11T05:24:03Z

Getting Started

Getting Lucene to work on your asp.net website isn't hard but there are a few tricks that help. We decided to use the Lucene.Net 2.1.0 release because it made updating the Lucene index easier via a new method on the IndexWriter object called UpdateDocument. This method deletes the specified document and then adds the new copy into the index. You can't download the Lucene.net 2.1.0 binary. Instead you will need to download the source via their subversion repository and then compile it.

Don't worry this is an easy step. Using your subversion client - I recommend TortiseSvn get the source by doing a checkout from this url: https://svn.apache.org/repos/asf/incubator/lucene.net/tags/Lucene.Net_2_1_0/

Next go into the directory: Lucene.Net_2_1_0\src\Lucene.Net You should find a Visual Studio solution file that matches your Visual Studio version. If you are using Visual Studio 2005 be sure to load Lucene.Net-2.1.0-VS2005.sln

Hit compile. The resulting Lucene.Net.dll in the bin/release folder is the dll you will need to reference in your Visual Studio project that will contain the Lucene code.

Creating the Lucene Index

Lucene creates a file based index that it uses to quickly return search results. We had to find a way to index all the pages in our system so that Lucene would have a way to search all of our content. In our case this includes all the articles, forum posts and of course house plans on the website. To make this happen we query our database, get back urls to all of our content and then send a webspider out to pull down the content from our site. That content is then parsed and fed to Lucene.

We developed three classes to make this all work. Most of the code is taken from examples or other kind souls who shared code. The first class, GeneralSearch.cs creates the index and provides the mechanism for searching it. The second class, HtmlDocument consists of code taken from Searcharoo a web spidering project written in C#. The HtmlDocument class handles parsing the html for us. Special thanks to Searcharoo for that code. I didn't want to write it. The last class is also borrowed from Searcharoo. It is called HtmlDownloader.cs and its task is to download pages from the site and then create a HtmlDocument from them.

GeneralSearch.cs

using System;
using System.Collections.Generic;
using System.Data;
using Core.Utils.Html;
using Lucene.Net.Analysis;
using Lucene.Net.Analysis.Standard;
using Lucene.Net.Documents;
using Lucene.Net.Index;
using Lucene.Net.QueryParsers;
using Lucene.Net.Search;

namespace Core.Search
{
    /// <summary>
    /// Wrapper for Lucene to perform a general search
    /// see:
    /// http://www.codeproject.com/KB/aspnet/DotLuceneSearch.aspx
    /// for more help about the methods used in this class
    /// </summary>
    public class GeneralSearch
    {
        private IndexWriter _Writer = null;
        private string _IndexDirectory;
        private List<string> _Errors = new List<string>();
        private int _TotalResults = 0;
        private int _Start = 1;
        private int _End = 10;
        
        /// <summary>
        /// General constructor method.
        /// </summary>
        /// <param name="indexDirectory">The directory where the index 
        /// is located.</param>
        public GeneralSearch(string indexDirectory)
        {
            _IndexDirectory = indexDirectory;                                    
        }
                
        /// <summary>
        /// List of errors that occured during indexing
        /// </summary>
        public List<string> Errors
        {
            get { return _Errors; }
            set { _Errors = value; }
        }

        /// <summary>
        /// Total number of hits return by the search
        /// </summary>
        public int TotalResults
        {
            get { return _TotalResults; }
            set { _TotalResults = value; }
        }

        /// <summary>
        /// The number of the record where the results begin.
        /// </summary>
        public int Start
        {
            get { return _Start; }
            set { _Start = value; }
        }

        /// <summary>
        /// The number of the record where the results end.
        /// </summary>
        public int End
        {
            get { return _End; }
            set { _End = value; }
        }

        /// <summary>
        /// Returns a table with matching results or null
        /// if the index does not exist.  This method will page the
        /// results.
        /// </summary>
        /// <param name="searchText">terms to search for</param>
        /// <param name="currentPage">The current results page</param>
        /// <param name="hitsPerPage">The number of hits to return for each results page</param>
        /// <returns>A datatable containing the number of results specified for the given page.</returns>
        public DataTable DoSearch(string searchText, int hitsPerPage, int currentPage)
        {
            if(!IndexReader.IndexExists(_IndexDirectory))
            {
                return null;    
            }

            string field = IndexedFields.Contents;

            IndexReader reader = IndexReader.Open(_IndexDirectory);
            IndexSearcher searcher = new IndexSearcher(reader); 
            Analyzer analyzer = new StandardAnalyzer();
            QueryParser parser = new QueryParser(field, analyzer);
            Query query = parser.Parse(searchText);
            
            Hits hits = searcher.Search(query);

            DataTable dt = new DataTable();
            dt.Columns.Add(IndexedFields.Url, typeof(string));
            dt.Columns.Add(IndexedFields.Title, typeof(string));
            //dt.Columns.Add(IndexedFields.Summary, typeof(string));
            dt.Columns.Add(IndexedFields.Contents, typeof(string));
            dt.Columns.Add(IndexedFields.Image, typeof(string));

            if(currentPage <= 0)
            {
                currentPage = 1;
            }
            Start = (currentPage-1) * hitsPerPage;
            End = System.Math.Min(hits.Length(), Start + hitsPerPage);
            TotalResults = hits.Length();
            for (int i = Start; i < End; i++)
            {
                // get the document from index
                Document doc = hits.Doc(i);
                
                DataRow row = dt.NewRow();
                row[IndexedFields.Url] = doc.Get(IndexedFields.Url);
                //row[IndexedFields.Summary] = doc.Get(IndexedFields.Summary);
                row[IndexedFields.Contents] = doc.Get(IndexedFields.Contents);
                row[IndexedFields.Title] = doc.Get(IndexedFields.Title);
                row[IndexedFields.Image] = doc.Get(IndexedFields.Image);

                dt.Rows.Add(row);
            }

            reader.Close();

            return dt;
        }

        /// <summary>
        /// Opens the index for writing
        /// </summary>
        public void OpenWriter()
        {
            bool create = false;
            if (!IndexReader.IndexExists(_IndexDirectory))
            {
                create = true;
            }

            _Writer = new IndexWriter(_IndexDirectory, new StandardAnalyzer(), create);
            _Writer.SetUseCompoundFile(true);
            _Writer.SetMaxFieldLength(1000000);
        }

        /// <summary>
        /// Closes and optimizes the index
        /// </summary>
        public void CloseWriter()
        {
            _Writer.Optimize();
            _Writer.Close();
        }

        /// <summary>
        /// Loads, parses and indexes an HTML file at a given url.
        /// </summary>
        /// <param name="url"></param>
        public void AddWebPage(string url)
        {
            HtmlDocument html = HtmlDownloader.Download(url);

            if (null != html)
            {
                // make a new, empty document
                Document doc = new Document();

                // Store the url
                doc.Add(new Field(IndexedFields.Url, url, Field.Store.YES, Field.Index.UN_TOKENIZED));

                // create a uid that will let us maintain the index incrementally
                doc.Add(new Field(IndexedFields.Uid, url, Field.Store.NO, Field.Index.UN_TOKENIZED));

                // Add the tag-stripped contents as a Reader-valued Text field so it will
                // get tokenized and indexed.
                doc.Add(new Field(IndexedFields.Contents, html.WordsOnly, Field.Store.YES, Field.Index.TOKENIZED));

                // Add the summary as a field that is stored and returned with
                // hit documents for display.
                //doc.Add(new Field(IndexedFields.Summary, html.Description, Field.Store.YES, Field.Index.NO));
                
                // Add the title as a field that it can be searched and that is stored.
                doc.Add(new Field(IndexedFields.Title, html.Title, Field.Store.YES, Field.Index.TOKENIZED));

                Term t = new Term(IndexedFields.Uid, url);
                _Writer.UpdateDocument(t, doc);
            }
            else
            {
                Errors.Add("Could not index " + url);
            }
        }

        /// <summary>
        /// Use this method to add a single page to the index.
        /// </summary>
        /// <remarks>
        /// If you are adding multiple pages use the AddPage method instead as it only opens and closes the index once.
        /// </remarks>
        /// <param name="url">The url for the given document.  The document will not be requested from
        /// this url.  Instead it will be used as a key to access the document within the index and 
        /// will be returned when the index is searched so that the document can be referenced by the
        /// client.</param>
        /// <param name="documentText">The contents of the document that is to be added to the index.</param>
        /// <param name="title">The title of the document to add to the index.</param>
        public void AddSinglePage(string url, string documentText, string title, string image)
        {
            OpenWriter();
            AddPage(url, documentText, title, image);
            CloseWriter();
        }

        /// <summary>
        /// Indexes the text of the given document, but does not request the document from the specified url.
        /// </summary>
        /// <remarks>
        /// Use this method to add a document to the index when you know it's contents and url.  This prevents
        /// a http download which can take longer.
        /// </remarks>
        /// <param name="url">The url for the given document.  The document will not be requested from
        /// this url.  Instead it will be used as a key to access the document within the index and 
        /// will be returned when the index is searched so that the document can be referenced by the
        /// client.</param>
        /// <param name="documentText">The contents of the document that is to be added to the index.</param>
        /// <param name="title">The title of the document to add to the index.</param>
        /// <param name="image">Image to include with search results</param>
        public void AddPage(string url, string documentText, string title, string image)
        {
            // make a new, empty document
            Document doc = new Document();

            // Store the url
            doc.Add(new Field(IndexedFields.Url, url, Field.Store.YES, Field.Index.UN_TOKENIZED));

            // create a uid that will let us maintain the index incrementally
            doc.Add(new Field(IndexedFields.Uid, url, Field.Store.NO, Field.Index.UN_TOKENIZED));

            // Add the tag-stripped contents as a Reader-valued Text field so it will
            // get tokenized and indexed.
            doc.Add(new Field(IndexedFields.Contents, documentText, Field.Store.YES, Field.Index.TOKENIZED));

            // Add the summary as a field that is stored and returned with
            // hit documents for display.
            //doc.Add(new Field(IndexedFields.Summary, documentDescription, Field.Store.YES, Field.Index.NO));

            // Add the title as a field that it can be searched and that is stored.
            doc.Add(new Field(IndexedFields.Title, title, Field.Store.YES, Field.Index.TOKENIZED));

            // Add the title as a field that it can be searched and that is stored.
            doc.Add(new Field(IndexedFields.Image, image, Field.Store.YES, Field.Index.TOKENIZED));

            
            Term t = new Term(IndexedFields.Uid, url);
            try
            {
                _Writer.UpdateDocument(t, doc);
            }
            catch(Exception ex)
            {
                Errors.Add(ex.Message);
            }
        }

        /// <summary>
        /// A list of fields available in the index
        /// </summary>
        public static class IndexedFields
        {
            public const string Url = "url";
            public const string Uid = "uid";
            public const string Contents = "contents";
            //public const string Summary = "summary";
            public const string Title = "title";
            public const string Image = "image";

        }
    }
}

HtmlDocument.cs

using System;
using System.Collections;
using System.Text.RegularExpressions;

namespace Core.Utils.Html
{
    
    /// <summary>
    /// This code was taken from:
    /// http://www.searcharoo.net/SearcharooV5/
    /// 
    /// Storage for parsed HTML data returned by ParsedHtmlData();
    /// </summary>
    /// <remarks>
    /// Arbitrary class to encapsulate just the properties we need 
    /// to index Html pages (Title, Meta tags, Keywords, etc).
    /// A 'generic' search engine would probably have a 'generic'
    /// document class, so maybe a future version of Searcharoo 
    /// will too...
    /// </remarks>
    public class HtmlDocument
    {
        #region Private fields: _Uri, _ContentType, _RobotIndexOK, _RobotFollowOK

        private int _SummaryCharacters = 350;
        private string _IgnoreRegionTagNoIndex = "";

        private string _All = String.Empty;
        private Uri _Uri;
        private String _ContentType;
        private string _Extension;
        private bool _RobotIndexOK = true;
        private bool _RobotFollowOK = true;
        private string _WordsOnly = string.Empty;
        /// <summary>MimeType so we know whether to try and parse the contents, eg. "text/html", "text/plain", etc</summary>
        private string _MimeType = String.Empty;
        /// <summary>Html &lt;title&gt; tag</summary>
        private String _Title = String.Empty;
        /// <summary>Html &lt;meta http-equiv='description'&gt; tag</summary>
        private string _Description = String.Empty;
        /// <summary>Length as reported by the server in the Http headers</summary>
        private long _Length;        
        
        #endregion

        public ArrayList LocalLinks;
        public ArrayList ExternalLinks;

        #region Public Properties: Uri, RobotIndexOK
        /// <summary>
        /// http://www.ietf.org/rfc/rfc2396.txt
        /// </summary>
        public Uri Uri
        {
            get { return _Uri; }
            set
            {
                _Uri = value;
            }
        }
        /// <summary>
        /// Whether a robot should index the text 
        /// found on this page, or just ignore it
        /// </summary>
        /// <remarks>
        /// Set when page META tags are parsed - no 'set' property
        /// More info:
        /// http://www.robotstxt.org/
        /// </remarks>
        public bool RobotIndexOK
        {
            get { return _RobotIndexOK; }
        }
        /// <summary>
        /// Whether a robot should follow any links 
        /// found on this page, or just ignore them
        /// </summary>
        /// <remarks>
        /// Set when page META tags are parsed - no 'set' property
        /// More info:
        /// http://www.robotstxt.org/
        /// </remarks>
        public bool RobotFollowOK
        {
            get { return _RobotFollowOK; }
        }

        public string Title
        {
            get { return _Title; }
            set { _Title = value; }
        }

        /// <summary>
        /// Whether to ignore sections of HTML wrapped in a special comment tag
        /// </summary>
        public bool IgnoreRegions
        {
            get { return _IgnoreRegionTagNoIndex.Length > 0; }
        }

        public string ContentType
        {
            get
            {
                return _ContentType;
            }
            set
            {
                _ContentType = value.ToString();
                string[] contentTypeArray = _ContentType.Split(';');
                // Set MimeType if it's blank
                if (_MimeType == String.Empty && contentTypeArray.Length >= 1)
                {
                    _MimeType = contentTypeArray[0];
                }
                // Set Encoding if it's blank
                if (Encoding == String.Empty && contentTypeArray.Length >= 2)
                {
                    int charsetpos = contentTypeArray[1].IndexOf("charset");
                    if (charsetpos > 0)
                    {
                        Encoding = contentTypeArray[1].Substring(charsetpos + 8, contentTypeArray[1].Length - charsetpos - 8);
                    }
                }
            }
        }

        public string MimeType
        {
            get { return _MimeType; }
            set { _MimeType = value; }
        }

        public string Extension
        {
            get { return _Extension; }
            set { _Extension = value; }
        }
        
        #endregion

        #region Public fields: Encoding, Keywords, All
        /// <summary>Encoding eg. "utf-8", "Shift_JIS", "iso-8859-1", "gb2312", etc</summary>
        public string Encoding = String.Empty;
        /// <summary>Html &lt;meta http-equiv='keywords'&gt; tag</summary>
        public string Keywords = String.Empty;

        /// <summary>
        /// Raw content of page, as downloaded from the server
        /// Html stripped to make up the 'wordsonly'
        /// </summary>
        public string Html
        {
            get { return _All; }
            set
            {
                _All = value;
                _WordsOnly = StripHtml(_All);
            }
        }
        public string WordsOnly
        {
            get { return this.Keywords + this._Description + this._WordsOnly; }
        }

        public virtual long Length
        {
            get { return _Length; }
            set { _Length = value; }
        }

        public string Description
        {
            get
            {
                // ### If no META DESC, grab start of file text ###
                if (String.Empty == this._Description)
                {
                    if (_WordsOnly.Length > _SummaryCharacters)
                    {
                        _Description = _WordsOnly.Substring(0, _SummaryCharacters);
                    }
                    else
                    {
                        _Description = WordsOnly;
                    }
                    _Description = Regex.Replace(_Description, @"\s+", " ").Trim();
                }
                // http://authors.aspalliance.com/stevesmith/articles/removewhitespace.asp
                return _Description;
            }
            set
            {
                _Description = Regex.Replace(value, @"\s+", " ").Trim();
            }
        }
        #endregion

        #region Public Methods: SetRobotDirective, ToString()
        /// <summary>
        /// Pass in a ROBOTS meta tag found while parsing, 
        /// and set HtmlDocument property/ies appropriately
        /// </summary>
        /// <remarks>
        /// More info:
        /// * Robots Exclusion Protocol *
        /// - for META tags
        /// http://www.robotstxt.org/wc/meta-user.html
        /// - for ROBOTS.TXT in the siteroot
        /// http://www.robotstxt.org/wc/norobots.html
        /// </remarks>
        public void SetRobotDirective(string robotMetaContent)
        {
            robotMetaContent = robotMetaContent.ToLower();
            if (robotMetaContent.IndexOf("none") >= 0)
            {
                // 'none' means you can't Index or Follow!
                _RobotIndexOK = false;
                _RobotFollowOK = false;
            }
            else
            {
                if (robotMetaContent.IndexOf("noindex") >= 0) { _RobotIndexOK = false; }
                if (robotMetaContent.IndexOf("nofollow") >= 0) { _RobotFollowOK = false; }
            }
        }

        /// <summary>
        /// For debugging - output all links found in the page
        /// </summary>
        public override string ToString()
        {
            string linkstring = "";
            foreach (object link in LocalLinks)
            {
                linkstring += Convert.ToString(link) + "\r\n";
            }
            return Title + "\r\n" + Description + "\r\n----------------\r\n" + linkstring + "\r\n----------------\r\n" + Html + "\r\n======================\r\n";
        }
        #endregion


        /// <summary>
        ///
        /// </summary>
        /// <remarks>
        /// "Original" link search Regex used by the code was from here
        /// http://www.dotnetjunkies.com/Tutorial/1B219C93-7702-4ADF-9106-DFFDF90914CF.dcik
        /// but it was not sophisticated enough to match all tag permutations
        ///
        /// whereas the Regex on this blog will parse ALL attributes from within tags...
        /// IMPORTANT when they're out of order, spaced out or over multiple lines
        /// http://blogs.worldnomads.com.au/matthewb/archive/2003/10/24/158.aspx
        /// http://blogs.worldnomads.com.au/matthewb/archive/2004/04/06/215.aspx
        ///
        /// http://www.experts-exchange.com/Programming/Programming_Languages/C_Sharp/Q_20848043.html
        /// </remarks>
        public void Parse()
        {
            string htmlData = this.Html;    // htmlData will be munged

            //xenomouse http://www.codeproject.com/aspnet/Spideroo.asp?msg=1271902#xx1271902xx
            if (string.IsNullOrEmpty(this.Title))
            {   // title may have been set previously... non-HTML file type (this will be refactored out, later)
                this.Title = Regex.Match(htmlData, @"(?<=<title[^\>]*>).*?(?=</title>)",
                    RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture).Value;
            }

            string metaKey = String.Empty, metaValue = String.Empty;
            foreach (Match metamatch in Regex.Matches(htmlData
                , @"<meta\s*(?:(?:\b(\w|-)+\b\s*(?:=\s*(?:""[^""]*""|'[^']*'|[^""'<> ]+)\s*)?)*)/?\s*>"
                , RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture))
            {
                metaKey = String.Empty;
                metaValue = String.Empty;
                // Loop through the attribute/value pairs inside the tag
                foreach (Match submetamatch in Regex.Matches(metamatch.Value.ToString()
                    , @"(?<name>\b(\w|-)+\b)\s*=\s*(""(?<value>[^""]*)""|'(?<value>[^']*)'|(?<value>[^""'<> ]+)\s*)+"
                    , RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture))
                {

                    if ("http-equiv" == submetamatch.Groups[1].ToString().ToLower())
                    {
                        metaKey = submetamatch.Groups[2].ToString();
                    }
                    if (("name" == submetamatch.Groups[1].ToString().ToLower())
                        && (metaKey == String.Empty))
                    { // if it's already set, HTTP-EQUIV takes precedence
                        metaKey = submetamatch.Groups[2].ToString();
                    }
                    if ("content" == submetamatch.Groups[1].ToString().ToLower())
                    {
                        metaValue = submetamatch.Groups[2].ToString();
                    }
                }
                switch (metaKey.ToLower())
                {
                    case "description":
                        this.Description = metaValue;
                        break;
                    case "keywords":
                    case "keyword":
                        this.Keywords = metaValue;
                        break;
                    case "robots":
                    case "robot":
                        this.SetRobotDirective(metaValue);
                        break;
                }
                //                ProgressEvent(this, new ProgressEventArgs(4, metaKey + " = " + metaValue));
            }

            string link = String.Empty;

            ArrayList linkLocal = new ArrayList();
            ArrayList linkExternal = new ArrayList();

            // http://msdn.microsoft.com/library/en-us/script56/html/js56jsgrpregexpsyntax.asp
            // Original Regex, just found <a href=""> links; and was "broken" by spaces, out-of-order, etc
            // @"(?<=<a\s+href="").*?(?=""\s*/?>)"
            // Looks for the src attribute of:
            // <A> anchor tags
            // <AREA> imagemap links
            // <FRAME> frameset links
            // <IFRAME> floating frames
            foreach (Match match in Regex.Matches(htmlData
                , @"(?<anchor><\s*(a|area|frame|iframe)\s*(?:(?:\b\w+\b\s*(?:=\s*(?:""[^""]*""|'[^']*'|[^""'<> ]+)\s*)?)*)?\s*>)"
                , RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture))
            {
                // Parse ALL attributes from within tags... IMPORTANT when they're out of order!!
                // in addition to the 'href' attribute, there might also be 'alt', 'class', 'style', 'area', etc...
                // there might also be 'spaces' between the attributes and they may be ", ', or unquoted
                link = String.Empty;
                //                ProgressEvent(this, new ProgressEventArgs(4, "Match:" + System.Web.HttpUtility.HtmlEncode(match.Value) + ""));
                foreach (Match submatch in Regex.Matches(match.Value.ToString()
                    , @"(?<name>\b\w+\b)\s*=\s*(""(?<value>[^""]*)""|'(?<value>[^']*)'|(?<value>[^""'<> \s]+)\s*)+"
                    , RegexOptions.IgnoreCase | RegexOptions.ExplicitCapture))
                {
                    // we're only interested in the href attribute (although in future maybe index the 'alt'/'title'?)
                    //                    ProgressEvent(this, new ProgressEventArgs(4, "Submatch: " + submatch.Groups[1].ToString() + "=" + submatch.Groups[2].ToString() + ""));
                    if ("href" == submatch.Groups[1].ToString().ToLower())
                    {
                        link = submatch.Groups[2].ToString();
                        if (link != "#") break; // break if this isn't just a placeholder href="#", which implies maybe an onclick attribute exists
                    }
                    if ("onclick" == submatch.Groups[1].ToString().ToLower())
                    {   // maybe try to parse some javascript in here

                        string jscript = submatch.Groups[2].ToString();
                        // some code here to extract a filename/link to follow from the onclick="_____"
                        // say it was onclick="window.location='top.htm'"
                        int firstApos = jscript.IndexOf("'");
                        int secondApos = jscript.IndexOf("'", firstApos + 1);
                        if (secondApos > firstApos)
                        {
                            link = jscript.Substring(firstApos + 1, secondApos - firstApos - 1);
                            break;  // break if we found something, ignoring any later href="" which may exist _after_ the onclick in the <a> element
                        }
                    }
                }
                // strip off internal links, so we don't index same page over again
                if (link.IndexOf("#") > -1)
                {
                    link = link.Substring(0, link.IndexOf("#"));
                }
                if (link.IndexOf("javascript:") == -1
                    && link.IndexOf("mailto:") == -1
                    && !link.StartsWith("#")
                    && link != String.Empty)
                {
                    if ((link.Length > 8) && (link.StartsWith("http://")
                        || link.StartsWith("https://")
                        || link.StartsWith("file://")
                        || link.StartsWith("//")
                        || link.StartsWith(@"\\")))
                    {
                        linkExternal.Add(link);
                        //                        ProgressEvent(this, new ProgressEventArgs(4, "External link: " + link));
                    }
                    else if (link.StartsWith("?"))
                    {
                        // it's possible to have /?query which sends the querystring to the
                        // 'default' page in a directory
                        linkLocal.Add(this.Uri.AbsolutePath + link);
                        //                        ProgressEvent(this, new ProgressEventArgs(4, "? Internal default page link: " + link));
                    }
                    else
                    {
                        linkLocal.Add(link);
                        //                        ProgressEvent(this, new ProgressEventArgs(4, "I Internal link: " + link));
                    }
                } // add each link to a collection
            } // foreach
            this.LocalLinks = linkLocal;
            this.ExternalLinks = linkExternal;
        } // Parse
        
        /// <summary>
        /// Stripping HTML
        /// http://www.4guysfromrolla.com/webtech/042501-1.shtml
        /// </summary>
        /// <remarks>
        /// Using regex to find tags without a trailing slash
        /// http://concepts.waetech.com/unclosed_tags/index.cfm
        ///
        /// http://msdn.microsoft.com/library/en-us/script56/html/js56jsgrpregexpsyntax.asp
        ///
        /// Replace html comment tags
        /// http://www.faqts.com/knowledge_base/view.phtml/aid/21761/fid/53
        /// </remarks>
        protected string StripHtml(string Html)
        {
            //Strips the <script> tags from the Html
            string scriptregex = @"<scr" + @"ipt[^>.]*>[\s\S]*?</sc" + @"ript>";
            System.Text.RegularExpressions.Regex scripts = new System.Text.RegularExpressions.Regex(scriptregex, RegexOptions.IgnoreCase | RegexOptions.Multiline | RegexOptions.ExplicitCapture);
            string scriptless = scripts.Replace(Html, " ");

            //Strips the <style> tags from the Html
            string styleregex = @"<style[^>.]*>[\s\S]*?</style>";
            System.Text.RegularExpressions.Regex styles = new System.Text.RegularExpressions.Regex(styleregex, RegexOptions.IgnoreCase | RegexOptions.Multiline | RegexOptions.ExplicitCapture);
            string styleless = styles.Replace(scriptless, " ");

            //Strips the <NOSEARCH> tags from the Html (where NOSEARCH is set in the web.config/Preferences class)
            //TODO: NOTE: this only applies to INDEXING the text - links are parsed before now, so they aren't "excluded" by the region!! (yet)
            string ignoreless = string.Empty;
            if (IgnoreRegions)
            {
                string noSearchStartTag = "<!--" + _IgnoreRegionTagNoIndex + "-->";
                string noSearchEndTag = "<!--/" + _IgnoreRegionTagNoIndex + "-->";
                string ignoreregex = noSearchStartTag + @"[\s\S]*?" + noSearchEndTag;
                System.Text.RegularExpressions.Regex ignores = new System.Text.RegularExpressions.Regex(ignoreregex, RegexOptions.IgnoreCase | RegexOptions.Multiline | RegexOptions.ExplicitCapture);
                ignoreless = ignores.Replace(styleless, " ");
            }
            else
            {
                ignoreless = styleless;
            }

            //Strips the <!--comment--> tags from the Html    
            //string commentregex = @"<!\-\-.*?\-\->";        // alternate suggestion from antonello franzil 
            string commentregex = @"<!(?:--[\s\S]*?--\s*)?>";
            System.Text.RegularExpressions.Regex comments = new System.Text.RegularExpressions.Regex(commentregex, RegexOptions.IgnoreCase | RegexOptions.Multiline | RegexOptions.ExplicitCapture);
            string commentless = comments.Replace(ignoreless, " ");

            //Strips the HTML tags from the Html
            System.Text.RegularExpressions.Regex objRegExp = new System.Text.RegularExpressions.Regex("<(.|\n)+?>", RegexOptions.IgnoreCase);

            //Replace all HTML tag matches with the empty string
            string output = objRegExp.Replace(commentless, " ");

            //Replace all _remaining_ < and > with &lt; and &gt;
            output = output.Replace("<", "&lt;");
            output = output.Replace(">", "&gt;");

            objRegExp = null;
            return output;
        }

    }
}

HtmlDownloader.cs

using System;

namespace Core.Utils.Html
{
    public static class HtmlDownloader
    {
        private static string _UserAgent = "Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; ThePlanCollection.com; robot)";
        private static int _RequestTimeout = 5;
        private static System.Net.CookieContainer _CookieContainer = new System.Net.CookieContainer();

        /// <summary>
        /// Attempts to download the Uri into the current document.
        /// </summary>
        /// <remarks>
        /// http://www.123aspx.com/redir.aspx?res=28320
        /// </remarks>
        public static HtmlDocument Download(string url)
        {
            Uri uri = new Uri(url);
            HtmlDocument doc = null;

            // Open the requested URL
            System.Net.HttpWebRequest req = (System.Net.HttpWebRequest)System.Net.WebRequest.Create(uri.AbsoluteUri);
            req.AllowAutoRedirect = true;
            req.MaximumAutomaticRedirections = 3;
            req.UserAgent = _UserAgent; //"Mozilla/6.0 (MSIE 6.0; Windows NT 5.1; Searcharoo.NET)";
            req.KeepAlive = true;
            req.Timeout = _RequestTimeout * 1000; //prefRequestTimeout 

            // SIMONJONES http://codeproject.com/aspnet/spideroo.asp?msg=1421158#xx1421158xx
            req.CookieContainer = new System.Net.CookieContainer();
            req.CookieContainer.Add(_CookieContainer.GetCookies(uri));

            // Get the stream from the returned web response
            System.Net.HttpWebResponse webresponse = null;

            try
            {
                webresponse = (System.Net.HttpWebResponse)req.GetResponse();
            }
            catch(Exception ex)
            {
                webresponse = null;
                Console.Write("request for url failed: {0} {1}", url, ex.Message);
            }

            if (webresponse != null)
            {
                webresponse.Cookies = req.CookieContainer.GetCookies(req.RequestUri);
                // handle cookies (need to do this incase we have any session cookies)
                foreach (System.Net.Cookie retCookie in webresponse.Cookies)
                {
                    bool cookieFound = false;
                    foreach (System.Net.Cookie oldCookie in _CookieContainer.GetCookies(uri))
                    {
                        if (retCookie.Name.Equals(oldCookie.Name))
                        {
                            oldCookie.Value = retCookie.Value;
                            cookieFound = true;
                        }
                    }
                    if (!cookieFound)
                    {
                        _CookieContainer.Add(retCookie);
                    }
                }

                doc = new HtmlDocument();
                
                doc.MimeType = ParseMimeType(webresponse.ContentType.ToString()).ToLower();
                doc.ContentType = ParseEncoding(webresponse.ToString()).ToLower();
                doc.Extension = ParseExtension(uri.AbsoluteUri);

                string enc = "utf-8"; // default
                if (webresponse.ContentEncoding != String.Empty)
                {
                    // Use the HttpHeader Content-Type in preference to the one set in META
                    doc.Encoding = webresponse.ContentEncoding;
                }
                else if (doc.Encoding == String.Empty)
                {
                    doc.Encoding = enc; // default
                }
                //http://www.c-sharpcorner.com/Code/2003/Dec/ReadingWebPageSources.asp
                System.IO.StreamReader stream = new System.IO.StreamReader
                    (webresponse.GetResponseStream(), System.Text.Encoding.GetEncoding(doc.Encoding));

                doc.Uri = webresponse.ResponseUri; // we *may* have been redirected... and we want the *final* URL
                doc.Length = webresponse.ContentLength;
                doc.Html = stream.ReadToEnd();
                stream.Close();

                doc.Parse();

                webresponse.Close();
            }

            return doc;
        }


        #region Private Methods: ParseExtension, ParseMimeType, ParseEncoding
        private static string ParseExtension(string filename)
        {
            return System.IO.Path.GetExtension(filename).ToLower();
        }

        private static string ParseMimeType(string contentType)
        {
            string mimeType = string.Empty;
            string[] contentTypeArray = contentType.Split(';');
            // Set MimeType if it's blank
            if (mimeType == String.Empty && contentTypeArray.Length >= 1)
            {
                mimeType = contentTypeArray[0];
            }
            return mimeType;
        }

        private static string ParseEncoding(string contentType)
        {
            string encoding = string.Empty;
            string[] contentTypeArray = contentType.Split(';');
            // Set Encoding if it's blank
            if (encoding == String.Empty && contentTypeArray.Length >= 2)
            {
                int charsetpos = contentTypeArray[1].IndexOf("charset");
                if (charsetpos > 0)
                {
                    encoding = contentTypeArray[1].Substring(charsetpos + 8, contentTypeArray[1].Length - charsetpos - 8);
                }
            }
            return encoding;
        }
        #endregion
    }
}

References:

Be Careful With Page.PreviousPage

2008-03-05T06:57:33Z

I found something interesting while testing the website tonight. The new search we implemented was not working on the pages that display house plans. I thought that it was some kind of routing error that could be attributed to the url rewriting. However, closer examination showed that this is the error:


'Page.PreviousPage' threw an exception of type 'System.Threading.ThreadAbortException'

For some reason ASP.Net doesn't like certain nested master pages and will throw this error. My solution was to remove the Page.PreviousPage reference and key off of a parameter value - in this case the value contained in the search box. All that I can figure is that the PreviousPage reference attempts to actually load the page which can result in errors and thus the exception. I haven't had this problem with any of the other pages on the site.

My fix may not be the one I want, but it is fixed.

XML Encoding Problems - Hexadecimal Value 0x1A, is an Invalid Character

2008-03-04T04:17:20Z

Recently I started noticing exceptions in my log files that stated "hexadecimal value 0x1A, is an invalid character". I did some searching and a lot of other people have had the same problem. It occurs when someone pastes in text that contains specific unicode characters that are invalid in XML. From what I gather this frequently happens when someone copies in text from Microsoft Word.

The trick was to find out which characters are invalid and then use a regular expression to remove them.

You can look at the specs for all the details:

But it is probably easier to just get the code:

        /// <summary>
        /// This removes characters that are invalid for xml encoding
        /// </summary>
        /// <param name="text">Text to be encoded.</param>
        /// <returns>Text with invalid xml characters removed.</returns>
        public static string CleanInvalidXmlChars(string text)
        {
            // From xml spec valid chars:
            // #x9 | #xA | #xD | [#x20-#xD7FF] | [#xE000-#xFFFD] | [#x10000-#x10FFFF]    
            // any Unicode character, excluding the surrogate blocks, FFFE, and FFFF.
            string re = @"[^\x09\x0A\x0D\x20-\xD7FF\xE000-\xFFFD\x10000-x10FFFF]";
            return Regex.Replace(text, re, "");
        }

Updated Forums

2008-02-25T07:16:15Z

We recently updated our forums. This was a big move as it required retiring our old forum software and migrating all the old posts to our new custom forums. We feel that the change will provide a better experience for everyone using the site.

If you have any questions or would like to make a comment feel free to post to the forum and let us know what you think!

One quick tip. If you paste a link to a house plan in the forum like this:

http://www.theplancollection.com/house-plans/home-plan-9218

the software will automatically add in the image of the house plan and create a working link.

The Forums are Broken

2008-02-14T07:27:05Z

The forums are busted at the moment. Sorry about that. We are currently using a piece of software called community server that we purchased so we didn't have to write our own. It has been nothing but trouble so now we are moving away from it to a simpler system. We are writing it so it will take a bit longer to finish, but you can expect to see it running within the week. The debate between whether to write your own software or to find something that exists is always a tough one. It is always nice to let someone else do the work for you, but over time it can actually be more cost effective to develop your own solution if:

What you need is only a small subset of the off the shelf software and you want to simplify the experience for your users.
You need a highly customized solution that would require extensive modification of existing software.
The off the shelf software you choose is not so great.
You can afford to do it.

In our case all the above criteria apply. I have written more than a few forums in the past in various programming languages so writing a custom one will only take me a few days. Now back to work I go.

Why Asp.net Sucks (Tonight)

2008-02-01T07:37:03Z

I have noticed a slight slow down on a few of the pages on the website so I figured that it was time to get in there and turn on some caching. Asp.net, the technology we use, is supposed to make this very simple. All you have to do is something like this:


<%@ OutputCache Duration="600" VaryByParam="none" %>

Then magic happens and your page is magically stored in memory and millions of people can view it at the same time. The problem is that every page on The Plan Collection contains a login control. It is the little piece of code in the upper right. If you turn on caching guess what happens. The first person to login and view a page gets their user account broadcast to everyone else that sees that page. That's right the designers of the LoginView Asp.net control were so short sighted that they didn't realize that that control should NEVER be cached.

There are other problems with caching in asp.net as well including a lovely problem with the varybycustom parameter when you do cross page posting. The fun part of that is that if you set the parameter in page one then post to page two where a second varybycustom parameter is set the framework coughs, sputters and throws an exception letting you know that the parameter has already been set. How about this guys. Instead of throwing an exception vary by two parameters?

Don't worry this code never made it to production. I am once again going to go in and deal with the caching programatically so that site will get a boost. However, it is frustrating when 'enterprise' frameworks like asp.net fail you in terrible ways. Come on guys even though you are Microsoft and the world is accustomed to being let down by you, you can do better.

Ping Feedburner, Ping-o-Matic, Google, Technorati and All the Other Cool Kids Using ASP.NET and Some Magic

2008-01-31T04:29:13Z

It has been a long time coming, but we have finally managed to migrate away from Community Server to our own custom blog software that is integrated directly into the web site. The reasons for the change are numerous including the need to customize the look and navigation of our blogging software and our desire to provide readers a more streamlined experience.

The final part of the migration to this new system was released tonight. This final piece being the ability to ping the various providers that watch new blog posts. The code required to do this is not difficult to write. I am posting it below in case anyone is interested in how it is done.

Ironically enough this post will be the first one to actually use the ping service. Here's a link that helped me figure out how to do the ping and here's the code:



using System;
using System.IO;
using System.Net;
using System.Text;
using System.Xml;
using ThePlanCollectionCore.Utils;

namespace ThePlanCollectionCore.DataFeeds
{
    /// <summary>
    /// Wrapper for xml rpc service
    /// that can ping ping-o-matic and other services
    /// when your blog is updated.
    /// </summary>
    static public class Ping
    {
        // List of services that accept an xml rpc ping.
        // If you surf around the net you will find many other
        // services but many of them are offline or 
        // appear to be junk.  Stick with the main ones.
        static private string [] _Services = {
            "http://rpc.pingomatic.com",
            "http://blogsearch.google.com/ping/RPC2",
            "http://www.blogsnow.com/ping",
            "http://www.wasalive.com/ping/"
            };


        /// <summary>
        /// Sends a ping to the above services using the given website name 
        /// and url
        /// </summary>
        /// <param name="websiteName">The name of the website or blog
        /// that has been updated.</param>
        /// <param name="websiteUrl">The url of the website or blog
        /// that has been updated.  This should be an absolute root url
        /// not a direct link to the individual page that has been updated.</param>
        public static void Send(string websiteName, string websiteUrl)
        {
            for (int i = 0; i < _Services.Length; i++)
            {
                try
                {
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(_Services[i]);
                    request.Method = "POST";
                    request.ContentType = "text/xml";
                    request.Timeout = 3000;

                    // Add in the updated page
                    BuildRequest(request, websiteName, websiteUrl);

                    // Send off the request
                    request.GetResponse();
                }
                catch (Exception ex)
                {
                    // Log the error.
                    Error err = new Error();
                    err.Title = "Ping Request Failure";
                    err.Body = "Ping to " + _Services[i] + " failed with error: " + ex.Message;
                    err.Save();
                }
            }
        }

        /// <summary>
        /// Adds the requred xml to the request.  This includes the website name and url
        /// along with the method name 'ping' that will tell the remote servers that
        /// and update has been made
        /// </summary>
        /// <param name="request">A http webrequest generated by the method 'Send' above.</param>
        /// <param name="websiteName">The name of the website or blog
        /// that has been updated.</param>
        /// <param name="websiteUrl">The url of the website or blog
        /// that has been updated.  This should be an absolute root url
        /// not a direct link to the individual page that has been updated.</param>
        private static void BuildRequest(HttpWebRequest request, string websiteName, string websiteUrl)
        {
            Stream stream = request.GetRequestStream();
            using (XmlTextWriter writer = new XmlTextWriter(stream, Encoding.ASCII))
            {
                writer.WriteStartDocument();
                writer.WriteStartElement("methodCall");
                writer.WriteElementString("methodName", "ping");
                writer.WriteStartElement("params");
                writer.WriteStartElement("param");
                
                // Add the name of your website here
                writer.WriteElementString("value", websiteName);
                writer.WriteEndElement();
                writer.WriteStartElement("param");

                // The absolute URL of your website - not the updated or new page
                writer.WriteElementString("value", websiteUrl);
                writer.WriteEndElement();
                writer.WriteEndElement();
                writer.WriteEndElement();
            }
        }
    }
}

Fixed The Main House Plan Pages

2008-01-29T18:19:30Z

The other update I made last night was with the main landing pages. Some of the pages - luxury house plans, country house plans, and arts and crafts house plans looked fine because we had edited those pages by hand. However, many of our other pages like cape code style house plans page and our colonial style house plans page looked terrible.

I was able to use our existing page processing system and create a template for our landing pages that is loaded dynamically and then sent through the processor. The template contains place holders that are replaced with values specific to the style. Those values are replaced on the fly and then the page is generated. The result is a much better looking page that the guys in the office don't have to maintain by hand.

Updated Image Processing

2008-01-29T08:56:56Z

Working on a website in the middle of the night as I usually do is always an interesting and lonely experience. The benefit is that you get a lot done because no one bothers you. The downside is that waking up the next day is never fun.

This evening I managed to image processing code for our site. Every image - hundreds of thousands of them are passed through that code. It is fun to watch on the server because you can see the images being generated and then cached on the fly. Hundreds of them are generated every second. Pretty cool eh?

The uncool part was that it has had a few problems this last week. We recently received some images from our designers that were in a new format. (Who would have thought that there are so many formats for gif and jpg). At any rate the image processor was choking on some of some of the unsupported formats, throwing an exception, and then spewing out the 'Image Not Found' thumbnail that it likes to spit out when it gets lost.

This of course drove everyone in the office crazy because, well the image was there but the computer was telling then it wasn't.

Its fixed now.

In case you wonder what made the difference here it is:

This was the exception:
A Graphics object cannot be created from an image that has an indexed pixel format.

Here's the documentation from Microsoft

and here is the code I used to deal with the issue:

        private Bitmap GetTempImage(int width, int height)
        {
            Bitmap tempImage;
            if(_CurrentImage.PixelFormat == PixelFormat.Undefined ||
               _CurrentImage.PixelFormat == PixelFormat.DontCare ||
               _CurrentImage.PixelFormat == PixelFormat.Format1bppIndexed ||
               _CurrentImage.PixelFormat == PixelFormat.Format4bppIndexed ||
               _CurrentImage.PixelFormat == PixelFormat.Format8bppIndexed ||
               _CurrentImage.PixelFormat == PixelFormat.Format16bppGrayScale ||
               _CurrentImage.PixelFormat == PixelFormat.Format16bppArgb1555)
            {
                // None of the above types are support so 
                //just create a bitmap that uses the default pixel format
                tempImage = new Bitmap(width, height);
            }
            else
            {
                tempImage = new Bitmap(width, height, _CurrentImage.PixelFormat);
            }
            return tempImage;
        }

Updated The Article System

2008-01-25T08:36:30Z

I updated the way we handle articles this evening. We used to use Commmunity Server from telligent systems for blogging, but I found that it was too cumbersome and way too difficult to update. The themes are terrible to maintain so I build a system that meets our specific needs.

Our new system is divided into topics that I hope our visitors will find relevant. Most of our main content will end up in the 'House Plans'. However, we hope to venture out a bit and include relevant content in the business section where we will talk about how we manage ThePlanCollection.com and our technology in this section. In addition, we have added a green building section where we hope to highlight methods of responsible homebuilding.

Enjoy!

New Technology Section

2008-01-25T08:29:40Z

As part of our efforts to become more interesting and more transparent we have added a couple of new sections to ThePlanCollection.com. This section will focus on the technology we use, updates to the website and other technical issues that we find are interesting or relevant. Hopefully everyone finds this useful.

The Plan Collection is on Facebook

2008-01-14T21:47:38Z

When you are a small business and don't have an enormous development team you have to be careful about technology decisions. Most of our time is spent ensuring that our application is secure and running properly. Some of our time is spent adding new features including making changes to the design of the interface so that the website looks nicer.

Then there is time that we spend working on search engine optimization (SEO). We prefer to spend our time trying to get organic traffic because one, it is the best traffic and two it is free if you dont count all the time we spend on SEO. Of course my justification for not counting SEO as a direct marketing expense is that it usually makes our site better when done right.

Every once in a while we have a moment to breathe and we do something that is fun and interesting and that will hopefully lead to new opportunities. To this end we have created a Facebook application that lets users save house plans and then share them with friends. Users can also add comments to house plans and then watch what others are doing/saying.

We realize that Facebook applications are no longer bleeding edge, but this is a chance to see if any sort of meaningful interaction will take place in our application and discover if Facebook can provide benefits to small business. We currently have around 1300 users which is nothing compared to the most popular applications, but given that our application is very niche specific and our target audience is smaller it is a great start. Many Facebook users are in college or right out of college. As they get get ready to purchase their first home our hope is that they will use their social network to get ideas and they will find us there.

If you want to try out the application you will need a Facebook account. Then all you need to do is click here to install our house plans application.

How beneficial social networks will be to businesses like ours is yet to be determined. Creating an application might just burn up more of our server resources and bandwidth or it might bring users who buy something. It might be a great, free marketing resource or it might just give a bunch of kids a place to post vulgarities. Whatever the outcome well keep posting to let you know what happens in this emerging space.

Need a Building? Just Add Water

2005-03-22T13:51:00Z

I thought this article was worth sharing with you.

Need a Building? Just Add Water

As an emergency shelter alternative to tents, which provide poor protection, and prefabricated buildings that are expensive to transport, two engineers working on their master’s degrees at the Royal College of Art in London have come up with a “building in a bag.” Erecting the “Concrete Canvas” structure entails adding water to a sack of cement-impregnated fabric and inflating it with air. The volume of the sack controls the water-to-cement ratio, eliminating the need for water measurement. The Nissen hut-shaped shelter is dry and ready to use in about 12 hours. The inventors said they thought of the approach after hearing about inflatable structures that are built around broken gas pipes to carry out repairs. “This gave us the idea of making a giant concrete eggshell for a shelter, using inflation to optimize the structure for a compressive load,” said co-inventor Peter Brewin. “Eggs are entirely compressive structures with enormous strength for a very thin wall.” A bag weighing about 500 pounds inflates into a structure with 172 square feet and would cost about $7,700. Full-scale production of the shelter is being planned. (www.wired.com)
Wired News (3/15/05); Rowan Hooper

Apologies in advance

2004-11-23T15:25:00Z

I haven't been posting very often latetly - again. We are currently working on a new version of software for The Plan Collection and it is taking up quite a bit of my time. Hopefully the new version will better meet the needs of everyone who visits. If you have any suggestions or features that you desire go ahead and post them in the comments and we'll do what we can to add them.

Thanks for coming

2004-07-02T13:44:00Z

As a software engineer there is nothing more gratifying then having people actually use the software you have created. Over the years as I have met new people, it is interesting to find how many have visited The Plan Collection. We receive many visitors each day from all around the world. When we moved our hosting services to a new provider this year our provider was shocked to see the amount of visitors that come each day. Whether you are looking for a home plan or just browsing for ideas thanks for coming.

About a blog

2004-06-29T13:17:00Z

When you are a geek and spend ninety percent of you time in a virtual world you forget that the rest of the world spends their time doing things like going outside or actually talking to people. Not everyone has run across the "blog" phenomenon before. Blog is a shortened version of the phrase web log. Blogs are website where individuals may record their daily observations about various topics. They are a great way to disseminate information on a more personal level. In addition, you will find that most blogs include a link to an RSS or an Atom feed. These links are usually orange buttons, but may not always be so conspicuous. If you click one of these links you will get a page of text that is difficult to read. The text on the resulting pages is intended for an RSS aggregator. These are programs that are capable of bringing the information from many different blogs or websites together into one place. This can be very useful if you read information from many sources as the aggregator will gather up all the news and allow you to view it without visiting each individual site. If you are like me and read news from hundreds of sources this can be a real time saver.

I personally use a program called SharpReader. You can download it for free at http://www.sharpreader.net/. There are many choices available. This article from PC World lists several good readers and gives a good introduction into how to find and use RSS.

On The Plan Collection website our blogs try to focus on home plans and how to build new homes. You will find that Jake's blog focuses on home designs while Jaren's blog will focus on how to build a home. Jake Smith's blog will display some of the more popular home plans from The Plan Collection. I will probably spend a lot of time complaining about building a new home. I will also include some technical information about the software that runs The Plan Collection. As time goes on other members of our organization may also join our effort.

Our hope is that you will find our website a welcoming place that is easy to use. We have long sought a better way to connect with those who use The Plan Collection. If there is anything you fell we are doing wrong or that we can do better we always welcome feedback. I look forward to interacting with all of you that share our passion for home design.

Justin