I have been asked to compare the differences between our search engine and Open-SearchEngine. I agree this is an important question that needs to be answered, so I decided to put together a comparison between the core DNN Search, Open-SearchEngine, and Venexus Search Engine. While my opinion of which is the best, is defintely biased toward our own product, I have tried to provide an in-depth look at the basics of how each search engine works, a feature matrix, and simple search results analysis. Without further ado, read on...
DotNetNuke Search (core project)
DNN Search is part of the DNN core that is installed and configured out of the box.
DotNetNuke Search consists of 4 main pieces:
The scheduled task initiates the process of indexing the modules, at the scheduled time interval. An iteration of all modules that support iSearchable is performed. During this process, text that is extracted from the module is cleaned, parsed, and added to search word and search items tables.
The search admin is for setting the maximum word length, minimum word length, option to include common words, and the option to include numbers.
A module or skin object can be used to provide the form for the search query. In module settings, you can use the default button, or an image. You do not have the option to change this image within the module, nor change the text. Styles can be used to make some look and feel changes, but it is limited. When a search is performed, the user is redirected to the Search Results page.
This module provides the search results. In the settings, you can set the maximum search results, results per page, maximum title length, maximum description length, and the option to show description. Results are limited to the exact word queried.
Oddly enough, there no longer appears to be a DNN forum for search, or a blog dedicated to it on the DotNetNuke website. However, a good place to find out more about the core module is ecktwo’s site. There is a lot of information about how all the pieces work together, as well as the bugs/issues of DotNetNuke Search. There is also a tutorial and report on DNN Search for DNN 4.
Open-SearchEngine
Open-Search Engine consists of 4 main pieces:
Test The scheduled task initiates the process of spidering, at the scheduled time interval. Lucene.Net handles indexing of the data.
- Search Engine Admin Module
This module provides an interface for configuring the search engine to your preferences. You can add a starting URL and by default, spidering is enabled. This allows you to offer multiple sites in your search engine. However, unless disabled, each time you run the process to update the index, all URLs are re-crawled. With many URLs on the site(s) you index, it can lead to a very long time between the completion of crawling and indexing runs.
A module or skin object can be used to provide the form for the search query. In module settings, you can use the default button, or an image. You also have the option to add “Search” as text or image before the textbox.
This module provides the search results. In the settings, you can set which sites are part of the results scope, maximum results per page, maximum title length, title link target, and the option to hide description.
Venexus Search Engine
The Venexus Search Engine is quite different than the other 2 solutions. The package includes 2 modules and requires MS SQL Server Full-Text Indexing. Like traditional crawlers, VSE can crawl and index a variety of data, but where the real difference is seen is in it's ability to also “crawl” and index RSS feeds. This is the key to keeping the search results up-to-date, while conserving server and bandwidth resources. Rather than recrawling and reindexing all content, "smart caching" is used to determine when RSS feeds need to be aggregated, and when non-syndicated content needs to be recrawled on the site.
The Venexus Search Engine consists of 2 main pieces:
The Seamus module is the “search engine aggregation module utilizing syndication”. On the first load of the module, Seamus iterates through the core DNN modules on all portals that support the iPortable interface. Seamus uses this “initial dump” to gather other URLs for the site. You also have the ability to add feeds to Seamus, not only for your site, but any external site. With “global crawler” enabled, any external site URLs that are discovered during crawling, are added to the queue as well. Using AJAX, Seamus performs crawling of 3 feeds and 3 URLs with each load. If the user remains on the page, using AJAX, Seamus will continue to crawl and save the data to the table for indexing. This decreases the load on the server by spreading the crawling and indexing across several user sessions, rather than a single scheduled task.
The Search module provides the search box, as well as the results. Using Microsoft SQL Server’s feature of Full-Text Indexing, data is indexed from the crawling and storing provided by Seamus. Within the settings you can specify the search button text or use you own custom image for the button, set maximum search length, set search bx size, maximum results, results per page, set maximum length of display URL, specify remote connection string (database other than DNN), specify portal specific search, or allow user to select between site or all of the web search.
Feature Comparison Matrix:
Below you will find a list of features for DNN Search, Open-SearchEngine, Venexus Search Engine Standard, and Venexus Search Engine PRO.
|
Feature |
DNN Search |
Open-SearchEngine |
Venexus Search Engine Standard |
Venexus Search Engine PRO |
|
Crawling Method |
Module Indexer (Must implement iSearchable) |
Custom URL crawler/spider (Must have starting URL for each site, with crawling enabled) |
Custom Crawler
(Uses iPortable interface, traditional URL crawler/spider, and RSS aggregation) |
Custom Crawler
(Uses iPortable interface, traditional URL crawler/spider, and RSS aggregation) |
|
Crawl and Index Start |
Requires DNN Scheduled Task |
Requires DNN Scheduled Task |
User Interactive (AJAX in aggregation module) |
User Interactive (AJAX in aggregation module) |
|
Global Crawler |
No |
No (Requires input of each domain) |
No |
Yes |
|
DNN User Impersonation |
No |
Yes |
No |
No (Version 2.0) |
|
Windows Authentication |
No |
Yes |
No |
No (Version 2.0) |
|
Exclude List |
No |
Yes |
Yes |
Yes |
|
Excel Documents |
No |
Yes |
No |
Yes |
|
PDF Files |
No |
Yes |
No |
Yes |
|
PowerPoints |
No |
Yes |
No |
Yes |
|
RTF Files |
No |
No |
No |
Yes |
|
Word Docs |
No |
Yes |
No |
Yes |
|
Index File System |
No |
Yes |
No |
No (Version 2.0) |
|
Index |
Table Driven Index |
Lucene.Net (flat file) |
Full-Text Indexing in SQL Server (flat file) |
Full-Text Indexing in SQL Server (flat file) |
|
RSS |
No |
No |
No |
Yes |
|
Enclosure Support (podcast/vodcast) |
No |
No |
No |
Yes |
|
Feed Discovery |
No |
No |
Yes |
Yes |
|
Smart Caching |
No |
No |
Yes |
Yes |
|
Allow users to add feeds |
No |
No |
No |
Yes |
|
Generates RSS Feed of latest items indexed |
No |
No |
Yes |
Yes |
|
Blog and Feed Aggregator Pinging |
No |
No |
No |
Yes |
|
Search Skin Object |
Yes |
Yes |
Yes |
Yes |
|
Utilize DNN Search Skin |
Yes |
No |
Yes |
Yes |
|
Modify search box and image |
No |
Yes |
Yes |
Yes |
|
Use Image or Text for Search button |
No |
Yes |
Yes |
Yes |
|
Portal(site) or Web search |
No |
No |
Yes |
Yes |
|
Keyword Highlighting |
No |
Yes |
Yes |
Yes |
|
Cached Version |
No |
No |
No |
No (Version 2.0) |
|
User Saved Searches |
No |
No |
No |
No (Version 2.0) |
|
Social Bookmarking |
No |
No |
No |
Yes |
|
Price |
Free |
$49 |
Free |
$199 |
Performance and Relevancy:
What about performance and the relevancy of the results? I setup a test site with 5 total pages of content and installed/configured DNN Search, Open-SearchEngine, and Venexus Search Engine on separate pages. I also installed PageGenerated module from Ventrian Systems to show page execution time. I am not sure of any accuracy for a benchmark here, but the following results are the best of 5 consecutive query executions against each search engine using "truman" without quotes as the search query. In reality, there are only 2 relevant pages associated with "truman". There is a link from the home page of the site with the text "Truman Doctrine" as a contextual link that directs the user to the full document about the "Truman Doctrine". Ideally, we should expect the document that is all about "truman" and his doctrine to be listed first:
DNN Search:
Best Execution Time: 0.218531 seconds
Results Returned: 1
Notes:
The only result returned is not the most relevant page on the site. In fact, the "Truman Doctrine" page is not even listed as a result. This must be because the word "truman" does not actually appear in the content of the text/html module on the Truman Doctrine page. There is "HARRY S. TRUMAN'S ADDRESS" in the content, but DNN Search can only return results where the query is spelled EXACTLY like something in the content.
Open-SearchEngine:
Best page Execution: 0.1093155 seconds
Returned Results: 10
Notes:
Notice the poor description and the fact that the true most relevant document (the "Truman Doctrine" page) is the 5th result. Also, there are several results of pages that have no information about "Truman" except for the link in the SolPartMenu. While it is good that the search engine is able to crawl the solpartmenu, it is unfortunate that the search engine weights pages that just have links in a menu higher than the most relevant result. The best page execution time was half that of DNN Search, which is excellent.
Venexus Search Engine:
Best Page Execution: 0.046866 seconds
Results Returned: 3
Notes:
Notice the first result is the actual document (the "Truman Doctrine" page) we are looking for. Also, page execution time is less than half the time than Open-SearchEngine and a quarter of the time compared to DNN Search.
Conclusion:
The implementation provided by the DNN core team for the built-in DotNetNuke Search suits the needs for many smaller sites. However, larger sites will quickly run into issues with memory consumption due to the way the module indexing is performed. The search architecture is limited and greatly impacts the performance of the site and the search results due to the indexing process and the direct SQL table queries that holds the words and index. Most likely this is due to the requirement for database independence, rather than poor design. If your site is small, needs database independence, and search results are helpful, but not really an important piece of your site, then this may be the best tool for you.
If you are looking for a traditional search engine crawler, with good scalability, and you require database server independence, and decent search results, Open-SearchEngine may be the solution for you. It is by far better than the core DNN Search, but relies on tradional crawling and indexing methods. Conservation of bandwidth and server resources are debatable since there is no method of smart caching available. The ability for this engine to index direcories of files is an important feature than neither DNN Search, nor VSE offer. However, the lack of RSS aggregation as the new medium for crawling and gathering new and updated data is a huge issue that will lead to stagnant search results without frequently reindexing all URLs. As evident from the simple search results analysis performed, most results are not really relevant, but it is better than not returning any true relevant results like DNN Search due to spelling differences. It just means your users will have plenty to click on before finding the correct document they are looking for. While execution time is certainly better than DNN Search, it is still significantly slower than the Venexus Search Engine execution time.
The Venexus Search Engine offers 2 versions, the standard (free version), and the Pro (not free version). The standard version still offers many of the features smaller sites require, including quick and relevant results, but does not include some of the nicer features of the Pro version like PDF and MS Office document indexing and blog and feed aggregation pinging service. Where VSE really shines is in its ability to provide and aggregate RSS feeds for inclusion in its index. The smart caching and user interactive crawling using AJAX distributes the load on the server and bandwidth. The major advantage and disadvantage of VSE is MS SQL Server Full-Text indexing. The disadvantage is that VSE is NOT database independant and requires Full-Text indexing enabled versions of MS SQL Server in order to operate. The advantage is that it uses Full-Text Indexing from MS SQL Server for more relevant and faster search results. We know VSE is scalable because it has been tested against a database of over 2 million indexed pages. The simple search results analysis shows that it is 4 times faster than DNN Search and 2 times faster than Open-SearchEngine. The actual search results speak for themselves, delivering the most relevant result as #1 and contextual links from the home page as supplemental results.
Picking the right search engine application is important for your website and now you should be armed with the knowledge of how each one operates, the differences in features between them, and the overall performance and relevancy of the search results.
I hope this answers everyone's questions concerning the differences between the 3 DotNetNuke Search Engines. Feel free to comment with questions or suggestion on how this post can be improved. If you know of a feature or difference that I missed, please let me know. While this post is quite lengthy, I plan on keeping it updated as a resource for those who would like to keep track of the differences between each DNN search engine.
We did a search engine optimization (SEO) campaign for a client a while back that required us to setup a few new domains with blogging software. We used a variety of different blogging software and noticed something very interesting...sites that performed pinging were crawled within 1-3 days and indexed within 2 weeks. All of the domains we used were brand new. In some cases, keywords that we were focusing on were in top 10 search results in 2 weeks as well. Needless to say, the client was happy.
So, we added a pinging feature as part of our DotNetNuke search engine module. Seamus, our search engine aggregation module, generates a RSS feed for the entire DNN portal. Anytime a tab/page is added or updated and Seamus finds this change, if the "Pinging Service" is enabled, it will ping 14 different blog and feed aggregation services.
So you are probably asking how does this work. Well, we send a "ping" to these aggregation services using XML-RPC. It simply tells these services that there is new content available on your site and to check the RSS feed. These services then will consume the feed and add it to their index. This provides a contextual link (assuming you actually use good titles for your pages) from these service sites directly to the content on your site. Pretty cool, huh?
Currently the following services are pinged with our module:
- BlogDigger.com
- BlogFlux.com
- Blogsearch.Google.com
- BlogRolling.com
- Bulkfeeds.net
- Feedburner.com
- Feedster.com
- IceRocket.com
- Pingomatic.com
- Syndic8.com
- Technorati.com
- My.Yahoo.com
- Weblogs.com
- Weblogalot.com
As a test, we dropped our search engine module on a new domain on 3/16/2007. With no links from external sites pointing to this domain, most if not all of the traffic was from our development at this point, averaging about 3-5 unique visitors a day (probably me, the client, and one or more of our developers). As of yesterday, the site had almost 150 unique visitors. Not bad for 2 weeks, considering we have done nothing else to this site other than add 10 total pages of content. This site is off to a great start and we have not even started a linking or submission campaign. See the graph below:
Also, checkout the following post on Marketing Pilgrim about faster indexing through pinging Google blog search that collaborates our results.
Ready to see the power of pinging blog and feed aggregators? Add a pinging service to your DNN site and watch the traffic roll in.
We issues a patch release today for the Venexus Search Engine, a DotNetNuke search engine module. This release has a few minor bug fixes and tweaks:
- Catalog creation in SQLDataProvider file moved to Search module (you still have to run the VenexusSearch module SQLDataProvider file through Host > SQL during new installation, but not for those who are upgrading).
- Queue Importance added to allow some URLs to be crawled sooner than others. If you are running a global search engine, we now give preference to certain domain extentions (.gov and .edu).
- Stores single Robots.txt file for domain instead of historical. Previously we stored a new robots.txt for each weekly check if it was updated. Now we only store one.
- Feed Title is now the portal/site name.
- Added XML-RPC ping for BlogFlux inclusion in our blog/feed aggregator pinging service. We had to remove one aggregator and added BlogFlux to the list (14 total)
Get the free version of our DNN search module.
We released the latest version of our search engine module last week. It has all of the features I mentioned in my previous post, plus the ability to add excluded URLs and partial URLs.
Here are the new specs:
Items marked with * are new
|
Features |
Standard Version |
Pro Version |
| Seamus Features |
|
|
| Maximum # of Pages |
500 |
Unlimited |
| Install on commercial site |
No |
Yes |
| Scheduled Index Updates |
Yes |
Yes |
| Announcements Module Support |
Yes |
Yes |
| Contacts Module Support |
Yes |
Yes |
| Events Module Support |
Yes |
Yes |
| FAQ Module Support |
Yes |
Yes |
| Links Module Support |
Yes |
Yes |
| Text/HTML Module Support |
Yes |
Yes |
| Index MS Excel Documents * |
No |
Yes |
| Index MS PowerPoint Documents * |
No |
Yes |
| Index MS Word Documents * |
No |
Yes |
| Index PDF Documents * |
No |
Yes |
| Index Rich Text Files * |
No |
Yes |
| Global Crawler * |
No |
Yes |
| Allows users to add feeds |
No |
Yes |
| Custom User Agent |
No |
Yes |
| Obeys Robots.txt |
Yes |
Yes |
| TTL Support |
Yes |
Yes |
| Feed and Queue Aggregation Using AJAX |
Yes |
Yes |
| Display Top X Latest Items |
Yes |
Yes |
| XSLT Support |
Yes |
Yes |
| Latest Items RSS Feed Generation |
Yes |
Yes |
| Portal Specific Feed |
Yes |
Yes |
| Enclosure/Podcast Support |
No |
Yes |
| Pinging Service |
No |
Yes |
| Exclude URLs * |
Yes |
Yes |
| |
|
|
| Search Features |
|
|
| Search Skin Object |
Yes |
Yes |
| Use Image or Text for Search button |
Yes |
Yes |
| and - (AND and OR) Support |
Yes |
Yes |
| Quoted Search Support |
Yes |
Yes |
| Keyword Highlighting |
Yes |
Yes |
| Obeys DNN Security |
Yes |
Yes |
| Social Bookmarking Support * |
No |
Yes |
| |
|
|
| Support |
|
|
| Issue Tracker |
Yes |
Yes |
| Email |
No |
Yes |
| Phone |
No |
1 Call |
| Price |
Free |
$199 Per Year
 |
You can download the free version here.
The Venexus Search Engine is a DotNetNuke search module, plus a whole lot more. Not only does our DNN search module index your portal, but also external sites. VSE crawls pages on your site, aggregates RSS feeds from other sites, and crawls any links for external websites, making it a full search engine module. Unlike the core DNN Search module that uses a scheduled task to perform index updates, VSE crawls and indexes content based on user request. Seamus can be configured for several different setups and displays, including the ability to hide the module on every page. When a page is loaded that has the Seamus module on it, Seamus will go out and grab 3 RSS feeds and 3 queued URLs and add any new or updated content to the index. In order to not delay the page loading for the end user, Seamus utilizes AJAX to make aggregation request, providing a seamless integration into your site.
Here are the pro features of VSE:
1.1 Pro Features
- Allow users to add their feeds
You can enable users to add feeds to the system.
Indexed items that have files associated with them are used as enclosures (aka podcast) in the feeds that Seamus generates.
When the pinging service is enabled, every time something new is added/updated on your site Seamus will “ping” several XML-RPC web services for blog and feed aggregation sites to notify them your portal has new content. The aggregators will then come to your site and aggregate your feed and provide links to their users to your site.
Here are a few services we ping:
You can set your own user-agent to specify your own crawler name. The default user-agent is “Seamus/1.1 PRO ( http://search.venexus.com)”.
The pro version allows you to be a global crawler. Any links found on your site, from aggregated news feeds, or from external links are crawled and indexed.
So, not only are you able to aggregate even more content with the Pro version when compared to the Standard version, but you also get the search engine optimization benefits of pinging all of the major blog and feed aggregation services. This provides you with links directly to your site, generating more web traffic. You can watch your page rank grow very quickly with this feature.
Since the release of the 1.1 version, we have steadily been working on the 1.2 version. We are now testing the latest version on our demo site: search.venexus.com.
1.2 Pro Features (March 1, 2007 Release)
You asked for it, so we added support for all of the most common Office document file types as well as PDF documents. We have added a new document-to-text converter to our crawler that is able to parse the actual text from these documents. So not only does Seamus crawl and index HTML, Text, and XML files, but also the following new formats:
1. Excel files
2. PDF files
3. PowerPoint files
4. Rich text files
5. Word documents
- Social Bookmarking Support
In the search results you can enable social bookmarking to allow users to easily add bookmarks to their favorite social bookmarking application/service. This allows user to easily find their favorite links to your site. Also, the sites that provide this service will generate a link to your site, giving you more traffic once again.
Here is an example of what it looks like:
Here are the supported sites:
1. Digg
2. del.icio.us
3. FURL
4. Reddit
5. Yahoo
6. Blinklist
7. Google
8. ma.gnolia
9. Shadows
10. Technorati
Ready for a real search engine for your site? Buy the Pro version here.
Stay tuned for more…
We released the Pro version of our DNN search engine module today.
Here is the breakdown of the feature comparison:
Venexus Search Engine Version Matrix
|
Features |
Standard Version |
Pro Version |
| Seamus Features
|
|
|
| Maximum # of Pages
|
500 |
Unlimited |
| Install on commercial site
|
No |
Yes |
| Scheduled Index Updates
|
Yes |
Yes |
| Announcements Module Support
|
Yes |
Yes |
| Contacts Module Support
|
Yes |
Yes |
| Events Module Support
|
Yes |
Yes |
| FAQ Module Support
|
Yes |
Yes |
| Links Module Support
|
Yes |
Yes |
| Text/HTML Module Support
|
Yes |
Yes |
| Allows users to add feeds
|
No |
Yes |
| Custom User Agent
|
No |
Yes |
| Obeys Robots.txt
|
Yes |
Yes |
| TTL Support
|
Yes |
Yes |
| Feed Aggregation Using AJAX
|
Yes |
Yes |
| Display Top X Latest Items
|
Yes |
Yes |
| XSLT Support
|
Yes |
Yes |
| Latest Items RSS Feed Generation
|
Yes |
Yes |
| Portal Specific Feed
|
Yes |
Yes |
| Enclosure/Podcast Support
|
No |
Yes |
| Pinging Service
|
No |
Yes |
|
|
|
|
| Search Features
|
|
|
| Search Skin Object
|
Yes |
Yes |
| Use Image or Text for Search button
|
Yes |
Yes |
| + and - (AND and OR) Support
|
Yes |
Yes |
| Quoted Search Support
|
Yes |
Yes |
| Keyword Highlighting
|
Yes |
Yes |
| Obeys DNN Security
|
Yes |
Yes |
|
|
|
|
| Support
|
|
|
| Issue Tracker
|
Yes |
Yes |
| Email
|
No |
Yes |
| Phone
|
No |
1 Call |
|
|
|
|
| Price
|
Free |
$199 Per Year
|
I will be discussing the features of the Pro version in a later post. Stay tuned...
 We have released the new version of the Venexus Search Engine. VSE Standard Version 1.1.0 has several bug fixes and shows some of the new features of the Pro version.
New standard features and bug fixes:
- VenexusSeamus - Changed TransformXSL to not create a temporary XML file
- VenexusSeamus - Modified Response.Charset
- VenexusSeamus - New Delete Tabs routine for removing deleted and expired tabs
- VenexusSeamus - Ability to reload default XSLT file
- VenexusSeamus - Shows total number of aggregated items
- VenexusSeamus - Gridview pagination
- VenexusSeamus - Link from Grid to show aggregation errors
- VenexusSeamus - Guid attribute added
- VenexusSeamus - application/rss+xml support
- VenexusSeamus - Automatic creation of fulltext index during installation (works for SQL Server Express too!)
- VenexusSearch - Support for DNN 4.4.1 and "search" URL parameter
- VenexusSearch - Non-authenticated postback issue resolved
- VenexusSearch - Limits URL length for display
- VenexusSearch - Quoted query support
If you have any issues with installation, configuration, or bugs, pleas post them in our issue tracker.
The first beta testers of the Venexus Search Engine were the guys from True Lawyers. They created a new portal in their DNN installation for Search.TrueLawyers.com. Search.TrueLawyers.com is a legal search engine.  Their instance of VSE has aggregated over 216,000 legal articles, news, and related site pages as of this morning. You can test this site and see that the speed of VSE is still great when considering the amount of data it has already indexed. When using the site, each time a page is loaded that has Seamus on it (and Seamus CAN be hidden on the page), 5 feeds are aggregated. Any new items in the feeds are added to the index immediately. While the user sits on the page, AJAX is used to pull more feeds. Plus, since the site is new and does not have much traffic...yet, they use a RSS reader to call the Seamus RSS feed, which grabs more data every 10 to 15 minutes. As you can imagine, their index is growing FAST! You can see the latest items Seamus has aggregated by visiting the True Lawyers Legal News Room.
So, not only does VSE work as a site search engine and multiple portal search engine, it also works as a full blown search engine, aggregating items from your DNN installation, as well as other sites that provides RSS feeds. One of the features we are working on for the 1.1 Pro version is the ability to index any website, regardless of having a RSS feed. You can now have the ability to build powerful niche websites that provide your users with lots of relevant information. Plus, with the RSS feed Seamus generates, you can set it to display items for only your website, allowing you to submit the link to many feed directories, providing search engine optimization. The 1.1 Pro version will ping many blog directories, greatly increasing traffic to your website, treating your entire website like a blog. And we all know that the other search engines are just eating up blog content, increasing the page rank of those sites over many traditional websites without feeds. Ready to try it out? You can download the release candidate here.
Stay tuned for more...
UPDATE 2/7/2007:
I just checked the total items indexed for this site again and it is now showing over 246,0000 items. So in 2 weeks, an extra 30k + items were indexed.
I know many of you have been patiently waiting for the release of the Venexus Search Engine. We have had several beta testers try out previous release candidates, and have several new tweaks in this release.
Seamus Additions:
-
Web.configless (No changes to web.config needed. Beta Testers should remove EntitySpaces web.config entries when installing this version)
-
Object Qualifier support (Thank you Barry White for testing this)
-
Index current tab (Seamus will index the tab it is on. You can add make Seamus invisible on the page by showing 0 items in the feed and unchecking Show Feed. Add to all pages on the site and Seamus will index and update the index when the page is updated)
-
Edit Feed Display (Only show feeds selected in Edit Feeds section. By default, all are show. Selecting feeds will filter news display of only items indexed from the feeds list)
Search Additions:
-
Web.configless
-
Object Qualifier support
-
"query" URL parameter (You can now use your existing default DNN search results page. Simply drop the module on the search results page and remove the default search results module. Utilize the DNN Search textbox in your skin with the power of Full-Text Indexing).
-
-
Form post fix (A fix was added that allows you to simply hit the enter key after adding your query, rather than forcing you to click on the button)
-
Allow user selected web or site search (allow your users to select whether their search is against the current portal or for all search results in the database)
-
URL Trim (Used to trim the URL display in the search results. Long URLs would stretch out the skin)
-
Search Query (Saves user queries and the number of "hits" for that query. This will be used in the pro version for "Top Searches" and "Latest Searches".)
Here is some information about our DotNetNuke search engine module. You can test it on our site here. We also have the latest version loaded on our DNN search site for finding DotNetNuke related pages and sites.
As for the official release, we are waiting on FlatBurger to fix an issue with their code protection that causes the module to generate an error after activation. We have been told that this may be fixed by Friday...we will see. In the meantime, please send us your thoughts on this release candidate. If you find any bugs, please post them to our issue tracker. If you have any suggestions for new features, please post them in the issue tracker or in the support forms.
REQUIREMENTS FOR VENEXUS SEARCH ENGINE
- DotNetNuke 4.3.5 or Higher (Yes it works with the DNN 4.4 release)
- SQL Server supporting Full-Text Indexing
- .Net full trust for EntitySpaces and Reflection usage
Now for the files....
Before installing this, you MUST read the instructions. You CANNOT just install both modules and expect it to work. You MUST configure fulltext indexing manually to get this to work. You will find instructions on performing this action in the Search Instructions and Configuration.
You can download both modules here. The file is also attached as an enclosure.
Please post your links here in a comment to show everyone how you are using the Venexus Search Engine.
UPDATED: Link to module downloads has been updated.
I know it has been over a week since the last post. Sorry to leave you hanging, but sometimes there are just not enough hours in a day. Anyway, without further ado, here is part two… SEAMUS.
At some point earlier this year, DNN Find became a different mission. We decided to build a full blown search engine for DotNetNuke. Not one that would just index a single DNN site, but one that would allow you to index all portals in a DNN installation AND information from external sites. And how would external site indexing best be handled? …via RSS feed aggregation of course.
Seamus is the first of the two modules that make up the Venexus Search Engine. SEAMUS = Search Engine Aggregation Module Utilizing Syndication. On a side note, there is also an obscure Pink Floyd song that not many know from the Meddle album, about an old hound dog by the same name. Our hound dog “fetches” data and stores it to a table that has enabled MS SQL Server full-text indexing. But before I go into the specifics, I think it is important to know about the framework.
We started with traditional DotNetNuke module development…until EntitySpaces was released. I’m an old ASP/VB developer and personally, it took me a bit to get my head wrapped around how ES worked, but once I figured it out, I was hooked. ES saves the day by automagically generating all the CRUD (create, read, update, delete). While very similar to the logic of a BusinessController and InfoObject, ES uses Collections and Entities. But, where I found ES the most useful is the Dynamic Queries you can write directly into the business logic.
For example, in Seamus we need to check the domain to see if it matches one we are already indexing:
Dim colDomains As New VenexusDomainCollection
colDomains.Query.Select(colDomains.Query.DomainName, colDomains.Query.DomainID)
colDomains.Query.Where(colDomains.Query.DomainName.Equal(GetDomainName(sURL)))
colDomains.Query.Load()
If colDomains.Count > 0 Then
‘a bunch of removed logic goes here..
End IF
With the colDomains.Query.Select, we are only returning the data we need rather than all columns. With the colDomains.Query.Where, I eliminated the need to:
- Write a stored proc just to retrieve by DomainName
- Iterate through the entire table, every row of all domains, just to find the one I am looking for.
I won’t even go into the performance gain of not having to loop through those rows of all columns, nor the time (even though it would be simple) to write a stored proc to pass in DomainName and have it return the DomainID.
Here is an example of adding a record to Seamus for a new feed:
Dim entFeed As New VenexusSeamus
entFeed.AddNew()
entFeed.Url = txtURL.Text
entFeed.Title = txtTitle.Text
entFeed.Account = txtAccount.Text
entFeed.Password = txtPassword.Text
entFeed.CacheTime = txtCacheTime.Text
entFeed.FeedTimeOut = txtTimeOut.Text
entFeed.DateAdded = Now()
entFeed.DateUpdated = "1/1/1901"
If chkActive.Checked = True Then
entFeed.IsActive = True
Else
entFeed.IsActive = False
End If
entFeed.Save()
Easy enough, eh?
And here is an update of a feed for Seamus:
Dim entFeed As New VenexusSeamus
entFeed.LoadByPrimaryKey(hidRSSID.Value)
entFeed.Url = txtURL.Text
entFeed.Title = txtTitle.Text
entFeed.Account = txtAccount.Text
entFeed .Password = txtPassword.Text
entFeed.CacheTime = txtCacheTime.Text
entFeed.FeedTimeOut = txtTimeOut.Text
entFeed.DateAdded = Now()
entFeed.DateUpdated = "1/1/1901"
If chkActive.Checked = True Then
entFeed.IsActive = True
Else
entFeed.IsActive = False
End If
entFeed .Save()
And a delete example:
Dim entFeed As New VenexusSeamus
entFeed.LoadByPrimaryKey(hidRSSID.Value)
entFeed.MarkAsDeleted()
entFeed.Save()
Yeah, it’s that easy. Makes you want to fire up your IDE eh?
Sure, I have used DAL Builder Pro, which was a huge time saver, but EntitySpaces made me to never want to develop any other way. Plus, last I checked, DAL Builder Pro was still only for DNN 3 development. The ease of generating the DAL and the ability to easily REgenerate the DAL if the database schema changes, makes ES the tool of choice for all of our module development. I cannot even begin to count the hours I have previously spent hand coding changes in a DAL due to spec changes. Oh how I wish I had all those hours back!
With the new DNN admin grid templates, it is just ridiculous how much code is generated before having to write the first line. The new template will generate an editable grid of the table(s), with sorting, paging, and search. If you are interested in .Net development (this is not just a DNN tool, it works for all .Net 2.0 development and using C# or VB.Net), you must check it out.
NOTE: Just so you know, we do not have any affiliation or partnership with EntitySpaces, we just think their tool rocks.
So, even though we had much of the initial Seamus development completed, we scrapped it and started development with ES. This will make future modifications and additions so much easier, saving time in the long run.
With that said, here is how Seamus works…
After you install Seamus, you can go into the module settings:
So in this example, the display for Seamus should show the top 10 items last indexed, each with a link to the actual item in the Title and using the “…More” link. A feed icon will also be displayed that provides a link to a RSS feed for the top 10 items.
Here is an example of the display:
Now while the above example does not show any local items (tabs or modules from this site), it does have items indexed from other sites. All of these items were from RSS feeds that were aggregated. As a module editor, you have the ability to manage external feeds (or local feeds if so desired, but we will go into more detail about how Seamus works shortly). But, if there were local items visible, they would only be visible if you have the proper permissions. Seamus checks permissions on any local site at the module and tab level for the display and the RSS feed.
Here is an example of the feeds we are are currently indexing on the Venexus Search Engine:
Here is the interface for adding new feeds:
Now we will get into how Seamus works…
First off, on the first load of Seamus, a dump of data from all modules supporting the IPortable interface (currently limited to DNN Core modules) is performed to ensure that there is data in the index. And every X hours (determined in module settings), the index is checked for new, updated, and deleted pages/modules.
Secondly, any feeds that have been added to Seamus are aggregated 5 at a time, order by last updated. And, while the user is sitting on the page, every 30 seconds that pass, 5 more feeds are aggregated via AJAX. This user interactive aggregating decreases the load on the server, rather than running as a scheduled task like the core DNN Search.
In order to save bandwidth, and to not tick off the owners of the websites you are aggregating data from, Seamus has what I call “smart caching”. Each time a feed is requested, if the information in the feed as not been updated, Seamus will increase the cache time. If the feed has been updated, it will request the same feed sooner than it had previously, decreasing the cache time. Over time, and based on the “average” a feed it updated, Seamus learns when to check again for updates, all while obeying TTLs.
Seamus will also index the current page/tab it is sitting on. Now you may be asking why you would index a page that displays items that have already been indexed. Well, Seamus can be setup to not display the top X items and/nor the RSS feed. Here is an example:
With the above Seamus settings and the module settings to display on all pages and set to not display the container or using an “invisible” container, when a user lands on any page of the site, the page is indexed. You can index your entire site by letting the users "crawl" the website. Also, when the page is updated, the index will be updated. Here is the module settings example:
So, not only does Seamus index all portals in the DNN installation by doing a dump of all modules that support the IPortable interface and individual page indexing based on user interaction, it will also aggregate and index data from other sites. This gives you the ability to create a full blown search engine for your niche. For example, let say you have a website about racing. You could have your entire DNN site indexed, along with aggregation of more racing data from the following sites:
http://www.sportsline.com/partners/feeds/rss/auto_news
http://rss.news.yahoo.com/imgrss/events/sp/042103autoformula
http://rss.cnn.com/rss/si_motorsports.rss
Not only are you able to display a list of the last items indexed in order to keep a page from becoming stagnant, you can also provide a RSS feed for your users, giving them a reason to return to your site. I will save a Seamus and SEO discussion for another time, but here is an example site for a legal search engine.
Speaking of time, I am once again out of it. Part III will be a discussion of the second module, the search form module. Stay tuned...
I know a lot of people have been waiting on this and it is literally been over 5 years in the making, but it is now time to tell the story of how the Venexus Search Engine came to be…
Bots, Crawlers, and Spiders, Oh My!
Once upon a time, long, long ago, well over 5 years ago anyway, but that’s like ancient history in terms of the web, I wrote a little script to rip down free fonts off of a font directory website, who shall remain nameless since they are still around today. FontGrabber.vbs crawled their entire website saving zip files of free font packages. If I remember correctly, it pulled down almost 5000 font packages in a few hours. What a time saver! And my crawler addiction began to set in…
MediaGrabber
The next crawler I wrote extracted data from an online database of live music recordings. I dumped about 10 to 12 thousand records into a custom media database. My crawling habit had now increased to an hour or 2 a week perfecting the use of HTTPGets using XMLHTTP and making modification to scrape other data from the site based on URL parameters.
Many variations of MediaGrabber were developed over the years for aggregating data. Some of the variations include:
-
PhotoGrabber - For consuming one of the stock photography buffett sites. An interesting note, the one we crawled, which will also remain nameless, started limiting the number of photo request per day the following month. I wonder if that had anything to do with what we were doing...hehe.
-
FDAUpdater - For pulling down pharmaceutical data from the FDA to be used on a pharmacy website. Enough said about that one.
-
CategoryDump - For pulling category names from Yahoo and DMOZ.
-
And others...
Madhatter
Madhatter was my first bot. It was a VBScript that sat in a Direct Connect P2P Server application. Madhatter started as a trigger bot. A user would type a message into the chat and if it contained keywords or phrases that matched a list of keywords and response(s), the bot would automatically reply with a random response from the list that was associated with that keyword. Over time, I added around 1000 different responses to about 400 keywords. Madhatter then received search capabilities. You could type +search <band> or +search <date> and it would return a top 100 list of media records from a database of about 20000 records that matched with a link pointing them to the website with the information. I then gave the ability for the Operators to allow Madhatter speak on their behalf. So in addition to Madhatter automatically responding, the operator would make new responses to the user messages via Madhatter. This worked so well, and I guess to some degree could be considered my first AI application, that many DC newbies really thought it was a live person responding to their messages, even when Madhatter was running solo. I even setup the bot so that if a user tried to send Madhatter a private message chat, it would display in the Operators chat. This led to untold hours of entertainment watching people talk to a rude, trash talking bot that would kick them off the hub if they responded in a derogatory manner. Just thinking about it again makes me want to write a DNN Bot, maybe not one as feisty as Madhatter. Or maybe “bot” interactive search anyone?
Tiny IntRAnet Crawler
I started working for Semiconductor Research Corporation in August 2001 as their Web Administrator/Developer. At that time they only had a website and a forums website. The forums website was using a product called SiteScope which was written in TCL, but we will not even go there in fear of recurring nightmares .The SRC main site was not built using a Content Management System, rather a Staging to Dev push of content. I think it was sometime in early 2002, I began writing my first true crawler that would consume all items in a domain.
The need was simple…with the amount of content we had on the site, there was bound to be broken links, missing images, orphaned files, and God forbid, 500 server errors. We needed something that would crawl the site and search for any issues, compare the file system, and generate a report for the Content Management Team. I was still using XMLHTTP component for grabbing the data until I found ASPTear. ASPTear proved to be faster and was the HTTP component of choice until I found NSoftware. NSoft’s HTTP component was far superior to any of the others for speed and with many more methods/objects that could be utilized.
SRC had a pretty big main site and we began developing 2 other websites to fall under the SRC umbrella. This lead to TIC 2.0, which crawled all 3 domains, and would (and probably still does) generate a report of any issues. With TIC now crawling more than one website and doing it dynamically (could jump from one domain and then the other with the FIFO [First In First Out] URL queue/stack), the need came to check the first link offsite. Why? In case the link moved (301 or 302), or was generating a 404. We have no control what some site may do to their content, but we sure wanted to know if our users were going to get an error if it was broken. TIC would find those problem links and let the CM Team know they needed to remove the link, or change the URL to the new redirect. Now comes TIC 3.0...
Tiny IntERnet Crawler
One night I was goofing around with TIC and decided to turn off the function that performs the domain or first link offsite check and just let it run…and run. And it did, all night long. When I got up the next morning, it had crawled almost 30,000 pages and had built a queue of over 100,000. Now I was hooked. How could I get more data and faster? Since TIC was a script and utilized a central database for the URL queue, instead of an in memory stack, I was able run multiple instances of the crawler. 10 instances of TIC 3.0 crawling brought my little home router to its knees. In fact, it choked and rolled over tits up. In three hours, over 110,000 pages were crawled, over 500,000 URLs queued, and had sucked down over a gig of data. Whoa…this was getting fun.
Over the next year or so I really was tweaking TIC quite a bit. I’d let it run for weeks at a time. I quickly realized I was going to run into a big problem…Disk space. The database was getting bloated and slowing down dramatically after it had indexed over 1 million pages and had over 5 more million queued. While those numbers are a drop in the bucket when compared to the 800 pound gorillas of search, it is still a lot of data for such a small operation. And, TIC would crawl anything, all file types. So I started curbing back what TIC looked for…all the way down to just XML. TIC, as the last version in use, now looks just for XML files anywhere on the Internet. Of course I added tweaks to check domain importance or linking page importance based on keywords and altered the queueing process so that TIC would not get stuck on a crappy domain. But that is a discussion for another time.
Tiny XML Spider
So with TIC crawling the web looking for XML files, TXS was developed to crawl and index the XML files TIC found. TXS runs continuously, iterating through all “approved” RSS feeds (about 2,500 of over 100,000). For each feed it parses through the articles and stores anything new to the database. If the feed has been updated, TXS will return in less time. Feeds that have not been updated will be crawled the next time after a longer duration. I call this “smart caching”, which will be discussed in the features of Seamus later on. TXS has aggregated over 1.7 million articles from only 2500 news feeds. Not bad considering how much other data we have to collect from feeds that have not been approved. We have been stuffing the aggregated data into a combination of DNN websites for SEO reasons.
DNNFind
DNNFind = DotNetNuke Fulltext INDexing. At some point about 2 years ago, and with TXS bringing in the data, we decided to build a DNN module that would perform a SQL Server fulltext index query against the aggregated data and return the results. While this is not a bot, crawler, or spider, it is a fundamental step of searching the data, which we will get into when discussing the search module of VSE.
DNN Spider
I started developing a standalone VB.Net application for crawling DotNetNuke websites. This was my first multi-threaded application. While similar to TIC, this application would allow 1 to many threads to be used to handle the crawling. What we found is that we can use the application for stress testing DotNetNuke websites by throwing a few hundred or thousand request at it. And, we can use multiple applications running on different servers to really pound away at a box. However, this got me thinking about distributing the load of crawling against the users of the website, which is why we are using AJAX to request more data from Seamus. More on that later on as well.
Okay, so you made it this far and you are probably asking why I have not even started to describe what the Venexus Search Engine does. Well, I think it is important to understand the background of the application and how it came to be. It’s not like we just came up with some flimsy half-brain ideas about how a search engine should be done, but rather years of trial and error. And, I want everyone to realize that our product is not going to disappear, but get stronger as we add more functionality from all of the code we have written over the years. With that said, here are the details...
Sorry, I am out of time and you will have to wait for Part II of this post.
In the meantime, if you want to see Venexus Search Engine in action, go to search.venexus.com. To read more about VSE, go here.
REQUIREMENTS FOR VENEXUS SEARCH ENGINE
- DotNetNuke 4.3.5
- SQL Server supporting Full-Text Indexing
- .Net full trust for EntitySpaces and Reflection usage
If you would like to test our release candidate, please reply in a comment to this post and I will send you the PA's.
If you are not familiar with Google Alerts, you should check it out. I have been tracking things from Google Alerts for at least 2 years, maybe longer. While I have noticed more things coming in for "DotNetNuke", starting on October 27th I noticed ALOT more alerts for "DotNetNuke" coming in. What did they all have in common? BLOGS. Also a few days ago, while looking into the activity of this blog, I noticed a new user agent I had not seen:
Feedfetcher-Google; (+http://www.google.com/feedfetcher.html)
If you go to that link, you are redirected to a FAQ page and at the bottom is a section called Feedfetcher. Here is an interesting Q and A:
How do I request that Google not retrieve some or all of my site's feeds?
Since Feedfetcher requests are all user-initiated, it does not follow the typical robots.txt guidelines for robots. For detailed information about how to prevent Feedfetcher from requesting all or part of your site, please see our removal instructions.
Very interesting. I was under the assumption that any "bot", and I will define Feedfetcher as a "bot" regardless of whether it is "user-initiated" or not, should obey robots.txt.
With that said, our feed aggregation module for Venexus Search Engine, called Seamus, does obey robots.txt. I am sure this discussion will come about with the release of VSE, so I decided to go ahead and post it now in preparation. And speaking of Venexus Search Engine...we have made the final compile and are finishing testing tonight...but more on that later.
A lot of DotNetNuke developers/webmasters forget the importance of traffic derived from their blogs or RSS feeds. Any seach engine optimization plan should include submitting links to popular blog/news feed directories.
For those who are unaware, you can enable syndication within several DotNetNuke modules. For example, in the Text/HTML module Settings > Page Settings > Basic Settings is Allow Syndicate checkbox. If this is enabled, a XML icon will appear on the rendered page.
There are also many DotNetNuke modules that provide RSS feeds for syndication. A couple of my favorite modules are Orizonti's NukeNews and Scott Schecter's NukeSyndicate. Both of these modules provide a RSS feed for consumption. NukeSyndicate allow for posting audio and video files, know as podcast and vodcast...more on this later 
Recently I spent a few hours submitting this blog to many directories. Most of the links below do not require reciprocal links, but in order to satisfy those who do and to provide a resource for others, here is the list I submitted our RSS feed to:
If you have any other suggestions, please do tell! Check back for updates....
I recently cleaned up our DNN default.aspx page by removing several meta tags. Many of the meta tags in default.aspx are useless and do nothing but make the payload bigger and pushes the main content further down the page. One of the meta tags removed was <META NAME="ROBOTS" CONTENT="INDEX, FOLLOW">. By default, all pages on a DNN site are Indexable and Crawlable. Search engines do not need to be told to index or follow a page. They will do this anyway as it is their nature. I suggest using this meta tag only if you do NOT want search engines to index or follow your page. However, it is important that if you modify default.aspx, make sure you set page to have read only access so that future DNN releases do not overwrite your "custom" page.
Now, we actually have a need for a Robots Meta Tag on a public page that we do not want crawled or index. So, to add the meta tag back in just for the page we do not want crawled or indexed, simply go into the Page Settings and expand Advanced Settings. In the Page Header Tags textarea, add <META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
Some other examples:
<meta name="robots" content="noindex,follow"> - this tells the search engines crawlers/spiders to not index the page but to follow any links that are on the page for possible indexing.
<meta name="robots" content="index,nofollow"> - this tells the search engines crawlers/spiders to index ths page but not to crawl any of the links on the page.
I recently posted some SQL statements on updating PageHeadText in the Tabs table for DotNetNuke and Google Analytics. I recently updated the SQL statement to exclude any deleted tabs.
UPDATE Tabs SET PageHeadText = '<script src="http://www.google-analytics.com/urchin.js" type="text/javascript">
</script>
<script type="text/javascript">
_uacct = "UA-xxxxxx-x";
urchinTracker();
</script>'
WHERE (PortalID = 0) AND (ParentID <> '38') AND (IsDeleted = 0)
This will automatically change all pages for the portal to include the Google Urchin JavaScript into the page head. This will also filter out certain pages. For example 38, will filter out any Admin pages in the database for our setup. IsDeleted = 0 makes sure the page is not deleted. This is alot quicker, especially for portals that have thousands of pages, than adding a module to each page or updating each page individually in the Page Settings > Advanced Settings > Page Header Tags.
Lee Sykes of DNN Creative approached me about doing an interview on search engine optimization and DotNetNuke. I jumped at the opportunity and am happy to announce the Issue 4 release of DNN Creative that includes the DNN and SEO interview. I will admit that I do not know everything about search engine optimization nor DotNetNuke, and have not been in the SEO game long. Nor do I have the stomach to live on the whim of Google or any other search engine algorithm change, but there are a LOT of basic things you can do to make your DNN site rank better in the search engines. I have seen many DNN sites that might look good to the user as far as look and feel are concerned, but the code looks long and poorly optimized in the source. This is disappointing because without the search engines really picking up on the DotNetNuke movement, as well as the search engine users finding sites built with DNN, the opponents of the darkside (damn hippy PHP coders) will continue their run at being the choice for data rich SE friendly sites. What I do know about SEO is from running many DNN sites and trying things to see how the search engines treat them. Also, having wrote several crawlers/spiders, I do know how the bots that retrieve and index webpages work. So, check out the interview in Issue 4 of DNN Creative and search engine optimize your DNN website.
As posted on Xfernal.com, here is the code for DotNetNuke Skin HTML Table Trick. This will help for search engine optimization by making the "Left Menu" aka LeftPane appear to the user as expected, but appear in the HTML code below the ContentPane. This will help getting the main content of the webpage above the navigation, making the content more important than the menu when indexed.
Here is the code: <TABLE cellSpacing=0 cellPadding=0 width="100%" border=0> <TBODY> <TR> <!-- skip to second column --> <TD vAlign=top align=middle width=180></TD> <TD class=ContentPane id=ContentPane vAlign=top width="100%" rowSpan=2 visible="false" runat="server"></TD> </TR> <TR> <TD class=LeftPane id=LeftPane vAlign=top align=middle width=180 visible="false" runat="server"></TD> </TR> </TBODY> </TABLE>
Out of the box, DNN does a couple of very bad things as far as search engine optimization is concerned. While I was learning how DNN worked, I had Google index one of our sites and it indexed every page on the site AND the unique URL page for Login, Register, Privacy Policy and Terms of Use, making my site index 5 times larger than it really was. Very bad! This redundant content could hurt search engine rankings...
The first concern is the links to Login and Register. By default, with search engine friendly URLs turned on, a unigue URL is created on EVERY page for the Login and Register. This can be quickly fixed...
The fix:
- Create a new page called Login
- Add the Account Login module to the page
- Create another page called Profile (you can call it whatever you want)
- Add the User Account module to the page
- Go to Admin > Site Settings
- Expand Advanced Settings
- Under Page Management select the page you first created for the Login Page
- For the User Page, select the Profile page you created
- Click the Update link at the bottom of the Site Settings page
Now, DNN will use the "hardcoded" links for all Login/Register links on the site.
The second badness DNN does is with it's default DotNetNuke skins. Within the skin is the DNN Tag/Token for Privacy Policy and Terms of Use. Once again, this is a unique URL for EVERY single page on the website for both pages. This one is a little more difficult and requires that you edit the skin. So, open up your favorite editor and modify the skin by removing the tags/tokens for "PRIVACY" and "TERMS" to use a real hardcoded links. While most skin developers include these tags/tokens in their skins, until the DNN core team fixes this issue, I would recommend removing them.
Update 1/5/2005:
I submitted an issue to the DNN Bug Tracker on 1/1/2005: DNNP-2289
No response yet...
|
Copyright © 2009 Venexus, Inc.. All rights reserved.
|
|