FindTheBest.com

Holy cow, how did it manage to hide from me the whole time? I don't remember how I came across this website, but I have to admit, when I spent a few minutes clicking here and there on it, I was truly dumbfounded.

It isn't really IT, but it is so damn close to what I'm working at. Sure, I knew there's a huge competition since I started, but all those freebases and wolfram|alphas were either different, ugly or hard to use. This is none of them. The only good thing for me is that the main turf for the website are mostly the States and I'm aiming higher.

On the other hand, they have a huge amount of pictures, awesome graphs, nice filtering options with excellent results interpretation, overwhelming wizard-like "assist me" function and much, much more.

When I shaked the surprise off, I started to pick features I like and think how to implement them into QetriX.

• Classifications should have default icon and picture.
• For attribs there should be an information about what value (lower/higher) is better. A weights for this would be nice too.
• User friendly filtering, using not just values and check boxes, but nice min-max sliders as well. Domains will be handy here.
• Entity should me even more specific, with charts and tables.
• Related entities are necessary.
• Besides the main entity it would be handy to attach small info-box for related entities too, such as manufacturer, superior area unit, homeland etc.
• User ratings and comments would be there again. Comments with rating also (up/down vote, likes...).

I assume it's also written in PHP, since /index.php returns no 404. I was trying to find their data sources, but I found only a simple statement: "We obtain our information from three sources: Public databases, primary sources (manufacturer websites) and expert sources." Well, sincerely I didn't expect much more, since data sources are the main business secrets here.

Something about Crawler

As I wrote earlier, I was confident I can collect data for the Particle Database by myself, but shortly I realized it's just too much. So after a struggle I decided to make a web crawler for harvesting semantic data.

The decision of target platform was quick - PHP is not suitable, so .NET. At first I tried to customize some of those existing open source crawlers, but it was a pain. So ultimately I wrote my own. I don't need any GUI, so simple console app was suitable.

It downloads plain text data (e.g. HTML page), using HttpWebRequest class, and extracts all links to other pages. For every use I can define a set of boundaries, so links out of these boundaries are throwed away. The rest are put into queue. Robots.txt and meta tags are a part of those boundaries as well.

Then crawler prepares data for parsing - strips off all unimportant parts and replaces or deletes some parts (like whitespaces). Then it checks for presence of defined string, indicating there are some data to extract. And finally, using regular expressions, structured data are extracted and send to server-side for further processing.

On server side is PHP script, which post-processes and saves data into database, finds relevant links to another entities and stores URL of data source to avoid duplicates.

Then crawler politely waits a while before loading next URL from queue, because I don't want to overload these servers. And... that's it! Simple, yet powerful.

MobiletriX

There are two main approaches for mobile version of a website or a web application (well, three, if you consider "none" as an option): responsive webdesign and separate mobile version.

I decided to do both. I really like the concept of responsive design, but I hate when I have to download something I won't use (graphics, HTML elements etc.).

Mobile version should be lightweight and may be little uglier by not having all the fancy bells and whistles, which on mobile devices just consumes bandwidth and processor time anyway.

Mobile users also usually want just to hop in, do anything quickly and hop out, especially when the mobile version isn't the main way of using your site/app. Therefore the original use cases might be little crooked and it's ok. Developer should not throw logs under user's legs, but stick with him instead.

Users mostly expect similar, but slightly different behavior between desktop, web and mobile app. Each of them is used in different situations, where user would appreciate different things. It's even determined by time of the day, but it would get too far :)

It's also a bad habit to emulate his device's interface a you can never satisfy all of them. And believe me, iPhone UI on Android phone looks extremely stupid.

Reengineering AGAIN?

And again :) I don't have to worry about anything right now, as everything is doing just fine with the previous version and nobody rely on the new one.

I wanted to add more functionality to the code and realized I pasted a lot of code in the core without further investigation what it really does and how good it is in sake of performance. And because I did some thinking about the object model, I decided to start over with no old code.

I know it will take a lot of time, but from my point it's time well wasted. Reengineering is kinda my way of "watching TV" - just relaxation without much thinking - I did all the thinking before, the thinking brought me to the point I decided to make it better. 

This time I'll introduce:

• Simple module model, along with the current model, which proved itself right.
• Improved class inherition schema.
• Better way of module loading.
• Better way of using Renderers (former Templates) in Components.
• Improved DataStores.
• Better way of using DataStores in Components.

I was very cautious about using different renderers in the past, because I knew it might cause a problem. This time I'll start using multiple renderers right away to make it work immediately without struggling in the future.