Well, I got many brilliant ideas for Quiky past few days. Most of them stays as TODO, to release first public version as soon as possible, but among them I got few killer features I can't wait to implement.
I tried Quiky in real life use case yesterday. I attended yet another great event, DevFest Praha 2013 by GUG CZ, and some of my notes I made using Quiky. Right there I figured out one of the killer features I don't want to write about, because I've never seen such feature anywhere else.
I also found I quite miss spell check, because I tend to use other text editors for this. Therefore I implemented NHunspell, but shortly after that I realized it would be nice to underline misspelled words. Quiky used plain multiline TextBox, which can't underline anything. I already had RichTextBox, but for future WYSIWYG only. After about a second of thinking about it I got rid of the plain TextBox and kept soloely the RichTextBox.
I was quite excited about the "switcheroo", because I didn't like the fact I have two different components for the same purpose.
Web side also got a lot of attention in the last few days. I focused on security and made few independent layers, especially when accessed from outside using Qiky API (like Quiky app).
My goal is to unify GUI for all platforms. Web will be responsive and it's not just for mobile usage. I still like Firefox's concept of a Sidebar, so I may be using Quiky there as well.
Monday, December 9, 2013
Monday, November 18, 2013
Staying low is darn hard
With Quiky I'm once again trapped in a feature loop. You know, you came up with something small and easy to make and during development you found many other (easy to make) features, that ultimately take much more time than the original idea combined. Or you figure out you gonna need this, and this, and that, so the original idea wasn't easy to make at all. That's what happened to me.
The PoC (Proof of Concept) was one evening stuff, but to make a product out of it it would need MUCH more grooming. Like settings form. I hate settings forms, it's always pain in the ass, because there's so much to do with uncertain result (of usage), but if you miss anything, you'll find an user, who will request that.
I had to stop coding, when I started playing with RichTextBox for WYSIWYG. I realized this is so much off the former road and that I need to go back to what's important.
Or you have a functionality on few lines of code, beautiful. But then it's time to sanitize all inputs, make some exception handling, value checking and other stuff, preventing the block of code from making unwanted stuff, suddenly you have a couple of new methods and from three lines are three screens of code.
Besides desktop app (which I can see I'll be reengineering in WPF, sooner or later) I made a nice progress with web app. Unfortunately I realized I'll have to redo the master code, because if I want the script to be cached, it can't show any content and therefore everything has to be made by XHR (AJAX). And it's harder I can't just delete the old code, because I want to preserve the original functionality for the improbable occasion some users may have JavaScript turned off. But I always took care of such events and this will be no exception.
Beacuse Quiky is getting serious, I registered a .net domain and created Facebook + Twitter pages.
The PoC (Proof of Concept) was one evening stuff, but to make a product out of it it would need MUCH more grooming. Like settings form. I hate settings forms, it's always pain in the ass, because there's so much to do with uncertain result (of usage), but if you miss anything, you'll find an user, who will request that.
I had to stop coding, when I started playing with RichTextBox for WYSIWYG. I realized this is so much off the former road and that I need to go back to what's important.
Or you have a functionality on few lines of code, beautiful. But then it's time to sanitize all inputs, make some exception handling, value checking and other stuff, preventing the block of code from making unwanted stuff, suddenly you have a couple of new methods and from three lines are three screens of code.
Besides desktop app (which I can see I'll be reengineering in WPF, sooner or later) I made a nice progress with web app. Unfortunately I realized I'll have to redo the master code, because if I want the script to be cached, it can't show any content and therefore everything has to be made by XHR (AJAX). And it's harder I can't just delete the old code, because I want to preserve the original functionality for the improbable occasion some users may have JavaScript turned off. But I always took care of such events and this will be no exception.
Beacuse Quiky is getting serious, I registered a .net domain and created Facebook + Twitter pages.
Thursday, November 14, 2013
Quiky
It has been a busy (and successful) month. First, I spent 2 fabulous weeks in Eastern and Southern Asia, exploring crown jewels of that area.
Right after that I attended Microsoft conference "MS Fest", where I've been introduced to MonoGame - an open source implementation of Microsoft XNA platform for easier game development. Because I have few games in mind for many years (and crashed'n'burned many times trying to make them come to life), I immediately tried to create the most ingenious of them. Thanks to wonderful YouTube tutorials from Coding Made Easy it was even easier.
I used Evernote to track all my ideas, but the lag between mobile and desktop sync was kind of frustrating. I had to deal with conflicts all the time and I got an idea to try (again) to make offline wiki editor for desktop. It supposed to be a tool for quick wiki editing, so I decided to call it "Quiky" (quick + wiki). To my surprise, quiky.net domain was available, so I took it.
For QetriX I created nice wiki parser in PHP in the past and it wasn't hard to convert the code into C#. I really love this language and .NET platform, I have to say.
After about 12 hrs the core stuff was done and I really like what I did. I made few design flaws here and there, but it wasn't hard to get rid of them. Application can work with multiple wikis, represented by a directory. Each page has it's name as filename and it's contents in the text file. When saved, a HTML file is generated thanks to my parser, and displayed in WebBrowser component.
OK, that's the desktop part. What about a mobile app? Because I have an iPhone and I don't want to jailbreak it, it was a no-go for me. Therefore I decided to try HTML5 with localStorage for offline situations.
I started slow and low, but quickly expanded into much more complex script. My main target was low data footprint for network transfers and I like to deal with such stuff (optimizations for speed and resources).
My goal was to create a web version of that desktop app, plus extend functionalities of the app to communicate with the web app. Web app will be able to merge changes line-by-line (I already have a working solution in my drawer, it was one of my little challenges :)
It will need few more hours and hours of testing, but I can feel I'm on right track there. I'm still thinking how to tie it into QetriX and the same applies for the game. I want it all to be into single environment.
Oh and the game went south a little bit :) I decided to stay low and make something simple at first, than to add more stuff and make the desired game. Until then, do you remember Tunneler and Atomic Bomberman? ;-)
Right after that I attended Microsoft conference "MS Fest", where I've been introduced to MonoGame - an open source implementation of Microsoft XNA platform for easier game development. Because I have few games in mind for many years (and crashed'n'burned many times trying to make them come to life), I immediately tried to create the most ingenious of them. Thanks to wonderful YouTube tutorials from Coding Made Easy it was even easier.
I used Evernote to track all my ideas, but the lag between mobile and desktop sync was kind of frustrating. I had to deal with conflicts all the time and I got an idea to try (again) to make offline wiki editor for desktop. It supposed to be a tool for quick wiki editing, so I decided to call it "Quiky" (quick + wiki). To my surprise, quiky.net domain was available, so I took it.
For QetriX I created nice wiki parser in PHP in the past and it wasn't hard to convert the code into C#. I really love this language and .NET platform, I have to say.
 |
| Quiky for Windows |
OK, that's the desktop part. What about a mobile app? Because I have an iPhone and I don't want to jailbreak it, it was a no-go for me. Therefore I decided to try HTML5 with localStorage for offline situations.
I started slow and low, but quickly expanded into much more complex script. My main target was low data footprint for network transfers and I like to deal with such stuff (optimizations for speed and resources).
My goal was to create a web version of that desktop app, plus extend functionalities of the app to communicate with the web app. Web app will be able to merge changes line-by-line (I already have a working solution in my drawer, it was one of my little challenges :)
It will need few more hours and hours of testing, but I can feel I'm on right track there. I'm still thinking how to tie it into QetriX and the same applies for the game. I want it all to be into single environment.
Oh and the game went south a little bit :) I decided to stay low and make something simple at first, than to add more stuff and make the desired game. Until then, do you remember Tunneler and Atomic Bomberman? ;-)
Saturday, October 12, 2013
Making progress
I'm aware I didn't publish a post in more than a month, but during the time I was (and still em) cracking on the final cut of QetriX for PHP.
So far I have the object and database model, I have HTML template for all major components: page, table, navigation, time sheet, form with controls etc.
I also determined file uploads and messaging/notifications thru the system. Still need to implement user accounts though, that involves user groups, roles and privileges as well. There's no general configuration at the moment, but it's also designed.
I tried to recycle some code from the previous version, but It was mostly useless, which made me to realize, how different these versions actually are, even if I'm doing pretty much the same.
It takes quite a lot of time, because I'm developing many different things at once - to make it as similar as possible across different platforms and languages it's not only PHP and HTML, but also C# with WPF and Java with GWT and Spring. I heard many great stuff about Node.js, so I'm going to include it to the process as well.
Everything I figure out I immediately publish on Wiki.
So far I have the object and database model, I have HTML template for all major components: page, table, navigation, time sheet, form with controls etc.
I also determined file uploads and messaging/notifications thru the system. Still need to implement user accounts though, that involves user groups, roles and privileges as well. There's no general configuration at the moment, but it's also designed.
I tried to recycle some code from the previous version, but It was mostly useless, which made me to realize, how different these versions actually are, even if I'm doing pretty much the same.
It takes quite a lot of time, because I'm developing many different things at once - to make it as similar as possible across different platforms and languages it's not only PHP and HTML, but also C# with WPF and Java with GWT and Spring. I heard many great stuff about Node.js, so I'm going to include it to the process as well.
Everything I figure out I immediately publish on Wiki.
Tuesday, September 10, 2013
Video from PowerPoint presentation
I've been looking for ages for some easy-to-use animation software, preferably vector, where I can design simple product videos or video tutorials. I tried many of them, but none of them was quite user friendly for a newbie.
I always liked drawing in Microsoft PowerPoint. Sometimes I created simple pictures or schemes in it, recently I drew a QetriX Logo. And then it hit me - why don't I use PowerPoint as an animator? It supports export presentation as WMV (Windows Media Video), even the final quality is not quite good and max output resolution is 960x720.
First step is to change slide size from default 4:3 to 16:9, using "Page Setup" button on a "Design" tab. If you do it later, the graphics will be deformed (following the change of an aspect ratio) and you'll have to change the scale of a size from 133 % back to 100 % (in Format Picture / Format Shape, then Size).
Second step is to define all objects in the slide. You can use various shapes, such as lines, circles, rectangles, polygons, arrows and much, much more. You can also add some text. It doesn't matter, if you choose Text Box or a Word Art - it's the same.
You can use more slides, or just a single one. I definitely recommend more slides, because sometimes on a single slide it gets quite complicated :)
Now add some (tasteful!) animations for shapes/pictures and transitions between slides as well. Less is more, don't get carried away and make something like 1990s website. You can make some gorgeous stuff, but you need to play with it a little.
I'm not going to describe the full process of creation, because there's tons of options and it's not that hard to figure it on your own.
You can add some music and/or narration as well.
When your job is done, choose "Create a video" in "Save & Send" option. Make sure quality is set to Computer & HD displays and click to "Create Video" button.
Now wait, until it's fully rendered and there you go.
I always liked drawing in Microsoft PowerPoint. Sometimes I created simple pictures or schemes in it, recently I drew a QetriX Logo. And then it hit me - why don't I use PowerPoint as an animator? It supports export presentation as WMV (Windows Media Video), even the final quality is not quite good and max output resolution is 960x720.
First step is to change slide size from default 4:3 to 16:9, using "Page Setup" button on a "Design" tab. If you do it later, the graphics will be deformed (following the change of an aspect ratio) and you'll have to change the scale of a size from 133 % back to 100 % (in Format Picture / Format Shape, then Size).
Second step is to define all objects in the slide. You can use various shapes, such as lines, circles, rectangles, polygons, arrows and much, much more. You can also add some text. It doesn't matter, if you choose Text Box or a Word Art - it's the same.
You can use more slides, or just a single one. I definitely recommend more slides, because sometimes on a single slide it gets quite complicated :)
Now add some (tasteful!) animations for shapes/pictures and transitions between slides as well. Less is more, don't get carried away and make something like 1990s website. You can make some gorgeous stuff, but you need to play with it a little.
I'm not going to describe the full process of creation, because there's tons of options and it's not that hard to figure it on your own.
You can add some music and/or narration as well.
When your job is done, choose "Create a video" in "Save & Send" option. Make sure quality is set to Computer & HD displays and click to "Create Video" button.
Now wait, until it's fully rendered and there you go.
Monday, August 26, 2013
Do I need a "flag"?
Few days ago I had an idea I can't get off my head: Do I really need the "flag" property? Now I use it to define importance of an entity and mostly to disable certain particles or entities, but all these things can be done by "order" property.
I don't use order for entities, so it may act as flag. 0 = disabled, 1 = new, 2 = unverified, 3 = verified. I use order for date or appearence length of an actor in particular movie, so it already has a value.
I feel there's no need for the "flag" anymore. It came from the time I thought I may stuck four different "values" into single particle, but this was just a wrong assuption. Everything should be in the value only, everything else is additional value. DateTime in order is great, but only as a "timestamp" for the value. And if I fill the timestamp, why would I want to hide it? I always can set order for the entity to zero and it won't show up anyway.
Sure, there might be situations, when I need to hide the particle without losing any information in the order property, but more often I will take advantage of smaller data model and less code in the framework.
I don't use order for entities, so it may act as flag. 0 = disabled, 1 = new, 2 = unverified, 3 = verified. I use order for date or appearence length of an actor in particular movie, so it already has a value.
I feel there's no need for the "flag" anymore. It came from the time I thought I may stuck four different "values" into single particle, but this was just a wrong assuption. Everything should be in the value only, everything else is additional value. DateTime in order is great, but only as a "timestamp" for the value. And if I fill the timestamp, why would I want to hide it? I always can set order for the entity to zero and it won't show up anyway.
Sure, there might be situations, when I need to hide the particle without losing any information in the order property, but more often I will take advantage of smaller data model and less code in the framework.
Wednesday, August 14, 2013
Hello World Showcase
One day I had an interesting idea. I already have QetriX project template in several languages for several platforms: PHP, JavaScript, C# (WinForms, WPF, ASPX), Java (Spring, Android) and Objective-C for iOS and OS X. Everything does the same and is coded in the same way.
Why not create a something like Hello World showcase? Usually, "Hello World" is just a simple piece of code to show how the programming language works. I have quite simple yet complete data solution, where I can demonstrate much more - the whole programming at glance. And not only the code, but different data management approaches (like databases) as well!
As I mentioned before, I'm very aware my programming habits is nothing to demonstrate. Also, each platform and language works differently and the same construction might be efficient in one language, but highly inefficient or even impossible to do in the other(s).
Therefore I decided to make more than one version for platforms where applicable - one in "compatibility mode", where all constructions will be the same (or at least very similar) across all languages and it will be easy to compare programming techniques between different languages and platforms, and one or more in "lang native mode", where the most efficient techniques and best practices will be in use.
In PHP there are many different approaches. Therefore I'll try to create one QetriX in pure functional programming, one in reasonably objective programming and one in pure OOP. I've just got an idea to even use different frameworks and CMS (CodeIgniter, Zend, Nette, Symfony, FatFree, CakePHP, WordPress, MediaWiki...) to recreate QetriX in them.
It would be nice to see the difference between them, how many lines of code and what performance results it will provide. I'm quite looking forward to do this.
Why not create a something like Hello World showcase? Usually, "Hello World" is just a simple piece of code to show how the programming language works. I have quite simple yet complete data solution, where I can demonstrate much more - the whole programming at glance. And not only the code, but different data management approaches (like databases) as well!
As I mentioned before, I'm very aware my programming habits is nothing to demonstrate. Also, each platform and language works differently and the same construction might be efficient in one language, but highly inefficient or even impossible to do in the other(s).
Therefore I decided to make more than one version for platforms where applicable - one in "compatibility mode", where all constructions will be the same (or at least very similar) across all languages and it will be easy to compare programming techniques between different languages and platforms, and one or more in "lang native mode", where the most efficient techniques and best practices will be in use.
In PHP there are many different approaches. Therefore I'll try to create one QetriX in pure functional programming, one in reasonably objective programming and one in pure OOP. I've just got an idea to even use different frameworks and CMS (CodeIgniter, Zend, Nette, Symfony, FatFree, CakePHP, WordPress, MediaWiki...) to recreate QetriX in them.
It would be nice to see the difference between them, how many lines of code and what performance results it will provide. I'm quite looking forward to do this.
Sunday, August 4, 2013
FindTheBest.com
Holy cow, how did it manage to hide from me the whole time? I don't remember how I came across this website, but I have to admit, when I spent a few minutes clicking here and there on it, I was truly dumbfounded.
It isn't really IT, but it is so damn close to what I'm working at. Sure, I knew there's a huge competition since I started, but all those freebases and wolfram|alphas were either different, ugly or hard to use. This is none of them. The only good thing for me is that the main turf for the website are mostly the States and I'm aiming higher.
On the other hand, they have a huge amount of pictures, awesome graphs, nice filtering options with excellent results interpretation, overwhelming wizard-like "assist me" function and much, much more.
When I shaked the surprise off, I started to pick features I like and think how to implement them into QetriX.
I assume it's also written in PHP, since /index.php returns no 404. I was trying to find their data sources, but I found only a simple statement: "We obtain our information from three sources: Public databases, primary sources (manufacturer websites) and expert sources." Well, sincerely I didn't expect much more, since data sources are the main business secrets here.
It isn't really IT, but it is so damn close to what I'm working at. Sure, I knew there's a huge competition since I started, but all those freebases and wolfram|alphas were either different, ugly or hard to use. This is none of them. The only good thing for me is that the main turf for the website are mostly the States and I'm aiming higher.
On the other hand, they have a huge amount of pictures, awesome graphs, nice filtering options with excellent results interpretation, overwhelming wizard-like "assist me" function and much, much more.
When I shaked the surprise off, I started to pick features I like and think how to implement them into QetriX.
- Classifications should have default icon and picture.
- For attribs there should be an information about what value (lower/higher) is better. A weights for this would be nice too.
- User friendly filtering, using not just values and check boxes, but nice min-max sliders as well. Domains will be handy here.
- Entity should me even more specific, with charts and tables.
- Related entities are necessary.
- Besides the main entity it would be handy to attach small info-box for related entities too, such as manufacturer, superior area unit, homeland etc.
- User ratings and comments would be there again. Comments with rating also (up/down vote, likes...).
I assume it's also written in PHP, since /index.php returns no 404. I was trying to find their data sources, but I found only a simple statement: "We obtain our information from three sources: Public databases, primary sources (manufacturer websites) and expert sources." Well, sincerely I didn't expect much more, since data sources are the main business secrets here.
Tuesday, July 23, 2013
Something about Crawler
As I wrote earlier, I was confident I can collect data for the Particle Database by myself, but shortly I realized it's just too much. So after a struggle I decided to make a web crawler for harvesting semantic data.
The decision of target platform was quick - PHP is not suitable, so .NET. At first I tried to customize some of those existing open source crawlers, but it was a pain. So ultimately I wrote my own. I don't need any GUI, so simple console app was suitable.
It downloads plain text data (e.g. HTML page), using HttpWebRequest class, and extracts all links to other pages. For every use I can define a set of boundaries, so links out of these boundaries are throwed away. The rest are put into queue. Robots.txt and meta tags are a part of those boundaries as well.
Then crawler prepares data for parsing - strips off all unimportant parts and replaces or deletes some parts (like whitespaces). Then it checks for presence of defined string, indicating there are some data to extract. And finally, using regular expressions, structured data are extracted and send to server-side for further processing.
On server side is PHP script, which post-processes and saves data into database, finds relevant links to another entities and stores URL of data source to avoid duplicates.
Then crawler politely waits a while before loading next URL from queue, because I don't want to overload these servers. And... that's it! Simple, yet powerful.
The decision of target platform was quick - PHP is not suitable, so .NET. At first I tried to customize some of those existing open source crawlers, but it was a pain. So ultimately I wrote my own. I don't need any GUI, so simple console app was suitable.
It downloads plain text data (e.g. HTML page), using HttpWebRequest class, and extracts all links to other pages. For every use I can define a set of boundaries, so links out of these boundaries are throwed away. The rest are put into queue. Robots.txt and meta tags are a part of those boundaries as well.
Then crawler prepares data for parsing - strips off all unimportant parts and replaces or deletes some parts (like whitespaces). Then it checks for presence of defined string, indicating there are some data to extract. And finally, using regular expressions, structured data are extracted and send to server-side for further processing.
On server side is PHP script, which post-processes and saves data into database, finds relevant links to another entities and stores URL of data source to avoid duplicates.
Then crawler politely waits a while before loading next URL from queue, because I don't want to overload these servers. And... that's it! Simple, yet powerful.
Tuesday, July 9, 2013
MobiletriX
There are two main approaches for mobile version of a website or a web application (well, three, if you consider "none" as an option): responsive webdesign and separate mobile version.
I decided to do both. I really like the concept of responsive design, but I hate when I have to download something I won't use (graphics, HTML elements etc.).
I decided to do both. I really like the concept of responsive design, but I hate when I have to download something I won't use (graphics, HTML elements etc.).
Mobile version should be lightweight and may be little uglier by not having all the fancy bells and whistles, which on mobile devices just consumes bandwidth and processor time anyway.
Mobile users also usually want just to hop in, do anything quickly and hop out, especially when the mobile version isn't the main way of using your site/app. Therefore the original use cases might be little crooked and it's ok. Developer should not throw logs under user's legs, but stick with him instead.
Users mostly expect similar, but slightly different behavior between desktop, web and mobile app. Each of them is used in different situations, where user would appreciate different things. It's even determined by time of the day, but it would get too far :)
It's also a bad habit to emulate his device's interface a you can never satisfy all of them. And believe me, iPhone UI on Android phone looks extremely stupid.
Wednesday, July 3, 2013
Reengineering AGAIN?
And again :) I don't have to worry about anything right now, as everything is doing just fine with the previous version and nobody rely on the new one.
I wanted to add more functionality to the code and realized I pasted a lot of code in the core without further investigation what it really does and how good it is in sake of performance. And because I did some thinking about the object model, I decided to start over with no old code.
I know it will take a lot of time, but from my point it's time well wasted. Reengineering is kinda my way of "watching TV" - just relaxation without much thinking - I did all the thinking before, the thinking brought me to the point I decided to make it better.
This time I'll introduce:
- Simple module model, along with the current model, which proved itself right.
- Improved class inherition schema.
- Better way of module loading.
- Better way of using Renderers (former Templates) in Components.
- Improved DataStores.
- Better way of using DataStores in Components.
I was very cautious about using different renderers in the past, because I knew it might cause a problem. This time I'll start using multiple renderers right away to make it work immediately without struggling in the future.
Tuesday, June 25, 2013
Limits of the FREE version
It wasn't easy for me to decide how limited the FREE version should be. Sure, I want users motivate to pay at least something, but I don't want to oust them completely.
I hate few things in desktop apps, even I understand their purpose. I hate grayed out menu options, just because PRO version has it. I hate even more the unavailable option isn't distinguished and I learn I have to pay first only after I click on the item.
I can live with some promo upon startup or closing the app, because usually I don't do it too often. But only if it's in a "polite" way, when I don't have to hunt the closing window nor close automatically opened tab in my browser.
I like x day trials, when the app works fully, but my problem with trial versions is, that it has to write the start date somewhere, obviously, and this thing stays on my computer even after I got rid of the software, again - obviously, but after that it's just garbage.
I hate online DRM, even it solves the problem of crap residing in my computer.
So, what DO I like? I like when the only limitation is I can't save my work locally, but I'm freely able to do anything it offers. After purchase there is a encrypted file with the license and this way I will do it with QetriX.
I hate few things in desktop apps, even I understand their purpose. I hate grayed out menu options, just because PRO version has it. I hate even more the unavailable option isn't distinguished and I learn I have to pay first only after I click on the item.
I can live with some promo upon startup or closing the app, because usually I don't do it too often. But only if it's in a "polite" way, when I don't have to hunt the closing window nor close automatically opened tab in my browser.
I like x day trials, when the app works fully, but my problem with trial versions is, that it has to write the start date somewhere, obviously, and this thing stays on my computer even after I got rid of the software, again - obviously, but after that it's just garbage.
I hate online DRM, even it solves the problem of crap residing in my computer.
So, what DO I like? I like when the only limitation is I can't save my work locally, but I'm freely able to do anything it offers. After purchase there is a encrypted file with the license and this way I will do it with QetriX.
And some polite ads for FREE users, even many of them know how to block them. I believe users can live with tasteful and just slightly disturbing (not by contents) ones.
Wednesday, June 19, 2013
Semantic Dictionary
One of my "unfinished businesses" is a semantic dictionary. Basically it will be a gigantic database of all words in all forms (as entities) and they'll have a set of attributes and relations between them, defining what kind of word it is (subject, verb...) and what relations between other words are (synonym, antonym, the same word in different language...).
I expect the primary usage in semantic search. The search engine will use some kind of universal meta language, to which all queries in natural languages will be transformed.
I expect the primary usage in semantic search. The search engine will use some kind of universal meta language, to which all queries in natural languages will be transformed.
I can't say how (and if) it will work exactly, it's just a thing I've been thinking of quite a long time. I'm aware there are some nuances and pickles (like the same word with different meaning not only in different languages) or how to deal with phrases in search queries (my former thought was just to analyse word by word, but It's not that easy).
These are some of expected unknowns I didn't make any further research about and therefore I don't have comprehensive overview of it.
And, last, but not least, it will be a nice performance test :)
Thursday, June 13, 2013
Moved wiki from Wikia
On Monday I moved our wiki from Wikia to our own system. Why? Well, Wikia is great, simple and intuitive. You can use many advanced features, but you also have to undergo ads and Wikia network. There is too much clutter for my taste.
Wikia was temporary for me anyway, but I thought I would migrate to system built on QetriX. Recently I decided to do some major improvements, so I didn't want to rush things. I already run few wiki instances on my own wiki engine, so I had no other reason to stay at Wikia.
I only had to migrate the current content (about 30 pages) and customize style sheet (CSS). I like it much more, than Wikia version, and also the URL is much nicer: wiki.qetrix.com. So take a look! :)
Wikia was temporary for me anyway, but I thought I would migrate to system built on QetriX. Recently I decided to do some major improvements, so I didn't want to rush things. I already run few wiki instances on my own wiki engine, so I had no other reason to stay at Wikia.
I only had to migrate the current content (about 30 pages) and customize style sheet (CSS). I like it much more, than Wikia version, and also the URL is much nicer: wiki.qetrix.com. So take a look! :)
Tuesday, June 11, 2013
Google Glass
People say that New York City you can either love or hate, noting in between. From buzzing around Google Glass I can say the same about this gadget. Even though I find myself on the side of "hate", as I see it as a major breakthrough into my privacy, from QetriX's point of view it's a major opportunity. It's exactly what I'm focusing to in this project - tiny pieces of important/interesting information on the go.
The true resolution of the display remains unknown, but for the apps Google recommends 640×360. There are also quite strict guidelines for creating apps, also made by Google, which may keep the environment homogeneous.
I can tell apart those opposing feelings easily, because my personal concern is about the built-in camera, whereas the opportunity is spurting from the head-mounted (or "heads up") display.
The true resolution of the display remains unknown, but for the apps Google recommends 640×360. There are also quite strict guidelines for creating apps, also made by Google, which may keep the environment homogeneous.
I'm working on use cases for small resolutions, because Glass is not the only such device. We're more hearing about various wearable devices these days, like bracelets or watches (Pebble).
Wednesday, June 5, 2013
Desktop App
When I was searching for inspiration thru AppStore, I found a great ToDo app, called Wunderlist2. I didn't need it, because I was already using Evernote for ToDos, but I like it's completely free and has mobile + web + desktop access - exactly what I admired Evernote for and what I was aiming at with QetriX.
But it was pretty neat. Like Evernote it looked the same on all platforms and unlike Evernote it featured beautiful desktop App. I was wondering how it's possible to achieve this look and my first thought was it's some kind of HTML5 wrapper. I knew Evernote was written in Qt and I made several apps in WinForms, but neither was an option. After a little googling I discovered WPF.
I came across WPF a while ago, when I looked for a simple way for 3D engine and I was playing with XNA little bit, but I thought "what the hell I would use this for" :)
WPF, like Android or ASP.NET, has GUI defined in XML and attached functionality in C# (in my case). I was overwhelmed by the GUI definitions, but fortunately I was able to figure out how the Wunderlist app is made, thanks to the WPF Inspector from CodePlex.
I don't rush with the app, but I'm glad I did these first steps and in the meantime I can think about the final product.
Wednesday, May 29, 2013
Less Frame, More Work
When I started to code QetriX in PHP, I didn't plan to release it as a framework. But with progress of the concept my perspective slightly changed.
How it works:
I don't want to look like I'm hiding something there. I know my coding habits are sometimes obsolete, but I'm strictly against today's beloved bloats, like giant classes in PHP even in situations, when simple function would do the job as well. I want my code quick and sleek, performance tests speaks loudly against OOP in PHP. In latest PHP builds it's better, but still little slower and more memory consuming.
On the other hand because I decided to unify the code, I have no choice but use objects. Making full OOP in PHP is like mounting wings on a car and trying to fly with that - it takes a lot of time and effort and the result is questionable. So I will try use the OOP wisely, only in situations, that otherwise would be problematic (like no object for "toolbox" or app itself).
Except for that, I don't want anyone to change anything in the code (PHP, JavaScript), everything is there for a reason. If you need anything to work in a different way, you may consider to look somewhere else. The framework is built to be "just enough" for the purpose it serves. I want full out-of-the-box experience.
First of all, the idea is cross platform, cross environment and cross language. It says just "what", not "how". Second of all, I think the idea may be used in a wrong way, so It's gonna be better, if I set the rules here and let fellow programmers just to follow them and stick with them.
Because all languages I'm focusing on are very similar, I decided to unify the code across all these langs. I'm talking about PHP, Java and C# here.
How it works:
- Install
- Profit
I don't want to look like I'm hiding something there. I know my coding habits are sometimes obsolete, but I'm strictly against today's beloved bloats, like giant classes in PHP even in situations, when simple function would do the job as well. I want my code quick and sleek, performance tests speaks loudly against OOP in PHP. In latest PHP builds it's better, but still little slower and more memory consuming.
On the other hand because I decided to unify the code, I have no choice but use objects. Making full OOP in PHP is like mounting wings on a car and trying to fly with that - it takes a lot of time and effort and the result is questionable. So I will try use the OOP wisely, only in situations, that otherwise would be problematic (like no object for "toolbox" or app itself).
Except for that, I don't want anyone to change anything in the code (PHP, JavaScript), everything is there for a reason. If you need anything to work in a different way, you may consider to look somewhere else. The framework is built to be "just enough" for the purpose it serves. I want full out-of-the-box experience.
During implementation it's possible to add a new module, but again - with deep respect to the ecosystem. The only part I'm completely OK about changes is CSS, but even there I created everything you may need.
Regardless, I'd love to hear a suggestions or constructive criticism (preferably with refactored code), if it stays in defined boundaries.
Regardless, I'd love to hear a suggestions or constructive criticism (preferably with refactored code), if it stays in defined boundaries.
Friday, May 17, 2013
Pythagorean theorem in Cue Editor
Last month I presented Cue language for scripting events in QetriX. I mentioned I created an online editor, but I didn't show it here. So here is a screenshot: (it's shrinked together as I wanted to save pixels here)
On the very left there is a block schema, where you can see a flow of all commands, because Cue utilizes a lot of "goto" commands and it may be easy to get lost in the code.
In the gray box, titled "Events", you can see the source code on the left an and debug output and trace on the right. Output is like a console, but here is nothing to print out. In Trace you can see what commands in what order (and with what result) has been executed.
"RESULT" is the value, what is returned into code after processing the Cue code.
In the code I defined two constants, but you can get those input values from QetriX, or e.g. GET, POST or COOKIE variables. So first two lines contains numbers 3 and 4. Pythagorean theorem for calculation the hypotenuse (variable "c") is: "c = sqrt(a² + b²)", so first of all we have to calculate square power for "a" and "b" (lines 3 and 4, showing two different ways how to achieve that, using "mul" and "pow" commands). Then we add those two results together and finally we calculate a square root of the sum.
There are no gotos or decisions, so the code is nicely straight - from Start ("S" circle) to End ("E" circle). Final result of the code is the same, as the result of the last line.
 |
| Pythagorean theorem in Cue Editor |
In the gray box, titled "Events", you can see the source code on the left an and debug output and trace on the right. Output is like a console, but here is nothing to print out. In Trace you can see what commands in what order (and with what result) has been executed.
"RESULT" is the value, what is returned into code after processing the Cue code.
In the code I defined two constants, but you can get those input values from QetriX, or e.g. GET, POST or COOKIE variables. So first two lines contains numbers 3 and 4. Pythagorean theorem for calculation the hypotenuse (variable "c") is: "c = sqrt(a² + b²)", so first of all we have to calculate square power for "a" and "b" (lines 3 and 4, showing two different ways how to achieve that, using "mul" and "pow" commands). Then we add those two results together and finally we calculate a square root of the sum.
There are no gotos or decisions, so the code is nicely straight - from Start ("S" circle) to End ("E" circle). Final result of the code is the same, as the result of the last line.
Tuesday, May 14, 2013
GeoQetriX
I've always been into geographic stuff and I'm not talking about geography back at school :) The very first version of what one day emerged as QetriX already gathered coordinates and I tried to visualize them on maps, especially custom made (calculated) from vectors. I bought an outdoor GPS device and visualized tracks from our long distance walks (sometimes more than 30 miles in a single day).
One day I grabbed few ortophoto tiles from Mapy.cz, Czech competitor to Google Maps, joined them into larger area and made math function to put POI (points of interest) icons on it precisely. Then I found a function to calculate a distance between two points, because on spherical object it's not just a calculation of hypotenuse, and used points for such calculations.
Years later, after I designed the Particle, I started to think about "geographical particle". I have to admit I still didn't make any further progress, I'm stuck with three tables: point, line and area. Pavel (he's a surveyor) told me about "topology", which consists right of these three components. There are points, line is a connection of two points and has an area on each side. It's easy to convert it into 3D afterwards just by adding 3rd coordinate to the point, because no matter how hard you try, you always end up with point-line-area. And I'd like to get rid at least of one table, but I can't see how. Pavel encouraged me to do so, referring to what I did about Particle. We'll see how successful will it be this time.
 |
| Old Q3X app with map module, utilizing Google Static Map API |
 |
| Map of POIs in QetriX two years ago (2011) |
Wednesday, May 8, 2013
Android App
Java has always been somehow unlucky for me, even though I have a lot of experience with similar languages, like C# and JavaScript. I tried it and failed several times over last five, six years. I was able to make the Hello World to work, but nothing beyond. This applies for Android as well, I remember installing the SDK two or three times in the past. Maybe because I used NetBeans and then Eclipse. But this time I tried IntelliJ IDEA (community edition) from JetBrains.
The first steps were obvious: File > New Project... > Android Application Module, fill the project form and create the project, package "com.qetrix". My test device is old Android 2.2, so I selected this one and API 8. But what now? I knew something about Activities but I had no idea what it's all about. I read somewhere it's like a Controller in MVC. OK, but what about view (something like Form in .NET)?
I tried to add new GUI Form, but it was dead end. I tried tutorial from JetBrains and I learned the GUI is defined as XML file in res/layout/ directory. I created new XML and - voila! Nice. I started adding controls and for me it was quite understable.
I created the screen with a list of databases (ListView with ArrayAdapter) and compiled it. After a while it launched an emulator and after it booter and I unlocked the screen, the app appeared. Fantastic! :)
Then I created new form as Login, set it as startup and launched the other from this one, using ".startActivity(myIntent);". This was a pivotal moment for me, from this point on it was just a matter of time, manuals and best practices.
The development for Android is really fun after all :)
The first steps were obvious: File > New Project... > Android Application Module, fill the project form and create the project, package "com.qetrix". My test device is old Android 2.2, so I selected this one and API 8. But what now? I knew something about Activities but I had no idea what it's all about. I read somewhere it's like a Controller in MVC. OK, but what about view (something like Form in .NET)?
I tried to add new GUI Form, but it was dead end. I tried tutorial from JetBrains and I learned the GUI is defined as XML file in res/layout/ directory. I created new XML and - voila! Nice. I started adding controls and for me it was quite understable.
I created the screen with a list of databases (ListView with ArrayAdapter) and compiled it. After a while it launched an emulator and after it booter and I unlocked the screen, the app appeared. Fantastic! :)
Then I created new form as Login, set it as startup and launched the other from this one, using ".startActivity(myIntent);". This was a pivotal moment for me, from this point on it was just a matter of time, manuals and best practices.
 |
| Login screen (Czech locale). Oldschool - I know :) |
The development for Android is really fun after all :)
Tuesday, April 30, 2013
Public Service
For a few years I had a vision to open QetriX for public, but not as a particle database, but more like a farm of databases. My inspiration was the concept of Wikia.com, where everyone can start his own wiki. I decided to do a similar thing, only with semantic/structured databases instead of wikis.
Crucial part is unfortunately my biggest weakness - to make it easy for the user. I began with stripping the user interface and ultimately reduced offered functionality to the thinnest core. My target was to make it "just enough". To keep the power in the backend and offer a bunch of presets, so to speak.
Application layers are: Databases > Classifications > Records (entities) > Attributes and relations.
In a database, user will be able to create classifications (in fact entity types, only available choice will be the name), attribute types (name, unit for numeric values, and data type) and relation types (name only).
Additionally, user can pick everything from preset templates, one set for home and the other for business. Templates can offer settings beyond the default available settings, like value range (e.g. 0-120 for age of a person).
For relations I dealt with unwanted entity types in suggest list, but how to get rid of them, when such thing is not available for this installation? Well, I made a little hack in the suggest algorithm, so when the name of relation type matches a name of any classification (entity type), only such entities are shown.
It's the first public release of QetriX aimed particularly to end users as content creators. QetriX had a public release two years ago, as Particle Database, but it was mainly to receive some feedback (which it did). Although I kinda like current style of QetriX, I feel this isn't "it". I can imagine it's far too off end-user's expectations, may be confusing and hard to understand. But I'm still trying to think out of the box, at least I figured out some nifty stuff this time :)
Crucial part is unfortunately my biggest weakness - to make it easy for the user. I began with stripping the user interface and ultimately reduced offered functionality to the thinnest core. My target was to make it "just enough". To keep the power in the backend and offer a bunch of presets, so to speak.
Application layers are: Databases > Classifications > Records (entities) > Attributes and relations.
In a database, user will be able to create classifications (in fact entity types, only available choice will be the name), attribute types (name, unit for numeric values, and data type) and relation types (name only).
Additionally, user can pick everything from preset templates, one set for home and the other for business. Templates can offer settings beyond the default available settings, like value range (e.g. 0-120 for age of a person).
For relations I dealt with unwanted entity types in suggest list, but how to get rid of them, when such thing is not available for this installation? Well, I made a little hack in the suggest algorithm, so when the name of relation type matches a name of any classification (entity type), only such entities are shown.
It's the first public release of QetriX aimed particularly to end users as content creators. QetriX had a public release two years ago, as Particle Database, but it was mainly to receive some feedback (which it did). Although I kinda like current style of QetriX, I feel this isn't "it". I can imagine it's far too off end-user's expectations, may be confusing and hard to understand. But I'm still trying to think out of the box, at least I figured out some nifty stuff this time :)
Tuesday, April 23, 2013
Cool Events
I tried to came up with a cool title for this hot spring day, so I hope this is cool enough ;-)
Event is triggered on a specific occasion. It can be specific timestamp (wildcards are possible) or any kind of manipulation with entity/attrib/relation type or even specific entity/attrib/relation.
Main purpose of an event is to launch associated script (set of commands) in a Cue scripting language, described earlier in this blog.
My first hunger for events was in my early project (2007), where everything was hardcoded. And except nobody has an idea how it actually works, when something was wrong, it was pain to correct it. For the next version I designed (in my head) a way, how to make it editable. Unfortunately there was nobody willing to pay the development, so the idea stayed just in my head.
I don't need any bells or whistles, I just need a tool for designing a list of consecutive commands and some minor evaluating on the way. It's like "when this, do that".
For timing events there will be a cron or a job (mostly every day or every hour), for data manipulation there will be some hooks hardcoded into CRUD methods. Nevertheless, the system has to be detachable, because I want to have system of independent layers (as I hate to execute unnecessary code over and over again).
And what is so cool about those events? I won't have to hardcode specific behavior any more, I'll just set a trigger (event) and write some commands to be executed. And that pays off.
Event is triggered on a specific occasion. It can be specific timestamp (wildcards are possible) or any kind of manipulation with entity/attrib/relation type or even specific entity/attrib/relation.
Main purpose of an event is to launch associated script (set of commands) in a Cue scripting language, described earlier in this blog.
My first hunger for events was in my early project (2007), where everything was hardcoded. And except nobody has an idea how it actually works, when something was wrong, it was pain to correct it. For the next version I designed (in my head) a way, how to make it editable. Unfortunately there was nobody willing to pay the development, so the idea stayed just in my head.
I don't need any bells or whistles, I just need a tool for designing a list of consecutive commands and some minor evaluating on the way. It's like "when this, do that".
For timing events there will be a cron or a job (mostly every day or every hour), for data manipulation there will be some hooks hardcoded into CRUD methods. Nevertheless, the system has to be detachable, because I want to have system of independent layers (as I hate to execute unnecessary code over and over again).
And what is so cool about those events? I won't have to hardcode specific behavior any more, I'll just set a trigger (event) and write some commands to be executed. And that pays off.
Thursday, April 18, 2013
Cue for events
Everything in QetriX is somehow unique or innovative. From my previous projects I knew I would need a simple scripting. I tried LUA, but it was too complex. I tried JavaScript, but it was even more complex :)
In my mind I already had a concept of simple scripting language. Every line will contain one command and, like in Assembler, there will be a keyword first, then arguments separated by space. I didn't care about spaces in text, because only language keywords would be allowed and they won't contain spaces anyway.
I call this language "Cue", because in English it has the same pronunciation as "Q" and it's meaning is similar to purpose of the language - to trigger an action.
From general perspective it's a bad language, because it has wildly disputed "goto" command. And even worse - "goto" here is the only command for structure, as there are no blocks (like curly braces in C) and all functions has to be coded in advance in platform's source code. From Cue you can only call them with arguments and retrieve their return value.
There are no variables in Cue, the result of each command is stored under a number of the line, where the command is. When you need the value, you can simply recall it by using hash and number of the line, e.g. #5 to get a value from the fifth line.
There are a major catch for editing the code, when number of lines changes due to add/remove lines in the middle of the code. For this reason there is a smart code editor "out of the box", which also draws a diagram of the source code (little help to manage all those gotos) and provides inline help for each command and it's arguments. It also allows to run the code and display a step trace.
Now let me show you how to code (gray comments are not part of the code):
The only thing I don't like (but I don't see any other option, so I dealt with it) are conditions; IF statements, so to speak. Because it technically consists of two parts - condition and then/else branches, I had to split it onto two lines. In the first one I eval the condition (and store true/false result), the second one is if-goto statement. There is no else branch, because if the condition is false, the parser goes to the next line (which is, in fact, the else branch).
I thought about more complex "if" statement, but only this way you can use multiple conditions clearly. The only hack is, that "if" is always if-goto, it's NOT an if-command, because it would violate the single-line-single-command concept.
The best usage though is to define actions between two states in a work flow. In QetriX you can define events, as a starting point for launching Cue script. Events are fired in specific occasion, but I'll write about it next time. For more info about Cue please visit our wiki.
In my mind I already had a concept of simple scripting language. Every line will contain one command and, like in Assembler, there will be a keyword first, then arguments separated by space. I didn't care about spaces in text, because only language keywords would be allowed and they won't contain spaces anyway.
I call this language "Cue", because in English it has the same pronunciation as "Q" and it's meaning is similar to purpose of the language - to trigger an action.
From general perspective it's a bad language, because it has wildly disputed "goto" command. And even worse - "goto" here is the only command for structure, as there are no blocks (like curly braces in C) and all functions has to be coded in advance in platform's source code. From Cue you can only call them with arguments and retrieve their return value.
There are no variables in Cue, the result of each command is stored under a number of the line, where the command is. When you need the value, you can simply recall it by using hash and number of the line, e.g. #5 to get a value from the fifth line.
There are a major catch for editing the code, when number of lines changes due to add/remove lines in the middle of the code. For this reason there is a smart code editor "out of the box", which also draws a diagram of the source code (little help to manage all those gotos) and provides inline help for each command and it's arguments. It also allows to run the code and display a step trace.
Now let me show you how to code (gray comments are not part of the code):
- add 5 4 // calculates 5 + 4 and stores the result (9) into line number
- sqrt #1 // calculates square root of the result from line 1 (=9)
The only thing I don't like (but I don't see any other option, so I dealt with it) are conditions; IF statements, so to speak. Because it technically consists of two parts - condition and then/else branches, I had to split it onto two lines. In the first one I eval the condition (and store true/false result), the second one is if-goto statement. There is no else branch, because if the condition is false, the parser goes to the next line (which is, in fact, the else branch).
I thought about more complex "if" statement, but only this way you can use multiple conditions clearly. The only hack is, that "if" is always if-goto, it's NOT an if-command, because it would violate the single-line-single-command concept.
 |
| QetriX Cue Code Editor |
The best usage though is to define actions between two states in a work flow. In QetriX you can define events, as a starting point for launching Cue script. Events are fired in specific occasion, but I'll write about it next time. For more info about Cue please visit our wiki.
Saturday, April 13, 2013
Flat File Database
Because I don't want excessive database use, I decided for FREE version of QetriX to use flat files as data storage.
Flat files are plain text files, where entries (rows) are delimited by Enter (Line Feed or \n in this case) and values on each rows are delimited by a special characters (CSV uses comma or semicolon, I prefer Tabs).
I noticed I was often quite lazy when writing data retrievals from database and instead of abstraction I used a lot of SQL code. I know this is a bad habit and I should use abstraction or ORM, but in this stage I prefer quickness over a nice'n'clean code (and I'm still balancing the most suitable object structure for datastore at the moment). I'm going to review the whole thing anyways. For this reason I use quite a lot inline CSS in HTML as well, it helps me to organize my CSS classes better afterwards.
Knowing this, I created a SQL parser for each of CRUD queries using regular expressions. Then I converted results into arrays and processed them over data. The most challenging one was a WHERE clause, which currently supports only AND operator (not OR). In all cases I consider the first column as an auto-increment primary key.
Aside I created routines for data retrieval and saving, I used multidimensional associative array for in-memory data storage. Because I wanted to keep the each table in a single file and for purpose of QetriX I don't care about data types, first line of each file will contain tab-separated column names. Later I found out that I'll be able to put some additional info about each column here too. Because column name mostly doesn't contain a space (in my case never), I can use space as a 2nd level separator. This way I can define primary key, default value (incl. auto increment, but with limitation I can't use spaces in strings) and even data types.
All data modifications I make on in-memory data and store changed structure into a file afterwards. If I want to modify anything, the use case is: load data > make changes > save data. There are some application logic for performance improvements, obviously.
At first I was somehow distrustful, because it worked weirdly good :) But later I discovered this was foul and gained a trust. It works really good, it's very simple and reliable.
As everything around QetriX, this solution didn't aim to completely replace database. It lacks most features of e.g. MySQL, it's just a workaround for purposes I don't want to use database and target data scope is rather small. Perfect for FREE account, where user can store 5 databases with 5 classifications and 5 attribute/relation types for each classification.
Flat files are plain text files, where entries (rows) are delimited by Enter (Line Feed or \n in this case) and values on each rows are delimited by a special characters (CSV uses comma or semicolon, I prefer Tabs).
I noticed I was often quite lazy when writing data retrievals from database and instead of abstraction I used a lot of SQL code. I know this is a bad habit and I should use abstraction or ORM, but in this stage I prefer quickness over a nice'n'clean code (and I'm still balancing the most suitable object structure for datastore at the moment). I'm going to review the whole thing anyways. For this reason I use quite a lot inline CSS in HTML as well, it helps me to organize my CSS classes better afterwards.
Knowing this, I created a SQL parser for each of CRUD queries using regular expressions. Then I converted results into arrays and processed them over data. The most challenging one was a WHERE clause, which currently supports only AND operator (not OR). In all cases I consider the first column as an auto-increment primary key.
Aside I created routines for data retrieval and saving, I used multidimensional associative array for in-memory data storage. Because I wanted to keep the each table in a single file and for purpose of QetriX I don't care about data types, first line of each file will contain tab-separated column names. Later I found out that I'll be able to put some additional info about each column here too. Because column name mostly doesn't contain a space (in my case never), I can use space as a 2nd level separator. This way I can define primary key, default value (incl. auto increment, but with limitation I can't use spaces in strings) and even data types.
All data modifications I make on in-memory data and store changed structure into a file afterwards. If I want to modify anything, the use case is: load data > make changes > save data. There are some application logic for performance improvements, obviously.
At first I was somehow distrustful, because it worked weirdly good :) But later I discovered this was foul and gained a trust. It works really good, it's very simple and reliable.
As everything around QetriX, this solution didn't aim to completely replace database. It lacks most features of e.g. MySQL, it's just a workaround for purposes I don't want to use database and target data scope is rather small. Perfect for FREE account, where user can store 5 databases with 5 classifications and 5 attribute/relation types for each classification.
Monday, April 8, 2013
Setting the main focus
Even I call the Particle Database "The Database of Everything", I had to set some boundaries and rules. I want to store there anything, what more than 100 different people can use and changes are not too often (better - never).
For weather it may be max/avg/min temperature for a day and month, but not precise temperature for specific date and time.
Similar situation is with currencies, we can update current exchange rate each day, but history has to go somewhere else. Table "ph" (particle history) is a good choice.
The best kind of data is precisely stated in Wikidi headline: places, people, products. We add parties.
Places are countries, further geographical/political divisions (counties, neighbourhoods), cities as well as villages, streets, buildings, bridges, lakes, rivers and creeks, roads, railway routes, public transportation lines and stops/stations, points of interest (restaurants, various premises and services) etc. Anything you can assign fixed coordinates or a set of coordinates to.
People are obvious, but with humongous respect to privacy! If it's not generally known, we don't want to file it.
Products are anything you can buy in any kind of store. We prefer if it has a barcode (EAN). So it can be groceries, office supplies, furniture (IKEA is a great example), electronics, appliances, cars, gardening tools, fashion, brands etc.
Parties are groups of people, so it could be companies, politic parties, ethnic groups, religions etc.
For weather it may be max/avg/min temperature for a day and month, but not precise temperature for specific date and time.
Similar situation is with currencies, we can update current exchange rate each day, but history has to go somewhere else. Table "ph" (particle history) is a good choice.
The best kind of data is precisely stated in Wikidi headline: places, people, products. We add parties.
Places are countries, further geographical/political divisions (counties, neighbourhoods), cities as well as villages, streets, buildings, bridges, lakes, rivers and creeks, roads, railway routes, public transportation lines and stops/stations, points of interest (restaurants, various premises and services) etc. Anything you can assign fixed coordinates or a set of coordinates to.
People are obvious, but with humongous respect to privacy! If it's not generally known, we don't want to file it.
Products are anything you can buy in any kind of store. We prefer if it has a barcode (EAN). So it can be groceries, office supplies, furniture (IKEA is a great example), electronics, appliances, cars, gardening tools, fashion, brands etc.
Parties are groups of people, so it could be companies, politic parties, ethnic groups, religions etc.
Saturday, March 30, 2013
Working with foreign tables
I consider this blog as an inspiration for others, not just as a promotion of QetriX ;-) so let me write a post about one of my former ideas for 3rd generation of my CMS (year 2007).
I was on a quest for automated database layer, so I came up with automated table joining without usage of "advanced" database features (designed for MyISAM engine).
From my former job I adopted a system of three-letter prefix for column names according to table name. The motivation for this was to have unique column names thruout the whole database schema. If I have table "users" (I use plural for table names), the prefix would be "usr", so primary key would be "usr_id". For PK in table "companies" it would be "com_id" and so on. If the table had similar name and the prefix was already taken, I'd use the next letter in order, so for "computers" along with "companies" it would be "com" and "cop" respectively.
For foreign keys I used even more specific approach, I used both table and prefix names from foreign table in the column name (plus "fk" suffix), so if user had relation to company, the column name in table "users" would be: "usr_companies_com_fk".
Thanks to this I was able to specify $db->getDataFrom("users") and it would return data from all joined tables as well. For this purpose I had cached database schema (like "information_schema.tables" in MySQL).
And that's it :)
Fun fact: The system was so automated, that when you deleted a row in one table, all referenced rows in all foreign tables has been deleted as well (incl. domains, users, locations etc). *palmface*
I was on a quest for automated database layer, so I came up with automated table joining without usage of "advanced" database features (designed for MyISAM engine).
From my former job I adopted a system of three-letter prefix for column names according to table name. The motivation for this was to have unique column names thruout the whole database schema. If I have table "users" (I use plural for table names), the prefix would be "usr", so primary key would be "usr_id". For PK in table "companies" it would be "com_id" and so on. If the table had similar name and the prefix was already taken, I'd use the next letter in order, so for "computers" along with "companies" it would be "com" and "cop" respectively.
For foreign keys I used even more specific approach, I used both table and prefix names from foreign table in the column name (plus "fk" suffix), so if user had relation to company, the column name in table "users" would be: "usr_companies_com_fk".
Thanks to this I was able to specify $db->getDataFrom("users") and it would return data from all joined tables as well. For this purpose I had cached database schema (like "information_schema.tables" in MySQL).
And that's it :)
Fun fact: The system was so automated, that when you deleted a row in one table, all referenced rows in all foreign tables has been deleted as well (incl. domains, users, locations etc). *palmface*
Monday, March 25, 2013
Pitching the idea
Like many developers, especially from my region, my crucial problem is to pitch my idea of the particle database. I crashed and burned many times before I realized I need to focus on just a few main features and don't try to explain the whole concept. But in this case, the main thing isn't that much exciting and everyone immediately sees i'm doing just an n-th copy of Freebase or something like that. Maybe I do, but not realized it yet...
However, I tried to sign up for startup event at local conference, but I didn't get their attention. At least the thought of I may get picked made me thinking more about what I do and how I can explain that to the crowd.
Then I tried to write few paragraphs about QetriX to local tech website's forum and got some responses. Fortunately all criticism was constructive and pushed me a leap forward. They mosťy pointed out that my UI is too complex and hard to understand, plus the database is quite empty. They got me thinking and ultimately I wrote a nice web crawler for semantic data extraction. I also redesigned the UI and dramatically reduced number of offered features, like comments or rating.
I slowly learned how to start simple and supress the urge to make everything perfect at once. Roll out is much more important than perfection. You can always add more features later. On the other hand, if you don't get user's attention in the beginning, you may not get a second chance from him. You have to determine your target group - geeks will demand features, but everyone else will demand simplicity. You can combine both, if you cleverly hide all the features and reveal them after the user is more experienced.
However, I tried to sign up for startup event at local conference, but I didn't get their attention. At least the thought of I may get picked made me thinking more about what I do and how I can explain that to the crowd.
Then I tried to write few paragraphs about QetriX to local tech website's forum and got some responses. Fortunately all criticism was constructive and pushed me a leap forward. They mosťy pointed out that my UI is too complex and hard to understand, plus the database is quite empty. They got me thinking and ultimately I wrote a nice web crawler for semantic data extraction. I also redesigned the UI and dramatically reduced number of offered features, like comments or rating.
I slowly learned how to start simple and supress the urge to make everything perfect at once. Roll out is much more important than perfection. You can always add more features later. On the other hand, if you don't get user's attention in the beginning, you may not get a second chance from him. You have to determine your target group - geeks will demand features, but everyone else will demand simplicity. You can combine both, if you cleverly hide all the features and reveal them after the user is more experienced.
Wednesday, March 20, 2013
Q-Logo
 |
| Sun Microsystems |
My requirements was quite clear:
- It should be stylized "Q",
- looks good (or at least is recognizible) as 16x16 px favicon,
- circle or square (not rectangular),
- bold lines, carvable for stickers,
- not too complex in vectors.
 |
| QeX (animated) |
I did some improvements, grooming and polishing, like determine width and thickness. The width would be 11 units, 1+3+3+3+1. The blue circle in the middle would have the same radius, as border of the outer blue rectangle; 1.5 units.
 |
 |
 |
| My brother's idea | My edit | Final version |
I really like it, no matter I keep hearing it looks like a Pac-Man ;-) I see a harddisk, which is great icon for data platform.
I tried to modify it over the time, for example it still bugs me that I didn't use golden ratio (which is quite hard to achieve in concentric design :) or that I didn't used guide circles better. But so far it always ended by rollback.
Friday, March 15, 2013
The Name
In 2001 we started our Internet forum, called "QeX". Letters Q and X has been chosen by my friend Pavel, because those two are the only letters in Czech language, which you can't pronounce as a single letter.
Domain "qx.cz" was already taken, but I remember we saw it available when we looked for the first time couple weeks before. So we decided to put "e" in between (for better pronunciation, to make a word out of it) and, as a remembrance of the original idea, we kept "Q" and "X" uppercase.
In 2006 I discovered a leetspeak and at that time I was looking for available short ".net" domain, which serves better for Internet stuff. I tried to convert "qex" into leet and I got "q3x". I tried different pronunciations and then it hit me: "Q-tri-X" - sounds like "Matrix". So I registered q3x.net. I saved this name for my emerging project.
One day I tried various speech plugins in different languages, like Google Translate or Text-to-Speech in Windows, and I found out, that transcription as "qetrix" sounded quite the same in all major languages, which was my wish for project name. Since then I used "qetrix" much more often.
I always wanted a ".com" domain, because it's the most common TLD and it's somehow the best choice for a new startup. I already pronounced "q3x" as "qetrix", and to my delight (but not a surprise) "qetrix" was TLD-wide available. It took me just a few secs to decide and I claimed generous amount of TLDs :)
I kept q3x.net a primary and all qetrix.* redirected to it. Almost a year later I decided to use "qetrix.com" as primary and everything else as redirect.
And this way the word "QetriX" was born.
Domain "qx.cz" was already taken, but I remember we saw it available when we looked for the first time couple weeks before. So we decided to put "e" in between (for better pronunciation, to make a word out of it) and, as a remembrance of the original idea, we kept "Q" and "X" uppercase.
In 2006 I discovered a leetspeak and at that time I was looking for available short ".net" domain, which serves better for Internet stuff. I tried to convert "qex" into leet and I got "q3x". I tried different pronunciations and then it hit me: "Q-tri-X" - sounds like "Matrix". So I registered q3x.net. I saved this name for my emerging project.
One day I tried various speech plugins in different languages, like Google Translate or Text-to-Speech in Windows, and I found out, that transcription as "qetrix" sounded quite the same in all major languages, which was my wish for project name. Since then I used "qetrix" much more often.
I always wanted a ".com" domain, because it's the most common TLD and it's somehow the best choice for a new startup. I already pronounced "q3x" as "qetrix", and to my delight (but not a surprise) "qetrix" was TLD-wide available. It took me just a few secs to decide and I claimed generous amount of TLDs :)
I kept q3x.net a primary and all qetrix.* redirected to it. Almost a year later I decided to use "qetrix.com" as primary and everything else as redirect.
And this way the word "QetriX" was born.
Monday, March 11, 2013
Everything makes sense now
I was quite happy about the total reengineering. I started in a brand new directory and created brand new scripts. It was totally new approach, but the puzzle finally clicked together.
My first neglected todo was renaming the key table. There was no reason for calling it "a", it was historical reason when everything was put into "attribute" table. Because I came up with the name "particle" for the data structure and I still like single-letter name for tables, I renamed "a" to "p". As a result I was unable to use anything I created so far right away, I had to dip into everything. And it was a good thing.
I knew everything will be either a list, detail or edit form. In my opinion this realization was the second most important invention for QetriX, after particle. I was able to stick with these three components and create a compact code, as I always wanted. It would require much more discipline from me to keep everything in order.
Even QetriX is more about methodology, the idea without implementation is quite worthless. So I decided I'll rebuild the system in PHP again, maybe later in C# .NET or it will be my case-study for Java, Android, iOS, HTML5 or whatever... :)
I designed a new object model. I don't like bloated object apps with giant code overhead, mostly because I didn't use advantages of OOP, like inheritation. All my code uses KISS approach (Keep It Simple, Stupid), so I didn't feel the urge to organize my functions into static methods. But I knew if I want to make the code open source one day, I need to abandon my 1990s coding habits :)
I was going to create something like framework, so for a few weeks I studied some of the most used - Nette, Zend, CodeIgniter, CakePHP, Symfony, FatFree (lightweight counterpart), plus MediaWiki and Wordpress. Closest "to my heart" was CodeIgniter.
I created URL parser (routing) first. It will combine class-method approach with URL rewriter (for particular entity, loaded from database). I established a basic set of rules for certain URL patterns for determining what part of URL will be detected as class, method, entity ID and parameters. If no class is found, system asks database for entity URL and if no correspoding URL is found, HTTP 404 is fired. And if URL contains /data/, system searches for corresponding file to offer for download.
After quite long and sometimes quite frustrating debug it works like a charm. I'm still thinking about reducing some lines in code though, I'll be happy to omit file_exists checks for certain cases as this code is executed all the time. On the other hand without this part working I was unable to continue in development, so eventually I sacrificed few milliseconds for sturdy and reliable code.
My first neglected todo was renaming the key table. There was no reason for calling it "a", it was historical reason when everything was put into "attribute" table. Because I came up with the name "particle" for the data structure and I still like single-letter name for tables, I renamed "a" to "p". As a result I was unable to use anything I created so far right away, I had to dip into everything. And it was a good thing.
I knew everything will be either a list, detail or edit form. In my opinion this realization was the second most important invention for QetriX, after particle. I was able to stick with these three components and create a compact code, as I always wanted. It would require much more discipline from me to keep everything in order.
Even QetriX is more about methodology, the idea without implementation is quite worthless. So I decided I'll rebuild the system in PHP again, maybe later in C# .NET or it will be my case-study for Java, Android, iOS, HTML5 or whatever... :)
I designed a new object model. I don't like bloated object apps with giant code overhead, mostly because I didn't use advantages of OOP, like inheritation. All my code uses KISS approach (Keep It Simple, Stupid), so I didn't feel the urge to organize my functions into static methods. But I knew if I want to make the code open source one day, I need to abandon my 1990s coding habits :)
I was going to create something like framework, so for a few weeks I studied some of the most used - Nette, Zend, CodeIgniter, CakePHP, Symfony, FatFree (lightweight counterpart), plus MediaWiki and Wordpress. Closest "to my heart" was CodeIgniter.
I created URL parser (routing) first. It will combine class-method approach with URL rewriter (for particular entity, loaded from database). I established a basic set of rules for certain URL patterns for determining what part of URL will be detected as class, method, entity ID and parameters. If no class is found, system asks database for entity URL and if no correspoding URL is found, HTTP 404 is fired. And if URL contains /data/, system searches for corresponding file to offer for download.
After quite long and sometimes quite frustrating debug it works like a charm. I'm still thinking about reducing some lines in code though, I'll be happy to omit file_exists checks for certain cases as this code is executed all the time. On the other hand without this part working I was unable to continue in development, so eventually I sacrificed few milliseconds for sturdy and reliable code.
Thursday, March 7, 2013
New Destinations
I was going to try the engine on fleet management and dispatching system. It was pretty much the target usage, so I was particularly happy about it.
First showcase was a small fiasco, because future users didn't like it. It was too different from their current program, which was slowly dying for several unimportant reasons. The platform was (and still is) very flexible, so I promised to bend it as far as I could to meet their requirements.
It took me about 6 months of non-full-time development and the progress was tremendous. I started as "Win95" and ended as "Win7". It took so long because I stumbled upon some major problems of initial design and I had to reengineer, twice(!). But much worse was, that end users needed, but didn't want the new system. They were happy with status quo and didn't want to make anything better or different. In fact, they requested to recreate their current system (Win32 app) exactly as it was.
I don't want to go into much detail, as it's irrelevant for this story. I want to write about changes this experience led to.
The most notable change was a new approach to generating a form. Since then only controls for value has been generated. Now controls are generated for relation, order and flag as well.
For datetime control I created extremely rich/smart, but quite buggy (I must admit) control. It was single input (textbox), but thanks to a lot of JavaScript around the control it checked, what part of datetime are you currently on and offered appropriate functionality. Like when you wrote "8" on "day" part, it knew there are up to 31 days and "8" is the final day, so it skipped to "month" part automatically (in Czech Republic we use "d.m.yyyy" date format). On the other hand, if you wrote "2" on the day part, you stayd on the day part, because the control didn't know, if you wanted to write just "2", or e.g. "24". And there are more such things.
For number control I allowed user to change value using up-down arrows and because on Czech keyboard you have to press Shift to write number, I checked for the other without-Shift characters and convert them into appropriate numbers.
I also had to think more about more functionality, like "suggest" for values, drawing tables, forms, pages etc. I noticed some repeating patterns, but unfortunately I didn't have time for deeper analysis. But I kept it in mind.
The project finally didn't happened, even after I finished the last request and killed the last reported bug. It wasn't so big for the hassle, I already built an intranet for our company on it, which saves a lot of my coworkers' time. After I dropped this load off my shoulders, I was able to dust off the knowledge for the final reengineering. But this time absolutely from scratch.
First showcase was a small fiasco, because future users didn't like it. It was too different from their current program, which was slowly dying for several unimportant reasons. The platform was (and still is) very flexible, so I promised to bend it as far as I could to meet their requirements.
It took me about 6 months of non-full-time development and the progress was tremendous. I started as "Win95" and ended as "Win7". It took so long because I stumbled upon some major problems of initial design and I had to reengineer, twice(!). But much worse was, that end users needed, but didn't want the new system. They were happy with status quo and didn't want to make anything better or different. In fact, they requested to recreate their current system (Win32 app) exactly as it was.
I don't want to go into much detail, as it's irrelevant for this story. I want to write about changes this experience led to.
The most notable change was a new approach to generating a form. Since then only controls for value has been generated. Now controls are generated for relation, order and flag as well.
For datetime control I created extremely rich/smart, but quite buggy (I must admit) control. It was single input (textbox), but thanks to a lot of JavaScript around the control it checked, what part of datetime are you currently on and offered appropriate functionality. Like when you wrote "8" on "day" part, it knew there are up to 31 days and "8" is the final day, so it skipped to "month" part automatically (in Czech Republic we use "d.m.yyyy" date format). On the other hand, if you wrote "2" on the day part, you stayd on the day part, because the control didn't know, if you wanted to write just "2", or e.g. "24". And there are more such things.
For number control I allowed user to change value using up-down arrows and because on Czech keyboard you have to press Shift to write number, I checked for the other without-Shift characters and convert them into appropriate numbers.
I also had to think more about more functionality, like "suggest" for values, drawing tables, forms, pages etc. I noticed some repeating patterns, but unfortunately I didn't have time for deeper analysis. But I kept it in mind.
The project finally didn't happened, even after I finished the last request and killed the last reported bug. It wasn't so big for the hassle, I already built an intranet for our company on it, which saves a lot of my coworkers' time. After I dropped this load off my shoulders, I was able to dust off the knowledge for the final reengineering. But this time absolutely from scratch.
Sunday, March 3, 2013
Birth of QetriX
So I was going to create an ultimate database model and try the second option, mentioned in one of my previous blog posts. Now I was skilled enough to see much further ahead and avoid any performance issues in advance.
Thanks to the decision I reduced the model from 18 tables to 6. But I'm getting ahead of myself.
I was fairly familiar with parent-child approach, utilizing hierarchical database model, so I decided to use it. Each row has "parent_fk" column; attribs and relations will point to entity's primary key there and if the parent_fk is the same, asi primary key, it's an entity.
For value I defined "Twitter's" varcahr(140) column / domain. Longer values would go to a different table, when needed.
Then it was time to deal with relations. My first idea was to put foreign key into the varchar column for attrib value, but it was no-go even for my anarchistic side, not mentioning it would give many invalid relations (with numeric values) and sooo many invalid JOINs (esp. with text values). Adding new numeric column to the table was inevitable, but later I realized it was a pivotal moment.
From some experiments emerged it would be useful to add order and flag/strenght/significance. I tried to combine it into single numeric column, but it wasn't safe and might collide in some cases. To keep the model compact I didn't want to add two columns, so I made a Solomon's decision and created decimal column, where integer part would be significance and fractional part order.
In MySQL I used FLOAT for this column, but later I found out, when I use a datetime in numeric format (yyyymmddhhmmss) as order, the precision isn't sufficient and the number gets crooked. After some testing I redesigned the datatype as DOUBLE(18,14). I figured I wouldn't need more than 4 digits for significance.
After I finished the "data" table, I moved to "type" table. Design of its structure was just about what I expected from the system. I had parent type, type and format od value (text, number, url, isbn, date, zip...), order in entity, max allowed occurencies in entity, units (for converting), max length of value etc.
A saw a major flaw - entities can't share the same attribute type, each entity type allows only own attribs. This led to a new table, but this one was fairly useful, so I didn't mind. After all, it was the third one and I made i optional.
I used this table for multiple purposes. Except for reason mentioned above, after I added a column for a relation, I was able to define a list of entity types, to only which the particular relation can point.
My employer gave me a golden opportunity to use my data platform on a real project, which was a fleet management system, and I was ready to take the challenge.
Thanks to the decision I reduced the model from 18 tables to 6. But I'm getting ahead of myself.
I was fairly familiar with parent-child approach, utilizing hierarchical database model, so I decided to use it. Each row has "parent_fk" column; attribs and relations will point to entity's primary key there and if the parent_fk is the same, asi primary key, it's an entity.
For value I defined "Twitter's" varcahr(140) column / domain. Longer values would go to a different table, when needed.
Then it was time to deal with relations. My first idea was to put foreign key into the varchar column for attrib value, but it was no-go even for my anarchistic side, not mentioning it would give many invalid relations (with numeric values) and sooo many invalid JOINs (esp. with text values). Adding new numeric column to the table was inevitable, but later I realized it was a pivotal moment.
From some experiments emerged it would be useful to add order and flag/strenght/significance. I tried to combine it into single numeric column, but it wasn't safe and might collide in some cases. To keep the model compact I didn't want to add two columns, so I made a Solomon's decision and created decimal column, where integer part would be significance and fractional part order.
In MySQL I used FLOAT for this column, but later I found out, when I use a datetime in numeric format (yyyymmddhhmmss) as order, the precision isn't sufficient and the number gets crooked. After some testing I redesigned the datatype as DOUBLE(18,14). I figured I wouldn't need more than 4 digits for significance.
After I finished the "data" table, I moved to "type" table. Design of its structure was just about what I expected from the system. I had parent type, type and format od value (text, number, url, isbn, date, zip...), order in entity, max allowed occurencies in entity, units (for converting), max length of value etc.
A saw a major flaw - entities can't share the same attribute type, each entity type allows only own attribs. This led to a new table, but this one was fairly useful, so I didn't mind. After all, it was the third one and I made i optional.
I used this table for multiple purposes. Except for reason mentioned above, after I added a column for a relation, I was able to define a list of entity types, to only which the particular relation can point.
My employer gave me a golden opportunity to use my data platform on a real project, which was a fleet management system, and I was ready to take the challenge.
Tuesday, February 26, 2013
Filling the database
Alpha and Omega of website, like QetriX Particle Database, is data, data, data.
My primary goal was to import all countries of the world with major/capital cities and all settlements in the Czech Republic with all main attributes into the database. It was quite easy task and it filled about 60 000 rows (about 13 500 entities). I used mostly a spreadsheet (OOo Calc), because I downloaded and consolidated some Excel files.
Then I found list of all streets in Prague, all first and last names in the Czech Republic, I gathered list of all mobile phone models, some aircrafts and cars, a lot of series with episodes and finally I got a list of almost all airports in the world. Altogether it was about 400 000 rows.
I used custom made XML and TSV parsers. TSV is my favorite, because unlike commas, semicolons, pipes etc., tabs usually don't appear in data. Plus, when I copy-paste data thru clipboard from Calc to import front end, it's already as TSV - values separated by tabs. I had to create some custom made PHP scripts as well, for more complex or unevenly structured data sets.
One day I found some good keywords to find all kind of lists of Czech companies, so I put some of them into the database as well, along with some Czech POIs. But it was quite hard to find source of POIs without licence... There are great websites for POIs, but I don't steal. Anyway, 600 000 rows.
Then I found free to use sources of detailed structured data for particular aircrafts, like serial numbers, registrations, types, built dates etc. I love flying and I was always curious about the age of the aircraft I was about to embark. Not because I was scared, it was more about current state and what I can expect on board. Thanks to this I was able to create a simple app just for this :) 800 000 rows.
After that I had a lot of unfinished data sheets in Calc - taxonomy with animal species, administrative divisions and subdivisions of countries and cities, movies, songs etc. I was confident this is a good approach, because the data will be pure, complete to certain level and without any significant duplicities or mistakes. I also defended this position in some disputation about amount of data the system provides. Even I had some previous experience with web crawlers and data parsers, I didn't expect I'll have to use it here (rather I didn't want to use it here). Boy, I was wrong! :) The more I defended it, the more I thought about it and the more I understood this is the only way for larger number of data in database.
My primary goal was to import all countries of the world with major/capital cities and all settlements in the Czech Republic with all main attributes into the database. It was quite easy task and it filled about 60 000 rows (about 13 500 entities). I used mostly a spreadsheet (OOo Calc), because I downloaded and consolidated some Excel files.
Then I found list of all streets in Prague, all first and last names in the Czech Republic, I gathered list of all mobile phone models, some aircrafts and cars, a lot of series with episodes and finally I got a list of almost all airports in the world. Altogether it was about 400 000 rows.
I used custom made XML and TSV parsers. TSV is my favorite, because unlike commas, semicolons, pipes etc., tabs usually don't appear in data. Plus, when I copy-paste data thru clipboard from Calc to import front end, it's already as TSV - values separated by tabs. I had to create some custom made PHP scripts as well, for more complex or unevenly structured data sets.
One day I found some good keywords to find all kind of lists of Czech companies, so I put some of them into the database as well, along with some Czech POIs. But it was quite hard to find source of POIs without licence... There are great websites for POIs, but I don't steal. Anyway, 600 000 rows.
Then I found free to use sources of detailed structured data for particular aircrafts, like serial numbers, registrations, types, built dates etc. I love flying and I was always curious about the age of the aircraft I was about to embark. Not because I was scared, it was more about current state and what I can expect on board. Thanks to this I was able to create a simple app just for this :) 800 000 rows.
After that I had a lot of unfinished data sheets in Calc - taxonomy with animal species, administrative divisions and subdivisions of countries and cities, movies, songs etc. I was confident this is a good approach, because the data will be pure, complete to certain level and without any significant duplicities or mistakes. I also defended this position in some disputation about amount of data the system provides. Even I had some previous experience with web crawlers and data parsers, I didn't expect I'll have to use it here (rather I didn't want to use it here). Boy, I was wrong! :) The more I defended it, the more I thought about it and the more I understood this is the only way for larger number of data in database.
Thursday, February 21, 2013
Prehistory of QetriX
 |
| QX Logo |
1.x
 |
| QeX Logo |
After signing in, user stepped into "Gate" - home page, where all important stuff resided. From Gate he could go to list of "clubs" (chat rooms), or directly into particular club, where he discussed.
Early logo of QX/QeX we used later as "helper" (like Clippy in MS Office) and appeared in what one day should be a learning center or help center. This logo and the following one were both entirely Pavel's work. Next and final logo contained "e" as well, not just Q and X, and all those three letters were combined together into a head of hedgehog, or maybe a fox.
 |
| The only picture of QeX v1 I found, as part of our promotion banner. (2001) |
2.x
Version 2 was like a typical Internet forum. We had discussion clubs about certain topics, where users can write their posts. Any user was eligible to create own club and manage it afterwards. Each club might contain a poll, a "home" (separated HTML text page attached to club via link) and a "minihome" (HTML text above posts, limited size).Pavel introduced a concept of "sheets". Service status, charts, daily info, news, weather, currency exchange rates... It was nice and small, sometimes even smaller, than the text itself, with HTML tags around. And this way we were able to use gradients and everything looked exactly alike in all web browsers - we're talking about IE 5/6 and Netscape Navigator here... :) Everything was in table layout with very little of basic CSS.
Pavel also introduced "visit-cards", containing all important info about user, like nick, profile picture, current online status and timestamp of last activity. Those cards were like a widget, which people can put into their websites.
On Gate there was also a bulletin board and system news. We had a tiny IFRAME, that reloads every minute and checked for new stuff (like incoming mail or friends online), no global support for XMLHttpRequest (later better known as AJAX) those days... :)
 |
| QeX v2 - Gate. See sheets on the right. (2002) |
3.x
3rd version was released in 2003 and unlike previous 2 versions, this one was almost entirely my work. I recreated the whole engine from scratch, but still using vlibTemplate with Pavel's customizations.Instead of clubs we had modules with various interfaces now. Module was just a shell around its data and interface determined how to interpret the content (data). We had interfaces, like section (list of child modules), discussion, monologue (form of discussion, where only owner can post), chat, gallery, game, plugin, book (one long post per page) etc.
Users could store desired posts to personal memo and post private messages to other users. We didn't need home/minihome any more, because if required, user was able to create child text module for his discussion module with tight link in between. System went from "clubs are managed by admin" to "modules are managed by the user-creator and his fellows". User was able to customize the Gate as well. We opened modules for anonymous users as well, in read-only mode.
Modules were categorized into a tree. We had no tags or anything though, everything was managed by hand. But I created a smart method for drag'n'drop (in 2003 on web it was really something special), so it was quick'n'easy.
Pavel's most significant idea later was to have a type of every piece of content, instead of having different interfaces. It would allow us to have posts, pictures, games, polls, tables... side by side in one module. In 2004 it was for us quite hard to achieve, not only because of hardware issues, but mostly because of lack of our skills.
 |
| QeX v3 - Gate, registered user (2003) |
 |
| QeX v3 - Section, anonymous user (2003) |
 |
| QeX v3 - Module "Discussion" |
Technically, users and modules have been merged into modules. In database we kept table "users", which contains auth info (username, password) and user's module ID. We put email into "module" table, so user was able to insert content into module by e-mail.
As a side project we created "WiH" - "Where is He". Because there were many websites, like QeX, in order to be able to contact particular user it would require to know, on which server is he on right now.
Retrospectively I think v3.2 was the most advanced version of QeX. And influenced QetriX at most. Too bad later Pavel held back in development and I was ending just by myself. He was more underground and I was more commercial, so maybe he realized we missed the right time for such underground project and lost a motivation.
Unfortunately we experienced a major system outage with database crash. It took some time to restore it and I decided it was time for another reengineering. We lost most of our users anyway.
4.x
In v4 I split major sections into 3rd level domains and created true multiportal - each of those domains acted like separate website with unique look. The module system concept was brilliant, there was no need to change it.I have to admit there were some great new ideas for v4. We put homogenous data in modules into "windows" and we enhanced terminology from "module" to "node".
User had been able to specify permanent windows, block unwanted windows, define own windows or even call windows from other nodes. Windows had position and state, user was able e.g. to rellocate or minimize them. I designed an amazing desktop-like interface for our new "windows" concept.
For the first time the user interface was much cleaner, no system lists or menus. Everything was up to the user, because of windows.
 |
| QeX v4 - Gate (2005) with the concept of windows |
Aftermath
After the outage of v3 we had no regular users, so we kind of let the original idea die. I started to focus on money-making projects, like my CMS, and kept alive only my personal website on QeX. And because the "www" domain was empty and abandoned, I created a web portfolio on it.In the years after I used QeX in various different ways, but none of them went to reincarnation of QeX. Nevertheless I often realized our (well, mostly Pavel's ;-) ideas for QeX are a goldmine, so from time to time I go thru documents "ideas for v4" and look for anything I'd be able to use today. Like my CMS used the module system and in fact, QetriX uses it too - entity is a module, entity type is an interface :)
I still think I may resurrect QeX one day again. After I graduated I became an employee, so portfolio website is not valid and the domain is available for something new once again. The only downside is it's .cz and no other TLD is available. So maybe something with just local ambitions.
Sunday, February 17, 2013
DB schema on a diet
OK, so I had 9 tables just for data, out of 18 in total. That's 9 different data structures programmer/code would have to deal with during data manipulations (CRUD). That's quite different from what I see as universal. It may be right from database development side, but ingenious ideas often come just because its author didn't know or care he's doing it wrong. And my layout was already quite wrong by storing numeric values as a text.
I decided to analyze table structure to find some similarities, allowing me to merge those similar tables. I was quite sure I'll find something in "type" tables, so I started with them. But it wasn't as easy as I thought. At the moment it was just primary key and name, but I already had in my head I'd like to store more specific stuff in each of them, like data validations, max length or value type (text, numeric, date...) for attribs.
What bothered me more though were those three m:n tables - because I couldn't do anything about them, as each of them was unique. The only way was to merge table for entities, attribs and relations into one, but they were just too different!
One day I talked about it with my best friend and former co-developer Pavel. I also pointed out the potential, which we looked for when we were creating QeX (more about it later). He encouraged me to go ahead and try to merge those tables. So I did. And the power of QetriX was born.
I decided to analyze table structure to find some similarities, allowing me to merge those similar tables. I was quite sure I'll find something in "type" tables, so I started with them. But it wasn't as easy as I thought. At the moment it was just primary key and name, but I already had in my head I'd like to store more specific stuff in each of them, like data validations, max length or value type (text, numeric, date...) for attribs.
What bothered me more though were those three m:n tables - because I couldn't do anything about them, as each of them was unique. The only way was to merge table for entities, attribs and relations into one, but they were just too different!
One day I talked about it with my best friend and former co-developer Pavel. I also pointed out the potential, which we looked for when we were creating QeX (more about it later). He encouraged me to go ahead and try to merge those tables. So I did. And the power of QetriX was born.
Wednesday, February 13, 2013
Semantic, baby!
OK, so I decided to create a semantic database. From relation database schema I had the basics - there's a row, row can have some columns and they may store some row's own values, or they can store a reference to some other row (in the same or different table). Years later I determined exact names I'll use for it: row = entity, column = attribute, reference = relation. And yes, any resemblance to ER model is purely intentional ;-)
My database model was quite obvious at this point: entity, attrib, relation, entity-attrib (m:n), entity-relation (m:n).
Then I started to think about what structure those tables should have. I started with "data" tables - entity, attrib and relation.
First two columns were quite obvious - integer primary key (id) + tinyint some kind of "type". Now I need something for data.
My first thought was: One column for text value + one for numeric value. And maybe one for datetime. Such model is good for performance and indexing, but I didn't like the code have to decide all the time where to put or look for a value and if I'd like to have one column always NULL. Single varchar column for value would be much better, but then I'll lose the performance advantage. But what about decimal numbers? Or dual values, like coordinates?
It felt like a Sophie's choice and after quite long thinking of all pros and cons I decided to sacrifice performance. I was going to store everything as a text, but the model will be pure.
Then I started to expand my thoughts about the whole schema little more. Relations should be able to contain attribs as well. So another table for m:n relation emerges. And all rels, ents and attrs would have their own type as well... there we go, another 3 tables. That's 9 now and that's it!
Those tables are just for data, nothing else. Sure, pretty much everything in QetriX is "data", but I needed more tables for specific purposes. Like translations, large text storage, secure file storage, activity log, change log, statistics, feedback or various caches. The schema grew big once again.
My database model was quite obvious at this point: entity, attrib, relation, entity-attrib (m:n), entity-relation (m:n).
Then I started to think about what structure those tables should have. I started with "data" tables - entity, attrib and relation.
First two columns were quite obvious - integer primary key (id) + tinyint some kind of "type". Now I need something for data.
My first thought was: One column for text value + one for numeric value. And maybe one for datetime. Such model is good for performance and indexing, but I didn't like the code have to decide all the time where to put or look for a value and if I'd like to have one column always NULL. Single varchar column for value would be much better, but then I'll lose the performance advantage. But what about decimal numbers? Or dual values, like coordinates?
It felt like a Sophie's choice and after quite long thinking of all pros and cons I decided to sacrifice performance. I was going to store everything as a text, but the model will be pure.
Then I started to expand my thoughts about the whole schema little more. Relations should be able to contain attribs as well. So another table for m:n relation emerges. And all rels, ents and attrs would have their own type as well... there we go, another 3 tables. That's 9 now and that's it!
Those tables are just for data, nothing else. Sure, pretty much everything in QetriX is "data", but I needed more tables for specific purposes. Like translations, large text storage, secure file storage, activity log, change log, statistics, feedback or various caches. The schema grew big once again.
Saturday, February 9, 2013
Finding a way
I understand everything in the world is somehow interconnected. I can buy a candy bar, manufactured by some company, whose HQ building has been opened the same day I graduated. If I want to cover all this by the data platform, the model must be really versatile.
I can imagine two approaches: Extensive and extensible database model, or universal data structure. Because of performance reasons I decided to go with the first option. My "proof of concept" was a lightweight encyclopedia for mobile devices, called "Kilopedia" (max size of an article was 1 kB - hence the name).
In MySQL I created an "advanced database structure", where each entity type has its own table. Every column (domain) in a table stored a value, as an attribute, or a foreign key (reference to a row in different table), as a relation. I had tables like "city", "person", "phone", "airport", "car", "fruit", etc.
User interface worked closely with table structure and data types. When I rendered a form for data entry, I called DESCRIBE on the database table and used Field and Type columns to create HTML INPUT elements with appropriate data validations. When I wanted to add a new attribute to the entity type, I simply entered its name and system processed ALTER TABLE ADD COLUMN on background.
It worked like a charm, I was quite happy about it. It was nice, easy, fast, reliable and efficient. Until I noticed I have hundreds of tables in my schema, which is OK in general, but not for me :) All the time I was on a quest to create simple, clean and compact schema. The downside was it would require some unpopular database denormalizations, maybe have an universal table with a lot of NULL columns. It would be nice to have some of the columns as a number, some as a varchar, datetime, bool etc. I tried to design such table, but even before I started working on a prototype, I already knew this is a dead end. So I scratched the whole idea. But even a bad idea could move thinking towards the final solution, or you experience something, which you'll be able to use in the future.
And one more thing I didn't like about Kilopedia. Content of the articles was language dependent, without a chance to be automatically translated - like in Wikipedia, it would require to create separated text for each language, content may vary in different languages and every change would have to be done in all languages. For me this was unacceptable and I knew the article part must go away. I wanted all changes to happen at once, in all languages and by a single modification. Only semantic values will do the trick.
I can imagine two approaches: Extensive and extensible database model, or universal data structure. Because of performance reasons I decided to go with the first option. My "proof of concept" was a lightweight encyclopedia for mobile devices, called "Kilopedia" (max size of an article was 1 kB - hence the name).
In MySQL I created an "advanced database structure", where each entity type has its own table. Every column (domain) in a table stored a value, as an attribute, or a foreign key (reference to a row in different table), as a relation. I had tables like "city", "person", "phone", "airport", "car", "fruit", etc.
User interface worked closely with table structure and data types. When I rendered a form for data entry, I called DESCRIBE on the database table and used Field and Type columns to create HTML INPUT elements with appropriate data validations. When I wanted to add a new attribute to the entity type, I simply entered its name and system processed ALTER TABLE ADD COLUMN on background.
It worked like a charm, I was quite happy about it. It was nice, easy, fast, reliable and efficient. Until I noticed I have hundreds of tables in my schema, which is OK in general, but not for me :) All the time I was on a quest to create simple, clean and compact schema. The downside was it would require some unpopular database denormalizations, maybe have an universal table with a lot of NULL columns. It would be nice to have some of the columns as a number, some as a varchar, datetime, bool etc. I tried to design such table, but even before I started working on a prototype, I already knew this is a dead end. So I scratched the whole idea. But even a bad idea could move thinking towards the final solution, or you experience something, which you'll be able to use in the future.
And one more thing I didn't like about Kilopedia. Content of the articles was language dependent, without a chance to be automatically translated - like in Wikipedia, it would require to create separated text for each language, content may vary in different languages and every change would have to be done in all languages. For me this was unacceptable and I knew the article part must go away. I wanted all changes to happen at once, in all languages and by a single modification. Only semantic values will do the trick.
Wednesday, February 6, 2013
Hello, world!
I've just started a blog and a wiki. Wiki is rather formal and serves as documentation as well, while blog is going to be more informal and less technical. I plan to share my ideas, thoughts, future plans and current progress here, maybe some feature sneak peeks or backstage info.
Feel free to join the project on Facebook, Twitter or Google+ and stay tuned.
I wrote a blog post here and there all the time, even knowing my readership usually consists just of me and some web crawlers :) But it helps me sort thoughts, remind stuff and such. And even those posts may sound like product promotion, I write them mostly to try to inspire others and share my accomplishments.
Umm, and what the hell em I talking about? Well, QetriX is a result of my long-term desire to create ultimate yet simple semantic data platform. After couple of years (and some dead ends) I created final design of Particle, as an elementary data unit and a base of the platform.
Every particle has certain properties, which allows us to classify them into entities, their attributes and relations between them. In fact, it's a different approach to RDF, utilizing ER model (I created my own paradigm, but it was hard to describe/get the idea to others, so I moved towards better known stuff).
I started QetriX Particle Database even although there are quite a few online structured knowledge bases, such as Freebase (incl. derived projects, like Wikidi), Yago, DBpedia, Wikipedia (esp. Wikidata) or even Wolfram|Alpha, but none of them allows or encourages to file insignificant entities, like local businesses, groceries etc. Or they were hard/unfriendly to use. I saw a hole in the market, which I was able to fill - even if it would be just for my own needs.
I'll go under the hood in some of my future posts; I don't want to shoot out all my ammunition right away ;-)
If you want to comment any of my posts, go ahead and don't hesitate. Even it may be a while since the post has been published, I didn't have any audience for some months so there was nobody to comment :) I'm still following all my posts and sometimes I do some "refactoring" in them.
Subscribe to:
Comments (Atom)