Aximsite.com Adds Review Engine - Should We?

[Jason Dunn]

<div class='os_post_top_link'><a href='http://www.aximsite.com/censura' target='_blank'>http://www.aximsite.com/censura</a><br /><br /></div>"With all the programs and accessories available for the Dell Axim and the PDA market, Buyers must rely on others experiences to make decisions on what to buy. With reviews done by Aximsite.com’s staff and the staff writers of other sites, the opinions can be somewhat biased by only one persons views. With that in mind, Aximsite.com has searched high and low for a solution to this problem. Chris Leckness, owner/webmaster of Aximsite, wanted to see something similar to Amazon or leading Electronic Retail Stores with user reviews and comments."<br /><br />Ok, I have to admit this make me a little jealous. :werenotworthy: I thought of this exact concept about a year ago and thought of how cool it would be to have as a feature on the site, but we never worked on it. Now that I see there's a commercial package I can buy for $149 a year, is this something you'd like to see here? I think there's a great natural synergy with software like this and a site with "thoughts" in the title. :mrgreen: What do you think? It wouldn't replace our awesome review team, but it would give us bredth and could easily become a killer resource. What do you think?

[crispeto]

I say the more resources the better.

[entropy1980]

what is the commercially available solution you speak of?

[Unreal32]

I'm all for it. One of the things I *used* to visit PDABuzz for was their extensive list of hardware reviews, so something like this would be a great addition (especially since I no longer visit PDABuzz!).

[jpaq]

Y E S

[Crystal Eitle]

Oooh! I love writing reviews. I say, go for it!

[Chris Spera]

Warning: The following is my opinion and does not in any way shape or form reflect the opinion of Jason Dunn, PPCThoughts Site Admins or Moderators. It is the experienced-based opinion of Christopher Spera, based on nearly 7 years of Professional, Paid, Writing Experiences.

I disagree (well someone had to start the great debate... :lol: ).

I write for 4 internet sites, including this one. (You'll see some of my work hit the site next month.) I've been freelancing for almost 7 years now; and while I agree, somewhat, in principle to the comment regarding bias or group-think, I have also attended the Shareware Industry Conference and spoken one-on-one with a great many shareware developers, from across many platforms.

They (Shareware Developers) are hungry for professional, published reviews. In fact, they'd almost kill for them. Professional Reviews used to be all over the Internet. Now, they are nearly impossible to find. According to many, the revenues gained by hosting professional reviews (whether compensated or non-compensated) don't warrant their hosting. They are generally too expensive to host because any advertising that may be placed on them doesn't get viewed. Professional reviews are, according to many of my collegues, too expensive too host.

User reviews on the other hand, are easy to host, don't take a lot of space, don't require the hosting of any graphics or screen shots; and most everyone will unfortunately read them.

User reviews tend to be hot and cold, black and white; and don't, in my humble opinion, offer much value. From my experience as a freelancer and from what I learned at last year's SIC, Developers want professional reviews. They want indepth analysis and comparisons. They want concrete, specific suggestions regarding implemented features. They want professional reviewers to tell them where there applications succeed as well as fail. They want their documentation read, if possible; or at the very least skimmed and constructive comments offered on the entire package. User reviews rarely, if EVER offer this kind of indepth analysis or information, and for the most part contain 3-5 sentences. (and many times, the word "SUCKS" comes to play in there somewhere...)

They can however, have a HUGE negative impact on sales, and that worries me. Many of the problems encountered by individuals that post negative reviews can be traced to, in my experience, user error. End users tend to just pop the top on an application without reading any of the installation instructions, Read Me's or other important documentation. They download and install in many casees, all in one step; and can in the blink of an eye screw up their computer, their Pocket PC, or both. They then blame the developer for their own ignorance. For example, a backup or Restore Point (if they run XP or Windows ME) doesn't exist or didn't get made before they installed the app in question...and that's the developer's fault (according to the end user).

As Sr. Content Editor for WUGNET.com, I can't tell you how many times I've been asked to help users unscrew their computers because they either installed an app that modified core OS DLL's or applied updates to apps that access core DLL's. Once they bump into these problems, they tend to blame the last app they installed and not themselves for failing to read the documentation the author provided.

Personally, I think PPC Thoughts has the best implementation of the correct solution; though many other sites have a similar model. Let the Review Team review the app and then let the world BLOG themselves silly commenting on it. Anyone and everyone can either support or refute the findings of the professional styled review, and readers still get the same effect as the User Review scenario; but with some substance and legitimate analysis provided by an individual whom Jason has cleared as a qualified member of the PPCThoughts Review Team.

This is the great debate, guys. Sites don't (necessarily) want to host professional reviews due to bandwidth, cost and overhead associated with them. Developers want professional, well-written reviews because they offer constructive criticism that positively effect sales. One, unfortunately cancels out the other...


Again, this is just my opinion, based upon many years of professional writing experience; and is not a rant or flame in any sense of the word.

To Jason, I would most respectfully and humbly recommend that we continue with the current model that the site has: The one that has made it the success it is today.

However, I welcome the debate that will undoubtedly come after I click the Submit Button...

Kind Regards,


Christopher Spera[/b]

[rfischer]

VERY cool. I'd even donate a couple bucks via PayPal for that! :D

[James Fee]

Oooh! I love writing reviews. I say, go for it!
Aximsite.com has a review contest...

http://www.aximsite.com/showarticle.php?s=&threadid=8720

All that is required is that you are a member of the site and you review software that works on the Dell Axim.

[Bob Anderson]

Information is power.

Consumers need power -- because the only say we can have is with our checkbook.

Therefore, I fully support the concept of user based ratings. I think that professional reviews are needed as well, but professionals can't always see and/or test everything!

Even some of the self-proclaimed professional reviewers, like MSNBC's Gary Krakow or the Wall Street Journal's Walt Mossberg, don't seem to cover all the territory, or have glaring ommissions, -- and sometimes their professional reviews seem a bit amateurish. So why not let the common Joe jump in with his/her two cents worth? I can't see how it can hurt.

And like at least one previous poster here... I'm willing to send a few bucks via PayPal to help finance the idea.

Jason... thanks for asking!

[lurch]

After reading Chris' post, I'm split.

On the one hand, I'm not developing these apps, I'm using them, so therefore I don't care what the developers want, I want to know what other people's experiences are with them, known problems, known successes, etc.. and I'll be the one to filter out what's user error and what's not.

But on the other hand, I do really like the current model of a real-deal review and then opening it up to posts so people can still express their views on it.

But on the FIRST hand, maybe at some point I'd like to do a more "professional-style" review and need a good place to post it, and would rather not post it in the same place as another review already done on it.

Hmm...

Here's the end result for me: I could go either way. :) Such a firm answer, I know...

[R K]

When I first read the title, I was thinking of a search engine that was made specifically for searching out reviews.
Imagine a database of products from different developers or manufacturers. Imagine clicking on an item and receiving options to read many different professional reviews on the item.
That's what I'd want.
I know that PocketPCThoughts receives a lot of mature readers, so I'm sure the "IT SUCKS" comments will be at a minimum, but I vote to disagree with the idea of a user-review database.

[Richard S]

I think I agree with Chris. While I like reading the opinion of individuals who have used the product at hand, I appreciate more the well thought out and extensive reviews that this site currently offers. If I want to see what folks think of the review, I read the discussion that follows. Too many times at CNET , for example, I have to wade through all sorts of inane comments just to get to some meat. I prefer when CNET has a review, and THEN I have the choice to read user opinions.

But hey, whatever works for the site admins. We're an intelligent enough group that I would expect most of the comments by folks to be fairly useful.

[darius779]

User reviews tend to be hot and cold, black and white; and don't, in my humble opinion, offer much value.

I agree with this in part, there will always be a place for more professional reviews and reviewers. However, when a larger group of users give their opinions, I value the combined result far more than any one person, professional or otherwise

[Chris Spera]

User reviews tend to be hot and cold, black and white; and don't, in my humble opinion, offer much value.

I agree with this in part, there will always be a place for more professional reviews and reviewers. However, when a larger group of users give their opinions, I value the combined result far more than any one person, professional or otherwise

I also agree with this in part. However, like someone else mentioned 2-3 posts above the one I'm quoting, you usually have to wade through a whole bunch of irrelevant commentary/ rants (read: noise) before you get to the meat that you're looking for.

I agree that if 50-100 users say something doesn't work then I'm going to back away from the app, however, I really have to wonder how many of those users installed the app correctly. I have to wonder how many of them read the installation instructions. CNet was one of the Members of the Press at last year's SIC in St. Louis.

As I recall, my good friend Max Green talked this up a lot because CNet was one of the pioneers of User Based Reviews. However, as a technically savvy CNet visitor, I really question the decision to tone down the indepth analysis that CNet is famous for (only in some cases, guys...not every time...) in favor of user based commentary.

In my opinion, you can't tone down professional reviews and offer users the chance to comment. You can't serve two masters: either go all the way to the wall with your professional reviews and then let users blog their comments; or let users blog their comments and have the professional reviewers moderate the blog.

HOWEVER..!! That's not a suggestion, and should not be taken as one for this site, (or any that I'm affiliated with... I won't moderate user based review blogs...I'm spread too thin as it is! :mrgreen: )

My point is that a site should do one or the other and not both. You either end up with skimpy professional reviews and a lot of people flaming the reviewer, or skimpy professional reviews and a lot of people flaming the reviewer...wait. Didn't I just say that?? 8O

Kind Regards,

Christopher Spera

[Paragon]

As Jason stated right off the top, there is no reason you can't do both.

The review team is an excellent concept. The members are for the most part everyday users that readers can relate to. They have been chosen for their ability to put together professional looking reviews that hit on everyday issues with the products they are reviewing. I think there are two very important reasons that this concept is working. First, there is a bond developing between the members of the team and the rest of the members here. The bond is building because there is a large amount of familiarity, and confidence in these people. They are outside the marketing "mill." Secondly, when a review is posted it is commented on in the forums. This means that not only do you get the reviewers views but you hear many other peoples views as well. This makes it much easier to make an informed buying decision that you can be confident of.

Having said all that doing something like the Aximsite is doing as well adds more voices, and opinions into the pot. It sure can't hurt.

Dave

[dh]

I have to agree with Paragon and say that ideally we would be able to read both the more professional reviews and also user ones. These would need to be set up in different areas of the site so that we can see clearly what type of review we are dealing with.

I certainly enjoy the very high quality of the reviews on PPCT, it would also be nice to be able to read views from a different perspective. (As of course does happen somewhat with the disscusions that take place in the forums after the reviews).

Other sites make very good use of user reviews, I'm thinking Mountain Bike Review, Amazon and others. I'm always interested to see what people think after actually forking out their money on a product. Like Chris pointed out, you do have to look through some junk to get to the real issues and yes the word "suck" does seem to be used a bit too much. Actually, my wife is in the process of trying to get a credit she is owed by US Cellular and I would for sure use the S word if reviewing their customer service.

To get back on topic, I think it would work really well, if user reviews could be posted right after the "pro review". Be good feedback for the developers as well to see if other users agree with the pro reviewers.

(just realized what a long sucky post this one is - sorry :drinking: )

[daveh]

I think professional and user reviews can coexist. The professional reviews on this site are exceptional. But, like most, I would love to see more.

I think that most of the regulars here bring more to the table than many of the professional reviews in the major pubs like the WSJ.

But for a review system to really be useful, there probably needs to be some type of reputation system in place.

[lurch]

But for a review system to really be useful, there probably needs to be some type of reputation system in place.
That's a great idea, in my mind... that way I could look at the user reviews, and perhaps sort by "reputation" or something to help filter out the junk? Or maybe only look at user reviews submitted by people with a certain level of rep.

[Janak Parekh]

But for a review system to really be useful, there probably needs to be some type of reputation system in place.
That's a great idea, in my mind... that way I could look at the user reviews, and perhaps sort by "reputation" or something to help filter out the junk? Or maybe only look at user reviews submitted by people with a certain level of rep.
epinions.com (www.epinions.com) has done a lot of original work in this. Unfortunately, most sites, like Amazon and C|Net, don't have any such support, and as a result you have to take their user reviews with a large grain of salt. Witness Handango reviews, for one.

--janak

[Jason Dunn]

Very interesting opinions everyone - keep them coming!

The problem with the reputation system is that it would need to be built into the system and I don't see it listed here (http://www.incursio.com/products/censura/index.html). Another issue with this is that I don't think we'd be able to tie the phpBB user database into the user database of this system without some heavy mods... 8O

[Paragon]

But for a review system to really be useful, there probably needs to be some type of reputation system in place.
That's a great idea, in my mind... that way I could look at the user reviews, and perhaps sort by "reputation" or something to help filter out the junk? Or maybe only look at user reviews submitted by people with a certain level of rep.

Isn't that exactly what is happening with the Thought's review team? With each review that one of them posts it builds their reputation...with you and me....The best way to build your own buyer confidence is to have confidence in the people who are advising you. The way that is done is by reading reviews from people you have become familiar with, and trust. You build their reputation in your mind. Reading reviews from people you are not familiar with, or are rated by people you are unfamiliar with doesn't build that confidence.....IMHO

Dave

[Chris Leckness]

I am sorry, I had to chime in. I am Chris of Aximsite.com, PocketpcThoughts.com used to be my favorite site (still is :wink: ) Jason does a super job here.

My opinion on the subject is simple, Once our engine gets so many "user" submitted reviews then there will be a baseline for a product from "users". Now I am no programmer and I am not really much more than a "user" myself, but I believe that software should be "user" proof. If 10 "users" review a product and 7 of 10 say they had troubles installing the software (instructions read or not) - Then I am going to believe that this software is not real "user" friendly.

I agree 95% on your thoughts (Chris) that "proffesional" reviews my uncover hidden trouble or test a product with a sharper eye. The other 5% seems to think that a "pro" reviewer also takes things the "user" may not know for granted.

As for a "user" review having a HUGE impact on sales... (If any of my advertisers are reading this, sorry - but it is my opinion) If a vendor/developer has a great product it will get great reviews and if it is an inferior product it will not. Something I am learning to deal with is that vendors do want their products reviewed but they do not want them to be candy coated. If a product is sub-par, a "pro" reviewer will mention it but tend to reference the short coming to something that is good for some, but not all....etc. All I have to do is remind myself often that I am not really selling anything.

The main point as Jason and others have mentioned is that they both SHOULD co-exsist. There is a pro opinion and multiple user opinions. I believe it will prove to be a super resource for my site and Jason's.

I do not personally mind the hosting costs of paid reviews as long as I do not pay hosting costs with my babies formula money...hehe.

&lt;&lt; --- Like Chris said earlier... these are not the opinions of Jason and his site or Aximsite as a whole.... Just mine as Aximsite's owner.... --- >>

BTW - Chris ! I am gonna be in Nashville in May, What is there fun to do there for a week?

[Janak Parekh]

The problem with the reputation system is that it would need to be built into the system and I don't see it listed here (http://www.incursio.com/products/censura/index.html). Another issue with this is that I don't think we'd be able to tie the phpBB user database into the user database of this system without some heavy mods... 8O
One could probably create a database alongside the phpBB user database, so that when phpBB is upgraded you wouldn't break everything. However, it's definitely not a trivial job. Would be cool to happen, but you're talking a month or two of heavy development, I think.

--janak

[SassKwatch]

They (Shareware Developers) are hungry for professional, published reviews. In fact, they'd almost kill for them. Professional Reviews used to be all over the Internet.

A lot of this depends on one's definition of 'Professional'. Unfortunately, 'Professional' all too often equates only to 'Paid', and has little to do with 'Quality'. I've read far too many 'Professional' reviews that left me knowing as little about the reviewed product as I did before, and feeling as though the writer was simply compiling words to draw a paycheck.

Now that said, I would throw in the caveat that I wouldn't apply that criticism to anyone at this site. In fact, it *is* the very 'Professional' approach that keeps me coming back here.

So, in general, I guess I'm in agreement with the notion the current setup here works great as is. But one thing that makes that especially so is the 'Discuss This Story' link attached to each article/review. No matter how hard a 'professional' reviewer might try to do a thorough review, it's virtually impossible for one person to test a product in the myriad different ways a hundred different users might. And I frequently find additional useful info in the followup discussions.

I can't tell you how many times I've been asked to help users unscrew their computers because they either installed an app that modified core OS DLL's or applied updates to apps that access core DLL's.

And how many of those apps and/or updates provided *ANY* kind of warning they were replacing or 'updating' core DLL's. In my experience, that almost never happens. And for that, developers are just as guilty, IMO, as are those who never bother to read the install intstructions and/or readme files. Any app that significantly alters *core* DLL's should be questioned *VERY* closely.

[pschultz]

User reviews tend to be hot and cold, black and white; and don't, in my humble opinion, offer much value.

I agree with this in part, there will always be a place for more professional reviews and reviewers. However, when a larger group of users give their opinions, I value the combined result far more than any one person, professional or otherwise
That's what polls are for.

[pschultz]

I like the current system now: a professional writes a review and then users can write their opinions in the forums.

[Crystal Eitle]

I'm with PSchultz, Chris Spera, et al.

I've got an idea - what if you started a new forum category called "Member Reviews" or something like that? It could be a place where we post reviews, or link to reviews on our own sites, but it wouldn't be a major/prominent part of PPC Thoughts, just something we could avail ourselves of if we wanted. That way it wouldn't be much work for you guys, and we could have a specific place to put our reviews.

[pschultz]

User reviews tend to be hot and cold, black and white; and don't, in my humble opinion, offer much value.
I agree that if 50-100 users say something doesn't work then I'm going to back away from the app, however, I really have to wonder how many of those users installed the app correctly. I have to wonder how many of them read the installation instructions.

I agree. It's hard to tell if the user had a problem with the software or the user just didn't install the software correctly. Professional reviews offer so much more value and information than user reviews.

[hesse]

Hi folks! My name is Young and I am in charge of the Aximsite.com review team.

I believe inviting regular users to leave their opinions about related products should be the first priority of any user community site. Sure, those sites, including Aximsite.com, have select number of dedicated review editors and they spend a lot of time testing products. However, they never get the experience that beginners or casual users get. I've written many review articles and it is very hard to think like novice because I am not. I think I read all my articles at least 20 times and make modifications before I publish just to make them more understandable to new users. But I still fail a lot of times.

The best way to address this problem is to let regular users leave their own experiences and opinions about the product. Many of them are not as experienced as review editors so they follow different steps when they try out products. And results are always surprising and valuable to both other users as well as vendors/developers. I am a lead web developer and I am always shocked by how Quality Assurance folks find bugs that I'd never thought of.

If vendors and developers are not willing to listen to users, then they will eventually fail. Marketing will give them needed exposures to users. But I believe it is post-sales support that will determine their success in the end.

Aximsite.com currently has a separate Staff Review section (http://www.aximsite.com/reviews/) where articles written by review editors are hosted. At the end of each article, links are provided to invite readers to leave their own ratings and short reviews. This still gives our users professional opinions while giving them chances to leave their own words about the product.

Oh, BTW, I would like to thank Jason and all Thoughts staff for creating an excellent site. This site is THE source of PPC news for me!!

[Pony99CA]

First, I think it would be very useful to have a way to find reviews of products more easily, be they written by the Review Team or regular members. Anything that makes finding something easier is a good thing, as long as people can still comment on the reviews. A reputation system is a nice, but not necessary, feature.

Second, I don't buy the concept that "professional" reviews and "lay" reviews are mutually exclusive. Right now, anybody can post a review in the (hopefully) appropriate forum, and other users can comment on it. The only difference with reviews written by the Review Team is the home page exposure and the template they have for reviews.

Third, let's not call reviews here "professional" unless they truly are. I know that some of the reviewers (like Chris Spera) are professional reviewers, but I bet many of the Review Team are not. "Professional" should mean either the writer gets paid for the review (which the Review Team does not, although they may get to keep some of the products) or that the writer writes reviews a significant part of his life.

The reviewers here are volunteers, and I bet most of them have jobs that don't involve writing reviews. The reviewers did have to go through a screening process involving writing one review and do have a template to follow (I know, because I tried out for the review team), but that does not make them professionals (again, I know, because I wouldn't consider myself a professional reviewer even if I had made the team).

So, to summarize, let's have a place where reviews, both from the Review Team and regular users, can be located. Make sure people can still comment on the reviews. If you can add a reputation system for reviewers (or for posters in general), that's fine, but not necessary.

By the way, would the review database only host complete reviews, or would it allow just linking to other reviews? If I put a review on my site, it would be redundant to put it here, too (and a waste of Jason's disk space), but people might still want to know about it and comment on it.

Steve

[albsilva]

I agree... but with ONE CONDITION: the reviewer has to include the serial number of the device he is reviewing to ensure that he's really reviewing a device, and not just posting his biased opinion.

Alberto Silva

[Andy Whiteford]

I mostly agree with Chris's comments on this subject. I think the current format works well because you get a single review that is structured, had time spent on it and offers a concise conclusion. The readers can then ask any questions that they feel are pertinent and add more body to the overall impression of the review that perhaps was missed or felt at the time of reviewing that it was not worth mentioning.

My impression of user reviews is that while many are well written, a good percentage can be a first impression or do not focus on the correct elements that will give a reader the information they require to help them forumulate a buying decision. I have also read many user reviews for a product and the overall 'score' of the product ends up in the 30 - 70% range with many differing views making it hard to come to a definite conclusion.

The 'professional' reviews you get here should be very focused with a clear final impression of the merits of the product. While many people may not agree with the review, at least they can follow and see the opinion of the reviewer. We are not paid for these reviews so professional is a loose term but do spend a lot of time and effort on these reviews and try to cover the main aspects that will assist a buying decision. We are human though and may miss things and we also are one opinion so I think the follow up feedback is important to get opinion from a different angle or even just for people to say they agree with the review. One other thing to be wary of regarding user reviews is a quick look at a product that may undermine a 'professional' review. I don't mean to say that the professional review is better, however the product is picked by a member of the review team because they have the relevant experience or knowledge of the subject matter to review that product and they set aside the time to conduct a thorough evaluation. It would be a shame if impressions of a product were tarnished by a few less thorough reviews as soon as a product is released. You only need to look in the forums to see user reviews of a product the day they have received it. While some products are easy to cover in a day, some are not but people still want to be the first to post regarding it.

[lurch]

Isn't that exactly what is happening with the Thought's review team? With each review that one of them posts it builds their reputation...with you and me.... &lt;snip> You build their reputation in your mind. Reading reviews from people you are not familiar with, or are rated by people you are unfamiliar with doesn't build that confidence.....IMHO
While this is true, I was thinking also about someone who comes to the site for the very first time.. yes they can assume (as I've done) that the reviewers have been selected because they can actually do a good review, and even though I can think of a few ways a ratings system could be hijacked slightly to "ratings bomb" (like a google bomb) someone, it's nice to see a tangible indicator of how trustworthy someone's review is.

As for the difficulty behind integrating the review software into the user database for this site, according to censura's site they integrate easily with phpBB (which I'm assuming also means the user database).
As for the whole issue with censura not providing the functionality for user "reputations", that's valid. But as a web developer who loves PHP, that's more of a positive thing than a negative thing to me. :mrgreen:

[AZMark]

Professional or not every review is going to be biased, unless you just want to list the spec sheet of the product.

How often have we read "Professional Reviews" with statements like "The Pocket PC is just too complex compared to Palm" ... "Palm has many more applications available for it's platform." Not too mention a few overly biased Pocket PC reviews on this site. :)

That aside, it's just a Jason decision on what he sees this site becoming. At first I loved this site, cuz it was just a blog on this crazy Canadian who, like me, was trying to learn all the in's and out's of the Pocket PC's. Now it's mainly a news site with some great back end dicussions.

Attention: "My opinion coming next"

I think that if Jason sees this as the end all be all Pocket PC site, then he would have to open up the user community side of things, to attract an even larger audience. Eventually doing user reviews, chat rooms, downloads etc. Or live with the fact that sites like Aximsite will pop up and fill needs that aren't offered here, and continue to provide only very targeted information to a smaller hi-tech audience.

AZMark

[Jason Dunn]

I think that if Jason sees this as the end all be all Pocket PC site, then he would have to open up the user community side of things, to attract an even larger audience. Eventually doing user reviews, chat rooms, downloads etc. Or live with the fact that sites like Aximsite will pop up and fill needs that aren't offered here, and continue to provide only very targeted information to a smaller hi-tech audience.


Your analysis of this site was very interesting to read, although your conclusions about Pocket PC Thoughts being over "targeted" information to a "smaller hi-tech audience" is pretty amusing. :lol: I never would have described this site in that way in a million years...

[Crystal Eitle]

I think that if Jason sees this as the end all be all Pocket PC site, then he would have to open up the user community side of things, to attract an even larger audience. Eventually doing user reviews, chat rooms, downloads etc. Or live with the fact that sites like Aximsite will pop up and fill needs that aren't offered here, and continue to provide only very targeted information to a smaller hi-tech audience.


Your analysis of this site was very interesting to read, although your conclusions about Pocket PC Thoughts being over "targeted" information to a "smaller hi-tech audience" is pretty amusing. :lol: I never would have described this site in that way in a million years...

No kidding. When I first started visiting Pocket PC Thoughts I was totally ignorant about most tech stuff, yet I found the forums here to be very welcoming and helpful.

[bbarker]

Many of the problems encountered by individuals that post negative reviews can be traced to, in my experience, user error. End users tend to just pop the top on an application without reading any of the installation instructions, Read Me's or other important documentation. They download and install in many casees, all in one step; and can in the blink of an eye screw up their computer, their Pocket PC, or both. They then blame the developer for their own ignorance. For example, a backup or Restore Point (if they run XP or Windows ME) doesn't exist or didn't get made before they installed the app in question...and that's the developer's fault (according to the end user).
Chris, I know you aren't a software developer, but developers often have that orientation. They design and build software that they understand how to use and often they write documentation that makes it clear how to use the software. (Actually, much of the documentation is not clear, but that's another topic.)

Unfortunately, that's not how most people learn and use software. They do it exactly as you describe. Great software is designed with that understanding. Usability is built in. That requires usability testing. Even a lone developer can test usability, initially with paper prototyping. (Yes, I'm a software designer, but anyone can do this and it's very inexpensive or even free.)

When I have designed software I've always assumed that if a user has problems in the first few minutes, he or she will set aside the software, intending to return later when there's more time. But unless the user is highly motivated, later never comes and we've lost that customer. One reason is that many users assume the software is logical but they are too dumb or illiterate to understand it, and people don't like feeling that way.

The wise user who is frustrated by new software blames the software. That may result in a negative user review. The "I'm too dumb" review is useful as well, though, because it warns off other beginners who may have problems. Yes, it may all be user error in the view of the developer. But it's really designer and developer error when a user can't figure out the software. The developer wants the user to read instructions but can't count on it. How many of us have tried software but just haven't been able to get our minds around it? But I'll bet the developer understood it and couldn't figure out why we were so stupid.

User reviews, then, are good for both the user and the developer. They let potential users know what others experienced and they tell developers (the smart ones, at least) what they need to do to improve their software. This feedback is invaluable. In pre-Internet days a good developer would crave such feedback. It may be embarrassing to have it out there for everyone to see, but that can be avoided through low-fidelity prototyping and low-cost usability testing. It's like quality control: you can spend the time to do it before you release the software or you can let real users identify your bugs and damage your reputation. Usability works the same way.

A lot of this depends on one's definition of 'Professional'. Unfortunately, 'Professional' all too often equates only to 'Paid', and has little to do with 'Quality'. I've read far too many 'Professional' reviews that left me knowing as little about the reviewed product as I did before, and feeling as though the writer was simply compiling words to draw a paycheck.
Professional review can be very superficial. I have often received far better feedback from actual users than from professional reviews. One reason is that users are drawn to the software because they have a personal or professional need they hope it will meet. They use the software to solve a real problem and may have extensively exercised its various features. A professional may not even be a potential user of the software and can miss aspects that are important to real users. Or they may not get the software's intent and really get things wrong. Have you ever read a Pocket PC review written by someone who has always used a Palm?

When I've read professional reviews of software I've designed or managed, I've been surprised by the rare review that got it right. Seriously. My primary interest in such reviews was what kind of publicity they might give us, good or bad. User feedback, on the other hand, was a far better indicator of how well the software worked. User reviews may not be well-structured, well-written, comprehensive or fair, but they are from real users. This compensates for many shortcomings.

Let me quickly add that most "professional" reviews found on enthusiast sites are different from those found in magazines and such publications. On these sites the reviewers normally are not paid and are reviewing software they've selected the same way a user would: because they have an actual interest in the software. I put much more stake in these reviews because they better represent a user's experience.

I haven't looked theCensura (http://www.incursio.com/products/censura/index.html) reviewing tool Jason is considering, but it might provide tools that can help structure user reviews. For example, we would get more structured user evaluations if people were asked to rate a product's Ease of learning, Ease of Use, Performance, Value and Overall Quality. Yes, there is a potential to stack the deck. I don't know how to address that. A review from someone who has been a registered member for at least three months and posted 50 times might be more valuable than the opinion of someone who just happened to show up around the date a new product is released.

[AZMark]

Not too sure about "over" targeted, just happens to be the niche it's ended up in. Remember when you asked how many of us were developers?

I'm sure many of the new users find a great wealth of information on this site and get many great answers from the community. But most that stay and continue to comment seem on the higher end even for the Pocket PC community.

The fact that this board talks about vertical market products (fire, medical etc), developer products, but has very limited coverage of games, themes, and the like draws at least a more mature crowd. Hence the reason I'm here I guess. I think I would trust the reviews of most of the people that come and visit PPCT on a regular basis.

Obviously you wanted this site to be more than a blog, but trying to attract a larger audience and keeping the user community feeling is a tough line walk. Either way I'm sure most of us are along for the journey.

AZMark

[AndrewShuttleworth]

I think user input is a great idea and if done right is a great way to build up more valuable content. I would like to see an extensive software database (although www.pocketpcmag.com already does a good job) and extensive up-to-date comparisons of software in the same categories (e.g. dictionaries, PIMs, input methods, databases). I think the only way to keep this really comprehensive and up-to-date is a collaborative solution with users adding and editing the data.

Andrew