Reviewers, impartiality, and comparisons...

[albsilva]

Hi,
After reading the review and the messages in this forum, I hope that future SPB products review to be more impartial... :roll:

Well, let's hope that the gotchas are taken into account in the next revision...

Regards,
Alberto Silva

[Jason Dunn]

After reading the review and the messages in this forum, I hope that future SPB products review to be more impartial... :roll:

Are you accusing Philip or writing an impartial review because of my involvement with Spb? I can assure you I had no involvement in Philip's review, nor would I have banned it from publication if it was un-flattering to the product. Accusing a volunteer reviewer of writing an impartial review is a very serious thing, and not something that I take lightly.

If you're saying that because Philip didn't mention some of the gotchas that other people here did, has it occurred to you that people use software differently, and that Philip simply didn't need some of the features other people are requesting here? Accusing him of writing a biased review simply because he didn't predict the shortcomings that other people might find in the product is both rude and assinine because it implies that he did it on purpose with some sort of ulterior motive.

I expected more of you Alberto. :|

[Dom]

LOL. I reckon we're all impartial reviewers. If we like something we expect everybody else will. Also others would be hard pressed to convince us that something else is better. Human nature and all that ...

To be fair if a new personal organiser was reviewed then I would be wary of any conlcusions made if it did not contain comparisons with Agenda Fusion and Pocket Informant, but that again is just my feeling. Certainly in the automotive world you would be hard pressed to find a review of a car without comparisons being made to other cars in the same class.
Dom

PS I am 100% impartial as I wrote Mastersoft Money. It's my personal favourite but ... I am very biased and have simple needs :wink:

[albsilva]

Sorry Jason and Philip,

It didn't seem as i wanted it to :oops: .

I've already read here great independent reviews of SPB software, but this particular review, no matter of who's company is the product from, didn't seem so professional as other did, like the last one about streaming video, but that's a personal opinion, and should have been directed only to the author.

My sincere apologies :| ,
Alberto Silva

[Jason Dunn]

To be fair if a new personal organiser was reviewed then I would be wary of any conlcusions made if it did not contain comparisons with Agenda Fusion and Pocket Informant, but that again is just my feeling.

I disagree - comparative reviews are VERY hard to do (look at how long this review is (http://www.pocketpcthoughts.com/articles.php?action=expand,16729)), and it every product we reviewed had to be compared to everything else on the market, we'd perhaps publish one review every two weeks. A product should be reviewed on its own merits, and if the product is lacking on its own merits, then the reviewer should comnent on that. If the reviewer is currently using another competing product, a certainly amount of comparisons will be natural, but I'm not going to ask my review team to make every review a comparison review.

Regarding car reviews, you might find comments like "The Audi handles better than the Celica I drove last week...", but you won't find a point-by-point comparison of every aspect of the Audio to the Celica, unless it's a "Head to Head" review. Again, it's a matter of resources. Plus, why waste space talking about product ABC if the reader came to read your review of product XYZ?

[Dom]

Jason, like reviews my comments are only my opinion. I personally would feel disappointed if I went to a top site such as yours and found a glowing report about a "new" product only to later find that an older one did things not included in the "new" product. But, it is just my opinion. I obviously don't read the same car mags as you but "Car" magazine is regarded Worldwide as one of the best and it ALWAYS compares :wink:
Dom

[Dom]

"Spb Finance claims to be the most ambitious personal finance program ever attempted on the Pocket PC platform" - a claim like this surely merits a comparison ?

Dom

[petvas]

When reviewing a product it is inevitable that there should be a comparison... When a new app comes into a mature market, everybody needs to know how it compares. Everything else is not important. Of course there should be a feature overview but a comparison is everything... It doesn't have to be very detailed, just the major differences...

[Philip Colmer]

"Spb Finance claims to be the most ambitious personal finance program ever attempted on the Pocket PC platform" - a claim like this surely merits a comparison ?
But who is best placed to make the comparison? If you read the review carefully, you'll see that I don't make a judgement on the claim because I don't believe I am in a position to do so.

I don't claim to be an authority on every single personal finance package you can get for the Pocket PC. In fact, I don't really claim to be an authority on anything :-)

All I do claim is that I try to write comprehensive reviews that provide you, the reader, with more information than you would get by just visiting the product's web site. If I miss something out that you feel I should have included then that is the beauty of you being able to post feedback on the review!

--Philip

[Philip Colmer]

When reviewing a product it is inevitable that there should be a comparison...
But against what? Prior to writing this review, I'd used desktop Microsoft Money and Money for the Pocket PC. Does that mean I can't write a review on Spb Finance simply because all I could compare it against is Money? I did use my experience of Microsoft Money to help me to highlight areas of missing functionality such as reconciliation and the lack of support for regular transactions but I admit that I don't drill down into Money's reports at all, which is why it never occurred to me that that would be something that people wanted to do.

Comparisons are very subjective and personal. What I may think is important to me may be very unimportant to you, and vice-versa. That is why I don't like them and would prefer to give you hard, cold facts and allow you to make up your own mind.

If I mess up on the facts, though, I can only apologise and try harder next time :D

--Philip

[Jason Dunn]

I personally would feel disappointed if I went to a top site such as yours and found a glowing report about a "new" product only to later find that an older one did things not included in the "new" product.

But if the reviewer doesn't need that feature, and thus doesn't notice it's missing, why does that invalidate the review in your eyes? Let's say for example that I review a 52x CD-R and love it - it performs well, and I recommend it. But if it doesn't burn T@atto images like the Yamaha, does that make my review less valid because I didn't notice or care about the "missing" feature?

Everyone here seems to be missing the point that a review is just one person's opinion - it's neither right nor wrong, but rather one person looking at a piece of software and evaluating it based on their needs. Ultimately it's up to individual consumer to decide whether they should spend their dollars on the product. A review can help guide the person in their decision, but it's virtually impossible to write a review that will address every factor that every reader wants in the depth they desire. I've been doing this for years, and it's just not possible.

Besides Dom, I hardly think you're an impartial observer in this matter. :wink:

[Jacob]

When reviewing a product it is inevitable that there should be a comparison... When a new app comes into a mature market, everybody needs to know how it compares. Everything else is not important. Of course there should be a feature overview but a comparison is everything... It doesn't have to be very detailed, just the major differences...

Although I do see the benefit of having a comparison, I don't see why a comparison would be necessary for the review to be valid or good.

This review is good in that it covers the major features of the software and shows much of what it can do. It's enough to give anyone who reads it enough information to help them decide if the software is worth looking at and possibly evaluating.

[petvas]

So, is it better just to know that Spb Finance is a good software? This is not enough. I want to know if it's the best.How can you compare it? You can begin by letting us know of all features... Then you can mention if there is something missing.. or a feature that is unique to the product in review... and also name the competitors and the features of these products (bullets). I can then judge by my own...

[Jason Dunn]

When reviewing a product it is inevitable that there should be a comparison... When a new app comes into a mature market, everybody needs to know how it compares. Everything else is not important. Of course there should be a feature overview but a comparison is everything... It doesn't have to be very detailed, just the major differences...

Tell you what: YOU write a detailed comparison of Spb Finance, Cash Organizer, and MasterSoft Money, and I'll gladly publish it! Please contact me when you've completed it.

Comparisons only occur if one of these conditions are met:

1) The reviewer is a user of a competing product, and thus has a deep familiarity with how one product works, and can compare it to the other. But this has it's own dangers: look at how badly the Pocket PC does when it's reviewed by a Palm user! Familiarity with one usually results in confusion with the other, and an unfair and biased review. I'd much rather have a reviewer using something for the first time, never having used anything else like it, in order to get a fair and balanced opinion about the program or product in isolation.

2) The reviewer downloads, installs, and heavily uses each program for several weeks in order to get a feel for what each product is capable of. That's what Doug with with his digital wallet review, and it took a tremendous amount of time and effort. Philip had Spb Finance for about one week, so this type of comparison just wasn't possible.

What you fail to understand is that it's usually impossible to find one TRUE victor in "application wars" where applications are compared in details. Look at Pocket Informant and AgendaFusion - each have people saying it's the "best" product, but each product obviously appeals to different types of people.

When I was reviewing the iPAQ 2215, I compared it to the speed of other Pocket PCs because I had an easy tool do to so, but I didn't go over each feature and say "I like the SD slot on the 5450 better than the 2215, but I like the d-pad on the 2215 better than the d-pad on the 5450." Who gives a damn if the SD slot on the 5450? It's not like you can magically attach to the 2215!

I can't help but notice that all of the people complaining here have no published articles that I can think of, so perhaps they should write a few articles and see what it's like before complaining about the work of others...:roll:

[Jacob]

So, is it better just to know that Spb Finance is a good software? This is not enough. I want to know if it's the best.

I never said that it is "better" to just know that at all - read my post(again if need be). I said that the review was useful.

The problem with that is only YOU can say whether it's the best for YOU.

There are very few pieces of software out there where personal preference doesn't come into play.

This review was very useful to me because it told me that the features of this application made it worth me evaluating it no matter what features the competitors have.

I'm not saying better or worse. I'm saying that a review that lists the features and describes the software can be good and useful.

I'm also saying that a comparison is good and useful - and sure, I'd love to see a comparison of all the competitors. However that won't make me say that this review was bad.

[petvas]

I am not complaining, I am just commenting on this thread. I do not expect to find a winner from a comparison... I just want the basic features of all major competitors mentioned. Is that too much to ask? I know that most reviewers don't have enough time and I respect that. Of course that doesn't mean that I do not like the review.. A good idea is to mention the major competitors. Give the user the option...

[Jason Dunn]

So, is it better just to know that Spb Finance is a good software? This is not enough. I want to know if it's the best.

The only way for YOU to know what's BEST is for YOU to download, install, and test ALL the software yourself. Why are you expecting a volunteer reviewer to do the work for you?

I can make the claim "The iPAQ 2215 is the bets Pocket PC on the market!", but that claim is only valid for me, because it meets my needs the best. Someone else might need a feature the 2215 doesn't have, or is looking for something at a lower price point.

Why are you expecting something that's impossible?

[Jason Dunn]

Can we stick to the subject please. The way things are at PPC Thoughts: articles and reviews are linked to a forum in which people can comment and add to the info in the original article, thus allowing readers to get input from all sides. And we all use it, so what's the prob?

Good use of the comment feature: people talking about what they think of a program, their own opinions on competing products, what they'd like to see added to the product, etc.

Bad use of the comment feature: criticising a volunteer reviewer for not writing the review the way it "should" have been written.

I encourage the first, but will fiercely oppose the second every time it occurs.

[petvas]

So, is it better just to know that Spb Finance is a good software? This is not enough. I want to know if it's the best.

The only way for YOU to know what's BEST is for YOU to download, install, and test ALL the software yourself. Why are you expecting a volunteer reviewer to do the work for you?

I can make the claim "The iPAQ 2215 is the bets Pocket PC on the market!", but that claim is only valid for me, because it meets my needs the best. Someone else might need a feature the 2215 doesn't have, or is looking for something at a lower price point.

Why are you expecting something that's impossible?
Jason, I believe you misunderstood me. I never said that the review is bad. Ii just found it incomplete...
I do not expect from a volunteer to dedicate lots of hours in order to get everything right. I respect their time and appreciate their efforts. I am just commenting on how a review must be done (according to me of course!!). I do not expect from a reviewer to tell me which software is better, I expect to hear about the features competing applications have. Then I can decide easier... I do not expect a full head to head review. I never said that...

[Philip Colmer]

In a recent posting ...

Jason, I believe you misunderstood me. I never said that the review is bad. Ii just found it incomplete...
Earlier on in this thread:
You can begin by letting us know of all features...
I believe I did that, to the best of my ability. This is certainly an area of reviewing that I try very hard to made as complete as possible without ending up having written a manual on how to use the product :D

Then you can mention if there is something missing.. or a feature that is unique to the product in review...
This is where it starts to get trickier.

You can only do this if you've got a frame of reference, either to alternative products (for both missing features or unique features) or to expectations that you've got that the product doesn't meet (in the case of missing features).

In order to highlight a feature that is unique, the reviewer must know every single product in the same genre, and know them inside out as well since, by definition, unique means that it doesn't exist anywhere else. The last thing I would want to happen is to claim that a product I'm reviewing has a unique feature only to find hundreds of PPCT readers posting details of products that have that self-same feature :oops:

In order to mention if something is missing, the reviewer must have an expectation for a feature in the first place. I did do this for two areas of functionality that I found to be missing - reconciliation and support for regular transactions. I did not mention that you couldn't drill down in the reports because it was not my expectation to be able to do so.

and also name the competitors and the features of these products (bullets). I can then judge by my own...
At first glance, this seems like it could be a reasonable request. However, once again, it requires that the reviewer knows of all available products. It also dates the review because the moment a new product comes along, the list of products is no longer accurate.

Please don't misunderstand me - you have raised what you believe are perfectly reasonable points. However, I don't believe that they are practical to implement beyond the comments I've made above.

--Philip

[jkendrick]

So, is it better just to know that Spb Finance is a good software? This is not enough. I want to know if it's the best.How can you compare it? You can begin by letting us know of all features... Then you can mention if there is something missing.. or a feature that is unique to the product in review... and also name the competitors and the features of these products (bullets). I can then judge by my own...

I held back for a while without jumping into this thread but as a reviewer, writer for industry publications, and the author of a review of this very program I can't be silent anymore.

Ask any 10 people what one feature would make any program "the best" and you'll likely get 10 different answers. There is no best in any software genre. I use PocketInformant because it does things the way I want but I also believe Agenda Fusion is a great piece of software. Thousands of users swear by each program so how can there be a "best"?

Reviewers don't set out to explain every single detail about a new program because it can't be done. There is nothing gained of a detailed comparison of every single program in a given category unless the review is also a detailed review of all the programs. This is not even feasible for so many reasons.

I just finished an article that's a detailed review of two programs head to head and you know what? At the end of the day they're both great programs but every single reader of this article will feel one is better than the other. And many readers will choose program A and the others program B. And they all will feel their choice is the "best".

Take reviews as they are offered- overview of a program's operation and features to help the reader make a little better decision for themselves.

If you want someone to make software decisions FOR you I'll be happy to send you a list of "best" programs in each category if you promise to buy all of them and use them.

:D

[Steven Cedrone]

Thread split from here... (http://www.pocketpcthoughts.com/forums/viewtopic.php?t=17513&postdays=0&postorder=asc&start=0)

Steven Cedrone
Community Moderator

[Fishie]

After reading the review and the messages in this forum, I hope that future SPB products review to be more impartial... :roll:

Accusing a volunteer reviewer of writing an impartial review is a very serious thing, and not something that I take lightly.

:|


Hahahaha
Hahahaha
Hahahahahahahaaaaaaaaaaaaa

[griph]

Accusing a volunteer reviewer of writing an impartial review is a very serious thing, and not something that I take lightly. :|

Hahahaha
Hahahaha
Hahahahahahahaaaaaaaaaaaaa

Well that was a clear comment -
OK Jason meant to refer to someone writing a partial (biased) review, rather than an impartial review. An easy typo.

I have been sitting back on this thread, but frankly some of the absolute (please excuse the language) b****cks (edited by Author) being talked about here beggards belief! 8O I was always taught to never look a gift horse in the mouth - someone produces a very useful review, but because it doesn't meet everyone's disparate 'requirements' the minority label it as biased or inadequate! SIMPLY NOT ON!

Most software is available for trial download - I dont expect anyone to do my job for me and review the whole range of available products that do similar functions in order for me to assess the usefulness of a particular product that is being reviewed. The review is useful in that it gives a fairly detailed general view including the main available features in that package. PERIOD! It does NOT need to do anything more. If you want a comparison (rather than a review) then do it yourself - and as suggested by Jason (if it is any good) it might get published. In other words put up or shut up!

I use Cash Organiser, but I will give you an immediate comment on comparison - Cash Organiser usually costs $49.95 (discounted to $29.95), SPB Finance costs $19.95 - now why would anyone expect that they would both do the same job to the same level of detail! If SPB's software, from the review seems to do what you want it do do - download the trial and check it out. Otherwise look at other products.

I think that is enough on this subject! :wink:

[Zensbikeshop]

I've just started writing reviews for PPCW.net and have found it significantly more difficult than I expected. To write a review of a complex piece of software such as SPB Finance must of taken considerable time and effort.

The thought of comparitively reviewing multiple finance programs fills me with terror! An remember this was a review of SPB Finance not a group review.

Reviews are great but the only way to decide whether a program is right for you is to use it. One of the great things about Pocket PC Software is that there generally a trial version availble so you can try before you buy.

Reviews are by definition one person's opinion.

Getting to the point of this thread; I read the review and detected no bias (and I'm paranoid :wink: ). I have always found Jason to be exceptionally upfront and open about his work for SPB and have never felt that Thoughts has been influenced in any way.

[jkendrick]

If this review had been for a product by any other publisher this thread would not even exist. I was wondering how long it would take before people started trying to find a bias somewhere.

[bazza]

One can only give an unbiased opinion about things that do not interest one, which is no doubt the reason an unbiased opinion is always valueless. The man who sees both sides of a question is a man who sees absolutely nothing.
Oscar Wilde (1856 - 1900)

Where there is no exaggeration there is no love, and where there is no love there is no understanding. It is only about things that do not interest one, that one can give a really unbiased opinion. . .
Oscar Wilde (1856 - 1900) Anglo-Irish playwright, novelist

:wink:

:werenotworthy: