Category Archives: Uncategorized

Modern Journalism, Politics, and Interpretation of Science

Posted on by
Reply

Recently there is news about the big break through in the research on the time line of the interbreeding between humans and Neanderthals. Since I am one of the unfortunate ones that happen to get the “Neanderthal” gene leading to diabetics, I am naturally very interested in such news.

Here is the original article from National Geographic: http://news.nationalgeographic.com/news/2014/10/141022-siberian-genome-ancient-science-discovery/

Here is quote from the original text:

“Genetically, the thigh bone’s owner appears equally related to modern-day Asians and Native Americans. Surprisingly, he appears to be about as closely related to them as to the 24,000-year-old Siberian boy or Stone Age European hunter-gatherers dated in other ancient DNA studies, notes paleoanthropologist Henry Harpending of the University of Utah in Salt Lake City, who was not part of the study.

Of modern groups, the Ust’-Ishim man is less closely related to Europeans, perhaps because today’s Europeans owe some of their ancestry to farmers who migrated there from the Middle East more than 10,000 years ago.”

When Fox News retweets about the story,
http://www.foxnews.com/science/2014/10/22/neanderthals-and-humans-first-mated-50000-years-ago-dna-reveals/

This is how they put it:

“Genetic analysis of DNA from the bone revealed this man was equally closely related to present-day Asians and to early Europeans. ”

Come on–Fox, the damned stupid bone was discovered in Siberia, the other side of the Caucasus Mountains, so isn’t it natural that it is more closely related to the East Asians than modern day Europeans? Why do you have to bend science like this–out of what? Some modern day inferiority complex?

Follow up on the Blood Sugar Test Strips

Posted on by
Reply

LifeScan’s customer service people are very efficient–they got back to me in just one day. After checking my personal information for 7 minutes, and then asking carefully about serial no of the blood sugar monitor and lot number of test strips, the first question they asked is, “what makes you want to test your blood sugar more than once?”

It is kind of funny, because at $1.5 per strip, I guess not many people can afford to do sanity checks like I did. Definitely not grandmas and grandpas who are on social security and medicare. And when your doctors/nurses/educators told you that you do need to test your blood sugar regularly and it is the best for you, who would doubt them? Even Mayo Clinic said so!Mayoclinic

With all the fuzz about biosensing, and all the advance in technologies, when can we have a more reliable method for testing blood? That will be a real ‘disrupting’ innovation–one that will not only destabilize a 174 Billion a year existing industry, but also benefit hundreds of millions of people around the world, including many people you know and maybe even your own grandma!

By the way, even though Mayo Clinic’s website never mentions how inaccurate these blood sugar tests are, American Association of Diabetes Educators clearly have done some research work, and here are the results they have dug up.

Here is a summary: even though that FDA only requires the glucose meters to show results within -25% to 25% of the true values for 95% of the times, most of these monitors fail to meet the standards in independent studies though they do get approved by the FDA. That means, for a person with a blood sugar of 100, it only requires the readings to be in the range of (75, 125) and they still fail it.

There is a new ISO standard coming requiring the monitors to be in the range for 99%, and FDA refuses to adopt it.

 

 

 

The Data Scientist’s Take on Dieting

Posted on by
Reply

I decided to take dieting the data science way and went out and got myself a blood sugar monitor, a One Touch Ultra Mini. Since some of my friends have been using One Touch for more than 10 years and really trust the brand, so I started to take tests without actually checking for the precision/error range of the equipment.

Definitely not the smart thing to do. In less than a week, I found something is seriously wrong with the readings I got. It started with one morning that I felt hungry only 3 hours after breakfast and decided to give my blood sugar a test.

The first number came out is 101, which was very suspicious. So my scientific training kicked in, and within 5 minutes, I took 6 readings from the tip of the same finger and got the following vastly different range of numbers:

10:52am 101
10:53am 88
10:55am 97
10:56am 96
10:57am 106
10:57am 95

The mean is 97.17, and the Standard deviation is only 5.52. Not that bad, LifeScan might argue. But the problem is that the monitor is one-reading only device. The range of the readings here is almost 18. If the number is 78, is number in the normal range as it suggested, or it could be 67 or 89? The first would mean the person is definitely going into hyperglycemia.

One the other hand, if the reading is 94, that number would mean that the person’s blood sugar is within the normal range, but adding the errors, the truth could be somewhere around 105, means he or she is definitely pre-diabetic.

By the law of large numbers, it is more likely that what I got is a normal blood sugar monitor than by the 0.1% almost negligible chance, it is a faulty one that skipped through all those supposed quality checks. So the conclusion we can draw here, is that, these blood sugar tests are extremely inaccurate.

Giving the amount of money that the health care system is spending on these test strips and monitors, I would rather say, that it is just a damn rip off by those pharmaceutical companies on the American public and the rest of the world.

Is MTurk for problems requiring years of training or special expertise?

Posted on by
Reply

We are trying to use Mechanical Turk to solve a problem that is different from most of the MTurk projects out there. First, to accomplish the task the Turker needs to have some ‘expertise’ in our subject domain. Second, we are trying to build a Machine Learning based model with very high precision requirement out of the data we collected from the Turkers. Instead of trying to find the model with the best precision/recall trade-off, we require our model to have very high precision (99.6%) and the best recall we can get for that precision level.

We are not sure yet whether MTurk is the way to go to get quality training/evaluation data. But we have been using it for almost two years. We spent a lot of money and efforts on collecting, cleaning and maintaining. So far it is somewhat meeting our expectation.

Here are some of the points:

1. Training and maintaining a population of Turkers with high accuracy:

We maintain continuous communications with the Turkers on TurkerNation. Whenever we set up a batch of HITs on MTurks, our researchers/analysts will monitor our TurkerNation board/thread to see if the Turkers have any questions about the HITs. If they do, we will try to answer the questions right away.

We salt our HITs with examples from our golden set, and use them to evaluate the Turkers.

We offer 100% bonus to Turkers who have accuracy of 100%. Most of the time none of the Turkers got 100% of the salted HITs correct. If this is the case, we always bonus the Turker with the highest accuracy by 100%. The rest of the Turkers will receive bonus based on their accuracy level. Usually we only bonus Turkers with accuracy higher than 80%, but if the batch is too hard, we may just bonus the top 20% or 30% of Turkers regardless of their precision.

After we evaluate each batch, we usually post messages about the accuracy information of each Turker and the amount of bonus they get on TurkerNation after each batch of HITs. The Turkers who read those posts can get an estimate of how well they perform compared to other Turkers.

To train the Turkers, we build and maintain a very detailed guideline page which takes some time to read, and edit it when our task changes or feedback from the Turkers suggests that we need to change and clarify our guideline. After a while, our guideline has become so long and it is very difficult for the Turkers to reference. To make it easier for the Turkers, in addition to the guideline pages, we create a set of example HITs with the correct answer and explanation. We add a link in the instruction section on our HITs, so the Turkers can go through them for reference very conveniently.

To further improve the accuracy of our Turkers, we expose a very powerful internal feature that we used in the model to the Turkers to help them make decisions for the ambiguous cases. This internal feature/score is so powerful, that some of the Turkers treat it as if it is the Golden Oracle, and are afraid of making decisions that are contradictory to the score. The pros: We did see a big increase in the accuracy of our Turkers. The cons: Sometimes the Turker just got lazy and made decisions by looking at that feature only. But most of the cases, our Turkers take our HITs very seriously and do take time working on the HITs very carefully.

2. A multiple tier labeling system

The first tier are the Turkers. Inside this tier we use a qualification score to separate them into Turkers and Super Turkers. Some of our jobs are only available to the Super Turkers. Usually we send out three types of jobs: a. Qualification jobs–these are offered to all Turkers who has an approval rate over 98%, and have done more than 500 HITs. These batches usually go parallel with our HITs for super Turkers. We use them to recruit more Super Turkers who do well on our HITs. These HITs are usually priced at 2 cent each. b. Super Turkers–these are the Turkers with qualification score over 8.  A majority of our HITs go to these Turkers. They pay 50% better than the ones for the general public at 3 cents. c. Super Super Turkers–Super Turkers who have done really well with us and with autoqual score over 25. We send some of our arbitration batches (batches of HITs that the super Turkers disagree on) to our Super Super Turkers, and pay them 5 cents for each HITs. These are only the base pay–in addition, there are always accuracy based bonuses.

The Super Turkers autoqual scores are adjusted based on their accuracy–the good ones will get an increase of 1 or 2 or even 3, and the bad ones will get an decrease of -1 or  -2. If their performance is really bad, we just disqualify them.

The second tier are our internal Data Raters. We hire people who usually have at least a bachelor degree to do the job on an hourly basis. They can work from home at any hours that are convenient to them and as many hours as they want. We bring them in for training when we see some big problems with their labeling results. The Data Raters work on two types of HITs: Hits the Super Turkers disagree on. We also put on MTurk the False Positives and False Negatives of our models and have the Data Raters work on them.

The third tier are our researchers, data analysts and QA team. These people go through the final false positives and false negatives together–usually many of these are very ambiguous cases–and discuss what the labels should be from their own perspectives. The things we learn from these sessions will become our new Label Guidelines and training examples.