I think simple open-source tools for this are coming, but they are a far cry from what you would want professionally.

May I ask: Are you worried about mixing open with closed source for reasons of protecting your intellectual property?

A commercial solution to that problem is being offered by Black Duck Software but of course you have to pay. Their tools can recognize open source code in your closed projects. Is that your need? (I’m honestly curious about that market.) Thanks!

Yep, no value judgement or even attempted explanation of different comment densities. Without further work, this would only get us into hot water…

Naturally, we all want quality comments, not boilerplate aut0-generated ones. At present, “our” parser (which is ohcount, to be precise), does not do a semantic analysis as to the content of comments. It would be another good step to add that information.

Personally, I don’t think that after more filtering we’ll only end up with junk. Why? We have some indications, for example, commits of SLoC 1 typically come with 2 comment lines. These should be comment lines with real contents. And from other work we know that more than 10% of all commits are 1 SLoC commits… so there is good commenting going on in open source.

What I’m really interested in, to be determined in future work, is of course how much commenting is needed to keep a project thriving, for example, how much does it help to get new people on board.

I am always appalled when I read code — and I read a lot of it — about the amount of useless, crappy and often auto-generated comments that sneak in the code. IDEs are often to blame for that, together with poor developer education on the value of comments.
I much prefer well written and readable code with no comments, than a page a comment-junk laced code.

I wonder about how much of these comments really mean something, ie add something of value to the code comprehension.
One thing to consider in future studies could be trying to discern between potentially valuable and useless comments, such as copyrights/licenses notices, changelogs and comment-junk. For the junk part, the length of structured comments (ie javadoc, perdoc, pydoc, phpdoc, etc) may be useful: it is often very short and prefixed by doc tags .
My hunch: once you have discounted the boilerplate and junk, there is not much left.
Is that a problem? Most likely not.