Igor Chilingarian and Ivan Zolotukhin open their recent paper on the botteneck in scientific computing with:
Present-day astronomical instruments and large surveys produce the data flow increasing exponentially in time. The CPU power required to analyse these data is also growing with the same pace following the Moore’s law; the same applies to the data storage volume per price unit. However, in astronomy we do not see the exponential avalanche of scientific results produced with this computational power.
After providing good and bad examples of code in projects and discussing issues with bad code, they conclude:
It turns out that all “good examples” were developed either by professional astronomers with very strong IT/CS background or by IT/CS professionals working closely with astronomers for years and understanding astronomy. One cannot simply hire an industrial software engineer to develop astronomical software and/or an archive and/or a database.
A possible solution is to change the teaching paradigm for students in astronomy. Basic courses in algorithms, programming, software development and maintenance have to be made mandatory in the education of modern astronomers and physicists; advanced courses should be recommended to some of them. The Fortran language is now obsolete and we have to accept this. Instead of teaching research students to Fortran programming, one should teach how to interface legacy Fortran code in C/C++.
As soon as this bottleneck is resolved, the avalanche of discoveries will loom.
I agree. What to do with all the existing astronomers, though? I don’t think it is too late to learn good practices for those interested.
Link via Astrocompute.
February 24th, 2011 at 4:56 pm
Their opinion of Fortran smacks of programming language elitism.
February 24th, 2011 at 6:08 pm
Yes, I don’t think Fortran is “obsolete” (whatever that means), but should we be teaching new students to program in Fortran?
February 24th, 2011 at 10:27 pm
I’m with Piper on this. I’m not convinced there are any major issues with fortran… it still is faster than C++ for the few cases in which I’ve directly compared them.
As long as some formal teaching of good coding practices is included, I don’t really care what language is used, as long as it’s a practical one (i.e. something used by astronomers).
However, when you’re talking about database programming, I think it’s much more important to follow very strict coding practices, and therefore databases meant for fortran or IDL should be avoided. I disagree with some of their good examples, though – casjobs, for example, was the bane of my existence for 6 months.
February 25th, 2011 at 8:17 am
[…] This post was mentioned on Twitter by Meg Schwamb and AstroBetter, Orson Winkler. Orson Winkler said: michaelgalloy.com — Bottleneck in scientific computing: It turns out that all “good examples” were de… http://bit.ly/e3k0FN #astronomy […]
February 25th, 2011 at 8:37 am
I do not think Fortran is that bad. However anyone still teaching how to code in Fortran 77 should be hanged, drawn and quartered, to begin with. It is one of the worst designed language I have ever seen. However, Fortran 90 and later is an almost entirely different language which is decently palatable
February 25th, 2011 at 10:08 am
the elitism is less about FORTRAN and more about distinguishing the use of FORTRAN as an epitome of the clearly evident practice/belief that one need not evolve one’s computing skill set throughout ones career. FORTRAN gets used as the punching bag because the current practice is to train students using only the skills we had and our advisors had and …. and that is FORTRAN.
one part of the argument about formalizing training in scientific computing is about the present day “data science” paradigm shift. which is probably true (i would argue it is true) and thus probably very important to the science. Yet the goals of formal training in the practice of scientific computing are to improve the science (whatever the science) and the reproducibility of that science as well as coupling the concepts of technological and scientific advancements such that “learning good [new] practices” in CS is no different than continuing to aggregate knowledge about your scientific field by reading the literature, using new observatories/cameras etc.
February 25th, 2011 at 11:48 am
I am always saying things similar to what is in that paper.
In my experience, the trouble is not really the choice of Fortran, but that often what people use is poorly coded F77. It is ignoring over 3 decades of developments, and without much consideration on how to write good code (that is, the code could be much better, even keeping to F77).
It is ok if somebody makes an informed decision to use Fortran. The problem is that many astronomers were never given the information on how to choose, and not even had a choice: typically astronomers are only taught (incomplete) F77, as if that was the only way to do it, and it was good forever.
I often note that astronomers typically are offered weak education in programming, while they spend many semesters learning, for instance, calculus. But when they work, much more often people spend a lot more time dealing with software than, say, solving differential equations by hand.
And while most recognize the need to keep up with new developments in astronomy, few seem to notice that computer science is making new things much faster than any other science, and that many of those developments could be very useful in astronomy.
February 25th, 2011 at 11:56 am
Another, similar, recent paper is
http://www.nature.com/news/2010/101013/full/467775a.html
Also, there is a recent special issue of Science:
http://www.sciencemag.org/site/special/data/